2024-09-21T09:35:50.960 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T09:35:50.986 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T09:35:51.072 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913661 branch: main description: upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} email: ceph-qa@ceph.com first-half-sequence: - cephadm.shell: env: - sha1 mon.a: - ceph config set mgr mgr/cephadm/daemon_cache_timeout 60 - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps - echo wait for minority of mons to upgrade - while ! ceph mon versions | grep $sha1 && ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error; do sleep 2 ; done - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for majority of mons to upgrade - 'while ! ceph mon versions | grep $sha1 | egrep '': [23]'' && ceph orch upgrade status | jq ''.in_progress'' | grep true && ! ceph orch upgrade status | jq ''.message'' | grep Error; do sleep 2 ; done' - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for all mons to upgrade - 'while ! ceph mon versions | grep $sha1 | grep '': 3'' && ceph orch upgrade status | jq ''.in_progress'' | grep true && ! ceph orch upgrade status | jq ''.message'' | grep Error; do sleep 2 ; done' - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for half of osds to upgrade - 'while ! ceph osd versions | grep $sha1 | egrep '': [45678]'' && ceph orch upgrade status | jq ''.in_progress'' | grep true && ! ceph orch upgrade status | jq ''.message'' | grep Error; do sleep 2 ; done' - ceph orch upgrade pause - ceph orch ps - ceph orch ps - ceph versions first-half-tasks: - workunit: branch: quincy clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' first_in_suite: false job_id: '7913661' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'run basic import/export cli tests for rbd ' - desc: 'run randomized correctness test for rados operations generate write load with rados bench ' - desc: 'run basic cls tests for rbd ' - desc: 'run basic import/export cli tests for rbd ' - desc: 'librbd C and C++ api tests ' - desc: 'randomized correctness test for rados operations on a replicated pool, using only reads, writes, and deletes ' - desc: 'randomized correctness test for rados operations on a replicated pool with snapshot operations ' - desc: 'run basic import/export cli tests for rbd ' name: teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true create_rbd_pool: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - OSD_DOWN - mons down - mon down - MON_DOWN - out of quorum - PG_DEGRADED - Reduced data availability - Degraded data redundancy - OSDMAP_FLAGS - PG_AVAILABILITY sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: main sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 owner: scheduled_teuthology@teuthology priority: 850 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b second-half-sequence: sequential: - cephadm.shell: env: - sha1 mon.a: - ceph orch upgrade resume - sleep 60 - echo wait for upgrade to complete - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch ps ; ceph versions ; sleep 30 ; done - echo upgrade complete - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 second-half-tasks: - workunit: branch: quincy clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' seed: 5357 sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 sleep_before_teardown: 0 stress-tasks: - thrashosds: aggressive_pg_num_changes: false chance_force_recovery: 0 chance_pgnum_grow: 1 chance_pgpnum_fix: 1 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 chance_thrash_pg_upmap_items: 0 disable_objectstore_tool_tests: true timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - print: '**** done end radosbench.yaml' - workunit: branch: quincy clients: client.0: - cls/test_cls_rbd.sh - print: '**** done cls/test_cls_rbd.sh 5-workload' - workunit: branch: quincy clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' - workunit: branch: quincy clients: client.0: - rbd/test_librbd.sh env: RBD_FEATURES: '61' - print: '**** done rbd/test_librbd.sh 7-workload' - full_sequential: - rados: clients: - client.0 objects: 500 op_weights: delete: 10 read: 45 write: 45 ops: 4000 write_append_excl: false - print: '**** done rados/readwrite 5-workload' - full_sequential: - rados: clients: - client.0 objects: 50 op_weights: delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 write_append_excl: false - print: '**** done rados/snaps-few-objects 5-workload' subset: 31/32 suite: upgrade suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 targets: smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH/nS8HxX+6jysQ9xYY5WADeQAuQB/5WrLBqGvnS5sXRC0+z0j5RugofkcwGpUENk96Hwo84Q2BKfCprj2U9usA= smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPbqvGAKqj1W9P/iYa6lwEC+tEzoJ86G8udougnGe27n+zsPWYEuba8tyZwcet4Q28jqA66ZtuCZ4xe4cmAzdD8= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install nvmetcli nvme-cli -y - install: branch: quincy exclude_packages: - ceph-volume - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy - cephadm.shell: mon.a: - ceph fs volume create foo - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph.healthy: null - print: '**** upgrading first half of cluster, with stress ****' - parallel: - first-half-tasks - first-half-sequence - print: '**** done upgrading first half of cluster ****' - ceph.healthy: null - print: '**** applying stress + thrashing to mixed-version cluster ****' - parallel: - stress-tasks - ceph.healthy: null - print: '**** finishing upgrade ****' - parallel: - second-half-tasks - second-half-sequence - ceph.healthy: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_03:08:26 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T09:35:51.073 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa; will attempt to use it 2024-09-21T09:35:51.073 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks 2024-09-21T09:35:51.073 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T09:35:51.075 INFO:teuthology.task.internal:Checking packages... 2024-09-21T09:35:51.093 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '62da54b4b83a96ca9976b7e1d0c8272a564a2208' 2024-09-21T09:35:51.093 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:35:51.093 INFO:teuthology.packaging:ref: None 2024-09-21T09:35:51.093 INFO:teuthology.packaging:tag: None 2024-09-21T09:35:51.093 INFO:teuthology.packaging:branch: main 2024-09-21T09:35:51.093 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:35:51.094 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-09-21T09:35:51.333 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5092.g62da54b4 2024-09-21T09:35:51.334 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T09:35:51.339 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T09:35:51.340 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T09:35:51.360 INFO:teuthology.task.internal:Saving configuration 2024-09-21T09:35:51.383 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T09:35:51.390 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T09:35:51.415 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913661', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 09:30:54.678135', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH/nS8HxX+6jysQ9xYY5WADeQAuQB/5WrLBqGvnS5sXRC0+z0j5RugofkcwGpUENk96Hwo84Q2BKfCprj2U9usA='} 2024-09-21T09:35:51.437 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913661', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 09:30:54.676766', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPbqvGAKqj1W9P/iYa6lwEC+tEzoJ86G8udougnGe27n+zsPWYEuba8tyZwcet4Q28jqA66ZtuCZ4xe4cmAzdD8='} 2024-09-21T09:35:51.437 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T09:35:51.443 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-09-21T09:35:51.443 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-09-21T09:35:51.443 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T09:35:51.505 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2d78a7c160>, signals=[15]) 2024-09-21T09:35:51.505 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T09:35:51.511 INFO:teuthology.task.internal:Opening connections... 2024-09-21T09:35:51.511 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-09-21T09:35:51.512 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:35:51.608 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi196.front.sepia.ceph.com 2024-09-21T09:35:51.609 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:35:51.694 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T09:35:51.702 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-09-21T09:35:51.721 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-09-21T09:35:51.721 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-09-21T09:35:51.778 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2024-09-21T09:35:51.779 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2024-09-21T09:35:51.779 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2024-09-21T09:35:51.779 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2024-09-21T09:35:51.779 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2024-09-21T09:35:51.779 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2024-09-21T09:35:51.779 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T09:35:51.779 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-09-21T09:35:51.779 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2024-09-21T09:35:51.779 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T09:35:51.779 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2024-09-21T09:35:51.779 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T09:35:51.779 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T09:35:51.779 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T09:35:51.780 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-09-21T09:35:51.800 DEBUG:teuthology.orchestra.run.smithi196:> uname -m 2024-09-21T09:35:51.819 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2024-09-21T09:35:51.819 DEBUG:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2024-09-21T09:35:51.877 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="CentOS Stream" 2024-09-21T09:35:51.877 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="9" 2024-09-21T09:35:51.877 INFO:teuthology.orchestra.run.smithi196.stdout:ID="centos" 2024-09-21T09:35:51.877 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="rhel fedora" 2024-09-21T09:35:51.878 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="9" 2024-09-21T09:35:51.878 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el9" 2024-09-21T09:35:51.878 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T09:35:51.878 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31" 2024-09-21T09:35:51.878 INFO:teuthology.orchestra.run.smithi196.stdout:LOGO="fedora-logo-icon" 2024-09-21T09:35:51.878 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T09:35:51.878 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://centos.org/" 2024-09-21T09:35:51.878 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T09:35:51.878 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T09:35:51.878 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T09:35:51.879 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2024-09-21T09:35:51.900 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T09:35:51.909 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T09:35:51.916 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T09:35:51.916 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-09-21T09:35:51.918 DEBUG:teuthology.orchestra.run.smithi196:> test '!' -e /home/ubuntu/cephtest 2024-09-21T09:35:51.935 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T09:35:51.941 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T09:35:51.941 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-09-21T09:35:51.975 DEBUG:teuthology.orchestra.run.smithi196:> test -z $(ls -A /var/lib/ceph) 2024-09-21T09:35:52.075 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T09:35:52.133 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T09:35:52.147 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T09:35:52.147 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T09:35:52.147 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T09:35:52.148 DEBUG:teuthology.orchestra.run.smithi196:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T09:35:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T09:35:52.165 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-09-21T09:35:52.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T09:35:52.166 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-09-21T09:35:52.222 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-503.el9.x86_64 2024-09-21T09:35:52.223 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.14.0-503.el9.x86_64 2024-09-21T09:35:52.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum install -y kernel 2024-09-21T09:35:52.224 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-503.el9.x86_64 2024-09-21T09:35:52.224 INFO:teuthology.task.kernel:Running kernel on smithi196: 5.14.0-503.el9.x86_64 2024-09-21T09:35:52.224 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum install -y kernel 2024-09-21T09:35:55.743 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.3 MB 00:01 2024-09-21T09:35:56.084 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.3 MB 00:02 2024-09-21T09:35:57.810 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 31 MB/s | 20 MB 00:00 2024-09-21T09:35:57.895 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 20 MB/s | 20 MB 00:01 2024-09-21T09:36:03.419 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 6.5 MB 00:01 2024-09-21T09:36:03.471 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-21T09:36:04.886 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 77 kB/s | 19 kB 00:00 2024-09-21T09:36:04.887 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 101 kB/s | 19 kB 00:00 2024-09-21T09:36:05.734 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 31 MB/s | 23 MB 00:00 2024-09-21T09:36:05.823 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-21T09:36:11.146 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-21T09:36:11.181 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-21T09:36:13.061 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T09:36:13.061 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T09:36:13.113 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T09:36:13.113 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T09:36:13.148 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:36:13.154 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:36:13.155 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-09-21T09:36:13.155 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:36:13.155 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-21T09:36:13.155 INFO:teuthology.orchestra.run.smithi196.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T09:36:13.155 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-21T09:36:13.155 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T09:36:13.155 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T09:36:13.155 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T09:36:13.155 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:36:13.155 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-21T09:36:13.155 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:36:13.155 INFO:teuthology.orchestra.run.smithi196.stdout:Install 4 Packages 2024-09-21T09:36:13.155 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:36:13.156 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 79 M 2024-09-21T09:36:13.156 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 126 M 2024-09-21T09:36:13.156 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-21T09:36:13.201 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:36:13.211 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:36:13.212 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-09-21T09:36:13.212 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:36:13.212 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-21T09:36:13.212 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T09:36:13.212 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-21T09:36:13.212 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T09:36:13.212 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T09:36:13.212 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T09:36:13.212 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:36:13.212 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-21T09:36:13.212 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:36:13.213 INFO:teuthology.orchestra.run.smithi098.stdout:Install 4 Packages 2024-09-21T09:36:13.213 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:36:13.213 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 79 M 2024-09-21T09:36:13.213 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 126 M 2024-09-21T09:36:13.213 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-21T09:36:13.350 INFO:teuthology.orchestra.run.smithi098.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 315 kB/s | 36 kB 00:00 2024-09-21T09:36:13.380 INFO:teuthology.orchestra.run.smithi196.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 386 kB/s | 36 kB 00:00 2024-09-21T09:36:13.772 INFO:teuthology.orchestra.run.smithi196.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-09-21T09:36:14.231 INFO:teuthology.orchestra.run.smithi196.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 37 MB/s | 35 MB 00:00 2024-09-21T09:36:14.490 INFO:teuthology.orchestra.run.smithi196.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 26 MB/s | 29 MB 00:01 2024-09-21T09:36:14.490 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:36:14.490 INFO:teuthology.orchestra.run.smithi196.stdout:Total 59 MB/s | 79 MB 00:01 2024-09-21T09:36:15.175 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-21T09:36:15.525 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-21T09:36:15.525 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-21T09:36:16.108 INFO:teuthology.orchestra.run.smithi098.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 5.5 MB/s | 16 MB 00:02 2024-09-21T09:36:16.528 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-21T09:36:16.528 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-21T09:36:18.824 INFO:teuthology.orchestra.run.smithi098.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 6.2 MB/s | 35 MB 00:05 2024-09-21T09:36:18.973 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-21T09:36:19.083 INFO:teuthology.orchestra.run.smithi098.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 5.0 MB/s | 29 MB 00:05 2024-09-21T09:36:19.084 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:36:19.084 INFO:teuthology.orchestra.run.smithi098.stdout:Total 13 MB/s | 79 MB 00:05 2024-09-21T09:36:19.677 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T09:36:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:36:19.761 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-21T09:36:20.106 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-21T09:36:20.106 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-21T09:36:21.066 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-21T09:36:21.066 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-21T09:36:22.492 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:36:22.618 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:36:23.501 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-21T09:36:24.135 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T09:36:24.218 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:36:26.965 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:36:27.097 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:36:27.459 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:36:27.568 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:36:31.916 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:36:31.956 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:36:32.444 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:36:36.797 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:37:10.332 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:37:10.373 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:37:10.742 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:37:10.742 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T09:37:10.742 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:37:10.742 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:37:11.602 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:37:11.602 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:37:11.602 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-21T09:37:11.602 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T09:37:11.603 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T09:37:11.603 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T09:37:11.603 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T09:37:11.603 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:37:11.603 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:37:11.942 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T09:37:12.035 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T09:37:12.036 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T09:37:12.036 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T09:37:12.036 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T09:37:12.036 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T09:37:12.036 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-21T09:37:12.036 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi196.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T09:37:12.037 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T09:37:12.095 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T09:37:12.095 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T09:37:12.095 DEBUG:teuthology.orchestra.run.smithi196:> sudo rpm -qi grub2-tools 2024-09-21T09:37:12.126 INFO:teuthology.orchestra.run.smithi196.stdout:Name : grub2-tools 2024-09-21T09:37:12.126 INFO:teuthology.orchestra.run.smithi196.stdout:Epoch : 1 2024-09-21T09:37:12.126 INFO:teuthology.orchestra.run.smithi196.stdout:Version : 2.06 2024-09-21T09:37:12.126 INFO:teuthology.orchestra.run.smithi196.stdout:Release : 82.el9 2024-09-21T09:37:12.126 INFO:teuthology.orchestra.run.smithi196.stdout:Architecture: x86_64 2024-09-21T09:37:12.127 INFO:teuthology.orchestra.run.smithi196.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T09:37:12.127 INFO:teuthology.orchestra.run.smithi196.stdout:Group : Unspecified 2024-09-21T09:37:12.127 INFO:teuthology.orchestra.run.smithi196.stdout:Size : 8274670 2024-09-21T09:37:12.127 INFO:teuthology.orchestra.run.smithi196.stdout:License : GPLv3+ 2024-09-21T09:37:12.127 INFO:teuthology.orchestra.run.smithi196.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T09:37:12.127 INFO:teuthology.orchestra.run.smithi196.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T09:37:12.127 INFO:teuthology.orchestra.run.smithi196.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T09:37:12.127 INFO:teuthology.orchestra.run.smithi196.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T09:37:12.127 INFO:teuthology.orchestra.run.smithi196.stdout:Packager : builder@centos.org 2024-09-21T09:37:12.127 INFO:teuthology.orchestra.run.smithi196.stdout:Vendor : CentOS 2024-09-21T09:37:12.127 INFO:teuthology.orchestra.run.smithi196.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T09:37:12.127 INFO:teuthology.orchestra.run.smithi196.stdout:Summary : Support tools for GRUB. 2024-09-21T09:37:12.127 INFO:teuthology.orchestra.run.smithi196.stdout:Description : 2024-09-21T09:37:12.127 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:37:12.128 INFO:teuthology.orchestra.run.smithi196.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T09:37:12.128 INFO:teuthology.orchestra.run.smithi196.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T09:37:12.128 INFO:teuthology.orchestra.run.smithi196.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T09:37:12.128 INFO:teuthology.orchestra.run.smithi196.stdout:hardware devices. 2024-09-21T09:37:12.128 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:37:12.128 INFO:teuthology.orchestra.run.smithi196.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T09:37:12.129 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T09:37:12.129 INFO:teuthology.task.kernel:Updating grub on smithi196 to boot 5.14.0-508.el9.x86_64 2024-09-21T09:37:12.129 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T09:37:12.785 INFO:teuthology.orchestra.run.smithi196.stderr:Generating grub configuration file ... 2024-09-21T09:37:14.122 INFO:teuthology.orchestra.run.smithi196.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T09:37:14.138 INFO:teuthology.orchestra.run.smithi196.stderr:done 2024-09-21T09:37:14.149 DEBUG:teuthology.orchestra.run.smithi196:> mktemp 2024-09-21T09:37:14.164 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.dPpi9tfM72 2024-09-21T09:37:14.164 DEBUG:teuthology.orchestra.run.smithi196:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dPpi9tfM72 2024-09-21T09:37:14.231 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.dPpi9tfM72 2024-09-21T09:37:14.373 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.dPpi9tfM72 is 6KB 2024-09-21T09:37:14.421 DEBUG:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.dPpi9tfM72 2024-09-21T09:37:14.436 DEBUG:teuthology.orchestra.run.smithi196:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T09:37:14.509 INFO:teuthology.orchestra.run.smithi196.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T09:37:14.509 INFO:teuthology.orchestra.run.smithi196.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T09:37:14.509 INFO:teuthology.orchestra.run.smithi196.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T09:37:14.509 INFO:teuthology.orchestra.run.smithi196.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T09:37:14.511 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T09:37:14.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo shutdown -r now 2024-09-21T09:37:15.203 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:37:15.243 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:37:15.595 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:37:15.595 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T09:37:15.595 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:37:15.595 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:37:16.435 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:37:16.436 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:37:16.436 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-21T09:37:16.436 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T09:37:16.436 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T09:37:16.436 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T09:37:16.436 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T09:37:16.436 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:37:16.436 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:37:16.790 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T09:37:16.865 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T09:37:16.865 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T09:37:16.866 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T09:37:16.866 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T09:37:16.866 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T09:37:16.866 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-21T09:37:16.866 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T09:37:16.866 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T09:37:16.924 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T09:37:16.924 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T09:37:16.925 DEBUG:teuthology.orchestra.run.smithi098:> sudo rpm -qi grub2-tools 2024-09-21T09:37:16.956 INFO:teuthology.orchestra.run.smithi098.stdout:Name : grub2-tools 2024-09-21T09:37:16.956 INFO:teuthology.orchestra.run.smithi098.stdout:Epoch : 1 2024-09-21T09:37:16.956 INFO:teuthology.orchestra.run.smithi098.stdout:Version : 2.06 2024-09-21T09:37:16.956 INFO:teuthology.orchestra.run.smithi098.stdout:Release : 82.el9 2024-09-21T09:37:16.956 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: x86_64 2024-09-21T09:37:16.957 INFO:teuthology.orchestra.run.smithi098.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T09:37:16.957 INFO:teuthology.orchestra.run.smithi098.stdout:Group : Unspecified 2024-09-21T09:37:16.957 INFO:teuthology.orchestra.run.smithi098.stdout:Size : 8274670 2024-09-21T09:37:16.957 INFO:teuthology.orchestra.run.smithi098.stdout:License : GPLv3+ 2024-09-21T09:37:16.957 INFO:teuthology.orchestra.run.smithi098.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T09:37:16.957 INFO:teuthology.orchestra.run.smithi098.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T09:37:16.957 INFO:teuthology.orchestra.run.smithi098.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T09:37:16.957 INFO:teuthology.orchestra.run.smithi098.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T09:37:16.957 INFO:teuthology.orchestra.run.smithi098.stdout:Packager : builder@centos.org 2024-09-21T09:37:16.957 INFO:teuthology.orchestra.run.smithi098.stdout:Vendor : CentOS 2024-09-21T09:37:16.957 INFO:teuthology.orchestra.run.smithi098.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T09:37:16.957 INFO:teuthology.orchestra.run.smithi098.stdout:Summary : Support tools for GRUB. 2024-09-21T09:37:16.957 INFO:teuthology.orchestra.run.smithi098.stdout:Description : 2024-09-21T09:37:16.957 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:37:16.958 INFO:teuthology.orchestra.run.smithi098.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T09:37:16.958 INFO:teuthology.orchestra.run.smithi098.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T09:37:16.958 INFO:teuthology.orchestra.run.smithi098.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T09:37:16.958 INFO:teuthology.orchestra.run.smithi098.stdout:hardware devices. 2024-09-21T09:37:16.958 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:37:16.958 INFO:teuthology.orchestra.run.smithi098.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T09:37:16.958 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T09:37:16.959 INFO:teuthology.task.kernel:Updating grub on smithi098 to boot 5.14.0-508.el9.x86_64 2024-09-21T09:37:16.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T09:37:17.604 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2024-09-21T09:37:18.877 INFO:teuthology.orchestra.run.smithi098.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T09:37:18.897 INFO:teuthology.orchestra.run.smithi098.stderr:done 2024-09-21T09:37:18.899 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-09-21T09:37:18.915 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.9oS8bjsL1I 2024-09-21T09:37:18.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.9oS8bjsL1I 2024-09-21T09:37:18.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.9oS8bjsL1I 2024-09-21T09:37:19.126 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.9oS8bjsL1I is 6KB 2024-09-21T09:37:19.175 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.9oS8bjsL1I 2024-09-21T09:37:19.190 DEBUG:teuthology.orchestra.run.smithi098:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T09:37:19.261 INFO:teuthology.orchestra.run.smithi098.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T09:37:19.261 INFO:teuthology.orchestra.run.smithi098.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T09:37:19.261 INFO:teuthology.orchestra.run.smithi098.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T09:37:19.261 INFO:teuthology.orchestra.run.smithi098.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T09:37:19.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T09:37:19.419 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2024-09-21T09:37:44.696 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-21T09:37:44.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-21T09:37:44.697 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:37:49.427 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-21T09:37:49.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-21T09:37:49.428 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:38:44.753 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T09:38:49.434 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T09:38:53.758 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-21T09:38:53.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:38:54.173 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-09-21T09:38:54.582 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-21T09:38:54.582 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-21T09:38:54.583 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-09-21T09:38:54.597 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-508.el9.x86_64 2024-09-21T09:38:54.597 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T09:38:54.597 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T09:38:54.597 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T09:38:55.598 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-21T09:38:55.598 DEBUG:teuthology.orchestra.run.smithi196:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T09:38:55.691 INFO:teuthology.orchestra.run.smithi196.stdout:ttyS1 2024-09-21T09:38:55.723 DEBUG:teuthology.parallel:result is None 2024-09-21T09:38:58.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-21T09:38:58.436 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:38:58.787 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-09-21T09:38:59.304 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-21T09:38:59.305 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-21T09:38:59.305 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-09-21T09:38:59.319 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-508.el9.x86_64 2024-09-21T09:38:59.319 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T09:38:59.319 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T09:38:59.320 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T09:39:00.321 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-21T09:39:00.321 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T09:39:00.429 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2024-09-21T09:39:00.463 DEBUG:teuthology.parallel:result is None 2024-09-21T09:39:00.464 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T09:39:00.471 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T09:39:00.471 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T09:39:00.473 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T09:39:00.492 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T09:39:00.504 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T09:39:00.511 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T09:39:00.511 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T09:39:00.529 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T09:39:00.614 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T09:39:00.619 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T09:39:00.619 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T09:39:00.621 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T09:39:00.651 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T09:39:00.663 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T09:39:00.666 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T09:39:00.677 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T09:39:00.679 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T09:39:00.689 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T09:39:00.689 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T09:39:00.691 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T09:39:00.734 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T09:39:00.742 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T09:39:00.742 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T09:39:00.755 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T09:39:00.792 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T09:39:00.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T09:39:00.898 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T09:39:00.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T09:39:01.058 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-21T09:39:01.059 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T09:39:01.124 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T09:39:01.170 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T09:39:01.225 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T09:39:01.310 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T09:39:01.354 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-21T09:39:01.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T09:39:01.421 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-09-21T09:39:01.423 DEBUG:teuthology.orchestra.run.smithi196:> sudo service rsyslog restart 2024-09-21T09:39:01.470 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T09:39:01.516 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T09:39:01.926 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T09:39:01.936 INFO:teuthology.task.internal:Starting timer... 2024-09-21T09:39:01.937 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T09:39:01.950 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T09:39:01.960 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-09-21T09:39:01.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-09-21T09:39:02.023 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-09-21T09:39:02.025 DEBUG:teuthology.orchestra.run.smithi196:> sudo service auditd rotate 2024-09-21T09:39:02.084 INFO:teuthology.orchestra.run.smithi196.stdout:Rotating logs: 2024-09-21T09:39:02.086 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T09:39:02.086 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-09-21T09:39:02.109 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-09-21T09:39:02.109 DEBUG:teuthology.orchestra.run.smithi196:> /usr/sbin/getenforce 2024-09-21T09:39:02.135 INFO:teuthology.orchestra.run.smithi196.stdout:Permissive 2024-09-21T09:39:02.135 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi098.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi196.front.sepia.ceph.com': 'permissive'} 2024-09-21T09:39:02.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-21T09:39:02.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T09:39:02.173 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-21T09:39:02.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T09:39:02.202 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T09:39:02.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-09-21T09:39:02.277 DEBUG:teuthology.orchestra.run.smithi196:> sudo /usr/sbin/setenforce permissive 2024-09-21T09:39:02.309 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T09:39:02.322 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-21T09:39:02.325 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-21T09:39:02.426 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T09:39:02.437 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T09:39:02.438 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi098.front.sepia.ceph.com,smithi196.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T09:42:42.648 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi098.front.sepia.ceph.com'), Remote(name='ubuntu@smithi196.front.sepia.ceph.com')] 2024-09-21T09:42:42.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-21T09:42:42.650 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:42:42.727 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-09-21T09:42:42.810 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-21T09:42:42.811 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-21T09:42:42.811 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:42:42.885 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-09-21T09:42:42.965 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-21T09:42:42.965 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T09:42:42.975 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T09:42:42.975 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T09:42:42.976 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T09:42:42.978 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T09:42:42.978 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T09:42:43.006 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T09:42:43.020 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T09:42:43.034 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T09:42:43.045 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-09-21T09:42:43.048 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T09:42:43.056 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-09-21T09:42:43.070 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T09:42:43.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: ntpd: command not found 2024-09-21T09:42:43.083 INFO:teuthology.orchestra.run.smithi196.stdout:506 Cannot talk to daemon 2024-09-21T09:42:43.084 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T09:42:43.097 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T09:42:43.111 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T09:42:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-09-21T09:42:43.150 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T09:42:43.150 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-09-21T09:42:43.150 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:42:43.150 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:42:43.150 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:42:43.150 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:42:43.167 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-09-21T09:42:43.170 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T09:42:43.171 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-09-21T09:42:43.171 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:42:43.171 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:42:43.171 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:42:43.171 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:42:43.172 INFO:teuthology.run_tasks:Running task pexec... 2024-09-21T09:42:43.181 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-21T09:42:43.181 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi098.front.sepia.ceph.com 2024-09-21T09:42:43.181 DEBUG:teuthology.orchestra.run.smithi098:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-21T09:42:43.181 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi196.front.sepia.ceph.com 2024-09-21T09:42:43.181 DEBUG:teuthology.orchestra.run.smithi196:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-21T09:42:43.542 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:42:43.551 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:42:43.552 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-09-21T09:42:43.552 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:42:43.552 INFO:teuthology.orchestra.run.smithi098.stdout:Removing: 2024-09-21T09:42:43.552 INFO:teuthology.orchestra.run.smithi098.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-21T09:42:43.552 INFO:teuthology.orchestra.run.smithi098.stdout:Removing unused dependencies: 2024-09-21T09:42:43.552 INFO:teuthology.orchestra.run.smithi098.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-21T09:42:43.552 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:42:43.552 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-21T09:42:43.552 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:42:43.552 INFO:teuthology.orchestra.run.smithi098.stdout:Remove 2 Packages 2024-09-21T09:42:43.552 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:42:43.552 INFO:teuthology.orchestra.run.smithi098.stdout:Freed space: 5.5 M 2024-09-21T09:42:43.552 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-21T09:42:43.556 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-21T09:42:43.556 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-21T09:42:43.561 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:42:43.564 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:42:43.564 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-09-21T09:42:43.564 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:42:43.564 INFO:teuthology.orchestra.run.smithi196.stdout:Removing: 2024-09-21T09:42:43.564 INFO:teuthology.orchestra.run.smithi196.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-21T09:42:43.564 INFO:teuthology.orchestra.run.smithi196.stdout:Removing unused dependencies: 2024-09-21T09:42:43.564 INFO:teuthology.orchestra.run.smithi196.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-21T09:42:43.564 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:42:43.564 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-21T09:42:43.564 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:42:43.564 INFO:teuthology.orchestra.run.smithi196.stdout:Remove 2 Packages 2024-09-21T09:42:43.564 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:42:43.565 INFO:teuthology.orchestra.run.smithi196.stdout:Freed space: 5.5 M 2024-09-21T09:42:43.565 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-21T09:42:43.568 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-21T09:42:43.568 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-21T09:42:43.621 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-21T09:42:43.621 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-21T09:42:43.629 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-21T09:42:43.629 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-21T09:42:43.756 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-21T09:42:43.780 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-21T09:42:43.844 INFO:teuthology.orchestra.run.smithi196.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-21T09:42:43.859 INFO:teuthology.orchestra.run.smithi098.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-21T09:42:43.901 INFO:teuthology.orchestra.run.smithi196.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T09:42:43.912 INFO:teuthology.orchestra.run.smithi098.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T09:42:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T09:42:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-21T09:42:44.417 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T09:42:44.417 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-21T09:42:44.646 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-21T09:42:44.646 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:42:44.646 INFO:teuthology.orchestra.run.smithi098.stdout:Removed: 2024-09-21T09:42:44.646 INFO:teuthology.orchestra.run.smithi098.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T09:42:44.647 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:42:44.647 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:42:44.661 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-21T09:42:44.661 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:42:44.661 INFO:teuthology.orchestra.run.smithi196.stdout:Removed: 2024-09-21T09:42:44.661 INFO:teuthology.orchestra.run.smithi196.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T09:42:44.661 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:42:44.661 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:42:45.486 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:47 ago on Sat 21 Sep 2024 09:41:58 AM UTC. 2024-09-21T09:42:45.503 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:47 ago on Sat 21 Sep 2024 09:41:58 AM UTC. 2024-09-21T09:42:45.616 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:42:45.616 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:42:45.616 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-09-21T09:42:45.616 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:42:45.617 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-21T09:42:45.617 INFO:teuthology.orchestra.run.smithi098.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-21T09:42:45.617 INFO:teuthology.orchestra.run.smithi098.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-21T09:42:45.617 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-21T09:42:45.617 INFO:teuthology.orchestra.run.smithi098.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-21T09:42:45.617 INFO:teuthology.orchestra.run.smithi098.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-21T09:42:45.617 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-21T09:42:45.617 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-21T09:42:45.617 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:42:45.617 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-21T09:42:45.617 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:42:45.617 INFO:teuthology.orchestra.run.smithi098.stdout:Install 6 Packages 2024-09-21T09:42:45.617 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:42:45.624 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 2.0 M 2024-09-21T09:42:45.624 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 9.2 M 2024-09-21T09:42:45.624 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-21T09:42:45.642 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:42:45.643 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:42:45.643 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-09-21T09:42:45.644 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:42:45.644 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-21T09:42:45.644 INFO:teuthology.orchestra.run.smithi196.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-21T09:42:45.644 INFO:teuthology.orchestra.run.smithi196.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-21T09:42:45.644 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-21T09:42:45.644 INFO:teuthology.orchestra.run.smithi196.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-21T09:42:45.644 INFO:teuthology.orchestra.run.smithi196.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-21T09:42:45.644 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-21T09:42:45.644 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-21T09:42:45.644 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:42:45.644 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-21T09:42:45.644 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:42:45.644 INFO:teuthology.orchestra.run.smithi196.stdout:Install 6 Packages 2024-09-21T09:42:45.645 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:42:45.651 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 2.0 M 2024-09-21T09:42:45.651 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 9.2 M 2024-09-21T09:42:45.651 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-21T09:42:45.783 INFO:teuthology.orchestra.run.smithi196.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 542 kB/s | 44 kB 00:00 2024-09-21T09:42:45.808 INFO:teuthology.orchestra.run.smithi196.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 988 kB/s | 106 kB 00:00 2024-09-21T09:42:45.818 INFO:teuthology.orchestra.run.smithi098.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 494 kB/s | 44 kB 00:00 2024-09-21T09:42:45.833 INFO:teuthology.orchestra.run.smithi196.stdout:(3/6): python3-configshell-1.1.30-1.el9.noarch. 1.4 MB/s | 72 kB 00:00 2024-09-21T09:42:45.852 INFO:teuthology.orchestra.run.smithi098.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 857 kB/s | 106 kB 00:00 2024-09-21T09:42:45.858 INFO:teuthology.orchestra.run.smithi196.stdout:(4/6): python3-kmod-0.9-32.el9.x86_64.rpm 1.6 MB/s | 84 kB 00:00 2024-09-21T09:42:45.894 INFO:teuthology.orchestra.run.smithi098.stdout:(3/6): python3-configshell-1.1.30-1.el9.noarch. 950 kB/s | 72 kB 00:00 2024-09-21T09:42:45.900 INFO:teuthology.orchestra.run.smithi196.stdout:(5/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 4.4 MB/s | 906 kB 00:00 2024-09-21T09:42:45.919 INFO:teuthology.orchestra.run.smithi098.stdout:(4/6): python3-kmod-0.9-32.el9.x86_64.rpm 1.2 MB/s | 84 kB 00:00 2024-09-21T09:42:45.942 INFO:teuthology.orchestra.run.smithi196.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 7.5 MB/s | 837 kB 00:00 2024-09-21T09:42:45.942 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:42:45.942 INFO:teuthology.orchestra.run.smithi196.stdout:Total 6.9 MB/s | 2.0 MB 00:00 2024-09-21T09:42:45.944 INFO:teuthology.orchestra.run.smithi098.stdout:(5/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 4.1 MB/s | 906 kB 00:00 2024-09-21T09:42:46.003 INFO:teuthology.orchestra.run.smithi098.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 7.5 MB/s | 837 kB 00:00 2024-09-21T09:42:46.004 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:42:46.004 INFO:teuthology.orchestra.run.smithi098.stdout:Total 5.3 MB/s | 2.0 MB 00:00 2024-09-21T09:42:46.015 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-21T09:42:46.028 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-21T09:42:46.028 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-21T09:42:46.075 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-21T09:42:46.089 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-21T09:42:46.089 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-21T09:42:46.311 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-21T09:42:46.312 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-21T09:42:46.350 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-21T09:42:46.351 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-21T09:42:46.731 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-21T09:42:46.766 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-21T09:42:46.820 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-21T09:42:46.847 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-21T09:42:46.854 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-21T09:42:46.882 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-21T09:42:46.883 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-21T09:42:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-21T09:42:47.089 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-21T09:42:47.123 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-21T09:42:47.131 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-21T09:42:47.159 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-21T09:42:47.579 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-21T09:42:47.609 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-21T09:42:47.610 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-21T09:42:47.647 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-21T09:42:48.594 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-21T09:42:48.594 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-21T09:42:48.594 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-21T09:42:48.594 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-21T09:42:48.594 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-21T09:42:48.594 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-21T09:42:48.654 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-21T09:42:48.654 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-21T09:42:48.654 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-21T09:42:48.654 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-21T09:42:48.654 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-21T09:42:48.654 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-21T09:42:48.928 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-21T09:42:48.928 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:42:48.928 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-21T09:42:48.928 INFO:teuthology.orchestra.run.smithi196.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T09:42:48.929 INFO:teuthology.orchestra.run.smithi196.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-21T09:42:48.929 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-21T09:42:48.929 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:42:48.929 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:42:49.025 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-21T09:42:49.025 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:42:49.025 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-21T09:42:49.025 INFO:teuthology.orchestra.run.smithi098.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T09:42:49.025 INFO:teuthology.orchestra.run.smithi098.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-21T09:42:49.025 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-21T09:42:49.025 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:42:49.025 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:42:49.282 DEBUG:teuthology.parallel:result is None 2024-09-21T09:42:49.351 DEBUG:teuthology.parallel:result is None 2024-09-21T09:42:49.351 INFO:teuthology.run_tasks:Running task install... 2024-09-21T09:42:49.368 DEBUG:teuthology.task.install:project ceph 2024-09-21T09:42:49.368 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}} 2024-09-21T09:42:49.368 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'} 2024-09-21T09:42:49.368 INFO:teuthology.task.install:Using flavor: default 2024-09-21T09:42:49.374 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T09:42:49.374 INFO:teuthology.task.install:extra packages: [] 2024-09-21T09:42:49.374 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'tag': None, 'wait_for_package': False} 2024-09-21T09:42:49.375 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:42:49.375 INFO:teuthology.packaging:ref: None 2024-09-21T09:42:49.375 INFO:teuthology.packaging:tag: None 2024-09-21T09:42:49.375 INFO:teuthology.packaging:branch: quincy 2024-09-21T09:42:49.375 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:42:49.375 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-21T09:42:49.376 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'tag': None, 'wait_for_package': False} 2024-09-21T09:42:49.377 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:42:49.377 INFO:teuthology.packaging:ref: None 2024-09-21T09:42:49.377 INFO:teuthology.packaging:tag: None 2024-09-21T09:42:49.377 INFO:teuthology.packaging:branch: quincy 2024-09-21T09:42:49.377 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:42:49.377 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-21T09:42:49.605 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/ 2024-09-21T09:42:49.606 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1649.gfbd1a6ab 2024-09-21T09:42:49.611 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/ 2024-09-21T09:42:49.611 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1649.gfbd1a6ab 2024-09-21T09:42:49.717 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T09:42:49.717 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-21T09:42:49.717 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T09:42:49.722 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T09:42:49.722 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-21T09:42:49.722 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T09:42:49.747 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T09:42:49.747 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:42:49.747 INFO:teuthology.packaging:ref: None 2024-09-21T09:42:49.747 INFO:teuthology.packaging:tag: None 2024-09-21T09:42:49.747 INFO:teuthology.packaging:branch: quincy 2024-09-21T09:42:49.747 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:42:49.747 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T09:42:49.752 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T09:42:49.752 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:42:49.752 INFO:teuthology.packaging:ref: None 2024-09-21T09:42:49.752 INFO:teuthology.packaging:tag: None 2024-09-21T09:42:49.752 INFO:teuthology.packaging:branch: quincy 2024-09-21T09:42:49.752 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:42:49.752 DEBUG:teuthology.orchestra.run.smithi196:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T09:42:49.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T09:42:49.819 DEBUG:teuthology.orchestra.run.smithi196:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T09:42:49.904 DEBUG:teuthology.orchestra.run.smithi196:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T09:42:49.906 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T09:42:49.937 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2024-09-21T09:42:49.939 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2024-09-21T09:42:49.969 INFO:teuthology.orchestra.run.smithi196.stdout:check_obsoletes = 1 2024-09-21T09:42:49.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum clean all 2024-09-21T09:42:50.173 INFO:teuthology.orchestra.run.smithi098.stdout:45 files removed 2024-09-21T09:42:50.200 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw 2024-09-21T09:42:50.201 INFO:teuthology.orchestra.run.smithi196.stdout:45 files removed 2024-09-21T09:42:50.227 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-radosgw 2024-09-21T09:42:50.735 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 308 kB/s | 77 kB 00:00 2024-09-21T09:42:50.791 INFO:teuthology.orchestra.run.smithi196.stdout:ceph packages for x86_64 279 kB/s | 77 kB 00:00 2024-09-21T09:42:51.003 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 48 kB/s | 11 kB 00:00 2024-09-21T09:42:51.083 INFO:teuthology.orchestra.run.smithi196.stdout:ceph noarch packages 43 kB/s | 11 kB 00:00 2024-09-21T09:42:51.270 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 7.7 kB/s | 1.8 kB 00:00 2024-09-21T09:42:51.350 INFO:teuthology.orchestra.run.smithi196.stdout:ceph source packages 7.7 kB/s | 1.8 kB 00:00 2024-09-21T09:42:51.879 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.3 MB 00:00 2024-09-21T09:42:52.934 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 5.3 MB/s | 8.3 MB 00:01 2024-09-21T09:42:54.643 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 32 MB/s | 20 MB 00:00 2024-09-21T09:42:54.947 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 10 MB/s | 20 MB 00:02 2024-09-21T09:42:59.127 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-21T09:42:59.214 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-21T09:43:00.620 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 120 kB/s | 19 kB 00:00 2024-09-21T09:43:00.715 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 19 kB 00:00 2024-09-21T09:43:01.578 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 31 MB/s | 23 MB 00:00 2024-09-21T09:43:01.690 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-21T09:43:06.984 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-09-21T09:43:07.088 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-09-21T09:43:09.233 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:43:09.235 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:43:09.235 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-21T09:43:09.235 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:43:09.235 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-21T09:43:09.236 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 14 M 2024-09-21T09:43:09.236 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2024-09-21T09:43:09.236 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 3.1 M 2024-09-21T09:43:09.236 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 2.9 M 2024-09-21T09:43:09.236 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-21T09:43:09.236 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 5.0 M 2024-09-21T09:43:09.236 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 22 M 2024-09-21T09:43:09.236 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 25 k 2024-09-21T09:43:09.236 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T09:43:09.236 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 636 k 2024-09-21T09:43:09.236 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T09:43:09.236 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T09:43:09.236 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T09:43:09.237 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 457 k 2024-09-21T09:43:09.237 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T09:43:09.237 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 4.9 M 2024-09-21T09:43:09.237 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T09:43:09.237 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T09:43:09.237 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 45 k 2024-09-21T09:43:09.237 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 120 k 2024-09-21T09:43:09.237 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 156 k 2024-09-21T09:43:09.237 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T09:43:09.237 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 320 k 2024-09-21T09:43:09.237 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 295 k 2024-09-21T09:43:09.237 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 99 k 2024-09-21T09:43:09.237 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T09:43:09.237 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:09.237 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-21T09:43:09.238 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:43:09.238 INFO:teuthology.orchestra.run.smithi098.stdout:Install 22 Packages 2024-09-21T09:43:09.238 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 2 Packages 2024-09-21T09:43:09.238 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:09.238 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 58 M 2024-09-21T09:43:09.238 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-21T09:43:09.335 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:43:09.337 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:43:09.337 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-21T09:43:09.337 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:43:09.337 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-21T09:43:09.337 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 14 M 2024-09-21T09:43:09.337 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrading: 2024-09-21T09:43:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: librados2 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 3.1 M 2024-09-21T09:43:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 2.9 M 2024-09-21T09:43:09.338 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-21T09:43:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 5.0 M 2024-09-21T09:43:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 22 M 2024-09-21T09:43:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 25 k 2024-09-21T09:43:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T09:43:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 636 k 2024-09-21T09:43:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T09:43:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T09:43:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T09:43:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 457 k 2024-09-21T09:43:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T09:43:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 4.9 M 2024-09-21T09:43:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T09:43:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T09:43:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 45 k 2024-09-21T09:43:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 120 k 2024-09-21T09:43:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 156 k 2024-09-21T09:43:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T09:43:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 320 k 2024-09-21T09:43:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 295 k 2024-09-21T09:43:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 99 k 2024-09-21T09:43:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T09:43:09.340 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:09.340 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-21T09:43:09.340 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:43:09.340 INFO:teuthology.orchestra.run.smithi196.stdout:Install 22 Packages 2024-09-21T09:43:09.340 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrade 2 Packages 2024-09-21T09:43:09.340 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:09.340 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 58 M 2024-09-21T09:43:09.341 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-21T09:43:09.811 INFO:teuthology.orchestra.run.smithi098.stdout:(1/24): ceph-base-17.2.7-1649.gfbd1a6ab.el9.x86 11 MB/s | 5.0 MB 00:00 2024-09-21T09:43:09.861 INFO:teuthology.orchestra.run.smithi098.stdout:(2/24): ceph-selinux-17.2.7-1649.gfbd1a6ab.el9. 505 kB/s | 25 kB 00:00 2024-09-21T09:43:09.953 INFO:teuthology.orchestra.run.smithi098.stdout:(3/24): libcephfs2-17.2.7-1649.gfbd1a6ab.el9.x8 6.8 MB/s | 636 kB 00:00 2024-09-21T09:43:10.045 INFO:teuthology.orchestra.run.smithi098.stdout:(4/24): libradosstriper1-17.2.7-1649.gfbd1a6ab. 4.9 MB/s | 457 kB 00:00 2024-09-21T09:43:10.403 INFO:teuthology.orchestra.run.smithi098.stdout:(5/24): librgw2-17.2.7-1649.gfbd1a6ab.el9.x86_6 14 MB/s | 4.9 MB 00:00 2024-09-21T09:43:10.595 INFO:teuthology.orchestra.run.smithi098.stdout:(6/24): ceph-radosgw-17.2.7-1649.gfbd1a6ab.el9. 12 MB/s | 14 MB 00:01 2024-09-21T09:43:10.620 INFO:teuthology.orchestra.run.smithi098.stdout:(7/24): python3-ceph-argparse-17.2.7-1649.gfbd1 208 kB/s | 45 kB 00:00 2024-09-21T09:43:10.646 INFO:teuthology.orchestra.run.smithi098.stdout:(8/24): python3-ceph-common-17.2.7-1649.gfbd1a6 2.3 MB/s | 120 kB 00:00 2024-09-21T09:43:10.854 INFO:teuthology.orchestra.run.smithi098.stdout:(9/24): ceph-common-17.2.7-1649.gfbd1a6ab.el9.x 15 MB/s | 22 MB 00:01 2024-09-21T09:43:10.855 INFO:teuthology.orchestra.run.smithi196.stdout:(1/24): ceph-base-17.2.7-1649.gfbd1a6ab.el9.x86 3.9 MB/s | 5.0 MB 00:01 2024-09-21T09:43:10.880 INFO:teuthology.orchestra.run.smithi098.stdout:(10/24): python3-cephfs-17.2.7-1649.gfbd1a6ab.e 603 kB/s | 156 kB 00:00 2024-09-21T09:43:10.897 INFO:teuthology.orchestra.run.smithi196.stdout:(2/24): ceph-selinux-17.2.7-1649.gfbd1a6ab.el9. 609 kB/s | 25 kB 00:00 2024-09-21T09:43:10.905 INFO:teuthology.orchestra.run.smithi098.stdout:(11/24): python3-rados-17.2.7-1649.gfbd1a6ab.el 1.2 MB/s | 320 kB 00:00 2024-09-21T09:43:10.938 INFO:teuthology.orchestra.run.smithi098.stdout:(12/24): python3-rbd-17.2.7-1649.gfbd1a6ab.el9. 3.4 MB/s | 295 kB 00:00 2024-09-21T09:43:10.964 INFO:teuthology.orchestra.run.smithi098.stdout:(13/24): python3-rgw-17.2.7-1649.gfbd1a6ab.el9. 1.2 MB/s | 99 kB 00:00 2024-09-21T09:43:11.006 INFO:teuthology.orchestra.run.smithi196.stdout:(3/24): libcephfs2-17.2.7-1649.gfbd1a6ab.el9.x8 5.7 MB/s | 636 kB 00:00 2024-09-21T09:43:11.031 INFO:teuthology.orchestra.run.smithi098.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 492 kB/s | 45 kB 00:00 2024-09-21T09:43:11.073 INFO:teuthology.orchestra.run.smithi098.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 956 kB/s | 160 kB 00:00 2024-09-21T09:43:11.098 INFO:teuthology.orchestra.run.smithi196.stdout:(4/24): libradosstriper1-17.2.7-1649.gfbd1a6ab. 4.9 MB/s | 457 kB 00:00 2024-09-21T09:43:11.106 INFO:teuthology.orchestra.run.smithi098.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 42 kB 00:00 2024-09-21T09:43:11.140 INFO:teuthology.orchestra.run.smithi098.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-21T09:43:11.174 INFO:teuthology.orchestra.run.smithi098.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-09-21T09:43:11.215 INFO:teuthology.orchestra.run.smithi098.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2024-09-21T09:43:11.241 INFO:teuthology.orchestra.run.smithi098.stdout:(20/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.2 MB/s | 308 kB 00:00 2024-09-21T09:43:11.266 INFO:teuthology.orchestra.run.smithi098.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 862 kB/s | 67 kB 00:00 2024-09-21T09:43:11.316 INFO:teuthology.orchestra.run.smithi098.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-21T09:43:11.417 INFO:teuthology.orchestra.run.smithi098.stdout:(23/24): librados2-17.2.7-1649.gfbd1a6ab.el9.x8 18 MB/s | 3.1 MB 00:00 2024-09-21T09:43:11.525 INFO:teuthology.orchestra.run.smithi098.stdout:(24/24): librbd1-17.2.7-1649.gfbd1a6ab.el9.x86_ 11 MB/s | 2.9 MB 00:00 2024-09-21T09:43:11.530 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:43:11.530 INFO:teuthology.orchestra.run.smithi098.stdout:Total 25 MB/s | 58 MB 00:02 2024-09-21T09:43:11.592 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-21T09:43:11.625 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-21T09:43:11.625 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-21T09:43:11.715 INFO:teuthology.orchestra.run.smithi196.stdout:(5/24): ceph-radosgw-17.2.7-1649.gfbd1a6ab.el9. 6.8 MB/s | 14 MB 00:02 2024-09-21T09:43:11.757 INFO:teuthology.orchestra.run.smithi196.stdout:(6/24): python3-ceph-argparse-17.2.7-1649.gfbd1 1.1 MB/s | 45 kB 00:00 2024-09-21T09:43:11.824 INFO:teuthology.orchestra.run.smithi196.stdout:(7/24): python3-ceph-common-17.2.7-1649.gfbd1a6 1.8 MB/s | 120 kB 00:00 2024-09-21T09:43:11.890 INFO:teuthology.orchestra.run.smithi196.stdout:(8/24): librgw2-17.2.7-1649.gfbd1a6ab.el9.x86_6 6.2 MB/s | 4.9 MB 00:00 2024-09-21T09:43:11.915 INFO:teuthology.orchestra.run.smithi196.stdout:(9/24): python3-cephfs-17.2.7-1649.gfbd1a6ab.el 1.7 MB/s | 156 kB 00:00 2024-09-21T09:43:12.106 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-21T09:43:12.106 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-21T09:43:12.166 INFO:teuthology.orchestra.run.smithi196.stdout:(10/24): ceph-common-17.2.7-1649.gfbd1a6ab.el9. 8.7 MB/s | 22 MB 00:02 2024-09-21T09:43:12.191 INFO:teuthology.orchestra.run.smithi196.stdout:(11/24): python3-rados-17.2.7-1649.gfbd1a6ab.el 1.0 MB/s | 320 kB 00:00 2024-09-21T09:43:12.225 INFO:teuthology.orchestra.run.smithi196.stdout:(12/24): python3-rgw-17.2.7-1649.gfbd1a6ab.el9. 1.7 MB/s | 99 kB 00:00 2024-09-21T09:43:12.250 INFO:teuthology.orchestra.run.smithi196.stdout:(13/24): python3-rbd-17.2.7-1649.gfbd1a6ab.el9. 881 kB/s | 295 kB 00:00 2024-09-21T09:43:12.367 INFO:teuthology.orchestra.run.smithi196.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 319 kB/s | 45 kB 00:00 2024-09-21T09:43:12.393 INFO:teuthology.orchestra.run.smithi196.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 796 kB/s | 160 kB 00:00 2024-09-21T09:43:12.451 INFO:teuthology.orchestra.run.smithi196.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-09-21T09:43:12.476 INFO:teuthology.orchestra.run.smithi196.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 506 kB/s | 42 kB 00:00 2024-09-21T09:43:12.502 INFO:teuthology.orchestra.run.smithi196.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.6 MB/s | 662 kB 00:00 2024-09-21T09:43:12.552 INFO:teuthology.orchestra.run.smithi196.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 646 kB/s | 49 kB 00:00 2024-09-21T09:43:12.577 INFO:teuthology.orchestra.run.smithi196.stdout:(20/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.4 MB/s | 308 kB 00:00 2024-09-21T09:43:12.603 INFO:teuthology.orchestra.run.smithi196.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 783 kB/s | 67 kB 00:00 2024-09-21T09:43:12.653 INFO:teuthology.orchestra.run.smithi196.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-21T09:43:12.778 INFO:teuthology.orchestra.run.smithi196.stdout:(23/24): librbd1-17.2.7-1649.gfbd1a6ab.el9.x86_ 17 MB/s | 2.9 MB 00:00 2024-09-21T09:43:12.978 INFO:teuthology.orchestra.run.smithi196.stdout:(24/24): librados2-17.2.7-1649.gfbd1a6ab.el9.x8 7.7 MB/s | 3.1 MB 00:00 2024-09-21T09:43:12.984 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:43:12.984 INFO:teuthology.orchestra.run.smithi196.stdout:Total 16 MB/s | 58 MB 00:03 2024-09-21T09:43:13.046 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-21T09:43:13.079 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-21T09:43:13.079 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-21T09:43:13.360 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-21T09:43:13.581 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-21T09:43:13.582 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-21T09:43:13.733 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-21T09:43:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/26 2024-09-21T09:43:13.907 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/26 2024-09-21T09:43:13.985 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/26 2024-09-21T09:43:14.033 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/26 2024-09-21T09:43:14.121 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/26 2024-09-21T09:43:14.180 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-21T09:43:14.213 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-21T09:43:14.650 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-21T09:43:14.677 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 8/26 2024-09-21T09:43:14.749 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 8/26 2024-09-21T09:43:14.784 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-21T09:43:14.786 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-21T09:43:14.833 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:17.2.7-1649.gfbd1a6ab.el9. 10/26 2024-09-21T09:43:14.877 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 11/26 2024-09-21T09:43:14.945 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 12/26 2024-09-21T09:43:14.972 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_6 13/26 2024-09-21T09:43:15.038 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_6 13/26 2024-09-21T09:43:15.078 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-21T09:43:15.123 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-21T09:43:15.126 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-21T09:43:15.155 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-21T09:43:15.231 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/26 2024-09-21T09:43:15.324 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/26 2024-09-21T09:43:15.409 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/26 2024-09-21T09:43:15.440 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/26 2024-09-21T09:43:15.450 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-21T09:43:15.493 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 18/26 2024-09-21T09:43:15.519 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/26 2024-09-21T09:43:15.567 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 18/26 2024-09-21T09:43:15.578 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-21T09:43:15.614 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-21T09:43:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 19/26 2024-09-21T09:43:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x8 20/26 2024-09-21T09:43:16.049 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-21T09:43:16.079 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 8/26 2024-09-21T09:43:16.142 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 8/26 2024-09-21T09:43:16.187 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-21T09:43:16.234 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-argparse-2:17.2.7-1649.gfbd1a6ab.el9. 10/26 2024-09-21T09:43:16.281 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 11/26 2024-09-21T09:43:16.345 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 12/26 2024-09-21T09:43:16.373 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_6 13/26 2024-09-21T09:43:16.439 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_6 13/26 2024-09-21T09:43:16.502 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-21T09:43:16.540 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-21T09:43:16.574 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-21T09:43:16.866 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-21T09:43:16.910 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 18/26 2024-09-21T09:43:17.000 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 18/26 2024-09-21T09:43:17.075 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 19/26 2024-09-21T09:43:17.104 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x8 20/26 2024-09-21T09:43:18.247 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 21/26 2024-09-21T09:43:18.292 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 21/26 2024-09-21T09:43:18.831 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 21/26 2024-09-21T09:43:19.013 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 22/26 2024-09-21T09:43:19.109 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 22/26 2024-09-21T09:43:19.109 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T09:43:19.109 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T09:43:19.110 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:19.147 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 23/26 2024-09-21T09:43:19.561 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 21/26 2024-09-21T09:43:19.601 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 21/26 2024-09-21T09:43:20.163 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 21/26 2024-09-21T09:43:20.196 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 22/26 2024-09-21T09:43:20.299 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 22/26 2024-09-21T09:43:20.299 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T09:43:20.299 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T09:43:20.299 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:20.339 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 23/26 2024-09-21T09:43:28.996 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 23/26 2024-09-21T09:43:28.996 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2024-09-21T09:43:28.996 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2024-09-21T09:43:28.996 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2024-09-21T09:43:28.996 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2024-09-21T09:43:28.996 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2024-09-21T09:43:28.996 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /root 2024-09-21T09:43:28.996 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2024-09-21T09:43:28.996 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:30.206 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 23/26 2024-09-21T09:43:30.207 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /sys 2024-09-21T09:43:30.207 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /proc 2024-09-21T09:43:30.207 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /mnt 2024-09-21T09:43:30.207 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /var/tmp 2024-09-21T09:43:30.207 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /home 2024-09-21T09:43:30.207 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /root 2024-09-21T09:43:30.207 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /tmp 2024-09-21T09:43:30.207 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:30.289 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 24/26 2024-09-21T09:43:30.367 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 24/26 2024-09-21T09:43:30.367 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:43:30.368 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T09:43:30.368 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T09:43:30.368 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T09:43:30.368 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:30.369 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-21T09:43:30.458 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-21T09:43:30.458 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-21T09:43:31.521 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 24/26 2024-09-21T09:43:31.600 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 24/26 2024-09-21T09:43:31.600 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:43:31.600 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T09:43:31.600 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T09:43:31.600 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T09:43:31.600 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:31.602 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-21T09:43:31.682 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-21T09:43:31.682 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-21T09:43:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-21T09:43:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/26 2024-09-21T09:43:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/26 2024-09-21T09:43:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/26 2024-09-21T09:43:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/26 2024-09-21T09:43:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 5/26 2024-09-21T09:43:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_6 6/26 2024-09-21T09:43:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 7/26 2024-09-21T09:43:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1649.gfbd1a6ab.el9. 8/26 2024-09-21T09:43:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x8 9/26 2024-09-21T09:43:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 10/26 2024-09-21T09:43:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 11/26 2024-09-21T09:43:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 12/26 2024-09-21T09:43:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 13/26 2024-09-21T09:43:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-21T09:43:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-21T09:43:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-21T09:43:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-21T09:43:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-21T09:43:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-21T09:43:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-21T09:43:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-21T09:43:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-21T09:43:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 23/26 2024-09-21T09:43:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-21T09:43:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 25/26 2024-09-21T09:43:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-21T09:43:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:32.712 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2024-09-21T09:43:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:32.712 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-21T09:43:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T09:43:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T09:43:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T09:43:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T09:43:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T09:43:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T09:43:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T09:43:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T09:43:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T09:43:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:32.714 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:43:32.925 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-21T09:43:32.925 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/26 2024-09-21T09:43:32.925 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/26 2024-09-21T09:43:32.925 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/26 2024-09-21T09:43:32.926 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/26 2024-09-21T09:43:32.926 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 5/26 2024-09-21T09:43:32.926 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_6 6/26 2024-09-21T09:43:32.926 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 7/26 2024-09-21T09:43:32.926 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1649.gfbd1a6ab.el9. 8/26 2024-09-21T09:43:32.926 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x8 9/26 2024-09-21T09:43:32.926 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 10/26 2024-09-21T09:43:32.926 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 11/26 2024-09-21T09:43:32.926 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 12/26 2024-09-21T09:43:32.926 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 13/26 2024-09-21T09:43:32.926 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-21T09:43:32.926 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-21T09:43:32.926 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-21T09:43:32.927 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-21T09:43:32.927 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-21T09:43:32.927 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-21T09:43:32.927 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-21T09:43:32.927 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-21T09:43:32.927 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-21T09:43:32.927 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 23/26 2024-09-21T09:43:32.927 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-21T09:43:32.927 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 25/26 2024-09-21T09:43:33.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-test 2024-09-21T09:43:33.697 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:27 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:43:33.766 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-21T09:43:33.766 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:33.766 INFO:teuthology.orchestra.run.smithi196.stdout:Upgraded: 2024-09-21T09:43:33.766 INFO:teuthology.orchestra.run.smithi196.stdout: librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:33.766 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:33.767 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-21T09:43:33.767 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:33.767 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:33.767 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:33.767 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:33.767 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T09:43:33.767 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:33.768 INFO:teuthology.orchestra.run.smithi196.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T09:43:33.768 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T09:43:33.768 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T09:43:33.768 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:33.768 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T09:43:33.768 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:33.768 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T09:43:33.769 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T09:43:33.769 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:33.769 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:33.769 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:33.769 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T09:43:33.769 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:33.770 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:33.770 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:33.770 INFO:teuthology.orchestra.run.smithi196.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T09:43:33.770 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:33.770 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:43:33.835 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:43:33.835 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:43:33.836 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-21T09:43:33.836 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:43:33.836 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-21T09:43:33.836 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 48 M 2024-09-21T09:43:33.836 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-21T09:43:33.836 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 163 k 2024-09-21T09:43:33.837 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T09:43:33.837 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T09:43:33.837 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:33.837 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-21T09:43:33.837 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:43:33.837 INFO:teuthology.orchestra.run.smithi098.stdout:Install 4 Packages 2024-09-21T09:43:33.838 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:33.838 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 48 M 2024-09-21T09:43:33.839 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 193 M 2024-09-21T09:43:33.839 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-21T09:43:34.189 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-test 2024-09-21T09:43:34.194 INFO:teuthology.orchestra.run.smithi098.stdout:(1/4): libcephsqlite-17.2.7-1649.gfbd1a6ab.el9. 708 kB/s | 163 kB 00:00 2024-09-21T09:43:34.603 INFO:teuthology.orchestra.run.smithi098.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 476 kB/s | 304 kB 00:00 2024-09-21T09:43:34.645 INFO:teuthology.orchestra.run.smithi098.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 141 kB/s | 64 kB 00:00 2024-09-21T09:43:34.770 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:27 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:43:34.902 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:43:34.902 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:43:34.902 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-21T09:43:34.902 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:43:34.903 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-21T09:43:34.903 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 48 M 2024-09-21T09:43:34.903 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-21T09:43:34.903 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 163 k 2024-09-21T09:43:34.903 INFO:teuthology.orchestra.run.smithi196.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T09:43:34.903 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T09:43:34.903 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:34.903 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-21T09:43:34.903 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:43:34.903 INFO:teuthology.orchestra.run.smithi196.stdout:Install 4 Packages 2024-09-21T09:43:34.903 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:34.904 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 48 M 2024-09-21T09:43:34.904 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 193 M 2024-09-21T09:43:34.904 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-21T09:43:35.060 INFO:teuthology.orchestra.run.smithi196.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.2 MB/s | 304 kB 00:00 2024-09-21T09:43:35.101 INFO:teuthology.orchestra.run.smithi196.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-21T09:43:35.238 INFO:teuthology.orchestra.run.smithi196.stdout:(3/4): libcephsqlite-17.2.7-1649.gfbd1a6ab.el9. 529 kB/s | 163 kB 00:00 2024-09-21T09:43:35.478 INFO:teuthology.orchestra.run.smithi098.stdout:(4/4): ceph-test-17.2.7-1649.gfbd1a6ab.el9.x86_ 32 MB/s | 48 MB 00:01 2024-09-21T09:43:35.481 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:43:35.481 INFO:teuthology.orchestra.run.smithi098.stdout:Total 29 MB/s | 48 MB 00:01 2024-09-21T09:43:35.509 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-21T09:43:35.528 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-21T09:43:35.528 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-21T09:43:35.896 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-21T09:43:35.896 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-21T09:43:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-21T09:43:36.510 INFO:teuthology.orchestra.run.smithi196.stdout:(4/4): ceph-test-17.2.7-1649.gfbd1a6ab.el9.x86_ 30 MB/s | 48 MB 00:01 2024-09-21T09:43:36.512 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:43:36.513 INFO:teuthology.orchestra.run.smithi196.stdout:Total 30 MB/s | 48 MB 00:01 2024-09-21T09:43:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T09:43:36.541 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-21T09:43:36.560 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-21T09:43:36.560 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-21T09:43:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T09:43:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/4 2024-09-21T09:43:36.926 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-21T09:43:36.927 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-21T09:43:37.468 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-21T09:43:37.558 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T09:43:37.601 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T09:43:37.629 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/4 2024-09-21T09:43:41.096 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/4 2024-09-21T09:43:41.203 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/4 2024-09-21T09:43:42.008 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/4 2024-09-21T09:43:42.084 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/4 2024-09-21T09:43:43.485 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/4 2024-09-21T09:43:43.485 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/4 2024-09-21T09:43:43.485 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/4 2024-09-21T09:43:43.485 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T09:43:43.791 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T09:43:43.791 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:43.791 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-21T09:43:43.791 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:43.791 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:43.791 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T09:43:43.791 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T09:43:43.791 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:43.792 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:43:44.071 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph 2024-09-21T09:43:44.338 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/4 2024-09-21T09:43:44.338 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/4 2024-09-21T09:43:44.338 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/4 2024-09-21T09:43:44.338 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T09:43:44.642 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T09:43:44.642 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:44.642 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-21T09:43:44.642 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:44.642 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:44.642 INFO:teuthology.orchestra.run.smithi196.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T09:43:44.642 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T09:43:44.642 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:44.642 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:43:44.656 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:38 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:43:44.790 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:43:44.793 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:43:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-21T09:43:44.793 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:43:44.793 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-21T09:43:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 6.5 k 2024-09-21T09:43:44.793 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-21T09:43:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 2.1 M 2024-09-21T09:43:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 1.4 M 2024-09-21T09:43:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 240 k 2024-09-21T09:43:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 3.9 M 2024-09-21T09:43:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 15 M 2024-09-21T09:43:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T09:43:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T09:43:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T09:43:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T09:43:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T09:43:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T09:43:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T09:43:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections 2024-09-21T09:43:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-21T09:43:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T09:43:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T09:43:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T09:43:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T09:43:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T09:43:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T09:43:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T09:43:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T09:43:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T09:43:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T09:43:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T09:43:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T09:43:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T09:43:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T09:43:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T09:43:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T09:43:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T09:43:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T09:43:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T09:43:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T09:43:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:44.795 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-21T09:43:44.796 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:43:44.796 INFO:teuthology.orchestra.run.smithi098.stdout:Install 34 Packages 2024-09-21T09:43:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:44.796 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 27 M 2024-09-21T09:43:44.796 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 96 M 2024-09-21T09:43:44.796 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-21T09:43:44.960 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph 2024-09-21T09:43:45.469 INFO:teuthology.orchestra.run.smithi098.stdout:(1/34): ceph-17.2.7-1649.gfbd1a6ab.el9.x86_64.r 53 kB/s | 6.5 kB 00:00 2024-09-21T09:43:45.537 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:38 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:43:45.654 INFO:teuthology.orchestra.run.smithi098.stdout:(2/34): ceph-mgr-17.2.7-1649.gfbd1a6ab.el9.x86_ 4.6 MB/s | 1.4 MB 00:00 2024-09-21T09:43:45.678 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:43:45.680 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:43:45.680 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-21T09:43:45.680 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:43:45.680 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-21T09:43:45.680 INFO:teuthology.orchestra.run.smithi196.stdout: ceph x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 6.5 k 2024-09-21T09:43:45.680 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-21T09:43:45.680 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 2.1 M 2024-09-21T09:43:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 1.4 M 2024-09-21T09:43:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 240 k 2024-09-21T09:43:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 3.9 M 2024-09-21T09:43:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 15 M 2024-09-21T09:43:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T09:43:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T09:43:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T09:43:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T09:43:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T09:43:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T09:43:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T09:43:45.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections 2024-09-21T09:43:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-21T09:43:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T09:43:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T09:43:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T09:43:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T09:43:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T09:43:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T09:43:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T09:43:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T09:43:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T09:43:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T09:43:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T09:43:45.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T09:43:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T09:43:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T09:43:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T09:43:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T09:43:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T09:43:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T09:43:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T09:43:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T09:43:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:45.683 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-21T09:43:45.683 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:43:45.683 INFO:teuthology.orchestra.run.smithi196.stdout:Install 34 Packages 2024-09-21T09:43:45.683 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:45.684 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 27 M 2024-09-21T09:43:45.684 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 96 M 2024-09-21T09:43:45.684 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-21T09:43:45.703 INFO:teuthology.orchestra.run.smithi098.stdout:(3/34): ceph-mds-17.2.7-1649.gfbd1a6ab.el9.x86_ 5.8 MB/s | 2.1 MB 00:00 2024-09-21T09:43:45.795 INFO:teuthology.orchestra.run.smithi098.stdout:(4/34): ceph-mon-17.2.7-1649.gfbd1a6ab.el9.x86_ 12 MB/s | 3.9 MB 00:00 2024-09-21T09:43:45.820 INFO:teuthology.orchestra.run.smithi098.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1649.gfbd1 2.0 MB/s | 240 kB 00:00 2024-09-21T09:43:46.104 INFO:teuthology.orchestra.run.smithi196.stdout:(1/34): ceph-17.2.7-1649.gfbd1a6ab.el9.x86_64.r 41 kB/s | 6.5 kB 00:00 2024-09-21T09:43:46.154 INFO:teuthology.orchestra.run.smithi098.stdout:(6/34): ceph-osd-17.2.7-1649.gfbd1a6ab.el9.x86_ 31 MB/s | 15 MB 00:00 2024-09-21T09:43:46.263 INFO:teuthology.orchestra.run.smithi196.stdout:(2/34): ceph-mgr-17.2.7-1649.gfbd1a6ab.el9.x86_ 4.4 MB/s | 1.4 MB 00:00 2024-09-21T09:43:46.313 INFO:teuthology.orchestra.run.smithi196.stdout:(3/34): ceph-mds-17.2.7-1649.gfbd1a6ab.el9.x86_ 5.6 MB/s | 2.1 MB 00:00 2024-09-21T09:43:46.380 INFO:teuthology.orchestra.run.smithi196.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1649.gfbd1 3.5 MB/s | 240 kB 00:00 2024-09-21T09:43:46.454 INFO:teuthology.orchestra.run.smithi098.stdout:(7/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 384 kB/s | 253 kB 00:00 2024-09-21T09:43:46.571 INFO:teuthology.orchestra.run.smithi098.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 1.1 MB/s | 135 kB 00:00 2024-09-21T09:43:46.572 INFO:teuthology.orchestra.run.smithi196.stdout:(5/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-21T09:43:46.613 INFO:teuthology.orchestra.run.smithi098.stdout:(9/34): python3-ply-3.11-14.el9.noarch.rpm 232 kB/s | 106 kB 00:00 2024-09-21T09:43:46.688 INFO:teuthology.orchestra.run.smithi098.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-09-21T09:43:46.756 INFO:teuthology.orchestra.run.smithi196.stdout:(6/34): ceph-osd-17.2.7-1649.gfbd1a6ab.el9.x86_ 31 MB/s | 15 MB 00:00 2024-09-21T09:43:46.763 INFO:teuthology.orchestra.run.smithi098.stdout:(11/34): python3-cryptography-36.0.1-4.el9.x86_ 1.3 MB/s | 1.2 MB 00:00 2024-09-21T09:43:46.814 INFO:teuthology.orchestra.run.smithi098.stdout:(12/34): python3-urllib3-1.26.5-6.el9.noarch.rp 1.0 MB/s | 215 kB 00:00 2024-09-21T09:43:46.831 INFO:teuthology.orchestra.run.smithi196.stdout:(7/34): ceph-mon-17.2.7-1649.gfbd1a6ab.el9.x86_ 5.4 MB/s | 3.9 MB 00:00 2024-09-21T09:43:46.839 INFO:teuthology.orchestra.run.smithi098.stdout:(13/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-21T09:43:46.864 INFO:teuthology.orchestra.run.smithi196.stdout:(8/34): python3-cryptography-36.0.1-4.el9.x86_6 4.2 MB/s | 1.2 MB 00:00 2024-09-21T09:43:46.865 INFO:teuthology.orchestra.run.smithi098.stdout:(14/34): python3-markupsafe-1.1.1-12.el9.x86_64 346 kB/s | 35 kB 00:00 2024-09-21T09:43:46.890 INFO:teuthology.orchestra.run.smithi098.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-21T09:43:46.923 INFO:teuthology.orchestra.run.smithi098.stdout:(16/34): python3-toml-0.10.2-6.el9.noarch.rpm 710 kB/s | 42 kB 00:00 2024-09-21T09:43:46.948 INFO:teuthology.orchestra.run.smithi196.stdout:(9/34): python3-requests-2.25.1-8.el9.noarch.rp 1.5 MB/s | 125 kB 00:00 2024-09-21T09:43:46.948 INFO:teuthology.orchestra.run.smithi098.stdout:(17/34): python3-packaging-20.9-5.el9.noarch.rp 575 kB/s | 77 kB 00:00 2024-09-21T09:43:46.973 INFO:teuthology.orchestra.run.smithi196.stdout:(10/34): python3-ply-3.11-14.el9.noarch.rpm 490 kB/s | 106 kB 00:00 2024-09-21T09:43:46.974 INFO:teuthology.orchestra.run.smithi098.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 517 kB/s | 43 kB 00:00 2024-09-21T09:43:46.998 INFO:teuthology.orchestra.run.smithi196.stdout:(11/34): python3-pycparser-2.20-6.el9.noarch.rp 808 kB/s | 135 kB 00:00 2024-09-21T09:43:46.999 INFO:teuthology.orchestra.run.smithi098.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 691 kB/s | 11 kB 00:00 2024-09-21T09:43:47.016 INFO:teuthology.orchestra.run.smithi098.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 1.2 MB/s | 18 kB 00:00 2024-09-21T09:43:47.026 INFO:teuthology.orchestra.run.smithi196.stdout:(12/34): python3-urllib3-1.26.5-6.el9.noarch.rp 2.7 MB/s | 215 kB 00:00 2024-09-21T09:43:47.050 INFO:teuthology.orchestra.run.smithi098.stdout:(21/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-21T09:43:47.067 INFO:teuthology.orchestra.run.smithi098.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-09-21T09:43:47.084 INFO:teuthology.orchestra.run.smithi098.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 580 kB/s | 19 kB 00:00 2024-09-21T09:43:47.109 INFO:teuthology.orchestra.run.smithi098.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 472 kB/s | 20 kB 00:00 2024-09-21T09:43:47.109 INFO:teuthology.orchestra.run.smithi196.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 314 kB/s | 35 kB 00:00 2024-09-21T09:43:47.134 INFO:teuthology.orchestra.run.smithi098.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-09-21T09:43:47.159 INFO:teuthology.orchestra.run.smithi098.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-21T09:43:47.160 INFO:teuthology.orchestra.run.smithi196.stdout:(14/34): python3-packaging-20.9-5.el9.noarch.rp 577 kB/s | 77 kB 00:00 2024-09-21T09:43:47.185 INFO:teuthology.orchestra.run.smithi098.stdout:(27/34): python3-portend-3.1.0-2.el9.noarch.rpm 664 kB/s | 16 kB 00:00 2024-09-21T09:43:47.193 INFO:teuthology.orchestra.run.smithi196.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 610 kB/s | 51 kB 00:00 2024-09-21T09:43:47.210 INFO:teuthology.orchestra.run.smithi098.stdout:(28/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-09-21T09:43:47.219 INFO:teuthology.orchestra.run.smithi196.stdout:(16/34): python3-toml-0.10.2-6.el9.noarch.rpm 711 kB/s | 42 kB 00:00 2024-09-21T09:43:47.235 INFO:teuthology.orchestra.run.smithi098.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-21T09:43:47.244 INFO:teuthology.orchestra.run.smithi196.stdout:(17/34): python3-mako-1.1.4-6.el9.noarch.rpm 636 kB/s | 172 kB 00:00 2024-09-21T09:43:47.260 INFO:teuthology.orchestra.run.smithi098.stdout:(30/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-21T09:43:47.294 INFO:teuthology.orchestra.run.smithi098.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-21T09:43:47.295 INFO:teuthology.orchestra.run.smithi196.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 431 kB/s | 43 kB 00:00 2024-09-21T09:43:47.320 INFO:teuthology.orchestra.run.smithi196.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 463 kB/s | 11 kB 00:00 2024-09-21T09:43:47.320 INFO:teuthology.orchestra.run.smithi098.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 798 kB/s | 20 kB 00:00 2024-09-21T09:43:47.345 INFO:teuthology.orchestra.run.smithi196.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-21T09:43:47.345 INFO:teuthology.orchestra.run.smithi098.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-09-21T09:43:47.370 INFO:teuthology.orchestra.run.smithi196.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-09-21T09:43:47.378 INFO:teuthology.orchestra.run.smithi098.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-09-21T09:43:47.383 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:43:47.383 INFO:teuthology.orchestra.run.smithi098.stdout:Total 11 MB/s | 27 MB 00:02 2024-09-21T09:43:47.395 INFO:teuthology.orchestra.run.smithi196.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-09-21T09:43:47.420 INFO:teuthology.orchestra.run.smithi196.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-21T09:43:47.446 INFO:teuthology.orchestra.run.smithi196.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-21T09:43:47.471 INFO:teuthology.orchestra.run.smithi196.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-09-21T09:43:47.496 INFO:teuthology.orchestra.run.smithi196.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-21T09:43:47.518 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-21T09:43:47.521 INFO:teuthology.orchestra.run.smithi196.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-21T09:43:47.547 INFO:teuthology.orchestra.run.smithi196.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-21T09:43:47.554 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-21T09:43:47.554 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-21T09:43:47.572 INFO:teuthology.orchestra.run.smithi196.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-21T09:43:47.598 INFO:teuthology.orchestra.run.smithi196.stdout:(30/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-21T09:43:47.623 INFO:teuthology.orchestra.run.smithi196.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-21T09:43:47.648 INFO:teuthology.orchestra.run.smithi196.stdout:(32/34): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-21T09:43:47.673 INFO:teuthology.orchestra.run.smithi196.stdout:(33/34): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-09-21T09:43:47.699 INFO:teuthology.orchestra.run.smithi196.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-21T09:43:47.703 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:43:47.703 INFO:teuthology.orchestra.run.smithi196.stdout:Total 13 MB/s | 27 MB 00:02 2024-09-21T09:43:47.832 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-21T09:43:47.869 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-21T09:43:47.869 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-21T09:43:47.916 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-21T09:43:47.916 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-21T09:43:48.239 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-21T09:43:48.239 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-21T09:43:49.182 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-21T09:43:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-21T09:43:49.281 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-21T09:43:49.313 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-21T09:43:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-21T09:43:49.449 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-21T09:43:49.492 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-21T09:43:49.510 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-21T09:43:49.532 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-21T09:43:49.542 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-21T09:43:49.568 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-21T09:43:49.575 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-21T09:43:49.598 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-21T09:43:49.632 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-21T09:43:49.666 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-21T09:43:49.668 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-21T09:43:49.706 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-21T09:43:49.741 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-21T09:43:49.746 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-21T09:43:49.776 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-21T09:43:49.785 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-21T09:43:49.821 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-21T09:43:49.824 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-21T09:43:49.851 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-21T09:43:49.856 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-21T09:43:49.885 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-21T09:43:49.919 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-21T09:43:49.937 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-21T09:43:49.959 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-21T09:43:49.994 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-21T09:43:50.029 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-21T09:43:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-21T09:43:50.084 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-21T09:43:50.117 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-21T09:43:50.173 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-21T09:43:50.190 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-21T09:43:50.226 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-21T09:43:50.265 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-21T09:43:50.313 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-21T09:43:50.337 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-21T09:43:50.369 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-21T09:43:50.417 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-21T09:43:50.487 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-21T09:43:50.525 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-21T09:43:50.564 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-21T09:43:50.575 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-21T09:43:50.619 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-21T09:43:50.638 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-21T09:43:50.689 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-21T09:43:50.777 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-21T09:43:50.810 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-21T09:43:50.834 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-21T09:43:50.880 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-21T09:43:50.926 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-21T09:43:50.950 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-21T09:43:51.039 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-21T09:43:51.079 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-21T09:43:51.082 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1649.gfbd1a6ab.el9. 29/34 2024-09-21T09:43:51.112 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 30/34 2024-09-21T09:43:51.137 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 30/34 2024-09-21T09:43:51.137 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:43:51.138 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T09:43:51.138 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T09:43:51.138 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T09:43:51.138 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:51.195 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-21T09:43:51.349 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1649.gfbd1a6ab.el9. 29/34 2024-09-21T09:43:51.381 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 30/34 2024-09-21T09:43:51.407 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 30/34 2024-09-21T09:43:51.407 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:43:51.407 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T09:43:51.408 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T09:43:51.408 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T09:43:51.408 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:52.489 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 31/34 2024-09-21T09:43:52.518 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 31/34 2024-09-21T09:43:52.518 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:43:52.518 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T09:43:52.518 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T09:43:52.519 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T09:43:52.519 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:52.766 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 31/34 2024-09-21T09:43:52.796 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 31/34 2024-09-21T09:43:52.796 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:43:52.796 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T09:43:52.796 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T09:43:52.796 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T09:43:52.796 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:52.890 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 32/34 2024-09-21T09:43:52.915 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 32/34 2024-09-21T09:43:52.915 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:43:52.915 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T09:43:52.915 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T09:43:52.915 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T09:43:52.915 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:53.116 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 33/34 2024-09-21T09:43:53.140 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 33/34 2024-09-21T09:43:53.140 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:43:53.140 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T09:43:53.140 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T09:43:53.140 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T09:43:53.140 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:53.204 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 34/34 2024-09-21T09:43:53.283 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 32/34 2024-09-21T09:43:53.308 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 32/34 2024-09-21T09:43:53.308 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:43:53.308 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T09:43:53.308 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T09:43:53.308 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T09:43:53.308 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:53.509 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 33/34 2024-09-21T09:43:53.533 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 33/34 2024-09-21T09:43:53.533 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:43:53.533 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T09:43:53.533 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T09:43:53.533 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T09:43:53.533 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:53.598 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 34/34 2024-09-21T09:43:54.747 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 34/34 2024-09-21T09:43:54.747 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/34 2024-09-21T09:43:54.747 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/34 2024-09-21T09:43:54.748 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/34 2024-09-21T09:43:54.748 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/34 2024-09-21T09:43:54.748 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 5/34 2024-09-21T09:43:54.748 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1649.gfbd1a6ab.el9. 6/34 2024-09-21T09:43:54.748 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-21T09:43:54.748 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-21T09:43:54.748 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-21T09:43:54.748 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-21T09:43:54.749 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-21T09:43:54.749 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-21T09:43:54.749 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-21T09:43:54.749 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-21T09:43:54.749 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-21T09:43:54.749 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-21T09:43:54.749 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-21T09:43:54.750 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-21T09:43:54.750 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-21T09:43:54.750 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-21T09:43:54.750 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-21T09:43:54.750 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-21T09:43:54.750 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-21T09:43:54.751 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-21T09:43:54.751 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-21T09:43:54.751 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-21T09:43:54.751 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-21T09:43:54.751 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-21T09:43:54.751 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-21T09:43:54.751 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-21T09:43:54.752 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-21T09:43:54.752 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-21T09:43:54.752 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-21T09:43:55.185 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 34/34 2024-09-21T09:43:55.185 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/34 2024-09-21T09:43:55.185 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/34 2024-09-21T09:43:55.186 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/34 2024-09-21T09:43:55.186 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/34 2024-09-21T09:43:55.186 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 5/34 2024-09-21T09:43:55.186 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1649.gfbd1a6ab.el9. 6/34 2024-09-21T09:43:55.186 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-21T09:43:55.187 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-21T09:43:55.187 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-21T09:43:55.187 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-21T09:43:55.187 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-21T09:43:55.187 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-21T09:43:55.188 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-21T09:43:55.188 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-21T09:43:55.188 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-21T09:43:55.188 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-21T09:43:55.189 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-21T09:43:55.189 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-21T09:43:55.189 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-21T09:43:55.189 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-21T09:43:55.189 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-21T09:43:55.190 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-21T09:43:55.190 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-21T09:43:55.190 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-21T09:43:55.190 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-21T09:43:55.190 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-21T09:43:55.191 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-21T09:43:55.191 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-21T09:43:55.191 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-21T09:43:55.191 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-21T09:43:55.191 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-21T09:43:55.191 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-21T09:43:55.192 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-21T09:43:55.768 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-21T09:43:55.768 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:55.769 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-21T09:43:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:43:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T09:43:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T09:43:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T09:43:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T09:43:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T09:43:55.769 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T09:43:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T09:43:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T09:43:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T09:43:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T09:43:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T09:43:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T09:43:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T09:43:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T09:43:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T09:43:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T09:43:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T09:43:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T09:43:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T09:43:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T09:43:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T09:43:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T09:43:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T09:43:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T09:43:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T09:43:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T09:43:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T09:43:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T09:43:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:55.771 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:43:56.182 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-base 2024-09-21T09:43:56.243 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-21T09:43:56.243 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:56.243 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-21T09:43:56.243 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:56.243 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:56.243 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:56.243 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:43:56.243 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:56.243 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:43:56.243 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T09:43:56.243 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T09:43:56.244 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T09:43:56.244 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T09:43:56.244 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T09:43:56.244 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T09:43:56.244 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T09:43:56.244 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T09:43:56.244 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T09:43:56.244 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T09:43:56.244 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T09:43:56.244 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T09:43:56.244 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T09:43:56.244 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T09:43:56.244 INFO:teuthology.orchestra.run.smithi196.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T09:43:56.245 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T09:43:56.245 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T09:43:56.245 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T09:43:56.245 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T09:43:56.245 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T09:43:56.245 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T09:43:56.245 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T09:43:56.245 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T09:43:56.245 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T09:43:56.245 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T09:43:56.245 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T09:43:56.245 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T09:43:56.246 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T09:43:56.246 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:56.246 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:43:56.627 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-base 2024-09-21T09:43:56.765 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:50 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:43:56.855 INFO:teuthology.orchestra.run.smithi098.stdout:Package ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:43:56.896 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:43:56.897 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-21T09:43:56.897 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:43:56.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install cephadm 2024-09-21T09:43:57.210 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:50 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:43:57.299 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:43:57.342 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:43:57.343 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-21T09:43:57.343 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:43:57.413 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install cephadm 2024-09-21T09:43:57.538 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:51 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:43:57.669 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:43:57.669 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:43:57.669 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-21T09:43:57.669 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:43:57.669 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-21T09:43:57.670 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 83 k 2024-09-21T09:43:57.670 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:57.670 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-21T09:43:57.670 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:43:57.670 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-21T09:43:57.670 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:57.670 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 83 k 2024-09-21T09:43:57.671 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 375 k 2024-09-21T09:43:57.671 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-21T09:43:57.857 INFO:teuthology.orchestra.run.smithi098.stdout:cephadm-17.2.7-1649.gfbd1a6ab.el9.noarch.rpm 443 kB/s | 83 kB 00:00 2024-09-21T09:43:57.858 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:43:57.858 INFO:teuthology.orchestra.run.smithi098.stdout:Total 439 kB/s | 83 kB 00:00 2024-09-21T09:43:57.858 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-21T09:43:57.863 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-21T09:43:57.863 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-21T09:43:57.865 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-21T09:43:57.866 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-21T09:43:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-21T09:43:57.990 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:50 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:43:58.121 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:43:58.121 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:43:58.121 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-21T09:43:58.121 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:43:58.121 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-21T09:43:58.121 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 83 k 2024-09-21T09:43:58.121 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:58.121 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-21T09:43:58.122 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:43:58.122 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-09-21T09:43:58.122 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:58.122 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 83 k 2024-09-21T09:43:58.122 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 375 k 2024-09-21T09:43:58.122 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-21T09:43:58.316 INFO:teuthology.orchestra.run.smithi196.stdout:cephadm-17.2.7-1649.gfbd1a6ab.el9.noarch.rpm 427 kB/s | 83 kB 00:00 2024-09-21T09:43:58.317 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:43:58.317 INFO:teuthology.orchestra.run.smithi196.stdout:Total 424 kB/s | 83 kB 00:00 2024-09-21T09:43:58.317 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-21T09:43:58.322 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-21T09:43:58.322 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-21T09:43:58.324 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-21T09:43:58.324 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-21T09:43:58.382 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:43:58.402 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-21T09:43:58.502 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:43:58.857 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:43:58.863 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:43:58.953 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:43:59.107 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:43:59.107 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:59.108 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-21T09:43:59.108 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:43:59.108 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:43:59.108 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:43:59.312 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:43:59.331 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T09:43:59.564 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:43:59.564 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:59.564 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-21T09:43:59.564 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:43:59.564 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:43:59.564 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:43:59.787 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T09:43:59.922 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:53 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:44:00.057 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:44:00.057 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:00.057 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-09-21T09:44:00.058 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:00.058 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-21T09:44:00.058 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 137 k 2024-09-21T09:44:00.058 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:00.058 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-21T09:44:00.058 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:00.058 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-21T09:44:00.058 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:00.058 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 137 k 2024-09-21T09:44:00.058 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 379 k 2024-09-21T09:44:00.058 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-21T09:44:00.264 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-immutable-object-cache-17.2.7-1649.gfbd1a6 667 kB/s | 137 kB 00:00 2024-09-21T09:44:00.265 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:44:00.265 INFO:teuthology.orchestra.run.smithi098.stdout:Total 662 kB/s | 137 kB 00:00 2024-09-21T09:44:00.265 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-21T09:44:00.273 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-21T09:44:00.273 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-21T09:44:00.331 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-21T09:44:00.332 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-21T09:44:00.374 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:53 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:44:00.492 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-21T09:44:00.511 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:44:00.512 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:00.512 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-09-21T09:44:00.512 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:00.512 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-21T09:44:00.512 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 137 k 2024-09-21T09:44:00.512 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:00.512 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-21T09:44:00.512 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:00.513 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-09-21T09:44:00.513 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:00.513 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 137 k 2024-09-21T09:44:00.513 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 379 k 2024-09-21T09:44:00.513 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-21T09:44:00.557 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab. 1/1 2024-09-21T09:44:00.582 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab. 1/1 2024-09-21T09:44:00.582 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:44:00.582 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T09:44:00.582 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:00.722 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-immutable-object-cache-17.2.7-1649.gfbd1a6 658 kB/s | 137 kB 00:00 2024-09-21T09:44:00.723 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:44:00.723 INFO:teuthology.orchestra.run.smithi196.stdout:Total 653 kB/s | 137 kB 00:00 2024-09-21T09:44:00.723 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-21T09:44:00.730 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-21T09:44:00.730 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-21T09:44:00.787 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-21T09:44:00.787 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-21T09:44:00.939 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-21T09:44:01.006 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab. 1/1 2024-09-21T09:44:01.031 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab. 1/1 2024-09-21T09:44:01.031 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:44:01.031 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T09:44:01.031 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:01.579 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab. 1/1 2024-09-21T09:44:01.579 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:01.580 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-21T09:44:01.580 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:44:01.580 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:01.580 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:44:01.792 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr 2024-09-21T09:44:02.009 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab. 1/1 2024-09-21T09:44:02.009 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:02.009 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-21T09:44:02.010 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:44:02.010 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:02.010 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:44:02.218 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr 2024-09-21T09:44:02.388 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:56 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:44:02.479 INFO:teuthology.orchestra.run.smithi098.stdout:Package ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:44:02.521 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:44:02.522 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-21T09:44:02.522 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:44:02.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T09:44:02.812 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:55 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:44:02.904 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:44:02.945 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:44:02.946 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-21T09:44:02.946 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:44:03.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T09:44:03.155 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:57 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:44:03.286 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:44:03.287 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:03.287 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-21T09:44:03.287 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:03.287 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-21T09:44:03.288 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 3.4 M 2024-09-21T09:44:03.288 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-21T09:44:03.288 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 24 k 2024-09-21T09:44:03.288 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 15 k 2024-09-21T09:44:03.288 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-21T09:44:03.288 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T09:44:03.288 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T09:44:03.288 INFO:teuthology.orchestra.run.smithi098.stdout:Installing weak dependencies: 2024-09-21T09:44:03.288 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-21T09:44:03.288 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:03.288 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-21T09:44:03.288 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:03.288 INFO:teuthology.orchestra.run.smithi098.stdout:Install 7 Packages 2024-09-21T09:44:03.288 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:03.289 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 3.7 M 2024-09-21T09:44:03.289 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 80 M 2024-09-21T09:44:03.289 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-21T09:44:03.590 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:56 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:44:03.722 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:44:03.722 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:03.723 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-21T09:44:03.723 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:03.723 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-21T09:44:03.723 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 3.4 M 2024-09-21T09:44:03.723 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-21T09:44:03.723 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 24 k 2024-09-21T09:44:03.723 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 15 k 2024-09-21T09:44:03.723 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-21T09:44:03.723 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T09:44:03.724 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T09:44:03.724 INFO:teuthology.orchestra.run.smithi196.stdout:Installing weak dependencies: 2024-09-21T09:44:03.724 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-21T09:44:03.724 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:03.724 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-21T09:44:03.724 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:03.724 INFO:teuthology.orchestra.run.smithi196.stdout:Install 7 Packages 2024-09-21T09:44:03.724 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:03.725 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 3.7 M 2024-09-21T09:44:03.725 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 80 M 2024-09-21T09:44:03.725 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-21T09:44:03.810 INFO:teuthology.orchestra.run.smithi098.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1649.gfbd 173 kB/s | 24 kB 00:00 2024-09-21T09:44:03.835 INFO:teuthology.orchestra.run.smithi098.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1649.gfbd1 88 kB/s | 15 kB 00:00 2024-09-21T09:44:03.877 INFO:teuthology.orchestra.run.smithi098.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-21T09:44:03.902 INFO:teuthology.orchestra.run.smithi098.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-21T09:44:03.909 INFO:teuthology.orchestra.run.smithi196.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1649.gfbd 161 kB/s | 24 kB 00:00 2024-09-21T09:44:03.928 INFO:teuthology.orchestra.run.smithi098.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 444 kB/s | 41 kB 00:00 2024-09-21T09:44:03.934 INFO:teuthology.orchestra.run.smithi196.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1649.gfbd1 83 kB/s | 15 kB 00:00 2024-09-21T09:44:03.962 INFO:teuthology.orchestra.run.smithi098.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-09-21T09:44:03.976 INFO:teuthology.orchestra.run.smithi196.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 136 kB/s | 9.0 kB 00:00 2024-09-21T09:44:04.001 INFO:teuthology.orchestra.run.smithi196.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-21T09:44:04.026 INFO:teuthology.orchestra.run.smithi196.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 445 kB/s | 41 kB 00:00 2024-09-21T09:44:04.051 INFO:teuthology.orchestra.run.smithi196.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-21T09:44:04.062 INFO:teuthology.orchestra.run.smithi098.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1649.gfbd1a6ab 8.5 MB/s | 3.4 MB 00:00 2024-09-21T09:44:04.065 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:44:04.065 INFO:teuthology.orchestra.run.smithi098.stdout:Total 4.7 MB/s | 3.7 MB 00:00 2024-09-21T09:44:04.066 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-21T09:44:04.074 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-21T09:44:04.074 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-21T09:44:04.160 INFO:teuthology.orchestra.run.smithi196.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1649.gfbd1a6ab 8.3 MB/s | 3.4 MB 00:00 2024-09-21T09:44:04.164 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:44:04.164 INFO:teuthology.orchestra.run.smithi196.stdout:Total 8.3 MB/s | 3.7 MB 00:00 2024-09-21T09:44:04.164 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-21T09:44:04.168 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-21T09:44:04.168 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-21T09:44:04.172 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-21T09:44:04.172 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-21T09:44:04.262 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-21T09:44:04.262 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-21T09:44:04.474 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-21T09:44:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-21T09:44:04.572 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-21T09:44:04.572 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-21T09:44:04.626 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-21T09:44:04.658 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-21T09:44:04.659 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-21T09:44:04.688 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1649.gfbd1a6ab.el9.n 5/7 2024-09-21T09:44:04.695 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-21T09:44:04.750 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-21T09:44:04.790 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-21T09:44:04.819 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1649.gfbd1a6ab.el9.n 5/7 2024-09-21T09:44:05.676 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1649.gfbd1a6ab.el9. 6/7 2024-09-21T09:44:05.708 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarc 7/7 2024-09-21T09:44:05.811 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1649.gfbd1a6ab.el9. 6/7 2024-09-21T09:44:05.856 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarc 7/7 2024-09-21T09:44:06.797 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarc 7/7 2024-09-21T09:44:06.797 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1649.gfbd1a6ab.el9. 1/7 2024-09-21T09:44:06.797 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarc 2/7 2024-09-21T09:44:06.797 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1649.gfbd1a6ab.el9.n 3/7 2024-09-21T09:44:06.798 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-21T09:44:06.798 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-21T09:44:06.798 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-21T09:44:06.895 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarc 7/7 2024-09-21T09:44:06.896 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1649.gfbd1a6ab.el9. 1/7 2024-09-21T09:44:06.896 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarc 2/7 2024-09-21T09:44:06.896 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1649.gfbd1a6ab.el9.n 3/7 2024-09-21T09:44:06.896 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-21T09:44:06.896 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-21T09:44:06.896 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-21T09:44:07.080 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-21T09:44:07.087 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:07.087 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-21T09:44:07.087 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:44:07.087 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:44:07.087 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:44:07.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-21T09:44:07.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-21T09:44:07.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T09:44:07.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T09:44:07.087 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:07.087 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:44:07.172 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-21T09:44:07.172 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:07.172 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-21T09:44:07.172 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:44:07.172 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:44:07.172 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:44:07.172 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-21T09:44:07.172 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-21T09:44:07.172 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T09:44:07.173 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T09:44:07.173 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:07.173 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:44:07.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T09:44:07.404 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T09:44:07.879 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:01 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:44:07.982 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:00 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:44:08.012 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:44:08.013 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-09-21T09:44:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-21T09:44:08.013 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-09-21T09:44:08.013 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-21T09:44:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 7.4 M 2024-09-21T09:44:08.013 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-21T09:44:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T09:44:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T09:44:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T09:44:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T09:44:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T09:44:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T09:44:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T09:44:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T09:44:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T09:44:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T09:44:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T09:44:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:08.014 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-21T09:44:08.014 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-09-21T09:44:08.015 INFO:teuthology.orchestra.run.smithi098.stdout:Install 12 Packages 2024-09-21T09:44:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:08.015 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 38 M 2024-09-21T09:44:08.015 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 212 M 2024-09-21T09:44:08.015 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-21T09:44:08.115 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:44:08.116 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================= 2024-09-21T09:44:08.116 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-21T09:44:08.116 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================= 2024-09-21T09:44:08.116 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-21T09:44:08.116 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 7.4 M 2024-09-21T09:44:08.116 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-21T09:44:08.117 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T09:44:08.117 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T09:44:08.117 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T09:44:08.117 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T09:44:08.117 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T09:44:08.117 INFO:teuthology.orchestra.run.smithi196.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T09:44:08.117 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T09:44:08.117 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T09:44:08.117 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T09:44:08.117 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T09:44:08.117 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T09:44:08.117 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:08.117 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-21T09:44:08.117 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================= 2024-09-21T09:44:08.118 INFO:teuthology.orchestra.run.smithi196.stdout:Install 12 Packages 2024-09-21T09:44:08.118 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:08.118 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 38 M 2024-09-21T09:44:08.118 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 212 M 2024-09-21T09:44:08.118 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-21T09:44:08.377 INFO:teuthology.orchestra.run.smithi196.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-21T09:44:08.419 INFO:teuthology.orchestra.run.smithi196.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.4 MB/s | 801 kB 00:00 2024-09-21T09:44:08.435 INFO:teuthology.orchestra.run.smithi196.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 558 kB/s | 33 kB 00:00 2024-09-21T09:44:08.469 INFO:teuthology.orchestra.run.smithi196.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 547 kB/s | 18 kB 00:00 2024-09-21T09:44:08.543 INFO:teuthology.orchestra.run.smithi196.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 24 MB/s | 3.0 MB 00:00 2024-09-21T09:44:08.562 INFO:teuthology.orchestra.run.smithi196.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 417 kB/s | 38 kB 00:00 2024-09-21T09:44:08.621 INFO:teuthology.orchestra.run.smithi098.stdout:(1/12): ceph-mgr-diskprediction-local-17.2.7-16 16 MB/s | 7.4 MB 00:00 2024-09-21T09:44:08.687 INFO:teuthology.orchestra.run.smithi196.stdout:(7/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 34 MB/s | 4.9 MB 00:00 2024-09-21T09:44:08.687 INFO:teuthology.orchestra.run.smithi098.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 493 kB/s | 33 kB 00:00 2024-09-21T09:44:08.712 INFO:teuthology.orchestra.run.smithi196.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-09-21T09:44:08.771 INFO:teuthology.orchestra.run.smithi196.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.2 MB/s | 250 kB 00:00 2024-09-21T09:44:08.813 INFO:teuthology.orchestra.run.smithi098.stdout:(3/12): libquadmath-11.5.0-2.el9.x86_64.rpm 293 kB/s | 191 kB 00:00 2024-09-21T09:44:08.838 INFO:teuthology.orchestra.run.smithi196.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 5.1 MB 00:00 2024-09-21T09:44:08.888 INFO:teuthology.orchestra.run.smithi098.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 243 kB/s | 18 kB 00:00 2024-09-21T09:44:08.947 INFO:teuthology.orchestra.run.smithi098.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 657 kB/s | 38 kB 00:00 2024-09-21T09:44:09.030 INFO:teuthology.orchestra.run.smithi098.stdout:(6/12): libgfortran-11.5.0-2.el9.x86_64.rpm 921 kB/s | 801 kB 00:00 2024-09-21T09:44:09.189 INFO:teuthology.orchestra.run.smithi098.stdout:(7/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 20 MB/s | 4.9 MB 00:00 2024-09-21T09:44:09.196 INFO:teuthology.orchestra.run.smithi196.stdout:(11/12): ceph-mgr-diskprediction-local-17.2.7-1 7.7 MB/s | 7.4 MB 00:00 2024-09-21T09:44:09.214 INFO:teuthology.orchestra.run.smithi098.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-09-21T09:44:09.323 INFO:teuthology.orchestra.run.smithi098.stdout:(9/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.7 MB/s | 3.0 MB 00:00 2024-09-21T09:44:09.349 INFO:teuthology.orchestra.run.smithi098.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.8 MB/s | 250 kB 00:00 2024-09-21T09:44:09.415 INFO:teuthology.orchestra.run.smithi098.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 5.1 MB 00:00 2024-09-21T09:44:09.513 INFO:teuthology.orchestra.run.smithi196.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-09-21T09:44:09.514 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:44:09.514 INFO:teuthology.orchestra.run.smithi196.stdout:Total 27 MB/s | 38 MB 00:01 2024-09-21T09:44:09.870 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-21T09:44:09.896 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-21T09:44:09.896 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-21T09:44:10.405 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-21T09:44:10.406 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-21T09:44:11.159 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-21T09:44:11.245 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T09:44:11.286 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T09:44:11.393 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T09:44:11.429 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T09:44:11.613 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T09:44:11.645 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T09:44:11.731 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T09:44:12.402 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T09:44:12.550 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T09:44:13.523 INFO:teuthology.orchestra.run.smithi098.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.9 MB/s | 16 MB 00:04 2024-09-21T09:44:13.525 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:44:13.525 INFO:teuthology.orchestra.run.smithi098.stdout:Total 6.9 MB/s | 38 MB 00:05 2024-09-21T09:44:13.847 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T09:44:13.882 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-21T09:44:13.908 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-21T09:44:13.908 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-21T09:44:14.424 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-21T09:44:14.425 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-21T09:44:14.873 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T09:44:14.902 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a 12/12 2024-09-21T09:44:15.184 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-21T09:44:15.270 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T09:44:15.313 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T09:44:15.421 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T09:44:15.456 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T09:44:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T09:44:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T09:44:15.752 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T09:44:16.414 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T09:44:16.536 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T09:44:17.609 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a 12/12 2024-09-21T09:44:17.609 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a 1/12 2024-09-21T09:44:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T09:44:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T09:44:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T09:44:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T09:44:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T09:44:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T09:44:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T09:44:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T09:44:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T09:44:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T09:44:17.795 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T09:44:18.054 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T09:44:18.054 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:18.054 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-21T09:44:18.054 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:44:18.054 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T09:44:18.054 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T09:44:18.054 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T09:44:18.054 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T09:44:18.055 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T09:44:18.055 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T09:44:18.055 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T09:44:18.055 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T09:44:18.055 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T09:44:18.055 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T09:44:18.055 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T09:44:18.055 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:18.055 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:44:18.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-rook 2024-09-21T09:44:18.826 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T09:44:18.864 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a 12/12 2024-09-21T09:44:19.007 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:12 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:44:19.141 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:44:19.142 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:19.142 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-21T09:44:19.142 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:19.142 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-21T09:44:19.142 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 47 k 2024-09-21T09:44:19.142 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-21T09:44:19.142 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T09:44:19.142 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T09:44:19.142 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T09:44:19.143 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T09:44:19.143 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T09:44:19.143 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T09:44:19.143 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T09:44:19.143 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T09:44:19.143 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T09:44:19.143 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T09:44:19.143 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T09:44:19.143 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T09:44:19.143 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:19.143 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-21T09:44:19.143 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:19.143 INFO:teuthology.orchestra.run.smithi196.stdout:Install 13 Packages 2024-09-21T09:44:19.144 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:19.144 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 2.2 M 2024-09-21T09:44:19.144 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 26 M 2024-09-21T09:44:19.144 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-21T09:44:19.544 INFO:teuthology.orchestra.run.smithi196.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 204 kB/s | 19 kB 00:00 2024-09-21T09:44:19.570 INFO:teuthology.orchestra.run.smithi196.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 214 kB/s | 26 kB 00:00 2024-09-21T09:44:19.628 INFO:teuthology.orchestra.run.smithi196.stdout:(3/13): ceph-mgr-rook-17.2.7-1649.gfbd1a6ab.el9 267 kB/s | 47 kB 00:00 2024-09-21T09:44:19.670 INFO:teuthology.orchestra.run.smithi196.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 222 kB 00:00 2024-09-21T09:44:19.695 INFO:teuthology.orchestra.run.smithi196.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-21T09:44:19.721 INFO:teuthology.orchestra.run.smithi196.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-21T09:44:19.762 INFO:teuthology.orchestra.run.smithi196.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-21T09:44:19.788 INFO:teuthology.orchestra.run.smithi196.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 211 kB/s | 14 kB 00:00 2024-09-21T09:44:19.813 INFO:teuthology.orchestra.run.smithi196.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 4.4 MB/s | 220 kB 00:00 2024-09-21T09:44:19.838 INFO:teuthology.orchestra.run.smithi196.stdout:(10/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.3 MB/s | 279 kB 00:00 2024-09-21T09:44:19.863 INFO:teuthology.orchestra.run.smithi196.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-09-21T09:44:19.905 INFO:teuthology.orchestra.run.smithi196.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 8.7 MB/s | 1.0 MB 00:00 2024-09-21T09:44:19.931 INFO:teuthology.orchestra.run.smithi196.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 974 kB/s | 90 kB 00:00 2024-09-21T09:44:19.934 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:44:19.934 INFO:teuthology.orchestra.run.smithi196.stdout:Total 2.8 MB/s | 2.2 MB 00:00 2024-09-21T09:44:19.999 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-21T09:44:20.014 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-21T09:44:20.014 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-21T09:44:20.100 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-21T09:44:20.102 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-21T09:44:20.622 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-21T09:44:20.787 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T09:44:20.840 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T09:44:20.896 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T09:44:20.933 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T09:44:20.975 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T09:44:21.085 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T09:44:21.198 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T09:44:21.251 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T09:44:21.517 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a 12/12 2024-09-21T09:44:21.517 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a 1/12 2024-09-21T09:44:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T09:44:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T09:44:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T09:44:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T09:44:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T09:44:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T09:44:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T09:44:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T09:44:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T09:44:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T09:44:21.914 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T09:44:21.915 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:21.915 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-21T09:44:21.915 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:44:21.915 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T09:44:21.915 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T09:44:21.915 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T09:44:21.915 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T09:44:21.915 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T09:44:21.915 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T09:44:21.915 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T09:44:21.916 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T09:44:21.916 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T09:44:21.916 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T09:44:21.916 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T09:44:21.916 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:21.916 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:44:21.961 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T09:44:22.015 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T09:44:22.049 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T09:44:22.095 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T09:44:22.119 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 13/13 2024-09-21T09:44:22.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-rook 2024-09-21T09:44:22.868 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:16 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:44:22.874 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 13/13 2024-09-21T09:44:22.874 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/13 2024-09-21T09:44:22.874 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T09:44:22.874 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T09:44:22.874 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T09:44:22.874 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T09:44:22.875 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T09:44:22.875 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T09:44:22.875 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T09:44:22.875 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T09:44:22.875 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T09:44:22.875 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T09:44:22.875 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T09:44:23.007 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:44:23.008 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:23.009 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-21T09:44:23.009 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:23.009 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-21T09:44:23.009 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 47 k 2024-09-21T09:44:23.009 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-21T09:44:23.009 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T09:44:23.009 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T09:44:23.009 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T09:44:23.009 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T09:44:23.009 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T09:44:23.010 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T09:44:23.010 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T09:44:23.010 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T09:44:23.010 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T09:44:23.010 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T09:44:23.010 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T09:44:23.010 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T09:44:23.010 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:23.010 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-21T09:44:23.010 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:23.010 INFO:teuthology.orchestra.run.smithi098.stdout:Install 13 Packages 2024-09-21T09:44:23.010 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:23.011 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 2.2 M 2024-09-21T09:44:23.011 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 26 M 2024-09-21T09:44:23.011 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-21T09:44:23.204 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T09:44:23.204 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:23.204 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-21T09:44:23.204 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:44:23.204 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T09:44:23.204 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T09:44:23.204 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T09:44:23.204 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T09:44:23.204 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T09:44:23.205 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T09:44:23.205 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T09:44:23.205 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T09:44:23.205 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T09:44:23.205 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T09:44:23.205 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T09:44:23.205 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T09:44:23.205 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:23.205 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:44:23.267 INFO:teuthology.orchestra.run.smithi098.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 239 kB/s | 19 kB 00:00 2024-09-21T09:44:23.292 INFO:teuthology.orchestra.run.smithi098.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 243 kB/s | 26 kB 00:00 2024-09-21T09:44:23.342 INFO:teuthology.orchestra.run.smithi098.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 222 kB 00:00 2024-09-21T09:44:23.376 INFO:teuthology.orchestra.run.smithi098.stdout:(4/13): ceph-mgr-rook-17.2.7-1649.gfbd1a6ab.el9 251 kB/s | 47 kB 00:00 2024-09-21T09:44:23.401 INFO:teuthology.orchestra.run.smithi098.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-21T09:44:23.426 INFO:teuthology.orchestra.run.smithi098.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.2 MB/s | 279 kB 00:00 2024-09-21T09:44:23.468 INFO:teuthology.orchestra.run.smithi098.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-21T09:44:23.479 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T09:44:23.493 INFO:teuthology.orchestra.run.smithi098.stdout:(8/13): python3-requests-oauthlib-1.3.0-12.el9. 456 kB/s | 54 kB 00:00 2024-09-21T09:44:23.519 INFO:teuthology.orchestra.run.smithi098.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 165 kB/s | 14 kB 00:00 2024-09-21T09:44:23.552 INFO:teuthology.orchestra.run.smithi098.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.2 MB/s | 59 kB 00:00 2024-09-21T09:44:23.578 INFO:teuthology.orchestra.run.smithi098.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 2.0 MB/s | 220 kB 00:00 2024-09-21T09:44:23.603 INFO:teuthology.orchestra.run.smithi098.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-21T09:44:23.645 INFO:teuthology.orchestra.run.smithi098.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-21T09:44:23.648 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:44:23.649 INFO:teuthology.orchestra.run.smithi098.stdout:Total 3.5 MB/s | 2.2 MB 00:00 2024-09-21T09:44:23.714 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-21T09:44:23.729 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-21T09:44:23.729 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-21T09:44:23.824 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-21T09:44:23.826 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-21T09:44:24.063 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:17 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:44:24.194 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:44:24.195 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:24.195 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-21T09:44:24.195 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:24.195 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-21T09:44:24.195 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 124 k 2024-09-21T09:44:24.195 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-21T09:44:24.195 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T09:44:24.195 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T09:44:24.195 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T09:44:24.195 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T09:44:24.195 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T09:44:24.195 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:24.196 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-21T09:44:24.196 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:24.196 INFO:teuthology.orchestra.run.smithi196.stdout:Install 6 Packages 2024-09-21T09:44:24.196 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:24.196 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 7.0 M 2024-09-21T09:44:24.196 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 33 M 2024-09-21T09:44:24.196 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-21T09:44:24.335 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-21T09:44:24.509 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T09:44:24.563 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T09:44:24.619 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T09:44:24.656 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T09:44:24.697 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T09:44:24.784 INFO:teuthology.orchestra.run.smithi196.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1649.gfbd1a6ab.e 608 kB/s | 124 kB 00:00 2024-09-21T09:44:24.800 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T09:44:24.809 INFO:teuthology.orchestra.run.smithi196.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-21T09:44:24.868 INFO:teuthology.orchestra.run.smithi196.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.4 MB/s | 548 kB 00:00 2024-09-21T09:44:24.893 INFO:teuthology.orchestra.run.smithi196.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 690 kB/s | 58 kB 00:00 2024-09-21T09:44:24.910 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T09:44:24.927 INFO:teuthology.orchestra.run.smithi196.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.3 MB/s | 74 kB 00:00 2024-09-21T09:44:24.957 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T09:44:25.691 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T09:44:25.747 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T09:44:25.780 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T09:44:25.802 INFO:teuthology.orchestra.run.smithi196.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.9 MB/s | 6.0 MB 00:01 2024-09-21T09:44:25.804 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:44:25.804 INFO:teuthology.orchestra.run.smithi196.stdout:Total 4.3 MB/s | 7.0 MB 00:01 2024-09-21T09:44:25.825 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T09:44:25.850 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 13/13 2024-09-21T09:44:25.874 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-21T09:44:25.885 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-21T09:44:25.886 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-21T09:44:25.984 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-21T09:44:25.985 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-21T09:44:26.302 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-21T09:44:26.466 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T09:44:26.513 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T09:44:26.666 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 13/13 2024-09-21T09:44:26.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/13 2024-09-21T09:44:26.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T09:44:26.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T09:44:26.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T09:44:26.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T09:44:26.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T09:44:26.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T09:44:26.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T09:44:26.667 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T09:44:26.667 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T09:44:26.667 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T09:44:26.667 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T09:44:27.003 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T09:44:27.003 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:27.003 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-21T09:44:27.003 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:44:27.003 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T09:44:27.003 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T09:44:27.004 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T09:44:27.004 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T09:44:27.004 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T09:44:27.004 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T09:44:27.004 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T09:44:27.004 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T09:44:27.004 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T09:44:27.004 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T09:44:27.004 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T09:44:27.004 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T09:44:27.004 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:27.004 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:44:27.028 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T09:44:27.123 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T09:44:27.186 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T09:44:27.216 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 6/6 2024-09-21T09:44:27.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T09:44:27.838 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:21 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:44:27.969 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:44:27.970 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-21T09:44:27.970 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:27.970 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-21T09:44:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 124 k 2024-09-21T09:44:27.970 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-21T09:44:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T09:44:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T09:44:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T09:44:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T09:44:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T09:44:27.971 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:27.971 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-21T09:44:27.971 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:27.971 INFO:teuthology.orchestra.run.smithi098.stdout:Install 6 Packages 2024-09-21T09:44:27.971 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:27.971 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 7.0 M 2024-09-21T09:44:27.971 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 33 M 2024-09-21T09:44:27.972 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-21T09:44:27.972 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 6/6 2024-09-21T09:44:27.972 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/6 2024-09-21T09:44:27.972 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T09:44:27.973 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T09:44:27.973 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T09:44:27.973 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T09:44:28.242 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T09:44:28.242 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:28.242 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-21T09:44:28.242 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:44:28.242 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T09:44:28.242 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T09:44:28.242 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T09:44:28.242 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T09:44:28.242 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T09:44:28.243 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:28.243 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:44:28.495 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-fuse 2024-09-21T09:44:28.573 INFO:teuthology.orchestra.run.smithi098.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-09-21T09:44:28.640 INFO:teuthology.orchestra.run.smithi098.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1649.gfbd1a6ab.e 567 kB/s | 124 kB 00:00 2024-09-21T09:44:28.673 INFO:teuthology.orchestra.run.smithi098.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 5.4 MB/s | 548 kB 00:00 2024-09-21T09:44:28.699 INFO:teuthology.orchestra.run.smithi098.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.1 MB/s | 74 kB 00:00 2024-09-21T09:44:28.724 INFO:teuthology.orchestra.run.smithi098.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 689 kB/s | 58 kB 00:00 2024-09-21T09:44:28.808 INFO:teuthology.orchestra.run.smithi098.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-09-21T09:44:28.810 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:44:28.811 INFO:teuthology.orchestra.run.smithi098.stdout:Total 8.3 MB/s | 7.0 MB 00:00 2024-09-21T09:44:28.884 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-21T09:44:28.895 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-21T09:44:28.895 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-21T09:44:28.995 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-21T09:44:28.997 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-21T09:44:29.081 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:22 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:44:29.213 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:44:29.214 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:29.214 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-21T09:44:29.214 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:29.214 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-21T09:44:29.215 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 813 k 2024-09-21T09:44:29.215 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-21T09:44:29.215 INFO:teuthology.orchestra.run.smithi196.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T09:44:29.215 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:29.215 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-21T09:44:29.215 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:29.215 INFO:teuthology.orchestra.run.smithi196.stdout:Install 2 Packages 2024-09-21T09:44:29.215 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:29.215 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 893 k 2024-09-21T09:44:29.215 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 2.6 M 2024-09-21T09:44:29.215 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-21T09:44:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-21T09:44:29.362 INFO:teuthology.orchestra.run.smithi196.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 637 kB/s | 80 kB 00:00 2024-09-21T09:44:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T09:44:29.512 INFO:teuthology.orchestra.run.smithi196.stdout:(2/2): ceph-fuse-17.2.7-1649.gfbd1a6ab.el9.x86_ 2.9 MB/s | 813 kB 00:00 2024-09-21T09:44:29.513 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:44:29.513 INFO:teuthology.orchestra.run.smithi196.stdout:Total 2.9 MB/s | 893 kB 00:00 2024-09-21T09:44:29.525 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-21T09:44:29.534 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T09:44:29.546 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-21T09:44:29.546 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-21T09:44:29.624 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-21T09:44:29.625 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-21T09:44:29.812 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-21T09:44:29.951 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T09:44:30.020 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/2 2024-09-21T09:44:30.068 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T09:44:30.155 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T09:44:30.211 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T09:44:30.239 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 6/6 2024-09-21T09:44:30.724 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/2 2024-09-21T09:44:30.724 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/2 2024-09-21T09:44:30.968 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T09:44:30.969 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:30.969 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-21T09:44:30.969 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T09:44:30.969 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:30.969 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:44:31.052 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 6/6 2024-09-21T09:44:31.052 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/6 2024-09-21T09:44:31.052 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T09:44:31.052 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T09:44:31.052 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T09:44:31.052 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T09:44:31.246 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados-devel 2024-09-21T09:44:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T09:44:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:31.335 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-21T09:44:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:44:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T09:44:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T09:44:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T09:44:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T09:44:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T09:44:31.336 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:31.336 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:44:31.590 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-fuse 2024-09-21T09:44:31.845 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:24 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:44:31.984 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:44:31.984 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:31.984 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-09-21T09:44:31.984 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:31.985 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-21T09:44:31.985 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 126 k 2024-09-21T09:44:31.985 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:31.985 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-21T09:44:31.985 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:31.985 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-09-21T09:44:31.985 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:31.985 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 126 k 2024-09-21T09:44:31.985 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 448 k 2024-09-21T09:44:31.985 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-21T09:44:32.170 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:26 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:44:32.201 INFO:teuthology.orchestra.run.smithi196.stdout:librados-devel-17.2.7-1649.gfbd1a6ab.el9.x86_64 586 kB/s | 126 kB 00:00 2024-09-21T09:44:32.201 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:44:32.202 INFO:teuthology.orchestra.run.smithi196.stdout:Total 582 kB/s | 126 kB 00:00 2024-09-21T09:44:32.202 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-21T09:44:32.207 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-21T09:44:32.207 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-21T09:44:32.270 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-21T09:44:32.271 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-21T09:44:32.303 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:44:32.304 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-21T09:44:32.304 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:32.304 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-21T09:44:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 813 k 2024-09-21T09:44:32.304 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-21T09:44:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T09:44:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:32.305 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-21T09:44:32.305 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:32.305 INFO:teuthology.orchestra.run.smithi098.stdout:Install 2 Packages 2024-09-21T09:44:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:32.305 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 893 k 2024-09-21T09:44:32.305 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 2.6 M 2024-09-21T09:44:32.305 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-21T09:44:32.419 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-21T09:44:32.515 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:32.545 INFO:teuthology.orchestra.run.smithi098.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 580 kB/s | 80 kB 00:00 2024-09-21T09:44:32.710 INFO:teuthology.orchestra.run.smithi098.stdout:(2/2): ceph-fuse-17.2.7-1649.gfbd1a6ab.el9.x86_ 2.6 MB/s | 813 kB 00:00 2024-09-21T09:44:32.711 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:44:32.711 INFO:teuthology.orchestra.run.smithi098.stdout:Total 2.1 MB/s | 893 kB 00:00 2024-09-21T09:44:32.722 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-21T09:44:32.745 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-21T09:44:32.745 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-21T09:44:32.824 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-21T09:44:32.825 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-21T09:44:32.947 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-21T09:44:33.156 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T09:44:33.200 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:33.200 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:33.200 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-21T09:44:33.200 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:44:33.200 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:33.200 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:44:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/2 2024-09-21T09:44:33.444 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs2 2024-09-21T09:44:33.905 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/2 2024-09-21T09:44:33.905 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/2 2024-09-21T09:44:34.048 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:27 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:44:34.139 INFO:teuthology.orchestra.run.smithi196.stdout:Package libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:44:34.180 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T09:44:34.180 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:34.180 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-21T09:44:34.180 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T09:44:34.181 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:34.181 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:44:34.182 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:44:34.184 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-21T09:44:34.184 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:44:34.246 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs-devel 2024-09-21T09:44:34.432 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados-devel 2024-09-21T09:44:34.831 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:27 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:44:34.964 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:44:34.964 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:34.964 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-09-21T09:44:34.965 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:34.965 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-21T09:44:34.965 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 25 k 2024-09-21T09:44:34.965 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:34.965 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-21T09:44:34.965 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:34.965 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-09-21T09:44:34.965 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:34.966 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 25 k 2024-09-21T09:44:34.966 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 110 k 2024-09-21T09:44:34.966 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-21T09:44:35.035 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:29 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:44:35.114 INFO:teuthology.orchestra.run.smithi196.stdout:libcephfs-devel-17.2.7-1649.gfbd1a6ab.el9.x86_6 170 kB/s | 25 kB 00:00 2024-09-21T09:44:35.114 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:44:35.114 INFO:teuthology.orchestra.run.smithi196.stdout:Total 168 kB/s | 25 kB 00:00 2024-09-21T09:44:35.114 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-21T09:44:35.117 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-21T09:44:35.117 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-21T09:44:35.118 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-21T09:44:35.119 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-21T09:44:35.169 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:44:35.170 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:35.170 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-09-21T09:44:35.170 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:35.170 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-21T09:44:35.170 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 126 k 2024-09-21T09:44:35.170 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:35.170 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-21T09:44:35.170 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:35.170 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-21T09:44:35.171 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:35.171 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 126 k 2024-09-21T09:44:35.171 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 448 k 2024-09-21T09:44:35.171 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-21T09:44:35.206 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-21T09:44:35.326 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:35.383 INFO:teuthology.orchestra.run.smithi098.stdout:librados-devel-17.2.7-1649.gfbd1a6ab.el9.x86_64 595 kB/s | 126 kB 00:00 2024-09-21T09:44:35.457 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:44:35.457 INFO:teuthology.orchestra.run.smithi098.stdout:Total 591 kB/s | 126 kB 00:00 2024-09-21T09:44:35.457 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-21T09:44:35.457 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-21T09:44:35.457 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-21T09:44:35.457 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-21T09:44:35.458 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-21T09:44:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-21T09:44:35.701 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:35.705 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:35.931 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:35.931 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:35.931 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-21T09:44:35.931 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:44:35.932 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:35.932 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:44:36.119 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados2 2024-09-21T09:44:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:36.459 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:36.459 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:36.459 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-21T09:44:36.459 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:44:36.459 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:36.459 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:44:36.700 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:29 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:44:36.727 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install libcephfs2 2024-09-21T09:44:36.790 INFO:teuthology.orchestra.run.smithi196.stdout:Package librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:44:36.833 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:44:36.834 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-21T09:44:36.834 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:44:36.903 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librbd1 2024-09-21T09:44:37.320 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:31 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:44:37.412 INFO:teuthology.orchestra.run.smithi098.stdout:Package libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:44:37.456 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:44:37.457 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-21T09:44:37.457 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:44:37.482 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:30 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:44:37.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install libcephfs-devel 2024-09-21T09:44:37.571 INFO:teuthology.orchestra.run.smithi196.stdout:Package librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:44:37.614 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:44:37.615 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-21T09:44:37.615 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:44:37.677 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rados 2024-09-21T09:44:38.100 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:32 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:44:38.232 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:44:38.232 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:38.232 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-09-21T09:44:38.232 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:38.232 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-21T09:44:38.232 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 25 k 2024-09-21T09:44:38.233 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:38.233 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-21T09:44:38.233 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:38.233 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-21T09:44:38.233 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:38.233 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 25 k 2024-09-21T09:44:38.233 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 110 k 2024-09-21T09:44:38.234 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-21T09:44:38.260 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:31 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:44:38.350 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:44:38.388 INFO:teuthology.orchestra.run.smithi098.stdout:libcephfs-devel-17.2.7-1649.gfbd1a6ab.el9.x86_6 162 kB/s | 25 kB 00:00 2024-09-21T09:44:38.389 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:44:38.389 INFO:teuthology.orchestra.run.smithi098.stdout:Total 160 kB/s | 25 kB 00:00 2024-09-21T09:44:38.389 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-21T09:44:38.393 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-21T09:44:38.393 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-21T09:44:38.394 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-21T09:44:38.394 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:44:38.394 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-21T09:44:38.395 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-21T09:44:38.395 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:44:38.457 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rgw 2024-09-21T09:44:38.483 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-21T09:44:38.593 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:38.973 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:39.041 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:32 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:44:39.130 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:44:39.172 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:44:39.173 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-21T09:44:39.173 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:44:39.198 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:39.198 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:39.198 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-21T09:44:39.198 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:44:39.198 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:39.198 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:44:39.240 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-cephfs 2024-09-21T09:44:39.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados2 2024-09-21T09:44:39.822 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:32 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:44:39.912 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:44:39.955 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:44:39.956 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-21T09:44:39.956 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:44:39.981 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:33 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:44:40.024 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rbd 2024-09-21T09:44:40.069 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:44:40.112 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:44:40.113 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-21T09:44:40.113 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:44:40.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librbd1 2024-09-21T09:44:40.602 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:33 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:44:40.691 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:44:40.734 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:44:40.735 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-21T09:44:40.735 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:44:40.762 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:34 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:44:40.798 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-fuse 2024-09-21T09:44:40.852 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:44:40.895 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:44:40.896 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-21T09:44:40.896 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:44:40.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rados 2024-09-21T09:44:41.391 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:34 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:44:41.523 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:44:41.523 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:41.523 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-09-21T09:44:41.523 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:41.524 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-21T09:44:41.524 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 82 k 2024-09-21T09:44:41.524 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:41.524 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-21T09:44:41.524 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:41.524 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-09-21T09:44:41.524 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:41.525 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 82 k 2024-09-21T09:44:41.525 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 222 k 2024-09-21T09:44:41.525 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-21T09:44:41.539 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:35 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:44:41.628 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:44:41.671 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:44:41.672 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-21T09:44:41.672 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:44:41.732 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-fuse-17.2.7-1649.gfbd1a6ab.el9.x86_64.rpm 394 kB/s | 82 kB 00:00 2024-09-21T09:44:41.733 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:44:41.733 INFO:teuthology.orchestra.run.smithi196.stdout:Total 392 kB/s | 82 kB 00:00 2024-09-21T09:44:41.733 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-21T09:44:41.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rgw 2024-09-21T09:44:41.740 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-21T09:44:41.740 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-21T09:44:41.803 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-21T09:44:41.803 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-21T09:44:41.945 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-21T09:44:42.054 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:42.316 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:36 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:44:42.405 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:44:42.412 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:42.447 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:44:42.448 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-21T09:44:42.448 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:44:42.506 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-cephfs 2024-09-21T09:44:42.652 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:42.652 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:42.652 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-21T09:44:42.652 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:44:42.652 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:42.652 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:44:42.891 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-mirror 2024-09-21T09:44:43.084 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:37 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:44:43.173 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:44:43.216 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:44:43.217 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-21T09:44:43.217 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:44:43.291 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rbd 2024-09-21T09:44:43.489 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:36 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:44:43.626 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:44:43.627 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:43.627 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-09-21T09:44:43.627 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:43.627 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-21T09:44:43.628 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 2.9 M 2024-09-21T09:44:43.628 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:43.628 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-21T09:44:43.628 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:43.628 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-09-21T09:44:43.628 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:43.628 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 2.9 M 2024-09-21T09:44:43.628 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 12 M 2024-09-21T09:44:43.628 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-21T09:44:43.869 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:37 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:44:43.958 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:44:43.961 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-mirror-17.2.7-1649.gfbd1a6ab.el9.x86_64.rpm 8.8 MB/s | 2.9 MB 00:00 2024-09-21T09:44:43.961 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:44:43.962 INFO:teuthology.orchestra.run.smithi196.stdout:Total 8.8 MB/s | 2.9 MB 00:00 2024-09-21T09:44:43.962 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-21T09:44:43.971 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-21T09:44:43.971 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-21T09:44:44.000 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:44:44.001 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-21T09:44:44.001 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:44:44.050 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-21T09:44:44.050 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-21T09:44:44.069 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-fuse 2024-09-21T09:44:44.460 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-21T09:44:44.511 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:44.536 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:44.536 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:44:44.537 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T09:44:44.537 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T09:44:44.537 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T09:44:44.537 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:44.649 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:38 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:44:44.781 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:44:44.781 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:44.781 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-09-21T09:44:44.781 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:44.781 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-21T09:44:44.782 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 82 k 2024-09-21T09:44:44.782 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:44.782 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-21T09:44:44.782 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:44.782 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-21T09:44:44.782 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:44.782 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 82 k 2024-09-21T09:44:44.783 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 222 k 2024-09-21T09:44:44.783 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-21T09:44:44.975 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-fuse-17.2.7-1649.gfbd1a6ab.el9.x86_64.rpm 425 kB/s | 82 kB 00:00 2024-09-21T09:44:44.976 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:44:44.976 INFO:teuthology.orchestra.run.smithi098.stdout:Total 423 kB/s | 82 kB 00:00 2024-09-21T09:44:44.976 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-21T09:44:44.983 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-21T09:44:44.983 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-21T09:44:45.046 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-21T09:44:45.046 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-21T09:44:45.197 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-21T09:44:45.297 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:45.659 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:45.659 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:45.659 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-21T09:44:45.659 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:44:45.659 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:45.660 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:44:45.693 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:45.873 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-nbd 2024-09-21T09:44:45.932 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:45.932 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:45.933 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-21T09:44:45.933 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:44:45.933 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:45.933 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:44:46.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-mirror 2024-09-21T09:44:46.473 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:39 ago on Sat 21 Sep 2024 09:43:07 AM UTC. 2024-09-21T09:44:46.605 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-21T09:44:46.606 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:46.606 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-09-21T09:44:46.606 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:46.607 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-21T09:44:46.607 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 165 k 2024-09-21T09:44:46.607 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:46.607 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-21T09:44:46.607 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-21T09:44:46.607 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-09-21T09:44:46.607 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:46.607 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 165 k 2024-09-21T09:44:46.607 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 477 k 2024-09-21T09:44:46.607 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-21T09:44:46.724 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:40 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:44:46.823 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-nbd-17.2.7-1649.gfbd1a6ab.el9.x86_64.rpm 764 kB/s | 165 kB 00:00 2024-09-21T09:44:46.823 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:44:46.824 INFO:teuthology.orchestra.run.smithi196.stdout:Total 760 kB/s | 165 kB 00:00 2024-09-21T09:44:46.824 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-21T09:44:46.831 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-21T09:44:46.832 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-21T09:44:46.859 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:44:46.860 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:46.860 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-09-21T09:44:46.860 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:46.860 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-21T09:44:46.860 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 2.9 M 2024-09-21T09:44:46.860 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:46.860 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-21T09:44:46.860 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:46.861 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-21T09:44:46.861 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:46.861 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 2.9 M 2024-09-21T09:44:46.861 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 12 M 2024-09-21T09:44:46.861 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-21T09:44:46.895 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-21T09:44:46.895 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-21T09:44:47.044 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-21T09:44:47.144 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:47.188 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-mirror-17.2.7-1649.gfbd1a6ab.el9.x86_64.rpm 9.0 MB/s | 2.9 MB 00:00 2024-09-21T09:44:47.189 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:44:47.189 INFO:teuthology.orchestra.run.smithi098.stdout:Total 8.9 MB/s | 2.9 MB 00:00 2024-09-21T09:44:47.189 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-21T09:44:47.198 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-21T09:44:47.198 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-21T09:44:47.278 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-21T09:44:47.278 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-21T09:44:47.586 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:47.691 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-21T09:44:47.755 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:47.780 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:47.780 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:44:47.780 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T09:44:47.781 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T09:44:47.781 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T09:44:47.781 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:47.830 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:47.830 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:47.831 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-21T09:44:47.831 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:44:47.831 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:44:47.831 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-21T09:44:48.061 DEBUG:teuthology.parallel:result is None 2024-09-21T09:44:48.891 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:48.892 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:48.892 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-21T09:44:48.892 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:44:48.892 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:48.892 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:44:49.100 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-nbd 2024-09-21T09:44:49.700 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:43 ago on Sat 21 Sep 2024 09:43:06 AM UTC. 2024-09-21T09:44:49.836 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-21T09:44:49.837 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:49.837 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-09-21T09:44:49.837 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:49.837 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-21T09:44:49.837 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 165 k 2024-09-21T09:44:49.837 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:49.838 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-21T09:44:49.838 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-21T09:44:49.838 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-21T09:44:49.838 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:49.838 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 165 k 2024-09-21T09:44:49.838 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 477 k 2024-09-21T09:44:49.838 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-21T09:44:50.051 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-nbd-17.2.7-1649.gfbd1a6ab.el9.x86_64.rpm 776 kB/s | 165 kB 00:00 2024-09-21T09:44:50.051 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:44:50.052 INFO:teuthology.orchestra.run.smithi098.stdout:Total 772 kB/s | 165 kB 00:00 2024-09-21T09:44:50.052 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-21T09:44:50.059 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-21T09:44:50.059 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-21T09:44:50.122 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-21T09:44:50.123 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-21T09:44:50.273 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-21T09:44:50.373 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:50.836 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:51.092 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:44:51.092 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:51.092 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-21T09:44:51.093 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:44:51.093 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:44:51.093 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-21T09:44:51.309 DEBUG:teuthology.parallel:result is None 2024-09-21T09:44:51.310 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:44:51.310 INFO:teuthology.packaging:ref: None 2024-09-21T09:44:51.310 INFO:teuthology.packaging:tag: None 2024-09-21T09:44:51.310 INFO:teuthology.packaging:branch: quincy 2024-09-21T09:44:51.310 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:44:51.310 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-21T09:44:51.539 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T09:44:51.560 INFO:teuthology.orchestra.run.smithi098.stdout:17.2.7-1649.gfbd1a6ab.el9 2024-09-21T09:44:51.560 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1649.gfbd1a6ab.el9 2024-09-21T09:44:51.560 INFO:teuthology.task.install:The correct ceph version 17.2.7-1649.gfbd1a6ab is installed. 2024-09-21T09:44:51.562 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:44:51.562 INFO:teuthology.packaging:ref: None 2024-09-21T09:44:51.562 INFO:teuthology.packaging:tag: None 2024-09-21T09:44:51.562 INFO:teuthology.packaging:branch: quincy 2024-09-21T09:44:51.562 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:44:51.562 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-21T09:44:51.784 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T09:44:51.810 INFO:teuthology.orchestra.run.smithi196.stdout:17.2.7-1649.gfbd1a6ab.el9 2024-09-21T09:44:51.810 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1649.gfbd1a6ab.el9 2024-09-21T09:44:51.810 INFO:teuthology.task.install:The correct ceph version 17.2.7-1649.gfbd1a6ab is installed. 2024-09-21T09:44:51.812 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T09:44:51.812 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-21T09:44:51.812 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T09:44:51.837 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-21T09:44:51.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T09:44:51.879 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T09:44:51.879 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-21T09:44:51.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T09:44:51.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T09:44:51.968 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-21T09:44:51.968 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T09:44:51.996 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T09:44:52.063 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T09:44:52.064 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-21T09:44:52.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T09:44:52.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T09:44:52.154 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-21T09:44:52.154 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T09:44:52.182 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T09:44:52.247 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T09:44:52.248 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-21T09:44:52.248 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T09:44:52.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T09:44:52.337 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-21T09:44:52.338 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T09:44:52.363 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T09:44:52.427 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-21T09:44:52.527 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'OSD_DOWN', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSDMAP_FLAGS', 'PG_AVAILABILITY'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'} 2024-09-21T09:44:52.527 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-09-21T09:44:52.528 INFO:tasks.cephadm:Cluster fsid is 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:44:52.528 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-21T09:44:52.528 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.98', 'mon.c': '[v2:172.21.15.98:3301,v1:172.21.15.98:6790]', 'mon.b': '172.21.15.196'} 2024-09-21T09:44:52.528 INFO:tasks.cephadm:First mon is mon.a on smithi098 2024-09-21T09:44:52.528 INFO:tasks.cephadm:First mgr is y 2024-09-21T09:44:52.528 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-21T09:44:52.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo hostname $(hostname -s) 2024-09-21T09:44:52.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo hostname $(hostname -s) 2024-09-21T09:44:52.580 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-09-21T09:44:52.580 DEBUG:teuthology.orchestra.run.smithi098:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T09:44:52.808 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r--. 1 ubuntu ubuntu 378585 Sep 21 09:44 /home/ubuntu/cephtest/cephadm 2024-09-21T09:44:52.808 DEBUG:teuthology.orchestra.run.smithi196:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T09:44:52.935 INFO:teuthology.orchestra.run.smithi196.stdout:-rw-r--r--. 1 ubuntu ubuntu 378585 Sep 21 09:44 /home/ubuntu/cephtest/cephadm 2024-09-21T09:44:52.935 DEBUG:teuthology.orchestra.run.smithi098:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T09:44:52.951 DEBUG:teuthology.orchestra.run.smithi196:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T09:44:52.977 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-09-21T09:44:52.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-09-21T09:44:52.995 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-09-21T09:44:54.570 INFO:teuthology.orchestra.run.smithi196.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-21T09:44:54.722 INFO:teuthology.orchestra.run.smithi098.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-21T09:45:20.481 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T09:45:20.481 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_version": "ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)", 2024-09-21T09:45:20.481 INFO:teuthology.orchestra.run.smithi196.stdout: "image_id": "fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9", 2024-09-21T09:45:20.481 INFO:teuthology.orchestra.run.smithi196.stdout: "repo_digests": [ 2024-09-21T09:45:20.481 INFO:teuthology.orchestra.run.smithi196.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2" 2024-09-21T09:45:20.481 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T09:45:20.481 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T09:45:24.707 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T09:45:24.707 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)", 2024-09-21T09:45:24.707 INFO:teuthology.orchestra.run.smithi098.stdout: "image_id": "fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9", 2024-09-21T09:45:24.708 INFO:teuthology.orchestra.run.smithi098.stdout: "repo_digests": [ 2024-09-21T09:45:24.708 INFO:teuthology.orchestra.run.smithi098.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2" 2024-09-21T09:45:24.708 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T09:45:24.708 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T09:45:24.727 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph 2024-09-21T09:45:24.771 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /etc/ceph 2024-09-21T09:45:24.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /etc/ceph 2024-09-21T09:45:24.836 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 777 /etc/ceph 2024-09-21T09:45:24.872 INFO:tasks.cephadm:Writing seed config... 2024-09-21T09:45:24.873 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-09-21T09:45:24.873 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-09-21T09:45:24.873 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-21T09:45:24.873 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-21T09:45:24.873 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-21T09:45:24.873 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-09-21T09:45:24.874 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-21T09:45:24.874 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-21T09:45:24.874 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-21T09:45:24.874 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-21T09:45:24.874 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-21T09:45:24.874 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-21T09:45:24.874 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-21T09:45:24.891 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 26a15512-77fe-11ef-baf6-efdc52797490 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-21T09:45:24.892 DEBUG:teuthology.orchestra.run.smithi098:mon.a> sudo journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@mon.a.service 2024-09-21T09:45:24.934 DEBUG:teuthology.orchestra.run.smithi098:mgr.y> sudo journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@mgr.y.service 2024-09-21T09:45:24.978 INFO:tasks.cephadm:Bootstrapping... 2024-09-21T09:45:24.978 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 26a15512-77fe-11ef-baf6-efdc52797490 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.98 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-21T09:45:25.176 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:45:25.176 INFO:teuthology.orchestra.run.smithi098.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '26a15512-77fe-11ef-baf6-efdc52797490', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.98', '--skip-admin-label'] 2024-09-21T09:45:25.200 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 5.2.2 2024-09-21T09:45:25.200 INFO:teuthology.orchestra.run.smithi098.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-21T09:45:25.200 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying podman|docker is present... 2024-09-21T09:45:25.221 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 5.2.2 2024-09-21T09:45:25.222 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying lvm2 is present... 2024-09-21T09:45:25.222 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying time synchronization is in place... 2024-09-21T09:45:25.230 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T09:45:25.230 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T09:45:25.236 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T09:45:25.236 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-09-21T09:45:25.244 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout enabled 2024-09-21T09:45:25.251 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout active 2024-09-21T09:45:25.251 INFO:teuthology.orchestra.run.smithi098.stdout:Unit chronyd.service is enabled and running 2024-09-21T09:45:25.251 INFO:teuthology.orchestra.run.smithi098.stdout:Repeating the final host check... 2024-09-21T09:45:25.272 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 5.2.2 2024-09-21T09:45:25.273 INFO:teuthology.orchestra.run.smithi098.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-21T09:45:25.273 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl is present 2024-09-21T09:45:25.273 INFO:teuthology.orchestra.run.smithi098.stdout:lvcreate is present 2024-09-21T09:45:25.279 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T09:45:25.279 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T09:45:25.286 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T09:45:25.286 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-09-21T09:45:25.293 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout enabled 2024-09-21T09:45:25.300 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout active 2024-09-21T09:45:25.300 INFO:teuthology.orchestra.run.smithi098.stdout:Unit chronyd.service is enabled and running 2024-09-21T09:45:25.300 INFO:teuthology.orchestra.run.smithi098.stdout:Host looks OK 2024-09-21T09:45:25.300 INFO:teuthology.orchestra.run.smithi098.stdout:Cluster fsid: 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:45:25.300 INFO:teuthology.orchestra.run.smithi098.stdout:Acquiring lock 139871664386976 on /run/cephadm/26a15512-77fe-11ef-baf6-efdc52797490.lock 2024-09-21T09:45:25.300 INFO:teuthology.orchestra.run.smithi098.stdout:Lock 139871664386976 acquired on /run/cephadm/26a15512-77fe-11ef-baf6-efdc52797490.lock 2024-09-21T09:45:25.301 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying IP 172.21.15.98 port 3300 ... 2024-09-21T09:45:25.301 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying IP 172.21.15.98 port 6789 ... 2024-09-21T09:45:25.301 INFO:teuthology.orchestra.run.smithi098.stdout:Base mon IP(s) is [172.21.15.98:3300, 172.21.15.98:6789], mon addrv is [v2:172.21.15.98:3300,v1:172.21.15.98:6789] 2024-09-21T09:45:25.304 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.98 metric 100 2024-09-21T09:45:25.304 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.98 metric 100 2024-09-21T09:45:25.307 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-21T09:45:25.307 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-21T09:45:25.307 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-21T09:45:25.310 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-21T09:45:25.311 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-21T09:45:25.311 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T09:45:25.311 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-21T09:45:25.311 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link noprefixroute 2024-09-21T09:45:25.311 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T09:45:25.311 INFO:teuthology.orchestra.run.smithi098.stdout:Mon IP `172.21.15.98` is in CIDR network `172.21.0.0/20` 2024-09-21T09:45:25.311 INFO:teuthology.orchestra.run.smithi098.stdout:Mon IP `172.21.15.98` is in CIDR network `172.21.0.0/20` 2024-09-21T09:45:25.311 INFO:teuthology.orchestra.run.smithi098.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-21T09:45:25.312 INFO:teuthology.orchestra.run.smithi098.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-21T09:45:25.312 INFO:teuthology.orchestra.run.smithi098.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-21T09:45:25.766 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 2024-09-21T09:45:25.766 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-21T09:45:25.766 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Getting image source signatures 2024-09-21T09:45:25.766 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying blob sha256:3ba00a516579693832719224d909a7cd5333511394d4f35f95e899e50f0bfaeb 2024-09-21T09:45:25.767 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-21T09:45:25.767 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying config sha256:fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 2024-09-21T09:45:25.767 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-21T09:45:26.688 INFO:teuthology.orchestra.run.smithi098.stdout:ceph: stdout ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable) 2024-09-21T09:45:26.688 INFO:teuthology.orchestra.run.smithi098.stdout:Ceph version: ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable) 2024-09-21T09:45:26.688 INFO:teuthology.orchestra.run.smithi098.stdout:Extracting ceph user uid/gid from container image... 2024-09-21T09:45:27.704 INFO:teuthology.orchestra.run.smithi098.stdout:stat: stdout 167 167 2024-09-21T09:45:27.704 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial keys... 2024-09-21T09:45:28.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQC4le5mqXp2AhAAwfqmbZXDoCemWJSuSEAzWw== 2024-09-21T09:45:29.574 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQC4le5mBwfxORAAx+LMyAJ3z9ijghg4Son7vw== 2024-09-21T09:45:30.480 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQC5le5mMudOMxAA7aTLJYHrrIEITJc5MtIbZQ== 2024-09-21T09:45:30.481 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial monmap... 2024-09-21T09:45:31.554 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T09:45:31.555 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-09-21T09:45:31.555 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:45:31.555 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T09:45:31.555 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool for a [v2:172.21.15.98:3300,v1:172.21.15.98:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T09:45:31.555 INFO:teuthology.orchestra.run.smithi098.stdout:setting min_mon_release = octopus 2024-09-21T09:45:31.555 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: set fsid to 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:45:31.555 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T09:45:31.555 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:45:31.555 INFO:teuthology.orchestra.run.smithi098.stdout:Creating mon... 2024-09-21T09:45:32.501 INFO:teuthology.orchestra.run.smithi098.stdout:create mon.a on 2024-09-21T09:45:32.706 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-21T09:45:32.845 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-21T09:45:32.997 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-26a15512-77fe-11ef-baf6-efdc52797490.target → /etc/systemd/system/ceph-26a15512-77fe-11ef-baf6-efdc52797490.target. 2024-09-21T09:45:32.998 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-26a15512-77fe-11ef-baf6-efdc52797490.target → /etc/systemd/system/ceph-26a15512-77fe-11ef-baf6-efdc52797490.target. 2024-09-21T09:45:33.171 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-26a15512-77fe-11ef-baf6-efdc52797490@mon.a 2024-09-21T09:45:33.171 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to reset failed state of unit ceph-26a15512-77fe-11ef-baf6-efdc52797490@mon.a.service: Unit ceph-26a15512-77fe-11ef-baf6-efdc52797490@mon.a.service not loaded. 2024-09-21T09:45:33.349 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-26a15512-77fe-11ef-baf6-efdc52797490.target.wants/ceph-26a15512-77fe-11ef-baf6-efdc52797490@mon.a.service → /etc/systemd/system/ceph-26a15512-77fe-11ef-baf6-efdc52797490@.service. 2024-09-21T09:45:33.892 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T09:45:33.892 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-09-21T09:45:33.899 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T09:45:33.899 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-09-21T09:45:33.899 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-09-21T09:45:33.899 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T09:45:33.900 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mon to start... 2024-09-21T09:45:33.900 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mon... 2024-09-21T09:45:35.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout cluster: 2024-09-21T09:45:35.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout id: 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:45:35.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-21T09:45:35.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-21T09:45:35.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout services: 2024-09-21T09:45:35.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.589685s) 2024-09-21T09:45:35.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-21T09:45:35.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-21T09:45:35.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-21T09:45:35.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout data: 2024-09-21T09:45:35.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-21T09:45:35.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-21T09:45:35.090 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-21T09:45:35.090 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout pgs: 2024-09-21T09:45:35.090 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-21T09:45:35.090 INFO:teuthology.orchestra.run.smithi098.stdout:mon is available 2024-09-21T09:45:35.090 INFO:teuthology.orchestra.run.smithi098.stdout:Assimilating anything we can from ceph.conf... 2024-09-21T09:45:36.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-21T09:45:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T09:45:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout fsid = 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:45:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-21T09:45:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.98:3300,v1:172.21.15.98:6789] 2024-09-21T09:45:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T09:45:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T09:45:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T09:45:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T09:45:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-21T09:45:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T09:45:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T09:45:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-21T09:45:36.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T09:45:36.080 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T09:45:36.080 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T09:45:36.080 INFO:teuthology.orchestra.run.smithi098.stdout:Generating new minimal ceph.conf... 2024-09-21T09:45:36.992 INFO:teuthology.orchestra.run.smithi098.stdout:Restarting the monitor... 2024-09-21T09:45:37.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:37 smithi098 systemd[1]: Stopping Ceph mon.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:45:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:37 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a[28871]: 2024-09-21T09:45:37.266+0000 7f749aa57640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T09:45:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:37 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a[28871]: 2024-09-21T09:45:37.266+0000 7f749aa57640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-21T09:45:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:37 smithi098 podman[29117]: 2024-09-21 09:45:37.51282198 +0000 UTC m=+0.361640394 container died 3673897ea7a09846ae3b28c65ac7371fa253fc9ad7ded0c0310a6997d583b20d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-fbd1a6a, org.label-schema.schema-version=1.0) 2024-09-21T09:45:38.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:37 smithi098 podman[29117]: 2024-09-21 09:45:37.883758341 +0000 UTC m=+0.732576752 container cleanup 3673897ea7a09846ae3b28c65ac7371fa253fc9ad7ded0c0310a6997d583b20d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=quincy-fbd1a6a, GIT_BRANCH=HEAD) 2024-09-21T09:45:38.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:37 smithi098 bash[29117]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a 2024-09-21T09:45:38.420 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 podman[29129]: 2024-09-21 09:45:38.109634029 +0000 UTC m=+0.598016067 container remove 3673897ea7a09846ae3b28c65ac7371fa253fc9ad7ded0c0310a6997d583b20d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=quincy-fbd1a6a, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-21T09:45:38.420 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-21T09:45:38.420 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 systemd[1]: Stopped Ceph mon.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:45:38.420 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 systemd[1]: Starting Ceph mon.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:45:38.672 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 podman[29220]: 2024-09-21 09:45:38.418860379 +0000 UTC m=+0.098991360 container create f501577d66cb7f2c3ae4a9f00ff86f8e580153126e1c088fd4f1612484a2be1f (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-21T09:45:38.672 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 podman[29220]: 2024-09-21 09:45:38.335217027 +0000 UTC m=+0.015348020 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph:quincy 2024-09-21T09:45:38.751 INFO:teuthology.orchestra.run.smithi098.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-21T09:45:38.960 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 podman[29220]: 2024-09-21 09:45:38.671457513 +0000 UTC m=+0.351588493 container init f501577d66cb7f2c3ae4a9f00ff86f8e580153126e1c088fd4f1612484a2be1f (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=quincy-fbd1a6a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T09:45:38.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 podman[29220]: 2024-09-21 09:45:38.67451553 +0000 UTC m=+0.354646513 container start f501577d66cb7f2c3ae4a9f00ff86f8e580153126e1c088fd4f1612484a2be1f (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=quincy-fbd1a6a, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T09:45:38.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T09:45:38.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable), process ceph-mon, pid 2 2024-09-21T09:45:38.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: pidfile_write: ignore empty --pid-file 2024-09-21T09:45:38.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: load: jerasure load: lrc 2024-09-21T09:45:38.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: RocksDB version: 6.15.5 2024-09-21T09:45:38.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-21T09:45:38.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Compile date Sep 20 2024 2024-09-21T09:45:38.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: DB SUMMARY 2024-09-21T09:45:38.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: DB Session ID: ZP34X80MAB0QNTNHBRWB 2024-09-21T09:45:38.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: CURRENT file: CURRENT 2024-09-21T09:45:38.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T09:45:38.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-09-21T09:45:38.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-21T09:45:38.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 83686 ; 2024-09-21T09:45:38.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.error_if_exists: 0 2024-09-21T09:45:38.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.create_if_missing: 0 2024-09-21T09:45:38.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.paranoid_checks: 1 2024-09-21T09:45:38.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T09:45:38.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.env: 0x55cd3ef89440 2024-09-21T09:45:38.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.fs: Posix File System 2024-09-21T09:45:38.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.info_log: 0x55cd3fabeec0 2024-09-21T09:45:38.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T09:45:38.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.statistics: (nil) 2024-09-21T09:45:38.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.use_fsync: 0 2024-09-21T09:45:38.963 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_log_file_size: 0 2024-09-21T09:45:38.963 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T09:45:38.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T09:45:38.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T09:45:38.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T09:45:38.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.allow_fallocate: 1 2024-09-21T09:45:38.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T09:45:38.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T09:45:38.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.use_direct_reads: 0 2024-09-21T09:45:38.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T09:45:38.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T09:45:38.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.db_log_dir: 2024-09-21T09:45:38.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-09-21T09:45:38.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T09:45:38.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T09:45:38.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T09:45:38.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T09:45:38.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T09:45:38.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T09:45:38.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T09:45:38.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T09:45:38.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.write_buffer_manager: 0x55cd3fb7c7e0 2024-09-21T09:45:38.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T09:45:38.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-21T09:45:38.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T09:45:38.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T09:45:38.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T09:45:38.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T09:45:38.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T09:45:38.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T09:45:38.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T09:45:38.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.unordered_write: 0 2024-09-21T09:45:38.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T09:45:38.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T09:45:38.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T09:45:38.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T09:45:38.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.row_cache: None 2024-09-21T09:45:38.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.wal_filter: None 2024-09-21T09:45:38.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T09:45:38.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T09:45:38.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.preserve_deletes: 0 2024-09-21T09:45:38.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.two_write_queues: 0 2024-09-21T09:45:38.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T09:45:38.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.atomic_flush: 0 2024-09-21T09:45:38.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T09:45:38.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T09:45:38.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T09:45:38.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.log_readahead_size: 0 2024-09-21T09:45:38.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T09:45:38.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T09:45:38.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T09:45:38.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T09:45:38.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T09:45:38.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T09:45:38.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_background_jobs: 2 2024-09-21T09:45:38.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_background_compactions: -1 2024-09-21T09:45:38.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_subcompactions: 1 2024-09-21T09:45:38.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T09:45:38.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T09:45:38.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T09:45:38.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T09:45:38.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T09:45:38.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T09:45:38.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T09:45:38.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T09:45:38.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_open_files: -1 2024-09-21T09:45:38.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T09:45:38.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T09:45:38.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T09:45:38.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T09:45:38.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_background_flushes: -1 2024-09-21T09:45:38.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Compression algorithms supported: 2024-09-21T09:45:38.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: kZSTD supported: 0 2024-09-21T09:45:38.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: kXpressCompression supported: 0 2024-09-21T09:45:38.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: kBZip2Compression supported: 0 2024-09-21T09:45:38.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T09:45:38.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: kLZ4Compression supported: 1 2024-09-21T09:45:38.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: kZlibCompression supported: 1 2024-09-21T09:45:38.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T09:45:38.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: kSnappyCompression supported: 1 2024-09-21T09:45:38.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T09:45:38.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-09-21T09:45:38.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-21T09:45:38.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T09:45:38.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.merge_operator: 2024-09-21T09:45:38.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.compaction_filter: None 2024-09-21T09:45:38.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T09:45:38.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T09:45:38.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T09:45:38.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T09:45:38.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cd3fa65a38) 2024-09-21T09:45:38.970 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks: 1 2024-09-21T09:45:38.970 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T09:45:38.970 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T09:45:38.971 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_top_level_index_and_filter: 1 2024-09-21T09:45:38.971 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_type: 0 2024-09-21T09:45:38.971 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_index_type: 0 2024-09-21T09:45:38.971 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_shortening: 1 2024-09-21T09:45:38.971 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T09:45:38.971 INFO:journalctl@ceph.mon.a.smithi098.stdout: hash_index_allow_collision: 1 2024-09-21T09:45:38.971 INFO:journalctl@ceph.mon.a.smithi098.stdout: checksum: 1 2024-09-21T09:45:38.971 INFO:journalctl@ceph.mon.a.smithi098.stdout: no_block_cache: 0 2024-09-21T09:45:38.971 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache: 0x55cd3fa9b1f0 2024-09-21T09:45:38.971 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_name: BinnedLRUCache 2024-09-21T09:45:38.971 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_options: 2024-09-21T09:45:38.971 INFO:journalctl@ceph.mon.a.smithi098.stdout: capacity : 536870912 2024-09-21T09:45:38.971 INFO:journalctl@ceph.mon.a.smithi098.stdout: num_shard_bits : 4 2024-09-21T09:45:38.971 INFO:journalctl@ceph.mon.a.smithi098.stdout: strict_capacity_limit : 0 2024-09-21T09:45:38.971 INFO:journalctl@ceph.mon.a.smithi098.stdout: high_pri_pool_ratio: 0.000 2024-09-21T09:45:38.972 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_compressed: (nil) 2024-09-21T09:45:38.972 INFO:journalctl@ceph.mon.a.smithi098.stdout: persistent_cache: (nil) 2024-09-21T09:45:38.972 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size: 4096 2024-09-21T09:45:38.972 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size_deviation: 10 2024-09-21T09:45:38.972 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_restart_interval: 16 2024-09-21T09:45:38.972 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_block_restart_interval: 1 2024-09-21T09:45:38.972 INFO:journalctl@ceph.mon.a.smithi098.stdout: metadata_block_size: 4096 2024-09-21T09:45:38.972 INFO:journalctl@ceph.mon.a.smithi098.stdout: partition_filters: 0 2024-09-21T09:45:38.972 INFO:journalctl@ceph.mon.a.smithi098.stdout: use_delta_encoding: 1 2024-09-21T09:45:38.972 INFO:journalctl@ceph.mon.a.smithi098.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-21T09:45:38.972 INFO:journalctl@ceph.mon.a.smithi098.stdout: whole_key_filtering: 1 2024-09-21T09:45:38.972 INFO:journalctl@ceph.mon.a.smithi098.stdout: verify_compression: 0 2024-09-21T09:45:38.972 INFO:journalctl@ceph.mon.a.smithi098.stdout: read_amp_bytes_per_bit: 0 2024-09-21T09:45:38.972 INFO:journalctl@ceph.mon.a.smithi098.stdout: format_version: 4 2024-09-21T09:45:38.973 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_index_compression: 1 2024-09-21T09:45:38.973 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_align: 0 2024-09-21T09:45:38.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T09:45:38.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T09:45:38.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.compression: NoCompression 2024-09-21T09:45:38.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T09:45:38.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T09:45:38.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T09:45:38.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.num_levels: 7 2024-09-21T09:45:38.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T09:45:38.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T09:45:38.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T09:45:38.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T09:45:38.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T09:45:38.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T09:45:38.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T09:45:38.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:45:38.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T09:45:38.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T09:45:38.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T09:45:38.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T09:45:38.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T09:45:38.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T09:45:38.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:45:38.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T09:45:38.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T09:45:38.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T09:45:38.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T09:45:38.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T09:45:38.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T09:45:38.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T09:45:38.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T09:45:38.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T09:45:38.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T09:45:38.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T09:45:38.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T09:45:38.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T09:45:38.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T09:45:38.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T09:45:38.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T09:45:38.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T09:45:38.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T09:45:38.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T09:45:38.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.arena_block_size: 4194304 2024-09-21T09:45:38.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T09:45:38.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T09:45:38.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-21T09:45:38.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T09:45:38.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T09:45:38.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T09:45:38.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T09:45:38.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T09:45:38.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T09:45:38.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T09:45:38.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T09:45:38.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T09:45:38.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T09:45:38.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T09:45:38.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T09:45:38.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.inplace_update_support: 0 2024-09-21T09:45:38.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T09:45:38.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T09:45:38.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T09:45:38.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T09:45:38.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.bloom_locality: 0 2024-09-21T09:45:38.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.max_successive_merges: 0 2024-09-21T09:45:38.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T09:45:38.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T09:45:38.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T09:45:38.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T09:45:38.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.ttl: 2592000 2024-09-21T09:45:38.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T09:45:38.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.enable_blob_files: false 2024-09-21T09:45:38.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.min_blob_size: 0 2024-09-21T09:45:38.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T09:45:38.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T09:45:38.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T09:45:38.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T09:45:38.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T09:45:38.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-09-21T09:45:38.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-09-21T09:45:38.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726911938697255, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-09-21T09:45:38.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-09-21T09:45:38.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-21T09:45:38.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726911938697958, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 80259, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 78501, "index_size": 220, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10130, "raw_average_key_size": 47, "raw_value_size": 72714, "raw_average_value_size": 341, "num_data_blocks": 10, "num_entries": 213, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726911938, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "c8237901-677b-4a88-bea9-c36a33f86681", "db_session_id": "ZP34X80MAB0QNTNHBRWB"}} 2024-09-21T09:45:38.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-09-21T09:45:38.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726911938698297, "job": 1, "event": "recovery_finished"} 2024-09-21T09:45:38.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T09:45:38.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55cd3face700 2024-09-21T09:45:38.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: DB pointer 0x55cd3fa9e000 2024-09-21T09:45:38.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-21T09:45:38.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-21T09:45:38.979 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** DB Stats ** 2024-09-21T09:45:38.979 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:45:38.979 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T09:45:38.979 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:45:38.979 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:45:38.979 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T09:45:38.980 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-21T09:45:38.980 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:45:38.980 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-09-21T09:45:38.980 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-09-21T09:45:38.980 INFO:journalctl@ceph.mon.a.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:45:38.980 INFO:journalctl@ceph.mon.a.smithi098.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:45:38.980 INFO:journalctl@ceph.mon.a.smithi098.stdout: L0 2/0 80.16 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.9 0.00 0.00 1 0.000 0 0 2024-09-21T09:45:38.980 INFO:journalctl@ceph.mon.a.smithi098.stdout: Sum 2/0 80.16 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.9 0.00 0.00 1 0.000 0 0 2024-09-21T09:45:38.980 INFO:journalctl@ceph.mon.a.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.9 0.00 0.00 1 0.000 0 0 2024-09-21T09:45:38.980 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-09-21T09:45:38.980 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-09-21T09:45:38.980 INFO:journalctl@ceph.mon.a.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:45:38.980 INFO:journalctl@ceph.mon.a.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:45:38.980 INFO:journalctl@ceph.mon.a.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 163.9 0.00 0.00 1 0.000 0 0 2024-09-21T09:45:38.980 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:45:38.981 INFO:journalctl@ceph.mon.a.smithi098.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T09:45:38.981 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:45:38.981 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:45:38.981 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:45:38.981 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:45:38.981 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative compaction: 0.00 GB write, 25.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:45:38.981 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval compaction: 0.00 GB write, 25.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:45:38.981 INFO:journalctl@ceph.mon.a.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:45:38.981 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-09-21T09:45:38.981 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:45:38.981 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-09-21T09:45:38.981 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-09-21T09:45:38.981 INFO:journalctl@ceph.mon.a.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:45:38.981 INFO:journalctl@ceph.mon.a.smithi098.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:45:38.982 INFO:journalctl@ceph.mon.a.smithi098.stdout: L0 2/0 80.16 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.9 0.00 0.00 1 0.000 0 0 2024-09-21T09:45:38.982 INFO:journalctl@ceph.mon.a.smithi098.stdout: Sum 2/0 80.16 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.9 0.00 0.00 1 0.000 0 0 2024-09-21T09:45:38.982 INFO:journalctl@ceph.mon.a.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-21T09:45:38.982 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-09-21T09:45:38.982 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-09-21T09:45:38.982 INFO:journalctl@ceph.mon.a.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:45:38.982 INFO:journalctl@ceph.mon.a.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:45:38.982 INFO:journalctl@ceph.mon.a.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 163.9 0.00 0.00 1 0.000 0 0 2024-09-21T09:45:38.982 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:45:38.982 INFO:journalctl@ceph.mon.a.smithi098.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T09:45:38.982 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:45:38.982 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:45:38.982 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:45:38.982 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:45:38.982 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative compaction: 0.00 GB write, 25.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:45:38.983 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:45:38.983 INFO:journalctl@ceph.mon.a.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:45:38.983 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-09-21T09:45:38.983 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:45:38.983 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: starting mon.a rank 0 at public addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] at bind addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:45:38.983 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: mon.a@-1(???) e1 preinit fsid 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:45:38.983 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: mon.a@-1(???).mds e1 new map 2024-09-21T09:45:38.983 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: mon.a@-1(???).mds e1 print_map 2024-09-21T09:45:38.983 INFO:journalctl@ceph.mon.a.smithi098.stdout: e1 2024-09-21T09:45:38.983 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T09:45:38.983 INFO:journalctl@ceph.mon.a.smithi098.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:45:38.983 INFO:journalctl@ceph.mon.a.smithi098.stdout: legacy client fscid: -1 2024-09-21T09:45:38.983 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-09-21T09:45:38.983 INFO:journalctl@ceph.mon.a.smithi098.stdout: No filesystems configured 2024-09-21T09:45:38.983 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T09:45:38.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:45:38.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:45:38.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:45:38.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-21T09:45:38.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 bash[29220]: f501577d66cb7f2c3ae4a9f00ff86f8e580153126e1c088fd4f1612484a2be1f 2024-09-21T09:45:38.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 systemd[1]: Started Ceph mon.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:45:38.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T09:45:38.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: monmap e1: 1 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:45:38.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: fsmap 2024-09-21T09:45:38.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T09:45:38.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:38 smithi098 ceph-mon[29239]: mgrmap e1: no daemons active 2024-09-21T09:45:39.740 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-21T09:45:39.740 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-21T09:45:39.740 INFO:teuthology.orchestra.run.smithi098.stdout:Creating mgr... 2024-09-21T09:45:39.740 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-21T09:45:39.741 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-21T09:45:39.917 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-26a15512-77fe-11ef-baf6-efdc52797490@mgr.y 2024-09-21T09:45:39.917 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to reset failed state of unit ceph-26a15512-77fe-11ef-baf6-efdc52797490@mgr.y.service: Unit ceph-26a15512-77fe-11ef-baf6-efdc52797490@mgr.y.service not loaded. 2024-09-21T09:45:40.072 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-26a15512-77fe-11ef-baf6-efdc52797490.target.wants/ceph-26a15512-77fe-11ef-baf6-efdc52797490@mgr.y.service → /etc/systemd/system/ceph-26a15512-77fe-11ef-baf6-efdc52797490@.service. 2024-09-21T09:45:40.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:40 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/1664804318' entity='client.admin' 2024-09-21T09:45:40.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:40 smithi098 podman[29452]: 2024-09-21 09:45:40.201305122 +0000 UTC m=+0.015558843 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph:quincy 2024-09-21T09:45:40.597 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T09:45:40.597 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-09-21T09:45:40.605 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T09:45:40.605 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-09-21T09:45:40.605 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-09-21T09:45:40.605 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T09:45:40.613 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T09:45:40.613 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-09-21T09:45:40.621 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T09:45:40.621 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-09-21T09:45:40.621 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-09-21T09:45:40.621 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-21T09:45:40.621 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr to start... 2024-09-21T09:45:40.621 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr... 2024-09-21T09:45:40.658 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:40 smithi098 podman[29452]: 2024-09-21 09:45:40.481802288 +0000 UTC m=+0.296056006 container init 93d2672b85f819d52f6903436147786fb17efe41650ab39c2cfe507b07f15104 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y, maintainer=Guillaume Abrioux , ceph=True, RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T09:45:40.658 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:40 smithi098 podman[29452]: 2024-09-21 09:45:40.484361033 +0000 UTC m=+0.298614744 container start 93d2672b85f819d52f6903436147786fb17efe41650ab39c2cfe507b07f15104 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y, RELEASE=quincy-fbd1a6a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-21T09:45:40.658 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:40 smithi098 bash[29452]: 93d2672b85f819d52f6903436147786fb17efe41650ab39c2cfe507b07f15104 2024-09-21T09:45:40.658 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:40 smithi098 systemd[1]: Started Ceph mgr.y for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:45:40.658 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:40.596+0000 7f62d01efec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:45:40.658 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:40.656+0000 7f62d01efec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:45:40.962 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:40.720+0000 7f62d01efec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:45:40.962 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:40.847+0000 7f62d01efec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:45:41.260 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:41 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/304964466' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-21T09:45:41.261 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:40.989+0000 7f62d01efec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:45:41.261 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:41 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:41.258+0000 7f62d01efec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:45:41.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:41 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:41.322+0000 7f62d01efec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:45:41.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:41 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:41.485+0000 7f62d01efec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:45:41.659 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-21T09:45:41.664 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:45:41.664 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "26a15512-77fe-11ef-baf6-efdc52797490", 2024-09-21T09:45:41.665 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T09:45:41.665 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T09:45:41.665 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T09:45:41.665 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T09:45:41.665 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:45:41.665 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T09:45:41.665 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T09:45:41.665 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-09-21T09:45:41.665 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:45:41.665 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T09:45:41.665 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T09:45:41.665 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:45:41.665 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-21T09:45:41.665 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T09:45:41.665 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:45:41.666 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-21T09:45:41.666 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T09:45:41.666 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:45:41.666 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T09:45:41.666 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:45:41.666 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T09:45:41.666 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T09:45:41.666 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T09:45:41.666 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T09:45:41.666 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T09:45:41.666 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T09:45:41.666 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:45:41.666 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T09:45:41.666 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T09:45:41.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T09:45:41.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T09:45:41.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T09:45:41.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T09:45:41.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T09:45:41.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T09:45:41.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T09:45:41.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:45:41.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T09:45:41.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:45:41.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T09:45:41.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T09:45:41.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:45:41.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T09:45:41.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T09:45:41.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T09:45:41.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T09:45:41.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T09:45:41.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T09:45:41.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T09:45:41.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:45:41.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T09:45:41.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:45:41.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T09:45:41.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:45:41.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T09:45:33.813914+0000", 2024-09-21T09:45:41.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T09:45:41.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:45:41.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T09:45:41.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:45:41.669 INFO:teuthology.orchestra.run.smithi098.stdout:mgr not available, waiting (1/15)... 2024-09-21T09:45:42.596 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:42 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:42.324+0000 7f62d01efec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:45:42.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:42 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:42.388+0000 7f62d01efec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:45:43.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:43 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:43.088+0000 7f62d01efec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:45:43.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:43 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:43.198+0000 7f62d01efec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:45:43.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:43 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:43.317+0000 7f62d01efec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:45:43.768 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:43 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:43.376+0000 7f62d01efec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:45:43.768 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:43 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:43.500+0000 7f62d01efec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:45:43.768 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:43 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:43.565+0000 7f62d01efec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:45:43.768 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:43 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:43.621+0000 7f62d01efec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:45:43.768 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:43 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:43.766+0000 7f62d01efec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:45:44.035 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:43 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:43.831+0000 7f62d01efec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:45:44.036 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:43 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:43.900+0000 7f62d01efec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:45:44.036 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:43 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:43.984+0000 7f62d01efec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:45:44.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:44 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2077262253' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-21T09:45:44.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:44 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:44.043+0000 7f62d01efec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:45:44.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:44 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:44.104+0000 7f62d01efec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:45:44.785 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-21T09:45:44.785 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:45:44.785 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "26a15512-77fe-11ef-baf6-efdc52797490", 2024-09-21T09:45:44.785 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T09:45:44.785 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T09:45:44.785 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T09:45:44.785 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T09:45:44.785 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:45:44.785 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T09:45:44.785 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T09:45:44.785 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-09-21T09:45:44.785 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:45:44.786 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T09:45:44.786 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T09:45:44.786 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:45:44.786 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-21T09:45:44.786 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T09:45:44.786 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:45:44.786 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-21T09:45:44.786 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T09:45:44.786 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:45:44.786 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T09:45:44.786 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:45:44.786 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T09:45:44.786 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T09:45:44.786 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T09:45:44.786 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T09:45:44.790 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T09:45:44.790 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T09:45:44.790 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:45:44.790 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T09:45:44.790 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T09:45:44.790 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T09:45:44.790 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T09:45:44.790 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T09:45:44.790 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T09:45:44.790 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T09:45:44.790 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T09:45:44.790 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T09:45:44.790 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:45:44.791 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T09:45:44.791 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:45:44.791 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T09:45:44.791 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T09:45:44.791 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:45:44.791 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T09:45:44.791 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T09:45:44.791 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T09:45:44.791 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T09:45:44.791 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T09:45:44.791 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T09:45:44.791 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T09:45:44.791 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:45:44.791 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T09:45:44.791 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:45:44.792 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T09:45:44.792 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:45:44.792 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T09:45:33.813914+0000", 2024-09-21T09:45:44.792 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T09:45:44.792 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:45:44.792 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T09:45:44.792 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:45:44.792 INFO:teuthology.orchestra.run.smithi098.stdout:mgr not available, waiting (2/15)... 2024-09-21T09:45:45.096 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:44 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:44.864+0000 7f62d01efec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:45:45.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:45 smithi098 ceph-mon[29239]: Activating manager daemon y 2024-09-21T09:45:45.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:45 smithi098 ceph-mon[29239]: mgrmap e2: y(active, starting, since 0.00295248s) 2024-09-21T09:45:45.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:45 smithi098 ceph-mon[29239]: from='mgr.14100 172.21.15.98:0/1078186096' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:45:45.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:45 smithi098 ceph-mon[29239]: from='mgr.14100 172.21.15.98:0/1078186096' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:45:45.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:45 smithi098 ceph-mon[29239]: from='mgr.14100 172.21.15.98:0/1078186096' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:45:45.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:45 smithi098 ceph-mon[29239]: from='mgr.14100 172.21.15.98:0/1078186096' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:45:45.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:45 smithi098 ceph-mon[29239]: from='mgr.14100 172.21.15.98:0/1078186096' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:45:45.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:45 smithi098 ceph-mon[29239]: Manager daemon y is now available 2024-09-21T09:45:45.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:45 smithi098 ceph-mon[29239]: from='mgr.14100 172.21.15.98:0/1078186096' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:45:45.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:45 smithi098 ceph-mon[29239]: from='mgr.14100 172.21.15.98:0/1078186096' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:45:45.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:45 smithi098 ceph-mon[29239]: from='mgr.14100 172.21.15.98:0/1078186096' entity='mgr.y' 2024-09-21T09:45:45.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:45 smithi098 ceph-mon[29239]: from='mgr.14100 172.21.15.98:0/1078186096' entity='mgr.y' 2024-09-21T09:45:45.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:45 smithi098 ceph-mon[29239]: from='mgr.14100 172.21.15.98:0/1078186096' entity='mgr.y' 2024-09-21T09:45:47.160 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:46 smithi098 ceph-mon[29239]: mgrmap e3: y(active, since 1.00662s) 2024-09-21T09:45:47.952 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-21T09:45:47.952 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:45:47.952 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "26a15512-77fe-11ef-baf6-efdc52797490", 2024-09-21T09:45:47.952 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T09:45:47.952 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T09:45:47.952 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T09:45:47.953 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T09:45:47.953 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:45:47.953 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T09:45:47.953 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T09:45:47.953 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-09-21T09:45:47.953 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:45:47.953 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T09:45:47.953 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T09:45:47.953 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:45:47.953 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-09-21T09:45:47.953 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T09:45:47.953 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:45:47.953 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-21T09:45:47.953 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T09:45:47.953 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:45:47.954 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T09:45:47.954 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:45:47.954 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T09:45:47.954 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T09:45:47.954 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T09:45:47.954 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T09:45:47.954 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T09:45:47.954 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T09:45:47.957 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:45:47.957 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T09:45:47.957 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T09:45:47.957 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T09:45:47.957 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T09:45:47.958 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T09:45:47.958 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T09:45:47.958 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T09:45:47.958 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T09:45:47.958 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T09:45:47.958 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:45:47.958 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T09:45:47.958 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:45:47.958 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T09:45:47.958 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T09:45:47.958 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:45:47.958 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T09:45:47.958 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T09:45:47.958 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T09:45:47.959 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T09:45:47.959 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T09:45:47.959 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T09:45:47.959 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T09:45:47.959 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:45:47.959 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T09:45:47.959 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:45:47.959 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T09:45:47.959 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:45:47.959 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T09:45:33.813914+0000", 2024-09-21T09:45:47.959 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T09:45:47.959 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:45:47.959 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T09:45:47.959 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:45:47.959 INFO:teuthology.orchestra.run.smithi098.stdout:mgr is available 2024-09-21T09:45:48.180 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:47 smithi098 ceph-mon[29239]: mgrmap e4: y(active, since 2s) 2024-09-21T09:45:48.180 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:47 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3512251088' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-21T09:45:49.032 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-21T09:45:49.032 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T09:45:49.032 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout fsid = 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:45:49.033 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-21T09:45:49.033 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T09:45:49.033 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T09:45:49.033 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T09:45:49.033 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T09:45:49.033 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-21T09:45:49.033 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T09:45:49.033 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T09:45:49.033 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-09-21T09:45:49.033 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T09:45:49.033 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T09:45:49.033 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T09:45:49.033 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling cephadm module... 2024-09-21T09:45:49.138 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:48 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3483570389' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-21T09:45:50.226 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:49 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/1508531057' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-21T09:45:50.227 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:49 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ignoring --setuser ceph since I am not root 2024-09-21T09:45:50.227 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:49 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ignoring --setgroup ceph since I am not root 2024-09-21T09:45:50.227 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:50.006+0000 7ff543b0eec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:45:50.227 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:50.060+0000 7ff543b0eec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:45:50.227 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:50.124+0000 7ff543b0eec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:45:50.511 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:50.250+0000 7ff543b0eec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:45:50.511 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:50.385+0000 7ff543b0eec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:45:50.766 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:50.646+0000 7ff543b0eec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:45:50.766 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:50.705+0000 7ff543b0eec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:45:51.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:50 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/1508531057' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-21T09:45:51.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:50 smithi098 ceph-mon[29239]: mgrmap e5: y(active, since 5s) 2024-09-21T09:45:51.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:50.871+0000 7ff543b0eec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:45:52.096 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:51 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2809185311' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-21T09:45:52.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:51 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:51.719+0000 7ff543b0eec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:45:52.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:51 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:51.783+0000 7ff543b0eec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:45:52.775 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:52 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:52.485+0000 7ff543b0eec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:45:52.775 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:52 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:52.596+0000 7ff543b0eec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:45:52.775 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:52 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:52.715+0000 7ff543b0eec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:45:53.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:52 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:52.773+0000 7ff543b0eec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:45:53.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:52 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:52.898+0000 7ff543b0eec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:45:53.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:52 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:52.964+0000 7ff543b0eec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:45:53.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:53 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:53.022+0000 7ff543b0eec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:45:53.368 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:53 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:53.156+0000 7ff543b0eec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:45:53.368 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:53 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:53.220+0000 7ff543b0eec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:45:53.368 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:53 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:53.286+0000 7ff543b0eec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:45:53.368 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:53 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:53.366+0000 7ff543b0eec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:45:53.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:53 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:53.423+0000 7ff543b0eec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:45:53.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:53 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:53.480+0000 7ff543b0eec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:45:54.363 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:54 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:54.235+0000 7ff543b0eec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:45:54.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:54 smithi098 ceph-mon[29239]: Active manager daemon y restarted 2024-09-21T09:45:54.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:54 smithi098 ceph-mon[29239]: Activating manager daemon y 2024-09-21T09:45:54.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:54 smithi098 ceph-mon[29239]: osdmap e2: 0 total, 0 up, 0 in 2024-09-21T09:45:54.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:54 smithi098 ceph-mon[29239]: mgrmap e6: y(active, starting, since 0.00300634s) 2024-09-21T09:45:54.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:54 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:45:54.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:54 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:45:54.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:54 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:45:54.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:54 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:45:54.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:54 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:45:54.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:54 smithi098 ceph-mon[29239]: Manager daemon y is now available 2024-09-21T09:45:54.365 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:54 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' 2024-09-21T09:45:54.365 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:54 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' 2024-09-21T09:45:54.365 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:54 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:45:54.365 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:54 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:45:54.365 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:54 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:45:54.365 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:54 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:45:55.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:55 smithi098 ceph-mon[29239]: Found migration_current of "None". Setting to last migration. 2024-09-21T09:45:55.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:55 smithi098 ceph-mon[29239]: mgrmap e7: y(active, since 1.00576s) 2024-09-21T09:45:56.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:56.204+0000 7ff4f5efd640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:09:45:56] ENGINE Error in HTTPServer.serve 2024-09-21T09:45:56.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: Traceback (most recent call last): 2024-09-21T09:45:56.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:45:56.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: self._connections.run(self.expiration_interval) 2024-09-21T09:45:56.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:45:56.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: self._run(expiration_interval) 2024-09-21T09:45:56.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:45:56.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:45:56.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:45:56.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:45:56.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:45:56.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: s = self.context.wrap_socket( 2024-09-21T09:45:56.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:45:56.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: return self.sslsocket_class._create( 2024-09-21T09:45:56.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:45:56.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: self.do_handshake() 2024-09-21T09:45:56.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:45:56.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: self._sslobj.do_handshake() 2024-09-21T09:45:56.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:45:56.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:45:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:45:57.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:57 smithi098 ceph-mon[29239]: [21/Sep/2024:09:45:56] ENGINE Bus STARTING 2024-09-21T09:45:57.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:57 smithi098 ceph-mon[29239]: [21/Sep/2024:09:45:56] ENGINE Serving on https://172.21.15.98:7150 2024-09-21T09:45:57.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:57 smithi098 ceph-mon[29239]: [21/Sep/2024:09:45:56] ENGINE Bus STARTED 2024-09-21T09:45:57.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:57 smithi098 ceph-mon[29239]: [21/Sep/2024:09:45:56] ENGINE Error in HTTPServer.serve 2024-09-21T09:45:57.598 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-09-21T09:45:57.598 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:45:57.598 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:45:57.598 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:45:57.598 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._run(expiration_interval) 2024-09-21T09:45:57.598 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:45:57.598 INFO:journalctl@ceph.mon.a.smithi098.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:45:57.598 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:45:57.598 INFO:journalctl@ceph.mon.a.smithi098.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:45:57.598 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:45:57.598 INFO:journalctl@ceph.mon.a.smithi098.stdout: s = self.context.wrap_socket( 2024-09-21T09:45:57.598 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:45:57.599 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.sslsocket_class._create( 2024-09-21T09:45:57.599 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:45:57.599 INFO:journalctl@ceph.mon.a.smithi098.stdout: self.do_handshake() 2024-09-21T09:45:57.599 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:45:57.599 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._sslobj.do_handshake() 2024-09-21T09:45:57.599 INFO:journalctl@ceph.mon.a.smithi098.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:45:57.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:57 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' 2024-09-21T09:45:57.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:57 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:45:57.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:45:57.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-21T09:45:57.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T09:45:57.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-21T09:45:57.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T09:45:57.807 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:45:57.807 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for the mgr to restart... 2024-09-21T09:45:57.807 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr epoch 5... 2024-09-21T09:45:58.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:45:58 smithi098 ceph-mon[29239]: mgrmap e8: y(active, since 2s) 2024-09-21T09:46:02.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:01 smithi098 ceph-mon[29239]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-21T09:46:02.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:01 smithi098 ceph-mon[29239]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-21T09:46:03.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:46:03.264 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 8, 2024-09-21T09:46:03.264 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T09:46:03.264 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:46:03.264 INFO:teuthology.orchestra.run.smithi098.stdout:mgr epoch 5 is available 2024-09-21T09:46:03.265 INFO:teuthology.orchestra.run.smithi098.stdout:Setting orchestrator backend to cephadm... 2024-09-21T09:46:05.026 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:04 smithi098 ceph-mon[29239]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:05.026 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:04 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' 2024-09-21T09:46:05.026 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:04 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:46:05.609 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-21T09:46:05.609 INFO:teuthology.orchestra.run.smithi098.stdout:Generating ssh key... 2024-09-21T09:46:06.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:05 smithi098 ceph-mon[29239]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:06.563 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:06 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: Generating public/private rsa key pair. 2024-09-21T09:46:06.563 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:06 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: Your identification has been saved in /tmp/tmp9dk27wyg/key 2024-09-21T09:46:06.563 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:06 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: Your public key has been saved in /tmp/tmp9dk27wyg/key.pub 2024-09-21T09:46:06.563 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:06 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: The key fingerprint is: 2024-09-21T09:46:06.563 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:06 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: SHA256:rslpVPpRKqT2h0QrDCW+WK7h1Fx3pBOae8ndNrvuH5c ceph-26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:46:06.563 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:06 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: The key's randomart image is: 2024-09-21T09:46:06.564 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:06 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: +---[RSA 3072]----+ 2024-09-21T09:46:06.564 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:06 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: | | 2024-09-21T09:46:06.564 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:06 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: | . . . . | 2024-09-21T09:46:06.564 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:06 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: | . o o + | 2024-09-21T09:46:06.564 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:06 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: | + +o+... | 2024-09-21T09:46:06.564 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:06 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: | +o+.+++So. | 2024-09-21T09:46:06.564 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:06 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: |o.oo=.*=o. + .| 2024-09-21T09:46:06.564 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:06 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: |oo . =.+... o. E | 2024-09-21T09:46:06.564 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:06 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: |.. .++o . o | 2024-09-21T09:46:06.564 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:06 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: | .=. o+o. | 2024-09-21T09:46:06.564 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:06 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: +----[SHA256]-----+ 2024-09-21T09:46:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:07 smithi098 ceph-mon[29239]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:07 smithi098 ceph-mon[29239]: Generating ssh key... 2024-09-21T09:46:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:07 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' 2024-09-21T09:46:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:07 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' 2024-09-21T09:46:08.423 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtDBLCxsDhq7Yu9Nvn5EzGDs/MO7IXvsWIkXoHSIBh7tlaJpy0vb/2387lp9UPEMCqSzI9Z8Ems4mjR+EeOEZCfKhVK6nsp9ZGXGpOAQo6Ram70B3uYqwnQt5hwOP0bfOrIYhNnpFAKHVOre/cjBAEBXkfTZiVuWSPbTytKOVZpgvw0J5jo1Jil50K7eTGSHhSMcWp3Eva9bttTf7R4ngTbozPCKDklO7dm9QEAQViuO7KFjkkquE34FnwhT6IphzT7ShMpy8LAMSyJuTMwToQddZidpu0O1AVGRh4jNS5Q0kji6SgQaL+u5UOfxUlXaEHxcxDF10G+7PGMLiszK+2DKZQgwpH3wji50VItCHV7onguBy1Ae++75tmBXpyVmmqu3eLoONPMn/I7w/I3kmdOVwTc0abwdYVUnuHs/rSvgIETyOgzEFDEphjO8nij+ThA8Do30M2NLX5j3RjpWZKtpFLHOm8OiXhzeZ0TVphxdTvJbOMoS9Mm95EDLp6dak= ceph-26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:46:08.424 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-21T09:46:08.424 INFO:teuthology.orchestra.run.smithi098.stdout:Adding key to root@localhost authorized_keys... 2024-09-21T09:46:08.433 INFO:teuthology.orchestra.run.smithi098.stdout:Adding host smithi098... 2024-09-21T09:46:08.546 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:08 smithi098 ceph-mon[29239]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:10.014 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:09 smithi098 ceph-mon[29239]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi098", "addr": "172.21.15.98", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:10.014 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:09 smithi098 ceph-mon[29239]: Deploying cephadm binary to smithi098 2024-09-21T09:46:12.280 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Added host 'smithi098' with addr '172.21.15.98' 2024-09-21T09:46:12.280 INFO:teuthology.orchestra.run.smithi098.stdout:Deploying unmanaged mon service... 2024-09-21T09:46:12.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:12 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' 2024-09-21T09:46:12.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:12 smithi098 ceph-mon[29239]: Added host smithi098 2024-09-21T09:46:12.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:12 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:46:13.622 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-21T09:46:13.623 INFO:teuthology.orchestra.run.smithi098.stdout:Deploying unmanaged mgr service... 2024-09-21T09:46:13.940 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:13 smithi098 ceph-mon[29239]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:13.940 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:13 smithi098 ceph-mon[29239]: Saving service mon spec with placement count:5 2024-09-21T09:46:13.940 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:13 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' 2024-09-21T09:46:14.806 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-21T09:46:15.394 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:15 smithi098 ceph-mon[29239]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:15.394 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:15 smithi098 ceph-mon[29239]: Saving service mgr spec with placement count:2 2024-09-21T09:46:15.394 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:15 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' 2024-09-21T09:46:15.394 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:15 smithi098 ceph-mon[29239]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:46:16.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:16 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2040531221' entity='client.admin' 2024-09-21T09:46:16.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:16 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' 2024-09-21T09:46:17.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:17 smithi098 ceph-mon[29239]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:46:17.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:17 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' 2024-09-21T09:46:17.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:17 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3830049197' entity='client.admin' 2024-09-21T09:46:18.377 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling the dashboard module... 2024-09-21T09:46:19.263 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:19 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' 2024-09-21T09:46:19.263 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:19 smithi098 ceph-mon[29239]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:46:19.263 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:19 smithi098 ceph-mon[29239]: from='mgr.14118 172.21.15.98:0/886212918' entity='mgr.y' 2024-09-21T09:46:20.270 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:20 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/4263709568' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-21T09:46:20.270 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ignoring --setuser ceph since I am not root 2024-09-21T09:46:20.271 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ignoring --setgroup ceph since I am not root 2024-09-21T09:46:20.271 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:20.149+0000 7f29f95d2ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:46:20.271 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:20.203+0000 7f29f95d2ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:46:20.271 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:20.268+0000 7f29f95d2ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:46:20.532 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:20.394+0000 7f29f95d2ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:46:20.793 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:20.530+0000 7f29f95d2ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:46:21.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:21 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/4263709568' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-21T09:46:21.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:21 smithi098 ceph-mon[29239]: mgrmap e9: y(active, since 25s) 2024-09-21T09:46:21.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:20.791+0000 7f29f95d2ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:46:21.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:20.850+0000 7f29f95d2ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:46:21.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:21.019+0000 7f29f95d2ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:46:21.954 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:21.887+0000 7f29f95d2ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:46:21.955 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:21.952+0000 7f29f95d2ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:46:21.999 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:46:21.999 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-21T09:46:22.000 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T09:46:22.000 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-21T09:46:22.000 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T09:46:22.000 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:46:22.000 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for the mgr to restart... 2024-09-21T09:46:22.000 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr epoch 9... 2024-09-21T09:46:22.231 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:22 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/295726105' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-21T09:46:22.964 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:22 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:22.673+0000 7f29f95d2ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:46:22.964 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:22 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:22.784+0000 7f29f95d2ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:46:22.964 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:22 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:22.902+0000 7f29f95d2ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:46:22.964 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:22 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:22.961+0000 7f29f95d2ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:46:23.345 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:23 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:23.087+0000 7f29f95d2ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:46:23.345 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:23 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:23.153+0000 7f29f95d2ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:46:23.345 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:23 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:23.210+0000 7f29f95d2ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:46:23.346 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:23 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:23.343+0000 7f29f95d2ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:46:23.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:23 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:23.411+0000 7f29f95d2ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:46:23.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:23 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:23.478+0000 7f29f95d2ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:46:23.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:23 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:23.562+0000 7f29f95d2ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:46:24.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:23 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:23.623+0000 7f29f95d2ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:46:24.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:23 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:23.681+0000 7f29f95d2ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:46:24.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:24 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:24.465+0000 7f29f95d2ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:46:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:24 smithi098 ceph-mon[29239]: Active manager daemon y restarted 2024-09-21T09:46:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:24 smithi098 ceph-mon[29239]: Activating manager daemon y 2024-09-21T09:46:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:24 smithi098 ceph-mon[29239]: osdmap e3: 0 total, 0 up, 0 in 2024-09-21T09:46:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:24 smithi098 ceph-mon[29239]: mgrmap e10: y(active, starting, since 0.00325809s) 2024-09-21T09:46:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:24 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:46:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:24 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:46:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:24 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:46:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:24 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:46:24.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:24 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:46:24.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:24 smithi098 ceph-mon[29239]: Manager daemon y is now available 2024-09-21T09:46:24.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:24 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:46:24.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:24 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:46:24.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:24 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:46:26.361 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:46:26.362 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-21T09:46:26.362 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T09:46:26.362 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:46:26.362 INFO:teuthology.orchestra.run.smithi098.stdout:mgr epoch 9 is available 2024-09-21T09:46:26.362 INFO:teuthology.orchestra.run.smithi098.stdout:Generating a dashboard self-signed certificate... 2024-09-21T09:46:26.635 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-mon[29239]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-21T09:46:26.636 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-mon[29239]: mgrmap e11: y(active, since 1.0055s) 2024-09-21T09:46:26.636 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-mon[29239]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-21T09:46:26.636 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:26.972 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:26.632+0000 7f29aa130640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:09:46:26] ENGINE Error in HTTPServer.serve 2024-09-21T09:46:26.972 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: Traceback (most recent call last): 2024-09-21T09:46:26.972 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:46:26.972 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: self._connections.run(self.expiration_interval) 2024-09-21T09:46:26.973 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:46:26.973 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: self._run(expiration_interval) 2024-09-21T09:46:26.973 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:46:26.973 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:46:26.973 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:46:26.973 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:46:26.973 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:46:26.973 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: s = self.context.wrap_socket( 2024-09-21T09:46:26.973 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:46:26.973 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: return self.sslsocket_class._create( 2024-09-21T09:46:26.973 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:46:26.973 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: self.do_handshake() 2024-09-21T09:46:26.973 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:46:26.973 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: self._sslobj.do_handshake() 2024-09-21T09:46:26.973 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:46:26.973 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:46:26 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:46:27.573 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-21T09:46:27.573 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial admin user... 2024-09-21T09:46:27.820 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:27 smithi098 ceph-mon[29239]: [21/Sep/2024:09:46:26] ENGINE Bus STARTING 2024-09-21T09:46:27.820 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:27 smithi098 ceph-mon[29239]: [21/Sep/2024:09:46:26] ENGINE Serving on https://172.21.15.98:7150 2024-09-21T09:46:27.821 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:27 smithi098 ceph-mon[29239]: [21/Sep/2024:09:46:26] ENGINE Bus STARTED 2024-09-21T09:46:27.821 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:27 smithi098 ceph-mon[29239]: [21/Sep/2024:09:46:26] ENGINE Error in HTTPServer.serve 2024-09-21T09:46:27.821 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-09-21T09:46:27.821 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:46:27.821 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:46:27.821 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:46:27.821 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._run(expiration_interval) 2024-09-21T09:46:27.821 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:46:27.821 INFO:journalctl@ceph.mon.a.smithi098.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:46:27.821 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:46:27.821 INFO:journalctl@ceph.mon.a.smithi098.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:46:27.821 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:46:27.821 INFO:journalctl@ceph.mon.a.smithi098.stdout: s = self.context.wrap_socket( 2024-09-21T09:46:27.821 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:46:27.822 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.sslsocket_class._create( 2024-09-21T09:46:27.822 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:46:27.822 INFO:journalctl@ceph.mon.a.smithi098.stdout: self.do_handshake() 2024-09-21T09:46:27.822 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:46:27.822 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._sslobj.do_handshake() 2024-09-21T09:46:27.822 INFO:journalctl@ceph.mon.a.smithi098.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:46:27.822 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:27 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:27.822 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:27 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:27.822 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:27 smithi098 ceph-mon[29239]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:27.822 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:27 smithi098 ceph-mon[29239]: mgrmap e12: y(active, since 2s) 2024-09-21T09:46:27.822 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:27 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:27.822 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:27 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:29.645 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:29 smithi098 ceph-mon[29239]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:29.645 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:29 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:29.647 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$s1Z2LNhEltOs/k.7ehZste3Ue8eluYI1xbdZz3hiHBqlv1ELQvxrK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726911988, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-21T09:46:29.647 INFO:teuthology.orchestra.run.smithi098.stdout:Fetching dashboard port number... 2024-09-21T09:46:30.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:30 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/1471785279' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-21T09:46:30.759 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 8443 2024-09-21T09:46:30.769 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T09:46:30.769 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-09-21T09:46:30.777 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T09:46:30.777 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-09-21T09:46:30.777 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-09-21T09:46:30.777 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-21T09:46:30.779 INFO:teuthology.orchestra.run.smithi098.stdout:Ceph Dashboard is now available at: 2024-09-21T09:46:30.779 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:46:30.779 INFO:teuthology.orchestra.run.smithi098.stdout: URL: https://smithi098.front.sepia.ceph.com:8443/ 2024-09-21T09:46:30.779 INFO:teuthology.orchestra.run.smithi098.stdout: User: admin 2024-09-21T09:46:30.779 INFO:teuthology.orchestra.run.smithi098.stdout: Password: 4w3dz4m5td 2024-09-21T09:46:30.779 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:46:30.779 INFO:teuthology.orchestra.run.smithi098.stdout:Saving cluster configuration to /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config directory 2024-09-21T09:46:30.780 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling autotune for osd_memory_target 2024-09-21T09:46:32.804 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:32 smithi098 ceph-mon[29239]: mgrmap e13: y(active, since 6s) 2024-09-21T09:46:33.040 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-21T09:46:33.040 INFO:teuthology.orchestra.run.smithi098.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-21T09:46:33.040 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:46:33.040 INFO:teuthology.orchestra.run.smithi098.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-21T09:46:33.040 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:46:33.040 INFO:teuthology.orchestra.run.smithi098.stdout:Or, if you are only running a single cluster on this host: 2024-09-21T09:46:33.040 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:46:33.040 INFO:teuthology.orchestra.run.smithi098.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-21T09:46:33.040 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:46:33.040 INFO:teuthology.orchestra.run.smithi098.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-21T09:46:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:46:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: ceph telemetry on 2024-09-21T09:46:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:46:33.041 INFO:teuthology.orchestra.run.smithi098.stdout:For more information see: 2024-09-21T09:46:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:46:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-09-21T09:46:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:46:33.041 INFO:teuthology.orchestra.run.smithi098.stdout:Bootstrap complete. 2024-09-21T09:46:33.074 INFO:tasks.cephadm:Fetching config... 2024-09-21T09:46:33.074 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-21T09:46:33.074 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-21T09:46:33.089 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-21T09:46:33.089 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-21T09:46:33.089 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-21T09:46:33.146 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-21T09:46:33.146 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-21T09:46:33.146 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/keyring of=/dev/stdout 2024-09-21T09:46:33.210 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-21T09:46:33.211 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-21T09:46:33.211 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-21T09:46:33.266 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-21T09:46:33.266 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtDBLCxsDhq7Yu9Nvn5EzGDs/MO7IXvsWIkXoHSIBh7tlaJpy0vb/2387lp9UPEMCqSzI9Z8Ems4mjR+EeOEZCfKhVK6nsp9ZGXGpOAQo6Ram70B3uYqwnQt5hwOP0bfOrIYhNnpFAKHVOre/cjBAEBXkfTZiVuWSPbTytKOVZpgvw0J5jo1Jil50K7eTGSHhSMcWp3Eva9bttTf7R4ngTbozPCKDklO7dm9QEAQViuO7KFjkkquE34FnwhT6IphzT7ShMpy8LAMSyJuTMwToQddZidpu0O1AVGRh4jNS5Q0kji6SgQaL+u5UOfxUlXaEHxcxDF10G+7PGMLiszK+2DKZQgwpH3wji50VItCHV7onguBy1Ae++75tmBXpyVmmqu3eLoONPMn/I7w/I3kmdOVwTc0abwdYVUnuHs/rSvgIETyOgzEFDEphjO8nij+ThA8Do30M2NLX5j3RjpWZKtpFLHOm8OiXhzeZ0TVphxdTvJbOMoS9Mm95EDLp6dak= ceph-26a15512-77fe-11ef-baf6-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T09:46:33.340 INFO:teuthology.orchestra.run.smithi098.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtDBLCxsDhq7Yu9Nvn5EzGDs/MO7IXvsWIkXoHSIBh7tlaJpy0vb/2387lp9UPEMCqSzI9Z8Ems4mjR+EeOEZCfKhVK6nsp9ZGXGpOAQo6Ram70B3uYqwnQt5hwOP0bfOrIYhNnpFAKHVOre/cjBAEBXkfTZiVuWSPbTytKOVZpgvw0J5jo1Jil50K7eTGSHhSMcWp3Eva9bttTf7R4ngTbozPCKDklO7dm9QEAQViuO7KFjkkquE34FnwhT6IphzT7ShMpy8LAMSyJuTMwToQddZidpu0O1AVGRh4jNS5Q0kji6SgQaL+u5UOfxUlXaEHxcxDF10G+7PGMLiszK+2DKZQgwpH3wji50VItCHV7onguBy1Ae++75tmBXpyVmmqu3eLoONPMn/I7w/I3kmdOVwTc0abwdYVUnuHs/rSvgIETyOgzEFDEphjO8nij+ThA8Do30M2NLX5j3RjpWZKtpFLHOm8OiXhzeZ0TVphxdTvJbOMoS9Mm95EDLp6dak= ceph-26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:46:33.351 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtDBLCxsDhq7Yu9Nvn5EzGDs/MO7IXvsWIkXoHSIBh7tlaJpy0vb/2387lp9UPEMCqSzI9Z8Ems4mjR+EeOEZCfKhVK6nsp9ZGXGpOAQo6Ram70B3uYqwnQt5hwOP0bfOrIYhNnpFAKHVOre/cjBAEBXkfTZiVuWSPbTytKOVZpgvw0J5jo1Jil50K7eTGSHhSMcWp3Eva9bttTf7R4ngTbozPCKDklO7dm9QEAQViuO7KFjkkquE34FnwhT6IphzT7ShMpy8LAMSyJuTMwToQddZidpu0O1AVGRh4jNS5Q0kji6SgQaL+u5UOfxUlXaEHxcxDF10G+7PGMLiszK+2DKZQgwpH3wji50VItCHV7onguBy1Ae++75tmBXpyVmmqu3eLoONPMn/I7w/I3kmdOVwTc0abwdYVUnuHs/rSvgIETyOgzEFDEphjO8nij+ThA8Do30M2NLX5j3RjpWZKtpFLHOm8OiXhzeZ0TVphxdTvJbOMoS9Mm95EDLp6dak= ceph-26a15512-77fe-11ef-baf6-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T09:46:33.479 INFO:teuthology.orchestra.run.smithi196.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtDBLCxsDhq7Yu9Nvn5EzGDs/MO7IXvsWIkXoHSIBh7tlaJpy0vb/2387lp9UPEMCqSzI9Z8Ems4mjR+EeOEZCfKhVK6nsp9ZGXGpOAQo6Ram70B3uYqwnQt5hwOP0bfOrIYhNnpFAKHVOre/cjBAEBXkfTZiVuWSPbTytKOVZpgvw0J5jo1Jil50K7eTGSHhSMcWp3Eva9bttTf7R4ngTbozPCKDklO7dm9QEAQViuO7KFjkkquE34FnwhT6IphzT7ShMpy8LAMSyJuTMwToQddZidpu0O1AVGRh4jNS5Q0kji6SgQaL+u5UOfxUlXaEHxcxDF10G+7PGMLiszK+2DKZQgwpH3wji50VItCHV7onguBy1Ae++75tmBXpyVmmqu3eLoONPMn/I7w/I3kmdOVwTc0abwdYVUnuHs/rSvgIETyOgzEFDEphjO8nij+ThA8Do30M2NLX5j3RjpWZKtpFLHOm8OiXhzeZ0TVphxdTvJbOMoS9Mm95EDLp6dak= ceph-26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:46:33.493 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-21T09:46:33.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:33 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3741173220' entity='client.admin' 2024-09-21T09:46:33.736 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:46:36.200 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:35 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:36.200 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:35 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:36.200 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:35 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:46:36.200 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:35 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:36.200 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:35 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:46:36.200 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:35 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:37 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/10668807' entity='client.admin' 2024-09-21T09:46:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:37 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:46:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:37 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:37.601 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-21T09:46:37.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-21T09:46:37.844 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:46:40.374 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi196 2024-09-21T09:46:40.374 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-21T09:46:40.374 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/etc/ceph/ceph.conf 2024-09-21T09:46:40.393 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-21T09:46:40.393 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:46:40.450 INFO:tasks.cephadm:Adding host smithi196 to orchestrator... 2024-09-21T09:46:40.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph orch host add smithi196 2024-09-21T09:46:40.692 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:46:40.907 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:40 smithi098 ceph-mon[29239]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:40.907 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:40 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:40.908 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:40 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:46:40.908 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:40 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:46:40.908 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:40 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:46:40.908 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:40 smithi098 ceph-mon[29239]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-21T09:46:40.908 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:40 smithi098 ceph-mon[29239]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:46:40.908 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:40 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:40.908 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:40 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:40.908 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:40 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:41.905 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:41 smithi098 ceph-mon[29239]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:46:41.905 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:41 smithi098 ceph-mon[29239]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:46:43.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:43 smithi098 ceph-mon[29239]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi196", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:44.346 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:44 smithi098 ceph-mon[29239]: Deploying cephadm binary to smithi196 2024-09-21T09:46:45.041 INFO:teuthology.orchestra.run.smithi098.stdout:Added host 'smithi196' with addr '172.21.15.196' 2024-09-21T09:46:45.629 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph orch host ls --format=json 2024-09-21T09:46:45.864 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:46:46.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:46 smithi098 ceph-mon[29239]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:46:46.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:46 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:46.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:46 smithi098 ceph-mon[29239]: Added host smithi196 2024-09-21T09:46:46.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:46 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:46:46.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:46 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:46.238 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:46 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:47.603 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:46:47.603 INFO:teuthology.orchestra.run.smithi098.stdout:[{"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}, {"addr": "172.21.15.196", "hostname": "smithi196", "labels": [], "status": ""}] 2024-09-21T09:46:47.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:47 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:47.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:47 smithi098 ceph-mon[29239]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:46:48.169 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-21T09:46:48.170 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd crush tunables default 2024-09-21T09:46:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:46:48.520 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:48 smithi098 ceph-mon[29239]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:46:49.775 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:49 smithi098 ceph-mon[29239]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:46:49.775 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:49 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:49.776 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:49 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:49.776 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:49 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:49.776 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:49 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:49.776 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:49 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:46:49.776 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:49 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:46:49.776 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:49 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:46:50.962 INFO:teuthology.orchestra.run.smithi098.stderr:adjusted tunables profile to default 2024-09-21T09:46:51.309 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:50 smithi098 ceph-mon[29239]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-21T09:46:51.309 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:50 smithi098 ceph-mon[29239]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:46:51.309 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:50 smithi098 ceph-mon[29239]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:46:51.309 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:50 smithi098 ceph-mon[29239]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:46:51.309 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:50 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:51.309 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:50 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:51.310 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:50 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:51.310 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:50 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/645294993' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-21T09:46:51.598 INFO:tasks.cephadm:Adding mon.a on smithi098 2024-09-21T09:46:51.598 INFO:tasks.cephadm:Adding mon.c on smithi098 2024-09-21T09:46:51.598 INFO:tasks.cephadm:Adding mon.b on smithi196 2024-09-21T09:46:51.598 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph orch apply mon '3;smithi098:172.21.15.98=a;smithi098:[v2:172.21.15.98:3301,v1:172.21.15.98:6790]=c;smithi196:172.21.15.196=b' 2024-09-21T09:46:51.838 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:46:52.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:51 smithi098 ceph-mon[29239]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:46:52.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:51 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/645294993' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T09:46:52.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:51 smithi098 ceph-mon[29239]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:46:53.528 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled mon update... 2024-09-21T09:46:54.107 DEBUG:teuthology.orchestra.run.smithi098:mon.c> sudo journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@mon.c.service 2024-09-21T09:46:54.110 DEBUG:teuthology.orchestra.run.smithi196:mon.b> sudo journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@mon.b.service 2024-09-21T09:46:54.113 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T09:46:54.113 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T09:46:54.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:53 smithi098 ceph-mon[29239]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:46:54.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:53 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:54.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:53 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:46:54.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:53 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:46:54.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:53 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:46:54.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:53 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:54.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:53 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:46:54.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:53 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:46:55.343 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:46:55.346 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:54 smithi098 ceph-mon[29239]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi098:[v2:172.21.15.98:3301,v1:172.21.15.98:6790]=c;smithi196:172.21.15.196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:55.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:54 smithi098 ceph-mon[29239]: Saving service mon spec with placement smithi098:172.21.15.98=a;smithi098:[v2:172.21.15.98:3301,v1:172.21.15.98:6790]=c;smithi196:172.21.15.196=b;count:3 2024-09-21T09:46:55.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:54 smithi098 ceph-mon[29239]: Deploying daemon mon.b on smithi196 2024-09-21T09:46:56.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:55 smithi098 ceph-mon[29239]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:46:57.520 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:57 smithi196 systemd[1]: Starting Ceph mon.b for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:46:58.070 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:46:58.071 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"26a15512-77fe-11ef-baf6-efdc52797490","modified":"2024-09-21T09:45:30.814529Z","created":"2024-09-21T09:45:30.814529Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T09:46:58.071 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-09-21T09:46:58.081 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 podman[32639]: 2024-09-21 09:46:57.920373875 +0000 UTC m=+0.017798519 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T09:46:58.081 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 podman[32639]: 2024-09-21 09:46:58.017483743 +0000 UTC m=+0.114908375 container create 4f1a8a66c2c869baf95034670b981005ccdf657396ee0d486e2e892e2b19bac8 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-b, org.label-schema.vendor=CentOS, RELEASE=quincy-fbd1a6a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731) 2024-09-21T09:46:58.346 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:46:57 smithi098 ceph-mon[29239]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:46:58.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 podman[32639]: 2024-09-21 09:46:58.345654429 +0000 UTC m=+0.443079061 container init 4f1a8a66c2c869baf95034670b981005ccdf657396ee0d486e2e892e2b19bac8 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=quincy-fbd1a6a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-21T09:46:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 podman[32639]: 2024-09-21 09:46:58.348892373 +0000 UTC m=+0.446317006 container start 4f1a8a66c2c869baf95034670b981005ccdf657396ee0d486e2e892e2b19bac8 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-b, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-21T09:46:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T09:46:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable), process ceph-mon, pid 2 2024-09-21T09:46:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: pidfile_write: ignore empty --pid-file 2024-09-21T09:46:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: load: jerasure load: lrc 2024-09-21T09:46:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: RocksDB version: 6.15.5 2024-09-21T09:46:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-21T09:46:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Compile date Sep 20 2024 2024-09-21T09:46:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: DB SUMMARY 2024-09-21T09:46:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: DB Session ID: YF1DW6CHSI7L5SPY4JW9 2024-09-21T09:46:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: CURRENT file: CURRENT 2024-09-21T09:46:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T09:46:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-09-21T09:46:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-21T09:46:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-09-21T09:46:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.error_if_exists: 0 2024-09-21T09:46:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.create_if_missing: 0 2024-09-21T09:46:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.paranoid_checks: 1 2024-09-21T09:46:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T09:46:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.env: 0x5612f7e9d440 2024-09-21T09:46:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.fs: Posix File System 2024-09-21T09:46:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.info_log: 0x5612fa13aec0 2024-09-21T09:46:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T09:46:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.statistics: (nil) 2024-09-21T09:46:58.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.use_fsync: 0 2024-09-21T09:46:58.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_log_file_size: 0 2024-09-21T09:46:58.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T09:46:58.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T09:46:58.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T09:46:58.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T09:46:58.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.allow_fallocate: 1 2024-09-21T09:46:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T09:46:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T09:46:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.use_direct_reads: 0 2024-09-21T09:46:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T09:46:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T09:46:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.db_log_dir: 2024-09-21T09:46:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-09-21T09:46:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T09:46:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T09:46:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T09:46:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T09:46:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T09:46:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T09:46:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T09:46:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T09:46:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.write_buffer_manager: 0x5612fa1f87e0 2024-09-21T09:46:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T09:46:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-21T09:46:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T09:46:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T09:46:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T09:46:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T09:46:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T09:46:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T09:46:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T09:46:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.unordered_write: 0 2024-09-21T09:46:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T09:46:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T09:46:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T09:46:58.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T09:46:58.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.row_cache: None 2024-09-21T09:46:58.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.wal_filter: None 2024-09-21T09:46:58.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T09:46:58.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T09:46:58.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.preserve_deletes: 0 2024-09-21T09:46:58.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.two_write_queues: 0 2024-09-21T09:46:58.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T09:46:58.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.atomic_flush: 0 2024-09-21T09:46:58.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T09:46:58.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T09:46:58.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T09:46:58.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.log_readahead_size: 0 2024-09-21T09:46:58.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T09:46:58.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T09:46:58.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T09:46:58.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T09:46:58.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T09:46:58.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T09:46:58.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_background_jobs: 2 2024-09-21T09:46:58.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_background_compactions: -1 2024-09-21T09:46:58.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_subcompactions: 1 2024-09-21T09:46:58.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T09:46:58.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T09:46:58.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T09:46:58.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T09:46:58.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T09:46:58.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T09:46:58.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T09:46:58.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T09:46:58.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_open_files: -1 2024-09-21T09:46:58.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T09:46:58.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T09:46:58.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T09:46:58.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T09:46:58.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_background_flushes: -1 2024-09-21T09:46:58.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Compression algorithms supported: 2024-09-21T09:46:58.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: kZSTD supported: 0 2024-09-21T09:46:58.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: kXpressCompression supported: 0 2024-09-21T09:46:58.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: kBZip2Compression supported: 0 2024-09-21T09:46:58.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T09:46:58.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: kLZ4Compression supported: 1 2024-09-21T09:46:58.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: kZlibCompression supported: 1 2024-09-21T09:46:58.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T09:46:58.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: kSnappyCompression supported: 1 2024-09-21T09:46:58.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T09:46:58.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-09-21T09:46:58.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-21T09:46:58.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T09:46:58.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.merge_operator: 2024-09-21T09:46:58.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.compaction_filter: None 2024-09-21T09:46:58.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T09:46:58.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T09:46:58.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T09:46:58.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T09:46:58.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5612fa0e1a38) 2024-09-21T09:46:58.649 INFO:journalctl@ceph.mon.b.smithi196.stdout: cache_index_and_filter_blocks: 1 2024-09-21T09:46:58.649 INFO:journalctl@ceph.mon.b.smithi196.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T09:46:58.649 INFO:journalctl@ceph.mon.b.smithi196.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T09:46:58.649 INFO:journalctl@ceph.mon.b.smithi196.stdout: pin_top_level_index_and_filter: 1 2024-09-21T09:46:58.649 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_type: 0 2024-09-21T09:46:58.649 INFO:journalctl@ceph.mon.b.smithi196.stdout: data_block_index_type: 0 2024-09-21T09:46:58.649 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_shortening: 1 2024-09-21T09:46:58.649 INFO:journalctl@ceph.mon.b.smithi196.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T09:46:58.649 INFO:journalctl@ceph.mon.b.smithi196.stdout: hash_index_allow_collision: 1 2024-09-21T09:46:58.649 INFO:journalctl@ceph.mon.b.smithi196.stdout: checksum: 1 2024-09-21T09:46:58.649 INFO:journalctl@ceph.mon.b.smithi196.stdout: no_block_cache: 0 2024-09-21T09:46:58.649 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache: 0x5612fa1171f0 2024-09-21T09:46:58.649 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_name: BinnedLRUCache 2024-09-21T09:46:58.649 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_options: 2024-09-21T09:46:58.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: capacity : 536870912 2024-09-21T09:46:58.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: num_shard_bits : 4 2024-09-21T09:46:58.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: strict_capacity_limit : 0 2024-09-21T09:46:58.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: high_pri_pool_ratio: 0.000 2024-09-21T09:46:58.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_compressed: (nil) 2024-09-21T09:46:58.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: persistent_cache: (nil) 2024-09-21T09:46:58.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_size: 4096 2024-09-21T09:46:58.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_size_deviation: 10 2024-09-21T09:46:58.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_restart_interval: 16 2024-09-21T09:46:58.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_block_restart_interval: 1 2024-09-21T09:46:58.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: metadata_block_size: 4096 2024-09-21T09:46:58.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: partition_filters: 0 2024-09-21T09:46:58.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: use_delta_encoding: 1 2024-09-21T09:46:58.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-21T09:46:58.651 INFO:journalctl@ceph.mon.b.smithi196.stdout: whole_key_filtering: 1 2024-09-21T09:46:58.651 INFO:journalctl@ceph.mon.b.smithi196.stdout: verify_compression: 0 2024-09-21T09:46:58.651 INFO:journalctl@ceph.mon.b.smithi196.stdout: read_amp_bytes_per_bit: 0 2024-09-21T09:46:58.651 INFO:journalctl@ceph.mon.b.smithi196.stdout: format_version: 4 2024-09-21T09:46:58.651 INFO:journalctl@ceph.mon.b.smithi196.stdout: enable_index_compression: 1 2024-09-21T09:46:58.651 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_align: 0 2024-09-21T09:46:58.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T09:46:58.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T09:46:58.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.compression: NoCompression 2024-09-21T09:46:58.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T09:46:58.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T09:46:58.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T09:46:58.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.num_levels: 7 2024-09-21T09:46:58.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T09:46:58.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T09:46:58.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T09:46:58.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T09:46:58.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T09:46:58.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T09:46:58.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T09:46:58.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:46:58.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T09:46:58.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T09:46:58.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T09:46:58.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T09:46:58.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T09:46:58.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T09:46:58.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:46:58.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T09:46:58.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T09:46:58.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T09:46:58.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T09:46:58.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T09:46:58.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T09:46:58.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T09:46:58.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T09:46:58.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T09:46:58.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T09:46:58.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T09:46:58.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T09:46:58.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T09:46:58.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T09:46:58.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T09:46:58.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T09:46:58.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T09:46:58.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T09:46:58.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T09:46:58.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.arena_block_size: 4194304 2024-09-21T09:46:58.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T09:46:58.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T09:46:58.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-21T09:46:58.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T09:46:58.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T09:46:58.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T09:46:58.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T09:46:58.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T09:46:58.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T09:46:58.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T09:46:58.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T09:46:58.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T09:46:58.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T09:46:58.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T09:46:58.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T09:46:58.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.inplace_update_support: 0 2024-09-21T09:46:58.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T09:46:58.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T09:46:58.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T09:46:58.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T09:46:58.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.bloom_locality: 0 2024-09-21T09:46:58.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.max_successive_merges: 0 2024-09-21T09:46:58.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T09:46:58.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T09:46:58.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T09:46:58.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T09:46:58.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.ttl: 2592000 2024-09-21T09:46:58.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T09:46:58.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.enable_blob_files: false 2024-09-21T09:46:58.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.min_blob_size: 0 2024-09-21T09:46:58.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T09:46:58.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T09:46:58.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T09:46:58.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T09:46:58.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T09:46:58.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-09-21T09:46:58.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-09-21T09:46:58.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912018373195, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-21T09:46:58.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-09-21T09:46:58.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-21T09:46:58.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912018373497, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1571, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726912018, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "b017cf38-cc8c-4ac1-8581-518c0ad88a40", "db_session_id": "YF1DW6CHSI7L5SPY4JW9"}} 2024-09-21T09:46:58.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-09-21T09:46:58.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912018373859, "job": 1, "event": "recovery_finished"} 2024-09-21T09:46:58.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T09:46:58.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x5612fa14a700 2024-09-21T09:46:58.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: DB pointer 0x5612fa11a000 2024-09-21T09:46:58.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-21T09:46:58.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-21T09:46:58.657 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** DB Stats ** 2024-09-21T09:46:58.657 INFO:journalctl@ceph.mon.b.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:46:58.657 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T09:46:58.657 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:46:58.657 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:46:58.657 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T09:46:58.658 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-21T09:46:58.658 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:46:58.658 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-21T09:46:58.658 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** Compaction Stats [default] ** 2024-09-21T09:46:58.658 INFO:journalctl@ceph.mon.b.smithi196.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:46:58.658 INFO:journalctl@ceph.mon.b.smithi196.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:46:58.658 INFO:journalctl@ceph.mon.b.smithi196.stdout: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:46:58.658 INFO:journalctl@ceph.mon.b.smithi196.stdout: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:46:58.658 INFO:journalctl@ceph.mon.b.smithi196.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:46:58.658 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-21T09:46:58.658 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** Compaction Stats [default] ** 2024-09-21T09:46:58.658 INFO:journalctl@ceph.mon.b.smithi196.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:46:58.658 INFO:journalctl@ceph.mon.b.smithi196.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:46:58.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:46:58.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:46:58.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T09:46:58.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:46:58.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:46:58.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:46:58.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:46:58.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:46:58.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:46:58.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:46:58.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-21T09:46:58.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:46:58.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-21T09:46:58.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** Compaction Stats [default] ** 2024-09-21T09:46:58.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:46:58.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:46:58.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:46:58.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:46:58.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-21T09:46:58.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-21T09:46:58.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** Compaction Stats [default] ** 2024-09-21T09:46:58.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:46:58.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:46:58.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:46:58.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:46:58.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T09:46:58.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:46:58.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:46:58.661 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:46:58.661 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:46:58.661 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:46:58.661 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:46:58.661 INFO:journalctl@ceph.mon.b.smithi196.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:46:58.661 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-21T09:46:58.661 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:46:58.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-21T09:46:58.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: using public_addr v2:172.21.15.196:0/0 -> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] 2024-09-21T09:46:58.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: starting mon.b rank -1 at public addrs [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] at bind addrs [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:46:58.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: mon.b@-1(???) e0 preinit fsid 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:46:58.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: mon.b@-1(synchronizing).mds e1 new map 2024-09-21T09:46:58.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-21T09:46:58.661 INFO:journalctl@ceph.mon.b.smithi196.stdout: e1 2024-09-21T09:46:58.662 INFO:journalctl@ceph.mon.b.smithi196.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T09:46:58.662 INFO:journalctl@ceph.mon.b.smithi196.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:46:58.662 INFO:journalctl@ceph.mon.b.smithi196.stdout: legacy client fscid: -1 2024-09-21T09:46:58.662 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-21T09:46:58.662 INFO:journalctl@ceph.mon.b.smithi196.stdout: No filesystems configured 2024-09-21T09:46:58.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T09:46:58.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T09:46:58.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T09:46:58.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T09:46:58.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T09:46:58.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T09:46:58.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T09:46:58.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:46:58.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:46:58.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:46:58.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:46:58.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:58.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:58.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:58.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:58.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:46:58.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:46:58.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:46:58.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-21T09:46:58.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:46:58.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:46:58.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:46:58.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:58.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:58.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:58.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/645294993' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-21T09:46:58.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:46:58.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/645294993' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T09:46:58.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:46:58.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:46:58.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:58.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:46:58.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:46:58.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:46:58.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:46:58.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:46:58.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:46:58.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi098:[v2:172.21.15.98:3301,v1:172.21.15.98:6790]=c;smithi196:172.21.15.196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:58.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: Saving service mon spec with placement smithi098:172.21.15.98=a;smithi098:[v2:172.21.15.98:3301,v1:172.21.15.98:6790]=c;smithi196:172.21.15.196=b;count:3 2024-09-21T09:46:58.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: Deploying daemon mon.b on smithi196 2024-09-21T09:46:58.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:46:58.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:46:58.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 ceph-mon[32670]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-21T09:46:58.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 bash[32639]: 4f1a8a66c2c869baf95034670b981005ccdf657396ee0d486e2e892e2b19bac8 2024-09-21T09:46:58.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:46:58 smithi196 systemd[1]: Started Ceph mon.b for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:47:00.058 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T09:47:00.059 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T09:47:00.312 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:47:03.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:47:03.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:03.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: mon.a calling monitor election 2024-09-21T09:47:03.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:03.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:03.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:03.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: mon.b calling monitor election 2024-09-21T09:47:03.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:03.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:03.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:03.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:03.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:03.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T09:47:03.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: monmap e2: 2 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:47:03.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: fsmap 2024-09-21T09:47:03.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:47:03.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: mgrmap e13: y(active, since 38s) 2024-09-21T09:47:03.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: overall HEALTH_OK 2024-09-21T09:47:03.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:03.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:03.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:03.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:47:03.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:03 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:47:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: mon.a calling monitor election 2024-09-21T09:47:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: mon.b calling monitor election 2024-09-21T09:47:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T09:47:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: monmap e2: 2 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:47:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: fsmap 2024-09-21T09:47:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:47:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: mgrmap e13: y(active, since 38s) 2024-09-21T09:47:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: overall HEALTH_OK 2024-09-21T09:47:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:47:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:03 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:04.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:04 smithi098 ceph-mon[29239]: Deploying daemon mon.c on smithi098 2024-09-21T09:47:04.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:04 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:04.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:04 smithi196 ceph-mon[32670]: Deploying daemon mon.c on smithi098 2024-09-21T09:47:04.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:04 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:04.998 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:47:04.998 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":2,"fsid":"26a15512-77fe-11ef-baf6-efdc52797490","modified":"2024-09-21T09:46:58.398132Z","created":"2024-09-21T09:45:30.814529Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T09:47:04.998 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 2 2024-09-21T09:47:05.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:05 smithi196 ceph-mon[32670]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:05.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:05 smithi196 ceph-mon[32670]: from='client.? 172.21.15.196:0/934987381' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:47:05.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:05 smithi098 ceph-mon[29239]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:05.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:05 smithi098 ceph-mon[29239]: from='client.? 172.21.15.196:0/934987381' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:47:06.555 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 podman[39934]: 2024-09-21 09:47:06.506737456 +0000 UTC m=+0.131700154 container create 54e9dc04184896a78fac2c92458f6a8f3f44f685c6e8183c8cf77dab2abf58a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2, RELEASE=quincy-fbd1a6a, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-21T09:47:06.595 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T09:47:06.595 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T09:47:06.806 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 podman[39934]: 2024-09-21 09:47:06.725866642 +0000 UTC m=+0.350829340 container init 54e9dc04184896a78fac2c92458f6a8f3f44f685c6e8183c8cf77dab2abf58a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-c, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-21T09:47:06.806 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 podman[39934]: 2024-09-21 09:47:06.729636364 +0000 UTC m=+0.354599065 container start 54e9dc04184896a78fac2c92458f6a8f3f44f685c6e8183c8cf77dab2abf58a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-c, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=quincy-fbd1a6a, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-21T09:47:06.806 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T09:47:06.806 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable), process ceph-mon, pid 2 2024-09-21T09:47:06.806 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: pidfile_write: ignore empty --pid-file 2024-09-21T09:47:06.806 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: load: jerasure load: lrc 2024-09-21T09:47:06.806 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: RocksDB version: 6.15.5 2024-09-21T09:47:06.806 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-21T09:47:06.806 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Compile date Sep 20 2024 2024-09-21T09:47:06.806 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: DB SUMMARY 2024-09-21T09:47:06.806 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: DB Session ID: E9PDL6Y2R0WJGXHNA3ID 2024-09-21T09:47:06.807 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: CURRENT file: CURRENT 2024-09-21T09:47:06.807 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T09:47:06.807 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-09-21T09:47:06.807 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-21T09:47:06.807 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-21T09:47:06.807 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.error_if_exists: 0 2024-09-21T09:47:06.807 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.create_if_missing: 0 2024-09-21T09:47:06.807 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.paranoid_checks: 1 2024-09-21T09:47:06.807 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T09:47:06.807 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.env: 0x55e8ac2d6440 2024-09-21T09:47:06.807 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.fs: Posix File System 2024-09-21T09:47:06.807 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.info_log: 0x55e8ae800ec0 2024-09-21T09:47:06.807 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T09:47:06.808 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.statistics: (nil) 2024-09-21T09:47:06.808 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.use_fsync: 0 2024-09-21T09:47:06.808 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_log_file_size: 0 2024-09-21T09:47:06.809 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T09:47:06.809 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T09:47:06.809 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T09:47:06.809 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T09:47:06.809 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.allow_fallocate: 1 2024-09-21T09:47:06.809 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T09:47:06.809 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T09:47:06.810 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.use_direct_reads: 0 2024-09-21T09:47:06.810 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T09:47:06.810 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T09:47:06.810 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.db_log_dir: 2024-09-21T09:47:06.810 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-09-21T09:47:06.810 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T09:47:06.810 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T09:47:06.810 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T09:47:06.810 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T09:47:06.810 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T09:47:06.810 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T09:47:06.810 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T09:47:06.810 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T09:47:06.811 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.write_buffer_manager: 0x55e8ae8be7b0 2024-09-21T09:47:06.811 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T09:47:06.811 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-21T09:47:06.811 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T09:47:06.811 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T09:47:06.811 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T09:47:06.811 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T09:47:06.811 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T09:47:06.811 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T09:47:06.811 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T09:47:06.811 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.unordered_write: 0 2024-09-21T09:47:06.811 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T09:47:06.811 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T09:47:06.811 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T09:47:06.812 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T09:47:06.812 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.row_cache: None 2024-09-21T09:47:06.812 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.wal_filter: None 2024-09-21T09:47:06.812 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T09:47:06.812 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T09:47:06.812 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.preserve_deletes: 0 2024-09-21T09:47:06.812 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.two_write_queues: 0 2024-09-21T09:47:06.812 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T09:47:06.812 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.atomic_flush: 0 2024-09-21T09:47:06.812 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T09:47:06.812 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T09:47:06.812 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T09:47:06.812 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.log_readahead_size: 0 2024-09-21T09:47:06.812 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T09:47:06.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T09:47:06.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T09:47:06.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T09:47:06.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T09:47:06.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T09:47:06.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_background_jobs: 2 2024-09-21T09:47:06.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_background_compactions: -1 2024-09-21T09:47:06.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_subcompactions: 1 2024-09-21T09:47:06.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T09:47:06.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T09:47:06.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T09:47:06.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T09:47:06.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T09:47:06.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T09:47:06.814 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T09:47:06.814 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T09:47:06.814 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_open_files: -1 2024-09-21T09:47:06.814 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T09:47:06.814 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T09:47:06.814 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T09:47:06.814 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T09:47:06.814 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_background_flushes: -1 2024-09-21T09:47:06.814 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Compression algorithms supported: 2024-09-21T09:47:06.814 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: kZSTD supported: 0 2024-09-21T09:47:06.814 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: kXpressCompression supported: 0 2024-09-21T09:47:06.814 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: kBZip2Compression supported: 0 2024-09-21T09:47:06.814 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T09:47:06.814 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: kLZ4Compression supported: 1 2024-09-21T09:47:06.815 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: kZlibCompression supported: 1 2024-09-21T09:47:06.815 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T09:47:06.815 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: kSnappyCompression supported: 1 2024-09-21T09:47:06.815 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T09:47:06.815 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-09-21T09:47:06.815 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-21T09:47:06.815 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T09:47:06.815 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.merge_operator: 2024-09-21T09:47:06.815 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.compaction_filter: None 2024-09-21T09:47:06.815 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T09:47:06.815 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T09:47:06.815 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T09:47:06.815 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T09:47:06.816 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e8ae7a7a38) 2024-09-21T09:47:06.816 INFO:journalctl@ceph.mon.c.smithi098.stdout: cache_index_and_filter_blocks: 1 2024-09-21T09:47:06.816 INFO:journalctl@ceph.mon.c.smithi098.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T09:47:06.816 INFO:journalctl@ceph.mon.c.smithi098.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T09:47:06.816 INFO:journalctl@ceph.mon.c.smithi098.stdout: pin_top_level_index_and_filter: 1 2024-09-21T09:47:06.816 INFO:journalctl@ceph.mon.c.smithi098.stdout: index_type: 0 2024-09-21T09:47:06.816 INFO:journalctl@ceph.mon.c.smithi098.stdout: data_block_index_type: 0 2024-09-21T09:47:06.816 INFO:journalctl@ceph.mon.c.smithi098.stdout: index_shortening: 1 2024-09-21T09:47:06.816 INFO:journalctl@ceph.mon.c.smithi098.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T09:47:06.816 INFO:journalctl@ceph.mon.c.smithi098.stdout: hash_index_allow_collision: 1 2024-09-21T09:47:06.816 INFO:journalctl@ceph.mon.c.smithi098.stdout: checksum: 1 2024-09-21T09:47:06.816 INFO:journalctl@ceph.mon.c.smithi098.stdout: no_block_cache: 0 2024-09-21T09:47:06.816 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_cache: 0x55e8ae7dd1f0 2024-09-21T09:47:06.817 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_cache_name: BinnedLRUCache 2024-09-21T09:47:06.817 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_cache_options: 2024-09-21T09:47:06.817 INFO:journalctl@ceph.mon.c.smithi098.stdout: capacity : 536870912 2024-09-21T09:47:06.817 INFO:journalctl@ceph.mon.c.smithi098.stdout: num_shard_bits : 4 2024-09-21T09:47:06.817 INFO:journalctl@ceph.mon.c.smithi098.stdout: strict_capacity_limit : 0 2024-09-21T09:47:06.817 INFO:journalctl@ceph.mon.c.smithi098.stdout: high_pri_pool_ratio: 0.000 2024-09-21T09:47:06.817 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_cache_compressed: (nil) 2024-09-21T09:47:06.817 INFO:journalctl@ceph.mon.c.smithi098.stdout: persistent_cache: (nil) 2024-09-21T09:47:06.817 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_size: 4096 2024-09-21T09:47:06.817 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_size_deviation: 10 2024-09-21T09:47:06.817 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_restart_interval: 16 2024-09-21T09:47:06.817 INFO:journalctl@ceph.mon.c.smithi098.stdout: index_block_restart_interval: 1 2024-09-21T09:47:06.817 INFO:journalctl@ceph.mon.c.smithi098.stdout: metadata_block_size: 4096 2024-09-21T09:47:06.817 INFO:journalctl@ceph.mon.c.smithi098.stdout: partition_filters: 0 2024-09-21T09:47:06.817 INFO:journalctl@ceph.mon.c.smithi098.stdout: use_delta_encoding: 1 2024-09-21T09:47:06.818 INFO:journalctl@ceph.mon.c.smithi098.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-21T09:47:06.818 INFO:journalctl@ceph.mon.c.smithi098.stdout: whole_key_filtering: 1 2024-09-21T09:47:06.818 INFO:journalctl@ceph.mon.c.smithi098.stdout: verify_compression: 0 2024-09-21T09:47:06.818 INFO:journalctl@ceph.mon.c.smithi098.stdout: read_amp_bytes_per_bit: 0 2024-09-21T09:47:06.818 INFO:journalctl@ceph.mon.c.smithi098.stdout: format_version: 4 2024-09-21T09:47:06.818 INFO:journalctl@ceph.mon.c.smithi098.stdout: enable_index_compression: 1 2024-09-21T09:47:06.818 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_align: 0 2024-09-21T09:47:06.818 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T09:47:06.818 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T09:47:06.818 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.compression: NoCompression 2024-09-21T09:47:06.818 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T09:47:06.818 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T09:47:06.818 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T09:47:06.819 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.num_levels: 7 2024-09-21T09:47:06.819 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T09:47:06.819 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T09:47:06.819 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T09:47:06.819 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T09:47:06.819 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T09:47:06.819 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T09:47:06.819 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T09:47:06.819 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:47:06.819 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T09:47:06.819 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T09:47:06.819 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T09:47:06.819 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T09:47:06.819 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T09:47:06.820 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T09:47:06.820 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:47:06.820 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T09:47:06.820 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T09:47:06.820 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T09:47:06.820 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T09:47:06.820 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T09:47:06.820 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T09:47:06.820 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T09:47:06.820 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T09:47:06.820 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T09:47:06.820 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T09:47:06.820 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T09:47:06.820 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T09:47:06.821 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T09:47:06.821 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T09:47:06.821 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T09:47:06.821 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T09:47:06.821 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T09:47:06.821 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T09:47:06.821 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T09:47:06.821 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.arena_block_size: 4194304 2024-09-21T09:47:06.821 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T09:47:06.821 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T09:47:06.821 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-21T09:47:06.821 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T09:47:06.821 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T09:47:06.822 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T09:47:06.822 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T09:47:06.822 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T09:47:06.822 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T09:47:06.822 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T09:47:06.822 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T09:47:06.822 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T09:47:06.822 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T09:47:06.822 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T09:47:06.822 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T09:47:06.822 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.inplace_update_support: 0 2024-09-21T09:47:06.822 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T09:47:06.822 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T09:47:06.822 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T09:47:06.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T09:47:06.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.bloom_locality: 0 2024-09-21T09:47:06.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.max_successive_merges: 0 2024-09-21T09:47:06.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T09:47:06.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T09:47:06.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T09:47:06.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T09:47:06.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.ttl: 2592000 2024-09-21T09:47:06.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T09:47:06.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.enable_blob_files: false 2024-09-21T09:47:06.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.min_blob_size: 0 2024-09-21T09:47:06.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T09:47:06.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T09:47:06.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T09:47:06.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T09:47:06.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T09:47:06.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-09-21T09:47:06.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-09-21T09:47:06.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912026753407, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-21T09:47:06.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-09-21T09:47:06.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-21T09:47:06.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912026753702, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1696, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726912026, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "1beb1b77-dbd2-426b-9c50-5de474484cfb", "db_session_id": "E9PDL6Y2R0WJGXHNA3ID"}} 2024-09-21T09:47:06.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-09-21T09:47:06.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912026754030, "job": 1, "event": "recovery_finished"} 2024-09-21T09:47:06.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T09:47:06.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55e8ae810700 2024-09-21T09:47:06.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: DB pointer 0x55e8ae7e0000 2024-09-21T09:47:06.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-21T09:47:06.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-21T09:47:06.825 INFO:journalctl@ceph.mon.c.smithi098.stdout: ** DB Stats ** 2024-09-21T09:47:06.825 INFO:journalctl@ceph.mon.c.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:47:06.825 INFO:journalctl@ceph.mon.c.smithi098.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T09:47:06.825 INFO:journalctl@ceph.mon.c.smithi098.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:47:06.825 INFO:journalctl@ceph.mon.c.smithi098.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:47:06.825 INFO:journalctl@ceph.mon.c.smithi098.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T09:47:06.825 INFO:journalctl@ceph.mon.c.smithi098.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-21T09:47:06.825 INFO:journalctl@ceph.mon.c.smithi098.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:47:06.825 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-09-21T09:47:06.825 INFO:journalctl@ceph.mon.c.smithi098.stdout: ** Compaction Stats [default] ** 2024-09-21T09:47:06.825 INFO:journalctl@ceph.mon.c.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:47:06.826 INFO:journalctl@ceph.mon.c.smithi098.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:47:06.826 INFO:journalctl@ceph.mon.c.smithi098.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.8 0.00 0.00 1 0.000 0 0 2024-09-21T09:47:06.826 INFO:journalctl@ceph.mon.c.smithi098.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.8 0.00 0.00 1 0.000 0 0 2024-09-21T09:47:06.826 INFO:journalctl@ceph.mon.c.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.8 0.00 0.00 1 0.000 0 0 2024-09-21T09:47:06.826 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-09-21T09:47:06.826 INFO:journalctl@ceph.mon.c.smithi098.stdout: ** Compaction Stats [default] ** 2024-09-21T09:47:06.826 INFO:journalctl@ceph.mon.c.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:47:06.826 INFO:journalctl@ceph.mon.c.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:47:06.826 INFO:journalctl@ceph.mon.c.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.8 0.00 0.00 1 0.000 0 0 2024-09-21T09:47:06.826 INFO:journalctl@ceph.mon.c.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:47:06.826 INFO:journalctl@ceph.mon.c.smithi098.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T09:47:06.826 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:47:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:47:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:47:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:47:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:47:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:47:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:47:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-09-21T09:47:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:47:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-09-21T09:47:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout: ** Compaction Stats [default] ** 2024-09-21T09:47:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:47:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:47:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.8 0.00 0.00 1 0.000 0 0 2024-09-21T09:47:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.8 0.00 0.00 1 0.000 0 0 2024-09-21T09:47:06.828 INFO:journalctl@ceph.mon.c.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-21T09:47:06.828 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-09-21T09:47:06.828 INFO:journalctl@ceph.mon.c.smithi098.stdout: ** Compaction Stats [default] ** 2024-09-21T09:47:06.828 INFO:journalctl@ceph.mon.c.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:47:06.828 INFO:journalctl@ceph.mon.c.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:47:06.828 INFO:journalctl@ceph.mon.c.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.8 0.00 0.00 1 0.000 0 0 2024-09-21T09:47:06.828 INFO:journalctl@ceph.mon.c.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:47:06.828 INFO:journalctl@ceph.mon.c.smithi098.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T09:47:06.828 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:47:06.828 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:47:06.828 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:47:06.828 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:47:06.828 INFO:journalctl@ceph.mon.c.smithi098.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:47:06.828 INFO:journalctl@ceph.mon.c.smithi098.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:47:06.829 INFO:journalctl@ceph.mon.c.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:47:06.829 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-09-21T09:47:06.829 INFO:journalctl@ceph.mon.c.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:47:06.829 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-21T09:47:06.829 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: using public_addrv [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-21T09:47:06.829 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: starting mon.c rank -1 at public addrs [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] at bind addrs [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:47:06.829 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: mon.c@-1(???) e0 preinit fsid 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:47:06.829 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: mon.c@-1(synchronizing).mds e1 new map 2024-09-21T09:47:06.829 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-21T09:47:06.829 INFO:journalctl@ceph.mon.c.smithi098.stdout: e1 2024-09-21T09:47:06.829 INFO:journalctl@ceph.mon.c.smithi098.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T09:47:06.829 INFO:journalctl@ceph.mon.c.smithi098.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:47:06.829 INFO:journalctl@ceph.mon.c.smithi098.stdout: legacy client fscid: -1 2024-09-21T09:47:06.829 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-09-21T09:47:06.830 INFO:journalctl@ceph.mon.c.smithi098.stdout: No filesystems configured 2024-09-21T09:47:06.830 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T09:47:06.830 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T09:47:06.830 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T09:47:06.830 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T09:47:06.830 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T09:47:06.830 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T09:47:06.830 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T09:47:06.830 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:47:06.830 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:47:06.830 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:47:06.830 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:06.830 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:47:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-21T09:47:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:47:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:47:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:47:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/645294993' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-21T09:47:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:06.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/645294993' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T09:47:06.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:47:06.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:06.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:06.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:47:06.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:06.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:06.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:06.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:47:06.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:06.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi098:[v2:172.21.15.98:3301,v1:172.21.15.98:6790]=c;smithi196:172.21.15.196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:47:06.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: Saving service mon spec with placement smithi098:172.21.15.98=a;smithi098:[v2:172.21.15.98:3301,v1:172.21.15.98:6790]=c;smithi196:172.21.15.196=b;count:3 2024-09-21T09:47:06.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: Deploying daemon mon.b on smithi196 2024-09-21T09:47:06.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:06.833 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:06.833 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:47:06.833 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:06.833 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: mon.a calling monitor election 2024-09-21T09:47:06.833 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:06.833 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:06.833 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:06.833 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: mon.b calling monitor election 2024-09-21T09:47:06.833 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:06.833 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:06.833 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:06.833 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:06.833 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:06.833 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T09:47:06.833 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: monmap e2: 2 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:47:06.833 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: fsmap 2024-09-21T09:47:06.834 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:47:06.834 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: mgrmap e13: y(active, since 38s) 2024-09-21T09:47:06.834 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: overall HEALTH_OK 2024-09-21T09:47:06.834 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:06.834 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:06.834 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:06.834 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:47:06.834 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:06.834 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: Deploying daemon mon.c on smithi098 2024-09-21T09:47:06.834 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:06.834 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:06.834 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: from='client.? 172.21.15.196:0/934987381' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:47:06.834 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 ceph-mon[39960]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-21T09:47:06.834 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 bash[39934]: 54e9dc04184896a78fac2c92458f6a8f3f44f685c6e8183c8cf77dab2abf58a7 2024-09-21T09:47:06.834 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:06 smithi098 systemd[1]: Started Ceph mon.c for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:47:06.845 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:47:12.092 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:47:12.092 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":3,"fsid":"26a15512-77fe-11ef-baf6-efdc52797490","modified":"2024-09-21T09:47:06.778233Z","created":"2024-09-21T09:45:30.814529Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-21T09:47:12.092 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 3 2024-09-21T09:47:12.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:12.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:47:12.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: mon.a calling monitor election 2024-09-21T09:47:12.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: mon.b calling monitor election 2024-09-21T09:47:12.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:12.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:47:12.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:47:12.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:12.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:47:12.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: mon.c calling monitor election 2024-09-21T09:47:12.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:47:12.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:12.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:47:12.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:47:12.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:47:12.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0],c=[v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:47:12.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: fsmap 2024-09-21T09:47:12.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:47:12.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: mgrmap e13: y(active, since 47s) 2024-09-21T09:47:12.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: overall HEALTH_OK 2024-09-21T09:47:12.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:12.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:12.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:12.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:12.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:12.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:47:12.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: mon.a calling monitor election 2024-09-21T09:47:12.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: mon.b calling monitor election 2024-09-21T09:47:12.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:12.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:47:12.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:47:12.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:12.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:47:12.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: mon.c calling monitor election 2024-09-21T09:47:12.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:47:12.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:12.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:47:12.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:47:12.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:47:12.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0],c=[v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:47:12.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: fsmap 2024-09-21T09:47:12.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:47:12.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: mgrmap e13: y(active, since 47s) 2024-09-21T09:47:12.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: overall HEALTH_OK 2024-09-21T09:47:12.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:12.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:12.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:12.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:12.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:12.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:47:12.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: mon.a calling monitor election 2024-09-21T09:47:12.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: mon.b calling monitor election 2024-09-21T09:47:12.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:47:12.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:47:12.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:47:12.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:12.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:47:12.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: mon.c calling monitor election 2024-09-21T09:47:12.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:47:12.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:12.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:47:12.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:47:12.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:47:12.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0],c=[v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:47:12.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: fsmap 2024-09-21T09:47:12.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:47:12.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: mgrmap e13: y(active, since 47s) 2024-09-21T09:47:12.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: overall HEALTH_OK 2024-09-21T09:47:12.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:12.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:12.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:12.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:12.689 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-21T09:47:12.689 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph config generate-minimal-conf 2024-09-21T09:47:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:47:13.056 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:12 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:47:13.057 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:12 smithi098 ceph-mon[39960]: from='client.? 172.21.15.196:0/1914528516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:47:13.057 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:12 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:47:13.057 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:12 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:47:13.057 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:12 smithi098 ceph-mon[29239]: from='client.? 172.21.15.196:0/1914528516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:47:13.057 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:12 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:47:13.076 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:12 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:47:13.076 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:12 smithi196 ceph-mon[32670]: from='client.? 172.21.15.196:0/1914528516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:47:13.076 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:12 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:47:14.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:13 smithi098 ceph-mon[29239]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:14.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:13 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:14.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:13 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:14.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:13 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:14.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:13 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:14.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:13 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:14.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:13 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:14.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:13 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:14.088 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:13 smithi098 ceph-mon[39960]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:14.088 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:13 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:14.088 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:13 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:14.088 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:13 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:14.088 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:13 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:14.088 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:13 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:14.088 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:13 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:14.088 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:13 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:14.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:13 smithi196 ceph-mon[32670]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:14.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:13 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:14.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:13 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:13 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:13 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:13 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:13 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:13 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:14.738 INFO:teuthology.orchestra.run.smithi098.stdout:# minimal ceph.conf for 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:47:14.738 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2024-09-21T09:47:14.738 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:47:14.738 INFO:teuthology.orchestra.run.smithi098.stdout: mon_host = [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-21T09:47:15.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:14 smithi196 ceph-mon[32670]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-21T09:47:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:14 smithi196 ceph-mon[32670]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-21T09:47:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:14 smithi196 ceph-mon[32670]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:47:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:14 smithi196 ceph-mon[32670]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:47:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:14 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:14 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:14 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:14 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:14 smithi196 ceph-mon[32670]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T09:47:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:14 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:47:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:14 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:47:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:14 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:14 smithi196 ceph-mon[32670]: Reconfiguring daemon mon.a on smithi098 2024-09-21T09:47:15.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:14 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:15.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:14 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/1167424151' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:15.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[29239]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-21T09:47:15.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[29239]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-21T09:47:15.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[29239]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:47:15.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[29239]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:47:15.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:15.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:15.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:15.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:15.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[29239]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T09:47:15.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:47:15.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:47:15.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:15.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[29239]: Reconfiguring daemon mon.a on smithi098 2024-09-21T09:47:15.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:15.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/1167424151' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:15.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[39960]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-21T09:47:15.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[39960]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-21T09:47:15.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[39960]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:47:15.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[39960]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:47:15.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:15.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:15.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:15.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:15.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[39960]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T09:47:15.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:47:15.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:47:15.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:15.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[39960]: Reconfiguring daemon mon.a on smithi098 2024-09-21T09:47:15.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:15.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:14 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/1167424151' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:15.775 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-21T09:47:15.775 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-21T09:47:15.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T09:47:15.803 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-21T09:47:15.803 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:47:15.869 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-21T09:47:15.869 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T09:47:15.902 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-21T09:47:15.902 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:47:15.972 INFO:tasks.cephadm:Adding mgr.y on smithi098 2024-09-21T09:47:15.972 INFO:tasks.cephadm:Adding mgr.x on smithi196 2024-09-21T09:47:15.972 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph orch apply mgr '2;smithi098=y;smithi196=x' 2024-09-21T09:47:16.246 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:15 smithi098 ceph-mon[29239]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:16.246 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:15 smithi098 ceph-mon[39960]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:16.247 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:47:16.367 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:15 smithi196 ceph-mon[32670]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:17.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:17 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:17.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:17 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:17.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:17 smithi196 ceph-mon[32670]: Reconfiguring mon.c (monmap changed)... 2024-09-21T09:47:17.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:17 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:47:17.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:17 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:47:17.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:17 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:17.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:17 smithi196 ceph-mon[32670]: Reconfiguring daemon mon.c on smithi098 2024-09-21T09:47:17.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:17 smithi196 ceph-mon[32670]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:17.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:17 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:17.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:17 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:17.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:17 smithi098 ceph-mon[39960]: Reconfiguring mon.c (monmap changed)... 2024-09-21T09:47:17.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:17 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:47:17.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:17 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:47:17.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:17 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:17.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:17 smithi098 ceph-mon[39960]: Reconfiguring daemon mon.c on smithi098 2024-09-21T09:47:17.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:17 smithi098 ceph-mon[39960]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:17.579 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:17 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:17.579 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:17 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:17.579 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:17 smithi098 ceph-mon[29239]: Reconfiguring mon.c (monmap changed)... 2024-09-21T09:47:17.579 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:17 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:47:17.579 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:17 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:47:17.579 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:17 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:17.579 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:17 smithi098 ceph-mon[29239]: Reconfiguring daemon mon.c on smithi098 2024-09-21T09:47:17.579 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:17 smithi098 ceph-mon[29239]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:17.934 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled mgr update... 2024-09-21T09:47:18.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:18 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:18.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:18 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:18.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:18 smithi196 ceph-mon[32670]: Reconfiguring mon.b (monmap changed)... 2024-09-21T09:47:18.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:18 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:47:18.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:18 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:47:18.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:18 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:18.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:18 smithi196 ceph-mon[32670]: Reconfiguring daemon mon.b on smithi196 2024-09-21T09:47:18.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:18 smithi196 ceph-mon[32670]: from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi098=y;smithi196=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:47:18.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:18 smithi196 ceph-mon[32670]: Saving service mgr spec with placement smithi098=y;smithi196=x;count:2 2024-09-21T09:47:18.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:18 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:19.048 DEBUG:teuthology.orchestra.run.smithi196:mgr.x> sudo journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@mgr.x.service 2024-09-21T09:47:19.051 DEBUG:tasks.cephadm:set 0 configs 2024-09-21T09:47:19.051 INFO:tasks.cephadm:Deploying OSDs... 2024-09-21T09:47:19.051 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-21T09:47:19.051 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T09:47:19.071 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T09:47:19.071 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-09-21T09:47:19.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:18 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:19.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:18 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:19.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:18 smithi098 ceph-mon[29239]: Reconfiguring mon.b (monmap changed)... 2024-09-21T09:47:19.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:18 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:47:19.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:18 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:47:19.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:18 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:19.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:18 smithi098 ceph-mon[29239]: Reconfiguring daemon mon.b on smithi196 2024-09-21T09:47:19.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:18 smithi098 ceph-mon[29239]: from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi098=y;smithi196=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:47:19.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:18 smithi098 ceph-mon[29239]: Saving service mgr spec with placement smithi098=y;smithi196=x;count:2 2024-09-21T09:47:19.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:18 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:19.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:18 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:19.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:18 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:19.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:18 smithi098 ceph-mon[39960]: Reconfiguring mon.b (monmap changed)... 2024-09-21T09:47:19.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:18 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:47:19.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:18 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:47:19.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:18 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:19.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:18 smithi098 ceph-mon[39960]: Reconfiguring daemon mon.b on smithi196 2024-09-21T09:47:19.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:18 smithi098 ceph-mon[39960]: from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi098=y;smithi196=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:47:19.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:18 smithi098 ceph-mon[39960]: Saving service mgr spec with placement smithi098=y;smithi196=x;count:2 2024-09-21T09:47:19.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:18 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:19.116 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T09:47:19.116 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:47:19.117 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 969 Links: 1 2024-09-21T09:47:19.117 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:47:19.117 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:47:19.117 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-09-21 09:47:06.211946084 +0000 2024-09-21T09:47:19.117 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-09-21 09:42:47.719528122 +0000 2024-09-21T09:47:19.117 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-09-21 09:42:47.719528122 +0000 2024-09-21T09:47:19.117 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-09-21 09:42:47.719528122 +0000 2024-09-21T09:47:19.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T09:47:19.185 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-09-21T09:47:19.185 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-09-21T09:47:19.185 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.00023791 s, 2.2 MB/s 2024-09-21T09:47:19.186 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T09:47:19.243 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-09-21T09:47:19.300 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T09:47:19.300 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:47:19.300 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 979 Links: 1 2024-09-21T09:47:19.300 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:47:19.300 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:47:19.300 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-09-21 09:47:06.212946070 +0000 2024-09-21T09:47:19.300 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-09-21 09:42:47.719528122 +0000 2024-09-21T09:47:19.300 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-09-21 09:42:47.719528122 +0000 2024-09-21T09:47:19.301 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-09-21 09:42:47.719528122 +0000 2024-09-21T09:47:19.301 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T09:47:19.364 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-09-21T09:47:19.365 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-09-21T09:47:19.365 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000209115 s, 2.4 MB/s 2024-09-21T09:47:19.366 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T09:47:19.423 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-09-21T09:47:19.482 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T09:47:19.482 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:47:19.482 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 980 Links: 1 2024-09-21T09:47:19.483 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:47:19.483 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:47:19.483 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-09-21 09:47:06.212946070 +0000 2024-09-21T09:47:19.483 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-09-21 09:42:47.719528122 +0000 2024-09-21T09:47:19.483 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-09-21 09:42:47.719528122 +0000 2024-09-21T09:47:19.483 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-09-21 09:42:47.719528122 +0000 2024-09-21T09:47:19.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T09:47:19.556 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-09-21T09:47:19.556 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-09-21T09:47:19.556 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000247814 s, 2.1 MB/s 2024-09-21T09:47:19.558 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T09:47:19.616 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-09-21T09:47:19.677 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T09:47:19.678 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:47:19.678 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 970 Links: 1 2024-09-21T09:47:19.678 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:47:19.678 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:47:19.678 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-09-21 09:47:06.212946070 +0000 2024-09-21T09:47:19.678 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-09-21 09:42:47.719528122 +0000 2024-09-21T09:47:19.679 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-09-21 09:42:47.719528122 +0000 2024-09-21T09:47:19.679 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-09-21 09:42:47.719528122 +0000 2024-09-21T09:47:19.679 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T09:47:19.745 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-09-21T09:47:19.745 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-09-21T09:47:19.745 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000255579 s, 2.0 MB/s 2024-09-21T09:47:19.746 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T09:47:19.807 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-21T09:47:19.807 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T09:47:19.833 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T09:47:19.833 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_1 2024-09-21T09:47:19.894 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T09:47:19.894 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:47:19.895 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 966 Links: 1 2024-09-21T09:47:19.895 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:47:19.895 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:47:19.895 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-09-21 09:46:57.307195993 +0000 2024-09-21T09:47:19.895 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-09-21 09:42:47.685731873 +0000 2024-09-21T09:47:19.895 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-09-21 09:42:47.685731873 +0000 2024-09-21T09:47:19.895 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-09-21 09:42:47.685731873 +0000 2024-09-21T09:47:19.895 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T09:47:19.961 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-09-21T09:47:19.961 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-09-21T09:47:19.961 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000241654 s, 2.1 MB/s 2024-09-21T09:47:19.962 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T09:47:20.022 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_2 2024-09-21T09:47:20.082 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T09:47:20.082 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:47:20.083 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 974 Links: 1 2024-09-21T09:47:20.083 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:47:20.083 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:47:20.083 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-09-21 09:46:57.308195971 +0000 2024-09-21T09:47:20.083 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-09-21 09:42:47.686731851 +0000 2024-09-21T09:47:20.083 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-09-21 09:42:47.686731851 +0000 2024-09-21T09:47:20.084 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-09-21 09:42:47.686731851 +0000 2024-09-21T09:47:20.084 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T09:47:20.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:19 smithi196 ceph-mon[32670]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:20.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:19 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:20.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:19 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:19 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:47:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:19 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:19 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:19 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:19 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:47:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:19 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T09:47:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:19 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:47:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:19 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:20.148 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-09-21T09:47:20.149 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-09-21T09:47:20.149 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000201525 s, 2.5 MB/s 2024-09-21T09:47:20.150 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T09:47:20.206 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_3 2024-09-21T09:47:20.261 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T09:47:20.262 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:47:20.262 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 982 Links: 1 2024-09-21T09:47:20.262 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:47:20.262 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:47:20.262 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-09-21 09:46:57.308195971 +0000 2024-09-21T09:47:20.262 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-09-21 09:42:47.688731807 +0000 2024-09-21T09:47:20.262 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-09-21 09:42:47.688731807 +0000 2024-09-21T09:47:20.262 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-09-21 09:42:47.688731807 +0000 2024-09-21T09:47:20.262 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T09:47:20.324 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-09-21T09:47:20.325 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-09-21T09:47:20.325 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000207603 s, 2.5 MB/s 2024-09-21T09:47:20.326 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T09:47:20.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:19 smithi098 ceph-mon[29239]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:20.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:19 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:20.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:19 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:20.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:19 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:47:20.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:19 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:20.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:19 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:20.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:19 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:20.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:19 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:47:20.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:19 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T09:47:20.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:19 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:47:20.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:19 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:20.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:19 smithi098 ceph-mon[39960]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:20.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:19 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:20.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:19 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:20.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:19 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:47:20.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:19 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:20.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:19 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:20.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:19 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:20.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:19 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:47:20.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:19 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T09:47:20.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:19 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:47:20.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:19 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:20.382 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_4 2024-09-21T09:47:20.436 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T09:47:20.436 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:47:20.436 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 975 Links: 1 2024-09-21T09:47:20.437 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:47:20.437 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:47:20.437 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-09-21 09:46:57.308195971 +0000 2024-09-21T09:47:20.437 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-09-21 09:42:47.686731851 +0000 2024-09-21T09:47:20.437 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-09-21 09:42:47.686731851 +0000 2024-09-21T09:47:20.437 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-09-21 09:42:47.686731851 +0000 2024-09-21T09:47:20.437 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T09:47:20.500 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-09-21T09:47:20.500 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-09-21T09:47:20.500 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000192305 s, 2.7 MB/s 2024-09-21T09:47:20.501 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T09:47:20.558 INFO:tasks.cephadm:Deploying osd.0 on smithi098 with /dev/vg_nvme/lv_4... 2024-09-21T09:47:20.558 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T09:47:20.795 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:47:21.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:20 smithi196 ceph-mon[32670]: Deploying daemon mgr.x on smithi196 2024-09-21T09:47:21.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:20 smithi098 ceph-mon[29239]: Deploying daemon mgr.x on smithi196 2024-09-21T09:47:21.248 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:20 smithi098 ceph-mon[39960]: Deploying daemon mgr.x on smithi196 2024-09-21T09:47:21.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:47:21 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:47:21.823+0000 7f2aeb25dec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:47:22.137 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:21 smithi098 ceph-mon[29239]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:22.137 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:21 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:22.137 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:21 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:22.137 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:21 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:22.137 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:21 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:22.138 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:21 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:47:22.138 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:21 smithi098 ceph-mon[39960]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:22.138 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:21 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:22.138 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:21 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:22.138 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:21 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:22.138 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:21 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:22.138 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:21 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:47:22.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:21 smithi196 ceph-mon[32670]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:22.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:21 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:22.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:21 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:22.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:21 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:22.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:21 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:22.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:21 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:47:22.381 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:47:21 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:47:21.955+0000 7f2aeb25dec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:47:22.381 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:47:22 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:47:22.102+0000 7f2aeb25dec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:47:22.638 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:47:22 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:47:22.379+0000 7f2aeb25dec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:47:22.638 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:47:22 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:47:22.442+0000 7f2aeb25dec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:47:22.638 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:47:22 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:47:22.609+0000 7f2aeb25dec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:47:23.164 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:47:23.184 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi098:vg_nvme/lv_4 2024-09-21T09:47:23.837 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:47:23 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:47:23.467+0000 7f2aeb25dec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:47:23.837 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:47:23 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:47:23.531+0000 7f2aeb25dec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:47:24.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:23 smithi098 ceph-mon[29239]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:24.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:23 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:24.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:23 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:24.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:23 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:24.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:23 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:24.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:23 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:24.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:23 smithi098 ceph-mon[29239]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T09:47:24.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:23 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:47:24.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:23 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:47:24.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:23 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:24.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:23 smithi098 ceph-mon[29239]: Reconfiguring daemon mgr.y on smithi098 2024-09-21T09:47:24.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:23 smithi098 ceph-mon[39960]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:24.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:23 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:24.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:23 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:24.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:23 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:24.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:23 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:24.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:23 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:24.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:23 smithi098 ceph-mon[39960]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T09:47:24.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:23 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:47:24.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:23 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:47:24.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:23 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:24.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:23 smithi098 ceph-mon[39960]: Reconfiguring daemon mgr.y on smithi098 2024-09-21T09:47:24.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:23 smithi196 ceph-mon[32670]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:24.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:23 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:24.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:23 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:24.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:23 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:24.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:23 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:24.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:23 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:24.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:23 smithi196 ceph-mon[32670]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T09:47:24.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:23 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:47:24.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:23 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:47:24.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:23 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:24.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:23 smithi196 ceph-mon[32670]: Reconfiguring daemon mgr.y on smithi098 2024-09-21T09:47:24.242 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:47:24.539 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:47:24 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:47:24.244+0000 7f2aeb25dec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:47:24.539 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:47:24 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:47:24.359+0000 7f2aeb25dec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:47:24.539 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:47:24 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:47:24.479+0000 7f2aeb25dec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:47:24.539 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:47:24 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:47:24.537+0000 7f2aeb25dec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:47:24.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:47:24 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:47:24.664+0000 7f2aeb25dec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:47:24.928 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:47:24 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:47:24.729+0000 7f2aeb25dec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:47:24.928 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:47:24 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:47:24.786+0000 7f2aeb25dec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:47:25.182 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:47:24 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:47:24.918+0000 7f2aeb25dec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:47:25.182 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:47:24 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:47:24.980+0000 7f2aeb25dec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:47:25.183 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:47:25 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:47:25.044+0000 7f2aeb25dec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:47:25.183 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:47:25 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:47:25.124+0000 7f2aeb25dec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:47:25.183 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:47:25 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:47:25.180+0000 7f2aeb25dec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:47:25.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:25 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:25.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:25 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:25.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:25 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:47:25.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:25 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:25.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:25 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:25.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:25 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:25.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:25 smithi098 ceph-mon[29239]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:25.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:25 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:25.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:25 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:25.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:25 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:25.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:25 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:47:25.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:25 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:25.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:25 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:25.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:25 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:25.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:25 smithi098 ceph-mon[39960]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:25.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:25 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:25.638 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:47:25 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:47:25.237+0000 7f2aeb25dec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:47:25.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:25 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:25.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:25 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:25 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:47:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:25 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:25 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:25 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:25 smithi196 ceph-mon[32670]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:25 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:26.270 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:47:25 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:47:25.980+0000 7f2aeb25dec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:47:26.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:26 smithi098 ceph-mon[39960]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:47:26.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:26 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:47:26.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:26 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:47:26.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:26 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:26.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:26 smithi098 ceph-mon[39960]: Standby manager daemon x started 2024-09-21T09:47:26.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:26 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/2575898668' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:47:26.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:26 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/2575898668' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:47:26.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:26 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/2575898668' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:47:26.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:26 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/2575898668' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:47:26.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:26 smithi098 ceph-mon[29239]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:47:26.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:26 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:47:26.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:26 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:47:26.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:26 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:26.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:26 smithi098 ceph-mon[29239]: Standby manager daemon x started 2024-09-21T09:47:26.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:26 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.196:0/2575898668' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:47:26.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:26 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.196:0/2575898668' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:47:26.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:26 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.196:0/2575898668' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:47:26.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:26 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.196:0/2575898668' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:47:26.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:26 smithi196 ceph-mon[32670]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:47:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:26 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:47:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:26 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:47:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:26 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:26 smithi196 ceph-mon[32670]: Standby manager daemon x started 2024-09-21T09:47:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:26 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/2575898668' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:47:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:26 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/2575898668' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:47:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:26 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/2575898668' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:47:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:26 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/2575898668' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:47:27.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:27 smithi098 ceph-mon[29239]: mgrmap e14: y(active, since 61s), standbys: x 2024-09-21T09:47:27.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:27 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:47:27.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:27 smithi098 ceph-mon[29239]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:27.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:27 smithi098 ceph-mon[39960]: mgrmap e14: y(active, since 61s), standbys: x 2024-09-21T09:47:27.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:27 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:47:27.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:27 smithi098 ceph-mon[39960]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:27.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:27 smithi196 ceph-mon[32670]: mgrmap e14: y(active, since 61s), standbys: x 2024-09-21T09:47:27.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:27 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:47:27.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:27 smithi196 ceph-mon[32670]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:28.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:28 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2221084851' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "579f13de-565d-4801-b65e-f27d1cde297a"}]: dispatch 2024-09-21T09:47:28.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:28 smithi098 ceph-mon[29239]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "579f13de-565d-4801-b65e-f27d1cde297a"}]: dispatch 2024-09-21T09:47:28.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:28 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2221084851' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "579f13de-565d-4801-b65e-f27d1cde297a"}]: dispatch 2024-09-21T09:47:28.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:28 smithi098 ceph-mon[39960]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "579f13de-565d-4801-b65e-f27d1cde297a"}]: dispatch 2024-09-21T09:47:28.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:28 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2221084851' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "579f13de-565d-4801-b65e-f27d1cde297a"}]: dispatch 2024-09-21T09:47:28.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:28 smithi196 ceph-mon[32670]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "579f13de-565d-4801-b65e-f27d1cde297a"}]: dispatch 2024-09-21T09:47:29.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:29 smithi098 ceph-mon[39960]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "579f13de-565d-4801-b65e-f27d1cde297a"}]': finished 2024-09-21T09:47:29.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:29 smithi098 ceph-mon[39960]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T09:47:29.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:29 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:47:29.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:29 smithi098 ceph-mon[39960]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:29.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:29 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/1809049843' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:47:29.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:29 smithi098 ceph-mon[29239]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "579f13de-565d-4801-b65e-f27d1cde297a"}]': finished 2024-09-21T09:47:29.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:29 smithi098 ceph-mon[29239]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T09:47:29.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:29 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:47:29.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:29 smithi098 ceph-mon[29239]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:29.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:29 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/1809049843' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:47:29.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:29 smithi196 ceph-mon[32670]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "579f13de-565d-4801-b65e-f27d1cde297a"}]': finished 2024-09-21T09:47:29.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:29 smithi196 ceph-mon[32670]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T09:47:29.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:29 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:47:29.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:29 smithi196 ceph-mon[32670]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:29.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:29 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/1809049843' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:47:31.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:31 smithi098 ceph-mon[29239]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:31.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:31 smithi098 ceph-mon[39960]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:31.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:31 smithi196 ceph-mon[32670]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:33.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:33 smithi098 ceph-mon[39960]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:33.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:33 smithi098 ceph-mon[29239]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:33.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:33 smithi196 ceph-mon[32670]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:34.822 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:34 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-21T09:47:34.822 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:34 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:34.822 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:34 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-21T09:47:34.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:34 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:34 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-21T09:47:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:34 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:35.645 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:35 smithi098 ceph-mon[29239]: Deploying daemon osd.0 on smithi098 2024-09-21T09:47:35.645 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:35 smithi098 ceph-mon[29239]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:35.646 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:35 smithi098 ceph-mon[39960]: Deploying daemon osd.0 on smithi098 2024-09-21T09:47:35.646 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:35 smithi098 ceph-mon[39960]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:35.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:35 smithi196 ceph-mon[32670]: Deploying daemon osd.0 on smithi098 2024-09-21T09:47:35.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:35 smithi196 ceph-mon[32670]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:37.657 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:37 smithi098 ceph-mon[29239]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:37.657 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:37 smithi098 ceph-mon[39960]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:37.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:37 smithi196 ceph-mon[32670]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:39.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:39 smithi098 ceph-mon[39960]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:39.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:39 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:47:39.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:39 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:39.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:39 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:39.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:39 smithi098 ceph-mon[29239]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:39.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:39 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:47:39.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:39 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:39.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:39 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:39 smithi196 ceph-mon[32670]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:39 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:47:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:39 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:39 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:41.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:41 smithi098 ceph-mon[29239]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:41.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:41 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:41.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:41 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:41.752 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:41 smithi098 ceph-mon[39960]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:41.752 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:41 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:41.752 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:41 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:41.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:41 smithi196 ceph-mon[32670]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:41.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:41 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:41.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:41 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:42.372 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 0 on host 'smithi098' 2024-09-21T09:47:42.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:42 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:42.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:42 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:42.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:42 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:42.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:42 smithi098 ceph-mon[29239]: from='osd.0 [v2:172.21.15.98:6802/506591636,v1:172.21.15.98:6803/506591636]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T09:47:42.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:42 smithi098 ceph-mon[29239]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T09:47:42.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:42 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:47:42.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:42 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:42.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:42 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:42.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:42 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:42.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:42 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:42.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:42 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:42.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:42 smithi098 ceph-mon[39960]: from='osd.0 [v2:172.21.15.98:6802/506591636,v1:172.21.15.98:6803/506591636]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T09:47:42.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:42 smithi098 ceph-mon[39960]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T09:47:42.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:42 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:47:42.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:42 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:42.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:42 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:42.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:42 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:42.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:42 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:42 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:42 smithi196 ceph-mon[32670]: from='osd.0 [v2:172.21.15.98:6802/506591636,v1:172.21.15.98:6803/506591636]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T09:47:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:42 smithi196 ceph-mon[32670]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T09:47:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:42 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:47:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:42 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:42 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:43.071 DEBUG:teuthology.orchestra.run.smithi098:osd.0> sudo journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.0.service 2024-09-21T09:47:43.073 INFO:tasks.cephadm:Deploying osd.1 on smithi098 with /dev/vg_nvme/lv_3... 2024-09-21T09:47:43.074 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T09:47:44.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:43 smithi098 ceph-mon[29239]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:44.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:43 smithi098 ceph-mon[29239]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T09:47:44.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:43 smithi098 ceph-mon[29239]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T09:47:44.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:43 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:47:44.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:43 smithi098 ceph-mon[29239]: from='osd.0 [v2:172.21.15.98:6802/506591636,v1:172.21.15.98:6803/506591636]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-21T09:47:44.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:43 smithi098 ceph-mon[29239]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-21T09:47:44.077 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:47:43 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0[43805]: 2024-09-21T09:47:43.750+0000 7fed37137640 -1 osd.0 0 waiting for initial osdmap 2024-09-21T09:47:44.077 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:47:43 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0[43805]: 2024-09-21T09:47:43.752+0000 7fed33f72640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:47:44.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:43 smithi098 ceph-mon[39960]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:44.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:43 smithi098 ceph-mon[39960]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T09:47:44.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:43 smithi098 ceph-mon[39960]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T09:47:44.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:43 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:47:44.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:43 smithi098 ceph-mon[39960]: from='osd.0 [v2:172.21.15.98:6802/506591636,v1:172.21.15.98:6803/506591636]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-21T09:47:44.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:43 smithi098 ceph-mon[39960]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-21T09:47:44.117 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:47:44.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:43 smithi196 ceph-mon[32670]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:44.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:43 smithi196 ceph-mon[32670]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T09:47:44.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:43 smithi196 ceph-mon[32670]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T09:47:44.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:43 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:47:44.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:43 smithi196 ceph-mon[32670]: from='osd.0 [v2:172.21.15.98:6802/506591636,v1:172.21.15.98:6803/506591636]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-21T09:47:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:43 smithi196 ceph-mon[32670]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-21T09:47:45.012 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:44 smithi098 ceph-mon[29239]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-21T09:47:45.012 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:44 smithi098 ceph-mon[29239]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T09:47:45.012 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:44 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:47:45.012 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:44 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:47:45.012 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:44 smithi098 ceph-mon[39960]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-21T09:47:45.012 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:44 smithi098 ceph-mon[39960]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T09:47:45.012 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:44 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:47:45.012 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:44 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:47:45.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:44 smithi196 ceph-mon[32670]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-21T09:47:45.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:44 smithi196 ceph-mon[32670]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T09:47:45.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:44 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:47:45.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:44 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:47:45.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:45 smithi098 ceph-mon[29239]: purged_snaps scrub starts 2024-09-21T09:47:45.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:45 smithi098 ceph-mon[29239]: purged_snaps scrub ok 2024-09-21T09:47:45.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:45 smithi098 ceph-mon[29239]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:45.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:45 smithi098 ceph-mon[29239]: osd.0 [v2:172.21.15.98:6802/506591636,v1:172.21.15.98:6803/506591636] boot 2024-09-21T09:47:45.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:45 smithi098 ceph-mon[29239]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T09:47:45.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:45 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:47:45.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:45 smithi098 ceph-mon[39960]: purged_snaps scrub starts 2024-09-21T09:47:45.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:45 smithi098 ceph-mon[39960]: purged_snaps scrub ok 2024-09-21T09:47:45.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:45 smithi098 ceph-mon[39960]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:45.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:45 smithi098 ceph-mon[39960]: osd.0 [v2:172.21.15.98:6802/506591636,v1:172.21.15.98:6803/506591636] boot 2024-09-21T09:47:45.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:45 smithi098 ceph-mon[39960]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T09:47:45.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:45 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:47:46.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:45 smithi196 ceph-mon[32670]: purged_snaps scrub starts 2024-09-21T09:47:46.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:45 smithi196 ceph-mon[32670]: purged_snaps scrub ok 2024-09-21T09:47:46.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:45 smithi196 ceph-mon[32670]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:47:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:45 smithi196 ceph-mon[32670]: osd.0 [v2:172.21.15.98:6802/506591636,v1:172.21.15.98:6803/506591636] boot 2024-09-21T09:47:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:45 smithi196 ceph-mon[32670]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T09:47:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:45 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:47:47.083 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:46 smithi098 ceph-mon[29239]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T09:47:47.083 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:46 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:47.083 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:46 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:47.083 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:46 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:47:47.084 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:46 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:47.084 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:46 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:47.084 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:46 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:47.084 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:46 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:47.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:46 smithi098 ceph-mon[39960]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T09:47:47.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:46 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:47.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:46 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:47.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:46 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:47:47.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:46 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:47.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:46 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:47.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:46 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:47.085 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:46 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:47.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:46 smithi196 ceph-mon[32670]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T09:47:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:46 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:46 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:46 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:47:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:46 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:46 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:46 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:46 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:47:47.652 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:47:47.671 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi098:vg_nvme/lv_3 2024-09-21T09:47:47.910 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:47:48.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:47 smithi098 ceph-mon[29239]: Detected new or changed devices on smithi098 2024-09-21T09:47:48.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:47 smithi098 ceph-mon[29239]: Adjusting osd_memory_target on smithi098 to 16064M 2024-09-21T09:47:48.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:47 smithi098 ceph-mon[29239]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:47:48.032 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:47 smithi098 ceph-mon[39960]: Detected new or changed devices on smithi098 2024-09-21T09:47:48.032 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:47 smithi098 ceph-mon[39960]: Adjusting osd_memory_target on smithi098 to 16064M 2024-09-21T09:47:48.032 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:47 smithi098 ceph-mon[39960]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:47:48.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:47 smithi196 ceph-mon[32670]: Detected new or changed devices on smithi098 2024-09-21T09:47:48.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:47 smithi196 ceph-mon[32670]: Adjusting osd_memory_target on smithi098 to 16064M 2024-09-21T09:47:48.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:47 smithi196 ceph-mon[32670]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:47:50.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:49 smithi098 ceph-mon[29239]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:47:50.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:49 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:47:50.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:49 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:47:50.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:49 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:50.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:49 smithi098 ceph-mon[39960]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:47:50.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:49 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:47:50.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:49 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:47:50.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:49 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:50.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:49 smithi196 ceph-mon[32670]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:47:50.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:49 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:47:50.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:49 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:47:50.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:49 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:51.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:50 smithi098 ceph-mon[29239]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:47:51.035 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:50 smithi098 ceph-mon[39960]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:47:51.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:50 smithi196 ceph-mon[32670]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:47:51.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:51 smithi098 ceph-mon[39960]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:47:51.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:51 smithi098 ceph-mon[29239]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:47:52.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:51 smithi196 ceph-mon[32670]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:47:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:52 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/127271226' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d31ac4e-dc1a-4051-8266-ba821106242f"}]: dispatch 2024-09-21T09:47:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:52 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/127271226' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d31ac4e-dc1a-4051-8266-ba821106242f"}]': finished 2024-09-21T09:47:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:52 smithi098 ceph-mon[29239]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T09:47:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:52 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:47:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:52 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3677930558' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:47:53.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:52 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/127271226' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d31ac4e-dc1a-4051-8266-ba821106242f"}]: dispatch 2024-09-21T09:47:53.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:52 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/127271226' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d31ac4e-dc1a-4051-8266-ba821106242f"}]': finished 2024-09-21T09:47:53.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:52 smithi098 ceph-mon[39960]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T09:47:53.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:52 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:47:53.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:52 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3677930558' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:47:53.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:52 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/127271226' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d31ac4e-dc1a-4051-8266-ba821106242f"}]: dispatch 2024-09-21T09:47:53.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:52 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/127271226' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d31ac4e-dc1a-4051-8266-ba821106242f"}]': finished 2024-09-21T09:47:53.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:52 smithi196 ceph-mon[32670]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T09:47:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:52 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:47:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:52 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3677930558' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:47:54.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:53 smithi098 ceph-mon[29239]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:47:54.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:53 smithi098 ceph-mon[39960]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:47:54.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:53 smithi196 ceph-mon[32670]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:47:55.886 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:55 smithi098 ceph-mon[29239]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:47:55.887 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:55 smithi098 ceph-mon[39960]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:47:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:55 smithi196 ceph-mon[32670]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:47:58.020 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:57 smithi098 ceph-mon[29239]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:47:58.020 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:57 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-21T09:47:58.020 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:57 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:58.020 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:57 smithi098 ceph-mon[39960]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:47:58.020 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:57 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-21T09:47:58.020 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:57 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:58.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:57 smithi196 ceph-mon[32670]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:47:58.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:57 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-21T09:47:58.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:57 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:59.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:58 smithi098 ceph-mon[29239]: Deploying daemon osd.1 on smithi098 2024-09-21T09:47:59.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:58 smithi098 ceph-mon[39960]: Deploying daemon osd.1 on smithi098 2024-09-21T09:47:59.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:58 smithi196 ceph-mon[32670]: Deploying daemon osd.1 on smithi098 2024-09-21T09:48:00.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:47:59 smithi098 ceph-mon[29239]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:48:00.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:47:59 smithi098 ceph-mon[39960]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:48:00.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:47:59 smithi196 ceph-mon[32670]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:48:02.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:01 smithi098 ceph-mon[29239]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:48:02.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:01 smithi098 ceph-mon[39960]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:48:02.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:01 smithi196 ceph-mon[32670]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:48:03.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:02 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:03.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:02 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:03.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:02 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:03.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:02 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:03.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:02 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:03.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:02 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:03.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:02 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:03.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:02 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:03.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:02 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:04.013 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:03 smithi098 ceph-mon[29239]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:48:04.013 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:03 smithi098 ceph-mon[39960]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:48:04.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:03 smithi196 ceph-mon[32670]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:48:05.957 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 1 on host 'smithi098' 2024-09-21T09:48:06.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:05 smithi098 ceph-mon[29239]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:48:06.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:05 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:06.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:05 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:06.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:05 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:06.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:05 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:48:06.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:05 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:06.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:05 smithi098 ceph-mon[29239]: from='osd.1 [v2:172.21.15.98:6810/4240601629,v1:172.21.15.98:6811/4240601629]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T09:48:06.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:05 smithi098 ceph-mon[29239]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T09:48:06.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:05 smithi098 ceph-mon[39960]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:48:06.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:05 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:06.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:05 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:06.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:05 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:06.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:05 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:48:06.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:05 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:06.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:05 smithi098 ceph-mon[39960]: from='osd.1 [v2:172.21.15.98:6810/4240601629,v1:172.21.15.98:6811/4240601629]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T09:48:06.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:05 smithi098 ceph-mon[39960]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T09:48:06.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:05 smithi196 ceph-mon[32670]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:48:06.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:05 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:05 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:05 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:05 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:48:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:05 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:05 smithi196 ceph-mon[32670]: from='osd.1 [v2:172.21.15.98:6810/4240601629,v1:172.21.15.98:6811/4240601629]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T09:48:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:05 smithi196 ceph-mon[32670]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T09:48:06.531 DEBUG:teuthology.orchestra.run.smithi098:osd.1> sudo journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.1.service 2024-09-21T09:48:06.533 INFO:tasks.cephadm:Deploying osd.2 on smithi098 with /dev/vg_nvme/lv_2... 2024-09-21T09:48:06.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T09:48:07.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:06 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:07.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:06 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:07.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:06 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:07.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:06 smithi098 ceph-mon[29239]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T09:48:07.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:06 smithi098 ceph-mon[29239]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T09:48:07.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:06 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:48:07.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:06 smithi098 ceph-mon[29239]: from='osd.1 [v2:172.21.15.98:6810/4240601629,v1:172.21.15.98:6811/4240601629]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-21T09:48:07.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:06 smithi098 ceph-mon[29239]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-21T09:48:07.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:07.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:07.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:07.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:06 smithi098 ceph-mon[39960]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T09:48:07.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:06 smithi098 ceph-mon[39960]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T09:48:07.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:48:07.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:06 smithi098 ceph-mon[39960]: from='osd.1 [v2:172.21.15.98:6810/4240601629,v1:172.21.15.98:6811/4240601629]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-21T09:48:07.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:06 smithi098 ceph-mon[39960]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-21T09:48:07.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:06 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:07.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:06 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:07.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:06 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:07.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:06 smithi196 ceph-mon[32670]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T09:48:07.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:06 smithi196 ceph-mon[32670]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T09:48:07.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:06 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:48:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:06 smithi196 ceph-mon[32670]: from='osd.1 [v2:172.21.15.98:6810/4240601629,v1:172.21.15.98:6811/4240601629]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-21T09:48:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:06 smithi196 ceph-mon[32670]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-21T09:48:07.247 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:48:07.449 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 09:48:07 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1[49687]: 2024-09-21T09:48:07.188+0000 7f757747e640 -1 osd.1 0 waiting for initial osdmap 2024-09-21T09:48:07.449 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 09:48:07 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1[49687]: 2024-09-21T09:48:07.190+0000 7f75702b1640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:48:08.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:07 smithi196 ceph-mon[32670]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:48:08.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:07 smithi196 ceph-mon[32670]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-21T09:48:08.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:07 smithi196 ceph-mon[32670]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T09:48:08.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:07 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:48:08.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:07 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:48:08.298 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:07 smithi098 ceph-mon[29239]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:48:08.298 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:07 smithi098 ceph-mon[29239]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-21T09:48:08.298 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:07 smithi098 ceph-mon[29239]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T09:48:08.298 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:07 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:48:08.298 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:07 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:48:08.298 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:07 smithi098 ceph-mon[39960]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:48:08.299 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:07 smithi098 ceph-mon[39960]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-21T09:48:08.299 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:07 smithi098 ceph-mon[39960]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T09:48:08.299 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:07 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:48:08.299 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:07 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:48:08.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:08 smithi098 ceph-mon[29239]: purged_snaps scrub starts 2024-09-21T09:48:08.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:08 smithi098 ceph-mon[29239]: purged_snaps scrub ok 2024-09-21T09:48:08.985 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:08 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:48:08.985 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:08 smithi098 ceph-mon[29239]: osd.1 [v2:172.21.15.98:6810/4240601629,v1:172.21.15.98:6811/4240601629] boot 2024-09-21T09:48:08.985 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:08 smithi098 ceph-mon[29239]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T09:48:08.985 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:08 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:48:08.985 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:08 smithi098 ceph-mon[39960]: purged_snaps scrub starts 2024-09-21T09:48:08.985 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:08 smithi098 ceph-mon[39960]: purged_snaps scrub ok 2024-09-21T09:48:08.985 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:08 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:48:08.985 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:08 smithi098 ceph-mon[39960]: osd.1 [v2:172.21.15.98:6810/4240601629,v1:172.21.15.98:6811/4240601629] boot 2024-09-21T09:48:08.985 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:08 smithi098 ceph-mon[39960]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T09:48:08.985 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:08 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:48:09.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:08 smithi196 ceph-mon[32670]: purged_snaps scrub starts 2024-09-21T09:48:09.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:08 smithi196 ceph-mon[32670]: purged_snaps scrub ok 2024-09-21T09:48:09.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:08 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:48:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:08 smithi196 ceph-mon[32670]: osd.1 [v2:172.21.15.98:6810/4240601629,v1:172.21.15.98:6811/4240601629] boot 2024-09-21T09:48:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:08 smithi196 ceph-mon[32670]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T09:48:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:08 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:48:09.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:09 smithi098 ceph-mon[29239]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:48:09.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:09 smithi098 ceph-mon[29239]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T09:48:09.932 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:09 smithi098 ceph-mon[39960]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:48:09.932 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:09 smithi098 ceph-mon[39960]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T09:48:10.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:09 smithi196 ceph-mon[32670]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:48:10.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:09 smithi196 ceph-mon[32670]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T09:48:11.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:11 smithi098 ceph-mon[29239]: Detected new or changed devices on smithi098 2024-09-21T09:48:11.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:11.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:11.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:11.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:11.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:11 smithi098 ceph-mon[29239]: Adjusting osd_memory_target on smithi098 to 8032M 2024-09-21T09:48:11.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:11.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:11.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:48:11.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:11.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:11 smithi098 ceph-mon[29239]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:11.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:11 smithi098 ceph-mon[39960]: Detected new or changed devices on smithi098 2024-09-21T09:48:11.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:11.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:11.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:11.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:11.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:11 smithi098 ceph-mon[39960]: Adjusting osd_memory_target on smithi098 to 8032M 2024-09-21T09:48:11.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:11.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:11.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:48:11.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:11.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:11 smithi098 ceph-mon[39960]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:11.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:11 smithi196 ceph-mon[32670]: Detected new or changed devices on smithi098 2024-09-21T09:48:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:11 smithi196 ceph-mon[32670]: Adjusting osd_memory_target on smithi098 to 8032M 2024-09-21T09:48:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:48:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:11 smithi196 ceph-mon[32670]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:11.689 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:48:11.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi098:vg_nvme/lv_2 2024-09-21T09:48:11.947 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:48:13.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:13 smithi098 ceph-mon[29239]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:13.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:13 smithi098 ceph-mon[39960]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:13.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:13 smithi196 ceph-mon[32670]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:14.807 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:14 smithi098 ceph-mon[29239]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:14.808 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:14 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:48:14.808 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:14 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:48:14.808 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:14 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:14.808 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:14 smithi098 ceph-mon[39960]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:14.808 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:14 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:48:14.808 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:14 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:48:14.808 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:14 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:14 smithi196 ceph-mon[32670]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:14 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:48:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:14 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:48:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:14 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:15.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:15 smithi196 ceph-mon[32670]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:15.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:15 smithi098 ceph-mon[29239]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:15.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:15 smithi098 ceph-mon[39960]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:16 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/1097920364' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62e22789-74be-4098-a9b9-62671fb4cac1"}]: dispatch 2024-09-21T09:48:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:16 smithi098 ceph-mon[29239]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62e22789-74be-4098-a9b9-62671fb4cac1"}]: dispatch 2024-09-21T09:48:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:16 smithi098 ceph-mon[29239]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62e22789-74be-4098-a9b9-62671fb4cac1"}]': finished 2024-09-21T09:48:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:16 smithi098 ceph-mon[29239]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T09:48:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:16 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:48:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:16 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/307156866' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:48:16.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:16 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/1097920364' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62e22789-74be-4098-a9b9-62671fb4cac1"}]: dispatch 2024-09-21T09:48:16.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:16 smithi098 ceph-mon[39960]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62e22789-74be-4098-a9b9-62671fb4cac1"}]: dispatch 2024-09-21T09:48:16.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:16 smithi098 ceph-mon[39960]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62e22789-74be-4098-a9b9-62671fb4cac1"}]': finished 2024-09-21T09:48:16.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:16 smithi098 ceph-mon[39960]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T09:48:16.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:16 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:48:16.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:16 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/307156866' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:48:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:16 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/1097920364' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62e22789-74be-4098-a9b9-62671fb4cac1"}]: dispatch 2024-09-21T09:48:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:16 smithi196 ceph-mon[32670]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62e22789-74be-4098-a9b9-62671fb4cac1"}]: dispatch 2024-09-21T09:48:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:16 smithi196 ceph-mon[32670]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62e22789-74be-4098-a9b9-62671fb4cac1"}]': finished 2024-09-21T09:48:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:16 smithi196 ceph-mon[32670]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T09:48:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:16 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:48:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:16 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/307156866' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:48:18.096 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:17 smithi098 ceph-mon[29239]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:18.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:17 smithi098 ceph-mon[39960]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:18.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:17 smithi196 ceph-mon[32670]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:20.054 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:19 smithi098 ceph-mon[29239]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:20.054 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:19 smithi098 ceph-mon[39960]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:20.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:19 smithi196 ceph-mon[32670]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:22.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:21 smithi098 ceph-mon[29239]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:22.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:21 smithi098 ceph-mon[39960]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:22.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:21 smithi196 ceph-mon[32670]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:22.996 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:22 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-21T09:48:22.996 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:22 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:22.996 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:22 smithi098 ceph-mon[39960]: Deploying daemon osd.2 on smithi098 2024-09-21T09:48:22.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:22 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-21T09:48:22.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:22 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:22.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:22 smithi098 ceph-mon[29239]: Deploying daemon osd.2 on smithi098 2024-09-21T09:48:23.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:22 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-21T09:48:23.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:22 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:22 smithi196 ceph-mon[32670]: Deploying daemon osd.2 on smithi098 2024-09-21T09:48:23.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:23 smithi098 ceph-mon[29239]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:23.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:23 smithi098 ceph-mon[39960]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:24.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:23 smithi196 ceph-mon[32670]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:26.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:25 smithi098 ceph-mon[29239]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:26.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:25 smithi098 ceph-mon[39960]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:26.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:25 smithi196 ceph-mon[32670]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:28.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:27 smithi098 ceph-mon[29239]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:28.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:27 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:28.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:27 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:28.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:27 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:28.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:27 smithi098 ceph-mon[39960]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:28.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:27 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:28.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:27 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:28.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:27 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:28.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:27 smithi196 ceph-mon[32670]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:28.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:27 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:28.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:27 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:28.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:27 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:29.827 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 2 on host 'smithi098' 2024-09-21T09:48:30.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:29 smithi098 ceph-mon[39960]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:30.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:29 smithi098 ceph-mon[29239]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:30.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:29 smithi196 ceph-mon[32670]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:30.397 DEBUG:teuthology.orchestra.run.smithi098:osd.2> sudo journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.2.service 2024-09-21T09:48:30.399 INFO:tasks.cephadm:Deploying osd.3 on smithi098 with /dev/vg_nvme/lv_1... 2024-09-21T09:48:30.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T09:48:30.711 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:48:31.091 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:30 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:31.091 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:30 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:31.091 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:30 smithi098 ceph-mon[29239]: from='osd.2 [v2:172.21.15.98:6818/2917126560,v1:172.21.15.98:6819/2917126560]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-21T09:48:31.091 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:30 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:31.091 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:30 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:31.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:30 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:31.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:30 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:31.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:30 smithi098 ceph-mon[39960]: from='osd.2 [v2:172.21.15.98:6818/2917126560,v1:172.21.15.98:6819/2917126560]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-21T09:48:31.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:30 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:31.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:30 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:31.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:30 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:30 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:30 smithi196 ceph-mon[32670]: from='osd.2 [v2:172.21.15.98:6818/2917126560,v1:172.21.15.98:6819/2917126560]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-21T09:48:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:30 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:30 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:32.024 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:31 smithi098 ceph-mon[29239]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:32.024 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:31 smithi098 ceph-mon[29239]: from='osd.2 [v2:172.21.15.98:6818/2917126560,v1:172.21.15.98:6819/2917126560]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T09:48:32.025 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:31 smithi098 ceph-mon[29239]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T09:48:32.025 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:31 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:48:32.025 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:31 smithi098 ceph-mon[29239]: from='osd.2 [v2:172.21.15.98:6818/2917126560,v1:172.21.15.98:6819/2917126560]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-21T09:48:32.025 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 09:48:31 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2[55266]: 2024-09-21T09:48:31.840+0000 7fe13f818640 -1 osd.2 0 waiting for initial osdmap 2024-09-21T09:48:32.025 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 09:48:31 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2[55266]: 2024-09-21T09:48:31.844+0000 7fe13864b640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:48:32.025 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:31 smithi098 ceph-mon[39960]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:32.025 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:31 smithi098 ceph-mon[39960]: from='osd.2 [v2:172.21.15.98:6818/2917126560,v1:172.21.15.98:6819/2917126560]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T09:48:32.026 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:31 smithi098 ceph-mon[39960]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T09:48:32.026 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:31 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:48:32.026 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:31 smithi098 ceph-mon[39960]: from='osd.2 [v2:172.21.15.98:6818/2917126560,v1:172.21.15.98:6819/2917126560]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-21T09:48:32.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:31 smithi196 ceph-mon[32670]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:32.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:31 smithi196 ceph-mon[32670]: from='osd.2 [v2:172.21.15.98:6818/2917126560,v1:172.21.15.98:6819/2917126560]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T09:48:32.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:31 smithi196 ceph-mon[32670]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T09:48:32.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:31 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:48:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:31 smithi196 ceph-mon[32670]: from='osd.2 [v2:172.21.15.98:6818/2917126560,v1:172.21.15.98:6819/2917126560]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-21T09:48:33.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:32 smithi098 ceph-mon[29239]: from='osd.2 [v2:172.21.15.98:6818/2917126560,v1:172.21.15.98:6819/2917126560]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-21T09:48:33.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:32 smithi098 ceph-mon[29239]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T09:48:33.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:32 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:48:33.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:32 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:48:33.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:32 smithi098 ceph-mon[39960]: from='osd.2 [v2:172.21.15.98:6818/2917126560,v1:172.21.15.98:6819/2917126560]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-21T09:48:33.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:32 smithi098 ceph-mon[39960]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T09:48:33.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:32 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:48:33.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:32 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:48:33.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:32 smithi196 ceph-mon[32670]: from='osd.2 [v2:172.21.15.98:6818/2917126560,v1:172.21.15.98:6819/2917126560]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-21T09:48:33.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:32 smithi196 ceph-mon[32670]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T09:48:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:32 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:48:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:32 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:48:33.923 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:48:33.949 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi098:vg_nvme/lv_1 2024-09-21T09:48:34.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:33 smithi098 ceph-mon[29239]: purged_snaps scrub starts 2024-09-21T09:48:34.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:33 smithi098 ceph-mon[29239]: purged_snaps scrub ok 2024-09-21T09:48:34.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:33 smithi098 ceph-mon[29239]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:34.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:33 smithi098 ceph-mon[29239]: osd.2 [v2:172.21.15.98:6818/2917126560,v1:172.21.15.98:6819/2917126560] boot 2024-09-21T09:48:34.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:33 smithi098 ceph-mon[29239]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T09:48:34.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:33 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:48:34.103 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:33 smithi098 ceph-mon[39960]: purged_snaps scrub starts 2024-09-21T09:48:34.103 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:33 smithi098 ceph-mon[39960]: purged_snaps scrub ok 2024-09-21T09:48:34.103 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:33 smithi098 ceph-mon[39960]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:34.103 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:33 smithi098 ceph-mon[39960]: osd.2 [v2:172.21.15.98:6818/2917126560,v1:172.21.15.98:6819/2917126560] boot 2024-09-21T09:48:34.103 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:33 smithi098 ceph-mon[39960]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T09:48:34.103 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:33 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:48:34.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:33 smithi196 ceph-mon[32670]: purged_snaps scrub starts 2024-09-21T09:48:34.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:33 smithi196 ceph-mon[32670]: purged_snaps scrub ok 2024-09-21T09:48:34.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:33 smithi196 ceph-mon[32670]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:48:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:33 smithi196 ceph-mon[32670]: osd.2 [v2:172.21.15.98:6818/2917126560,v1:172.21.15.98:6819/2917126560] boot 2024-09-21T09:48:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:33 smithi196 ceph-mon[32670]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T09:48:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:33 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:48:34.446 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:48:35.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:34 smithi098 ceph-mon[29239]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T09:48:35.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:34 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-21T09:48:35.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:34 smithi098 ceph-mon[39960]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T09:48:35.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:34 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-21T09:48:35.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:34 smithi196 ceph-mon[32670]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T09:48:35.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:34 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-21T09:48:36.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[29239]: pgmap v74: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:36.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-21T09:48:36.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[29239]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T09:48:36.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:48:36.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:36.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:36.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:36.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:36.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:36.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:36.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:36.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:48:36.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:36.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[39960]: pgmap v74: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:36.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-21T09:48:36.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[39960]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T09:48:36.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:48:36.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:36.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:36.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:36.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:36.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:36.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:36.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:36.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:48:36.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:35 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:36.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:35 smithi196 ceph-mon[32670]: pgmap v74: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:35 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-21T09:48:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:35 smithi196 ceph-mon[32670]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T09:48:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:35 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:48:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:35 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:35 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:35 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:35 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:35 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:35 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:35 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:35 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:48:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:35 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:36.499 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:48:36 smithi098 sudo[58980]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:48:36.499 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:48:36 smithi098 sudo[58980]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:48:36.499 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:48:36 smithi098 sudo[58980]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:48:36.500 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:48:36 smithi098 sudo[58980]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:48:36.828 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:48:36 smithi098 sudo[58987]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:48:36.828 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:48:36 smithi098 sudo[58987]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:48:36.828 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:48:36 smithi098 sudo[58987]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:48:36.828 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:48:36 smithi098 sudo[58987]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:48:37.092 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 09:48:36 smithi098 sudo[59012]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:48:37.092 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 09:48:36 smithi098 sudo[59012]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:48:37.092 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 09:48:36 smithi098 sudo[59012]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:48:37.092 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 09:48:36 smithi098 sudo[59012]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:48:37.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:36 smithi098 ceph-mon[29239]: Detected new or changed devices on smithi098 2024-09-21T09:48:37.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:36 smithi098 ceph-mon[29239]: Adjusting osd_memory_target on smithi098 to 5354M 2024-09-21T09:48:37.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:36 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:37.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:36 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:48:37.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:36 smithi098 ceph-mon[29239]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T09:48:37.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:36 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:37.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:36 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:48:37.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:36 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:37.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:36 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:37.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:36 smithi098 ceph-mon[39960]: Detected new or changed devices on smithi098 2024-09-21T09:48:37.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:36 smithi098 ceph-mon[39960]: Adjusting osd_memory_target on smithi098 to 5354M 2024-09-21T09:48:37.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:36 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:37.094 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:36 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:48:37.094 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:36 smithi098 ceph-mon[39960]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T09:48:37.094 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:36 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:37.094 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:36 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:48:37.094 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:36 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:37.094 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:36 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:37.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:36 smithi196 ceph-mon[32670]: Detected new or changed devices on smithi098 2024-09-21T09:48:37.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:36 smithi196 ceph-mon[32670]: Adjusting osd_memory_target on smithi098 to 5354M 2024-09-21T09:48:37.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:36 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:36 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:48:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:36 smithi196 ceph-mon[32670]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T09:48:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:36 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:36 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:48:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:36 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:36 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:37.347 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 09:48:37 smithi098 sudo[59082]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:48:37.347 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 09:48:37 smithi098 sudo[59082]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:48:37.347 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 09:48:37 smithi098 sudo[59082]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:48:37.347 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 09:48:37 smithi098 sudo[59082]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:48:37.613 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 09:48:37 smithi098 sudo[59140]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:48:37.613 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 09:48:37 smithi098 sudo[59140]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:48:37.613 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 09:48:37 smithi098 sudo[59140]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:48:37.613 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 09:48:37 smithi098 sudo[59140]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:48:37.877 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:37 smithi098 sudo[59167]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:48:37.877 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:37 smithi098 sudo[59167]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:48:37.877 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:37 smithi098 sudo[59167]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:48:37.877 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:37 smithi098 sudo[59167]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:48:37.877 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 09:48:37 smithi098 sudo[59161]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:48:37.877 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 09:48:37 smithi098 sudo[59161]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:48:37.878 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 09:48:37 smithi098 sudo[59161]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:48:37.878 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 09:48:37 smithi098 sudo[59161]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:48:38.153 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:38 smithi098 sudo[59170]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:48:38.153 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:38 smithi098 sudo[59170]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:48:38.153 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:38 smithi098 sudo[59170]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:48:38.153 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:38 smithi098 sudo[59170]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:48:38.153 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:38 smithi098 ceph-mon[29239]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:38.153 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:38 smithi098 ceph-mon[29239]: osdmap e22: 3 total, 3 up, 3 in 2024-09-21T09:48:38.154 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:38 smithi098 ceph-mon[29239]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T09:48:38.154 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:38 smithi098 ceph-mon[29239]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:38.154 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:38 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:48:38.154 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:38 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:48:38.154 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:38 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:38.154 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:38 smithi098 ceph-mon[39960]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:38.154 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:38 smithi098 ceph-mon[39960]: osdmap e22: 3 total, 3 up, 3 in 2024-09-21T09:48:38.154 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:38 smithi098 ceph-mon[39960]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T09:48:38.154 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:38 smithi098 ceph-mon[39960]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:38.154 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:38 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:48:38.154 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:38 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:48:38.154 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:38 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:38.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:38 smithi196 ceph-mon[32670]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:38.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:38 smithi196 ceph-mon[32670]: osdmap e22: 3 total, 3 up, 3 in 2024-09-21T09:48:38.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:38 smithi196 ceph-mon[32670]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T09:48:38.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:38 smithi196 ceph-mon[32670]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:38.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:38 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:48:38.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:38 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:48:38.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:38 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:38.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:38 smithi196 sudo[35156]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:48:38.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:38 smithi196 sudo[35156]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:48:38.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:38 smithi196 sudo[35156]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:48:38.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:38 smithi196 sudo[35156]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:48:38.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:38 smithi196 sudo[35159]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:48:38.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:38 smithi196 sudo[35159]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:48:38.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:38 smithi196 sudo[35159]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:48:38.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:38 smithi196 sudo[35159]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:48:39.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:38 smithi098 sudo[59221]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:48:39.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:38 smithi098 sudo[59221]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:48:39.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:38 smithi098 sudo[59221]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:48:39.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:38 smithi098 sudo[59221]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:48:39.416 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[29239]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:48:39.416 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[29239]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:48:39.416 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:48:39.416 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:48:39.416 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:48:39.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[29239]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:48:39.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:48:39.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:48:39.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:48:39.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[29239]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:39.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[29239]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:48:39.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:39.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:48:39.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:48:39.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:48:39.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2238919243' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f653a6-69b0-437a-960d-c3fb036708be"}]: dispatch 2024-09-21T09:48:39.417 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:39 smithi098 sudo[59244]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:48:39.418 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:39 smithi098 sudo[59244]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:48:39.418 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:39 smithi098 sudo[59244]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:48:39.418 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:39 smithi098 sudo[59244]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:48:39.418 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[39960]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:48:39.418 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[39960]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:48:39.418 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:48:39.418 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:48:39.418 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:48:39.418 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[39960]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:48:39.418 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:48:39.418 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:48:39.418 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:48:39.418 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[39960]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:39.419 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[39960]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:48:39.419 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:39.419 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:48:39.419 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:48:39.419 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:48:39.419 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:39 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2238919243' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f653a6-69b0-437a-960d-c3fb036708be"}]: dispatch 2024-09-21T09:48:39.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:39 smithi196 ceph-mon[32670]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:48:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:39 smithi196 ceph-mon[32670]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:48:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:39 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:48:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:39 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:48:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:39 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:48:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:39 smithi196 ceph-mon[32670]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:48:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:39 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:48:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:39 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:48:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:39 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:48:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:39 smithi196 ceph-mon[32670]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:39 smithi196 ceph-mon[32670]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:48:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:39 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:39 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:48:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:39 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:48:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:39 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:48:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:39 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2238919243' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f653a6-69b0-437a-960d-c3fb036708be"}]: dispatch 2024-09-21T09:48:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:40 smithi098 ceph-mon[29239]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:48:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:40 smithi098 ceph-mon[29239]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:48:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:40 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2238919243' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77f653a6-69b0-437a-960d-c3fb036708be"}]': finished 2024-09-21T09:48:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:40 smithi098 ceph-mon[29239]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T09:48:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:40 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:48:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:40 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3453846880' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:48:40.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:40 smithi098 ceph-mon[39960]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:48:40.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:40 smithi098 ceph-mon[39960]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:48:40.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:40 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2238919243' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77f653a6-69b0-437a-960d-c3fb036708be"}]': finished 2024-09-21T09:48:40.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:40 smithi098 ceph-mon[39960]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T09:48:40.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:40 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:48:40.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:40 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3453846880' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:48:40.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:40 smithi196 ceph-mon[32670]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:48:40.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:40 smithi196 ceph-mon[32670]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:48:40.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:40 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2238919243' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77f653a6-69b0-437a-960d-c3fb036708be"}]': finished 2024-09-21T09:48:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:40 smithi196 ceph-mon[32670]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T09:48:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:40 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:48:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:40 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3453846880' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:48:41.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:41 smithi098 ceph-mon[29239]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:41.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:41 smithi098 ceph-mon[39960]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:41.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:41 smithi196 ceph-mon[32670]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:43.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:43 smithi098 ceph-mon[39960]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:43.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:43 smithi098 ceph-mon[29239]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:43.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:43 smithi196 ceph-mon[32670]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:45.774 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:45 smithi098 ceph-mon[29239]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:45.774 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:45 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-21T09:48:45.774 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:45 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:45.775 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:45 smithi098 ceph-mon[39960]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:45.775 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:45 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-21T09:48:45.775 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:45 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:45.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:45 smithi196 ceph-mon[32670]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:45.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:45 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-21T09:48:45.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:45 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:46.779 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:46 smithi098 ceph-mon[39960]: Deploying daemon osd.3 on smithi098 2024-09-21T09:48:46.779 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:46 smithi098 ceph-mon[29239]: Deploying daemon osd.3 on smithi098 2024-09-21T09:48:46.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:46 smithi196 ceph-mon[32670]: Deploying daemon osd.3 on smithi098 2024-09-21T09:48:47.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:47 smithi098 ceph-mon[29239]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:47.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:47 smithi098 ceph-mon[39960]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:47.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:47 smithi196 ceph-mon[32670]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:49.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:49 smithi098 ceph-mon[29239]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:49.704 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:49 smithi098 ceph-mon[39960]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:49.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:49 smithi196 ceph-mon[32670]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:50.768 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:50 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:50.768 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:50 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:50.768 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:50 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:50.769 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:50 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:50.769 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:50 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:50.769 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:50 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:50 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:50 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:50 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:51.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:51 smithi098 ceph-mon[29239]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:51.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:51 smithi098 ceph-mon[39960]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:51.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:51 smithi196 ceph-mon[32670]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:53.576 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 3 on host 'smithi098' 2024-09-21T09:48:54.137 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:53 smithi098 ceph-mon[29239]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:54.137 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:53 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:54.137 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:53 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:54.137 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:53 smithi098 ceph-mon[29239]: from='osd.3 [v2:172.21.15.98:6826/4022556179,v1:172.21.15.98:6827/4022556179]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T09:48:54.137 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:53 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:54.137 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:53 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:48:54.138 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:53 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:54.138 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:53 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:54.138 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:53 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:54.138 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:53 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:54.138 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:53 smithi098 ceph-mon[39960]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:54.138 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:53 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:54.139 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:53 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:54.139 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:53 smithi098 ceph-mon[39960]: from='osd.3 [v2:172.21.15.98:6826/4022556179,v1:172.21.15.98:6827/4022556179]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T09:48:54.139 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:53 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:54.139 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:53 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:48:54.139 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:53 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:54.139 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:53 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:54.139 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:53 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:54.140 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:53 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:53 smithi196 ceph-mon[32670]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:53 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:53 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:53 smithi196 ceph-mon[32670]: from='osd.3 [v2:172.21.15.98:6826/4022556179,v1:172.21.15.98:6827/4022556179]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T09:48:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:53 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:53 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:48:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:53 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:54.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:53 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:48:54.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:53 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:54.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:53 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:54.160 DEBUG:teuthology.orchestra.run.smithi098:osd.3> sudo journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.3.service 2024-09-21T09:48:54.163 INFO:tasks.cephadm:Deploying osd.4 on smithi196 with /dev/vg_nvme/lv_4... 2024-09-21T09:48:54.163 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T09:48:54.408 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:48:55.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:54 smithi196 ceph-mon[32670]: from='osd.3 [v2:172.21.15.98:6826/4022556179,v1:172.21.15.98:6827/4022556179]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T09:48:55.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:54 smithi196 ceph-mon[32670]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T09:48:55.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:54 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:48:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:54 smithi196 ceph-mon[32670]: from='osd.3 [v2:172.21.15.98:6826/4022556179,v1:172.21.15.98:6827/4022556179]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-21T09:48:55.294 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:54 smithi098 ceph-mon[29239]: from='osd.3 [v2:172.21.15.98:6826/4022556179,v1:172.21.15.98:6827/4022556179]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T09:48:55.294 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:54 smithi098 ceph-mon[29239]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T09:48:55.295 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:54 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:48:55.295 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:54 smithi098 ceph-mon[29239]: from='osd.3 [v2:172.21.15.98:6826/4022556179,v1:172.21.15.98:6827/4022556179]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-21T09:48:55.295 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:54 smithi098 ceph-mon[39960]: from='osd.3 [v2:172.21.15.98:6826/4022556179,v1:172.21.15.98:6827/4022556179]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T09:48:55.295 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:54 smithi098 ceph-mon[39960]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T09:48:55.295 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:54 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:48:55.295 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:54 smithi098 ceph-mon[39960]: from='osd.3 [v2:172.21.15.98:6826/4022556179,v1:172.21.15.98:6827/4022556179]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-09-21T09:48:55.296 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 09:48:54 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3[60910]: 2024-09-21T09:48:54.892+0000 7f75a861f640 -1 osd.3 0 waiting for initial osdmap 2024-09-21T09:48:55.296 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 09:48:54 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3[60910]: 2024-09-21T09:48:54.896+0000 7f75a1c31640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:48:56.024 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:55 smithi196 ceph-mon[32670]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:56.024 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:55 smithi196 ceph-mon[32670]: from='osd.3 [v2:172.21.15.98:6826/4022556179,v1:172.21.15.98:6827/4022556179]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-21T09:48:56.024 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:55 smithi196 ceph-mon[32670]: osdmap e25: 4 total, 3 up, 4 in 2024-09-21T09:48:56.024 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:55 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:48:56.024 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:55 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:48:56.025 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:55 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:48:56.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:55 smithi098 ceph-mon[29239]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:56.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:55 smithi098 ceph-mon[29239]: from='osd.3 [v2:172.21.15.98:6826/4022556179,v1:172.21.15.98:6827/4022556179]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-21T09:48:56.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:55 smithi098 ceph-mon[29239]: osdmap e25: 4 total, 3 up, 4 in 2024-09-21T09:48:56.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:55 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:48:56.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:55 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:48:56.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:55 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:48:56.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:55 smithi098 ceph-mon[39960]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:56.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:55 smithi098 ceph-mon[39960]: from='osd.3 [v2:172.21.15.98:6826/4022556179,v1:172.21.15.98:6827/4022556179]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-09-21T09:48:56.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:55 smithi098 ceph-mon[39960]: osdmap e25: 4 total, 3 up, 4 in 2024-09-21T09:48:56.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:55 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:48:56.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:55 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:48:56.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:55 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:48:56.734 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:48:56.754 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi196:vg_nvme/lv_4 2024-09-21T09:48:57.001 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:48:57.122 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:56 smithi196 ceph-mon[32670]: purged_snaps scrub starts 2024-09-21T09:48:57.122 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:56 smithi196 ceph-mon[32670]: purged_snaps scrub ok 2024-09-21T09:48:57.122 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:56 smithi196 ceph-mon[32670]: osd.3 [v2:172.21.15.98:6826/4022556179,v1:172.21.15.98:6827/4022556179] boot 2024-09-21T09:48:57.122 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:56 smithi196 ceph-mon[32670]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T09:48:57.122 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:56 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:48:57.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:56 smithi098 ceph-mon[29239]: purged_snaps scrub starts 2024-09-21T09:48:57.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:56 smithi098 ceph-mon[29239]: purged_snaps scrub ok 2024-09-21T09:48:57.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:56 smithi098 ceph-mon[29239]: osd.3 [v2:172.21.15.98:6826/4022556179,v1:172.21.15.98:6827/4022556179] boot 2024-09-21T09:48:57.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:56 smithi098 ceph-mon[29239]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T09:48:57.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:56 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:48:57.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:56 smithi098 ceph-mon[39960]: purged_snaps scrub starts 2024-09-21T09:48:57.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:56 smithi098 ceph-mon[39960]: purged_snaps scrub ok 2024-09-21T09:48:57.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:56 smithi098 ceph-mon[39960]: osd.3 [v2:172.21.15.98:6826/4022556179,v1:172.21.15.98:6827/4022556179] boot 2024-09-21T09:48:57.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:56 smithi098 ceph-mon[39960]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T09:48:57.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:56 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:48:58.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:57 smithi196 ceph-mon[32670]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:58.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:57 smithi196 ceph-mon[32670]: Detected new or changed devices on smithi098 2024-09-21T09:48:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:57 smithi196 ceph-mon[32670]: osdmap e27: 4 total, 4 up, 4 in 2024-09-21T09:48:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:57 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:57 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:57 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:57 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:57 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:57 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:57 smithi196 ceph-mon[32670]: Adjusting osd_memory_target on smithi098 to 4016M 2024-09-21T09:48:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:57 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:57 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:57 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:48:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:57 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:58.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[29239]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:58.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[29239]: Detected new or changed devices on smithi098 2024-09-21T09:48:58.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[29239]: osdmap e27: 4 total, 4 up, 4 in 2024-09-21T09:48:58.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:58.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:58.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:58.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:58.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:58.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:58.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[29239]: Adjusting osd_memory_target on smithi098 to 4016M 2024-09-21T09:48:58.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:58.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:58.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:48:58.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:58.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[39960]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:48:58.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[39960]: Detected new or changed devices on smithi098 2024-09-21T09:48:58.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[39960]: osdmap e27: 4 total, 4 up, 4 in 2024-09-21T09:48:58.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:58.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:58.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:58.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:58.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:58.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:48:58.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[39960]: Adjusting osd_memory_target on smithi098 to 4016M 2024-09-21T09:48:58.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:58.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:58.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:48:58.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:57 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:48:59.232 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:58 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:48:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:58 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:48:59.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:58 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:59.351 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:58 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:48:59.351 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:58 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:48:59.351 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:58 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:59.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:58 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:48:59.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:58 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:48:59.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:58 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:48:59.979 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:59 smithi196 ceph-mon[32670]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:48:59.979 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:48:59 smithi196 ceph-mon[32670]: from='client.24184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:00.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:59 smithi098 ceph-mon[29239]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:00.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:48:59 smithi098 ceph-mon[29239]: from='client.24184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:00.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:59 smithi098 ceph-mon[39960]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:00.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:48:59 smithi098 ceph-mon[39960]: from='client.24184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:01.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:00 smithi196 ceph-mon[32670]: from='client.? 172.21.15.196:0/2385658486' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28ec4928-7f00-46a9-b8c5-95f0b7e6be75"}]: dispatch 2024-09-21T09:49:01.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:00 smithi196 ceph-mon[32670]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28ec4928-7f00-46a9-b8c5-95f0b7e6be75"}]: dispatch 2024-09-21T09:49:01.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:00 smithi196 ceph-mon[32670]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "28ec4928-7f00-46a9-b8c5-95f0b7e6be75"}]': finished 2024-09-21T09:49:01.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:00 smithi196 ceph-mon[32670]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T09:49:01.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:00 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:49:01.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:00 smithi098 ceph-mon[29239]: from='client.? 172.21.15.196:0/2385658486' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28ec4928-7f00-46a9-b8c5-95f0b7e6be75"}]: dispatch 2024-09-21T09:49:01.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:00 smithi098 ceph-mon[29239]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28ec4928-7f00-46a9-b8c5-95f0b7e6be75"}]: dispatch 2024-09-21T09:49:01.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:00 smithi098 ceph-mon[29239]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "28ec4928-7f00-46a9-b8c5-95f0b7e6be75"}]': finished 2024-09-21T09:49:01.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:00 smithi098 ceph-mon[29239]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T09:49:01.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:00 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:49:01.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:00 smithi098 ceph-mon[39960]: from='client.? 172.21.15.196:0/2385658486' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28ec4928-7f00-46a9-b8c5-95f0b7e6be75"}]: dispatch 2024-09-21T09:49:01.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:00 smithi098 ceph-mon[39960]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28ec4928-7f00-46a9-b8c5-95f0b7e6be75"}]: dispatch 2024-09-21T09:49:01.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:00 smithi098 ceph-mon[39960]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "28ec4928-7f00-46a9-b8c5-95f0b7e6be75"}]': finished 2024-09-21T09:49:01.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:00 smithi098 ceph-mon[39960]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T09:49:01.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:00 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:49:02.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:01 smithi098 ceph-mon[29239]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:02.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:01 smithi098 ceph-mon[29239]: from='client.? 172.21.15.196:0/1398797337' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:49:02.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:01 smithi098 ceph-mon[39960]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:02.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:01 smithi098 ceph-mon[39960]: from='client.? 172.21.15.196:0/1398797337' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:49:02.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:01 smithi196 ceph-mon[32670]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:02.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:01 smithi196 ceph-mon[32670]: from='client.? 172.21.15.196:0/1398797337' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:49:04.101 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:03 smithi196 ceph-mon[32670]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:04.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:03 smithi098 ceph-mon[29239]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:04.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:03 smithi098 ceph-mon[39960]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:06.014 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:05 smithi196 ceph-mon[32670]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:06.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:05 smithi098 ceph-mon[29239]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:06.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:05 smithi098 ceph-mon[39960]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:07.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:06 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-21T09:49:07.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:06 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:07.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:06 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-21T09:49:07.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:06 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:07.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-21T09:49:07.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:06 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:08.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:07 smithi196 ceph-mon[32670]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:08.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:07 smithi196 ceph-mon[32670]: Deploying daemon osd.4 on smithi196 2024-09-21T09:49:08.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:07 smithi098 ceph-mon[29239]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:08.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:07 smithi098 ceph-mon[29239]: Deploying daemon osd.4 on smithi196 2024-09-21T09:49:08.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:07 smithi098 ceph-mon[39960]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:08.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:07 smithi098 ceph-mon[39960]: Deploying daemon osd.4 on smithi196 2024-09-21T09:49:10.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:09 smithi196 ceph-mon[32670]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:10.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:09 smithi098 ceph-mon[29239]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:10.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:09 smithi098 ceph-mon[39960]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:12.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:11 smithi196 ceph-mon[32670]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:11 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:12.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:11 smithi098 ceph-mon[29239]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:12.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:12.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:12.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:11 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:12.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:11 smithi098 ceph-mon[39960]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:12.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:12.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:12.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:11 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:14.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:13 smithi196 ceph-mon[32670]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:14.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:13 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:14.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:13 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:14.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:13 smithi098 ceph-mon[29239]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:14.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:13 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:14.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:13 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:14.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:13 smithi098 ceph-mon[39960]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:14.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:13 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:14.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:13 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:14.524 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 4 on host 'smithi196' 2024-09-21T09:49:15.100 DEBUG:teuthology.orchestra.run.smithi196:osd.4> sudo journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.4.service 2024-09-21T09:49:15.102 INFO:tasks.cephadm:Deploying osd.5 on smithi196 with /dev/vg_nvme/lv_3... 2024-09-21T09:49:15.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T09:49:15.395 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:49:15.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:15 smithi196 ceph-mon[32670]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:15.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:15 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:15.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:15 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:15.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:15 smithi196 ceph-mon[32670]: from='osd.4 [v2:172.21.15.196:6800/158468112,v1:172.21.15.196:6801/158468112]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:49:15.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:15 smithi196 ceph-mon[32670]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:49:15.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:15 smithi098 ceph-mon[39960]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:15.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:15 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:15.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:15 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:15.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:15 smithi098 ceph-mon[39960]: from='osd.4 [v2:172.21.15.196:6800/158468112,v1:172.21.15.196:6801/158468112]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:49:15.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:15 smithi098 ceph-mon[39960]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:49:15.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:15 smithi098 ceph-mon[29239]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:15.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:15 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:15.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:15 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:15.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:15 smithi098 ceph-mon[29239]: from='osd.4 [v2:172.21.15.196:6800/158468112,v1:172.21.15.196:6801/158468112]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:49:15.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:15 smithi098 ceph-mon[29239]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:49:16.742 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:16 smithi196 ceph-mon[32670]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T09:49:16.742 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:16 smithi196 ceph-mon[32670]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T09:49:16.742 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:16 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:49:16.742 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:16 smithi196 ceph-mon[32670]: from='osd.4 [v2:172.21.15.196:6800/158468112,v1:172.21.15.196:6801/158468112]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:49:16.743 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:16 smithi196 ceph-mon[32670]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:49:16.743 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 09:49:16 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4[37445]: 2024-09-21T09:49:16.532+0000 7fcc1b1ce640 -1 osd.4 0 waiting for initial osdmap 2024-09-21T09:49:16.743 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 09:49:16 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4[37445]: 2024-09-21T09:49:16.536+0000 7fcc15003640 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:49:16.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:16 smithi098 ceph-mon[39960]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T09:49:16.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:16 smithi098 ceph-mon[39960]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T09:49:16.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:16 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:49:16.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:16 smithi098 ceph-mon[39960]: from='osd.4 [v2:172.21.15.196:6800/158468112,v1:172.21.15.196:6801/158468112]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:49:16.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:16 smithi098 ceph-mon[39960]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:49:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:16 smithi098 ceph-mon[29239]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T09:49:16.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:16 smithi098 ceph-mon[29239]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T09:49:16.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:16 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:49:16.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:16 smithi098 ceph-mon[29239]: from='osd.4 [v2:172.21.15.196:6800/158468112,v1:172.21.15.196:6801/158468112]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:49:16.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:16 smithi098 ceph-mon[29239]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:49:17.812 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:17 smithi196 ceph-mon[32670]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:17.812 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:17 smithi196 ceph-mon[32670]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-21T09:49:17.812 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:17 smithi196 ceph-mon[32670]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T09:49:17.812 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:17 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:49:17.812 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:17 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:49:17.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:17 smithi098 ceph-mon[29239]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:17.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:17 smithi098 ceph-mon[29239]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-21T09:49:17.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:17 smithi098 ceph-mon[29239]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T09:49:17.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:17 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:49:17.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:17 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:49:17.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:17 smithi098 ceph-mon[39960]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:17.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:17 smithi098 ceph-mon[39960]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-21T09:49:17.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:17 smithi098 ceph-mon[39960]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T09:49:17.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:17 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:49:17.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:17 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:49:18.721 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:49:18.741 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi196:vg_nvme/lv_3 2024-09-21T09:49:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:18 smithi098 ceph-mon[29239]: purged_snaps scrub starts 2024-09-21T09:49:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:18 smithi098 ceph-mon[29239]: purged_snaps scrub ok 2024-09-21T09:49:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:18 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:49:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:18 smithi098 ceph-mon[29239]: osd.4 [v2:172.21.15.196:6800/158468112,v1:172.21.15.196:6801/158468112] boot 2024-09-21T09:49:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:18 smithi098 ceph-mon[29239]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T09:49:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:18 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:49:18.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:18 smithi098 ceph-mon[39960]: purged_snaps scrub starts 2024-09-21T09:49:18.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:18 smithi098 ceph-mon[39960]: purged_snaps scrub ok 2024-09-21T09:49:18.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:18 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:49:18.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:18 smithi098 ceph-mon[39960]: osd.4 [v2:172.21.15.196:6800/158468112,v1:172.21.15.196:6801/158468112] boot 2024-09-21T09:49:18.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:18 smithi098 ceph-mon[39960]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T09:49:18.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:18 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:49:18.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:18 smithi196 ceph-mon[32670]: purged_snaps scrub starts 2024-09-21T09:49:18.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:18 smithi196 ceph-mon[32670]: purged_snaps scrub ok 2024-09-21T09:49:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:18 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:49:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:18 smithi196 ceph-mon[32670]: osd.4 [v2:172.21.15.196:6800/158468112,v1:172.21.15.196:6801/158468112] boot 2024-09-21T09:49:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:18 smithi196 ceph-mon[32670]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T09:49:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:18 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:49:19.276 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:49:19.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:19 smithi098 ceph-mon[29239]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:19.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:19 smithi098 ceph-mon[29239]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T09:49:19.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:19 smithi098 ceph-mon[39960]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:19.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:19 smithi098 ceph-mon[39960]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T09:49:19.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:19 smithi196 ceph-mon[32670]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:49:19.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:19 smithi196 ceph-mon[32670]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T09:49:20.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:20 smithi098 ceph-mon[29239]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T09:49:20.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:20 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:20.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:20 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:20.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:20 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:49:20.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:20 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:20.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:20 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:20.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:20 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:20.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:20 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:20.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:20 smithi098 ceph-mon[39960]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T09:49:20.851 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:20 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:20.851 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:20 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:20.851 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:20 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:49:20.851 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:20 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:20.851 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:20 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:20.851 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:20 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:20.851 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:20 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:20.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:20 smithi196 ceph-mon[32670]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T09:49:20.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:20 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:20.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:20 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:20.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:20 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:49:20.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:20 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:20.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:20 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:20.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:20 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:20.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:20 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:21.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:21 smithi098 ceph-mon[29239]: Detected new or changed devices on smithi196 2024-09-21T09:49:21.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:21 smithi098 ceph-mon[29239]: Adjusting osd_memory_target on smithi196 to 17088M 2024-09-21T09:49:21.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:21 smithi098 ceph-mon[29239]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:49:21.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:21 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:21.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:21 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:21.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:21 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:21.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:21 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:21.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:21 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:49:21.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:21 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:49:21.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:21 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:21.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:21 smithi098 ceph-mon[39960]: Detected new or changed devices on smithi196 2024-09-21T09:49:21.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:21 smithi098 ceph-mon[39960]: Adjusting osd_memory_target on smithi196 to 17088M 2024-09-21T09:49:21.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:21 smithi098 ceph-mon[39960]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:49:21.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:21 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:21.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:21 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:21.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:21 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:21.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:21 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:21.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:21 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:49:21.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:21 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:49:21.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:21 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:21.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:21 smithi196 ceph-mon[32670]: Detected new or changed devices on smithi196 2024-09-21T09:49:21.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:21 smithi196 ceph-mon[32670]: Adjusting osd_memory_target on smithi196 to 17088M 2024-09-21T09:49:21.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:21 smithi196 ceph-mon[32670]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:49:21.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:21 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:21 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:21 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:21 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:21 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:49:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:21 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:49:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:21 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:22.804 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:22 smithi196 ceph-mon[32670]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:22 smithi098 ceph-mon[29239]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:22.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:22 smithi098 ceph-mon[39960]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:23.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:23 smithi098 ceph-mon[29239]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:49:23.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:23 smithi098 ceph-mon[39960]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:49:23.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:23 smithi196 ceph-mon[32670]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:49:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:24 smithi098 ceph-mon[29239]: from='client.? 172.21.15.196:0/1470038002' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cdcc44eb-e27f-4d23-9ecb-03bebdc82c96"}]: dispatch 2024-09-21T09:49:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:24 smithi098 ceph-mon[29239]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cdcc44eb-e27f-4d23-9ecb-03bebdc82c96"}]: dispatch 2024-09-21T09:49:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:24 smithi098 ceph-mon[29239]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cdcc44eb-e27f-4d23-9ecb-03bebdc82c96"}]': finished 2024-09-21T09:49:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:24 smithi098 ceph-mon[29239]: osdmap e34: 6 total, 5 up, 6 in 2024-09-21T09:49:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:24 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:49:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:24 smithi098 ceph-mon[29239]: from='client.? 172.21.15.196:0/3541704098' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:49:24.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:24 smithi098 ceph-mon[39960]: from='client.? 172.21.15.196:0/1470038002' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cdcc44eb-e27f-4d23-9ecb-03bebdc82c96"}]: dispatch 2024-09-21T09:49:24.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:24 smithi098 ceph-mon[39960]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cdcc44eb-e27f-4d23-9ecb-03bebdc82c96"}]: dispatch 2024-09-21T09:49:24.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:24 smithi098 ceph-mon[39960]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cdcc44eb-e27f-4d23-9ecb-03bebdc82c96"}]': finished 2024-09-21T09:49:24.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:24 smithi098 ceph-mon[39960]: osdmap e34: 6 total, 5 up, 6 in 2024-09-21T09:49:24.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:24 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:49:24.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:24 smithi098 ceph-mon[39960]: from='client.? 172.21.15.196:0/3541704098' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:49:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:24 smithi196 ceph-mon[32670]: from='client.? 172.21.15.196:0/1470038002' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cdcc44eb-e27f-4d23-9ecb-03bebdc82c96"}]: dispatch 2024-09-21T09:49:24.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:24 smithi196 ceph-mon[32670]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cdcc44eb-e27f-4d23-9ecb-03bebdc82c96"}]: dispatch 2024-09-21T09:49:24.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:24 smithi196 ceph-mon[32670]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cdcc44eb-e27f-4d23-9ecb-03bebdc82c96"}]': finished 2024-09-21T09:49:24.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:24 smithi196 ceph-mon[32670]: osdmap e34: 6 total, 5 up, 6 in 2024-09-21T09:49:24.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:24 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:49:24.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:24 smithi196 ceph-mon[32670]: from='client.? 172.21.15.196:0/3541704098' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:49:25.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:25 smithi098 ceph-mon[29239]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T09:49:25.968 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:25 smithi098 ceph-mon[39960]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T09:49:26.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:25 smithi196 ceph-mon[32670]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T09:49:27.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:27 smithi196 ceph-mon[32670]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T09:49:28.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:27 smithi098 ceph-mon[29239]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T09:49:28.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:27 smithi098 ceph-mon[39960]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T09:49:30.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:29 smithi098 ceph-mon[29239]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-09-21T09:49:30.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:29 smithi098 ceph-mon[39960]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-09-21T09:49:30.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:29 smithi196 ceph-mon[32670]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-09-21T09:49:30.805 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:30 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-21T09:49:30.805 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:30 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:30.805 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:30 smithi196 ceph-mon[32670]: Deploying daemon osd.5 on smithi196 2024-09-21T09:49:31.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:30 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-21T09:49:31.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:30 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:31.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:30 smithi098 ceph-mon[29239]: Deploying daemon osd.5 on smithi196 2024-09-21T09:49:31.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:30 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-21T09:49:31.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:30 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:31.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:30 smithi098 ceph-mon[39960]: Deploying daemon osd.5 on smithi196 2024-09-21T09:49:31.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:31 smithi196 ceph-mon[32670]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:49:32.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:31 smithi098 ceph-mon[29239]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:49:32.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:31 smithi098 ceph-mon[39960]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:49:34.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:33 smithi098 ceph-mon[29239]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:49:34.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:33 smithi098 ceph-mon[39960]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:49:34.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:33 smithi196 ceph-mon[32670]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:49:34.973 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:34 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:34.974 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:34 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:34.974 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:34 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:35.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:34 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:35.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:34 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:35.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:34 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:35.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:34 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:35.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:34 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:35.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:34 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:35.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:35 smithi098 ceph-mon[29239]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-21T09:49:35.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:35 smithi098 ceph-mon[39960]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-21T09:49:36.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:35 smithi196 ceph-mon[32670]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-21T09:49:37.759 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 5 on host 'smithi196' 2024-09-21T09:49:38.317 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:38 smithi196 ceph-mon[32670]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:49:38.317 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:38 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:38.317 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:38 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:38.317 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:38 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:38.317 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:38 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:38.317 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:38 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:38.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:38 smithi196 ceph-mon[32670]: from='osd.5 [v2:172.21.15.196:6808/2341878784,v1:172.21.15.196:6809/2341878784]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:49:38.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:38 smithi196 ceph-mon[32670]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:49:38.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:38 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:38.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:38 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:38.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:38 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:38.343 DEBUG:teuthology.orchestra.run.smithi196:osd.5> sudo journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.5.service 2024-09-21T09:49:38.345 INFO:tasks.cephadm:Deploying osd.6 on smithi196 with /dev/vg_nvme/lv_2... 2024-09-21T09:49:38.345 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T09:49:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:38 smithi098 ceph-mon[29239]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:49:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:38 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:38 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:38 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:38 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:38 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:38 smithi098 ceph-mon[29239]: from='osd.5 [v2:172.21.15.196:6808/2341878784,v1:172.21.15.196:6809/2341878784]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:49:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:38 smithi098 ceph-mon[29239]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:49:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:38 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:38 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:38 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:38.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:38 smithi098 ceph-mon[39960]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:49:38.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:38 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:38.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:38 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:38.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:38 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:38.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:38 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:38.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:38 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:38.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:38 smithi098 ceph-mon[39960]: from='osd.5 [v2:172.21.15.196:6808/2341878784,v1:172.21.15.196:6809/2341878784]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:49:38.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:38 smithi098 ceph-mon[39960]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:49:38.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:38 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:38.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:38 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:38.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:38 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:39.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:39 smithi098 ceph-mon[29239]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T09:49:39.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:39 smithi098 ceph-mon[29239]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T09:49:39.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:39 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:49:39.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:39 smithi098 ceph-mon[29239]: from='osd.5 [v2:172.21.15.196:6808/2341878784,v1:172.21.15.196:6809/2341878784]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:49:39.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:39 smithi098 ceph-mon[29239]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:49:39.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:39 smithi098 ceph-mon[39960]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T09:49:39.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:39 smithi098 ceph-mon[39960]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T09:49:39.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:39 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:49:39.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:39 smithi098 ceph-mon[39960]: from='osd.5 [v2:172.21.15.196:6808/2341878784,v1:172.21.15.196:6809/2341878784]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:49:39.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:39 smithi098 ceph-mon[39960]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:49:39.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:39 smithi196 ceph-mon[32670]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T09:49:39.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:39 smithi196 ceph-mon[32670]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T09:49:39.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:39 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:49:39.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:39 smithi196 ceph-mon[32670]: from='osd.5 [v2:172.21.15.196:6808/2341878784,v1:172.21.15.196:6809/2341878784]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:49:39.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:39 smithi196 ceph-mon[32670]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:49:39.389 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 09:49:39 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T09:49:39.091+0000 7f2b38d8c640 -1 osd.5 0 waiting for initial osdmap 2024-09-21T09:49:39.389 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 09:49:39 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T09:49:39.096+0000 7f2b34bc5640 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:49:39.842 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:49:40.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:40 smithi098 ceph-mon[29239]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:49:40.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:40 smithi098 ceph-mon[29239]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-21T09:49:40.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:40 smithi098 ceph-mon[29239]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T09:49:40.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:40 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:49:40.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:40 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:49:40.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:40 smithi098 ceph-mon[39960]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:49:40.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:40 smithi098 ceph-mon[39960]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-21T09:49:40.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:40 smithi098 ceph-mon[39960]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T09:49:40.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:40 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:49:40.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:40 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:49:40.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:40 smithi196 ceph-mon[32670]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:49:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:40 smithi196 ceph-mon[32670]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-21T09:49:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:40 smithi196 ceph-mon[32670]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T09:49:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:40 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:49:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:40 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:49:41.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:41 smithi196 ceph-mon[32670]: purged_snaps scrub starts 2024-09-21T09:49:41.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:41 smithi196 ceph-mon[32670]: purged_snaps scrub ok 2024-09-21T09:49:41.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:41 smithi196 ceph-mon[32670]: osd.5 [v2:172.21.15.196:6808/2341878784,v1:172.21.15.196:6809/2341878784] boot 2024-09-21T09:49:41.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:41 smithi196 ceph-mon[32670]: osdmap e37: 6 total, 6 up, 6 in 2024-09-21T09:49:41.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:41 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:49:41.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:41 smithi098 ceph-mon[29239]: purged_snaps scrub starts 2024-09-21T09:49:41.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:41 smithi098 ceph-mon[29239]: purged_snaps scrub ok 2024-09-21T09:49:41.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:41 smithi098 ceph-mon[29239]: osd.5 [v2:172.21.15.196:6808/2341878784,v1:172.21.15.196:6809/2341878784] boot 2024-09-21T09:49:41.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:41 smithi098 ceph-mon[29239]: osdmap e37: 6 total, 6 up, 6 in 2024-09-21T09:49:41.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:41 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:49:41.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:41 smithi098 ceph-mon[39960]: purged_snaps scrub starts 2024-09-21T09:49:41.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:41 smithi098 ceph-mon[39960]: purged_snaps scrub ok 2024-09-21T09:49:41.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:41 smithi098 ceph-mon[39960]: osd.5 [v2:172.21.15.196:6808/2341878784,v1:172.21.15.196:6809/2341878784] boot 2024-09-21T09:49:41.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:41 smithi098 ceph-mon[39960]: osdmap e37: 6 total, 6 up, 6 in 2024-09-21T09:49:41.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:41 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:49:42.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:42 smithi196 ceph-mon[32670]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:49:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:42 smithi196 ceph-mon[32670]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T09:49:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:42 smithi196 ceph-mon[32670]: Detected new or changed devices on smithi196 2024-09-21T09:49:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:42 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:42 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:42 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:49:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:42 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:49:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:42 smithi196 ceph-mon[32670]: Adjusting osd_memory_target on smithi196 to 8544M 2024-09-21T09:49:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:42 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:42 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:42 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:42 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:42.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[29239]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:49:42.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[29239]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T09:49:42.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[29239]: Detected new or changed devices on smithi196 2024-09-21T09:49:42.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:42.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:42.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:49:42.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:49:42.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[29239]: Adjusting osd_memory_target on smithi196 to 8544M 2024-09-21T09:49:42.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:42.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:42.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:42.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:42.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[39960]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:49:42.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[39960]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T09:49:42.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[39960]: Detected new or changed devices on smithi196 2024-09-21T09:49:42.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:42.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:42.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:49:42.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:49:42.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[39960]: Adjusting osd_memory_target on smithi196 to 8544M 2024-09-21T09:49:42.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:42.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:42.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:49:42.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:42 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:42.874 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:49:42.894 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi196:vg_nvme/lv_2 2024-09-21T09:49:43.132 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:49:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:43 smithi196 ceph-mon[32670]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T09:49:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:43 smithi196 ceph-mon[32670]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:49:43.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:43 smithi098 ceph-mon[29239]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T09:49:43.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:43 smithi098 ceph-mon[29239]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:49:43.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:43 smithi098 ceph-mon[39960]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T09:49:43.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:43 smithi098 ceph-mon[39960]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:49:44.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:44 smithi196 ceph-mon[32670]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T09:49:44.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:44 smithi098 ceph-mon[29239]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T09:49:44.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:44 smithi098 ceph-mon[39960]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T09:49:45.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:45 smithi196 ceph-mon[32670]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:49:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:45 smithi196 ceph-mon[32670]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:45 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:49:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:45 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:49:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:45 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:45.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:45 smithi098 ceph-mon[29239]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:49:45.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:45 smithi098 ceph-mon[29239]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:45.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:45 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:49:45.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:45 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:49:45.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:45 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:45.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:45 smithi098 ceph-mon[39960]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:49:45.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:45 smithi098 ceph-mon[39960]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:45.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:45 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:49:45.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:45 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:49:45.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:45 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:46.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:46 smithi196 ceph-mon[32670]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T09:49:46.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:46 smithi196 ceph-mon[32670]: Cluster is now healthy 2024-09-21T09:49:47.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:46 smithi098 ceph-mon[29239]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T09:49:47.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:46 smithi098 ceph-mon[29239]: Cluster is now healthy 2024-09-21T09:49:47.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:46 smithi098 ceph-mon[39960]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T09:49:47.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:46 smithi098 ceph-mon[39960]: Cluster is now healthy 2024-09-21T09:49:47.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:47 smithi196 ceph-mon[32670]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:49:47.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:47 smithi196 ceph-mon[32670]: from='client.? 172.21.15.196:0/1035357982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "566cfb44-1b21-45fc-b053-7205b1c10d4a"}]: dispatch 2024-09-21T09:49:47.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:47 smithi196 ceph-mon[32670]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "566cfb44-1b21-45fc-b053-7205b1c10d4a"}]: dispatch 2024-09-21T09:49:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:47 smithi196 ceph-mon[32670]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "566cfb44-1b21-45fc-b053-7205b1c10d4a"}]': finished 2024-09-21T09:49:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:47 smithi196 ceph-mon[32670]: osdmap e40: 7 total, 6 up, 7 in 2024-09-21T09:49:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:47 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:49:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:47 smithi196 ceph-mon[32670]: from='client.? 172.21.15.196:0/3882435676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:49:48.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:47 smithi098 ceph-mon[29239]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:49:48.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:47 smithi098 ceph-mon[29239]: from='client.? 172.21.15.196:0/1035357982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "566cfb44-1b21-45fc-b053-7205b1c10d4a"}]: dispatch 2024-09-21T09:49:48.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:47 smithi098 ceph-mon[29239]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "566cfb44-1b21-45fc-b053-7205b1c10d4a"}]: dispatch 2024-09-21T09:49:48.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:47 smithi098 ceph-mon[29239]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "566cfb44-1b21-45fc-b053-7205b1c10d4a"}]': finished 2024-09-21T09:49:48.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:47 smithi098 ceph-mon[29239]: osdmap e40: 7 total, 6 up, 7 in 2024-09-21T09:49:48.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:47 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:49:48.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:47 smithi098 ceph-mon[29239]: from='client.? 172.21.15.196:0/3882435676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:49:48.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:47 smithi098 ceph-mon[39960]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:49:48.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:47 smithi098 ceph-mon[39960]: from='client.? 172.21.15.196:0/1035357982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "566cfb44-1b21-45fc-b053-7205b1c10d4a"}]: dispatch 2024-09-21T09:49:48.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:47 smithi098 ceph-mon[39960]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "566cfb44-1b21-45fc-b053-7205b1c10d4a"}]: dispatch 2024-09-21T09:49:48.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:47 smithi098 ceph-mon[39960]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "566cfb44-1b21-45fc-b053-7205b1c10d4a"}]': finished 2024-09-21T09:49:48.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:47 smithi098 ceph-mon[39960]: osdmap e40: 7 total, 6 up, 7 in 2024-09-21T09:49:48.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:47 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:49:48.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:47 smithi098 ceph-mon[39960]: from='client.? 172.21.15.196:0/3882435676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:49:49.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:49 smithi196 ceph-mon[32670]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-09-21T09:49:50.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:49 smithi098 ceph-mon[29239]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-09-21T09:49:50.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:49 smithi098 ceph-mon[39960]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-09-21T09:49:51.769 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:51 smithi196 ceph-mon[32670]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-21T09:49:52.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:51 smithi098 ceph-mon[39960]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-21T09:49:52.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:51 smithi098 ceph-mon[29239]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-21T09:49:53.761 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:53 smithi196 ceph-mon[32670]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:49:53.761 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:53 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-21T09:49:53.762 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:53 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:53.762 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:53 smithi196 ceph-mon[32670]: Deploying daemon osd.6 on smithi196 2024-09-21T09:49:54.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:53 smithi098 ceph-mon[39960]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:49:54.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:53 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-21T09:49:54.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:53 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:54.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:53 smithi098 ceph-mon[39960]: Deploying daemon osd.6 on smithi196 2024-09-21T09:49:54.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:53 smithi098 ceph-mon[29239]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:49:54.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:53 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-21T09:49:54.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:53 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:49:54.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:53 smithi098 ceph-mon[29239]: Deploying daemon osd.6 on smithi196 2024-09-21T09:49:55.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:55 smithi196 ceph-mon[32670]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:49:55.899 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:55 smithi098 ceph-mon[29239]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:49:55.899 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:55 smithi098 ceph-mon[39960]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:49:57.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:57 smithi196 ceph-mon[32670]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:49:57.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:57 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:57.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:57 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:57.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:57 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:58.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:57 smithi098 ceph-mon[39960]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:49:58.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:57 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:58.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:57 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:58.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:57 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:58.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:57 smithi098 ceph-mon[29239]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:49:58.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:57 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:49:58.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:57 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:58.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:57 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:49:59.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:49:59 smithi196 ceph-mon[32670]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:50:00.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:49:59 smithi098 ceph-mon[39960]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:50:00.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:49:59 smithi098 ceph-mon[29239]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:50:00.697 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 6 on host 'smithi196' 2024-09-21T09:50:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:00 smithi196 ceph-mon[32670]: overall HEALTH_OK 2024-09-21T09:50:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:00 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:00 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:00 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:00 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:00 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:00 smithi196 ceph-mon[32670]: from='osd.6 [v2:172.21.15.196:6816/3059702155,v1:172.21.15.196:6817/3059702155]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:50:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:00 smithi196 ceph-mon[32670]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:50:01.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:00 smithi098 ceph-mon[29239]: overall HEALTH_OK 2024-09-21T09:50:01.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:00 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:01.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:00 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:01.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:00 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:01.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:00 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:01.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:00 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:01.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:00 smithi098 ceph-mon[29239]: from='osd.6 [v2:172.21.15.196:6816/3059702155,v1:172.21.15.196:6817/3059702155]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:50:01.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:00 smithi098 ceph-mon[29239]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:50:01.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:00 smithi098 ceph-mon[39960]: overall HEALTH_OK 2024-09-21T09:50:01.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:00 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:01.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:00 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:01.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:00 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:01.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:00 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:01.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:00 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:01.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:00 smithi098 ceph-mon[39960]: from='osd.6 [v2:172.21.15.196:6816/3059702155,v1:172.21.15.196:6817/3059702155]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:50:01.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:00 smithi098 ceph-mon[39960]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:50:01.363 DEBUG:teuthology.orchestra.run.smithi196:osd.6> sudo journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.6.service 2024-09-21T09:50:01.365 INFO:tasks.cephadm:Deploying osd.7 on smithi196 with /dev/vg_nvme/lv_1... 2024-09-21T09:50:01.365 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T09:50:01.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:01 smithi196 ceph-mon[32670]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:50:01.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:01 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:01 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:01 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:01 smithi196 ceph-mon[32670]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T09:50:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:01 smithi196 ceph-mon[32670]: osdmap e41: 7 total, 6 up, 7 in 2024-09-21T09:50:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:01 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:50:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:01 smithi196 ceph-mon[32670]: from='osd.6 [v2:172.21.15.196:6816/3059702155,v1:172.21.15.196:6817/3059702155]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:50:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:01 smithi196 ceph-mon[32670]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:50:02.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:01 smithi098 ceph-mon[29239]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:50:02.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:01 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:02.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:01 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:02.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:01 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:02.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:01 smithi098 ceph-mon[29239]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T09:50:02.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:01 smithi098 ceph-mon[29239]: osdmap e41: 7 total, 6 up, 7 in 2024-09-21T09:50:02.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:01 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:50:02.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:01 smithi098 ceph-mon[29239]: from='osd.6 [v2:172.21.15.196:6816/3059702155,v1:172.21.15.196:6817/3059702155]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:50:02.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:01 smithi098 ceph-mon[29239]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:50:02.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:01 smithi098 ceph-mon[39960]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:50:02.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:01 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:02.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:01 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:02.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:01 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:02.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:01 smithi098 ceph-mon[39960]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T09:50:02.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:01 smithi098 ceph-mon[39960]: osdmap e41: 7 total, 6 up, 7 in 2024-09-21T09:50:02.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:01 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:50:02.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:01 smithi098 ceph-mon[39960]: from='osd.6 [v2:172.21.15.196:6816/3059702155,v1:172.21.15.196:6817/3059702155]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:50:02.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:01 smithi098 ceph-mon[39960]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:50:02.388 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 09:50:02 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T09:50:02.028+0000 7f1861881640 -1 osd.6 0 waiting for initial osdmap 2024-09-21T09:50:02.389 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 09:50:02 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T09:50:02.033+0000 7f185aeb5640 -1 osd.6 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:50:02.496 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:50:03.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:03 smithi196 ceph-mon[32670]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-21T09:50:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:03 smithi196 ceph-mon[32670]: osdmap e42: 7 total, 6 up, 7 in 2024-09-21T09:50:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:03 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:50:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:03 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:50:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:03 smithi098 ceph-mon[29239]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-21T09:50:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:03 smithi098 ceph-mon[29239]: osdmap e42: 7 total, 6 up, 7 in 2024-09-21T09:50:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:03 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:50:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:03 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:50:03.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:03 smithi098 ceph-mon[39960]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-21T09:50:03.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:03 smithi098 ceph-mon[39960]: osdmap e42: 7 total, 6 up, 7 in 2024-09-21T09:50:03.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:03 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:50:03.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:03 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:50:04.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:04 smithi098 ceph-mon[29239]: purged_snaps scrub starts 2024-09-21T09:50:04.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:04 smithi098 ceph-mon[29239]: purged_snaps scrub ok 2024-09-21T09:50:04.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:04 smithi098 ceph-mon[29239]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:50:04.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:04 smithi098 ceph-mon[29239]: osd.6 [v2:172.21.15.196:6816/3059702155,v1:172.21.15.196:6817/3059702155] boot 2024-09-21T09:50:04.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:04 smithi098 ceph-mon[29239]: osdmap e43: 7 total, 7 up, 7 in 2024-09-21T09:50:04.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:04 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:50:04.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:04 smithi098 ceph-mon[29239]: osdmap e44: 7 total, 7 up, 7 in 2024-09-21T09:50:04.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:04 smithi098 ceph-mon[39960]: purged_snaps scrub starts 2024-09-21T09:50:04.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:04 smithi098 ceph-mon[39960]: purged_snaps scrub ok 2024-09-21T09:50:04.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:04 smithi098 ceph-mon[39960]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:50:04.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:04 smithi098 ceph-mon[39960]: osd.6 [v2:172.21.15.196:6816/3059702155,v1:172.21.15.196:6817/3059702155] boot 2024-09-21T09:50:04.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:04 smithi098 ceph-mon[39960]: osdmap e43: 7 total, 7 up, 7 in 2024-09-21T09:50:04.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:04 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:50:04.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:04 smithi098 ceph-mon[39960]: osdmap e44: 7 total, 7 up, 7 in 2024-09-21T09:50:04.367 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:04 smithi196 ceph-mon[32670]: purged_snaps scrub starts 2024-09-21T09:50:04.367 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:04 smithi196 ceph-mon[32670]: purged_snaps scrub ok 2024-09-21T09:50:04.367 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:04 smithi196 ceph-mon[32670]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:50:04.367 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:04 smithi196 ceph-mon[32670]: osd.6 [v2:172.21.15.196:6816/3059702155,v1:172.21.15.196:6817/3059702155] boot 2024-09-21T09:50:04.368 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:04 smithi196 ceph-mon[32670]: osdmap e43: 7 total, 7 up, 7 in 2024-09-21T09:50:04.368 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:04 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:50:04.368 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:04 smithi196 ceph-mon[32670]: osdmap e44: 7 total, 7 up, 7 in 2024-09-21T09:50:06.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:05 smithi196 ceph-mon[32670]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:06.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:05 smithi196 ceph-mon[32670]: osdmap e45: 7 total, 7 up, 7 in 2024-09-21T09:50:06.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:05 smithi196 ceph-mon[32670]: Detected new or changed devices on smithi196 2024-09-21T09:50:06.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:05 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:06.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:05 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:06.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:05 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:06.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:05 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:06.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:05 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:06.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:05 smithi196 ceph-mon[32670]: Adjusting osd_memory_target on smithi196 to 5696M 2024-09-21T09:50:06.004 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:05 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:06.004 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:05 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:06.004 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:05 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:06.004 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:05 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:06.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[29239]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:06.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[29239]: osdmap e45: 7 total, 7 up, 7 in 2024-09-21T09:50:06.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[29239]: Detected new or changed devices on smithi196 2024-09-21T09:50:06.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:06.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:06.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:06.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:06.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:06.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[29239]: Adjusting osd_memory_target on smithi196 to 5696M 2024-09-21T09:50:06.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:06.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:06.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:06.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:06.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[39960]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:06.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[39960]: osdmap e45: 7 total, 7 up, 7 in 2024-09-21T09:50:06.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[39960]: Detected new or changed devices on smithi196 2024-09-21T09:50:06.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:06.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:06.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:06.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:06.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:06.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[39960]: Adjusting osd_memory_target on smithi196 to 5696M 2024-09-21T09:50:06.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:06.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:06.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:06.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:05 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:06.563 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-21T09:50:06.584 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi196:vg_nvme/lv_1 2024-09-21T09:50:06.825 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:50:07.843 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:07 smithi196 ceph-mon[32670]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:08.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:07 smithi098 ceph-mon[29239]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:08.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:07 smithi098 ceph-mon[39960]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:09.004 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:08 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:50:09.004 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:08 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:50:09.004 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:08 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:09.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:08 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:50:09.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:08 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:50:09.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:08 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:09.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:08 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:50:09.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:08 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:50:09.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:08 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:09.832 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:09 smithi196 ceph-mon[32670]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:09.832 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:09 smithi196 ceph-mon[32670]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:10.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:09 smithi098 ceph-mon[29239]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:10.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:09 smithi098 ceph-mon[29239]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:10.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:09 smithi098 ceph-mon[39960]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:10.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:09 smithi098 ceph-mon[39960]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:11.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:10 smithi098 ceph-mon[29239]: from='client.? 172.21.15.196:0/2715389873' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "228404c1-3910-4933-b023-55aa8b01aee4"}]: dispatch 2024-09-21T09:50:11.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:10 smithi098 ceph-mon[29239]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "228404c1-3910-4933-b023-55aa8b01aee4"}]: dispatch 2024-09-21T09:50:11.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:10 smithi098 ceph-mon[29239]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "228404c1-3910-4933-b023-55aa8b01aee4"}]': finished 2024-09-21T09:50:11.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:10 smithi098 ceph-mon[29239]: osdmap e46: 8 total, 7 up, 8 in 2024-09-21T09:50:11.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:10 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:50:11.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:10 smithi098 ceph-mon[39960]: from='client.? 172.21.15.196:0/2715389873' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "228404c1-3910-4933-b023-55aa8b01aee4"}]: dispatch 2024-09-21T09:50:11.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:10 smithi098 ceph-mon[39960]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "228404c1-3910-4933-b023-55aa8b01aee4"}]: dispatch 2024-09-21T09:50:11.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:10 smithi098 ceph-mon[39960]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "228404c1-3910-4933-b023-55aa8b01aee4"}]': finished 2024-09-21T09:50:11.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:10 smithi098 ceph-mon[39960]: osdmap e46: 8 total, 7 up, 8 in 2024-09-21T09:50:11.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:10 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:50:11.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:10 smithi196 ceph-mon[32670]: from='client.? 172.21.15.196:0/2715389873' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "228404c1-3910-4933-b023-55aa8b01aee4"}]: dispatch 2024-09-21T09:50:11.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:10 smithi196 ceph-mon[32670]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "228404c1-3910-4933-b023-55aa8b01aee4"}]: dispatch 2024-09-21T09:50:11.118 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:10 smithi196 ceph-mon[32670]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "228404c1-3910-4933-b023-55aa8b01aee4"}]': finished 2024-09-21T09:50:11.118 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:10 smithi196 ceph-mon[32670]: osdmap e46: 8 total, 7 up, 8 in 2024-09-21T09:50:11.118 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:10 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:50:12.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:11 smithi098 ceph-mon[29239]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:12.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:11 smithi098 ceph-mon[29239]: from='client.? 172.21.15.196:0/2022666111' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:50:12.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:11 smithi098 ceph-mon[39960]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:12.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:11 smithi098 ceph-mon[39960]: from='client.? 172.21.15.196:0/2022666111' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:50:12.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:11 smithi196 ceph-mon[32670]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:11 smithi196 ceph-mon[32670]: from='client.? 172.21.15.196:0/2022666111' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:50:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:13 smithi196 ceph-mon[32670]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:14.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:13 smithi098 ceph-mon[29239]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:14.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:13 smithi098 ceph-mon[39960]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:15.949 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:15 smithi196 ceph-mon[32670]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:16.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:15 smithi098 ceph-mon[29239]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:16.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:15 smithi098 ceph-mon[39960]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:17.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:17 smithi196 ceph-mon[32670]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:17.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:17 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-21T09:50:17.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:17 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:17 smithi196 ceph-mon[32670]: Deploying daemon osd.7 on smithi196 2024-09-21T09:50:18.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:17 smithi098 ceph-mon[29239]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:18.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:17 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-21T09:50:18.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:17 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:18.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:17 smithi098 ceph-mon[29239]: Deploying daemon osd.7 on smithi196 2024-09-21T09:50:18.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:17 smithi098 ceph-mon[39960]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:18.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:17 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-21T09:50:18.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:17 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:18.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:17 smithi098 ceph-mon[39960]: Deploying daemon osd.7 on smithi196 2024-09-21T09:50:20.056 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:19 smithi196 ceph-mon[32670]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:20.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:19 smithi098 ceph-mon[39960]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:20.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:19 smithi098 ceph-mon[29239]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:21.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:21 smithi196 ceph-mon[32670]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:22.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:21 smithi098 ceph-mon[29239]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:22.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:21 smithi098 ceph-mon[39960]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:22 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:22 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:22 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:22 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:22 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:22 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:23.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:22 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:23.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:22 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:23.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:22 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:24.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:23 smithi196 ceph-mon[32670]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:24.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:23 smithi098 ceph-mon[29239]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:24.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:23 smithi098 ceph-mon[39960]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:25.162 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 7 on host 'smithi196' 2024-09-21T09:50:25.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:25 smithi196 ceph-mon[32670]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:25.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:25 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:25.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:25 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:25.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:25 smithi196 ceph-mon[32670]: from='osd.7 [v2:172.21.15.196:6824/2155020234,v1:172.21.15.196:6825/2155020234]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:50:25.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:25 smithi196 ceph-mon[32670]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:50:25.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:25 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:25.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:25 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:25.762 DEBUG:teuthology.orchestra.run.smithi196:osd.7> sudo journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.7.service 2024-09-21T09:50:25.764 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-21T09:50:25.764 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd stat -f json 2024-09-21T09:50:25.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:25 smithi098 ceph-mon[29239]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:25.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:25 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:25.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:25 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:25.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:25 smithi098 ceph-mon[29239]: from='osd.7 [v2:172.21.15.196:6824/2155020234,v1:172.21.15.196:6825/2155020234]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:50:25.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:25 smithi098 ceph-mon[29239]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:50:25.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:25 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:25.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:25 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:25.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:25 smithi098 ceph-mon[39960]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:25.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:25 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:25.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:25 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:25.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:25 smithi098 ceph-mon[39960]: from='osd.7 [v2:172.21.15.196:6824/2155020234,v1:172.21.15.196:6825/2155020234]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:50:25.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:25 smithi098 ceph-mon[39960]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:50:25.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:25 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:25.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:25 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:26.022 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:50:27.009 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:26 smithi196 ceph-mon[32670]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T09:50:27.009 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:26 smithi196 ceph-mon[32670]: osdmap e47: 8 total, 7 up, 8 in 2024-09-21T09:50:27.009 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:26 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:50:27.009 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:26 smithi196 ceph-mon[32670]: from='osd.7 [v2:172.21.15.196:6824/2155020234,v1:172.21.15.196:6825/2155020234]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:50:27.009 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:26 smithi196 ceph-mon[32670]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:50:27.010 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 09:50:26 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T09:50:26.913+0000 7efe63fc4640 -1 osd.7 0 waiting for initial osdmap 2024-09-21T09:50:27.010 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 09:50:26 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T09:50:26.919+0000 7efe605fe640 -1 osd.7 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:50:27.174 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:26 smithi098 ceph-mon[29239]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T09:50:27.174 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:26 smithi098 ceph-mon[29239]: osdmap e47: 8 total, 7 up, 8 in 2024-09-21T09:50:27.174 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:26 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:50:27.174 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:26 smithi098 ceph-mon[29239]: from='osd.7 [v2:172.21.15.196:6824/2155020234,v1:172.21.15.196:6825/2155020234]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:50:27.174 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:26 smithi098 ceph-mon[29239]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:50:27.175 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:26 smithi098 ceph-mon[39960]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T09:50:27.175 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:26 smithi098 ceph-mon[39960]: osdmap e47: 8 total, 7 up, 8 in 2024-09-21T09:50:27.175 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:26 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:50:27.175 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:26 smithi098 ceph-mon[39960]: from='osd.7 [v2:172.21.15.196:6824/2155020234,v1:172.21.15.196:6825/2155020234]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:50:27.175 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:26 smithi098 ceph-mon[39960]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-21T09:50:27.822 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:50:28.017 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:27 smithi196 ceph-mon[32670]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:28.017 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:27 smithi196 ceph-mon[32670]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-21T09:50:28.017 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:27 smithi196 ceph-mon[32670]: osdmap e48: 8 total, 7 up, 8 in 2024-09-21T09:50:28.017 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:27 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:50:28.017 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:27 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:50:28.017 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:27 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2280051801' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T09:50:28.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:27 smithi098 ceph-mon[29239]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:28.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:27 smithi098 ceph-mon[29239]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-21T09:50:28.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:27 smithi098 ceph-mon[29239]: osdmap e48: 8 total, 7 up, 8 in 2024-09-21T09:50:28.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:27 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:50:28.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:27 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:50:28.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:27 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2280051801' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T09:50:28.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:27 smithi098 ceph-mon[39960]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:50:28.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:27 smithi098 ceph-mon[39960]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-21T09:50:28.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:27 smithi098 ceph-mon[39960]: osdmap e48: 8 total, 7 up, 8 in 2024-09-21T09:50:28.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:27 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:50:28.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:27 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:50:28.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:27 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2280051801' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T09:50:28.556 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":48,"num_osds":8,"num_up_osds":7,"osd_up_since":1726912203,"num_in_osds":8,"osd_in_since":1726912210,"num_remapped_pgs":0} 2024-09-21T09:50:29.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[29239]: purged_snaps scrub starts 2024-09-21T09:50:29.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[29239]: purged_snaps scrub ok 2024-09-21T09:50:29.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[29239]: osd.7 [v2:172.21.15.196:6824/2155020234,v1:172.21.15.196:6825/2155020234] boot 2024-09-21T09:50:29.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[29239]: osdmap e49: 8 total, 8 up, 8 in 2024-09-21T09:50:29.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:50:29.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:29.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:29.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:29.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:29.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:29.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:29.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:29.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:29.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:29.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:29.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:29.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:29.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:29.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:29.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[29239]: osdmap e50: 8 total, 8 up, 8 in 2024-09-21T09:50:29.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[39960]: purged_snaps scrub starts 2024-09-21T09:50:29.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[39960]: purged_snaps scrub ok 2024-09-21T09:50:29.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[39960]: osd.7 [v2:172.21.15.196:6824/2155020234,v1:172.21.15.196:6825/2155020234] boot 2024-09-21T09:50:29.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[39960]: osdmap e49: 8 total, 8 up, 8 in 2024-09-21T09:50:29.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:50:29.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:29.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:29.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:29.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:29.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:29.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:29.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:29.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:29.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:29.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:29.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:29.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:29.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:29.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:29.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:28 smithi098 ceph-mon[39960]: osdmap e50: 8 total, 8 up, 8 in 2024-09-21T09:50:29.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:28 smithi196 ceph-mon[32670]: purged_snaps scrub starts 2024-09-21T09:50:29.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:28 smithi196 ceph-mon[32670]: purged_snaps scrub ok 2024-09-21T09:50:29.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:28 smithi196 ceph-mon[32670]: osd.7 [v2:172.21.15.196:6824/2155020234,v1:172.21.15.196:6825/2155020234] boot 2024-09-21T09:50:29.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:28 smithi196 ceph-mon[32670]: osdmap e49: 8 total, 8 up, 8 in 2024-09-21T09:50:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:28 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:50:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:28 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:28 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:28 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:28 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:28 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:28 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:50:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:28 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:28 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:28 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:28 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:28 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:28 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:28 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:28 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:28 smithi196 ceph-mon[32670]: osdmap e50: 8 total, 8 up, 8 in 2024-09-21T09:50:29.558 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd stat -f json 2024-09-21T09:50:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:50:30.186 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:29 smithi098 ceph-mon[39960]: Detected new or changed devices on smithi196 2024-09-21T09:50:30.186 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:29 smithi098 ceph-mon[39960]: Adjusting osd_memory_target on smithi196 to 4272M 2024-09-21T09:50:30.186 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:29 smithi098 ceph-mon[39960]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:30.186 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:29 smithi098 ceph-mon[39960]: osdmap e51: 8 total, 8 up, 8 in 2024-09-21T09:50:30.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:29 smithi098 ceph-mon[29239]: Detected new or changed devices on smithi196 2024-09-21T09:50:30.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:29 smithi098 ceph-mon[29239]: Adjusting osd_memory_target on smithi196 to 4272M 2024-09-21T09:50:30.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:29 smithi098 ceph-mon[29239]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:30.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:29 smithi098 ceph-mon[29239]: osdmap e51: 8 total, 8 up, 8 in 2024-09-21T09:50:30.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:29 smithi196 ceph-mon[32670]: Detected new or changed devices on smithi196 2024-09-21T09:50:30.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:29 smithi196 ceph-mon[32670]: Adjusting osd_memory_target on smithi196 to 4272M 2024-09-21T09:50:30.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:29 smithi196 ceph-mon[32670]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:30.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:29 smithi196 ceph-mon[32670]: osdmap e51: 8 total, 8 up, 8 in 2024-09-21T09:50:31.706 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:50:32.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:31 smithi098 ceph-mon[29239]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:32.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:31 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/1105783596' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T09:50:32.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:31 smithi098 ceph-mon[39960]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:32.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:31 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/1105783596' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T09:50:32.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:31 smithi196 ceph-mon[32670]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:31 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/1105783596' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T09:50:32.400 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":51,"num_osds":8,"num_up_osds":8,"osd_up_since":1726912227,"num_in_osds":8,"osd_in_since":1726912210,"num_remapped_pgs":0} 2024-09-21T09:50:32.401 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T09:50:32.644 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:50:33.982 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:33 smithi098 ceph-mon[29239]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:33.982 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:33 smithi098 ceph-mon[39960]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:34.191 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:33 smithi196 ceph-mon[32670]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:34.330 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:50:34.330 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":51,"fsid":"26a15512-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:45:33.813487+0000","modified":"2024-09-21T09:50:29.745690+0000","last_up_change":"2024-09-21T09:50:27.903155+0000","last_in_change":"2024-09-21T09:50:10.725457+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:48:34.781592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"579f13de-565d-4801-b65e-f27d1cde297a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":506591636},{"type":"v1","addr":"172.21.15.98:6803","nonce":506591636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":506591636},{"type":"v1","addr":"172.21.15.98:6805","nonce":506591636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":506591636},{"type":"v1","addr":"172.21.15.98:6809","nonce":506591636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":506591636},{"type":"v1","addr":"172.21.15.98:6807","nonce":506591636}]},"public_addr":"172.21.15.98:6803/506591636","cluster_addr":"172.21.15.98:6805/506591636","heartbeat_back_addr":"172.21.15.98:6809/506591636","heartbeat_front_addr":"172.21.15.98:6807/506591636","state":["exists","up"]},{"osd":1,"uuid":"6d31ac4e-dc1a-4051-8266-ba821106242f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":4240601629},{"type":"v1","addr":"172.21.15.98:6811","nonce":4240601629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":4240601629},{"type":"v1","addr":"172.21.15.98:6813","nonce":4240601629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4240601629},{"type":"v1","addr":"172.21.15.98:6817","nonce":4240601629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":4240601629},{"type":"v1","addr":"172.21.15.98:6815","nonce":4240601629}]},"public_addr":"172.21.15.98:6811/4240601629","cluster_addr":"172.21.15.98:6813/4240601629","heartbeat_back_addr":"172.21.15.98:6817/4240601629","heartbeat_front_addr":"172.21.15.98:6815/4240601629","state":["exists","up"]},{"osd":2,"uuid":"62e22789-74be-4098-a9b9-62671fb4cac1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2917126560},{"type":"v1","addr":"172.21.15.98:6819","nonce":2917126560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2917126560},{"type":"v1","addr":"172.21.15.98:6821","nonce":2917126560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2917126560},{"type":"v1","addr":"172.21.15.98:6825","nonce":2917126560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2917126560},{"type":"v1","addr":"172.21.15.98:6823","nonce":2917126560}]},"public_addr":"172.21.15.98:6819/2917126560","cluster_addr":"172.21.15.98:6821/2917126560","heartbeat_back_addr":"172.21.15.98:6825/2917126560","heartbeat_front_addr":"172.21.15.98:6823/2917126560","state":["exists","up"]},{"osd":3,"uuid":"77f653a6-69b0-437a-960d-c3fb036708be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":4022556179},{"type":"v1","addr":"172.21.15.98:6827","nonce":4022556179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":4022556179},{"type":"v1","addr":"172.21.15.98:6829","nonce":4022556179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4022556179},{"type":"v1","addr":"172.21.15.98:6833","nonce":4022556179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":4022556179},{"type":"v1","addr":"172.21.15.98:6831","nonce":4022556179}]},"public_addr":"172.21.15.98:6827/4022556179","cluster_addr":"172.21.15.98:6829/4022556179","heartbeat_back_addr":"172.21.15.98:6833/4022556179","heartbeat_front_addr":"172.21.15.98:6831/4022556179","state":["exists","up"]},{"osd":4,"uuid":"28ec4928-7f00-46a9-b8c5-95f0b7e6be75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6801","nonce":158468112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6803","nonce":158468112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6807","nonce":158468112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6805","nonce":158468112}]},"public_addr":"172.21.15.196:6801/158468112","cluster_addr":"172.21.15.196:6803/158468112","heartbeat_back_addr":"172.21.15.196:6807/158468112","heartbeat_front_addr":"172.21.15.196:6805/158468112","state":["exists","up"]},{"osd":5,"uuid":"cdcc44eb-e27f-4d23-9ecb-03bebdc82c96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6809","nonce":2341878784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6811","nonce":2341878784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6815","nonce":2341878784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6813","nonce":2341878784}]},"public_addr":"172.21.15.196:6809/2341878784","cluster_addr":"172.21.15.196:6811/2341878784","heartbeat_back_addr":"172.21.15.196:6815/2341878784","heartbeat_front_addr":"172.21.15.196:6813/2341878784","state":["exists","up"]},{"osd":6,"uuid":"566cfb44-1b21-45fc-b053-7205b1c10d4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6817","nonce":3059702155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6819","nonce":3059702155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6823","nonce":3059702155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6821","nonce":3059702155}]},"public_addr":"172.21.15.196:6817/3059702155","cluster_addr":"172.21.15.196:6819/3059702155","heartbeat_back_addr":"172.21.15.196:6823/3059702155","heartbeat_front_addr":"172.21.15.196:6821/3059702155","state":["exists","up"]},{"osd":7,"uuid":"228404c1-3910-4933-b023-55aa8b01aee4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6825","nonce":2155020234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6827","nonce":2155020234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6831","nonce":2155020234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6829","nonce":2155020234}]},"public_addr":"172.21.15.196:6825/2155020234","cluster_addr":"172.21.15.196:6827/2155020234","heartbeat_back_addr":"172.21.15.196:6831/2155020234","heartbeat_front_addr":"172.21.15.196:6829/2155020234","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:47:43.263977+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:06.738746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:30.883578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:54.216866+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:49:15.599276+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:49:38.581033+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:01.504502+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:25.964220+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/227493886":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/3737889208":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/262018661":"2024-09-22T09:46:24.467967+0000","172.21.15.98:6801/440385038":"2024-09-22T09:46:24.467967+0000","172.21.15.98:6800/440385038":"2024-09-22T09:46:24.467967+0000","172.21.15.98:6801/1427334238":"2024-09-22T09:45:54.237657+0000","172.21.15.98:6800/1427334238":"2024-09-22T09:45:54.237657+0000","172.21.15.98:0/3656555554":"2024-09-22T09:45:54.237657+0000","172.21.15.98:0/2075804383":"2024-09-22T09:45:54.237657+0000","172.21.15.98:0/2144473621":"2024-09-22T09:45:54.237657+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T09:50:35.039 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T09:48:34.781592+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-09-21T09:50:35.039 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool get .mgr pg_num 2024-09-21T09:50:35.210 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:34 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3954729366' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:50:35.285 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:50:35.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:34 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3954729366' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:50:35.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:34 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3954729366' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:50:36.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:35 smithi098 ceph-mon[29239]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:36.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:35 smithi098 ceph-mon[39960]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:36.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:35 smithi196 ceph-mon[32670]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:37.112 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-09-21T09:50:37.863 INFO:tasks.cephadm:Adding prometheus.a on smithi196 2024-09-21T09:50:37.864 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph orch apply prometheus '1;smithi196=a' 2024-09-21T09:50:38.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:37 smithi098 ceph-mon[29239]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:38.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:37 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3107209256' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-21T09:50:38.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:37 smithi098 ceph-mon[39960]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:38.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:37 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3107209256' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-21T09:50:38.119 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:50:38.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:37 smithi196 ceph-mon[32670]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:38.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:37 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3107209256' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-21T09:50:39.996 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled prometheus update... 2024-09-21T09:50:40.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:39 smithi098 ceph-mon[29239]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:40.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:39 smithi098 ceph-mon[39960]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:40.362 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:39 smithi196 ceph-mon[32670]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:40.646 DEBUG:teuthology.orchestra.run.smithi196:prometheus.a> sudo journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@prometheus.a.service 2024-09-21T09:50:40.649 INFO:tasks.cephadm:Adding node-exporter.a on smithi098 2024-09-21T09:50:40.650 INFO:tasks.cephadm:Adding node-exporter.b on smithi196 2024-09-21T09:50:40.650 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph orch apply node-exporter '2;smithi098=a;smithi196=b' 2024-09-21T09:50:40.935 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:50:41.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:40 smithi196 ceph-mon[32670]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:41.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:40 smithi196 ceph-mon[32670]: Saving service prometheus spec with placement smithi196=a;count:1 2024-09-21T09:50:41.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:40 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:41.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:40 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:41.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:40 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:41.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:40 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:41.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:40 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:41.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:40 smithi196 ceph-mon[32670]: Deploying daemon prometheus.a on smithi196 2024-09-21T09:50:41.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:40 smithi098 ceph-mon[29239]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:41.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:40 smithi098 ceph-mon[29239]: Saving service prometheus spec with placement smithi196=a;count:1 2024-09-21T09:50:41.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:40 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:41.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:40 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:41.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:40 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:41.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:40 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:41.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:40 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:41.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:40 smithi098 ceph-mon[29239]: Deploying daemon prometheus.a on smithi196 2024-09-21T09:50:41.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:40 smithi098 ceph-mon[39960]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:41.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:40 smithi098 ceph-mon[39960]: Saving service prometheus spec with placement smithi196=a;count:1 2024-09-21T09:50:41.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:40 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:41.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:40 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:41.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:40 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:41.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:40 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:50:41.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:40 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:41.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:40 smithi098 ceph-mon[39960]: Deploying daemon prometheus.a on smithi196 2024-09-21T09:50:42.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:42 smithi196 ceph-mon[32670]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:42.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:42 smithi098 ceph-mon[29239]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:42.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:42 smithi098 ceph-mon[39960]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:43.132 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled node-exporter update... 2024-09-21T09:50:44.104 DEBUG:teuthology.orchestra.run.smithi098:node-exporter.a> sudo journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@node-exporter.a.service 2024-09-21T09:50:44.108 DEBUG:teuthology.orchestra.run.smithi196:node-exporter.b> sudo journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@node-exporter.b.service 2024-09-21T09:50:44.111 INFO:tasks.cephadm:Adding alertmanager.a on smithi098 2024-09-21T09:50:44.111 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph orch apply alertmanager '1;smithi098=a' 2024-09-21T09:50:44.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:44 smithi098 ceph-mon[29239]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:44.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:44 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:44.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:44 smithi098 ceph-mon[39960]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:44.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:44 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:44.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:44 smithi196 ceph-mon[32670]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:44.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:44 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:44.474 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:50:45.244 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:45 smithi196 ceph-mon[32670]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi098=a;smithi196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:45.244 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:45 smithi196 ceph-mon[32670]: Saving service node-exporter spec with placement smithi098=a;smithi196=b;count:2 2024-09-21T09:50:45.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:45 smithi098 ceph-mon[29239]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi098=a;smithi196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:45.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:45 smithi098 ceph-mon[29239]: Saving service node-exporter spec with placement smithi098=a;smithi196=b;count:2 2024-09-21T09:50:45.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:45 smithi098 ceph-mon[39960]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi098=a;smithi196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:45.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:45 smithi098 ceph-mon[39960]: Saving service node-exporter spec with placement smithi098=a;smithi196=b;count:2 2024-09-21T09:50:46.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:46 smithi098 ceph-mon[29239]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:46.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:46 smithi098 ceph-mon[39960]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:46.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:46 smithi196 ceph-mon[32670]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:48.343 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:48 smithi196 ceph-mon[32670]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:48.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:48 smithi098 ceph-mon[29239]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:48.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:48 smithi098 ceph-mon[39960]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:48.780 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled alertmanager update... 2024-09-21T09:50:50.067 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:49 smithi196 ceph-mon[32670]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:50.067 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:49 smithi196 ceph-mon[32670]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:50.067 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:49 smithi196 ceph-mon[32670]: Saving service alertmanager spec with placement smithi098=a;count:1 2024-09-21T09:50:50.067 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:49 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:50.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:49 smithi098 ceph-mon[29239]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:50.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:49 smithi098 ceph-mon[29239]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:50.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:49 smithi098 ceph-mon[29239]: Saving service alertmanager spec with placement smithi098=a;count:1 2024-09-21T09:50:50.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:49 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:50.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:49 smithi098 ceph-mon[39960]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:50.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:49 smithi098 ceph-mon[39960]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:50.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:49 smithi098 ceph-mon[39960]: Saving service alertmanager spec with placement smithi098=a;count:1 2024-09-21T09:50:50.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:49 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:50.248 DEBUG:teuthology.orchestra.run.smithi098:alertmanager.a> sudo journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@alertmanager.a.service 2024-09-21T09:50:50.250 INFO:tasks.cephadm:Adding grafana.a on smithi196 2024-09-21T09:50:50.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph orch apply grafana '1;smithi196=a' 2024-09-21T09:50:51.876 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:51 smithi196 ceph-mon[32670]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:52.059 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:50:52.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:51 smithi098 ceph-mon[29239]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:52.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:51 smithi098 ceph-mon[39960]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:53.809 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled grafana update... 2024-09-21T09:50:54.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:53 smithi098 ceph-mon[29239]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:54.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:53 smithi098 ceph-mon[39960]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:54.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:53 smithi196 ceph-mon[32670]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:54.477 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:50:54 smithi196 systemd[1]: Started Ceph prometheus.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:50:54.793 DEBUG:teuthology.orchestra.run.smithi196:grafana.a> sudo journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@grafana.a.service 2024-09-21T09:50:54.795 INFO:tasks.cephadm:Setting up client nodes... 2024-09-21T09:50:54.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T09:50:55.045 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:50:55.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:54 smithi098 ceph-mon[29239]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:55.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:54 smithi098 ceph-mon[29239]: Saving service grafana spec with placement smithi196=a;count:1 2024-09-21T09:50:55.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:54 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:55.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:54 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:55.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:54 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:55.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:54 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:55.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:54 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-21T09:50:55.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:54 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:55.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:54 smithi098 ceph-mon[39960]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:55.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:54 smithi098 ceph-mon[39960]: Saving service grafana spec with placement smithi196=a;count:1 2024-09-21T09:50:55.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:54 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:55.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:54 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:55.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:54 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:55.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:54 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:55.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:54 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-21T09:50:55.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:54 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:55.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:54 smithi196 ceph-mon[32670]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:55.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:54 smithi196 ceph-mon[32670]: Saving service grafana spec with placement smithi196=a;count:1 2024-09-21T09:50:55.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:54 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:54 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:54 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:54 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:54 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-21T09:50:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:54 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' 2024-09-21T09:50:55.770 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:55 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ignoring --setuser ceph since I am not root 2024-09-21T09:50:55.771 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:55 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ignoring --setgroup ceph since I am not root 2024-09-21T09:50:55.771 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:55 smithi196 ceph-mgr[34506]: -- 172.21.15.196:0/3881465214 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d454b04340 con 0x55d454b10c00 2024-09-21T09:50:55.771 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:55 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:55.524+0000 7fa9ef53bec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:50:55.771 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:55 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:55.579+0000 7fa9ef53bec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:50:55.771 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:55 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:55.644+0000 7fa9ef53bec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:50:55.771 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:55 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:55.767+0000 7fa9ef53bec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:50:55.772 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:55 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ignoring --setuser ceph since I am not root 2024-09-21T09:50:55.772 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:55 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ignoring --setgroup ceph since I am not root 2024-09-21T09:50:55.772 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:55 smithi098 ceph-mgr[29471]: -- 172.21.15.98:0/4250471660 <== mon.2 v2:172.21.15.98:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5579377e4340 con 0x5579377f0800 2024-09-21T09:50:55.772 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:55 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:55.525+0000 7fec7a80cec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:50:55.772 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:55 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:55.579+0000 7fec7a80cec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:50:55.772 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:55 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:55.644+0000 7fec7a80cec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:50:55.772 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:55 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:55.768+0000 7fec7a80cec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:50:56.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:55 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:55.905+0000 7fec7a80cec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:50:56.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:55 smithi098 ceph-mon[29239]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:56.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:55 smithi098 ceph-mon[29239]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T09:50:56.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:55 smithi098 ceph-mon[29239]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-21T09:50:56.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:55 smithi098 ceph-mon[39960]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:56.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:55 smithi098 ceph-mon[39960]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T09:50:56.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:55 smithi098 ceph-mon[39960]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-21T09:50:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:55 smithi196 ceph-mon[32670]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:55 smithi196 ceph-mon[32670]: from='mgr.14150 172.21.15.98:0/148226246' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T09:50:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:55 smithi196 ceph-mon[32670]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-21T09:50:56.139 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:55 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:55.904+0000 7fa9ef53bec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:50:56.468 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:56.162+0000 7fec7a80cec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:50:56.468 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:56.224+0000 7fec7a80cec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:50:56.469 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:56 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:56.388+0000 7fec7a80cec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:50:56.638 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:56 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:56.157+0000 7fa9ef53bec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:50:56.638 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:56 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:56.215+0000 7fa9ef53bec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:50:56.638 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:56 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:56.373+0000 7fa9ef53bec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:50:56.920 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2024-09-21T09:50:56.920 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAAl+5muYyHNhAAnfEKBy/6i0sYmMcjlkv8rQ== 2024-09-21T09:50:57.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:56 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/125721818' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:50:57.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:56 smithi098 ceph-mon[29239]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:50:57.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:56 smithi098 ceph-mon[29239]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:50:57.244 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:56 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/125721818' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:50:57.244 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:56 smithi098 ceph-mon[39960]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:50:57.244 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:56 smithi098 ceph-mon[39960]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:50:57.275 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:57 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:57.208+0000 7fa9ef53bec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:50:57.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:56 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/125721818' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:50:57.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:56 smithi196 ceph-mon[32670]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:50:57.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:56 smithi196 ceph-mon[32670]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:50:57.528 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:57 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:57.273+0000 7fec7a80cec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:50:57.528 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:57 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:57.336+0000 7fec7a80cec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:50:57.553 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-21T09:50:57.553 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-21T09:50:57.553 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T09:50:57.593 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T09:50:57.615 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:57 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:57.272+0000 7fa9ef53bec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:50:57.849 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:50:58.247 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:58 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:58.005+0000 7fa9ef53bec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:50:58.247 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:58 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:58.120+0000 7fa9ef53bec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:50:58.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:58.064+0000 7fec7a80cec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:50:58.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:58.179+0000 7fec7a80cec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:50:58.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:58.302+0000 7fec7a80cec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:50:58.502 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:58 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:58.243+0000 7fa9ef53bec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:50:58.502 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:58 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:58.306+0000 7fa9ef53bec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:50:58.502 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:58 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:58.432+0000 7fa9ef53bec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:50:58.502 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:58 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:58.499+0000 7fa9ef53bec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:50:58.617 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:58.362+0000 7fec7a80cec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:50:58.617 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:58.490+0000 7fec7a80cec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:50:58.617 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:58.557+0000 7fec7a80cec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:50:58.617 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:58.615+0000 7fec7a80cec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:50:58.762 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:58 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:58.557+0000 7fa9ef53bec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:50:58.762 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:58 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:58.694+0000 7fa9ef53bec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:50:58.885 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:58.752+0000 7fec7a80cec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:50:58.885 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:58.819+0000 7fec7a80cec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:50:58.885 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:58.882+0000 7fec7a80cec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:50:59.014 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:58 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:58.758+0000 7fa9ef53bec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:50:59.014 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:58 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:58.825+0000 7fa9ef53bec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:50:59.015 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:58 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:58.907+0000 7fa9ef53bec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:50:59.015 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:58 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:58.965+0000 7fa9ef53bec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:50:59.274 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:59 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:59.024+0000 7fa9ef53bec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:50:59.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:58.962+0000 7fec7a80cec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:50:59.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:59.018+0000 7fec7a80cec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:50:59.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:59.073+0000 7fec7a80cec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:50:59.699 INFO:teuthology.orchestra.run.smithi196.stdout:[client.1] 2024-09-21T09:50:59.700 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQADl+5mY3pkKRAA7ZiNK9De4pkER7Pl0H4ovw== 2024-09-21T09:50:59.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:59 smithi196 ceph-mon[32670]: from='client.? 172.21.15.196:0/3912798045' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:50:59.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:59 smithi196 ceph-mon[32670]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:50:59.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:50:59 smithi196 ceph-mon[32670]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:50:59.991 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:59 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:59.803+0000 7fa9ef53bec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:50:59.991 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:59 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: [21/Sep/2024:09:50:59] ENGINE Bus STARTING 2024-09-21T09:50:59.991 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:59 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: CherryPy Checker: 2024-09-21T09:50:59.991 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:59 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: The Application mounted at '' has an empty config. 2024-09-21T09:50:59.991 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:59 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: 2024-09-21T09:50:59.991 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:59 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: [21/Sep/2024:09:50:59] ENGINE Serving on http://:::9283 2024-09-21T09:50:59.991 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:50:59 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: [21/Sep/2024:09:50:59] ENGINE Bus STARTED 2024-09-21T09:50:59.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:59 smithi098 ceph-mon[29239]: from='client.? 172.21.15.196:0/3912798045' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:50:59.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:59 smithi098 ceph-mon[29239]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:50:59.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:50:59 smithi098 ceph-mon[29239]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:50:59.999 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:50:59.812+0000 7fec7a80cec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:50:59.999 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:50:59] ENGINE Bus STARTING 2024-09-21T09:50:59.999 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: CherryPy Checker: 2024-09-21T09:50:59.999 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: The Application mounted at '' has an empty config. 2024-09-21T09:50:59.999 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:51:00.000 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:59 smithi098 ceph-mon[39960]: from='client.? 172.21.15.196:0/3912798045' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:51:00.000 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:59 smithi098 ceph-mon[39960]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:51:00.000 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:50:59 smithi098 ceph-mon[39960]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:51:00.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:50:59] ENGINE Serving on http://:::9283 2024-09-21T09:51:00.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:50:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:50:59] ENGINE Bus STARTED 2024-09-21T09:51:00.370 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-21T09:51:00.370 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-21T09:51:00.370 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-21T09:51:00.448 INFO:tasks.cephadm:Creating RBD pool 2024-09-21T09:51:00.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool create rbd 8 2024-09-21T09:51:01.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: Standby manager daemon x restarted 2024-09-21T09:51:01.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: Standby manager daemon x started 2024-09-21T09:51:01.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.196:0/2795218522' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:51:01.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.196:0/2795218522' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:51:01.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.196:0/2795218522' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:51:01.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.196:0/2795218522' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:51:01.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: Active manager daemon y restarted 2024-09-21T09:51:01.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: Activating manager daemon y 2024-09-21T09:51:01.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: osdmap e52: 8 total, 8 up, 8 in 2024-09-21T09:51:01.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: mgrmap e17: y(active, starting, since 0.0066084s), standbys: x 2024-09-21T09:51:01.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:51:01.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:01.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:01.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:51:01.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:51:01.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:51:01.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:51:01.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:51:01.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:51:01.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:51:01.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:51:01.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:51:01.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:51:01.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:51:01.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:51:01.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:51:01.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: Manager daemon y is now available 2024-09-21T09:51:01.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:01.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:01.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:51:01.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:51:01.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:51:01.045 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: Standby manager daemon x restarted 2024-09-21T09:51:01.045 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: Standby manager daemon x started 2024-09-21T09:51:01.045 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/2795218522' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:51:01.045 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/2795218522' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:51:01.045 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/2795218522' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:51:01.045 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/2795218522' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:51:01.045 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: Active manager daemon y restarted 2024-09-21T09:51:01.045 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: Activating manager daemon y 2024-09-21T09:51:01.045 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: osdmap e52: 8 total, 8 up, 8 in 2024-09-21T09:51:01.045 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: mgrmap e17: y(active, starting, since 0.0066084s), standbys: x 2024-09-21T09:51:01.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:51:01.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:01.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:01.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:51:01.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:51:01.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:51:01.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:51:01.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:51:01.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:51:01.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:51:01.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:51:01.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:51:01.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:51:01.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:51:01.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:51:01.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:51:01.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: Manager daemon y is now available 2024-09-21T09:51:01.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:01.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:01.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:51:01.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:51:01.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:51:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: Standby manager daemon x restarted 2024-09-21T09:51:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: Standby manager daemon x started 2024-09-21T09:51:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/2795218522' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:51:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/2795218522' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:51:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/2795218522' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:51:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/2795218522' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:51:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: Active manager daemon y restarted 2024-09-21T09:51:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: Activating manager daemon y 2024-09-21T09:51:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: osdmap e52: 8 total, 8 up, 8 in 2024-09-21T09:51:01.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: mgrmap e17: y(active, starting, since 0.0066084s), standbys: x 2024-09-21T09:51:01.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:51:01.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:01.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:01.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:51:01.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:51:01.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:51:01.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:51:01.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:51:01.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:51:01.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:51:01.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:51:01.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:51:01.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:51:01.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:51:01.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:51:01.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:51:01.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: Manager daemon y is now available 2024-09-21T09:51:01.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:01.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:01.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:51:01.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:51:01.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:51:01.305 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:01.348 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:51:01.057+0000 7fec1d5cf640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:09:51:01] ENGINE Error in HTTPServer.serve 2024-09-21T09:51:01.348 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: Traceback (most recent call last): 2024-09-21T09:51:01.348 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:51:01.348 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: self._connections.run(self.expiration_interval) 2024-09-21T09:51:01.348 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:51:01.349 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: self._run(expiration_interval) 2024-09-21T09:51:01.349 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:51:01.349 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:51:01.349 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:51:01.349 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:51:01.349 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:51:01.349 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: s = self.context.wrap_socket( 2024-09-21T09:51:01.349 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:51:01.349 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: return self.sslsocket_class._create( 2024-09-21T09:51:01.349 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:51:01.349 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: self.do_handshake() 2024-09-21T09:51:01.349 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:51:01.349 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: self._sslobj.do_handshake() 2024-09-21T09:51:01.349 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:51:01.350 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:51:01.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:01 smithi196 ceph-mon[32670]: mgrmap e18: y(active, since 1.01046s), standbys: x 2024-09-21T09:51:01.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:01 smithi196 ceph-mon[32670]: [21/Sep/2024:09:51:00] ENGINE Bus STARTING 2024-09-21T09:51:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:01 smithi196 ceph-mon[32670]: [21/Sep/2024:09:51:01] ENGINE Serving on https://172.21.15.98:7150 2024-09-21T09:51:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:01 smithi196 ceph-mon[32670]: [21/Sep/2024:09:51:01] ENGINE Bus STARTED 2024-09-21T09:51:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:01 smithi196 ceph-mon[32670]: [21/Sep/2024:09:51:01] ENGINE Error in HTTPServer.serve 2024-09-21T09:51:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-21T09:51:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:51:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:51:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:51:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._run(expiration_interval) 2024-09-21T09:51:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:51:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:51:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:51:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:51:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:51:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout: s = self.context.wrap_socket( 2024-09-21T09:51:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:51:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.sslsocket_class._create( 2024-09-21T09:51:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:51:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout: self.do_handshake() 2024-09-21T09:51:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:51:01.891 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._sslobj.do_handshake() 2024-09-21T09:51:01.891 INFO:journalctl@ceph.mon.b.smithi196.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:51:01.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:01 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:02.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-mon[39960]: mgrmap e18: y(active, since 1.01046s), standbys: x 2024-09-21T09:51:02.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-mon[39960]: [21/Sep/2024:09:51:00] ENGINE Bus STARTING 2024-09-21T09:51:02.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-mon[39960]: [21/Sep/2024:09:51:01] ENGINE Serving on https://172.21.15.98:7150 2024-09-21T09:51:02.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-mon[39960]: [21/Sep/2024:09:51:01] ENGINE Bus STARTED 2024-09-21T09:51:02.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-mon[39960]: [21/Sep/2024:09:51:01] ENGINE Error in HTTPServer.serve 2024-09-21T09:51:02.097 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-09-21T09:51:02.097 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:51:02.097 INFO:journalctl@ceph.mon.c.smithi098.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:51:02.097 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:51:02.098 INFO:journalctl@ceph.mon.c.smithi098.stdout: self._run(expiration_interval) 2024-09-21T09:51:02.098 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:51:02.098 INFO:journalctl@ceph.mon.c.smithi098.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:51:02.098 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:51:02.098 INFO:journalctl@ceph.mon.c.smithi098.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:51:02.098 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:51:02.098 INFO:journalctl@ceph.mon.c.smithi098.stdout: s = self.context.wrap_socket( 2024-09-21T09:51:02.098 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:51:02.098 INFO:journalctl@ceph.mon.c.smithi098.stdout: return self.sslsocket_class._create( 2024-09-21T09:51:02.098 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:51:02.098 INFO:journalctl@ceph.mon.c.smithi098.stdout: self.do_handshake() 2024-09-21T09:51:02.098 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:51:02.098 INFO:journalctl@ceph.mon.c.smithi098.stdout: self._sslobj.do_handshake() 2024-09-21T09:51:02.098 INFO:journalctl@ceph.mon.c.smithi098.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:51:02.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:02.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-mon[29239]: mgrmap e18: y(active, since 1.01046s), standbys: x 2024-09-21T09:51:02.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-mon[29239]: [21/Sep/2024:09:51:00] ENGINE Bus STARTING 2024-09-21T09:51:02.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-mon[29239]: [21/Sep/2024:09:51:01] ENGINE Serving on https://172.21.15.98:7150 2024-09-21T09:51:02.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-mon[29239]: [21/Sep/2024:09:51:01] ENGINE Bus STARTED 2024-09-21T09:51:02.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-mon[29239]: [21/Sep/2024:09:51:01] ENGINE Error in HTTPServer.serve 2024-09-21T09:51:02.099 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-09-21T09:51:02.099 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:51:02.099 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:51:02.099 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:51:02.099 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._run(expiration_interval) 2024-09-21T09:51:02.099 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:51:02.100 INFO:journalctl@ceph.mon.a.smithi098.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:51:02.100 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:51:02.100 INFO:journalctl@ceph.mon.a.smithi098.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:51:02.100 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:51:02.100 INFO:journalctl@ceph.mon.a.smithi098.stdout: s = self.context.wrap_socket( 2024-09-21T09:51:02.100 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:51:02.100 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.sslsocket_class._create( 2024-09-21T09:51:02.100 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:51:02.100 INFO:journalctl@ceph.mon.a.smithi098.stdout: self.do_handshake() 2024-09-21T09:51:02.100 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:51:02.100 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._sslobj.do_handshake() 2024-09-21T09:51:02.100 INFO:journalctl@ceph.mon.a.smithi098.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:51:02.100 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:01 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:03 smithi098 ceph-mon[29239]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:03 smithi098 ceph-mon[29239]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-21T09:51:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:03 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:03 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:03 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:03 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:03 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:03 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:03 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:03 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:03.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:03 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:03.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:03 smithi098 ceph-mon[39960]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:03.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:03 smithi098 ceph-mon[39960]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-21T09:51:03.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:03 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:03.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:03 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:03.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:03 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:03.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:03 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:03.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:03 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:03.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:03 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:03.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:03 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:03.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:03 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:03.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:03 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:03.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:03 smithi196 ceph-mon[32670]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:03 smithi196 ceph-mon[32670]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-21T09:51:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:03 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:03 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:03 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:03 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:03 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:03 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:03 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:03 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:03 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:04.131 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'rbd' created 2024-09-21T09:51:04.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:04 smithi196 ceph-mon[32670]: Adjusting osd_memory_target on smithi196 to 4016M 2024-09-21T09:51:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:04 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:04 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:04 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2404726597' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-21T09:51:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:04 smithi196 ceph-mon[32670]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-21T09:51:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:04 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:04 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:04 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:04 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:04.518 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:04 smithi098 ceph-mon[29239]: Adjusting osd_memory_target on smithi196 to 4016M 2024-09-21T09:51:04.519 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:04 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:04.519 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:04 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:04.519 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:04 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2404726597' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-21T09:51:04.519 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:04 smithi098 ceph-mon[29239]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-21T09:51:04.519 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:04 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:04.520 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:04 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:04.520 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:04 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:04.520 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:04 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:04.520 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:04 smithi098 ceph-mon[39960]: Adjusting osd_memory_target on smithi196 to 4016M 2024-09-21T09:51:04.520 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:04 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:04.520 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:04 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:04.520 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:04 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2404726597' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-21T09:51:04.520 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:04 smithi098 ceph-mon[39960]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-21T09:51:04.521 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:04 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:04.521 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:04 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:04.521 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:04 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:04.521 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:04 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:04.955 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- rbd pool init rbd 2024-09-21T09:51:05.236 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:05.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[29239]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-21T09:51:05.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[29239]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-21T09:51:05.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[29239]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:05.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[29239]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:51:05.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[29239]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:51:05.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[29239]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-21T09:51:05.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[29239]: osdmap e53: 8 total, 8 up, 8 in 2024-09-21T09:51:05.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[29239]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:51:05.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[29239]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:51:05.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[29239]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:51:05.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[29239]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:51:05.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:05.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:05.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:05.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:05.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:05.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[29239]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-21T09:51:05.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[39960]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-21T09:51:05.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[39960]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-21T09:51:05.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[39960]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:05.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[39960]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:51:05.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[39960]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:51:05.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[39960]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-21T09:51:05.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[39960]: osdmap e53: 8 total, 8 up, 8 in 2024-09-21T09:51:05.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[39960]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:51:05.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[39960]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:51:05.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[39960]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:51:05.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[39960]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:51:05.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:05.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:05.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:05.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:05.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:05.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:05 smithi098 ceph-mon[39960]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-21T09:51:05.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:05 smithi196 ceph-mon[32670]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-21T09:51:05.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:05 smithi196 ceph-mon[32670]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-21T09:51:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:05 smithi196 ceph-mon[32670]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:05 smithi196 ceph-mon[32670]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:51:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:05 smithi196 ceph-mon[32670]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:51:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:05 smithi196 ceph-mon[32670]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-21T09:51:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:05 smithi196 ceph-mon[32670]: osdmap e53: 8 total, 8 up, 8 in 2024-09-21T09:51:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:05 smithi196 ceph-mon[32670]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:51:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:05 smithi196 ceph-mon[32670]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:51:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:05 smithi196 ceph-mon[32670]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:51:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:05 smithi196 ceph-mon[32670]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:51:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:05 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:05 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:05 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:05 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:05 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:05 smithi196 ceph-mon[32670]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-21T09:51:06.554 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:06 smithi196 ceph-mon[32670]: Deploying daemon alertmanager.a on smithi098 2024-09-21T09:51:06.555 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:06 smithi196 ceph-mon[32670]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T09:51:06.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:06 smithi098 ceph-mon[29239]: Deploying daemon alertmanager.a on smithi098 2024-09-21T09:51:06.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:06 smithi098 ceph-mon[29239]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T09:51:06.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:06 smithi098 ceph-mon[39960]: Deploying daemon alertmanager.a on smithi098 2024-09-21T09:51:06.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:06 smithi098 ceph-mon[39960]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T09:51:06.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:51:06 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:51:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:51:07.481 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:07 smithi098 ceph-mon[29239]: pgmap v8: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:07.482 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:07 smithi098 ceph-mon[29239]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:51:07.482 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:07 smithi098 ceph-mon[29239]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T09:51:07.482 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:07 smithi098 ceph-mon[39960]: pgmap v8: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:07.482 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:07 smithi098 ceph-mon[39960]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:51:07.482 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:07 smithi098 ceph-mon[39960]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T09:51:07.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:07 smithi196 ceph-mon[32670]: pgmap v8: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:07.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:07 smithi196 ceph-mon[32670]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:51:07.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:07 smithi196 ceph-mon[32670]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T09:51:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:08 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/4266816680' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:51:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:08 smithi098 ceph-mon[29239]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:51:08.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:08 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/4266816680' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:51:08.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:08 smithi098 ceph-mon[39960]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:51:08.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:08 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/4266816680' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:51:08.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:08 smithi196 ceph-mon[32670]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:51:09.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:09 smithi098 ceph-mon[29239]: pgmap v10: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T09:51:09.410 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:09 smithi098 ceph-mon[29239]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-21T09:51:09.410 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:09 smithi098 ceph-mon[29239]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T09:51:09.410 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:09 smithi098 ceph-mon[39960]: pgmap v10: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T09:51:09.410 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:09 smithi098 ceph-mon[39960]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-21T09:51:09.410 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:09 smithi098 ceph-mon[39960]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T09:51:09.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:09 smithi196 ceph-mon[32670]: pgmap v10: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T09:51:09.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:09 smithi196 ceph-mon[32670]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-21T09:51:09.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:09 smithi196 ceph-mon[32670]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T09:51:10.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:10 smithi098 ceph-mon[29239]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T09:51:10.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:10 smithi098 ceph-mon[39960]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T09:51:10.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:10 smithi196 ceph-mon[32670]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T09:51:11.308 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:11 smithi098 ceph-mon[29239]: pgmap v13: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T09:51:11.308 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:11 smithi098 ceph-mon[29239]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:51:11.308 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:11 smithi098 ceph-mon[29239]: Cluster is now healthy 2024-09-21T09:51:11.308 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:11 smithi098 ceph-mon[29239]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T09:51:11.309 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:11 smithi098 ceph-mon[39960]: pgmap v13: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T09:51:11.309 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:11 smithi098 ceph-mon[39960]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:51:11.309 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:11 smithi098 ceph-mon[39960]: Cluster is now healthy 2024-09-21T09:51:11.309 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:11 smithi098 ceph-mon[39960]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T09:51:11.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:11 smithi196 ceph-mon[32670]: pgmap v13: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T09:51:11.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:11 smithi196 ceph-mon[32670]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:51:11.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:11 smithi196 ceph-mon[32670]: Cluster is now healthy 2024-09-21T09:51:11.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:11 smithi196 ceph-mon[32670]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T09:51:12.229 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph config log 1 --format=json 2024-09-21T09:51:12.918 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:13.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:13 smithi098 ceph-mon[29239]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 360 B/s wr, 0 op/s 2024-09-21T09:51:13.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:13 smithi098 ceph-mon[39960]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 360 B/s wr, 0 op/s 2024-09-21T09:51:13.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:13 smithi196 ceph-mon[32670]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 360 B/s wr, 0 op/s 2024-09-21T09:51:14.727 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:51:14 smithi098 podman[68367]: 2024-09-21 09:51:14.725910323 +0000 UTC m=+0.523031403 volume create 5e2c9bc9a8ceb4f7b6db272a663a1e389c5a93118b0101317fe9a957877f4d6c 2024-09-21T09:51:14.815 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:51:15.097 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:51:14 smithi098 podman[68367]: 2024-09-21 09:51:14.834702623 +0000 UTC m=+0.631823694 container create 31b7c93087d5da407e07433409ff954986cc6afa3bb567fddc4c5993cd7239b7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:51:15.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:15 smithi196 ceph-mon[32670]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-09-21T09:51:15.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:15 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/1292542800' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-21T09:51:15.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:15 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:15.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:15 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:51:15.448 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:15 smithi098 ceph-mon[29239]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-09-21T09:51:15.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:15 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/1292542800' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-21T09:51:15.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:15 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:15.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:15 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:51:15.449 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:15 smithi098 ceph-mon[39960]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-09-21T09:51:15.449 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:15 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/1292542800' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-21T09:51:15.449 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:15 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:15.449 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:15 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:51:15.450 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:51:15 smithi098 podman[68367]: 2024-09-21 09:51:15.347780233 +0000 UTC m=+1.144901309 container init 31b7c93087d5da407e07433409ff954986cc6afa3bb567fddc4c5993cd7239b7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:51:15.450 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:51:15 smithi098 podman[68367]: 2024-09-21 09:51:15.350723864 +0000 UTC m=+1.147844943 container start 31b7c93087d5da407e07433409ff954986cc6afa3bb567fddc4c5993cd7239b7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:51:15.450 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:51:15 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[68424]: ts=2024-09-21T09:51:15.362Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-21T09:51:15.450 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:51:15 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[68424]: ts=2024-09-21T09:51:15.363Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-21T09:51:15.450 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:51:15 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[68424]: ts=2024-09-21T09:51:15.364Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.98 port=9094 2024-09-21T09:51:15.450 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:51:15 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[68424]: ts=2024-09-21T09:51:15.365Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-21T09:51:15.450 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:51:15 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[68424]: ts=2024-09-21T09:51:15.387Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T09:51:15.450 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:51:15 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[68424]: ts=2024-09-21T09:51:15.388Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T09:51:15.450 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:51:15 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[68424]: ts=2024-09-21T09:51:15.390Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-21T09:51:15.450 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:51:15 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[68424]: ts=2024-09-21T09:51:15.390Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-21T09:51:15.450 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:51:15 smithi098 bash[68367]: 31b7c93087d5da407e07433409ff954986cc6afa3bb567fddc4c5993cd7239b7 2024-09-21T09:51:15.847 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:51:15 smithi098 systemd[1]: Started Ceph alertmanager.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:51:15.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:15 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:51:15] ENGINE Bus STOPPING 2024-09-21T09:51:15.959 INFO:teuthology.orchestra.run.smithi098.stdout:[{"version":18,"timestamp":"2024-09-21T09:51:02.710250+0000","name":"","changes":[{"name":"osd/host:smithi196/osd_memory_target","previous_value":"4479528960","new_value":"4211093504"}]}] 2024-09-21T09:51:15.959 INFO:tasks.ceph_manager:config epoch is 18 2024-09-21T09:51:15.959 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T09:51:15.959 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-21T09:51:15.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph mgr dump --format=json 2024-09-21T09:51:16.204 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:16.325 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:51:16] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:51:16.325 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:51:16] ENGINE Bus STOPPED 2024-09-21T09:51:16.325 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:51:16] ENGINE Bus STARTING 2024-09-21T09:51:16.325 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:51:16] ENGINE Serving on http://:::9283 2024-09-21T09:51:16.325 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:51:16] ENGINE Bus STARTED 2024-09-21T09:51:16.582 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:16.582 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:16.582 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:16.582 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:16.582 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:16.583 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:16.583 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:51:16.583 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-mon[29239]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:51:16.583 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:16.583 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-mon[29239]: Deploying daemon grafana.a on smithi196 2024-09-21T09:51:16.583 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-mon[29239]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 266 B/s wr, 0 op/s 2024-09-21T09:51:16.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:16.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:16.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:16.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:16.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:16.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:16.584 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:51:16.584 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-mon[39960]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:51:16.584 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:16.584 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-mon[39960]: Deploying daemon grafana.a on smithi196 2024-09-21T09:51:16.584 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:16 smithi098 ceph-mon[39960]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 266 B/s wr, 0 op/s 2024-09-21T09:51:16.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:51:16 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:51:16] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:51:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:16 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:16 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:16 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:16 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:16 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:16 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:16 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:51:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:16 smithi196 ceph-mon[32670]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:51:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:16 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:16 smithi196 ceph-mon[32670]: Deploying daemon grafana.a on smithi196 2024-09-21T09:51:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:16 smithi196 ceph-mon[32670]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 266 B/s wr, 0 op/s 2024-09-21T09:51:17.668 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:51:17 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[68424]: ts=2024-09-21T09:51:17.365Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000575004s 2024-09-21T09:51:18.079 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:51:18.710 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":20,"flags":0,"active_gid":24323,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":1387543382},{"type":"v1","addr":"172.21.15.98:6801","nonce":1387543382}]},"active_addr":"172.21.15.98:6801/1387543382","active_change":"2024-09-21T09:50:59.815532+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":24325,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.98:8443/","prometheus":"http://172.21.15.98:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":52,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":3466051265}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1093274465}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1829140501}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":900860885}]}]} 2024-09-21T09:51:18.715 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-21T09:51:18.715 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T09:51:18.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T09:51:18.958 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:19.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:18 smithi098 ceph-mon[29239]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 235 B/s wr, 0 op/s 2024-09-21T09:51:19.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:18 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/1804227569' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:51:19.079 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:18 smithi098 ceph-mon[29239]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T09:51:19.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:18 smithi098 ceph-mon[39960]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 235 B/s wr, 0 op/s 2024-09-21T09:51:19.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:18 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/1804227569' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:51:19.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:18 smithi098 ceph-mon[39960]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T09:51:19.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:18 smithi196 ceph-mon[32670]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 235 B/s wr, 0 op/s 2024-09-21T09:51:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:18 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/1804227569' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:51:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:18 smithi196 ceph-mon[32670]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T09:51:20.771 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:51:20.771 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":59,"fsid":"26a15512-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:45:33.813487+0000","modified":"2024-09-21T09:51:18.751019+0000","last_up_change":"2024-09-21T09:50:27.903155+0000","last_in_change":"2024-09-21T09:50:10.725457+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:48:34.781592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:51:03.408228+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":58,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"579f13de-565d-4801-b65e-f27d1cde297a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":506591636},{"type":"v1","addr":"172.21.15.98:6803","nonce":506591636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":506591636},{"type":"v1","addr":"172.21.15.98:6805","nonce":506591636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":506591636},{"type":"v1","addr":"172.21.15.98:6809","nonce":506591636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":506591636},{"type":"v1","addr":"172.21.15.98:6807","nonce":506591636}]},"public_addr":"172.21.15.98:6803/506591636","cluster_addr":"172.21.15.98:6805/506591636","heartbeat_back_addr":"172.21.15.98:6809/506591636","heartbeat_front_addr":"172.21.15.98:6807/506591636","state":["exists","up"]},{"osd":1,"uuid":"6d31ac4e-dc1a-4051-8266-ba821106242f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":4240601629},{"type":"v1","addr":"172.21.15.98:6811","nonce":4240601629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":4240601629},{"type":"v1","addr":"172.21.15.98:6813","nonce":4240601629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4240601629},{"type":"v1","addr":"172.21.15.98:6817","nonce":4240601629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":4240601629},{"type":"v1","addr":"172.21.15.98:6815","nonce":4240601629}]},"public_addr":"172.21.15.98:6811/4240601629","cluster_addr":"172.21.15.98:6813/4240601629","heartbeat_back_addr":"172.21.15.98:6817/4240601629","heartbeat_front_addr":"172.21.15.98:6815/4240601629","state":["exists","up"]},{"osd":2,"uuid":"62e22789-74be-4098-a9b9-62671fb4cac1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2917126560},{"type":"v1","addr":"172.21.15.98:6819","nonce":2917126560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2917126560},{"type":"v1","addr":"172.21.15.98:6821","nonce":2917126560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2917126560},{"type":"v1","addr":"172.21.15.98:6825","nonce":2917126560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2917126560},{"type":"v1","addr":"172.21.15.98:6823","nonce":2917126560}]},"public_addr":"172.21.15.98:6819/2917126560","cluster_addr":"172.21.15.98:6821/2917126560","heartbeat_back_addr":"172.21.15.98:6825/2917126560","heartbeat_front_addr":"172.21.15.98:6823/2917126560","state":["exists","up"]},{"osd":3,"uuid":"77f653a6-69b0-437a-960d-c3fb036708be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":4022556179},{"type":"v1","addr":"172.21.15.98:6827","nonce":4022556179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":4022556179},{"type":"v1","addr":"172.21.15.98:6829","nonce":4022556179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4022556179},{"type":"v1","addr":"172.21.15.98:6833","nonce":4022556179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":4022556179},{"type":"v1","addr":"172.21.15.98:6831","nonce":4022556179}]},"public_addr":"172.21.15.98:6827/4022556179","cluster_addr":"172.21.15.98:6829/4022556179","heartbeat_back_addr":"172.21.15.98:6833/4022556179","heartbeat_front_addr":"172.21.15.98:6831/4022556179","state":["exists","up"]},{"osd":4,"uuid":"28ec4928-7f00-46a9-b8c5-95f0b7e6be75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6801","nonce":158468112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6803","nonce":158468112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6807","nonce":158468112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6805","nonce":158468112}]},"public_addr":"172.21.15.196:6801/158468112","cluster_addr":"172.21.15.196:6803/158468112","heartbeat_back_addr":"172.21.15.196:6807/158468112","heartbeat_front_addr":"172.21.15.196:6805/158468112","state":["exists","up"]},{"osd":5,"uuid":"cdcc44eb-e27f-4d23-9ecb-03bebdc82c96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6809","nonce":2341878784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6811","nonce":2341878784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6815","nonce":2341878784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6813","nonce":2341878784}]},"public_addr":"172.21.15.196:6809/2341878784","cluster_addr":"172.21.15.196:6811/2341878784","heartbeat_back_addr":"172.21.15.196:6815/2341878784","heartbeat_front_addr":"172.21.15.196:6813/2341878784","state":["exists","up"]},{"osd":6,"uuid":"566cfb44-1b21-45fc-b053-7205b1c10d4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6817","nonce":3059702155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6819","nonce":3059702155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6823","nonce":3059702155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6821","nonce":3059702155}]},"public_addr":"172.21.15.196:6817/3059702155","cluster_addr":"172.21.15.196:6819/3059702155","heartbeat_back_addr":"172.21.15.196:6823/3059702155","heartbeat_front_addr":"172.21.15.196:6821/3059702155","state":["exists","up"]},{"osd":7,"uuid":"228404c1-3910-4933-b023-55aa8b01aee4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6825","nonce":2155020234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6827","nonce":2155020234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6831","nonce":2155020234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6829","nonce":2155020234}]},"public_addr":"172.21.15.196:6825/2155020234","cluster_addr":"172.21.15.196:6827/2155020234","heartbeat_back_addr":"172.21.15.196:6831/2155020234","heartbeat_front_addr":"172.21.15.196:6829/2155020234","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:47:43.263977+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:06.738746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:30.883578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:54.216866+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:49:15.599276+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:49:38.581033+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:01.504502+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:25.964220+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/4139436413":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/2144473621":"2024-09-22T09:45:54.237657+0000","172.21.15.98:0/2075804383":"2024-09-22T09:45:54.237657+0000","172.21.15.98:0/675672768":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/3656555554":"2024-09-22T09:45:54.237657+0000","172.21.15.98:6800/440385038":"2024-09-22T09:46:24.467967+0000","172.21.15.98:6801/440385038":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/4208010999":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/262018661":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/3737889208":"2024-09-22T09:46:24.467967+0000","172.21.15.98:6800/89887073":"2024-09-22T09:50:59.815356+0000","172.21.15.98:6801/1427334238":"2024-09-22T09:45:54.237657+0000","172.21.15.98:0/227493886":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/881612124":"2024-09-22T09:50:59.815356+0000","172.21.15.98:6800/1427334238":"2024-09-22T09:45:54.237657+0000","172.21.15.98:6801/89887073":"2024-09-22T09:50:59.815356+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T09:51:21.060 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:20 smithi098 ceph-mon[29239]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 211 B/s wr, 0 op/s 2024-09-21T09:51:21.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:20 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:21.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:20 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/97133043' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:51:21.061 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:20 smithi098 ceph-mon[39960]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 211 B/s wr, 0 op/s 2024-09-21T09:51:21.061 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:20 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:21.061 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:20 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/97133043' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:51:21.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:20 smithi196 ceph-mon[32670]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 211 B/s wr, 0 op/s 2024-09-21T09:51:21.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:20 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:21.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:20 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/97133043' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:51:21.482 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T09:51:21.482 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T09:51:21.727 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:23.082 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:22 smithi098 ceph-mon[29239]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:23.082 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:22 smithi098 ceph-mon[39960]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:23.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:22 smithi196 ceph-mon[32670]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:23.469 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:51:23.469 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":59,"fsid":"26a15512-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:45:33.813487+0000","modified":"2024-09-21T09:51:18.751019+0000","last_up_change":"2024-09-21T09:50:27.903155+0000","last_in_change":"2024-09-21T09:50:10.725457+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:48:34.781592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:51:03.408228+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":58,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"579f13de-565d-4801-b65e-f27d1cde297a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":506591636},{"type":"v1","addr":"172.21.15.98:6803","nonce":506591636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":506591636},{"type":"v1","addr":"172.21.15.98:6805","nonce":506591636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":506591636},{"type":"v1","addr":"172.21.15.98:6809","nonce":506591636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":506591636},{"type":"v1","addr":"172.21.15.98:6807","nonce":506591636}]},"public_addr":"172.21.15.98:6803/506591636","cluster_addr":"172.21.15.98:6805/506591636","heartbeat_back_addr":"172.21.15.98:6809/506591636","heartbeat_front_addr":"172.21.15.98:6807/506591636","state":["exists","up"]},{"osd":1,"uuid":"6d31ac4e-dc1a-4051-8266-ba821106242f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":4240601629},{"type":"v1","addr":"172.21.15.98:6811","nonce":4240601629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":4240601629},{"type":"v1","addr":"172.21.15.98:6813","nonce":4240601629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4240601629},{"type":"v1","addr":"172.21.15.98:6817","nonce":4240601629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":4240601629},{"type":"v1","addr":"172.21.15.98:6815","nonce":4240601629}]},"public_addr":"172.21.15.98:6811/4240601629","cluster_addr":"172.21.15.98:6813/4240601629","heartbeat_back_addr":"172.21.15.98:6817/4240601629","heartbeat_front_addr":"172.21.15.98:6815/4240601629","state":["exists","up"]},{"osd":2,"uuid":"62e22789-74be-4098-a9b9-62671fb4cac1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2917126560},{"type":"v1","addr":"172.21.15.98:6819","nonce":2917126560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2917126560},{"type":"v1","addr":"172.21.15.98:6821","nonce":2917126560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2917126560},{"type":"v1","addr":"172.21.15.98:6825","nonce":2917126560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2917126560},{"type":"v1","addr":"172.21.15.98:6823","nonce":2917126560}]},"public_addr":"172.21.15.98:6819/2917126560","cluster_addr":"172.21.15.98:6821/2917126560","heartbeat_back_addr":"172.21.15.98:6825/2917126560","heartbeat_front_addr":"172.21.15.98:6823/2917126560","state":["exists","up"]},{"osd":3,"uuid":"77f653a6-69b0-437a-960d-c3fb036708be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":4022556179},{"type":"v1","addr":"172.21.15.98:6827","nonce":4022556179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":4022556179},{"type":"v1","addr":"172.21.15.98:6829","nonce":4022556179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4022556179},{"type":"v1","addr":"172.21.15.98:6833","nonce":4022556179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":4022556179},{"type":"v1","addr":"172.21.15.98:6831","nonce":4022556179}]},"public_addr":"172.21.15.98:6827/4022556179","cluster_addr":"172.21.15.98:6829/4022556179","heartbeat_back_addr":"172.21.15.98:6833/4022556179","heartbeat_front_addr":"172.21.15.98:6831/4022556179","state":["exists","up"]},{"osd":4,"uuid":"28ec4928-7f00-46a9-b8c5-95f0b7e6be75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6801","nonce":158468112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6803","nonce":158468112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6807","nonce":158468112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6805","nonce":158468112}]},"public_addr":"172.21.15.196:6801/158468112","cluster_addr":"172.21.15.196:6803/158468112","heartbeat_back_addr":"172.21.15.196:6807/158468112","heartbeat_front_addr":"172.21.15.196:6805/158468112","state":["exists","up"]},{"osd":5,"uuid":"cdcc44eb-e27f-4d23-9ecb-03bebdc82c96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6809","nonce":2341878784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6811","nonce":2341878784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6815","nonce":2341878784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6813","nonce":2341878784}]},"public_addr":"172.21.15.196:6809/2341878784","cluster_addr":"172.21.15.196:6811/2341878784","heartbeat_back_addr":"172.21.15.196:6815/2341878784","heartbeat_front_addr":"172.21.15.196:6813/2341878784","state":["exists","up"]},{"osd":6,"uuid":"566cfb44-1b21-45fc-b053-7205b1c10d4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6817","nonce":3059702155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6819","nonce":3059702155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6823","nonce":3059702155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6821","nonce":3059702155}]},"public_addr":"172.21.15.196:6817/3059702155","cluster_addr":"172.21.15.196:6819/3059702155","heartbeat_back_addr":"172.21.15.196:6823/3059702155","heartbeat_front_addr":"172.21.15.196:6821/3059702155","state":["exists","up"]},{"osd":7,"uuid":"228404c1-3910-4933-b023-55aa8b01aee4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6825","nonce":2155020234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6827","nonce":2155020234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6831","nonce":2155020234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6829","nonce":2155020234}]},"public_addr":"172.21.15.196:6825/2155020234","cluster_addr":"172.21.15.196:6827/2155020234","heartbeat_back_addr":"172.21.15.196:6831/2155020234","heartbeat_front_addr":"172.21.15.196:6829/2155020234","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:47:43.263977+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:06.738746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:30.883578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:54.216866+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:49:15.599276+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:49:38.581033+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:01.504502+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:25.964220+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/4139436413":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/2144473621":"2024-09-22T09:45:54.237657+0000","172.21.15.98:0/2075804383":"2024-09-22T09:45:54.237657+0000","172.21.15.98:0/675672768":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/3656555554":"2024-09-22T09:45:54.237657+0000","172.21.15.98:6800/440385038":"2024-09-22T09:46:24.467967+0000","172.21.15.98:6801/440385038":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/4208010999":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/262018661":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/3737889208":"2024-09-22T09:46:24.467967+0000","172.21.15.98:6800/89887073":"2024-09-22T09:50:59.815356+0000","172.21.15.98:6801/1427334238":"2024-09-22T09:45:54.237657+0000","172.21.15.98:0/227493886":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/881612124":"2024-09-22T09:50:59.815356+0000","172.21.15.98:6800/1427334238":"2024-09-22T09:45:54.237657+0000","172.21.15.98:6801/89887073":"2024-09-22T09:50:59.815356+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T09:51:24.120 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:23 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/961907789' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:51:24.121 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:23 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/961907789' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:51:24.121 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:23 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/961907789' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:51:24.146 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T09:51:24.146 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T09:51:24.147 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T09:51:24.147 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T09:51:24.147 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T09:51:24.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T09:51:24.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T09:51:24.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T09:51:24.674 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:24.674 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:24.675 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:24.675 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:24.678 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:24.678 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:24.683 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:24.690 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:25.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:24 smithi098 ceph-mon[29239]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:25.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:24 smithi098 ceph-mon[39960]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:25.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:24 smithi196 ceph-mon[32670]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:25.646 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:51:25 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[68424]: ts=2024-09-21T09:51:25.368Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003211145s 2024-09-21T09:51:26.553 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:26 smithi196 podman[61265]: 2024-09-21 09:51:26.45467944 +0000 UTC m=+0.119794955 container create be7bf40319e5f8baccf8af3613c924d045b02c9ada77f4a8fb4a67ca718a0dc7 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, description=Ceph Grafana Container, maintainer=Paul Cuzner , io.openshift.expose-services=, build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.buildah.version=1.32.0, name=ubi8, com.redhat.component=ubi8-container, distribution-scope=public) 2024-09-21T09:51:26.883 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:51:26 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:51:26] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:51:27.064 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:26 smithi098 ceph-mon[29239]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:27.064 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:26 smithi098 ceph-mon[39960]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:26 smithi196 ceph-mon[32670]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:27.562 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 podman[61265]: 2024-09-21 09:51:27.288134932 +0000 UTC m=+0.953250410 container init be7bf40319e5f8baccf8af3613c924d045b02c9ada77f4a8fb4a67ca718a0dc7 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a, maintainer=Paul Cuzner , build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, architecture=x86_64, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, name=ubi8, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vendor=Red Hat, Inc., io.openshift.expose-services=, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0) 2024-09-21T09:51:27.562 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 podman[61265]: 2024-09-21 09:51:27.291772503 +0000 UTC m=+0.956887975 container start be7bf40319e5f8baccf8af3613c924d045b02c9ada77f4a8fb4a67ca718a0dc7 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, release=239.1651231664, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, version=8.5, distribution-scope=public, description=Ceph Grafana Container, io.openshift.expose-services=, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-09-21T09:51:27.562 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=settings t=2024-09-21T09:51:27.448918349Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-21T09:51:27.562 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=settings t=2024-09-21T09:51:27.449094962Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-21T09:51:27.562 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=settings t=2024-09-21T09:51:27.449129545Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-21T09:51:27.562 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=settings t=2024-09-21T09:51:27.449136925Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-21T09:51:27.562 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=settings t=2024-09-21T09:51:27.449142885Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-21T09:51:27.562 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=settings t=2024-09-21T09:51:27.449148815Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-21T09:51:27.562 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=settings t=2024-09-21T09:51:27.449154592Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-21T09:51:27.562 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=settings t=2024-09-21T09:51:27.449160282Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-21T09:51:27.563 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=settings t=2024-09-21T09:51:27.449166121Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-21T09:51:27.563 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=settings t=2024-09-21T09:51:27.449171723Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-21T09:51:27.563 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=settings t=2024-09-21T09:51:27.449177189Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-21T09:51:27.563 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=settings t=2024-09-21T09:51:27.449182581Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-21T09:51:27.563 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=settings t=2024-09-21T09:51:27.449189094Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-21T09:51:27.563 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=settings t=2024-09-21T09:51:27.449195991Z level=info msg="App mode production" 2024-09-21T09:51:27.563 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=sqlstore t=2024-09-21T09:51:27.449240525Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-21T09:51:27.563 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=sqlstore t=2024-09-21T09:51:27.449256091Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-21T09:51:27.563 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.450086836Z level=info msg="Starting DB migrations" 2024-09-21T09:51:27.563 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.450156602Z level=info msg="Executing migration" id="create migration_log table" 2024-09-21T09:51:27.563 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.450663001Z level=info msg="Executing migration" id="create user table" 2024-09-21T09:51:27.563 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.451109845Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-21T09:51:27.563 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.45159849Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-21T09:51:27.563 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.452417014Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-21T09:51:27.564 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.453253366Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-21T09:51:27.564 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.454187582Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-21T09:51:27.564 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.454980319Z level=info msg="Executing migration" id="create user table v2" 2024-09-21T09:51:27.564 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.455997448Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-21T09:51:27.564 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.456727899Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-21T09:51:27.564 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.457623774Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-21T09:51:27.564 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.458475387Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-21T09:51:27.564 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.459423264Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-21T09:51:27.565 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.460007865Z level=info msg="Executing migration" id="Update user table charset" 2024-09-21T09:51:27.565 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.460857263Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-21T09:51:27.565 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.461723985Z level=info msg="Executing migration" id="Add missing user data" 2024-09-21T09:51:27.565 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.462680723Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-21T09:51:27.565 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.463282989Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-21T09:51:27.565 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.464181392Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-21T09:51:27.565 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.464898267Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-21T09:51:27.565 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.466344394Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-21T09:51:27.565 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.467078803Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-21T09:51:27.565 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.468065064Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-21T09:51:27.565 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.468677722Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-21T09:51:27.565 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.469715Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-21T09:51:27.565 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.472501039Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-21T09:51:27.565 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.475196754Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-21T09:51:27.565 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.477975015Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-21T09:51:27.566 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.48061574Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-21T09:51:27.566 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.483325182Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-21T09:51:27.566 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.486207001Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-21T09:51:27.566 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.489022835Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-21T09:51:27.566 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.491919076Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-21T09:51:27.566 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.494641173Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-21T09:51:27.566 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.497559667Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-21T09:51:27.566 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.500336672Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-21T09:51:27.566 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.503132201Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-21T09:51:27.566 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.505805259Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-21T09:51:27.566 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.508567827Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-21T09:51:27.566 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.511495508Z level=info msg="Executing migration" id="create star table" 2024-09-21T09:51:27.566 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.514138532Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-21T09:51:27.566 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.560462732Z level=info msg="Executing migration" id="create org table v1" 2024-09-21T09:51:27.567 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.560959152Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-21T09:51:27.812 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.563772442Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-21T09:51:27.812 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.566405457Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-21T09:51:27.813 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.569059301Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-21T09:51:27.813 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.569468203Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-21T09:51:27.813 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.572233412Z level=info msg="Executing migration" id="Update org table charset" 2024-09-21T09:51:27.813 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.574976062Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-21T09:51:27.813 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.577439769Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-21T09:51:27.813 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.577748366Z level=info msg="Executing migration" id="create dashboard table" 2024-09-21T09:51:27.813 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.580618208Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-21T09:51:27.813 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.583379774Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-21T09:51:27.813 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.586011036Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-21T09:51:27.813 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.586417956Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-21T09:51:27.813 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.58917477Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-21T09:51:27.813 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.59193842Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-21T09:51:27.813 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.594448819Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-21T09:51:27.813 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.594861159Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-21T09:51:27.814 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.597604121Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-21T09:51:27.814 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.598453697Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-21T09:51:27.814 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.599326695Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-21T09:51:27.815 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.59971228Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-21T09:51:27.815 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.600589849Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-21T09:51:27.815 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.601333134Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-21T09:51:27.815 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.602157781Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-21T09:51:27.815 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.602673095Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-21T09:51:27.815 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.603606034Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-21T09:51:27.815 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.604382512Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-21T09:51:27.815 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.605250603Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-21T09:51:27.815 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.605664193Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-21T09:51:27.815 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.606404121Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-21T09:51:27.815 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.606709422Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-21T09:51:27.815 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.607301093Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-21T09:51:27.816 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.608129132Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-21T09:51:27.816 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.608928863Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-21T09:51:27.816 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.609712559Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-21T09:51:27.816 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.610573124Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-21T09:51:27.816 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.611535583Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-21T09:51:27.816 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.61208142Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-21T09:51:27.816 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.6129756Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-21T09:51:27.816 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.613970845Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-21T09:51:27.816 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.614637169Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-21T09:51:27.816 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.615600615Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-21T09:51:27.816 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.616620304Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-21T09:51:27.816 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.617356172Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-21T09:51:27.816 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.618151796Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-21T09:51:27.816 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.61899604Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-21T09:51:27.817 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.619918221Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-21T09:51:27.817 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.62084665Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-21T09:51:27.817 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.621554735Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-21T09:51:27.817 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.622375464Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-21T09:51:27.817 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.62270618Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-21T09:51:27.817 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.623288748Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-21T09:51:27.817 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.624176784Z level=info msg="Executing migration" id="create data_source table" 2024-09-21T09:51:27.817 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.627147763Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-21T09:51:27.817 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.630157643Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-21T09:51:27.817 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.632988614Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-21T09:51:27.817 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.635698546Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-21T09:51:27.817 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.638217754Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-21T09:51:27.817 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.641041533Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-21T09:51:27.818 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.643957472Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-21T09:51:27.818 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.64666796Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-21T09:51:27.818 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.649552267Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-21T09:51:27.818 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.649935528Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-21T09:51:27.818 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.650809256Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-21T09:51:27.818 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.651731178Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-21T09:51:27.818 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.652474854Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-21T09:51:27.818 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.653378401Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-21T09:51:27.818 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.654006554Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-21T09:51:27.818 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.654343183Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-21T09:51:27.818 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.655186349Z level=info msg="Executing migration" id="Add uid column" 2024-09-21T09:51:27.818 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.655738442Z level=info msg="Executing migration" id="Update uid value" 2024-09-21T09:51:27.818 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.656078458Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-21T09:51:27.818 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.657106387Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-21T09:51:27.819 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.657759434Z level=info msg="Executing migration" id="create api_key table" 2024-09-21T09:51:27.819 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.658765402Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-21T09:51:27.819 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.659488913Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-21T09:51:27.819 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.660511619Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-21T09:51:27.819 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.663320409Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-21T09:51:27.819 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.665987555Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-21T09:51:27.819 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.66639052Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-21T09:51:27.819 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.669195549Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-21T09:51:27.819 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.670271953Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-21T09:51:27.819 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.671907373Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-21T09:51:27.819 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.674573321Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-21T09:51:27.819 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.674985615Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-21T09:51:27.819 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.67793499Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-21T09:51:27.819 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.680474375Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-21T09:51:27.820 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.680951291Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-21T09:51:27.820 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.681238187Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-21T09:51:27.820 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.684026958Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-21T09:51:27.820 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.684589975Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-21T09:51:27.820 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.686921619Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-21T09:51:27.820 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.687454203Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-21T09:51:27.820 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.689643674Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-21T09:51:27.820 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.690064832Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-21T09:51:27.820 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.690446502Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-21T09:51:27.820 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.692995895Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-21T09:51:27.820 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.69345086Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-21T09:51:27.820 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.695643697Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-21T09:51:27.820 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.69815449Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-21T09:51:27.820 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.698468179Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-21T09:51:27.821 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.701126345Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-21T09:51:27.821 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.701680409Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-21T09:51:27.821 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.703861752Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-21T09:51:27.821 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.706605776Z level=info msg="Executing migration" id="create quota table v1" 2024-09-21T09:51:27.821 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.707079959Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-21T09:51:27.821 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.709682046Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-21T09:51:27.821 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.712111962Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-21T09:51:27.821 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.712548043Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-21T09:51:27.821 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.712961167Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-21T09:51:27.821 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.715458376Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-21T09:51:27.821 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.718136651Z level=info msg="Executing migration" id="create session table" 2024-09-21T09:51:27.821 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.718600774Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-21T09:51:27.821 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.718915204Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-21T09:51:27.821 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.719282591Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-21T09:51:27.822 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.71969832Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-21T09:51:27.822 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.722569401Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-21T09:51:27.822 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.722918204Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-21T09:51:27.822 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.723212388Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-21T09:51:27.822 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.72348612Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-21T09:51:27.822 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.723794843Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-21T09:51:27.822 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.724200596Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-21T09:51:27.822 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.725004996Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-21T09:51:27.822 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.725614708Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-21T09:51:27.822 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.725925789Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-21T09:51:27.822 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.726539723Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-21T09:51:27.822 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.727399577Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-21T09:51:27.822 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.728220661Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-21T09:51:27.822 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.728870159Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-21T09:51:27.823 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.729379322Z level=info msg="Executing migration" id="create alert table v1" 2024-09-21T09:51:27.823 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.730171738Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-21T09:51:27.823 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.730998545Z level=info msg="Executing migration" id="add index alert state" 2024-09-21T09:51:27.823 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.731422595Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-21T09:51:27.823 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.731904994Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-21T09:51:27.823 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.732295128Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-21T09:51:27.823 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.732779492Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-21T09:51:27.824 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.733224019Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-21T09:51:27.824 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.7346634Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-21T09:51:27.824 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.735061346Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-21T09:51:27.824 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.735514348Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-21T09:51:27.824 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.735821883Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-21T09:51:27.824 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.736202045Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-21T09:51:27.824 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.736567435Z level=info msg="Executing migration" id="Add column is_default" 2024-09-21T09:51:27.824 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.737194837Z level=info msg="Executing migration" id="Add column frequency" 2024-09-21T09:51:27.824 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.737811094Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-21T09:51:27.824 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.738432588Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-21T09:51:27.825 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.739088152Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-21T09:51:27.825 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.739525959Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-21T09:51:27.825 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.73979153Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-21T09:51:27.825 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.740101252Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-21T09:51:27.825 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.740511832Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-21T09:51:27.825 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.740989683Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-21T09:51:27.825 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.741377714Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-21T09:51:27.825 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.741743141Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-21T09:51:27.825 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.742162748Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-21T09:51:27.825 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.742795088Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-21T09:51:27.825 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.743456993Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-21T09:51:27.825 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.743781348Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-21T09:51:27.825 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.744231542Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-21T09:51:27.825 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.744654036Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-21T09:51:27.826 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.745330015Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-21T09:51:27.826 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.745634036Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-21T09:51:27.826 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.746069943Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-21T09:51:27.826 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.746504259Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-21T09:51:27.826 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.746813932Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-21T09:51:27.826 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.747264595Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-21T09:51:27.826 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.747685193Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-21T09:51:27.826 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.748117531Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-21T09:51:27.826 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.748556129Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-21T09:51:27.826 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.749054728Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-21T09:51:27.826 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.749523399Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-21T09:51:27.826 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.749789123Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-21T09:51:27.826 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.750450569Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-21T09:51:27.826 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.753567142Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-21T09:51:27.827 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.754381909Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-21T09:51:27.827 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.755243523Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-21T09:51:27.827 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.755908243Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-21T09:51:27.827 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.756907175Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-21T09:51:27.827 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.758631316Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-21T09:51:27.827 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.759044604Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-21T09:51:27.827 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.7600579Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-21T09:51:27.827 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.760687897Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-21T09:51:27.827 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.761568135Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-21T09:51:27.827 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.761887089Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-21T09:51:27.827 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.762738096Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-21T09:51:27.827 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.763593025Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-21T09:51:27.827 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.764458672Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-21T09:51:27.828 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.764879923Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-21T09:51:27.828 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.765714433Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-21T09:51:27.828 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.766532688Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-21T09:51:27.828 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.767417765Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-21T09:51:27.828 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.76773132Z level=info msg="Executing migration" id="Move region to single row" 2024-09-21T09:51:27.828 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.768507123Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-21T09:51:27.828 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.769487907Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-21T09:51:27.828 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.770206068Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-21T09:51:27.828 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.770644461Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-21T09:51:27.828 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.773596659Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-21T09:51:27.828 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.776244915Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-21T09:51:27.828 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.77892968Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-21T09:51:27.828 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.779247688Z level=info msg="Executing migration" id="create test_data table" 2024-09-21T09:51:27.828 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.782091653Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-21T09:51:27.829 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.78251941Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-21T09:51:27.829 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.78539955Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-21T09:51:27.829 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.785830496Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-21T09:51:27.829 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.788580479Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-21T09:51:27.829 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.791489534Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-21T09:51:27.829 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.793973128Z level=info msg="Executing migration" id="create team table" 2024-09-21T09:51:27.829 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.794621324Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-21T09:51:27.829 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.797244474Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-21T09:51:27.829 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.799966262Z level=info msg="Executing migration" id="create team member table" 2024-09-21T09:51:27.829 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.802626154Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-21T09:51:27.829 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.80325582Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-21T09:51:27.829 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.806035009Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-21T09:51:27.829 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.808630585Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-21T09:51:27.829 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.811133191Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-21T09:51:27.830 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.811861404Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-21T09:51:27.830 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.814569909Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-21T09:51:28.063 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.817467148Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-21T09:51:28.063 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.819996426Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-21T09:51:28.063 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.820419054Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-21T09:51:28.063 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.823289467Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-21T09:51:28.063 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.825983924Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-21T09:51:28.063 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.828694088Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-21T09:51:28.063 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.831134221Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-21T09:51:28.063 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.833850655Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-21T09:51:28.063 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.83667315Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-21T09:51:28.063 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.839157728Z level=info msg="Executing migration" id="create tag table" 2024-09-21T09:51:28.063 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.841934837Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-21T09:51:28.064 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.842754874Z level=info msg="Executing migration" id="create login attempt table" 2024-09-21T09:51:28.064 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.843710997Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-21T09:51:28.064 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.844411781Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-21T09:51:28.064 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.845259921Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-21T09:51:28.064 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.847117695Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-21T09:51:28.064 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.847964998Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-21T09:51:28.064 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.848798905Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-21T09:51:28.064 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.849719522Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-21T09:51:28.065 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.850468868Z level=info msg="Executing migration" id="create user auth table" 2024-09-21T09:51:28.065 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.851311046Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-21T09:51:28.065 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.852166074Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-21T09:51:28.065 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.853063723Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-21T09:51:28.065 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.853984088Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-21T09:51:28.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.854764495Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-21T09:51:28.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.855641139Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-21T09:51:28.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.85650045Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-21T09:51:28.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.857295672Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-21T09:51:28.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.858118332Z level=info msg="Executing migration" id="create server_lock table" 2024-09-21T09:51:28.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.859128574Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-21T09:51:28.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.860060474Z level=info msg="Executing migration" id="create user auth token table" 2024-09-21T09:51:28.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.860750641Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-21T09:51:28.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.861688577Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-21T09:51:28.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.862424517Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-21T09:51:28.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.86328751Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-21T09:51:28.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.86412774Z level=info msg="Executing migration" id="create cache_data table" 2024-09-21T09:51:28.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.864993364Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-21T09:51:28.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.865828008Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-21T09:51:28.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.866674536Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-21T09:51:28.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.867527511Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-21T09:51:28.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.868491076Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-21T09:51:28.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.86918132Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-21T09:51:28.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.870198001Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-21T09:51:28.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.870948141Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-21T09:51:28.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.871724834Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-21T09:51:28.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.872653647Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-21T09:51:28.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.875509097Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-21T09:51:28.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.878330496Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-21T09:51:28.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.881057145Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-21T09:51:28.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.886614846Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-21T09:51:28.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.887493448Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-21T09:51:28.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.887950195Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-21T09:51:28.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.890944456Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-21T09:51:28.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.893638794Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-21T09:51:28.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.89654298Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-21T09:51:28.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.897081023Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-21T09:51:28.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.899985339Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-21T09:51:28.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.902770821Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-21T09:51:28.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.905668024Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-21T09:51:28.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.906169713Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-21T09:51:28.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.909004516Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-21T09:51:28.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.911796633Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-21T09:51:28.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.914676123Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-21T09:51:28.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.917355106Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-21T09:51:28.069 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.921298074Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-21T09:51:28.069 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.924668913Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-21T09:51:28.069 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.927464315Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-21T09:51:28.069 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.928035442Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-21T09:51:28.069 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.930798247Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-21T09:51:28.069 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.933494677Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-21T09:51:28.069 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.936172448Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-21T09:51:28.069 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.936665516Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-21T09:51:28.069 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.939422778Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-21T09:51:28.069 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.942085849Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-21T09:51:28.069 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.944789527Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-21T09:51:28.069 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.945658028Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-21T09:51:28.069 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.948156073Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-21T09:51:28.069 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.950795597Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-21T09:51:28.070 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.953504337Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-21T09:51:28.070 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.954363805Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-21T09:51:28.070 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.957110539Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-21T09:51:28.070 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.957973762Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-21T09:51:28.070 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.958811112Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-21T09:51:28.070 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.959647707Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-21T09:51:28.070 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.960584604Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-21T09:51:28.070 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.961180805Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-21T09:51:28.070 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.961661864Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-21T09:51:28.070 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.962672974Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-21T09:51:28.070 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.963157134Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-21T09:51:28.070 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.964062557Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-21T09:51:28.070 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.964923588Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-21T09:51:28.070 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.96576117Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-21T09:51:28.071 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.966626852Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-21T09:51:28.071 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.967499959Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-21T09:51:28.071 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.968455114Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-21T09:51:28.071 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.969127081Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-21T09:51:28.071 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.970056802Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-21T09:51:28.071 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.97092452Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-21T09:51:28.071 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.971804233Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-21T09:51:28.071 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.972626811Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-21T09:51:28.071 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.973518834Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-21T09:51:28.071 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.974361054Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-21T09:51:28.071 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.975243958Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-21T09:51:28.071 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.976234178Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-21T09:51:28.071 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.976671596Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-21T09:51:28.071 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.97753607Z level=info msg="Executing migration" id="create alert_image table" 2024-09-21T09:51:28.072 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.978495799Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-21T09:51:28.072 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.97921425Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-21T09:51:28.072 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.979655258Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-21T09:51:28.072 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.980557983Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-21T09:51:28.072 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.981287894Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-21T09:51:28.072 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.981371243Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-21T09:51:28.072 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.982128342Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-21T09:51:28.072 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.982494113Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-21T09:51:28.072 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.983343723Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-21T09:51:28.072 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.983455232Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-21T09:51:28.072 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.983575964Z level=info msg="alerts found to migrate" alerts=0 2024-09-21T09:51:28.072 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.98401286Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-21T09:51:28.072 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.984465655Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-21T09:51:28.072 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.987327034Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-21T09:51:28.073 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.987764107Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-21T09:51:28.073 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.990475197Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-21T09:51:28.073 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.993232082Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-21T09:51:28.073 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.995840105Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-21T09:51:28.073 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.996269188Z level=info msg="Executing migration" id="create data_keys table" 2024-09-21T09:51:28.073 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:27.999045702Z level=info msg="Executing migration" id="create secrets table" 2024-09-21T09:51:28.073 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.001742383Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-21T09:51:28.073 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.00605423Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-21T09:51:28.073 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.006996543Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-21T09:51:28.073 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.009432294Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-21T09:51:28.073 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.013630135Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-21T09:51:28.073 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.017835787Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-21T09:51:28.073 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.01827863Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-21T09:51:28.073 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.02103843Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-21T09:51:28.073 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.023758096Z level=info msg="Executing migration" id="create permission table" 2024-09-21T09:51:28.074 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.026561661Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-21T09:51:28.074 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.02708972Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-21T09:51:28.074 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.02796194Z level=info msg="Executing migration" id="create role table" 2024-09-21T09:51:28.074 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.028715848Z level=info msg="Executing migration" id="add column display_name" 2024-09-21T09:51:28.074 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.029648576Z level=info msg="Executing migration" id="add column group_name" 2024-09-21T09:51:28.074 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.030624464Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-21T09:51:28.074 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.031479874Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-21T09:51:28.074 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.03226323Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-21T09:51:28.074 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.033079086Z level=info msg="Executing migration" id="create team role table" 2024-09-21T09:51:28.074 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.033500007Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-21T09:51:28.074 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.034295788Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-21T09:51:28.074 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.03516215Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-21T09:51:28.074 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.035838529Z level=info msg="Executing migration" id="create user role table" 2024-09-21T09:51:28.075 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.036258511Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-21T09:51:28.075 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.037037378Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-21T09:51:28.075 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.037957515Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-21T09:51:28.075 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.038597238Z level=info msg="Executing migration" id="create builtin role table" 2024-09-21T09:51:28.075 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.039395941Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-21T09:51:28.075 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.040386934Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-21T09:51:28.076 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.041130185Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-21T09:51:28.076 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.042153093Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-21T09:51:28.076 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.042730202Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-21T09:51:28.076 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.043482515Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-21T09:51:28.076 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.043894587Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-21T09:51:28.076 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.044320038Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-21T09:51:28.076 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.044695201Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-21T09:51:28.076 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.045130379Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-21T09:51:28.076 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.046094011Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-21T09:51:28.076 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.046493764Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-21T09:51:28.076 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.046943338Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-21T09:51:28.076 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.047227751Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-21T09:51:28.076 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.047603945Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-21T09:51:28.076 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.04805096Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-21T09:51:28.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.048405024Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-21T09:51:28.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.048673319Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-21T09:51:28.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.049058459Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-21T09:51:28.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.049441284Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-21T09:51:28.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.049862821Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-21T09:51:28.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.050811038Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-21T09:51:28.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.051115358Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-21T09:51:28.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.051525059Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-21T09:51:28.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.051969093Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-21T09:51:28.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.052412663Z level=info msg="Executing migration" id="add correlation config column" 2024-09-21T09:51:28.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.053396461Z level=info msg="Executing migration" id="create entity_events table" 2024-09-21T09:51:28.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.053817635Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-21T09:51:28.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.054608927Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-21T09:51:28.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.05470491Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-21T09:51:28.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.055016926Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T09:51:28.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.055116494Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T09:51:28.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.055412186Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-21T09:51:28.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.05580624Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-21T09:51:28.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.056980083Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-21T09:51:28.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.057552025Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T09:51:28.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.058129752Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-21T09:51:28.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.058598505Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-21T09:51:28.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.059025787Z level=info msg="Executing migration" id="Drop public config table" 2024-09-21T09:51:28.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.059428617Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-21T09:51:28.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.059817335Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-21T09:51:28.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.060248362Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-21T09:51:28.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.060689389Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-21T09:51:28.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.061502132Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-21T09:51:28.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.064512858Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-21T09:51:28.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.06550532Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-21T09:51:28.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.066487419Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-21T09:51:28.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.066859032Z level=info msg="Executing migration" id="add share column" 2024-09-21T09:51:28.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.06783785Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-21T09:51:28.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.068278689Z level=info msg="Executing migration" id="create file table" 2024-09-21T09:51:28.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.068694113Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-21T09:51:28.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.069493149Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-21T09:51:28.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.069961504Z level=info msg="Executing migration" id="create file_meta table" 2024-09-21T09:51:28.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.070793503Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-21T09:51:28.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.071240648Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-21T09:51:28.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.072085324Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-21T09:51:28.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.073037077Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-21T09:51:28.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.07377638Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-21T09:51:28.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.074209751Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-21T09:51:28.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.075216245Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-21T09:51:28.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.076083852Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-21T09:51:28.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.076923979Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-21T09:51:28.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.077832037Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-21T09:51:28.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.080771388Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-21T09:51:28.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.083385522Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-21T09:51:28.391 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.086145162Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-21T09:51:28.391 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.087167439Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-21T09:51:28.391 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.089932714Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-21T09:51:28.391 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.099844538Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-21T09:51:28.391 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.102631079Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-21T09:51:28.391 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.103179038Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-21T09:51:28.391 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.106594575Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-21T09:51:28.391 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.109268811Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-21T09:51:28.392 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.112039741Z level=info msg="Executing migration" id="create folder table" 2024-09-21T09:51:28.392 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.112561042Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-21T09:51:28.392 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.115604973Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-21T09:51:28.392 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.118092654Z level=info msg="Executing migration" id="Update folder title length" 2024-09-21T09:51:28.392 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.120813953Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-21T09:51:28.392 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=migrator t=2024-09-21T09:51:28.121303781Z level=info msg="migrations completed" performed=485 skipped=0 duration=671.165403ms 2024-09-21T09:51:28.392 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=sqlstore t=2024-09-21T09:51:28.121702821Z level=info msg="Created default organization" 2024-09-21T09:51:28.392 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=plugin.loader t=2024-09-21T09:51:28.179548801Z level=info msg="Plugin registered" pluginID=input 2024-09-21T09:51:28.392 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 bash[61265]: be7bf40319e5f8baccf8af3613c924d045b02c9ada77f4a8fb4a67ca718a0dc7 2024-09-21T09:51:28.392 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 systemd[1]: Started Ceph grafana.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:51:28.392 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=plugin.loader t=2024-09-21T09:51:28.207359362Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-21T09:51:28.392 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=plugin.loader t=2024-09-21T09:51:28.207388118Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-21T09:51:28.392 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=secrets t=2024-09-21T09:51:28.207428786Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-21T09:51:28.392 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=query_data t=2024-09-21T09:51:28.209215344Z level=info msg="Query Service initialization" 2024-09-21T09:51:28.393 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=live.push_http t=2024-09-21T09:51:28.211616109Z level=info msg="Live Push Gateway initialization" 2024-09-21T09:51:29.276 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=infra.usagestats.collector t=2024-09-21T09:51:28.992400968Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-21T09:51:29.276 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=provisioning.datasources t=2024-09-21T09:51:28.992732135Z level=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-09-21T09:51:29.276 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:29 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=provisioning.datasources t=2024-09-21T09:51:29.024854367Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-21T09:51:29.277 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:29 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=provisioning.datasources t=2024-09-21T09:51:29.05662497Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-21T09:51:29.277 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:29 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=provisioning.alerting t=2024-09-21T09:51:29.07823706Z level=info msg="starting to provision alerting" 2024-09-21T09:51:29.277 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:29 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=provisioning.alerting t=2024-09-21T09:51:29.078273319Z level=info msg="finished to provision alerting" 2024-09-21T09:51:29.277 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:29 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=grafanaStorageLogger t=2024-09-21T09:51:29.078438282Z level=info msg="storage starting" 2024-09-21T09:51:29.277 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:29 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=ngalert.state.manager t=2024-09-21T09:51:29.078508636Z level=info msg="Warming state cache for startup" 2024-09-21T09:51:29.277 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:29 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=http.server t=2024-09-21T09:51:29.080951552Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-21T09:51:29.277 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:29 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=sqlstore.transactions t=2024-09-21T09:51:29.088849961Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T09:51:29.277 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:29 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=ngalert.state.manager t=2024-09-21T09:51:29.089146489Z level=info msg="State cache has been initialized" states=0 duration=10.635933ms 2024-09-21T09:51:29.277 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:29 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=ticker t=2024-09-21T09:51:29.089273092Z level=info msg=starting first_tick=2024-09-21T09:51:30Z 2024-09-21T09:51:29.277 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:29 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=ngalert.multiorg.alertmanager t=2024-09-21T09:51:29.089338127Z level=info msg="starting MultiOrg Alertmanager" 2024-09-21T09:51:29.277 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:29 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=sqlstore.transactions t=2024-09-21T09:51:29.128876694Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T09:51:29.277 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:29 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=sqlstore.transactions t=2024-09-21T09:51:29.160738919Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T09:51:29.277 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:51:29 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[61279]: logger=sqlstore.transactions t=2024-09-21T09:51:29.171140253Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-21T09:51:29.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:29 smithi098 ceph-mon[29239]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:29.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:29 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:29.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:29 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:29.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:29 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:29.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:29 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:29.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:29 smithi098 ceph-mon[29239]: Deploying daemon node-exporter.a on smithi098 2024-09-21T09:51:29.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:29 smithi098 ceph-mon[39960]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:29.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:29 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:29.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:29 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:29.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:29 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:29.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:29 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:29.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:29 smithi098 ceph-mon[39960]: Deploying daemon node-exporter.a on smithi098 2024-09-21T09:51:29.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:29 smithi196 ceph-mon[32670]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:29.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:29 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:29.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:29 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:29.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:29 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:29.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:29 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:29.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:29 smithi196 ceph-mon[32670]: Deploying daemon node-exporter.a on smithi098 2024-09-21T09:51:30.460 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:30 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:51:30.460 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:30 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:30.461 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:30 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:51:30.461 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:30 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:30.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:30 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:51:30.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:30 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:31.529 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:31 smithi098 ceph-mon[39960]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:31.530 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:31 smithi098 ceph-mon[29239]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:31.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:31 smithi196 ceph-mon[32670]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:33.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:33 smithi098 ceph-mon[29239]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:33.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:33 smithi098 ceph-mon[39960]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:33.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:33 smithi196 ceph-mon[32670]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:35.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:35 smithi098 ceph-mon[29239]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:35.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:35 smithi098 ceph-mon[39960]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:35.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:35 smithi196 ceph-mon[32670]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:35.842 INFO:teuthology.orchestra.run.smithi098.stdout:111669149730 2024-09-21T09:51:35.842 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T09:51:36.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:51:36 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:51:36] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:51:36.988 INFO:teuthology.orchestra.run.smithi098.stdout:77309411368 2024-09-21T09:51:36.988 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T09:51:37.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:37 smithi098 ceph-mon[29239]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:37.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:37 smithi098 ceph-mon[39960]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:37 smithi196 ceph-mon[32670]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:37.906 INFO:teuthology.orchestra.run.smithi098.stdout:55834574892 2024-09-21T09:51:37.906 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T09:51:39.315 INFO:teuthology.orchestra.run.smithi098.stdout:34359738417 2024-09-21T09:51:39.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T09:51:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:39 smithi196 ceph-mon[32670]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:39.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:39 smithi098 ceph-mon[29239]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:39.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:39 smithi098 ceph-mon[39960]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:40.896 INFO:teuthology.orchestra.run.smithi098.stdout:133143986206 2024-09-21T09:51:40.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T09:51:41.558 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:41 smithi098 ceph-mon[29239]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:41.558 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:41 smithi098 ceph-mon[39960]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:41.559 INFO:teuthology.orchestra.run.smithi098.stdout:184683593748 2024-09-21T09:51:41.559 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T09:51:41.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:41 smithi196 ceph-mon[32670]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:41.948 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:41.949 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:41.950 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:42.760 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:51:42 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[70687]: ts=2024-09-21T09:51:42.659Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T09:51:42.760 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:51:42 smithi098 bash[70280]: 20cd79611b0362b56c8bad8dcc0a64480eb00efa8643cefddacd10040739d2be 2024-09-21T09:51:43.073 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:51:42 smithi098 systemd[1]: Started Ceph node-exporter.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:51:43.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:43 smithi098 ceph-mon[39960]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:43.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:43 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:43.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:43 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:43.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:43 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:43.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:43 smithi098 ceph-mon[29239]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:43.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:43 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:43.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:43 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:43.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:43 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:43.514 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:43 smithi196 ceph-mon[32670]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:43.514 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:43 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:43.514 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:43 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:43.514 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:43 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:43.889 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:43 smithi196 bash[61545]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-21T09:51:44.304 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:44 smithi196 bash[61545]: Getting image source signatures 2024-09-21T09:51:44.304 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:44 smithi196 bash[61545]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-21T09:51:44.304 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:44 smithi196 bash[61545]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-21T09:51:44.304 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:44 smithi196 bash[61545]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-21T09:51:44.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:44 smithi098 ceph-mon[29239]: Deploying daemon node-exporter.b on smithi196 2024-09-21T09:51:44.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:44 smithi098 ceph-mon[39960]: Deploying daemon node-exporter.b on smithi196 2024-09-21T09:51:44.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:44 smithi196 ceph-mon[32670]: Deploying daemon node-exporter.b on smithi196 2024-09-21T09:51:45.329 INFO:teuthology.orchestra.run.smithi098.stdout:210453397521 2024-09-21T09:51:45.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T09:51:45.554 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:45 smithi098 ceph-mon[29239]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:45.554 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:45 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:51:45.555 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:45 smithi098 ceph-mon[39960]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:45.555 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:45 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:51:45.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:45 smithi196 ceph-mon[32670]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:45 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:51:45.639 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:45 smithi196 bash[61545]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-21T09:51:45.639 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:45 smithi196 bash[61545]: Writing manifest to image destination 2024-09-21T09:51:46.086 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:45 smithi196 podman[61545]: 2024-09-21 09:51:45.767870008 +0000 UTC m=+2.160204421 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-21T09:51:46.086 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:45 smithi196 podman[61545]: 2024-09-21 09:51:45.874997138 +0000 UTC m=+2.267331545 container create 5f6e262b57853085b863d3a9ffaec4a79a7180f1ea92ca868701f2d59e383909 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:51:46.087 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 podman[61545]: 2024-09-21 09:51:46.08531958 +0000 UTC m=+2.477653999 container init 5f6e262b57853085b863d3a9ffaec4a79a7180f1ea92ca868701f2d59e383909 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:51:46.373 INFO:teuthology.orchestra.run.smithi098.stdout:111669149733 2024-09-21T09:51:46.388 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 podman[61545]: 2024-09-21 09:51:46.088799737 +0000 UTC m=+2.481134142 container start 5f6e262b57853085b863d3a9ffaec4a79a7180f1ea92ca868701f2d59e383909 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:51:46.389 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.093Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-21T09:51:46.389 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.093Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-21T09:51:46.389 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.093Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T09:51:46.389 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.093Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T09:51:46.389 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T09:51:46.389 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T09:51:46.389 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T09:51:46.389 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T09:51:46.389 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T09:51:46.389 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T09:51:46.389 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T09:51:46.390 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T09:51:46.390 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T09:51:46.390 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T09:51:46.390 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T09:51:46.390 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T09:51:46.390 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T09:51:46.390 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T09:51:46.391 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T09:51:46.391 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T09:51:46.391 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T09:51:46.391 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T09:51:46.391 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T09:51:46.391 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T09:51:46.391 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T09:51:46.391 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T09:51:46.391 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T09:51:46.391 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T09:51:46.391 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T09:51:46.391 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T09:51:46.391 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T09:51:46.391 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T09:51:46.392 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T09:51:46.392 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T09:51:46.392 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T09:51:46.392 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T09:51:46.392 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T09:51:46.392 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T09:51:46.392 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T09:51:46.392 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T09:51:46.392 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T09:51:46.392 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T09:51:46.392 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T09:51:46.392 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T09:51:46.392 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T09:51:46.392 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T09:51:46.392 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T09:51:46.393 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T09:51:46.393 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T09:51:46.393 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T09:51:46.393 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T09:51:46.393 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-21T09:51:46.393 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[61623]: ts=2024-09-21T09:51:46.094Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T09:51:46.393 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 bash[61545]: 5f6e262b57853085b863d3a9ffaec4a79a7180f1ea92ca868701f2d59e383909 2024-09-21T09:51:46.393 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:51:46 smithi196 systemd[1]: Started Ceph node-exporter.b for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:51:46.712 INFO:teuthology.orchestra.run.smithi098.stdout:158913789979 2024-09-21T09:51:46.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T09:51:46.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:51:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:51:46] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:51:47.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:47 smithi098 ceph-mon[39960]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:47.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:47 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:47.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:47 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:47.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:47 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:47.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:47 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:47.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:47 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:47.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:47 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:47.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:47 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3007710803' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:51:47.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:47 smithi098 ceph-mon[29239]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:47.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:47 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:47.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:47 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:47.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:47 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:47.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:47 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:47.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:47 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:47.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:47 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:47.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:47 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3007710803' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:51:47.473 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149730 got 111669149733 for osd.3 2024-09-21T09:51:47.474 DEBUG:teuthology.parallel:result is None 2024-09-21T09:51:47.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:47 smithi196 ceph-mon[32670]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:47 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:47 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:47 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:47 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:47 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:47 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:47 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3007710803' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:51:49.012 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:49.012 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:49.014 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:49.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:49 smithi098 ceph-mon[29239]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:49.506 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:49 smithi098 ceph-mon[39960]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:49.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:49 smithi196 ceph-mon[32670]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:50.651 INFO:teuthology.orchestra.run.smithi098.stdout:55834574895 2024-09-21T09:51:50.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:50 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:50.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:50 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:50.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:50 smithi098 ceph-mon[29239]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:50.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:50 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:50.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:50 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/1542309603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:51:50.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:50 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:50.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:50 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:50.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:50 smithi098 ceph-mon[39960]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:50.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:50 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:50.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:50 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/1542309603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:51:51.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:50 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:51.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:50 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:50 smithi196 ceph-mon[32670]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:50 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:51:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:50 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/1542309603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:51:51.505 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:52.261 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574892 got 55834574895 for osd.1 2024-09-21T09:51:52.261 DEBUG:teuthology.parallel:result is None 2024-09-21T09:51:53.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:52 smithi098 ceph-mon[29239]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:53.208 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:52 smithi098 ceph-mon[39960]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:53.316 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:53.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:52 smithi196 ceph-mon[32670]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:55.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:54 smithi098 ceph-mon[29239]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:55.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:54 smithi098 ceph-mon[39960]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:55.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:54 smithi196 ceph-mon[32670]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:55.770 INFO:teuthology.orchestra.run.smithi098.stdout:34359738420 2024-09-21T09:51:55.956 INFO:teuthology.orchestra.run.smithi098.stdout:133143986211 2024-09-21T09:51:55.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:55 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2825557666' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:51:55.957 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:55 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2825557666' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:51:56.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:55 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2825557666' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:51:56.420 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738417 got 34359738420 for osd.0 2024-09-21T09:51:56.421 DEBUG:teuthology.parallel:result is None 2024-09-21T09:51:56.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:51:56 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:51:56] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:51:57.011 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:56 smithi098 ceph-mon[29239]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:57.012 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:56 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/1191301423' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:51:57.012 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:56 smithi098 ceph-mon[39960]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:57.012 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:56 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/1191301423' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:51:57.035 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986206 got 133143986211 for osd.4 2024-09-21T09:51:57.035 DEBUG:teuthology.parallel:result is None 2024-09-21T09:51:57.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:56 smithi196 ceph-mon[32670]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:57.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:56 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/1191301423' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:51:58.385 INFO:teuthology.orchestra.run.smithi098.stdout:184683593752 2024-09-21T09:51:59.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:58 smithi098 ceph-mon[29239]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:59.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:58 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3074012705' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:51:59.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:58 smithi098 ceph-mon[39960]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:59.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:58 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3074012705' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:51:59.209 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593748 got 184683593752 for osd.6 2024-09-21T09:51:59.209 DEBUG:teuthology.parallel:result is None 2024-09-21T09:51:59.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:58 smithi196 ceph-mon[32670]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:51:59.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:58 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3074012705' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:51:59.998 INFO:teuthology.orchestra.run.smithi098.stdout:77309411372 2024-09-21T09:51:59.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:51:59 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:51:59.998 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:51:59 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:52:00.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:51:59 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:52:00.818 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411368 got 77309411372 for osd.2 2024-09-21T09:52:00.818 DEBUG:teuthology.parallel:result is None 2024-09-21T09:52:01.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:00 smithi098 ceph-mon[29239]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:01.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:00 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2120969346' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:52:01.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:00 smithi098 ceph-mon[39960]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:01.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:00 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2120969346' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:52:01.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:00 smithi196 ceph-mon[32670]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:01.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:00 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2120969346' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:52:02.260 INFO:teuthology.orchestra.run.smithi098.stdout:158913789983 2024-09-21T09:52:02.449 INFO:teuthology.orchestra.run.smithi098.stdout:210453397525 2024-09-21T09:52:03.021 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:02 smithi098 ceph-mon[29239]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:03.022 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:02 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/723671342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:52:03.022 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:02 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/4162880182' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:52:03.022 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:02 smithi098 ceph-mon[39960]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:03.022 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:02 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/723671342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:52:03.022 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:02 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/4162880182' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:52:03.044 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789979 got 158913789983 for osd.5 2024-09-21T09:52:03.044 DEBUG:teuthology.parallel:result is None 2024-09-21T09:52:03.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:02 smithi196 ceph-mon[32670]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:03.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:02 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/723671342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:52:03.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:02 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/4162880182' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:52:04.151 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397521 got 210453397525 for osd.7 2024-09-21T09:52:04.152 DEBUG:teuthology.parallel:result is None 2024-09-21T09:52:04.152 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T09:52:04.152 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T09:52:04.954 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:05.191 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:04 smithi196 ceph-mon[32670]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:05.191 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:04 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:05.191 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:04 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:05.196 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:04 smithi098 ceph-mon[29239]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:05.196 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:04 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:05.196 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:04 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:05.196 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:04 smithi098 ceph-mon[39960]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:05.196 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:04 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:05.197 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:04 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:06.043 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:05 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:06.043 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:05 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:06.043 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:05 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:06.043 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:05 smithi098 ceph-mon[39960]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T09:52:06.044 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:05 smithi098 ceph-mon[39960]: Reconfiguring daemon alertmanager.a on smithi098 2024-09-21T09:52:06.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:05 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:06.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:05 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:06.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:05 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:06.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:05 smithi098 ceph-mon[29239]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T09:52:06.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:05 smithi098 ceph-mon[29239]: Reconfiguring daemon alertmanager.a on smithi098 2024-09-21T09:52:06.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:05 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:05 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:05 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:05 smithi196 ceph-mon[32670]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T09:52:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:05 smithi196 ceph-mon[32670]: Reconfiguring daemon alertmanager.a on smithi098 2024-09-21T09:52:06.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:52:06 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:52:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:52:07.472 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:52:07.473 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T09:52:07.473 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:07 smithi098 ceph-mon[29239]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:07.473 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:07 smithi098 ceph-mon[39960]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:07.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:07 smithi196 ceph-mon[32670]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:09.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:09 smithi098 ceph-mon[29239]: from='client.24494 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:52:09.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:09 smithi098 ceph-mon[29239]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:09.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:09 smithi098 ceph-mon[39960]: from='client.24494 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:52:09.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:09 smithi098 ceph-mon[39960]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:09.348 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:09 smithi098 systemd[1]: Stopping Ceph alertmanager.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:52:09.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:09 smithi196 ceph-mon[32670]: from='client.24494 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:52:09.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:09 smithi196 ceph-mon[32670]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:10.099 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-09-21T09:52:05.837544+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":115,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385164,"kb_used_data":6316,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747576052,"statfs":{"total":767960285184,"available":765517877248,"internally_reserved":0,"allocated":6467584,"data_stored":3738920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001972"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:51:19.554273+0000","last_change":"2024-09-21T09:51:10.174279+0000","last_active":"2024-09-21T09:51:19.554273+0000","last_peered":"2024-09-21T09:51:19.554273+0000","last_clean":"2024-09-21T09:51:19.554273+0000","last_became_active":"2024-09-21T09:51:05.133025+0000","last_became_peered":"2024-09-21T09:51:05.133025+0000","last_unstale":"2024-09-21T09:51:19.554273+0000","last_undegraded":"2024-09-21T09:51:19.554273+0000","last_fullsized":"2024-09-21T09:51:19.554273+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:47.204645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029041499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:51:18.756261+0000","last_change":"2024-09-21T09:51:10.174262+0000","last_active":"2024-09-21T09:51:18.756261+0000","last_peered":"2024-09-21T09:51:18.756261+0000","last_clean":"2024-09-21T09:51:18.756261+0000","last_became_active":"2024-09-21T09:51:05.429405+0000","last_became_peered":"2024-09-21T09:51:05.429405+0000","last_unstale":"2024-09-21T09:51:18.756261+0000","last_undegraded":"2024-09-21T09:51:18.756261+0000","last_fullsized":"2024-09-21T09:51:18.756261+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:57:49.920758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000351963,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:51:18.806808+0000","last_change":"2024-09-21T09:51:11.140616+0000","last_active":"2024-09-21T09:51:18.806808+0000","last_peered":"2024-09-21T09:51:18.806808+0000","last_clean":"2024-09-21T09:51:18.806808+0000","last_became_active":"2024-09-21T09:51:05.429200+0000","last_became_peered":"2024-09-21T09:51:05.429200+0000","last_unstale":"2024-09-21T09:51:18.806808+0000","last_undegraded":"2024-09-21T09:51:18.806808+0000","last_fullsized":"2024-09-21T09:51:18.806808+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:29:34.651310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031258499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:51:18.756275+0000","last_change":"2024-09-21T09:51:10.174269+0000","last_active":"2024-09-21T09:51:18.756275+0000","last_peered":"2024-09-21T09:51:18.756275+0000","last_clean":"2024-09-21T09:51:18.756275+0000","last_became_active":"2024-09-21T09:51:05.134200+0000","last_became_peered":"2024-09-21T09:51:05.134200+0000","last_unstale":"2024-09-21T09:51:18.756275+0000","last_undegraded":"2024-09-21T09:51:18.756275+0000","last_fullsized":"2024-09-21T09:51:18.756275+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:38:40.009638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036908299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":21,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:51:19.554217+0000","last_change":"2024-09-21T09:51:10.173383+0000","last_active":"2024-09-21T09:51:19.554217+0000","last_peered":"2024-09-21T09:51:19.554217+0000","last_clean":"2024-09-21T09:51:19.554217+0000","last_became_active":"2024-09-21T09:51:05.134269+0000","last_became_peered":"2024-09-21T09:51:05.134269+0000","last_unstale":"2024-09-21T09:51:19.554217+0000","last_undegraded":"2024-09-21T09:51:19.554217+0000","last_fullsized":"2024-09-21T09:51:19.554217+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:23.879314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091501400000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:51:19.554082+0000","last_change":"2024-09-21T09:51:10.168481+0000","last_active":"2024-09-21T09:51:19.554082+0000","last_peered":"2024-09-21T09:51:19.554082+0000","last_clean":"2024-09-21T09:51:19.554082+0000","last_became_active":"2024-09-21T09:51:05.134406+0000","last_became_peered":"2024-09-21T09:51:05.134406+0000","last_unstale":"2024-09-21T09:51:19.554082+0000","last_undegraded":"2024-09-21T09:51:19.554082+0000","last_fullsized":"2024-09-21T09:51:19.554082+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:14:34.828436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000364877,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:51:18.806815+0000","last_change":"2024-09-21T09:51:11.140693+0000","last_active":"2024-09-21T09:51:18.806815+0000","last_peered":"2024-09-21T09:51:18.806815+0000","last_clean":"2024-09-21T09:51:18.806815+0000","last_became_active":"2024-09-21T09:51:05.134160+0000","last_became_peered":"2024-09-21T09:51:05.134160+0000","last_unstale":"2024-09-21T09:51:18.806815+0000","last_undegraded":"2024-09-21T09:51:18.806815+0000","last_fullsized":"2024-09-21T09:51:18.806815+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:20.004781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028573500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":20,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:51:19.554192+0000","last_change":"2024-09-21T09:51:10.172804+0000","last_active":"2024-09-21T09:51:19.554192+0000","last_peered":"2024-09-21T09:51:19.554192+0000","last_clean":"2024-09-21T09:51:19.554192+0000","last_became_active":"2024-09-21T09:51:05.135491+0000","last_became_peered":"2024-09-21T09:51:05.135491+0000","last_unstale":"2024-09-21T09:51:19.554192+0000","last_undegraded":"2024-09-21T09:51:19.554192+0000","last_fullsized":"2024-09-21T09:51:19.554192+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:27:30.172694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034123299999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"52'79","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-21T09:51:11.140280+0000","last_change":"2024-09-21T09:50:29.757187+0000","last_active":"2024-09-21T09:51:11.140280+0000","last_peered":"2024-09-21T09:51:11.140280+0000","last_clean":"2024-09-21T09:51:11.140280+0000","last_became_active":"2024-09-21T09:50:29.751734+0000","last_became_peered":"2024-09-21T09:50:29.751734+0000","last_unstale":"2024-09-21T09:51:11.140280+0000","last_undegraded":"2024-09-21T09:51:11.140280+0000","last_fullsized":"2024-09-21T09:51:11.140280+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T09:48:34.850872+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:20:45.636784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397526,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298508,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446644,"statfs":{"total":95995035648,"available":95689363456,"internally_reserved":0,"allocated":1179648,"data_stored":836456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:51:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.56699999999999995}]},{"osd":1,"last update":"Sat Sep 21 09:51:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.45500000000000002}]},{"osd":2,"last update":"Sat Sep 21 09:51:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.49099999999999999}]},{"osd":3,"last update":"Sat Sep 21 09:51:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.442}]},{"osd":4,"last update":"Sat Sep 21 09:51:28 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.51100000000000001}]},{"osd":5,"last update":"Sat Sep 21 09:51:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.52800000000000002}]},{"osd":6,"last update":"Sat Sep 21 09:51:28 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.42599999999999999}]}]},{"osd":6,"up_from":43,"seq":184683593754,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298512,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446640,"statfs":{"total":95995035648,"available":95689359360,"internally_reserved":0,"allocated":1183744,"data_stored":836475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:52:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48499999999999999,"15min":0.48399999999999999},"min":{"1min":0.35399999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.498,"5min":0.48599999999999999,"15min":0.48399999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.57299999999999995}]},{"osd":1,"last update":"Sat Sep 21 09:52:05 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51000000000000001,"15min":0.51100000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.63900000000000001,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.49199999999999999,"15min":0.48999999999999999},"min":{"1min":0.34300000000000003,"5min":0.309,"15min":0.309},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.66200000000000003}]},{"osd":2,"last update":"Sat Sep 21 09:52:05 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.505,"15min":0.501},"min":{"1min":0.41299999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.65000000000000002,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.499,"15min":0.497},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.62}]},{"osd":3,"last update":"Sat Sep 21 09:52:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.498,"15min":0.49399999999999999},"min":{"1min":0.39200000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.50800000000000001,"15min":0.504},"min":{"1min":0.38600000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.59899999999999998}]},{"osd":4,"last update":"Sat Sep 21 09:52:05 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.39300000000000002,"15min":0.38400000000000001},"min":{"1min":0.33900000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.42699999999999999,"15min":0.41899999999999998},"min":{"1min":0.32200000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.55700000000000005}]},{"osd":5,"last update":"Sat Sep 21 09:52:05 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.40300000000000002,"15min":0.39700000000000002},"min":{"1min":0.28999999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.40200000000000002,"15min":0.38800000000000001},"min":{"1min":0.30399999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.52200000000000002}]},{"osd":7,"last update":"Sat Sep 21 09:51:27 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.42499999999999999}]}]},{"osd":5,"up_from":37,"seq":158913789984,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":585728,"data_stored":245792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:51:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.51900000000000002,"15min":0.51100000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70899999999999996,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.505,"15min":0.504},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.63300000000000001}]},{"osd":1,"last update":"Sat Sep 21 09:51:41 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.53700000000000003,"15min":0.53100000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71699999999999997,"5min":0.748,"15min":0.748},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.51600000000000001,"15min":0.51200000000000001},"min":{"1min":0.36299999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.60099999999999998}]},{"osd":2,"last update":"Sat Sep 21 09:51:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.51300000000000001,"15min":0.51000000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70299999999999996,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.53500000000000003,"15min":0.52600000000000002},"min":{"1min":0.41699999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73699999999999999,"5min":0.747,"15min":0.747},"last":0.71999999999999997}]},{"osd":3,"last update":"Sat Sep 21 09:51:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.47999999999999998,"15min":0.47199999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55700000000000005,"15min":0.55200000000000005},"min":{"1min":0.42299999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.70399999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.73899999999999999}]},{"osd":4,"last update":"Sat Sep 21 09:51:41 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.372,"15min":0.36099999999999999},"min":{"1min":0.30199999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.439},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.38200000000000001,"15min":0.36799999999999999},"min":{"1min":0.30199999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.52200000000000002}]},{"osd":6,"last update":"Sat Sep 21 09:51:03 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.57699999999999996}]},{"osd":7,"last update":"Sat Sep 21 09:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.55100000000000005}]}]},{"osd":4,"up_from":31,"seq":133143986213,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":585728,"data_stored":246088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70999999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.37,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.75600000000000001,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.53500000000000003}]},{"osd":1,"last update":"Sat Sep 21 09:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.53700000000000003,"15min":0.53900000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.71499999999999997,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.746,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.52000000000000002}]},{"osd":2,"last update":"Sat Sep 21 09:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55900000000000005,"15min":0.56499999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.67200000000000004,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.502,"5min":0.50600000000000001,"15min":0.50700000000000001},"min":{"1min":0.35299999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.69999999999999996,"5min":0.879,"15min":0.879},"last":0.48199999999999998}]},{"osd":3,"last update":"Sat Sep 21 09:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53200000000000003,"15min":0.53600000000000003},"min":{"1min":0.377,"5min":0.375,"15min":0.375},"max":{"1min":0.66200000000000003,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.502},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.378,"5min":0.375,"15min":0.375},"max":{"1min":0.75700000000000001,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.54900000000000004}]},{"osd":5,"last update":"Sat Sep 21 09:51:41 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.38200000000000001,"15min":0.377},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.39300000000000002,"15min":0.39100000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.74199999999999999,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.35699999999999998}]},{"osd":6,"last update":"Sat Sep 21 09:52:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.42399999999999999,"15min":0.41599999999999998},"min":{"1min":0.315,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.67200000000000004,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.42299999999999999,"15min":0.41499999999999998},"min":{"1min":0.30199999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.372}]},{"osd":7,"last update":"Sat Sep 21 09:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.434}]}]},{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":585728,"data_stored":246088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35699999999999998},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.53800000000000003,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.379},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.39400000000000002,"15min":0.39600000000000002},"min":{"1min":0.29899999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.52500000000000002,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.34699999999999998}]},{"osd":1,"last update":"Sat Sep 21 09:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.35799999999999998,"15min":0.35699999999999998},"min":{"1min":0.23899999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.48999999999999999,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.36599999999999999,"15min":0.36799999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.33000000000000002}]},{"osd":2,"last update":"Sat Sep 21 09:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41299999999999998,"15min":0.40799999999999997},"min":{"1min":0.30099999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.41099999999999998,"15min":0.41799999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.56000000000000005,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.30099999999999999}]},{"osd":4,"last update":"Sat Sep 21 09:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54400000000000004,"15min":0.54300000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.51800000000000002,"15min":0.51200000000000001},"min":{"1min":0.41699999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.627}]},{"osd":5,"last update":"Sat Sep 21 09:51:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.52700000000000002,"15min":0.51600000000000001},"min":{"1min":0.42599999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55200000000000005,"15min":0.54600000000000004},"min":{"1min":0.41899999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.64800000000000002}]},{"osd":6,"last update":"Sat Sep 21 09:51:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.67800000000000005}]},{"osd":7,"last update":"Sat Sep 21 09:51:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.66000000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":585728,"data_stored":246088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35199999999999998,"15min":0.34799999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.25},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.35099999999999998,"15min":0.34200000000000003},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.311}]},{"osd":1,"last update":"Sat Sep 21 09:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40100000000000002,"15min":0.39300000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.40999999999999998,"15min":0.40400000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.32700000000000001}]},{"osd":3,"last update":"Sat Sep 21 09:52:02 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39600000000000002,"15min":0.39400000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.54000000000000004,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.40699999999999997,"15min":0.41099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.56499999999999995,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.34000000000000002}]},{"osd":4,"last update":"Sat Sep 21 09:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54200000000000004,"15min":0.53900000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.55500000000000005,"15min":0.55200000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.44900000000000001}]},{"osd":5,"last update":"Sat Sep 21 09:51:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.497},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80800000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.437}]},{"osd":6,"last update":"Sat Sep 21 09:51:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.53500000000000003}]},{"osd":7,"last update":"Sat Sep 21 09:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.46000000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297924,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447228,"statfs":{"total":95995035648,"available":95689961472,"internally_reserved":0,"allocated":581632,"data_stored":245477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33600000000000002,"15min":0.32600000000000001},"min":{"1min":0.21199999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.47199999999999998,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.373},{"interface":"front","average":{"1min":0.38,"5min":0.34399999999999997,"15min":0.32500000000000001},"min":{"1min":0.22900000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.32400000000000001}]},{"osd":2,"last update":"Sat Sep 21 09:51:41 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.39200000000000002,"15min":0.39000000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.432},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.378,"15min":0.376},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.53000000000000003,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.29799999999999999}]},{"osd":3,"last update":"Sat Sep 21 09:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.42999999999999999,"15min":0.434},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.432,"15min":0.435},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.40799999999999997}]},{"osd":4,"last update":"Sat Sep 21 09:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.56200000000000006,"15min":0.56799999999999995},"min":{"1min":0.41999999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.63300000000000001},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.64900000000000002,"15min":0.58899999999999997},"min":{"1min":0.45400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":8.4100000000000001,"5min":8.4100000000000001,"15min":8.4100000000000001},"last":0.71599999999999997}]},{"osd":5,"last update":"Sat Sep 21 09:51:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57499999999999996,"15min":0.57899999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.80100000000000005,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.63800000000000001}]},{"osd":6,"last update":"Sat Sep 21 09:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57999999999999996,"15min":0.57799999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.92700000000000005,"15min":0.98499999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.876,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.60599999999999998}]},{"osd":7,"last update":"Sat Sep 21 09:51:28 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.65800000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298508,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446644,"statfs":{"total":95995035648,"available":95689363456,"internally_reserved":0,"allocated":1179648,"data_stored":836456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 09:51:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.308,"15min":0.28799999999999998},"min":{"1min":0.20799999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.44900000000000001,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.33700000000000002,"15min":0.31900000000000001},"min":{"1min":0.23799999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.34499999999999997}]},{"osd":2,"last update":"Sat Sep 21 09:51:35 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.34899999999999998,"15min":0.34499999999999997},"min":{"1min":0.249,"5min":0.22,"15min":0.22},"max":{"1min":0.49399999999999999,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.378,"15min":0.37},"min":{"1min":0.27600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.47099999999999997}]},{"osd":3,"last update":"Sat Sep 21 09:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41499999999999998,"15min":0.41099999999999998},"min":{"1min":0.318,"5min":0.254,"15min":0.254},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.438},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.39100000000000001,"15min":0.38700000000000001},"min":{"1min":0.27100000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.57199999999999995,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.317}]},{"osd":4,"last update":"Sat Sep 21 09:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":1.1499999999999999,"15min":1.254},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.70299999999999996,"5min":15.444000000000001,"15min":15.444000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.505,"5min":0.51100000000000001,"15min":0.51200000000000001},"min":{"1min":0.39800000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.628,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.64000000000000001}]},{"osd":5,"last update":"Sat Sep 21 09:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.749,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.41099999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.59399999999999997}]},{"osd":6,"last update":"Sat Sep 21 09:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.45500000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55000000000000004,"15min":0.54800000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.55700000000000005}]},{"osd":7,"last update":"Sat Sep 21 09:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.621},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.51800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T09:52:10.100 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T09:52:10.730 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[68424]: ts=2024-09-21T09:52:10.418Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-21T09:52:10.730 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:10 smithi098 podman[72650]: 2024-09-21 09:52:10.438737156 +0000 UTC m=+0.143455212 container died 31b7c93087d5da407e07433409ff954986cc6afa3bb567fddc4c5993cd7239b7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:52:10.901 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:11.071 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:10 smithi098 podman[72650]: 2024-09-21 09:52:10.81848699 +0000 UTC m=+0.523205033 container cleanup 31b7c93087d5da407e07433409ff954986cc6afa3bb567fddc4c5993cd7239b7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:52:11.071 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:10 smithi098 bash[72650]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a 2024-09-21T09:52:11.324 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:11 smithi098 podman[72677]: 2024-09-21 09:52:11.069744458 +0000 UTC m=+0.629609401 container remove 31b7c93087d5da407e07433409ff954986cc6afa3bb567fddc4c5993cd7239b7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:52:11.324 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:11 smithi098 podman[72677]: 2024-09-21 09:52:11.145315133 +0000 UTC m=+0.705180032 volume remove 5e2c9bc9a8ceb4f7b6db272a663a1e389c5a93118b0101317fe9a957877f4d6c 2024-09-21T09:52:11.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:11 smithi098 ceph-mon[29239]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:11.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:11 smithi098 ceph-mon[39960]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:11.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:11 smithi196 ceph-mon[32670]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:11.847 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:11 smithi098 podman[72703]: 2024-09-21 09:52:11.574502483 +0000 UTC m=+0.732368255 container died 5aefeffbebbb3f333d294087a0271be4930f2ce2e3a6ef5f9b14bd56af7940df (image=quay.ceph.io/ceph-ci/ceph:quincy, name=peaceful_haibt, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=quincy-fbd1a6a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T09:52:12.539 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:12 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-21T09:52:12.539 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:12 smithi098 systemd[1]: Stopped Ceph alertmanager.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:52:12.539 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:12 smithi098 systemd[1]: Starting Ceph alertmanager.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:52:12.910 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:52:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T09:52:13.326 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:12 smithi098 podman[72865]: 2024-09-21 09:52:12.829722637 +0000 UTC m=+0.015572244 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-21T09:52:13.327 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:12 smithi098 podman[72865]: 2024-09-21 09:52:12.934925441 +0000 UTC m=+0.120775034 volume create 58ec910acbe9cbe9c8e97e60a1b9a7b61b7b87b75871d370cbf4cad9a4e68ae2 2024-09-21T09:52:13.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:13 smithi098 ceph-mon[39960]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:13.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:13 smithi098 ceph-mon[29239]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:13.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:13 smithi196 ceph-mon[32670]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:13.673 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2024-09-21T09:52:11.838312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":115,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385164,"kb_used_data":6316,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747576052,"statfs":{"total":767960285184,"available":765517877248,"internally_reserved":0,"allocated":6467584,"data_stored":3738920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001736"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:51:19.554273+0000","last_change":"2024-09-21T09:51:10.174279+0000","last_active":"2024-09-21T09:51:19.554273+0000","last_peered":"2024-09-21T09:51:19.554273+0000","last_clean":"2024-09-21T09:51:19.554273+0000","last_became_active":"2024-09-21T09:51:05.133025+0000","last_became_peered":"2024-09-21T09:51:05.133025+0000","last_unstale":"2024-09-21T09:51:19.554273+0000","last_undegraded":"2024-09-21T09:51:19.554273+0000","last_fullsized":"2024-09-21T09:51:19.554273+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:47.204645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029041499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:51:18.756261+0000","last_change":"2024-09-21T09:51:10.174262+0000","last_active":"2024-09-21T09:51:18.756261+0000","last_peered":"2024-09-21T09:51:18.756261+0000","last_clean":"2024-09-21T09:51:18.756261+0000","last_became_active":"2024-09-21T09:51:05.429405+0000","last_became_peered":"2024-09-21T09:51:05.429405+0000","last_unstale":"2024-09-21T09:51:18.756261+0000","last_undegraded":"2024-09-21T09:51:18.756261+0000","last_fullsized":"2024-09-21T09:51:18.756261+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:57:49.920758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000351963,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:51:18.806808+0000","last_change":"2024-09-21T09:51:11.140616+0000","last_active":"2024-09-21T09:51:18.806808+0000","last_peered":"2024-09-21T09:51:18.806808+0000","last_clean":"2024-09-21T09:51:18.806808+0000","last_became_active":"2024-09-21T09:51:05.429200+0000","last_became_peered":"2024-09-21T09:51:05.429200+0000","last_unstale":"2024-09-21T09:51:18.806808+0000","last_undegraded":"2024-09-21T09:51:18.806808+0000","last_fullsized":"2024-09-21T09:51:18.806808+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:29:34.651310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031258499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:51:18.756275+0000","last_change":"2024-09-21T09:51:10.174269+0000","last_active":"2024-09-21T09:51:18.756275+0000","last_peered":"2024-09-21T09:51:18.756275+0000","last_clean":"2024-09-21T09:51:18.756275+0000","last_became_active":"2024-09-21T09:51:05.134200+0000","last_became_peered":"2024-09-21T09:51:05.134200+0000","last_unstale":"2024-09-21T09:51:18.756275+0000","last_undegraded":"2024-09-21T09:51:18.756275+0000","last_fullsized":"2024-09-21T09:51:18.756275+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:38:40.009638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036908299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":21,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:51:19.554217+0000","last_change":"2024-09-21T09:51:10.173383+0000","last_active":"2024-09-21T09:51:19.554217+0000","last_peered":"2024-09-21T09:51:19.554217+0000","last_clean":"2024-09-21T09:51:19.554217+0000","last_became_active":"2024-09-21T09:51:05.134269+0000","last_became_peered":"2024-09-21T09:51:05.134269+0000","last_unstale":"2024-09-21T09:51:19.554217+0000","last_undegraded":"2024-09-21T09:51:19.554217+0000","last_fullsized":"2024-09-21T09:51:19.554217+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:23.879314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091501400000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:51:19.554082+0000","last_change":"2024-09-21T09:51:10.168481+0000","last_active":"2024-09-21T09:51:19.554082+0000","last_peered":"2024-09-21T09:51:19.554082+0000","last_clean":"2024-09-21T09:51:19.554082+0000","last_became_active":"2024-09-21T09:51:05.134406+0000","last_became_peered":"2024-09-21T09:51:05.134406+0000","last_unstale":"2024-09-21T09:51:19.554082+0000","last_undegraded":"2024-09-21T09:51:19.554082+0000","last_fullsized":"2024-09-21T09:51:19.554082+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:14:34.828436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000364877,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:51:18.806815+0000","last_change":"2024-09-21T09:51:11.140693+0000","last_active":"2024-09-21T09:51:18.806815+0000","last_peered":"2024-09-21T09:51:18.806815+0000","last_clean":"2024-09-21T09:51:18.806815+0000","last_became_active":"2024-09-21T09:51:05.134160+0000","last_became_peered":"2024-09-21T09:51:05.134160+0000","last_unstale":"2024-09-21T09:51:18.806815+0000","last_undegraded":"2024-09-21T09:51:18.806815+0000","last_fullsized":"2024-09-21T09:51:18.806815+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:20.004781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028573500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":20,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:51:19.554192+0000","last_change":"2024-09-21T09:51:10.172804+0000","last_active":"2024-09-21T09:51:19.554192+0000","last_peered":"2024-09-21T09:51:19.554192+0000","last_clean":"2024-09-21T09:51:19.554192+0000","last_became_active":"2024-09-21T09:51:05.135491+0000","last_became_peered":"2024-09-21T09:51:05.135491+0000","last_unstale":"2024-09-21T09:51:19.554192+0000","last_undegraded":"2024-09-21T09:51:19.554192+0000","last_fullsized":"2024-09-21T09:51:19.554192+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:27:30.172694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034123299999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"52'79","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-21T09:51:11.140280+0000","last_change":"2024-09-21T09:50:29.757187+0000","last_active":"2024-09-21T09:51:11.140280+0000","last_peered":"2024-09-21T09:51:11.140280+0000","last_clean":"2024-09-21T09:51:11.140280+0000","last_became_active":"2024-09-21T09:50:29.751734+0000","last_became_peered":"2024-09-21T09:50:29.751734+0000","last_unstale":"2024-09-21T09:51:11.140280+0000","last_undegraded":"2024-09-21T09:51:11.140280+0000","last_fullsized":"2024-09-21T09:51:11.140280+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T09:48:34.850872+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:20:45.636784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397527,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298508,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446644,"statfs":{"total":95995035648,"available":95689363456,"internally_reserved":0,"allocated":1179648,"data_stored":836456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:51:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.38400000000000001}]},{"osd":1,"last update":"Sat Sep 21 09:51:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.442}]},{"osd":2,"last update":"Sat Sep 21 09:51:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.45900000000000002}]},{"osd":3,"last update":"Sat Sep 21 09:51:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.56499999999999995}]},{"osd":4,"last update":"Sat Sep 21 09:51:28 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.315},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.34200000000000003}]},{"osd":5,"last update":"Sat Sep 21 09:51:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.52800000000000002}]},{"osd":6,"last update":"Sat Sep 21 09:51:28 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.50700000000000001}]}]},{"osd":6,"up_from":43,"seq":184683593755,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298512,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446640,"statfs":{"total":95995035648,"available":95689359360,"internally_reserved":0,"allocated":1183744,"data_stored":836475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:52:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48499999999999999,"15min":0.48399999999999999},"min":{"1min":0.35399999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.48599999999999999,"15min":0.48399999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.36799999999999999}]},{"osd":1,"last update":"Sat Sep 21 09:52:05 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51000000000000001,"15min":0.51100000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.63900000000000001,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.49199999999999999,"15min":0.48999999999999999},"min":{"1min":0.34300000000000003,"5min":0.309,"15min":0.309},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.42599999999999999}]},{"osd":2,"last update":"Sat Sep 21 09:52:05 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.505,"15min":0.501},"min":{"1min":0.41299999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.65000000000000002,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.499,"15min":0.497},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.438}]},{"osd":3,"last update":"Sat Sep 21 09:52:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.498,"15min":0.49399999999999999},"min":{"1min":0.39200000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.50800000000000001,"15min":0.504},"min":{"1min":0.38600000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.52700000000000002}]},{"osd":4,"last update":"Sat Sep 21 09:52:05 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.39300000000000002,"15min":0.38400000000000001},"min":{"1min":0.33900000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.42699999999999999,"15min":0.41899999999999998},"min":{"1min":0.32200000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.45700000000000002}]},{"osd":5,"last update":"Sat Sep 21 09:52:05 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.40300000000000002,"15min":0.39700000000000002},"min":{"1min":0.28999999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.40200000000000002,"15min":0.38800000000000001},"min":{"1min":0.30399999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.40600000000000003}]},{"osd":7,"last update":"Sat Sep 21 09:51:27 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.34999999999999998}]}]},{"osd":5,"up_from":37,"seq":158913789985,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":585728,"data_stored":245792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:51:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.51900000000000002,"15min":0.51100000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70899999999999996,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.623},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.505,"15min":0.504},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.502}]},{"osd":1,"last update":"Sat Sep 21 09:51:41 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.53700000000000003,"15min":0.53100000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71699999999999997,"5min":0.748,"15min":0.748},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.51600000000000001,"15min":0.51200000000000001},"min":{"1min":0.36299999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.52700000000000002}]},{"osd":2,"last update":"Sat Sep 21 09:51:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.51300000000000001,"15min":0.51000000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70299999999999996,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.53500000000000003,"15min":0.52600000000000002},"min":{"1min":0.41699999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73699999999999999,"5min":0.747,"15min":0.747},"last":0.46100000000000002}]},{"osd":3,"last update":"Sat Sep 21 09:51:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.47999999999999998,"15min":0.47199999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55700000000000005,"15min":0.55200000000000005},"min":{"1min":0.42299999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.70399999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.53800000000000003}]},{"osd":4,"last update":"Sat Sep 21 09:51:41 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.372,"15min":0.36099999999999999},"min":{"1min":0.30199999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.38200000000000001,"15min":0.36799999999999999},"min":{"1min":0.30199999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.42799999999999999}]},{"osd":6,"last update":"Sat Sep 21 09:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.40799999999999997,"15min":0.39000000000000001},"min":{"1min":0.371,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.379},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.45600000000000002,"15min":0.434},"min":{"1min":0.38500000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.60599999999999998}]},{"osd":7,"last update":"Sat Sep 21 09:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.47499999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986214,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":585728,"data_stored":246088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70999999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.37,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.75600000000000001,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.45800000000000002}]},{"osd":1,"last update":"Sat Sep 21 09:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.53700000000000003,"15min":0.53900000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.71499999999999997,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.746,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.52300000000000002}]},{"osd":2,"last update":"Sat Sep 21 09:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55900000000000005,"15min":0.56499999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.67200000000000004,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.502,"5min":0.50600000000000001,"15min":0.50700000000000001},"min":{"1min":0.35299999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.69999999999999996,"5min":0.879,"15min":0.879},"last":0.48699999999999999}]},{"osd":3,"last update":"Sat Sep 21 09:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53200000000000003,"15min":0.53600000000000003},"min":{"1min":0.377,"5min":0.375,"15min":0.375},"max":{"1min":0.66200000000000003,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.378,"5min":0.375,"15min":0.375},"max":{"1min":0.75700000000000001,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.42299999999999999}]},{"osd":5,"last update":"Sat Sep 21 09:51:41 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.38200000000000001,"15min":0.377},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.39300000000000002,"15min":0.39100000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.74199999999999999,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.497}]},{"osd":6,"last update":"Sat Sep 21 09:52:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.42399999999999999,"15min":0.41599999999999998},"min":{"1min":0.315,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.67200000000000004,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.42299999999999999,"15min":0.41499999999999998},"min":{"1min":0.30199999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.35899999999999999}]},{"osd":7,"last update":"Sat Sep 21 09:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.439},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.50700000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":585728,"data_stored":246088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35699999999999998},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.53800000000000003,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.39400000000000002,"15min":0.39600000000000002},"min":{"1min":0.29899999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.52500000000000002,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.253}]},{"osd":1,"last update":"Sat Sep 21 09:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.35799999999999998,"15min":0.35699999999999998},"min":{"1min":0.23899999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.48999999999999999,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.36599999999999999,"15min":0.36799999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.33200000000000002}]},{"osd":2,"last update":"Sat Sep 21 09:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41299999999999998,"15min":0.40799999999999997},"min":{"1min":0.30099999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.41099999999999998,"15min":0.41799999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.56000000000000005,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.313}]},{"osd":4,"last update":"Sat Sep 21 09:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54400000000000004,"15min":0.54300000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.51800000000000002,"15min":0.51200000000000001},"min":{"1min":0.41699999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.64900000000000002}]},{"osd":5,"last update":"Sat Sep 21 09:51:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.52700000000000002,"15min":0.51600000000000001},"min":{"1min":0.42599999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55200000000000005,"15min":0.54600000000000004},"min":{"1min":0.41899999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.51100000000000001}]},{"osd":6,"last update":"Sat Sep 21 09:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.39400000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.73799999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.55500000000000005,"15min":0.55100000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56499999999999995}]},{"osd":7,"last update":"Sat Sep 21 09:51:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.60999999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":585728,"data_stored":246088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35199999999999998,"15min":0.34799999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.35099999999999998,"15min":0.34200000000000003},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.27100000000000002}]},{"osd":1,"last update":"Sat Sep 21 09:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40100000000000002,"15min":0.39300000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.40999999999999998,"15min":0.40400000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.32400000000000001}]},{"osd":3,"last update":"Sat Sep 21 09:52:02 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39600000000000002,"15min":0.39400000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.54000000000000004,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.31},{"interface":"front","average":{"1min":0.371,"5min":0.40699999999999997,"15min":0.41099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.56499999999999995,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.34100000000000003}]},{"osd":4,"last update":"Sat Sep 21 09:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54200000000000004,"15min":0.53900000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.55500000000000005,"15min":0.55200000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.63100000000000001}]},{"osd":5,"last update":"Sat Sep 21 09:51:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80800000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.64100000000000001}]},{"osd":6,"last update":"Sat Sep 21 09:52:08 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":0.70799999999999996,"15min":0.629},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":10.569000000000001,"5min":10.569000000000001,"15min":10.569000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.57299999999999995,"15min":0.57999999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.78700000000000003,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.60999999999999999}]},{"osd":7,"last update":"Sat Sep 21 09:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.65400000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297924,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447228,"statfs":{"total":95995035648,"available":95689961472,"internally_reserved":0,"allocated":581632,"data_stored":245477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33600000000000002,"15min":0.32600000000000001},"min":{"1min":0.21199999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.47199999999999998,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.38,"5min":0.34399999999999997,"15min":0.32500000000000001},"min":{"1min":0.22900000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.35299999999999998}]},{"osd":2,"last update":"Sat Sep 21 09:51:41 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.39200000000000002,"15min":0.39000000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.434},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.378,"15min":0.376},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.53000000000000003,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.39100000000000001}]},{"osd":3,"last update":"Sat Sep 21 09:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.42999999999999999,"15min":0.434},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.432,"15min":0.435},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.41299999999999998}]},{"osd":4,"last update":"Sat Sep 21 09:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.56200000000000006,"15min":0.56799999999999995},"min":{"1min":0.41999999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.60899999999999999},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.64900000000000002,"15min":0.58899999999999997},"min":{"1min":0.45400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":8.4100000000000001,"5min":8.4100000000000001,"15min":8.4100000000000001},"last":0.68799999999999994}]},{"osd":5,"last update":"Sat Sep 21 09:51:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57499999999999996,"15min":0.57899999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.80100000000000005,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.67700000000000005}]},{"osd":6,"last update":"Sat Sep 21 09:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57999999999999996,"15min":0.57799999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.628},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.92700000000000005,"15min":0.98499999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.876,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.64900000000000002}]},{"osd":7,"last update":"Sat Sep 21 09:51:28 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.66100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298508,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446644,"statfs":{"total":95995035648,"available":95689363456,"internally_reserved":0,"allocated":1179648,"data_stored":836456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 09:51:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.308,"15min":0.28799999999999998},"min":{"1min":0.20799999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.44900000000000001,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.375,"5min":0.33700000000000002,"15min":0.31900000000000001},"min":{"1min":0.23799999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.35899999999999999}]},{"osd":2,"last update":"Sat Sep 21 09:51:35 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.34899999999999998,"15min":0.34499999999999997},"min":{"1min":0.249,"5min":0.22,"15min":0.22},"max":{"1min":0.49399999999999999,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.378,"15min":0.37},"min":{"1min":0.27600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.30499999999999999}]},{"osd":3,"last update":"Sat Sep 21 09:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41499999999999998,"15min":0.41099999999999998},"min":{"1min":0.318,"5min":0.254,"15min":0.254},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.39100000000000001,"15min":0.38700000000000001},"min":{"1min":0.27100000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.57199999999999995,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.32300000000000001}]},{"osd":4,"last update":"Sat Sep 21 09:51:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":1.1499999999999999,"15min":1.254},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.70299999999999996,"5min":15.444000000000001,"15min":15.444000000000001},"last":0.436},{"interface":"front","average":{"1min":0.505,"5min":0.51100000000000001,"15min":0.51200000000000001},"min":{"1min":0.39800000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.628,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.49199999999999999}]},{"osd":5,"last update":"Sat Sep 21 09:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.749,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.41099999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.41499999999999998}]},{"osd":6,"last update":"Sat Sep 21 09:52:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.45500000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55000000000000004,"15min":0.54800000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.45700000000000002}]},{"osd":7,"last update":"Sat Sep 21 09:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.51900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T09:52:13.675 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T09:52:13.675 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T09:52:13.675 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-21T09:52:13.675 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph health --format=json 2024-09-21T09:52:13.949 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:13 smithi098 podman[72865]: 2024-09-21 09:52:13.773498081 +0000 UTC m=+0.959347670 container create 9ae44cc24819ab5583009a5147d8f3dbada4d905d3c96cc4700032b023c48a7c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:52:14.235 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:14.335 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:14 smithi098 podman[72865]: 2024-09-21 09:52:14.0685938 +0000 UTC m=+1.254443405 container init 9ae44cc24819ab5583009a5147d8f3dbada4d905d3c96cc4700032b023c48a7c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:52:14.335 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:14 smithi098 podman[72865]: 2024-09-21 09:52:14.071429156 +0000 UTC m=+1.257278747 container start 9ae44cc24819ab5583009a5147d8f3dbada4d905d3c96cc4700032b023c48a7c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:52:14.335 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:14 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[72927]: ts=2024-09-21T09:52:14.082Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-21T09:52:14.335 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:14 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[72927]: ts=2024-09-21T09:52:14.083Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-21T09:52:14.335 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:14 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[72927]: ts=2024-09-21T09:52:14.084Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.98 port=9094 2024-09-21T09:52:14.335 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:14 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[72927]: ts=2024-09-21T09:52:14.085Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-21T09:52:14.335 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:14 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[72927]: ts=2024-09-21T09:52:14.107Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T09:52:14.336 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:14 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[72927]: ts=2024-09-21T09:52:14.108Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T09:52:14.336 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:14 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[72927]: ts=2024-09-21T09:52:14.111Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-21T09:52:14.336 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:14 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[72927]: ts=2024-09-21T09:52:14.111Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-21T09:52:14.336 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:14 smithi098 bash[72865]: 9ae44cc24819ab5583009a5147d8f3dbada4d905d3c96cc4700032b023c48a7c 2024-09-21T09:52:14.336 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:14 smithi098 systemd[1]: Started Ceph alertmanager.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:52:14.593 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:14 smithi098 ceph-mon[39960]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:52:14.593 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:14 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:14.594 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:14 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:14.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:14 smithi098 ceph-mon[29239]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:52:14.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:14 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:14.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:14 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:14.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:14 smithi196 ceph-mon[32670]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:52:14.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:14 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:14.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:14 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:15.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:15 smithi098 ceph-mon[29239]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:15.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:15 smithi098 ceph-mon[29239]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:52:15.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:15 smithi098 ceph-mon[29239]: Reconfiguring daemon prometheus.a on smithi196 2024-09-21T09:52:15.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:15 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:52:15.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:15 smithi098 ceph-mon[39960]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:15.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:15 smithi098 ceph-mon[39960]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:52:15.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:15 smithi098 ceph-mon[39960]: Reconfiguring daemon prometheus.a on smithi196 2024-09-21T09:52:15.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:15 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:52:15.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:15 smithi196 ceph-mon[32670]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:15.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:15 smithi196 ceph-mon[32670]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:52:15.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:15 smithi196 ceph-mon[32670]: Reconfiguring daemon prometheus.a on smithi196 2024-09-21T09:52:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:15 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:52:16.236 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:52:16.236 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T09:52:16.236 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:16 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[72927]: ts=2024-09-21T09:52:16.085Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000407827s 2024-09-21T09:52:16.554 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:16 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2948965946' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:52:16.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:16 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2948965946' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:52:16.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:16 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2948965946' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:52:16.881 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:52:16 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:52:16] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:52:17.016 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-21T09:52:17.016 INFO:tasks.cephadm:Setup complete, yielding 2024-09-21T09:52:17.016 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-21T09:52:17.025 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-09-21T09:52:17.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- bash -c 'ceph fs volume create foo' 2024-09-21T09:52:17.277 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:17.568 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:17 smithi196 ceph-mon[32670]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:17.568 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:17 smithi196 systemd[1]: Stopping Ceph prometheus.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:52:17.592 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:17 smithi098 ceph-mon[29239]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:17.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:17 smithi098 ceph-mon[39960]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:17.820 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:17 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[59307]: ts=2024-09-21T09:52:17.566Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T09:52:17.820 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:17 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[59307]: ts=2024-09-21T09:52:17.566Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-21T09:52:17.820 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:17 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[59307]: ts=2024-09-21T09:52:17.566Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-21T09:52:17.820 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:17 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[59307]: ts=2024-09-21T09:52:17.566Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T09:52:17.820 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:17 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[59307]: ts=2024-09-21T09:52:17.566Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-21T09:52:17.820 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:17 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[59307]: ts=2024-09-21T09:52:17.566Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T09:52:17.821 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:17 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[59307]: ts=2024-09-21T09:52:17.566Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-21T09:52:17.821 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:17 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[59307]: ts=2024-09-21T09:52:17.566Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-21T09:52:17.821 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:17 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[59307]: ts=2024-09-21T09:52:17.567Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-21T09:52:17.821 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:17 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[59307]: ts=2024-09-21T09:52:17.567Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T09:52:17.821 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:17 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[59307]: ts=2024-09-21T09:52:17.567Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-21T09:52:17.821 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:17 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[59307]: ts=2024-09-21T09:52:17.567Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-21T09:52:17.821 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:17 smithi196 podman[62377]: 2024-09-21 09:52:17.577123797 +0000 UTC m=+0.105404090 container died d5828454fa60028edbec2cfa6f2c7ccf1311c4777f0f2a915afdb94762ffdd26 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:52:18.110 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:17 smithi196 podman[62377]: 2024-09-21 09:52:17.89956877 +0000 UTC m=+0.427849068 container cleanup d5828454fa60028edbec2cfa6f2c7ccf1311c4777f0f2a915afdb94762ffdd26 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:52:18.110 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:17 smithi196 bash[62377]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T09:52:18.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 podman[62389]: 2024-09-21 09:52:18.109195149 +0000 UTC m=+0.530743384 container remove d5828454fa60028edbec2cfa6f2c7ccf1311c4777f0f2a915afdb94762ffdd26 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:52:18.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T09:52:18.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 systemd[1]: Stopped Ceph prometheus.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:52:18.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Consumed 1.013s CPU time. 2024-09-21T09:52:18.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 systemd[1]: Starting Ceph prometheus.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:52:18.807 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 podman[62482]: 2024-09-21 09:52:18.437594273 +0000 UTC m=+0.016677125 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-21T09:52:18.807 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 podman[62482]: 2024-09-21 09:52:18.560226085 +0000 UTC m=+0.139308938 container create 93f520945f7b39a8f14a902593e50f8177aeb3dc71fdc6dce5ae7f8c5cebc91b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:52:18.807 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 podman[62482]: 2024-09-21 09:52:18.779139795 +0000 UTC m=+0.358222640 container init 93f520945f7b39a8f14a902593e50f8177aeb3dc71fdc6dce5ae7f8c5cebc91b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:52:18.807 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 podman[62482]: 2024-09-21 09:52:18.78179071 +0000 UTC m=+0.360873556 container start 93f520945f7b39a8f14a902593e50f8177aeb3dc71fdc6dce5ae7f8c5cebc91b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:52:19.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:52:18.805Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-21T09:52:19.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:52:18.805Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-21T09:52:19.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:52:18.805Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi196 (none))" 2024-09-21T09:52:19.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:52:18.805Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T09:52:19.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:52:18.805Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T09:52:19.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:52:18.808Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T09:52:19.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:52:18.808Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-21T09:52:19.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:52:18.809Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T09:52:19.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:52:18.809Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T09:52:19.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:52:18.811Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T09:52:19.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:52:18.811Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=4.548µs 2024-09-21T09:52:19.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:52:18.811Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T09:52:19.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:52:18.812Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-21T09:52:19.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:52:18.812Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-21T09:52:19.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:52:18.812Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=70.356µs wal_replay_duration=1.03505ms wbl_replay_duration=86ns total_replay_duration=1.303279ms 2024-09-21T09:52:19.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:52:18.814Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T09:52:19.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:52:18.814Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-21T09:52:19.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:52:18.814Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T09:52:19.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:52:18.828Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.510193ms db_storage=722ns remote_storage=1.082µs web_handler=362ns query_engine=557ns scrape=383.287µs scrape_sd=25.312µs notify=24.65µs notify_sd=6.087µs rules=13.834251ms tracing=5.972µs 2024-09-21T09:52:19.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:52:18.828Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-21T09:52:19.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:52:18.828Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T09:52:19.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 bash[62482]: 93f520945f7b39a8f14a902593e50f8177aeb3dc71fdc6dce5ae7f8c5cebc91b 2024-09-21T09:52:19.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:52:18 smithi196 systemd[1]: Started Ceph prometheus.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:52:19.297 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:52:19] ENGINE Bus STOPPING 2024-09-21T09:52:19.298 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:52:19] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:52:19.298 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:52:19] ENGINE Bus STOPPED 2024-09-21T09:52:19.298 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:52:19] ENGINE Bus STARTING 2024-09-21T09:52:19.298 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:52:19] ENGINE Serving on http://:::9283 2024-09-21T09:52:19.298 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:52:19] ENGINE Bus STARTED 2024-09-21T09:52:19.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:52:19] ENGINE Bus STOPPING 2024-09-21T09:52:19.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[29239]: pgmap v49: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:19.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-21T09:52:19.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:19.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:19.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:52:19.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi098.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:52:19.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:19.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:52:19.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:52:19.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:19.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:52:19.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:52:19.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:19.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:19.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[39960]: pgmap v49: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:19.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-21T09:52:19.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:19.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:19.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:52:19.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi098.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:52:19.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:19.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:52:19.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:52:19.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:19.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:52:19.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:52:19.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:19.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:19.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:19 smithi196 ceph-mon[32670]: pgmap v49: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:19.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:19 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-21T09:52:19.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:19 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:19 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:19 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:52:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:19 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi098.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:52:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:19 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:19 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:52:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:19 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:52:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:19 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:19 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:52:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:19 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:52:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:19 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:19 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:19.905 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:52:19] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:52:19.905 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:52:19] ENGINE Bus STOPPED 2024-09-21T09:52:19.905 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:52:19] ENGINE Bus STARTING 2024-09-21T09:52:19.905 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:52:19] ENGINE Serving on http://:::9283 2024-09-21T09:52:19.905 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:52:19] ENGINE Bus STARTED 2024-09-21T09:52:20.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:52:19] ENGINE Bus STOPPING 2024-09-21T09:52:20.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:52:19] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:52:20.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:52:19] ENGINE Bus STOPPED 2024-09-21T09:52:20.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:52:19 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:52:19] ENGINE Bus STARTING 2024-09-21T09:52:20.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:52:20] ENGINE Serving on http://:::9283 2024-09-21T09:52:20.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:52:20] ENGINE Bus STARTED 2024-09-21T09:52:20.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:20 smithi196 ceph-mon[32670]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:20 smithi196 ceph-mon[32670]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:52:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:20 smithi196 ceph-mon[32670]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi098.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:52:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:20 smithi196 ceph-mon[32670]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:52:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:20 smithi196 ceph-mon[32670]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:52:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:20 smithi196 ceph-mon[32670]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:52:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:20 smithi196 ceph-mon[32670]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:52:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:20 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-21T09:52:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:20 smithi196 ceph-mon[32670]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T09:52:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:20 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-21T09:52:20.708 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-mon[39960]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:20.708 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-mon[39960]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:52:20.708 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-mon[39960]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi098.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:52:20.708 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-mon[39960]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:52:20.708 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-mon[39960]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:52:20.708 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-mon[39960]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:52:20.708 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-mon[39960]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:52:20.708 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-21T09:52:20.708 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-mon[39960]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T09:52:20.708 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-21T09:52:20.708 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-mon[29239]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:20.709 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-mon[29239]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:52:20.709 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-mon[29239]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi098.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:52:20.709 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-mon[29239]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:52:20.709 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-mon[29239]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:52:20.709 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-mon[29239]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:52:20.709 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-mon[29239]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:52:20.709 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-21T09:52:20.709 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-mon[29239]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T09:52:20.709 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-21T09:52:20.709 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a[29235]: 2024-09-21T09:52:20.375+0000 7fc966092640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T09:52:21.023 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-09-21T09:52:21.282 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:21.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:21 smithi196 ceph-mon[32670]: pgmap v51: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:21 smithi196 ceph-mon[32670]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:52:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:21 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-21T09:52:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:21 smithi196 ceph-mon[32670]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T09:52:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:21 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-21T09:52:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:21 smithi196 ceph-mon[32670]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T09:52:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:21 smithi196 ceph-mon[32670]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T09:52:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:21 smithi196 ceph-mon[32670]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T09:52:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:21 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-21T09:52:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:21 smithi196 ceph-mon[32670]: fsmap foo:0 2024-09-21T09:52:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:21 smithi196 ceph-mon[32670]: Saving service mds.foo spec with placement count:2 2024-09-21T09:52:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:21 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:21.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[29239]: pgmap v51: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:21.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[29239]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:52:21.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-21T09:52:21.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[29239]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T09:52:21.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-21T09:52:21.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[29239]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T09:52:21.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[29239]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T09:52:21.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[29239]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T09:52:21.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-21T09:52:21.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[29239]: fsmap foo:0 2024-09-21T09:52:21.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[29239]: Saving service mds.foo spec with placement count:2 2024-09-21T09:52:21.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:21.751 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[39960]: pgmap v51: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:21.752 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[39960]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:52:21.752 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-21T09:52:21.752 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[39960]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T09:52:21.752 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-21T09:52:21.752 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[39960]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T09:52:21.752 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[39960]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T09:52:21.752 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[39960]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T09:52:21.752 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-21T09:52:21.752 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[39960]: fsmap foo:0 2024-09-21T09:52:21.752 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[39960]: Saving service mds.foo spec with placement count:2 2024-09-21T09:52:21.752 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:21 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:22.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:22 smithi196 ceph-mon[32670]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T09:52:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:22 smithi098 ceph-mon[29239]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T09:52:22.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:22 smithi098 ceph-mon[39960]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T09:52:23.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:23 smithi098 ceph-mon[39960]: pgmap v55: 73 pgs: 7 creating+peering, 66 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:23.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:23 smithi098 ceph-mon[39960]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:52:23.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:23 smithi098 ceph-mon[39960]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T09:52:23.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:23 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:23.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:23 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:23.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:23 smithi098 ceph-mon[29239]: pgmap v55: 73 pgs: 7 creating+peering, 66 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:23.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:23 smithi098 ceph-mon[29239]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:52:23.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:23 smithi098 ceph-mon[29239]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T09:52:23.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:23 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:23.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:23 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:23.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:23 smithi196 ceph-mon[32670]: pgmap v55: 73 pgs: 7 creating+peering, 66 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:23.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:23 smithi196 ceph-mon[32670]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:52:23.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:23 smithi196 ceph-mon[32670]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T09:52:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:23 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:23 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:24.347 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:52:24 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[72927]: ts=2024-09-21T09:52:24.087Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002064118s 2024-09-21T09:52:25.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:24 smithi098 ceph-mon[29239]: pgmap v57: 73 pgs: 7 creating+peering, 66 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:25.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:24 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:25.256 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:24 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:25.256 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:24 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:25.256 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:24 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:25.256 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:24 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:25.256 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:24 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi098.stcwsu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:52:25.256 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:24 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi098.stcwsu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:52:25.256 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:24 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:25.256 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:24 smithi098 ceph-mon[29239]: Deploying daemon mds.foo.smithi098.stcwsu on smithi098 2024-09-21T09:52:25.256 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:24 smithi098 ceph-mon[39960]: pgmap v57: 73 pgs: 7 creating+peering, 66 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:25.256 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:24 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:25.256 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:24 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:25.257 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:24 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:25.257 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:24 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:25.257 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:24 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:25.257 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:24 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi098.stcwsu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:52:25.257 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:24 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi098.stcwsu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:52:25.257 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:24 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:25.257 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:24 smithi098 ceph-mon[39960]: Deploying daemon mds.foo.smithi098.stcwsu on smithi098 2024-09-21T09:52:25.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:24 smithi196 ceph-mon[32670]: pgmap v57: 73 pgs: 7 creating+peering, 66 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:24 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:24 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:24 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:24 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:24 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:24 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi098.stcwsu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:52:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:24 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi098.stcwsu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:52:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:24 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:24 smithi196 ceph-mon[32670]: Deploying daemon mds.foo.smithi098.stcwsu on smithi098 2024-09-21T09:52:26.570 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-09-21T09:52:26.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:52:26 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:52:26] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:52:27.005 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:26 smithi098 ceph-mon[39960]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:27.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:26 smithi098 ceph-mon[29239]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:26 smithi196 ceph-mon[32670]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:28.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:28.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:28.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:28.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi196.ugblah", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:52:28.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi196.ugblah", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:52:28.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:28.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[39960]: Deploying daemon mds.foo.smithi196.ugblah on smithi196 2024-09-21T09:52:28.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[39960]: mds.? [v2:172.21.15.98:6834/2705511271,v1:172.21.15.98:6835/2705511271] up:boot 2024-09-21T09:52:28.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[39960]: daemon mds.foo.smithi098.stcwsu assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-21T09:52:28.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[39960]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T09:52:28.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[39960]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T09:52:28.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[39960]: Cluster is now healthy 2024-09-21T09:52:28.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[39960]: fsmap foo:0 1 up:standby 2024-09-21T09:52:28.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi098.stcwsu"}]: dispatch 2024-09-21T09:52:28.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[39960]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:creating} 2024-09-21T09:52:28.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[39960]: daemon mds.foo.smithi098.stcwsu is now active in filesystem foo as rank 0 2024-09-21T09:52:28.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:28.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:28.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:28.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi196.ugblah", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:52:28.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi196.ugblah", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:52:28.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:28.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[29239]: Deploying daemon mds.foo.smithi196.ugblah on smithi196 2024-09-21T09:52:28.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[29239]: mds.? [v2:172.21.15.98:6834/2705511271,v1:172.21.15.98:6835/2705511271] up:boot 2024-09-21T09:52:28.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[29239]: daemon mds.foo.smithi098.stcwsu assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-21T09:52:28.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[29239]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T09:52:28.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[29239]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T09:52:28.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[29239]: Cluster is now healthy 2024-09-21T09:52:28.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[29239]: fsmap foo:0 1 up:standby 2024-09-21T09:52:28.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi098.stcwsu"}]: dispatch 2024-09-21T09:52:28.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[29239]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:creating} 2024-09-21T09:52:28.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:28 smithi098 ceph-mon[29239]: daemon mds.foo.smithi098.stcwsu is now active in filesystem foo as rank 0 2024-09-21T09:52:28.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:28 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:28.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:28 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:28.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:28 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:28.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:28 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi196.ugblah", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:52:28.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:28 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi196.ugblah", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:52:28.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:28 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:28.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:28 smithi196 ceph-mon[32670]: Deploying daemon mds.foo.smithi196.ugblah on smithi196 2024-09-21T09:52:28.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:28 smithi196 ceph-mon[32670]: mds.? [v2:172.21.15.98:6834/2705511271,v1:172.21.15.98:6835/2705511271] up:boot 2024-09-21T09:52:28.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:28 smithi196 ceph-mon[32670]: daemon mds.foo.smithi098.stcwsu assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-21T09:52:28.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:28 smithi196 ceph-mon[32670]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T09:52:28.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:28 smithi196 ceph-mon[32670]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T09:52:28.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:28 smithi196 ceph-mon[32670]: Cluster is now healthy 2024-09-21T09:52:28.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:28 smithi196 ceph-mon[32670]: fsmap foo:0 1 up:standby 2024-09-21T09:52:28.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:28 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi098.stcwsu"}]: dispatch 2024-09-21T09:52:28.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:28 smithi196 ceph-mon[32670]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:creating} 2024-09-21T09:52:28.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:28 smithi196 ceph-mon[32670]: daemon mds.foo.smithi098.stcwsu is now active in filesystem foo as rank 0 2024-09-21T09:52:29.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:29 smithi196 ceph-mon[32670]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:29 smithi196 ceph-mon[32670]: mds.? [v2:172.21.15.98:6834/2705511271,v1:172.21.15.98:6835/2705511271] up:active 2024-09-21T09:52:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:29 smithi196 ceph-mon[32670]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 2024-09-21T09:52:29.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:29 smithi098 ceph-mon[29239]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:29.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:29 smithi098 ceph-mon[29239]: mds.? [v2:172.21.15.98:6834/2705511271,v1:172.21.15.98:6835/2705511271] up:active 2024-09-21T09:52:29.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:29 smithi098 ceph-mon[29239]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 2024-09-21T09:52:29.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:29 smithi098 ceph-mon[39960]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:29.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:29 smithi098 ceph-mon[39960]: mds.? [v2:172.21.15.98:6834/2705511271,v1:172.21.15.98:6835/2705511271] up:active 2024-09-21T09:52:29.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:29 smithi098 ceph-mon[39960]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 2024-09-21T09:52:29.814 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-09-21T09:52:29.823 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T09:52:29.823 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-21T09:52:29.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph mgr dump --format=json 2024-09-21T09:52:30.083 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:30.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:52:30] "GET /metrics HTTP/1.1" 200 239115 "" "Prometheus/2.43.0" 2024-09-21T09:52:30.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:30.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:30.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:30.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:30.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:30.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:30.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[29239]: pgmap v60: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:30.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:52:30.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:30.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[29239]: mds.? [v2:172.21.15.196:6832/3677035285,v1:172.21.15.196:6833/3677035285] up:boot 2024-09-21T09:52:30.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[29239]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 1 up:standby 2024-09-21T09:52:30.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi196.ugblah"}]: dispatch 2024-09-21T09:52:30.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[29239]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 1 up:standby 2024-09-21T09:52:30.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:30.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:30.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:30.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:30.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:30.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:30.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[39960]: pgmap v60: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:30.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:52:30.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:30.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[39960]: mds.? [v2:172.21.15.196:6832/3677035285,v1:172.21.15.196:6833/3677035285] up:boot 2024-09-21T09:52:30.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[39960]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 1 up:standby 2024-09-21T09:52:30.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi196.ugblah"}]: dispatch 2024-09-21T09:52:30.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:30 smithi098 ceph-mon[39960]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 1 up:standby 2024-09-21T09:52:31.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:30 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:31.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:30 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:30 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:30 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:30 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:30 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:30 smithi196 ceph-mon[32670]: pgmap v60: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:30 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:52:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:30 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:30 smithi196 ceph-mon[32670]: mds.? [v2:172.21.15.196:6832/3677035285,v1:172.21.15.196:6833/3677035285] up:boot 2024-09-21T09:52:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:30 smithi196 ceph-mon[32670]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 1 up:standby 2024-09-21T09:52:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:30 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi196.ugblah"}]: dispatch 2024-09-21T09:52:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:30 smithi196 ceph-mon[32670]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 1 up:standby 2024-09-21T09:52:32.800 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:52:33.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:32 smithi098 ceph-mon[29239]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-09-21T09:52:33.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:32 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/966700871' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:52:33.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:32 smithi098 ceph-mon[39960]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-09-21T09:52:33.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:32 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/966700871' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:52:33.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:32 smithi196 ceph-mon[32670]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-09-21T09:52:33.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:32 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/966700871' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:52:33.519 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":20,"flags":0,"active_gid":24323,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":1387543382},{"type":"v1","addr":"172.21.15.98:6801","nonce":1387543382}]},"active_addr":"172.21.15.98:6801/1387543382","active_change":"2024-09-21T09:50:59.815532+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":24325,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.98:8443/","prometheus":"http://172.21.15.98:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":52,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":3466051265}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1093274465}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1829140501}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":900860885}]}]} 2024-09-21T09:52:33.522 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-21T09:52:33.522 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T09:52:33.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T09:52:34.048 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:33 smithi098 ceph-mon[29239]: mds.? [v2:172.21.15.98:6834/2705511271,v1:172.21.15.98:6835/2705511271] up:active 2024-09-21T09:52:34.048 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:33 smithi098 ceph-mon[29239]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 1 up:standby 2024-09-21T09:52:34.048 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:33 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:34.048 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:33 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:34.048 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:33 smithi098 ceph-mon[39960]: mds.? [v2:172.21.15.98:6834/2705511271,v1:172.21.15.98:6835/2705511271] up:active 2024-09-21T09:52:34.048 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:33 smithi098 ceph-mon[39960]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 1 up:standby 2024-09-21T09:52:34.049 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:33 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:34.049 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:33 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:34.338 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:33 smithi196 ceph-mon[32670]: mds.? [v2:172.21.15.98:6834/2705511271,v1:172.21.15.98:6835/2705511271] up:active 2024-09-21T09:52:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:33 smithi196 ceph-mon[32670]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 1 up:standby 2024-09-21T09:52:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:33 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:34.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:33 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:35.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:35 smithi098 ceph-mon[29239]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:52:35.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:35 smithi098 ceph-mon[29239]: mds.? [v2:172.21.15.196:6832/3677035285,v1:172.21.15.196:6833/3677035285] up:standby 2024-09-21T09:52:35.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:35 smithi098 ceph-mon[29239]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 1 up:standby 2024-09-21T09:52:35.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:35 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:35.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:35 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:35.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:35 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:35.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:35 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:35.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:35 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:35.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:35 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:35.491 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:35 smithi098 ceph-mon[39960]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:52:35.491 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:35 smithi098 ceph-mon[39960]: mds.? [v2:172.21.15.196:6832/3677035285,v1:172.21.15.196:6833/3677035285] up:standby 2024-09-21T09:52:35.491 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:35 smithi098 ceph-mon[39960]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 1 up:standby 2024-09-21T09:52:35.491 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:35 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:35.491 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:35 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:35.492 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:35 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:35.492 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:35 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:35.492 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:35 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:35.492 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:35 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:35 smithi196 ceph-mon[32670]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:52:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:35 smithi196 ceph-mon[32670]: mds.? [v2:172.21.15.196:6832/3677035285,v1:172.21.15.196:6833/3677035285] up:standby 2024-09-21T09:52:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:35 smithi196 ceph-mon[32670]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 1 up:standby 2024-09-21T09:52:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:35 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:35 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:35 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:35 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:35 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:35 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:36.771 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:52:36 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:52:36] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:52:37.005 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:52:37.005 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":64,"fsid":"26a15512-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:45:33.813487+0000","modified":"2024-09-21T09:52:22.381635+0000","last_up_change":"2024-09-21T09:50:27.903155+0000","last_in_change":"2024-09-21T09:50:10.725457+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:48:34.781592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:51:03.408228+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":58,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:52:19.135018+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:52:19.606631+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}}}],"osds":[{"osd":0,"uuid":"579f13de-565d-4801-b65e-f27d1cde297a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":506591636},{"type":"v1","addr":"172.21.15.98:6803","nonce":506591636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":506591636},{"type":"v1","addr":"172.21.15.98:6805","nonce":506591636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":506591636},{"type":"v1","addr":"172.21.15.98:6809","nonce":506591636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":506591636},{"type":"v1","addr":"172.21.15.98:6807","nonce":506591636}]},"public_addr":"172.21.15.98:6803/506591636","cluster_addr":"172.21.15.98:6805/506591636","heartbeat_back_addr":"172.21.15.98:6809/506591636","heartbeat_front_addr":"172.21.15.98:6807/506591636","state":["exists","up"]},{"osd":1,"uuid":"6d31ac4e-dc1a-4051-8266-ba821106242f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":4240601629},{"type":"v1","addr":"172.21.15.98:6811","nonce":4240601629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":4240601629},{"type":"v1","addr":"172.21.15.98:6813","nonce":4240601629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4240601629},{"type":"v1","addr":"172.21.15.98:6817","nonce":4240601629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":4240601629},{"type":"v1","addr":"172.21.15.98:6815","nonce":4240601629}]},"public_addr":"172.21.15.98:6811/4240601629","cluster_addr":"172.21.15.98:6813/4240601629","heartbeat_back_addr":"172.21.15.98:6817/4240601629","heartbeat_front_addr":"172.21.15.98:6815/4240601629","state":["exists","up"]},{"osd":2,"uuid":"62e22789-74be-4098-a9b9-62671fb4cac1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2917126560},{"type":"v1","addr":"172.21.15.98:6819","nonce":2917126560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2917126560},{"type":"v1","addr":"172.21.15.98:6821","nonce":2917126560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2917126560},{"type":"v1","addr":"172.21.15.98:6825","nonce":2917126560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2917126560},{"type":"v1","addr":"172.21.15.98:6823","nonce":2917126560}]},"public_addr":"172.21.15.98:6819/2917126560","cluster_addr":"172.21.15.98:6821/2917126560","heartbeat_back_addr":"172.21.15.98:6825/2917126560","heartbeat_front_addr":"172.21.15.98:6823/2917126560","state":["exists","up"]},{"osd":3,"uuid":"77f653a6-69b0-437a-960d-c3fb036708be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":4022556179},{"type":"v1","addr":"172.21.15.98:6827","nonce":4022556179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":4022556179},{"type":"v1","addr":"172.21.15.98:6829","nonce":4022556179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4022556179},{"type":"v1","addr":"172.21.15.98:6833","nonce":4022556179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":4022556179},{"type":"v1","addr":"172.21.15.98:6831","nonce":4022556179}]},"public_addr":"172.21.15.98:6827/4022556179","cluster_addr":"172.21.15.98:6829/4022556179","heartbeat_back_addr":"172.21.15.98:6833/4022556179","heartbeat_front_addr":"172.21.15.98:6831/4022556179","state":["exists","up"]},{"osd":4,"uuid":"28ec4928-7f00-46a9-b8c5-95f0b7e6be75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6801","nonce":158468112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6803","nonce":158468112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6807","nonce":158468112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6805","nonce":158468112}]},"public_addr":"172.21.15.196:6801/158468112","cluster_addr":"172.21.15.196:6803/158468112","heartbeat_back_addr":"172.21.15.196:6807/158468112","heartbeat_front_addr":"172.21.15.196:6805/158468112","state":["exists","up"]},{"osd":5,"uuid":"cdcc44eb-e27f-4d23-9ecb-03bebdc82c96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6809","nonce":2341878784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6811","nonce":2341878784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6815","nonce":2341878784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6813","nonce":2341878784}]},"public_addr":"172.21.15.196:6809/2341878784","cluster_addr":"172.21.15.196:6811/2341878784","heartbeat_back_addr":"172.21.15.196:6815/2341878784","heartbeat_front_addr":"172.21.15.196:6813/2341878784","state":["exists","up"]},{"osd":6,"uuid":"566cfb44-1b21-45fc-b053-7205b1c10d4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6817","nonce":3059702155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6819","nonce":3059702155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6823","nonce":3059702155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6821","nonce":3059702155}]},"public_addr":"172.21.15.196:6817/3059702155","cluster_addr":"172.21.15.196:6819/3059702155","heartbeat_back_addr":"172.21.15.196:6823/3059702155","heartbeat_front_addr":"172.21.15.196:6821/3059702155","state":["exists","up"]},{"osd":7,"uuid":"228404c1-3910-4933-b023-55aa8b01aee4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6825","nonce":2155020234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6827","nonce":2155020234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6831","nonce":2155020234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6829","nonce":2155020234}]},"public_addr":"172.21.15.196:6825/2155020234","cluster_addr":"172.21.15.196:6827/2155020234","heartbeat_back_addr":"172.21.15.196:6831/2155020234","heartbeat_front_addr":"172.21.15.196:6829/2155020234","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:47:43.263977+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:06.738746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:30.883578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:54.216866+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:49:15.599276+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:49:38.581033+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:01.504502+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:25.964220+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/4139436413":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/2144473621":"2024-09-22T09:45:54.237657+0000","172.21.15.98:0/2075804383":"2024-09-22T09:45:54.237657+0000","172.21.15.98:0/675672768":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/3656555554":"2024-09-22T09:45:54.237657+0000","172.21.15.98:6800/440385038":"2024-09-22T09:46:24.467967+0000","172.21.15.98:6801/440385038":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/4208010999":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/262018661":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/3737889208":"2024-09-22T09:46:24.467967+0000","172.21.15.98:6800/89887073":"2024-09-22T09:50:59.815356+0000","172.21.15.98:6801/1427334238":"2024-09-22T09:45:54.237657+0000","172.21.15.98:0/227493886":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/881612124":"2024-09-22T09:50:59.815356+0000","172.21.15.98:6800/1427334238":"2024-09-22T09:45:54.237657+0000","172.21.15.98:6801/89887073":"2024-09-22T09:50:59.815356+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T09:52:37.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:37 smithi098 ceph-mon[29239]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:52:37.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:37 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3907987027' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:52:37.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:37 smithi098 ceph-mon[39960]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:52:37.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:37 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3907987027' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:52:37.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:37 smithi196 ceph-mon[32670]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:52:37.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:37 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3907987027' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:52:38.135 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T09:52:38.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T09:52:38.392 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:39.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:39 smithi196 ceph-mon[32670]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:52:39.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:39 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:39.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:39 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:39.711 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:39 smithi098 ceph-mon[29239]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:52:39.711 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:39 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:39.711 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:39 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:39.712 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:39 smithi098 ceph-mon[39960]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:52:39.712 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:39 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:39.712 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:39 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:40.271 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:52:40.271 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":64,"fsid":"26a15512-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:45:33.813487+0000","modified":"2024-09-21T09:52:22.381635+0000","last_up_change":"2024-09-21T09:50:27.903155+0000","last_in_change":"2024-09-21T09:50:10.725457+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:48:34.781592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:51:03.408228+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":58,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:52:19.135018+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:52:19.606631+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}}}],"osds":[{"osd":0,"uuid":"579f13de-565d-4801-b65e-f27d1cde297a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":506591636},{"type":"v1","addr":"172.21.15.98:6803","nonce":506591636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":506591636},{"type":"v1","addr":"172.21.15.98:6805","nonce":506591636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":506591636},{"type":"v1","addr":"172.21.15.98:6809","nonce":506591636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":506591636},{"type":"v1","addr":"172.21.15.98:6807","nonce":506591636}]},"public_addr":"172.21.15.98:6803/506591636","cluster_addr":"172.21.15.98:6805/506591636","heartbeat_back_addr":"172.21.15.98:6809/506591636","heartbeat_front_addr":"172.21.15.98:6807/506591636","state":["exists","up"]},{"osd":1,"uuid":"6d31ac4e-dc1a-4051-8266-ba821106242f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":4240601629},{"type":"v1","addr":"172.21.15.98:6811","nonce":4240601629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":4240601629},{"type":"v1","addr":"172.21.15.98:6813","nonce":4240601629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4240601629},{"type":"v1","addr":"172.21.15.98:6817","nonce":4240601629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":4240601629},{"type":"v1","addr":"172.21.15.98:6815","nonce":4240601629}]},"public_addr":"172.21.15.98:6811/4240601629","cluster_addr":"172.21.15.98:6813/4240601629","heartbeat_back_addr":"172.21.15.98:6817/4240601629","heartbeat_front_addr":"172.21.15.98:6815/4240601629","state":["exists","up"]},{"osd":2,"uuid":"62e22789-74be-4098-a9b9-62671fb4cac1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2917126560},{"type":"v1","addr":"172.21.15.98:6819","nonce":2917126560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2917126560},{"type":"v1","addr":"172.21.15.98:6821","nonce":2917126560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2917126560},{"type":"v1","addr":"172.21.15.98:6825","nonce":2917126560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2917126560},{"type":"v1","addr":"172.21.15.98:6823","nonce":2917126560}]},"public_addr":"172.21.15.98:6819/2917126560","cluster_addr":"172.21.15.98:6821/2917126560","heartbeat_back_addr":"172.21.15.98:6825/2917126560","heartbeat_front_addr":"172.21.15.98:6823/2917126560","state":["exists","up"]},{"osd":3,"uuid":"77f653a6-69b0-437a-960d-c3fb036708be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":4022556179},{"type":"v1","addr":"172.21.15.98:6827","nonce":4022556179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":4022556179},{"type":"v1","addr":"172.21.15.98:6829","nonce":4022556179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4022556179},{"type":"v1","addr":"172.21.15.98:6833","nonce":4022556179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":4022556179},{"type":"v1","addr":"172.21.15.98:6831","nonce":4022556179}]},"public_addr":"172.21.15.98:6827/4022556179","cluster_addr":"172.21.15.98:6829/4022556179","heartbeat_back_addr":"172.21.15.98:6833/4022556179","heartbeat_front_addr":"172.21.15.98:6831/4022556179","state":["exists","up"]},{"osd":4,"uuid":"28ec4928-7f00-46a9-b8c5-95f0b7e6be75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6801","nonce":158468112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6803","nonce":158468112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6807","nonce":158468112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6805","nonce":158468112}]},"public_addr":"172.21.15.196:6801/158468112","cluster_addr":"172.21.15.196:6803/158468112","heartbeat_back_addr":"172.21.15.196:6807/158468112","heartbeat_front_addr":"172.21.15.196:6805/158468112","state":["exists","up"]},{"osd":5,"uuid":"cdcc44eb-e27f-4d23-9ecb-03bebdc82c96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6809","nonce":2341878784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6811","nonce":2341878784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6815","nonce":2341878784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6813","nonce":2341878784}]},"public_addr":"172.21.15.196:6809/2341878784","cluster_addr":"172.21.15.196:6811/2341878784","heartbeat_back_addr":"172.21.15.196:6815/2341878784","heartbeat_front_addr":"172.21.15.196:6813/2341878784","state":["exists","up"]},{"osd":6,"uuid":"566cfb44-1b21-45fc-b053-7205b1c10d4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6817","nonce":3059702155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6819","nonce":3059702155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6823","nonce":3059702155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6821","nonce":3059702155}]},"public_addr":"172.21.15.196:6817/3059702155","cluster_addr":"172.21.15.196:6819/3059702155","heartbeat_back_addr":"172.21.15.196:6823/3059702155","heartbeat_front_addr":"172.21.15.196:6821/3059702155","state":["exists","up"]},{"osd":7,"uuid":"228404c1-3910-4933-b023-55aa8b01aee4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6825","nonce":2155020234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6827","nonce":2155020234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6831","nonce":2155020234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6829","nonce":2155020234}]},"public_addr":"172.21.15.196:6825/2155020234","cluster_addr":"172.21.15.196:6827/2155020234","heartbeat_back_addr":"172.21.15.196:6831/2155020234","heartbeat_front_addr":"172.21.15.196:6829/2155020234","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:47:43.263977+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:06.738746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:30.883578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:54.216866+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:49:15.599276+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:49:38.581033+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:01.504502+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:25.964220+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/4139436413":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/2144473621":"2024-09-22T09:45:54.237657+0000","172.21.15.98:0/2075804383":"2024-09-22T09:45:54.237657+0000","172.21.15.98:0/675672768":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/3656555554":"2024-09-22T09:45:54.237657+0000","172.21.15.98:6800/440385038":"2024-09-22T09:46:24.467967+0000","172.21.15.98:6801/440385038":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/4208010999":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/262018661":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/3737889208":"2024-09-22T09:46:24.467967+0000","172.21.15.98:6800/89887073":"2024-09-22T09:50:59.815356+0000","172.21.15.98:6801/1427334238":"2024-09-22T09:45:54.237657+0000","172.21.15.98:0/227493886":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/881612124":"2024-09-22T09:50:59.815356+0000","172.21.15.98:6800/1427334238":"2024-09-22T09:45:54.237657+0000","172.21.15.98:6801/89887073":"2024-09-22T09:50:59.815356+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T09:52:40.910 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:52:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:52:40] "GET /metrics HTTP/1.1" 200 239115 "" "Prometheus/2.43.0" 2024-09-21T09:52:41.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:40 smithi098 ceph-mon[29239]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:52:41.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:40 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:41.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:40 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3812425096' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:52:41.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:40 smithi098 ceph-mon[39960]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:52:41.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:40 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:41.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:40 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3812425096' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:52:41.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:40 smithi196 ceph-mon[32670]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:52:41.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:40 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:41.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:40 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3812425096' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:52:41.450 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T09:52:41.450 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T09:52:41.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T09:52:41.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T09:52:41.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T09:52:41.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T09:52:41.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T09:52:41.453 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T09:52:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:42.628 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:42.667 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:42.667 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:42.897 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:43.019 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:42 smithi098 ceph-mon[29239]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:52:43.019 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:42 smithi098 ceph-mon[39960]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:52:43.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:42 smithi196 ceph-mon[32670]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:52:43.536 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:44.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:43 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:44.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:43 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:44.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:43 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:44.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:43 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:44.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:43 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:44.256 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:43 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:44.256 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:43 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:44.256 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:43 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:44.256 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:43 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:44.256 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:43 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:44.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:43 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:44.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:43 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:44.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:43 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:43 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:43 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:52:45.312 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:44 smithi098 ceph-mon[29239]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-21T09:52:45.312 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:44 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:52:45.312 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:44 smithi098 ceph-mon[39960]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-21T09:52:45.313 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:44 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:52:45.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:44 smithi196 ceph-mon[32670]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-21T09:52:45.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:44 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:52:46.889 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:52:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:52:46] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:52:47.048 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:46 smithi098 ceph-mon[29239]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-21T09:52:47.049 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:46 smithi098 ceph-mon[39960]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-21T09:52:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:46 smithi196 ceph-mon[32670]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-21T09:52:48.764 INFO:teuthology.orchestra.run.smithi098.stdout:34359738432 2024-09-21T09:52:48.765 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T09:52:49.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:48 smithi098 ceph-mon[29239]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:49.080 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:48 smithi098 ceph-mon[39960]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:49.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:48 smithi196 ceph-mon[32670]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:50.716 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:52:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:52:50] "GET /metrics HTTP/1.1" 200 239607 "" "Prometheus/2.43.0" 2024-09-21T09:52:51.244 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:50 smithi098 ceph-mon[39960]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:51.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:50 smithi098 ceph-mon[29239]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:51.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:50 smithi196 ceph-mon[32670]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:53.064 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:52 smithi098 ceph-mon[29239]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:53.064 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:52 smithi098 ceph-mon[39960]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:53.089 INFO:teuthology.orchestra.run.smithi098.stdout:77309411384 2024-09-21T09:52:53.089 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T09:52:53.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:52 smithi196 ceph-mon[32670]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:55.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:54 smithi098 ceph-mon[29239]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:55.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:54 smithi098 ceph-mon[39960]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:55.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:54 smithi196 ceph-mon[32670]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:56.329 INFO:teuthology.orchestra.run.smithi098.stdout:158913789994 2024-09-21T09:52:56.330 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T09:52:56.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:52:56 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:52:56] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:52:57.196 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:56 smithi098 ceph-mon[29239]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:57.196 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:56 smithi098 ceph-mon[39960]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:57.220 INFO:teuthology.orchestra.run.smithi098.stdout:133143986223 2024-09-21T09:52:57.220 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T09:52:57.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:56 smithi196 ceph-mon[32670]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:57.782 INFO:teuthology.orchestra.run.smithi098.stdout:111669149748 2024-09-21T09:52:57.782 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T09:52:58.853 INFO:teuthology.orchestra.run.smithi098.stdout:210453397537 2024-09-21T09:52:58.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T09:52:59.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:58 smithi098 ceph-mon[29239]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:59.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:58 smithi098 ceph-mon[39960]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:52:59.315 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:59.315 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:59.315 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:59.350 INFO:teuthology.orchestra.run.smithi098.stdout:184683593764 2024-09-21T09:52:59.351 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T09:52:59.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:58 smithi196 ceph-mon[32670]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:00.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:59 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-21T09:53:00.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:59 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-09-21T09:53:00.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:59 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-09-21T09:53:00.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:52:59 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:53:00.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:59 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-21T09:53:00.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:59 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-09-21T09:53:00.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:59 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-09-21T09:53:00.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:52:59 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:53:00.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:59 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-21T09:53:00.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:59 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-09-21T09:53:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:59 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-09-21T09:53:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:52:59 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:53:00.921 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:53:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:53:00] "GET /metrics HTTP/1.1" 200 239586 "" "Prometheus/2.43.0" 2024-09-21T09:53:00.994 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:01.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:00 smithi098 ceph-mon[29239]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:01.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-21T09:53:01.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-21T09:53:01.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:00 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-21T09:53:01.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:00 smithi098 ceph-mon[29239]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T09:53:01.341 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:00 smithi098 ceph-mon[39960]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:01.341 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-21T09:53:01.341 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-21T09:53:01.341 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:00 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-21T09:53:01.341 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:00 smithi098 ceph-mon[39960]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T09:53:01.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:00 smithi196 ceph-mon[32670]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:01.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-21T09:53:01.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-21T09:53:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:00 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-21T09:53:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:00 smithi196 ceph-mon[32670]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T09:53:02.131 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:01 smithi098 ceph-mon[29239]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T09:53:02.131 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:01 smithi098 ceph-mon[39960]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T09:53:02.169 INFO:teuthology.orchestra.run.smithi098.stdout:55834574911 2024-09-21T09:53:02.169 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T09:53:02.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:01 smithi196 ceph-mon[32670]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T09:53:03.191 INFO:teuthology.orchestra.run.smithi098.stdout:158913789996 2024-09-21T09:53:03.303 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:02 smithi098 ceph-mon[39960]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:03.303 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:02 smithi098 ceph-mon[29239]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:03.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:02 smithi196 ceph-mon[32670]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:04.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:04 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/4244943666' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:53:04.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:04 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/4244943666' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:53:04.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:04 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/4244943666' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:53:04.819 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:04.819 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:05.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:05 smithi098 ceph-mon[29239]: pgmap v79: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:05.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:05 smithi098 ceph-mon[39960]: pgmap v79: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:05.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:05 smithi196 ceph-mon[32670]: pgmap v79: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:05.768 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789994 got 158913789996 for osd.5 2024-09-21T09:53:05.768 DEBUG:teuthology.parallel:result is None 2024-09-21T09:53:06.198 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:06.581 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:06.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:53:06 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:53:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:53:07.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:07 smithi098 ceph-mon[29239]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:07.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:07 smithi098 ceph-mon[39960]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:07.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:07 smithi196 ceph-mon[32670]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:07.459 INFO:teuthology.orchestra.run.smithi098.stdout:34359738436 2024-09-21T09:53:07.637 INFO:teuthology.orchestra.run.smithi098.stdout:77309411387 2024-09-21T09:53:08.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:08 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3549800803' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:53:08.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:08 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/1237234153' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:53:08.285 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:08 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3549800803' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:53:08.285 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:08 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/1237234153' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:53:08.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:08 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3549800803' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:53:08.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:08 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/1237234153' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:53:08.419 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411384 got 77309411387 for osd.2 2024-09-21T09:53:08.419 DEBUG:teuthology.parallel:result is None 2024-09-21T09:53:09.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:09 smithi098 ceph-mon[29239]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:09.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:09 smithi098 ceph-mon[39960]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:09.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:09 smithi196 ceph-mon[32670]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:11.014 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:53:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:53:10] "GET /metrics HTTP/1.1" 200 239586 "" "Prometheus/2.43.0" 2024-09-21T09:53:11.155 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738432 got 34359738436 for osd.0 2024-09-21T09:53:11.155 DEBUG:teuthology.parallel:result is None 2024-09-21T09:53:11.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:11 smithi098 ceph-mon[29239]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:11.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:11 smithi098 ceph-mon[39960]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:11.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:11 smithi196 ceph-mon[32670]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:12.547 INFO:teuthology.orchestra.run.smithi098.stdout:111669149751 2024-09-21T09:53:13.179 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:13 smithi098 ceph-mon[29239]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:13.520 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:13 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/156386885' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:53:13.521 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:13 smithi098 ceph-mon[39960]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:13.521 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:13 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/156386885' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:53:13.521 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:13 smithi196 ceph-mon[32670]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:13.521 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:13 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/156386885' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:53:13.827 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149748 got 111669149751 for osd.3 2024-09-21T09:53:13.827 DEBUG:teuthology.parallel:result is None 2024-09-21T09:53:14.173 INFO:teuthology.orchestra.run.smithi098.stdout:184683593768 2024-09-21T09:53:14.955 INFO:teuthology.orchestra.run.smithi098.stdout:210453397540 2024-09-21T09:53:15.287 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:15 smithi098 ceph-mon[29239]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:15.288 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:15 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2328198445' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:53:15.288 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:15 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:53:15.288 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:15 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2227087686' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:53:15.288 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:15 smithi098 ceph-mon[39960]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:15.288 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:15 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2328198445' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:53:15.288 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:15 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:53:15.288 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:15 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2227087686' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:53:15.312 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593764 got 184683593768 for osd.6 2024-09-21T09:53:15.312 DEBUG:teuthology.parallel:result is None 2024-09-21T09:53:15.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:15 smithi196 ceph-mon[32670]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:15 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2328198445' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:53:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:15 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:53:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:15 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2227087686' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:53:16.313 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397537 got 210453397540 for osd.7 2024-09-21T09:53:16.313 DEBUG:teuthology.parallel:result is None 2024-09-21T09:53:16.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:53:16 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:53:16] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:53:17.165 INFO:teuthology.orchestra.run.smithi098.stdout:55834574914 2024-09-21T09:53:17.177 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:17 smithi098 ceph-mon[29239]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:17.177 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:17 smithi098 ceph-mon[39960]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:17.364 INFO:teuthology.orchestra.run.smithi098.stdout:133143986228 2024-09-21T09:53:17.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:17 smithi196 ceph-mon[32670]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:18.025 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574911 got 55834574914 for osd.1 2024-09-21T09:53:18.026 DEBUG:teuthology.parallel:result is None 2024-09-21T09:53:18.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:18 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3585881337' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:53:18.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:18 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2694022930' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:53:18.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:18 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3585881337' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:53:18.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:18 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2694022930' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:53:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:18 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3585881337' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:53:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:18 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2694022930' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:53:18.502 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986223 got 133143986228 for osd.4 2024-09-21T09:53:18.502 DEBUG:teuthology.parallel:result is None 2024-09-21T09:53:18.502 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T09:53:18.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T09:53:18.745 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:19.173 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:19 smithi098 ceph-mon[29239]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:19.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:19 smithi098 ceph-mon[39960]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:19.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:19 smithi196 ceph-mon[32670]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:20.537 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:53:20.539 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T09:53:20.846 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:53:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:53:20] "GET /metrics HTTP/1.1" 200 239651 "" "Prometheus/2.43.0" 2024-09-21T09:53:21.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:21 smithi098 ceph-mon[29239]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:21.320 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:21 smithi098 ceph-mon[39960]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:21.344 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-09-21T09:53:19.852025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592725,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":160,"num_write_kb":1643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121,"ondisk_log_size":121,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385932,"kb_used_data":7084,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575284,"statfs":{"total":767960285184,"available":765517090816,"internally_reserved":0,"allocated":7254016,"data_stored":4149350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002348"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994525+0000","last_change":"2024-09-21T09:52:22.011483+0000","last_active":"2024-09-21T09:52:59.994525+0000","last_peered":"2024-09-21T09:52:59.994525+0000","last_clean":"2024-09-21T09:52:59.994525+0000","last_became_active":"2024-09-21T09:52:22.011213+0000","last_became_peered":"2024-09-21T09:52:22.011213+0000","last_unstale":"2024-09-21T09:52:59.994525+0000","last_undegraded":"2024-09-21T09:52:59.994525+0000","last_fullsized":"2024-09-21T09:52:59.994525+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:21:14.207687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:53:00.352301+0000","last_change":"2024-09-21T09:52:20.374958+0000","last_active":"2024-09-21T09:53:00.352301+0000","last_peered":"2024-09-21T09:53:00.352301+0000","last_clean":"2024-09-21T09:53:00.352301+0000","last_became_active":"2024-09-21T09:52:20.374657+0000","last_became_peered":"2024-09-21T09:52:20.374657+0000","last_unstale":"2024-09-21T09:53:00.352301+0000","last_undegraded":"2024-09-21T09:53:00.352301+0000","last_fullsized":"2024-09-21T09:53:00.352301+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:14.351511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993557+0000","last_change":"2024-09-21T09:52:20.723312+0000","last_active":"2024-09-21T09:52:59.993557+0000","last_peered":"2024-09-21T09:52:59.993557+0000","last_clean":"2024-09-21T09:52:59.993557+0000","last_became_active":"2024-09-21T09:52:20.722299+0000","last_became_peered":"2024-09-21T09:52:20.722299+0000","last_unstale":"2024-09-21T09:52:59.993557+0000","last_undegraded":"2024-09-21T09:52:59.993557+0000","last_fullsized":"2024-09-21T09:52:59.993557+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:08:16.074195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990052+0000","last_change":"2024-09-21T09:52:20.378674+0000","last_active":"2024-09-21T09:52:59.990052+0000","last_peered":"2024-09-21T09:52:59.990052+0000","last_clean":"2024-09-21T09:52:59.990052+0000","last_became_active":"2024-09-21T09:52:20.378419+0000","last_became_peered":"2024-09-21T09:52:20.378419+0000","last_unstale":"2024-09-21T09:52:59.990052+0000","last_undegraded":"2024-09-21T09:52:59.990052+0000","last_fullsized":"2024-09-21T09:52:59.990052+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:56:47.522459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990721+0000","last_change":"2024-09-21T09:52:20.722626+0000","last_active":"2024-09-21T09:52:59.990721+0000","last_peered":"2024-09-21T09:52:59.990721+0000","last_clean":"2024-09-21T09:52:59.990721+0000","last_became_active":"2024-09-21T09:52:20.722261+0000","last_became_peered":"2024-09-21T09:52:20.722261+0000","last_unstale":"2024-09-21T09:52:59.990721+0000","last_undegraded":"2024-09-21T09:52:59.990721+0000","last_fullsized":"2024-09-21T09:52:59.990721+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:09:15.051105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:53:00.352160+0000","last_change":"2024-09-21T09:52:20.371775+0000","last_active":"2024-09-21T09:53:00.352160+0000","last_peered":"2024-09-21T09:53:00.352160+0000","last_clean":"2024-09-21T09:53:00.352160+0000","last_became_active":"2024-09-21T09:52:20.371509+0000","last_became_peered":"2024-09-21T09:52:20.371509+0000","last_unstale":"2024-09-21T09:53:00.352160+0000","last_undegraded":"2024-09-21T09:53:00.352160+0000","last_fullsized":"2024-09-21T09:53:00.352160+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:43:21.822524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994672+0000","last_change":"2024-09-21T09:52:22.011502+0000","last_active":"2024-09-21T09:52:59.994672+0000","last_peered":"2024-09-21T09:52:59.994672+0000","last_clean":"2024-09-21T09:52:59.994672+0000","last_became_active":"2024-09-21T09:52:22.011224+0000","last_became_peered":"2024-09-21T09:52:22.011224+0000","last_unstale":"2024-09-21T09:52:59.994672+0000","last_undegraded":"2024-09-21T09:52:59.994672+0000","last_fullsized":"2024-09-21T09:52:59.994672+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:35:55.160339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":22,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.992946+0000","last_change":"2024-09-21T09:52:20.377238+0000","last_active":"2024-09-21T09:52:59.992946+0000","last_peered":"2024-09-21T09:52:59.992946+0000","last_clean":"2024-09-21T09:52:59.992946+0000","last_became_active":"2024-09-21T09:52:20.376971+0000","last_became_peered":"2024-09-21T09:52:20.376971+0000","last_unstale":"2024-09-21T09:52:59.992946+0000","last_undegraded":"2024-09-21T09:52:59.992946+0000","last_fullsized":"2024-09-21T09:52:59.992946+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:41.805752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:53:00.352486+0000","last_change":"2024-09-21T09:52:20.378860+0000","last_active":"2024-09-21T09:53:00.352486+0000","last_peered":"2024-09-21T09:53:00.352486+0000","last_clean":"2024-09-21T09:53:00.352486+0000","last_became_active":"2024-09-21T09:52:20.378709+0000","last_became_peered":"2024-09-21T09:52:20.378709+0000","last_unstale":"2024-09-21T09:53:00.352486+0000","last_undegraded":"2024-09-21T09:53:00.352486+0000","last_fullsized":"2024-09-21T09:53:00.352486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:42.724812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993436+0000","last_change":"2024-09-21T09:52:20.402753+0000","last_active":"2024-09-21T09:52:59.993436+0000","last_peered":"2024-09-21T09:52:59.993436+0000","last_clean":"2024-09-21T09:52:59.993436+0000","last_became_active":"2024-09-21T09:52:20.402462+0000","last_became_peered":"2024-09-21T09:52:20.402462+0000","last_unstale":"2024-09-21T09:52:59.993436+0000","last_undegraded":"2024-09-21T09:52:59.993436+0000","last_fullsized":"2024-09-21T09:52:59.993436+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:14.036869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:53:00.352419+0000","last_change":"2024-09-21T09:52:20.375637+0000","last_active":"2024-09-21T09:53:00.352419+0000","last_peered":"2024-09-21T09:53:00.352419+0000","last_clean":"2024-09-21T09:53:00.352419+0000","last_became_active":"2024-09-21T09:52:20.375414+0000","last_became_peered":"2024-09-21T09:52:20.375414+0000","last_unstale":"2024-09-21T09:53:00.352419+0000","last_undegraded":"2024-09-21T09:53:00.352419+0000","last_fullsized":"2024-09-21T09:53:00.352419+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:20:05.958817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990075+0000","last_change":"2024-09-21T09:52:20.722766+0000","last_active":"2024-09-21T09:52:59.990075+0000","last_peered":"2024-09-21T09:52:59.990075+0000","last_clean":"2024-09-21T09:52:59.990075+0000","last_became_active":"2024-09-21T09:52:20.722482+0000","last_became_peered":"2024-09-21T09:52:20.722482+0000","last_unstale":"2024-09-21T09:52:59.990075+0000","last_undegraded":"2024-09-21T09:52:59.990075+0000","last_fullsized":"2024-09-21T09:52:59.990075+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:55:00.154858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:53:00.352486+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T09:53:00.352486+0000","last_peered":"2024-09-21T09:53:00.352486+0000","last_clean":"2024-09-21T09:53:00.352486+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T09:53:00.352486+0000","last_undegraded":"2024-09-21T09:53:00.352486+0000","last_fullsized":"2024-09-21T09:53:00.352486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.992513+0000","last_change":"2024-09-21T09:52:20.402892+0000","last_active":"2024-09-21T09:52:59.992513+0000","last_peered":"2024-09-21T09:52:59.992513+0000","last_clean":"2024-09-21T09:52:59.992513+0000","last_became_active":"2024-09-21T09:52:20.402615+0000","last_became_peered":"2024-09-21T09:52:20.402615+0000","last_unstale":"2024-09-21T09:52:59.992513+0000","last_undegraded":"2024-09-21T09:52:59.992513+0000","last_fullsized":"2024-09-21T09:52:59.992513+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:50.196101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993739+0000","last_change":"2024-09-21T09:52:20.376738+0000","last_active":"2024-09-21T09:52:59.993739+0000","last_peered":"2024-09-21T09:52:59.993739+0000","last_clean":"2024-09-21T09:52:59.993739+0000","last_became_active":"2024-09-21T09:52:20.376471+0000","last_became_peered":"2024-09-21T09:52:20.376471+0000","last_unstale":"2024-09-21T09:52:59.993739+0000","last_undegraded":"2024-09-21T09:52:59.993739+0000","last_fullsized":"2024-09-21T09:52:59.993739+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:05:33.184281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990200+0000","last_change":"2024-09-21T09:52:20.395015+0000","last_active":"2024-09-21T09:52:59.990200+0000","last_peered":"2024-09-21T09:52:59.990200+0000","last_clean":"2024-09-21T09:52:59.990200+0000","last_became_active":"2024-09-21T09:52:20.394611+0000","last_became_peered":"2024-09-21T09:52:20.394611+0000","last_unstale":"2024-09-21T09:52:59.990200+0000","last_undegraded":"2024-09-21T09:52:59.990200+0000","last_fullsized":"2024-09-21T09:52:59.990200+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:18:45.242576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994525+0000","last_change":"2024-09-21T09:52:20.381302+0000","last_active":"2024-09-21T09:52:59.994525+0000","last_peered":"2024-09-21T09:52:59.994525+0000","last_clean":"2024-09-21T09:52:59.994525+0000","last_became_active":"2024-09-21T09:52:20.379978+0000","last_became_peered":"2024-09-21T09:52:20.379978+0000","last_unstale":"2024-09-21T09:52:59.994525+0000","last_undegraded":"2024-09-21T09:52:59.994525+0000","last_fullsized":"2024-09-21T09:52:59.994525+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:41:52.152320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990240+0000","last_change":"2024-09-21T09:52:20.723137+0000","last_active":"2024-09-21T09:52:59.990240+0000","last_peered":"2024-09-21T09:52:59.990240+0000","last_clean":"2024-09-21T09:52:59.990240+0000","last_became_active":"2024-09-21T09:52:20.722788+0000","last_became_peered":"2024-09-21T09:52:20.722788+0000","last_unstale":"2024-09-21T09:52:59.990240+0000","last_undegraded":"2024-09-21T09:52:59.990240+0000","last_fullsized":"2024-09-21T09:52:59.990240+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:17:13.290156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.992617+0000","last_change":"2024-09-21T09:52:20.375736+0000","last_active":"2024-09-21T09:52:59.992617+0000","last_peered":"2024-09-21T09:52:59.992617+0000","last_clean":"2024-09-21T09:52:59.992617+0000","last_became_active":"2024-09-21T09:52:20.375473+0000","last_became_peered":"2024-09-21T09:52:20.375473+0000","last_unstale":"2024-09-21T09:52:59.992617+0000","last_undegraded":"2024-09-21T09:52:59.992617+0000","last_fullsized":"2024-09-21T09:52:59.992617+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:18.364939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990451+0000","last_change":"2024-09-21T09:52:20.393294+0000","last_active":"2024-09-21T09:52:59.990451+0000","last_peered":"2024-09-21T09:52:59.990451+0000","last_clean":"2024-09-21T09:52:59.990451+0000","last_became_active":"2024-09-21T09:52:20.392996+0000","last_became_peered":"2024-09-21T09:52:20.392996+0000","last_unstale":"2024-09-21T09:52:59.990451+0000","last_undegraded":"2024-09-21T09:52:59.990451+0000","last_fullsized":"2024-09-21T09:52:59.990451+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:15:18.225257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990278+0000","last_change":"2024-09-21T09:52:20.372268+0000","last_active":"2024-09-21T09:52:59.990278+0000","last_peered":"2024-09-21T09:52:59.990278+0000","last_clean":"2024-09-21T09:52:59.990278+0000","last_became_active":"2024-09-21T09:52:20.371948+0000","last_became_peered":"2024-09-21T09:52:20.371948+0000","last_unstale":"2024-09-21T09:52:59.990278+0000","last_undegraded":"2024-09-21T09:52:59.990278+0000","last_fullsized":"2024-09-21T09:52:59.990278+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:05:05.460719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990560+0000","last_change":"2024-09-21T09:52:20.391232+0000","last_active":"2024-09-21T09:52:59.990560+0000","last_peered":"2024-09-21T09:52:59.990560+0000","last_clean":"2024-09-21T09:52:59.990560+0000","last_became_active":"2024-09-21T09:52:20.390947+0000","last_became_peered":"2024-09-21T09:52:20.390947+0000","last_unstale":"2024-09-21T09:52:59.990560+0000","last_undegraded":"2024-09-21T09:52:59.990560+0000","last_fullsized":"2024-09-21T09:52:59.990560+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:03:11.273766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994355+0000","last_change":"2024-09-21T09:52:20.376219+0000","last_active":"2024-09-21T09:52:59.994355+0000","last_peered":"2024-09-21T09:52:59.994355+0000","last_clean":"2024-09-21T09:52:59.994355+0000","last_became_active":"2024-09-21T09:52:20.375957+0000","last_became_peered":"2024-09-21T09:52:20.375957+0000","last_unstale":"2024-09-21T09:52:59.994355+0000","last_undegraded":"2024-09-21T09:52:59.994355+0000","last_fullsized":"2024-09-21T09:52:59.994355+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:53.883620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.992702+0000","last_change":"2024-09-21T09:52:20.722803+0000","last_active":"2024-09-21T09:52:59.992702+0000","last_peered":"2024-09-21T09:52:59.992702+0000","last_clean":"2024-09-21T09:52:59.992702+0000","last_became_active":"2024-09-21T09:52:20.722527+0000","last_became_peered":"2024-09-21T09:52:20.722527+0000","last_unstale":"2024-09-21T09:52:59.992702+0000","last_undegraded":"2024-09-21T09:52:59.992702+0000","last_fullsized":"2024-09-21T09:52:59.992702+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:31.995729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"52'79","reported_seq":56,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:53:00.352114+0000","last_change":"2024-09-21T09:50:29.757187+0000","last_active":"2024-09-21T09:53:00.352114+0000","last_peered":"2024-09-21T09:53:00.352114+0000","last_clean":"2024-09-21T09:53:00.352114+0000","last_became_active":"2024-09-21T09:50:29.751734+0000","last_became_peered":"2024-09-21T09:50:29.751734+0000","last_unstale":"2024-09-21T09:53:00.352114+0000","last_undegraded":"2024-09-21T09:53:00.352114+0000","last_fullsized":"2024-09-21T09:53:00.352114+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T09:48:34.850872+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:20:45.636784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":27,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993655+0000","last_change":"2024-09-21T09:51:10.172804+0000","last_active":"2024-09-21T09:52:59.993655+0000","last_peered":"2024-09-21T09:52:59.993655+0000","last_clean":"2024-09-21T09:52:59.993655+0000","last_became_active":"2024-09-21T09:51:05.135491+0000","last_became_peered":"2024-09-21T09:51:05.135491+0000","last_unstale":"2024-09-21T09:52:59.993655+0000","last_undegraded":"2024-09-21T09:52:59.993655+0000","last_fullsized":"2024-09-21T09:52:59.993655+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:27:30.172694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034123299999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994414+0000","last_change":"2024-09-21T09:52:20.373618+0000","last_active":"2024-09-21T09:52:59.994414+0000","last_peered":"2024-09-21T09:52:59.994414+0000","last_clean":"2024-09-21T09:52:59.994414+0000","last_became_active":"2024-09-21T09:52:20.373365+0000","last_became_peered":"2024-09-21T09:52:20.373365+0000","last_unstale":"2024-09-21T09:52:59.994414+0000","last_undegraded":"2024-09-21T09:52:59.994414+0000","last_fullsized":"2024-09-21T09:52:59.994414+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:19.839696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.992759+0000","last_change":"2024-09-21T09:52:20.403534+0000","last_active":"2024-09-21T09:52:59.992759+0000","last_peered":"2024-09-21T09:52:59.992759+0000","last_clean":"2024-09-21T09:52:59.992759+0000","last_became_active":"2024-09-21T09:52:20.403258+0000","last_became_peered":"2024-09-21T09:52:20.403258+0000","last_unstale":"2024-09-21T09:52:59.992759+0000","last_undegraded":"2024-09-21T09:52:59.992759+0000","last_fullsized":"2024-09-21T09:52:59.992759+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:55.374343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:53:00.352056+0000","last_change":"2024-09-21T09:51:11.140693+0000","last_active":"2024-09-21T09:53:00.352056+0000","last_peered":"2024-09-21T09:53:00.352056+0000","last_clean":"2024-09-21T09:53:00.352056+0000","last_became_active":"2024-09-21T09:51:05.134160+0000","last_became_peered":"2024-09-21T09:51:05.134160+0000","last_unstale":"2024-09-21T09:53:00.352056+0000","last_undegraded":"2024-09-21T09:53:00.352056+0000","last_fullsized":"2024-09-21T09:53:00.352056+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:20.004781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028573500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.989892+0000","last_change":"2024-09-21T09:52:20.379256+0000","last_active":"2024-09-21T09:52:59.989892+0000","last_peered":"2024-09-21T09:52:59.989892+0000","last_clean":"2024-09-21T09:52:59.989892+0000","last_became_active":"2024-09-21T09:52:20.378963+0000","last_became_peered":"2024-09-21T09:52:20.378963+0000","last_unstale":"2024-09-21T09:52:59.989892+0000","last_undegraded":"2024-09-21T09:52:59.989892+0000","last_fullsized":"2024-09-21T09:52:59.989892+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:42:11.194772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990660+0000","last_change":"2024-09-21T09:52:20.394708+0000","last_active":"2024-09-21T09:52:59.990660+0000","last_peered":"2024-09-21T09:52:59.990660+0000","last_clean":"2024-09-21T09:52:59.990660+0000","last_became_active":"2024-09-21T09:52:20.394423+0000","last_became_peered":"2024-09-21T09:52:20.394423+0000","last_unstale":"2024-09-21T09:52:59.990660+0000","last_undegraded":"2024-09-21T09:52:59.990660+0000","last_fullsized":"2024-09-21T09:52:59.990660+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:46:16.384329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.989862+0000","last_change":"2024-09-21T09:51:10.168481+0000","last_active":"2024-09-21T09:52:59.989862+0000","last_peered":"2024-09-21T09:52:59.989862+0000","last_clean":"2024-09-21T09:52:59.989862+0000","last_became_active":"2024-09-21T09:51:05.134406+0000","last_became_peered":"2024-09-21T09:51:05.134406+0000","last_unstale":"2024-09-21T09:52:59.989862+0000","last_undegraded":"2024-09-21T09:52:59.989862+0000","last_fullsized":"2024-09-21T09:52:59.989862+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:14:34.828436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000364877,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.992199+0000","last_change":"2024-09-21T09:52:20.369967+0000","last_active":"2024-09-21T09:52:59.992199+0000","last_peered":"2024-09-21T09:52:59.992199+0000","last_clean":"2024-09-21T09:52:59.992199+0000","last_became_active":"2024-09-21T09:52:20.369681+0000","last_became_peered":"2024-09-21T09:52:20.369681+0000","last_unstale":"2024-09-21T09:52:59.992199+0000","last_undegraded":"2024-09-21T09:52:59.992199+0000","last_fullsized":"2024-09-21T09:52:59.992199+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:16:11.034455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:53:01.058799+0000","last_change":"2024-09-21T09:53:01.058799+0000","last_active":"2024-09-21T09:53:01.058799+0000","last_peered":"2024-09-21T09:53:01.058799+0000","last_clean":"2024-09-21T09:53:01.058799+0000","last_became_active":"2024-09-21T09:53:01.058487+0000","last_became_peered":"2024-09-21T09:53:01.058487+0000","last_unstale":"2024-09-21T09:53:01.058799+0000","last_undegraded":"2024-09-21T09:53:01.058799+0000","last_fullsized":"2024-09-21T09:53:01.058799+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:37.423153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":28,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993547+0000","last_change":"2024-09-21T09:51:10.173383+0000","last_active":"2024-09-21T09:52:59.993547+0000","last_peered":"2024-09-21T09:52:59.993547+0000","last_clean":"2024-09-21T09:52:59.993547+0000","last_became_active":"2024-09-21T09:51:05.134269+0000","last_became_peered":"2024-09-21T09:51:05.134269+0000","last_unstale":"2024-09-21T09:52:59.993547+0000","last_undegraded":"2024-09-21T09:52:59.993547+0000","last_fullsized":"2024-09-21T09:52:59.993547+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:23.879314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091501400000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990623+0000","last_change":"2024-09-21T09:52:20.375323+0000","last_active":"2024-09-21T09:52:59.990623+0000","last_peered":"2024-09-21T09:52:59.990623+0000","last_clean":"2024-09-21T09:52:59.990623+0000","last_became_active":"2024-09-21T09:52:20.375036+0000","last_became_peered":"2024-09-21T09:52:20.375036+0000","last_unstale":"2024-09-21T09:52:59.990623+0000","last_undegraded":"2024-09-21T09:52:59.990623+0000","last_fullsized":"2024-09-21T09:52:59.990623+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:07.500769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:53:01.929555+0000","last_change":"2024-09-21T09:53:01.929555+0000","last_active":"2024-09-21T09:53:01.929555+0000","last_peered":"2024-09-21T09:53:01.929555+0000","last_clean":"2024-09-21T09:53:01.929555+0000","last_became_active":"2024-09-21T09:53:01.929292+0000","last_became_peered":"2024-09-21T09:53:01.929292+0000","last_unstale":"2024-09-21T09:53:01.929555+0000","last_undegraded":"2024-09-21T09:53:01.929555+0000","last_fullsized":"2024-09-21T09:53:01.929555+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:26:07.291513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993917+0000","last_change":"2024-09-21T09:52:20.376173+0000","last_active":"2024-09-21T09:52:59.993917+0000","last_peered":"2024-09-21T09:52:59.993917+0000","last_clean":"2024-09-21T09:52:59.993917+0000","last_became_active":"2024-09-21T09:52:20.375977+0000","last_became_peered":"2024-09-21T09:52:20.375977+0000","last_unstale":"2024-09-21T09:52:59.993917+0000","last_undegraded":"2024-09-21T09:52:59.993917+0000","last_fullsized":"2024-09-21T09:52:59.993917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:26:30.599655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.992767+0000","last_change":"2024-09-21T09:51:10.174269+0000","last_active":"2024-09-21T09:52:59.992767+0000","last_peered":"2024-09-21T09:52:59.992767+0000","last_clean":"2024-09-21T09:52:59.992767+0000","last_became_active":"2024-09-21T09:51:05.134200+0000","last_became_peered":"2024-09-21T09:51:05.134200+0000","last_unstale":"2024-09-21T09:52:59.992767+0000","last_undegraded":"2024-09-21T09:52:59.992767+0000","last_fullsized":"2024-09-21T09:52:59.992767+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:38:40.009638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036908299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.992325+0000","last_change":"2024-09-21T09:52:20.393526+0000","last_active":"2024-09-21T09:52:59.992325+0000","last_peered":"2024-09-21T09:52:59.992325+0000","last_clean":"2024-09-21T09:52:59.992325+0000","last_became_active":"2024-09-21T09:52:20.393130+0000","last_became_peered":"2024-09-21T09:52:20.393130+0000","last_unstale":"2024-09-21T09:52:59.992325+0000","last_undegraded":"2024-09-21T09:52:59.992325+0000","last_fullsized":"2024-09-21T09:52:59.992325+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:56.289484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.992693+0000","last_change":"2024-09-21T09:52:20.371750+0000","last_active":"2024-09-21T09:52:59.992693+0000","last_peered":"2024-09-21T09:52:59.992693+0000","last_clean":"2024-09-21T09:52:59.992693+0000","last_became_active":"2024-09-21T09:52:20.371479+0000","last_became_peered":"2024-09-21T09:52:20.371479+0000","last_unstale":"2024-09-21T09:52:59.992693+0000","last_undegraded":"2024-09-21T09:52:59.992693+0000","last_fullsized":"2024-09-21T09:52:59.992693+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:50:13.674878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:53:00.352458+0000","last_change":"2024-09-21T09:51:11.140616+0000","last_active":"2024-09-21T09:53:00.352458+0000","last_peered":"2024-09-21T09:53:00.352458+0000","last_clean":"2024-09-21T09:53:00.352458+0000","last_became_active":"2024-09-21T09:51:05.429200+0000","last_became_peered":"2024-09-21T09:51:05.429200+0000","last_unstale":"2024-09-21T09:53:00.352458+0000","last_undegraded":"2024-09-21T09:53:00.352458+0000","last_fullsized":"2024-09-21T09:53:00.352458+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:29:34.651310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031258499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990263+0000","last_change":"2024-09-21T09:52:20.403016+0000","last_active":"2024-09-21T09:52:59.990263+0000","last_peered":"2024-09-21T09:52:59.990263+0000","last_clean":"2024-09-21T09:52:59.990263+0000","last_became_active":"2024-09-21T09:52:20.402670+0000","last_became_peered":"2024-09-21T09:52:20.402670+0000","last_unstale":"2024-09-21T09:52:59.990263+0000","last_undegraded":"2024-09-21T09:52:59.990263+0000","last_fullsized":"2024-09-21T09:52:59.990263+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:38.432145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994758+0000","last_change":"2024-09-21T09:52:20.381501+0000","last_active":"2024-09-21T09:52:59.994758+0000","last_peered":"2024-09-21T09:52:59.994758+0000","last_clean":"2024-09-21T09:52:59.994758+0000","last_became_active":"2024-09-21T09:52:20.380059+0000","last_became_peered":"2024-09-21T09:52:20.380059+0000","last_unstale":"2024-09-21T09:52:59.994758+0000","last_undegraded":"2024-09-21T09:52:59.994758+0000","last_fullsized":"2024-09-21T09:52:59.994758+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:20:11.443657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993005+0000","last_change":"2024-09-21T09:51:10.174262+0000","last_active":"2024-09-21T09:52:59.993005+0000","last_peered":"2024-09-21T09:52:59.993005+0000","last_clean":"2024-09-21T09:52:59.993005+0000","last_became_active":"2024-09-21T09:51:05.429405+0000","last_became_peered":"2024-09-21T09:51:05.429405+0000","last_unstale":"2024-09-21T09:52:59.993005+0000","last_undegraded":"2024-09-21T09:52:59.993005+0000","last_fullsized":"2024-09-21T09:52:59.993005+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:57:49.920758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000351963,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994274+0000","last_change":"2024-09-21T09:52:21.388589+0000","last_active":"2024-09-21T09:52:59.994274+0000","last_peered":"2024-09-21T09:52:59.994274+0000","last_clean":"2024-09-21T09:52:59.994274+0000","last_became_active":"2024-09-21T09:52:21.388371+0000","last_became_peered":"2024-09-21T09:52:21.388371+0000","last_unstale":"2024-09-21T09:52:59.994274+0000","last_undegraded":"2024-09-21T09:52:59.994274+0000","last_fullsized":"2024-09-21T09:52:59.994274+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:16:02.942895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990787+0000","last_change":"2024-09-21T09:52:20.376117+0000","last_active":"2024-09-21T09:52:59.990787+0000","last_peered":"2024-09-21T09:52:59.990787+0000","last_clean":"2024-09-21T09:52:59.990787+0000","last_became_active":"2024-09-21T09:52:20.375329+0000","last_became_peered":"2024-09-21T09:52:20.375329+0000","last_unstale":"2024-09-21T09:52:59.990787+0000","last_undegraded":"2024-09-21T09:52:59.990787+0000","last_fullsized":"2024-09-21T09:52:59.990787+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:05:21.158761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":26,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993096+0000","last_change":"2024-09-21T09:51:10.174279+0000","last_active":"2024-09-21T09:52:59.993096+0000","last_peered":"2024-09-21T09:52:59.993096+0000","last_clean":"2024-09-21T09:52:59.993096+0000","last_became_active":"2024-09-21T09:51:05.133025+0000","last_became_peered":"2024-09-21T09:51:05.133025+0000","last_unstale":"2024-09-21T09:52:59.993096+0000","last_undegraded":"2024-09-21T09:52:59.993096+0000","last_fullsized":"2024-09-21T09:52:59.993096+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:47.204645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029041499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.989981+0000","last_change":"2024-09-21T09:52:20.391439+0000","last_active":"2024-09-21T09:52:59.989981+0000","last_peered":"2024-09-21T09:52:59.989981+0000","last_clean":"2024-09-21T09:52:59.989981+0000","last_became_active":"2024-09-21T09:52:20.391157+0000","last_became_peered":"2024-09-21T09:52:20.391157+0000","last_unstale":"2024-09-21T09:52:59.989981+0000","last_undegraded":"2024-09-21T09:52:59.989981+0000","last_fullsized":"2024-09-21T09:52:59.989981+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:36.788330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990526+0000","last_change":"2024-09-21T09:52:20.391844+0000","last_active":"2024-09-21T09:52:59.990526+0000","last_peered":"2024-09-21T09:52:59.990526+0000","last_clean":"2024-09-21T09:52:59.990526+0000","last_became_active":"2024-09-21T09:52:20.391561+0000","last_became_peered":"2024-09-21T09:52:20.391561+0000","last_unstale":"2024-09-21T09:52:59.990526+0000","last_undegraded":"2024-09-21T09:52:59.990526+0000","last_fullsized":"2024-09-21T09:52:59.990526+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:37.669003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"64'6","reported_seq":20,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990343+0000","last_change":"2024-09-21T09:52:20.371668+0000","last_active":"2024-09-21T09:52:59.990343+0000","last_peered":"2024-09-21T09:52:59.990343+0000","last_clean":"2024-09-21T09:52:59.990343+0000","last_became_active":"2024-09-21T09:52:20.371389+0000","last_became_peered":"2024-09-21T09:52:20.371389+0000","last_unstale":"2024-09-21T09:52:59.990343+0000","last_undegraded":"2024-09-21T09:52:59.990343+0000","last_fullsized":"2024-09-21T09:52:59.990343+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:10.013020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1552,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.992338+0000","last_change":"2024-09-21T09:52:20.391921+0000","last_active":"2024-09-21T09:52:59.992338+0000","last_peered":"2024-09-21T09:52:59.992338+0000","last_clean":"2024-09-21T09:52:59.992338+0000","last_became_active":"2024-09-21T09:52:20.391648+0000","last_became_peered":"2024-09-21T09:52:20.391648+0000","last_unstale":"2024-09-21T09:52:59.992338+0000","last_undegraded":"2024-09-21T09:52:59.992338+0000","last_fullsized":"2024-09-21T09:52:59.992338+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:10.365545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":17,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:53:00.352300+0000","last_change":"2024-09-21T09:52:20.377816+0000","last_active":"2024-09-21T09:53:00.352300+0000","last_peered":"2024-09-21T09:53:00.352300+0000","last_clean":"2024-09-21T09:53:00.352300+0000","last_became_active":"2024-09-21T09:52:20.377550+0000","last_became_peered":"2024-09-21T09:52:20.377550+0000","last_unstale":"2024-09-21T09:53:00.352300+0000","last_undegraded":"2024-09-21T09:53:00.352300+0000","last_fullsized":"2024-09-21T09:53:00.352300+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:16:36.207459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993926+0000","last_change":"2024-09-21T09:52:20.376061+0000","last_active":"2024-09-21T09:52:59.993926+0000","last_peered":"2024-09-21T09:52:59.993926+0000","last_clean":"2024-09-21T09:52:59.993926+0000","last_became_active":"2024-09-21T09:52:20.375801+0000","last_became_peered":"2024-09-21T09:52:20.375801+0000","last_unstale":"2024-09-21T09:52:59.993926+0000","last_undegraded":"2024-09-21T09:52:59.993926+0000","last_fullsized":"2024-09-21T09:52:59.993926+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:15:54.908397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994332+0000","last_change":"2024-09-21T09:52:21.547147+0000","last_active":"2024-09-21T09:52:59.994332+0000","last_peered":"2024-09-21T09:52:59.994332+0000","last_clean":"2024-09-21T09:52:59.994332+0000","last_became_active":"2024-09-21T09:52:21.546878+0000","last_became_peered":"2024-09-21T09:52:21.546878+0000","last_unstale":"2024-09-21T09:52:59.994332+0000","last_undegraded":"2024-09-21T09:52:59.994332+0000","last_fullsized":"2024-09-21T09:52:59.994332+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:41.758824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990717+0000","last_change":"2024-09-21T09:52:20.378644+0000","last_active":"2024-09-21T09:52:59.990717+0000","last_peered":"2024-09-21T09:52:59.990717+0000","last_clean":"2024-09-21T09:52:59.990717+0000","last_became_active":"2024-09-21T09:52:20.378375+0000","last_became_peered":"2024-09-21T09:52:20.378375+0000","last_unstale":"2024-09-21T09:52:59.990717+0000","last_undegraded":"2024-09-21T09:52:59.990717+0000","last_fullsized":"2024-09-21T09:52:59.990717+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:00:45.021261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994261+0000","last_change":"2024-09-21T09:52:22.010842+0000","last_active":"2024-09-21T09:52:59.994261+0000","last_peered":"2024-09-21T09:52:59.994261+0000","last_clean":"2024-09-21T09:52:59.994261+0000","last_became_active":"2024-09-21T09:52:22.010543+0000","last_became_peered":"2024-09-21T09:52:22.010543+0000","last_unstale":"2024-09-21T09:52:59.994261+0000","last_undegraded":"2024-09-21T09:52:59.994261+0000","last_fullsized":"2024-09-21T09:52:59.994261+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:45:05.955965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993091+0000","last_change":"2024-09-21T09:52:20.392713+0000","last_active":"2024-09-21T09:52:59.993091+0000","last_peered":"2024-09-21T09:52:59.993091+0000","last_clean":"2024-09-21T09:52:59.993091+0000","last_became_active":"2024-09-21T09:52:20.392414+0000","last_became_peered":"2024-09-21T09:52:20.392414+0000","last_unstale":"2024-09-21T09:52:59.993091+0000","last_undegraded":"2024-09-21T09:52:59.993091+0000","last_fullsized":"2024-09-21T09:52:59.993091+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:37:00.959645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994340+0000","last_change":"2024-09-21T09:52:20.381561+0000","last_active":"2024-09-21T09:52:59.994340+0000","last_peered":"2024-09-21T09:52:59.994340+0000","last_clean":"2024-09-21T09:52:59.994340+0000","last_became_active":"2024-09-21T09:52:20.381193+0000","last_became_peered":"2024-09-21T09:52:20.381193+0000","last_unstale":"2024-09-21T09:52:59.994340+0000","last_undegraded":"2024-09-21T09:52:59.994340+0000","last_fullsized":"2024-09-21T09:52:59.994340+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:05:47.715444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990595+0000","last_change":"2024-09-21T09:52:20.724722+0000","last_active":"2024-09-21T09:52:59.990595+0000","last_peered":"2024-09-21T09:52:59.990595+0000","last_clean":"2024-09-21T09:52:59.990595+0000","last_became_active":"2024-09-21T09:52:20.724458+0000","last_became_peered":"2024-09-21T09:52:20.724458+0000","last_unstale":"2024-09-21T09:52:59.990595+0000","last_undegraded":"2024-09-21T09:52:59.990595+0000","last_fullsized":"2024-09-21T09:52:59.990595+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:32.897754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.992205+0000","last_change":"2024-09-21T09:52:20.377605+0000","last_active":"2024-09-21T09:52:59.992205+0000","last_peered":"2024-09-21T09:52:59.992205+0000","last_clean":"2024-09-21T09:52:59.992205+0000","last_became_active":"2024-09-21T09:52:20.376848+0000","last_became_peered":"2024-09-21T09:52:20.376848+0000","last_unstale":"2024-09-21T09:52:59.992205+0000","last_undegraded":"2024-09-21T09:52:59.992205+0000","last_fullsized":"2024-09-21T09:52:59.992205+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:16:32.126379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994575+0000","last_change":"2024-09-21T09:52:22.010825+0000","last_active":"2024-09-21T09:52:59.994575+0000","last_peered":"2024-09-21T09:52:59.994575+0000","last_clean":"2024-09-21T09:52:59.994575+0000","last_became_active":"2024-09-21T09:52:22.010539+0000","last_became_peered":"2024-09-21T09:52:22.010539+0000","last_unstale":"2024-09-21T09:52:59.994575+0000","last_undegraded":"2024-09-21T09:52:59.994575+0000","last_fullsized":"2024-09-21T09:52:59.994575+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:53.018658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990083+0000","last_change":"2024-09-21T09:52:20.373508+0000","last_active":"2024-09-21T09:52:59.990083+0000","last_peered":"2024-09-21T09:52:59.990083+0000","last_clean":"2024-09-21T09:52:59.990083+0000","last_became_active":"2024-09-21T09:52:20.373191+0000","last_became_peered":"2024-09-21T09:52:20.373191+0000","last_unstale":"2024-09-21T09:52:59.990083+0000","last_undegraded":"2024-09-21T09:52:59.990083+0000","last_fullsized":"2024-09-21T09:52:59.990083+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:49:04.982379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990183+0000","last_change":"2024-09-21T09:52:20.724765+0000","last_active":"2024-09-21T09:52:59.990183+0000","last_peered":"2024-09-21T09:52:59.990183+0000","last_clean":"2024-09-21T09:52:59.990183+0000","last_became_active":"2024-09-21T09:52:20.724362+0000","last_became_peered":"2024-09-21T09:52:20.724362+0000","last_unstale":"2024-09-21T09:52:59.990183+0000","last_undegraded":"2024-09-21T09:52:59.990183+0000","last_fullsized":"2024-09-21T09:52:59.990183+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:40.063634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990388+0000","last_change":"2024-09-21T09:52:20.374193+0000","last_active":"2024-09-21T09:52:59.990388+0000","last_peered":"2024-09-21T09:52:59.990388+0000","last_clean":"2024-09-21T09:52:59.990388+0000","last_became_active":"2024-09-21T09:52:20.373860+0000","last_became_peered":"2024-09-21T09:52:20.373860+0000","last_unstale":"2024-09-21T09:52:59.990388+0000","last_undegraded":"2024-09-21T09:52:59.990388+0000","last_fullsized":"2024-09-21T09:52:59.990388+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:05:32.105795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990411+0000","last_change":"2024-09-21T09:52:20.722903+0000","last_active":"2024-09-21T09:52:59.990411+0000","last_peered":"2024-09-21T09:52:59.990411+0000","last_clean":"2024-09-21T09:52:59.990411+0000","last_became_active":"2024-09-21T09:52:20.722337+0000","last_became_peered":"2024-09-21T09:52:20.722337+0000","last_unstale":"2024-09-21T09:52:59.990411+0000","last_undegraded":"2024-09-21T09:52:59.990411+0000","last_fullsized":"2024-09-21T09:52:59.990411+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:32.585565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993648+0000","last_change":"2024-09-21T09:52:20.369353+0000","last_active":"2024-09-21T09:52:59.993648+0000","last_peered":"2024-09-21T09:52:59.993648+0000","last_clean":"2024-09-21T09:52:59.993648+0000","last_became_active":"2024-09-21T09:52:20.369058+0000","last_became_peered":"2024-09-21T09:52:20.369058+0000","last_unstale":"2024-09-21T09:52:59.993648+0000","last_undegraded":"2024-09-21T09:52:59.993648+0000","last_fullsized":"2024-09-21T09:52:59.993648+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:44:59.238491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990037+0000","last_change":"2024-09-21T09:52:20.723224+0000","last_active":"2024-09-21T09:52:59.990037+0000","last_peered":"2024-09-21T09:52:59.990037+0000","last_clean":"2024-09-21T09:52:59.990037+0000","last_became_active":"2024-09-21T09:52:20.722308+0000","last_became_peered":"2024-09-21T09:52:20.722308+0000","last_unstale":"2024-09-21T09:52:59.990037+0000","last_undegraded":"2024-09-21T09:52:59.990037+0000","last_fullsized":"2024-09-21T09:52:59.990037+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:43:23.115705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993908+0000","last_change":"2024-09-21T09:52:20.371638+0000","last_active":"2024-09-21T09:52:59.993908+0000","last_peered":"2024-09-21T09:52:59.993908+0000","last_clean":"2024-09-21T09:52:59.993908+0000","last_became_active":"2024-09-21T09:52:20.371383+0000","last_became_peered":"2024-09-21T09:52:20.371383+0000","last_unstale":"2024-09-21T09:52:59.993908+0000","last_undegraded":"2024-09-21T09:52:59.993908+0000","last_fullsized":"2024-09-21T09:52:59.993908+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:33:45.384397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994490+0000","last_change":"2024-09-21T09:52:21.386491+0000","last_active":"2024-09-21T09:52:59.994490+0000","last_peered":"2024-09-21T09:52:59.994490+0000","last_clean":"2024-09-21T09:52:59.994490+0000","last_became_active":"2024-09-21T09:52:21.386086+0000","last_became_peered":"2024-09-21T09:52:21.386086+0000","last_unstale":"2024-09-21T09:52:59.994490+0000","last_undegraded":"2024-09-21T09:52:59.994490+0000","last_fullsized":"2024-09-21T09:52:59.994490+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:16.979789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994708+0000","last_change":"2024-09-21T09:52:20.375466+0000","last_active":"2024-09-21T09:52:59.994708+0000","last_peered":"2024-09-21T09:52:59.994708+0000","last_clean":"2024-09-21T09:52:59.994708+0000","last_became_active":"2024-09-21T09:52:20.374169+0000","last_became_peered":"2024-09-21T09:52:20.374169+0000","last_unstale":"2024-09-21T09:52:59.994708+0000","last_undegraded":"2024-09-21T09:52:59.994708+0000","last_fullsized":"2024-09-21T09:52:59.994708+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:13:29.821368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":23,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:53:01.596702+0000","last_change":"2024-09-21T09:53:01.596702+0000","last_active":"2024-09-21T09:53:01.596702+0000","last_peered":"2024-09-21T09:53:01.596702+0000","last_clean":"2024-09-21T09:53:01.596702+0000","last_became_active":"2024-09-21T09:53:01.596342+0000","last_became_peered":"2024-09-21T09:53:01.596342+0000","last_unstale":"2024-09-21T09:53:01.596702+0000","last_undegraded":"2024-09-21T09:53:01.596702+0000","last_fullsized":"2024-09-21T09:53:01.596702+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:17:48.203293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":18,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990681+0000","last_change":"2024-09-21T09:52:20.375908+0000","last_active":"2024-09-21T09:52:59.990681+0000","last_peered":"2024-09-21T09:52:59.990681+0000","last_clean":"2024-09-21T09:52:59.990681+0000","last_became_active":"2024-09-21T09:52:20.375644+0000","last_became_peered":"2024-09-21T09:52:20.375644+0000","last_unstale":"2024-09-21T09:52:59.990681+0000","last_undegraded":"2024-09-21T09:52:59.990681+0000","last_fullsized":"2024-09-21T09:52:59.990681+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:39:22.510856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2338,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397541,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298620,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446532,"statfs":{"total":95995035648,"available":95689248768,"internally_reserved":0,"allocated":1294336,"data_stored":888651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.627},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67200000000000004,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.60299999999999998}]},{"osd":1,"last update":"Sat Sep 21 09:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.54700000000000004,"15min":0.55500000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.55600000000000005,"15min":0.56200000000000006},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.71599999999999997,"5min":0.745,"15min":0.745},"last":0.66800000000000004}]},{"osd":2,"last update":"Sat Sep 21 09:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52800000000000002,"15min":0.53200000000000003},"min":{"1min":0.39500000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.55500000000000005}]},{"osd":3,"last update":"Sat Sep 21 09:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.54200000000000004,"15min":0.54600000000000004},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56499999999999995,"15min":0.56699999999999995},"min":{"1min":0.442,"5min":0.371,"15min":0.371},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.59199999999999997}]},{"osd":4,"last update":"Sat Sep 21 09:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.40300000000000002,"15min":0.40100000000000002},"min":{"1min":0.30399999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.504},{"interface":"front","average":{"1min":0.433,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.58199999999999996,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.53700000000000003}]},{"osd":5,"last update":"Sat Sep 21 09:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.42599999999999999,"15min":0.41899999999999998},"min":{"1min":0.32600000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.46700000000000003,"15min":0.46200000000000002},"min":{"1min":0.32300000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.63,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.64000000000000001}]},{"osd":6,"last update":"Sat Sep 21 09:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.47899999999999998,"15min":0.47399999999999998},"min":{"1min":0.36499999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47699999999999998,"15min":0.47499999999999998},"min":{"1min":0.33000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.51800000000000002}]}]},{"osd":6,"up_from":43,"seq":184683593769,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298612,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446540,"statfs":{"total":95995035648,"available":95689256960,"internally_reserved":0,"allocated":1286144,"data_stored":888524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.48799999999999999,"15min":0.48499999999999999},"min":{"1min":0.32500000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.623,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48599999999999999,"15min":0.48399999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69199999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.433}]},{"osd":1,"last update":"Sat Sep 21 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51000000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.58299999999999996,"15min":0.52000000000000002},"min":{"1min":0.372,"5min":0.309,"15min":0.309},"max":{"1min":8.8780000000000001,"5min":8.8780000000000001,"15min":8.8780000000000001},"last":0.58399999999999996}]},{"osd":2,"last update":"Sat Sep 21 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.50600000000000001,"15min":0.501},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.501,"15min":0.498},"min":{"1min":0.432,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.5}]},{"osd":3,"last update":"Sat Sep 21 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.502,"15min":0.495},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.64200000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52000000000000002,"15min":0.50800000000000001},"min":{"1min":0.42499999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.55500000000000005}]},{"osd":4,"last update":"Sat Sep 21 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.40000000000000002,"15min":0.38600000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.57399999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.433,"15min":0.42099999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.62,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.36899999999999999}]},{"osd":5,"last update":"Sat Sep 21 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.41499999999999998,"15min":0.40100000000000002},"min":{"1min":0.29199999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.42299999999999999,"15min":0.39500000000000002},"min":{"1min":0.35099999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.46400000000000002}]},{"osd":7,"last update":"Sat Sep 21 09:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.33700000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.434,"15min":0.42399999999999999},"min":{"1min":0.31900000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.48399999999999999}]}]},{"osd":5,"up_from":37,"seq":158913789999,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298024,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447128,"statfs":{"total":95995035648,"available":95689859072,"internally_reserved":0,"allocated":679936,"data_stored":296859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:52:41 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52900000000000003,"15min":0.51400000000000001},"min":{"1min":0.28599999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.68999999999999995,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50800000000000001,"15min":0.505},"min":{"1min":0.35999999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.67100000000000004,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.502}]},{"osd":1,"last update":"Sat Sep 21 09:52:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54000000000000004,"15min":0.53200000000000003},"min":{"1min":0.33700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71799999999999997,"5min":0.748,"15min":0.748},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51800000000000002,"15min":0.51200000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.53700000000000003}]},{"osd":2,"last update":"Sat Sep 21 09:52:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51600000000000001,"15min":0.51100000000000001},"min":{"1min":0.41599999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.67900000000000005,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54200000000000004,"15min":0.52800000000000002},"min":{"1min":0.374,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.71999999999999997,"5min":0.747,"15min":0.747},"last":0.65300000000000002}]},{"osd":3,"last update":"Sat Sep 21 09:52:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.48899999999999999,"15min":0.47499999999999998},"min":{"1min":0.38,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65900000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56100000000000005,"15min":0.55300000000000005},"min":{"1min":0.39900000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.73899999999999999,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.47799999999999998}]},{"osd":4,"last update":"Sat Sep 21 09:52:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.39400000000000002,"15min":0.36799999999999999},"min":{"1min":0.32900000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.40500000000000003,"15min":0.376},"min":{"1min":0.317,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.64200000000000002}]},{"osd":6,"last update":"Sat Sep 21 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.41799999999999998,"15min":0.39400000000000002},"min":{"1min":0.29299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.625,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.47299999999999998,"15min":0.439},"min":{"1min":0.30199999999999999,"5min":0.252,"15min":0.252},"max":{"1min":0.69799999999999995,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.625}]},{"osd":7,"last update":"Sat Sep 21 09:52:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.52300000000000002,"15min":0.51600000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.48399999999999999,"15min":0.47799999999999998},"min":{"1min":0.34000000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.61399999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986228,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298036,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447116,"statfs":{"total":95995035648,"available":95689846784,"internally_reserved":0,"allocated":696320,"data_stored":298213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:52:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.52000000000000002,"15min":0.52800000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.66100000000000003,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.38,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67900000000000005,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.503}]},{"osd":1,"last update":"Sat Sep 21 09:52:22 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53000000000000003,"15min":0.53700000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.72699999999999998,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.54000000000000004,"15min":0.54400000000000004},"min":{"1min":0.373,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.70599999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.57499999999999996}]},{"osd":2,"last update":"Sat Sep 21 09:52:22 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.54600000000000004,"15min":0.56000000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.623,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.624},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.65400000000000003,"5min":0.879,"15min":0.879},"last":0.41799999999999998}]},{"osd":3,"last update":"Sat Sep 21 09:52:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53500000000000003},"min":{"1min":0.379,"5min":0.375,"15min":0.375},"max":{"1min":0.72999999999999998,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.53200000000000003,"15min":0.53500000000000003},"min":{"1min":0.39200000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.68100000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.60799999999999998}]},{"osd":5,"last update":"Sat Sep 21 09:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.39400000000000002,"15min":0.38100000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.621,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.39900000000000002,"15min":0.39300000000000002},"min":{"1min":0.27800000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.65900000000000003,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.65100000000000002}]},{"osd":6,"last update":"Sat Sep 21 09:53:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.436,"15min":0.41999999999999998},"min":{"1min":0.31,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.432,"15min":0.41799999999999998},"min":{"1min":0.32200000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.64000000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.46000000000000002}]},{"osd":7,"last update":"Sat Sep 21 09:52:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.5,"15min":0.502},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68899999999999995,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.61399999999999999,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.58999999999999997}]}]},{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298020,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447132,"statfs":{"total":95995035648,"available":95689863168,"internally_reserved":0,"allocated":675840,"data_stored":296605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35999999999999999,"15min":0.35699999999999998},"min":{"1min":0.24299999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.48899999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38800000000000001,"15min":0.39400000000000002},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.498,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.254}]},{"osd":1,"last update":"Sat Sep 21 09:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.35299999999999998,"15min":0.35499999999999998},"min":{"1min":0.24199999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.44700000000000001,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.317},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36599999999999999,"15min":0.36799999999999999},"min":{"1min":0.29599999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.51100000000000001,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.22600000000000001}]},{"osd":2,"last update":"Sat Sep 21 09:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.41099999999999998,"15min":0.40699999999999997},"min":{"1min":0.29799999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.56200000000000006,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.377,"5min":0.40200000000000002,"15min":0.41499999999999998},"min":{"1min":0.23000000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.53800000000000003,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.33500000000000002}]},{"osd":4,"last update":"Sat Sep 21 09:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.55200000000000005,"15min":0.54600000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.70099999999999996,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.504},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.53100000000000003,"15min":0.51600000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.442}]},{"osd":5,"last update":"Sat Sep 21 09:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.54000000000000004,"15min":0.52000000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.75600000000000001,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.436},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55200000000000005,"15min":0.54600000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70999999999999996,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.55400000000000005}]},{"osd":6,"last update":"Sat Sep 21 09:53:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.69299999999999995,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55900000000000005,"15min":0.55300000000000005},"min":{"1min":0.443,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.746,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.52400000000000002}]},{"osd":7,"last update":"Sat Sep 21 09:52:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61399999999999999,"15min":0.61299999999999999},"min":{"1min":0.49099999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.78100000000000003,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.51500000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298016,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447136,"statfs":{"total":95995035648,"available":95689867264,"internally_reserved":0,"allocated":679936,"data_stored":297119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:52:43 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.34599999999999997,"15min":0.34599999999999997},"min":{"1min":0.245,"5min":0.22,"15min":0.22},"max":{"1min":0.41599999999999998,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.34899999999999998,"15min":0.34100000000000003},"min":{"1min":0.26300000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.41099999999999998,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.252}]},{"osd":1,"last update":"Sat Sep 21 09:52:43 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.38700000000000001,"15min":0.38800000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.47999999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.39900000000000002,"15min":0.40100000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.45900000000000002,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.34799999999999998}]},{"osd":3,"last update":"Sat Sep 21 09:53:03 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.38400000000000001,"15min":0.39000000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.495,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.314},{"interface":"front","average":{"1min":0.38,"5min":0.40000000000000002,"15min":0.40899999999999997},"min":{"1min":0.27300000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.56299999999999994,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.26700000000000002}]},{"osd":4,"last update":"Sat Sep 21 09:52:20 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.53400000000000003,"15min":0.53600000000000003},"min":{"1min":0.39000000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.70899999999999996,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42599999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73799999999999999,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.57099999999999995}]},{"osd":5,"last update":"Sat Sep 21 09:52:51 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.55300000000000005,"15min":0.56399999999999995},"min":{"1min":0.38300000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.69899999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56100000000000005,"15min":0.57599999999999996},"min":{"1min":0.39400000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.66500000000000004,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":3.423}]},{"osd":6,"last update":"Sat Sep 21 09:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.69399999999999995,"15min":0.624},"min":{"1min":0.40500000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.76700000000000002,"5min":10.569000000000001,"15min":10.569000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56299999999999994,"15min":0.57699999999999996},"min":{"1min":0.373,"5min":0.371,"15min":0.371},"max":{"1min":0.68000000000000005,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.46300000000000002}]},{"osd":7,"last update":"Sat Sep 21 09:52:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.58699999999999997,"15min":0.59099999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.75900000000000001,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.41399999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77500000000000002,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.56100000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298004,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447148,"statfs":{"total":95995035648,"available":95689879552,"internally_reserved":0,"allocated":667648,"data_stored":295904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34200000000000003,"15min":0.32800000000000001},"min":{"1min":0.20499999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.48999999999999999,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.35399999999999998,"15min":0.32800000000000001},"min":{"1min":0.25700000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.22500000000000001}]},{"osd":2,"last update":"Sat Sep 21 09:52:45 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39300000000000002,"15min":0.39100000000000001},"min":{"1min":0.28899999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.375,"15min":0.375},"min":{"1min":0.24099999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.55500000000000005,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.37}]},{"osd":3,"last update":"Sat Sep 21 09:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.41699999999999998,"15min":0.42999999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.625,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.317},{"interface":"front","average":{"1min":0.375,"5min":0.41999999999999998,"15min":0.43099999999999999},"min":{"1min":0.27600000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.59499999999999997,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.29699999999999999}]},{"osd":4,"last update":"Sat Sep 21 09:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.59699999999999998,"15min":0.57899999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.65400000000000003,"15min":0.58999999999999997},"min":{"1min":0.38600000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73299999999999998,"5min":8.4100000000000001,"15min":8.4100000000000001},"last":0.55400000000000005}]},{"osd":5,"last update":"Sat Sep 21 09:52:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":0.67100000000000004,"15min":0.61099999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.44400000000000001}]},{"osd":6,"last update":"Sat Sep 21 09:53:05 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58399999999999996,"15min":0.57999999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.83399999999999996,"15min":0.95399999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.78100000000000003,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.47399999999999998}]},{"osd":7,"last update":"Sat Sep 21 09:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.59599999999999997,"15min":0.59999999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76700000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.44700000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.53300000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298600,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446552,"statfs":{"total":95995035648,"available":95689269248,"internally_reserved":0,"allocated":1273856,"data_stored":887475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 09:52:14 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.33100000000000002,"15min":0.29499999999999998},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.379},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.35399999999999998,"15min":0.32400000000000001},"min":{"1min":0.24399999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.50900000000000001}]},{"osd":2,"last update":"Sat Sep 21 09:52:38 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.34499999999999997,"15min":0.34300000000000003},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.46300000000000002,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.377,"15min":0.36899999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.55700000000000005,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.49299999999999999}]},{"osd":3,"last update":"Sat Sep 21 09:53:10 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.27300000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.58699999999999997,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.38700000000000001,"15min":0.38600000000000001},"min":{"1min":0.26700000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.51500000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.47199999999999998}]},{"osd":4,"last update":"Sat Sep 21 09:52:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.999,"15min":1.204},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.73899999999999999,"5min":15.444000000000001,"15min":15.444000000000001},"last":0.754},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.82699999999999996}]},{"osd":5,"last update":"Sat Sep 21 09:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.56299999999999994,"15min":0.55500000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.56100000000000005,"15min":0.55200000000000005},"min":{"1min":0.375,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.919,"5min":1.919,"15min":1.919},"last":0.77300000000000002}]},{"osd":6,"last update":"Sat Sep 21 09:53:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.57199999999999995,"15min":0.56699999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55600000000000005,"15min":0.55000000000000004},"min":{"1min":0.40899999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.92000000000000004}]},{"osd":7,"last update":"Sat Sep 21 09:52:32 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.56000000000000005,"15min":0.54600000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.57599999999999996,"15min":0.57799999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.73099999999999998,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.93500000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T09:53:21.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T09:53:21.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:21 smithi196 ceph-mon[32670]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:21.591 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:22.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:22 smithi098 ceph-mon[29239]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:53:22.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:22 smithi098 ceph-mon[39960]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:53:22.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:22 smithi196 ceph-mon[32670]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:53:23.083 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:23 smithi098 ceph-mon[29239]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:23.083 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:23 smithi098 ceph-mon[39960]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:23.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:23 smithi196 ceph-mon[32670]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:23.479 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:53:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T09:53:24.205 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-09-21T09:53:21.852334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592725,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":160,"num_write_kb":1643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121,"ondisk_log_size":121,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385932,"kb_used_data":7084,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575284,"statfs":{"total":767960285184,"available":765517090816,"internally_reserved":0,"allocated":7254016,"data_stored":4149350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002380"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994525+0000","last_change":"2024-09-21T09:52:22.011483+0000","last_active":"2024-09-21T09:52:59.994525+0000","last_peered":"2024-09-21T09:52:59.994525+0000","last_clean":"2024-09-21T09:52:59.994525+0000","last_became_active":"2024-09-21T09:52:22.011213+0000","last_became_peered":"2024-09-21T09:52:22.011213+0000","last_unstale":"2024-09-21T09:52:59.994525+0000","last_undegraded":"2024-09-21T09:52:59.994525+0000","last_fullsized":"2024-09-21T09:52:59.994525+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:21:14.207687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:53:00.352301+0000","last_change":"2024-09-21T09:52:20.374958+0000","last_active":"2024-09-21T09:53:00.352301+0000","last_peered":"2024-09-21T09:53:00.352301+0000","last_clean":"2024-09-21T09:53:00.352301+0000","last_became_active":"2024-09-21T09:52:20.374657+0000","last_became_peered":"2024-09-21T09:52:20.374657+0000","last_unstale":"2024-09-21T09:53:00.352301+0000","last_undegraded":"2024-09-21T09:53:00.352301+0000","last_fullsized":"2024-09-21T09:53:00.352301+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:14.351511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993557+0000","last_change":"2024-09-21T09:52:20.723312+0000","last_active":"2024-09-21T09:52:59.993557+0000","last_peered":"2024-09-21T09:52:59.993557+0000","last_clean":"2024-09-21T09:52:59.993557+0000","last_became_active":"2024-09-21T09:52:20.722299+0000","last_became_peered":"2024-09-21T09:52:20.722299+0000","last_unstale":"2024-09-21T09:52:59.993557+0000","last_undegraded":"2024-09-21T09:52:59.993557+0000","last_fullsized":"2024-09-21T09:52:59.993557+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:08:16.074195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990052+0000","last_change":"2024-09-21T09:52:20.378674+0000","last_active":"2024-09-21T09:52:59.990052+0000","last_peered":"2024-09-21T09:52:59.990052+0000","last_clean":"2024-09-21T09:52:59.990052+0000","last_became_active":"2024-09-21T09:52:20.378419+0000","last_became_peered":"2024-09-21T09:52:20.378419+0000","last_unstale":"2024-09-21T09:52:59.990052+0000","last_undegraded":"2024-09-21T09:52:59.990052+0000","last_fullsized":"2024-09-21T09:52:59.990052+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:56:47.522459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990721+0000","last_change":"2024-09-21T09:52:20.722626+0000","last_active":"2024-09-21T09:52:59.990721+0000","last_peered":"2024-09-21T09:52:59.990721+0000","last_clean":"2024-09-21T09:52:59.990721+0000","last_became_active":"2024-09-21T09:52:20.722261+0000","last_became_peered":"2024-09-21T09:52:20.722261+0000","last_unstale":"2024-09-21T09:52:59.990721+0000","last_undegraded":"2024-09-21T09:52:59.990721+0000","last_fullsized":"2024-09-21T09:52:59.990721+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:09:15.051105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:53:00.352160+0000","last_change":"2024-09-21T09:52:20.371775+0000","last_active":"2024-09-21T09:53:00.352160+0000","last_peered":"2024-09-21T09:53:00.352160+0000","last_clean":"2024-09-21T09:53:00.352160+0000","last_became_active":"2024-09-21T09:52:20.371509+0000","last_became_peered":"2024-09-21T09:52:20.371509+0000","last_unstale":"2024-09-21T09:53:00.352160+0000","last_undegraded":"2024-09-21T09:53:00.352160+0000","last_fullsized":"2024-09-21T09:53:00.352160+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:43:21.822524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994672+0000","last_change":"2024-09-21T09:52:22.011502+0000","last_active":"2024-09-21T09:52:59.994672+0000","last_peered":"2024-09-21T09:52:59.994672+0000","last_clean":"2024-09-21T09:52:59.994672+0000","last_became_active":"2024-09-21T09:52:22.011224+0000","last_became_peered":"2024-09-21T09:52:22.011224+0000","last_unstale":"2024-09-21T09:52:59.994672+0000","last_undegraded":"2024-09-21T09:52:59.994672+0000","last_fullsized":"2024-09-21T09:52:59.994672+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:35:55.160339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":22,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.992946+0000","last_change":"2024-09-21T09:52:20.377238+0000","last_active":"2024-09-21T09:52:59.992946+0000","last_peered":"2024-09-21T09:52:59.992946+0000","last_clean":"2024-09-21T09:52:59.992946+0000","last_became_active":"2024-09-21T09:52:20.376971+0000","last_became_peered":"2024-09-21T09:52:20.376971+0000","last_unstale":"2024-09-21T09:52:59.992946+0000","last_undegraded":"2024-09-21T09:52:59.992946+0000","last_fullsized":"2024-09-21T09:52:59.992946+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:41.805752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:53:00.352486+0000","last_change":"2024-09-21T09:52:20.378860+0000","last_active":"2024-09-21T09:53:00.352486+0000","last_peered":"2024-09-21T09:53:00.352486+0000","last_clean":"2024-09-21T09:53:00.352486+0000","last_became_active":"2024-09-21T09:52:20.378709+0000","last_became_peered":"2024-09-21T09:52:20.378709+0000","last_unstale":"2024-09-21T09:53:00.352486+0000","last_undegraded":"2024-09-21T09:53:00.352486+0000","last_fullsized":"2024-09-21T09:53:00.352486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:42.724812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993436+0000","last_change":"2024-09-21T09:52:20.402753+0000","last_active":"2024-09-21T09:52:59.993436+0000","last_peered":"2024-09-21T09:52:59.993436+0000","last_clean":"2024-09-21T09:52:59.993436+0000","last_became_active":"2024-09-21T09:52:20.402462+0000","last_became_peered":"2024-09-21T09:52:20.402462+0000","last_unstale":"2024-09-21T09:52:59.993436+0000","last_undegraded":"2024-09-21T09:52:59.993436+0000","last_fullsized":"2024-09-21T09:52:59.993436+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:14.036869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:53:00.352419+0000","last_change":"2024-09-21T09:52:20.375637+0000","last_active":"2024-09-21T09:53:00.352419+0000","last_peered":"2024-09-21T09:53:00.352419+0000","last_clean":"2024-09-21T09:53:00.352419+0000","last_became_active":"2024-09-21T09:52:20.375414+0000","last_became_peered":"2024-09-21T09:52:20.375414+0000","last_unstale":"2024-09-21T09:53:00.352419+0000","last_undegraded":"2024-09-21T09:53:00.352419+0000","last_fullsized":"2024-09-21T09:53:00.352419+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:20:05.958817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990075+0000","last_change":"2024-09-21T09:52:20.722766+0000","last_active":"2024-09-21T09:52:59.990075+0000","last_peered":"2024-09-21T09:52:59.990075+0000","last_clean":"2024-09-21T09:52:59.990075+0000","last_became_active":"2024-09-21T09:52:20.722482+0000","last_became_peered":"2024-09-21T09:52:20.722482+0000","last_unstale":"2024-09-21T09:52:59.990075+0000","last_undegraded":"2024-09-21T09:52:59.990075+0000","last_fullsized":"2024-09-21T09:52:59.990075+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:55:00.154858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:53:00.352486+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T09:53:00.352486+0000","last_peered":"2024-09-21T09:53:00.352486+0000","last_clean":"2024-09-21T09:53:00.352486+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T09:53:00.352486+0000","last_undegraded":"2024-09-21T09:53:00.352486+0000","last_fullsized":"2024-09-21T09:53:00.352486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.992513+0000","last_change":"2024-09-21T09:52:20.402892+0000","last_active":"2024-09-21T09:52:59.992513+0000","last_peered":"2024-09-21T09:52:59.992513+0000","last_clean":"2024-09-21T09:52:59.992513+0000","last_became_active":"2024-09-21T09:52:20.402615+0000","last_became_peered":"2024-09-21T09:52:20.402615+0000","last_unstale":"2024-09-21T09:52:59.992513+0000","last_undegraded":"2024-09-21T09:52:59.992513+0000","last_fullsized":"2024-09-21T09:52:59.992513+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:50.196101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993739+0000","last_change":"2024-09-21T09:52:20.376738+0000","last_active":"2024-09-21T09:52:59.993739+0000","last_peered":"2024-09-21T09:52:59.993739+0000","last_clean":"2024-09-21T09:52:59.993739+0000","last_became_active":"2024-09-21T09:52:20.376471+0000","last_became_peered":"2024-09-21T09:52:20.376471+0000","last_unstale":"2024-09-21T09:52:59.993739+0000","last_undegraded":"2024-09-21T09:52:59.993739+0000","last_fullsized":"2024-09-21T09:52:59.993739+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:05:33.184281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990200+0000","last_change":"2024-09-21T09:52:20.395015+0000","last_active":"2024-09-21T09:52:59.990200+0000","last_peered":"2024-09-21T09:52:59.990200+0000","last_clean":"2024-09-21T09:52:59.990200+0000","last_became_active":"2024-09-21T09:52:20.394611+0000","last_became_peered":"2024-09-21T09:52:20.394611+0000","last_unstale":"2024-09-21T09:52:59.990200+0000","last_undegraded":"2024-09-21T09:52:59.990200+0000","last_fullsized":"2024-09-21T09:52:59.990200+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:18:45.242576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994525+0000","last_change":"2024-09-21T09:52:20.381302+0000","last_active":"2024-09-21T09:52:59.994525+0000","last_peered":"2024-09-21T09:52:59.994525+0000","last_clean":"2024-09-21T09:52:59.994525+0000","last_became_active":"2024-09-21T09:52:20.379978+0000","last_became_peered":"2024-09-21T09:52:20.379978+0000","last_unstale":"2024-09-21T09:52:59.994525+0000","last_undegraded":"2024-09-21T09:52:59.994525+0000","last_fullsized":"2024-09-21T09:52:59.994525+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:41:52.152320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990240+0000","last_change":"2024-09-21T09:52:20.723137+0000","last_active":"2024-09-21T09:52:59.990240+0000","last_peered":"2024-09-21T09:52:59.990240+0000","last_clean":"2024-09-21T09:52:59.990240+0000","last_became_active":"2024-09-21T09:52:20.722788+0000","last_became_peered":"2024-09-21T09:52:20.722788+0000","last_unstale":"2024-09-21T09:52:59.990240+0000","last_undegraded":"2024-09-21T09:52:59.990240+0000","last_fullsized":"2024-09-21T09:52:59.990240+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:17:13.290156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.992617+0000","last_change":"2024-09-21T09:52:20.375736+0000","last_active":"2024-09-21T09:52:59.992617+0000","last_peered":"2024-09-21T09:52:59.992617+0000","last_clean":"2024-09-21T09:52:59.992617+0000","last_became_active":"2024-09-21T09:52:20.375473+0000","last_became_peered":"2024-09-21T09:52:20.375473+0000","last_unstale":"2024-09-21T09:52:59.992617+0000","last_undegraded":"2024-09-21T09:52:59.992617+0000","last_fullsized":"2024-09-21T09:52:59.992617+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:18.364939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990451+0000","last_change":"2024-09-21T09:52:20.393294+0000","last_active":"2024-09-21T09:52:59.990451+0000","last_peered":"2024-09-21T09:52:59.990451+0000","last_clean":"2024-09-21T09:52:59.990451+0000","last_became_active":"2024-09-21T09:52:20.392996+0000","last_became_peered":"2024-09-21T09:52:20.392996+0000","last_unstale":"2024-09-21T09:52:59.990451+0000","last_undegraded":"2024-09-21T09:52:59.990451+0000","last_fullsized":"2024-09-21T09:52:59.990451+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:15:18.225257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990278+0000","last_change":"2024-09-21T09:52:20.372268+0000","last_active":"2024-09-21T09:52:59.990278+0000","last_peered":"2024-09-21T09:52:59.990278+0000","last_clean":"2024-09-21T09:52:59.990278+0000","last_became_active":"2024-09-21T09:52:20.371948+0000","last_became_peered":"2024-09-21T09:52:20.371948+0000","last_unstale":"2024-09-21T09:52:59.990278+0000","last_undegraded":"2024-09-21T09:52:59.990278+0000","last_fullsized":"2024-09-21T09:52:59.990278+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:05:05.460719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990560+0000","last_change":"2024-09-21T09:52:20.391232+0000","last_active":"2024-09-21T09:52:59.990560+0000","last_peered":"2024-09-21T09:52:59.990560+0000","last_clean":"2024-09-21T09:52:59.990560+0000","last_became_active":"2024-09-21T09:52:20.390947+0000","last_became_peered":"2024-09-21T09:52:20.390947+0000","last_unstale":"2024-09-21T09:52:59.990560+0000","last_undegraded":"2024-09-21T09:52:59.990560+0000","last_fullsized":"2024-09-21T09:52:59.990560+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:03:11.273766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994355+0000","last_change":"2024-09-21T09:52:20.376219+0000","last_active":"2024-09-21T09:52:59.994355+0000","last_peered":"2024-09-21T09:52:59.994355+0000","last_clean":"2024-09-21T09:52:59.994355+0000","last_became_active":"2024-09-21T09:52:20.375957+0000","last_became_peered":"2024-09-21T09:52:20.375957+0000","last_unstale":"2024-09-21T09:52:59.994355+0000","last_undegraded":"2024-09-21T09:52:59.994355+0000","last_fullsized":"2024-09-21T09:52:59.994355+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:53.883620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.992702+0000","last_change":"2024-09-21T09:52:20.722803+0000","last_active":"2024-09-21T09:52:59.992702+0000","last_peered":"2024-09-21T09:52:59.992702+0000","last_clean":"2024-09-21T09:52:59.992702+0000","last_became_active":"2024-09-21T09:52:20.722527+0000","last_became_peered":"2024-09-21T09:52:20.722527+0000","last_unstale":"2024-09-21T09:52:59.992702+0000","last_undegraded":"2024-09-21T09:52:59.992702+0000","last_fullsized":"2024-09-21T09:52:59.992702+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:31.995729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"52'79","reported_seq":56,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:53:00.352114+0000","last_change":"2024-09-21T09:50:29.757187+0000","last_active":"2024-09-21T09:53:00.352114+0000","last_peered":"2024-09-21T09:53:00.352114+0000","last_clean":"2024-09-21T09:53:00.352114+0000","last_became_active":"2024-09-21T09:50:29.751734+0000","last_became_peered":"2024-09-21T09:50:29.751734+0000","last_unstale":"2024-09-21T09:53:00.352114+0000","last_undegraded":"2024-09-21T09:53:00.352114+0000","last_fullsized":"2024-09-21T09:53:00.352114+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T09:48:34.850872+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:20:45.636784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":27,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993655+0000","last_change":"2024-09-21T09:51:10.172804+0000","last_active":"2024-09-21T09:52:59.993655+0000","last_peered":"2024-09-21T09:52:59.993655+0000","last_clean":"2024-09-21T09:52:59.993655+0000","last_became_active":"2024-09-21T09:51:05.135491+0000","last_became_peered":"2024-09-21T09:51:05.135491+0000","last_unstale":"2024-09-21T09:52:59.993655+0000","last_undegraded":"2024-09-21T09:52:59.993655+0000","last_fullsized":"2024-09-21T09:52:59.993655+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:27:30.172694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034123299999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994414+0000","last_change":"2024-09-21T09:52:20.373618+0000","last_active":"2024-09-21T09:52:59.994414+0000","last_peered":"2024-09-21T09:52:59.994414+0000","last_clean":"2024-09-21T09:52:59.994414+0000","last_became_active":"2024-09-21T09:52:20.373365+0000","last_became_peered":"2024-09-21T09:52:20.373365+0000","last_unstale":"2024-09-21T09:52:59.994414+0000","last_undegraded":"2024-09-21T09:52:59.994414+0000","last_fullsized":"2024-09-21T09:52:59.994414+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:19.839696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.992759+0000","last_change":"2024-09-21T09:52:20.403534+0000","last_active":"2024-09-21T09:52:59.992759+0000","last_peered":"2024-09-21T09:52:59.992759+0000","last_clean":"2024-09-21T09:52:59.992759+0000","last_became_active":"2024-09-21T09:52:20.403258+0000","last_became_peered":"2024-09-21T09:52:20.403258+0000","last_unstale":"2024-09-21T09:52:59.992759+0000","last_undegraded":"2024-09-21T09:52:59.992759+0000","last_fullsized":"2024-09-21T09:52:59.992759+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:55.374343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:53:00.352056+0000","last_change":"2024-09-21T09:51:11.140693+0000","last_active":"2024-09-21T09:53:00.352056+0000","last_peered":"2024-09-21T09:53:00.352056+0000","last_clean":"2024-09-21T09:53:00.352056+0000","last_became_active":"2024-09-21T09:51:05.134160+0000","last_became_peered":"2024-09-21T09:51:05.134160+0000","last_unstale":"2024-09-21T09:53:00.352056+0000","last_undegraded":"2024-09-21T09:53:00.352056+0000","last_fullsized":"2024-09-21T09:53:00.352056+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:20.004781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028573500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.989892+0000","last_change":"2024-09-21T09:52:20.379256+0000","last_active":"2024-09-21T09:52:59.989892+0000","last_peered":"2024-09-21T09:52:59.989892+0000","last_clean":"2024-09-21T09:52:59.989892+0000","last_became_active":"2024-09-21T09:52:20.378963+0000","last_became_peered":"2024-09-21T09:52:20.378963+0000","last_unstale":"2024-09-21T09:52:59.989892+0000","last_undegraded":"2024-09-21T09:52:59.989892+0000","last_fullsized":"2024-09-21T09:52:59.989892+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:42:11.194772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990660+0000","last_change":"2024-09-21T09:52:20.394708+0000","last_active":"2024-09-21T09:52:59.990660+0000","last_peered":"2024-09-21T09:52:59.990660+0000","last_clean":"2024-09-21T09:52:59.990660+0000","last_became_active":"2024-09-21T09:52:20.394423+0000","last_became_peered":"2024-09-21T09:52:20.394423+0000","last_unstale":"2024-09-21T09:52:59.990660+0000","last_undegraded":"2024-09-21T09:52:59.990660+0000","last_fullsized":"2024-09-21T09:52:59.990660+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:46:16.384329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.989862+0000","last_change":"2024-09-21T09:51:10.168481+0000","last_active":"2024-09-21T09:52:59.989862+0000","last_peered":"2024-09-21T09:52:59.989862+0000","last_clean":"2024-09-21T09:52:59.989862+0000","last_became_active":"2024-09-21T09:51:05.134406+0000","last_became_peered":"2024-09-21T09:51:05.134406+0000","last_unstale":"2024-09-21T09:52:59.989862+0000","last_undegraded":"2024-09-21T09:52:59.989862+0000","last_fullsized":"2024-09-21T09:52:59.989862+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:14:34.828436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000364877,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.992199+0000","last_change":"2024-09-21T09:52:20.369967+0000","last_active":"2024-09-21T09:52:59.992199+0000","last_peered":"2024-09-21T09:52:59.992199+0000","last_clean":"2024-09-21T09:52:59.992199+0000","last_became_active":"2024-09-21T09:52:20.369681+0000","last_became_peered":"2024-09-21T09:52:20.369681+0000","last_unstale":"2024-09-21T09:52:59.992199+0000","last_undegraded":"2024-09-21T09:52:59.992199+0000","last_fullsized":"2024-09-21T09:52:59.992199+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:16:11.034455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:53:01.058799+0000","last_change":"2024-09-21T09:53:01.058799+0000","last_active":"2024-09-21T09:53:01.058799+0000","last_peered":"2024-09-21T09:53:01.058799+0000","last_clean":"2024-09-21T09:53:01.058799+0000","last_became_active":"2024-09-21T09:53:01.058487+0000","last_became_peered":"2024-09-21T09:53:01.058487+0000","last_unstale":"2024-09-21T09:53:01.058799+0000","last_undegraded":"2024-09-21T09:53:01.058799+0000","last_fullsized":"2024-09-21T09:53:01.058799+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:37.423153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":28,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993547+0000","last_change":"2024-09-21T09:51:10.173383+0000","last_active":"2024-09-21T09:52:59.993547+0000","last_peered":"2024-09-21T09:52:59.993547+0000","last_clean":"2024-09-21T09:52:59.993547+0000","last_became_active":"2024-09-21T09:51:05.134269+0000","last_became_peered":"2024-09-21T09:51:05.134269+0000","last_unstale":"2024-09-21T09:52:59.993547+0000","last_undegraded":"2024-09-21T09:52:59.993547+0000","last_fullsized":"2024-09-21T09:52:59.993547+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:23.879314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091501400000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990623+0000","last_change":"2024-09-21T09:52:20.375323+0000","last_active":"2024-09-21T09:52:59.990623+0000","last_peered":"2024-09-21T09:52:59.990623+0000","last_clean":"2024-09-21T09:52:59.990623+0000","last_became_active":"2024-09-21T09:52:20.375036+0000","last_became_peered":"2024-09-21T09:52:20.375036+0000","last_unstale":"2024-09-21T09:52:59.990623+0000","last_undegraded":"2024-09-21T09:52:59.990623+0000","last_fullsized":"2024-09-21T09:52:59.990623+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:07.500769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:53:01.929555+0000","last_change":"2024-09-21T09:53:01.929555+0000","last_active":"2024-09-21T09:53:01.929555+0000","last_peered":"2024-09-21T09:53:01.929555+0000","last_clean":"2024-09-21T09:53:01.929555+0000","last_became_active":"2024-09-21T09:53:01.929292+0000","last_became_peered":"2024-09-21T09:53:01.929292+0000","last_unstale":"2024-09-21T09:53:01.929555+0000","last_undegraded":"2024-09-21T09:53:01.929555+0000","last_fullsized":"2024-09-21T09:53:01.929555+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:26:07.291513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993917+0000","last_change":"2024-09-21T09:52:20.376173+0000","last_active":"2024-09-21T09:52:59.993917+0000","last_peered":"2024-09-21T09:52:59.993917+0000","last_clean":"2024-09-21T09:52:59.993917+0000","last_became_active":"2024-09-21T09:52:20.375977+0000","last_became_peered":"2024-09-21T09:52:20.375977+0000","last_unstale":"2024-09-21T09:52:59.993917+0000","last_undegraded":"2024-09-21T09:52:59.993917+0000","last_fullsized":"2024-09-21T09:52:59.993917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:26:30.599655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.992767+0000","last_change":"2024-09-21T09:51:10.174269+0000","last_active":"2024-09-21T09:52:59.992767+0000","last_peered":"2024-09-21T09:52:59.992767+0000","last_clean":"2024-09-21T09:52:59.992767+0000","last_became_active":"2024-09-21T09:51:05.134200+0000","last_became_peered":"2024-09-21T09:51:05.134200+0000","last_unstale":"2024-09-21T09:52:59.992767+0000","last_undegraded":"2024-09-21T09:52:59.992767+0000","last_fullsized":"2024-09-21T09:52:59.992767+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:38:40.009638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036908299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.992325+0000","last_change":"2024-09-21T09:52:20.393526+0000","last_active":"2024-09-21T09:52:59.992325+0000","last_peered":"2024-09-21T09:52:59.992325+0000","last_clean":"2024-09-21T09:52:59.992325+0000","last_became_active":"2024-09-21T09:52:20.393130+0000","last_became_peered":"2024-09-21T09:52:20.393130+0000","last_unstale":"2024-09-21T09:52:59.992325+0000","last_undegraded":"2024-09-21T09:52:59.992325+0000","last_fullsized":"2024-09-21T09:52:59.992325+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:56.289484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.992693+0000","last_change":"2024-09-21T09:52:20.371750+0000","last_active":"2024-09-21T09:52:59.992693+0000","last_peered":"2024-09-21T09:52:59.992693+0000","last_clean":"2024-09-21T09:52:59.992693+0000","last_became_active":"2024-09-21T09:52:20.371479+0000","last_became_peered":"2024-09-21T09:52:20.371479+0000","last_unstale":"2024-09-21T09:52:59.992693+0000","last_undegraded":"2024-09-21T09:52:59.992693+0000","last_fullsized":"2024-09-21T09:52:59.992693+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:50:13.674878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:53:00.352458+0000","last_change":"2024-09-21T09:51:11.140616+0000","last_active":"2024-09-21T09:53:00.352458+0000","last_peered":"2024-09-21T09:53:00.352458+0000","last_clean":"2024-09-21T09:53:00.352458+0000","last_became_active":"2024-09-21T09:51:05.429200+0000","last_became_peered":"2024-09-21T09:51:05.429200+0000","last_unstale":"2024-09-21T09:53:00.352458+0000","last_undegraded":"2024-09-21T09:53:00.352458+0000","last_fullsized":"2024-09-21T09:53:00.352458+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:29:34.651310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031258499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990263+0000","last_change":"2024-09-21T09:52:20.403016+0000","last_active":"2024-09-21T09:52:59.990263+0000","last_peered":"2024-09-21T09:52:59.990263+0000","last_clean":"2024-09-21T09:52:59.990263+0000","last_became_active":"2024-09-21T09:52:20.402670+0000","last_became_peered":"2024-09-21T09:52:20.402670+0000","last_unstale":"2024-09-21T09:52:59.990263+0000","last_undegraded":"2024-09-21T09:52:59.990263+0000","last_fullsized":"2024-09-21T09:52:59.990263+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:38.432145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994758+0000","last_change":"2024-09-21T09:52:20.381501+0000","last_active":"2024-09-21T09:52:59.994758+0000","last_peered":"2024-09-21T09:52:59.994758+0000","last_clean":"2024-09-21T09:52:59.994758+0000","last_became_active":"2024-09-21T09:52:20.380059+0000","last_became_peered":"2024-09-21T09:52:20.380059+0000","last_unstale":"2024-09-21T09:52:59.994758+0000","last_undegraded":"2024-09-21T09:52:59.994758+0000","last_fullsized":"2024-09-21T09:52:59.994758+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:20:11.443657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993005+0000","last_change":"2024-09-21T09:51:10.174262+0000","last_active":"2024-09-21T09:52:59.993005+0000","last_peered":"2024-09-21T09:52:59.993005+0000","last_clean":"2024-09-21T09:52:59.993005+0000","last_became_active":"2024-09-21T09:51:05.429405+0000","last_became_peered":"2024-09-21T09:51:05.429405+0000","last_unstale":"2024-09-21T09:52:59.993005+0000","last_undegraded":"2024-09-21T09:52:59.993005+0000","last_fullsized":"2024-09-21T09:52:59.993005+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:57:49.920758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000351963,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994274+0000","last_change":"2024-09-21T09:52:21.388589+0000","last_active":"2024-09-21T09:52:59.994274+0000","last_peered":"2024-09-21T09:52:59.994274+0000","last_clean":"2024-09-21T09:52:59.994274+0000","last_became_active":"2024-09-21T09:52:21.388371+0000","last_became_peered":"2024-09-21T09:52:21.388371+0000","last_unstale":"2024-09-21T09:52:59.994274+0000","last_undegraded":"2024-09-21T09:52:59.994274+0000","last_fullsized":"2024-09-21T09:52:59.994274+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:16:02.942895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990787+0000","last_change":"2024-09-21T09:52:20.376117+0000","last_active":"2024-09-21T09:52:59.990787+0000","last_peered":"2024-09-21T09:52:59.990787+0000","last_clean":"2024-09-21T09:52:59.990787+0000","last_became_active":"2024-09-21T09:52:20.375329+0000","last_became_peered":"2024-09-21T09:52:20.375329+0000","last_unstale":"2024-09-21T09:52:59.990787+0000","last_undegraded":"2024-09-21T09:52:59.990787+0000","last_fullsized":"2024-09-21T09:52:59.990787+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:05:21.158761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":26,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993096+0000","last_change":"2024-09-21T09:51:10.174279+0000","last_active":"2024-09-21T09:52:59.993096+0000","last_peered":"2024-09-21T09:52:59.993096+0000","last_clean":"2024-09-21T09:52:59.993096+0000","last_became_active":"2024-09-21T09:51:05.133025+0000","last_became_peered":"2024-09-21T09:51:05.133025+0000","last_unstale":"2024-09-21T09:52:59.993096+0000","last_undegraded":"2024-09-21T09:52:59.993096+0000","last_fullsized":"2024-09-21T09:52:59.993096+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:47.204645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029041499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.989981+0000","last_change":"2024-09-21T09:52:20.391439+0000","last_active":"2024-09-21T09:52:59.989981+0000","last_peered":"2024-09-21T09:52:59.989981+0000","last_clean":"2024-09-21T09:52:59.989981+0000","last_became_active":"2024-09-21T09:52:20.391157+0000","last_became_peered":"2024-09-21T09:52:20.391157+0000","last_unstale":"2024-09-21T09:52:59.989981+0000","last_undegraded":"2024-09-21T09:52:59.989981+0000","last_fullsized":"2024-09-21T09:52:59.989981+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:36.788330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990526+0000","last_change":"2024-09-21T09:52:20.391844+0000","last_active":"2024-09-21T09:52:59.990526+0000","last_peered":"2024-09-21T09:52:59.990526+0000","last_clean":"2024-09-21T09:52:59.990526+0000","last_became_active":"2024-09-21T09:52:20.391561+0000","last_became_peered":"2024-09-21T09:52:20.391561+0000","last_unstale":"2024-09-21T09:52:59.990526+0000","last_undegraded":"2024-09-21T09:52:59.990526+0000","last_fullsized":"2024-09-21T09:52:59.990526+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:37.669003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"64'6","reported_seq":20,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990343+0000","last_change":"2024-09-21T09:52:20.371668+0000","last_active":"2024-09-21T09:52:59.990343+0000","last_peered":"2024-09-21T09:52:59.990343+0000","last_clean":"2024-09-21T09:52:59.990343+0000","last_became_active":"2024-09-21T09:52:20.371389+0000","last_became_peered":"2024-09-21T09:52:20.371389+0000","last_unstale":"2024-09-21T09:52:59.990343+0000","last_undegraded":"2024-09-21T09:52:59.990343+0000","last_fullsized":"2024-09-21T09:52:59.990343+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:10.013020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1552,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.992338+0000","last_change":"2024-09-21T09:52:20.391921+0000","last_active":"2024-09-21T09:52:59.992338+0000","last_peered":"2024-09-21T09:52:59.992338+0000","last_clean":"2024-09-21T09:52:59.992338+0000","last_became_active":"2024-09-21T09:52:20.391648+0000","last_became_peered":"2024-09-21T09:52:20.391648+0000","last_unstale":"2024-09-21T09:52:59.992338+0000","last_undegraded":"2024-09-21T09:52:59.992338+0000","last_fullsized":"2024-09-21T09:52:59.992338+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:10.365545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":17,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:53:00.352300+0000","last_change":"2024-09-21T09:52:20.377816+0000","last_active":"2024-09-21T09:53:00.352300+0000","last_peered":"2024-09-21T09:53:00.352300+0000","last_clean":"2024-09-21T09:53:00.352300+0000","last_became_active":"2024-09-21T09:52:20.377550+0000","last_became_peered":"2024-09-21T09:52:20.377550+0000","last_unstale":"2024-09-21T09:53:00.352300+0000","last_undegraded":"2024-09-21T09:53:00.352300+0000","last_fullsized":"2024-09-21T09:53:00.352300+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:16:36.207459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993926+0000","last_change":"2024-09-21T09:52:20.376061+0000","last_active":"2024-09-21T09:52:59.993926+0000","last_peered":"2024-09-21T09:52:59.993926+0000","last_clean":"2024-09-21T09:52:59.993926+0000","last_became_active":"2024-09-21T09:52:20.375801+0000","last_became_peered":"2024-09-21T09:52:20.375801+0000","last_unstale":"2024-09-21T09:52:59.993926+0000","last_undegraded":"2024-09-21T09:52:59.993926+0000","last_fullsized":"2024-09-21T09:52:59.993926+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:15:54.908397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994332+0000","last_change":"2024-09-21T09:52:21.547147+0000","last_active":"2024-09-21T09:52:59.994332+0000","last_peered":"2024-09-21T09:52:59.994332+0000","last_clean":"2024-09-21T09:52:59.994332+0000","last_became_active":"2024-09-21T09:52:21.546878+0000","last_became_peered":"2024-09-21T09:52:21.546878+0000","last_unstale":"2024-09-21T09:52:59.994332+0000","last_undegraded":"2024-09-21T09:52:59.994332+0000","last_fullsized":"2024-09-21T09:52:59.994332+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:41.758824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990717+0000","last_change":"2024-09-21T09:52:20.378644+0000","last_active":"2024-09-21T09:52:59.990717+0000","last_peered":"2024-09-21T09:52:59.990717+0000","last_clean":"2024-09-21T09:52:59.990717+0000","last_became_active":"2024-09-21T09:52:20.378375+0000","last_became_peered":"2024-09-21T09:52:20.378375+0000","last_unstale":"2024-09-21T09:52:59.990717+0000","last_undegraded":"2024-09-21T09:52:59.990717+0000","last_fullsized":"2024-09-21T09:52:59.990717+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:00:45.021261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994261+0000","last_change":"2024-09-21T09:52:22.010842+0000","last_active":"2024-09-21T09:52:59.994261+0000","last_peered":"2024-09-21T09:52:59.994261+0000","last_clean":"2024-09-21T09:52:59.994261+0000","last_became_active":"2024-09-21T09:52:22.010543+0000","last_became_peered":"2024-09-21T09:52:22.010543+0000","last_unstale":"2024-09-21T09:52:59.994261+0000","last_undegraded":"2024-09-21T09:52:59.994261+0000","last_fullsized":"2024-09-21T09:52:59.994261+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:45:05.955965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993091+0000","last_change":"2024-09-21T09:52:20.392713+0000","last_active":"2024-09-21T09:52:59.993091+0000","last_peered":"2024-09-21T09:52:59.993091+0000","last_clean":"2024-09-21T09:52:59.993091+0000","last_became_active":"2024-09-21T09:52:20.392414+0000","last_became_peered":"2024-09-21T09:52:20.392414+0000","last_unstale":"2024-09-21T09:52:59.993091+0000","last_undegraded":"2024-09-21T09:52:59.993091+0000","last_fullsized":"2024-09-21T09:52:59.993091+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:37:00.959645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994340+0000","last_change":"2024-09-21T09:52:20.381561+0000","last_active":"2024-09-21T09:52:59.994340+0000","last_peered":"2024-09-21T09:52:59.994340+0000","last_clean":"2024-09-21T09:52:59.994340+0000","last_became_active":"2024-09-21T09:52:20.381193+0000","last_became_peered":"2024-09-21T09:52:20.381193+0000","last_unstale":"2024-09-21T09:52:59.994340+0000","last_undegraded":"2024-09-21T09:52:59.994340+0000","last_fullsized":"2024-09-21T09:52:59.994340+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:05:47.715444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990595+0000","last_change":"2024-09-21T09:52:20.724722+0000","last_active":"2024-09-21T09:52:59.990595+0000","last_peered":"2024-09-21T09:52:59.990595+0000","last_clean":"2024-09-21T09:52:59.990595+0000","last_became_active":"2024-09-21T09:52:20.724458+0000","last_became_peered":"2024-09-21T09:52:20.724458+0000","last_unstale":"2024-09-21T09:52:59.990595+0000","last_undegraded":"2024-09-21T09:52:59.990595+0000","last_fullsized":"2024-09-21T09:52:59.990595+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:32.897754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.992205+0000","last_change":"2024-09-21T09:52:20.377605+0000","last_active":"2024-09-21T09:52:59.992205+0000","last_peered":"2024-09-21T09:52:59.992205+0000","last_clean":"2024-09-21T09:52:59.992205+0000","last_became_active":"2024-09-21T09:52:20.376848+0000","last_became_peered":"2024-09-21T09:52:20.376848+0000","last_unstale":"2024-09-21T09:52:59.992205+0000","last_undegraded":"2024-09-21T09:52:59.992205+0000","last_fullsized":"2024-09-21T09:52:59.992205+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:16:32.126379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994575+0000","last_change":"2024-09-21T09:52:22.010825+0000","last_active":"2024-09-21T09:52:59.994575+0000","last_peered":"2024-09-21T09:52:59.994575+0000","last_clean":"2024-09-21T09:52:59.994575+0000","last_became_active":"2024-09-21T09:52:22.010539+0000","last_became_peered":"2024-09-21T09:52:22.010539+0000","last_unstale":"2024-09-21T09:52:59.994575+0000","last_undegraded":"2024-09-21T09:52:59.994575+0000","last_fullsized":"2024-09-21T09:52:59.994575+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:53.018658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990083+0000","last_change":"2024-09-21T09:52:20.373508+0000","last_active":"2024-09-21T09:52:59.990083+0000","last_peered":"2024-09-21T09:52:59.990083+0000","last_clean":"2024-09-21T09:52:59.990083+0000","last_became_active":"2024-09-21T09:52:20.373191+0000","last_became_peered":"2024-09-21T09:52:20.373191+0000","last_unstale":"2024-09-21T09:52:59.990083+0000","last_undegraded":"2024-09-21T09:52:59.990083+0000","last_fullsized":"2024-09-21T09:52:59.990083+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:49:04.982379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990183+0000","last_change":"2024-09-21T09:52:20.724765+0000","last_active":"2024-09-21T09:52:59.990183+0000","last_peered":"2024-09-21T09:52:59.990183+0000","last_clean":"2024-09-21T09:52:59.990183+0000","last_became_active":"2024-09-21T09:52:20.724362+0000","last_became_peered":"2024-09-21T09:52:20.724362+0000","last_unstale":"2024-09-21T09:52:59.990183+0000","last_undegraded":"2024-09-21T09:52:59.990183+0000","last_fullsized":"2024-09-21T09:52:59.990183+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:40.063634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990388+0000","last_change":"2024-09-21T09:52:20.374193+0000","last_active":"2024-09-21T09:52:59.990388+0000","last_peered":"2024-09-21T09:52:59.990388+0000","last_clean":"2024-09-21T09:52:59.990388+0000","last_became_active":"2024-09-21T09:52:20.373860+0000","last_became_peered":"2024-09-21T09:52:20.373860+0000","last_unstale":"2024-09-21T09:52:59.990388+0000","last_undegraded":"2024-09-21T09:52:59.990388+0000","last_fullsized":"2024-09-21T09:52:59.990388+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:05:32.105795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990411+0000","last_change":"2024-09-21T09:52:20.722903+0000","last_active":"2024-09-21T09:52:59.990411+0000","last_peered":"2024-09-21T09:52:59.990411+0000","last_clean":"2024-09-21T09:52:59.990411+0000","last_became_active":"2024-09-21T09:52:20.722337+0000","last_became_peered":"2024-09-21T09:52:20.722337+0000","last_unstale":"2024-09-21T09:52:59.990411+0000","last_undegraded":"2024-09-21T09:52:59.990411+0000","last_fullsized":"2024-09-21T09:52:59.990411+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:32.585565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993648+0000","last_change":"2024-09-21T09:52:20.369353+0000","last_active":"2024-09-21T09:52:59.993648+0000","last_peered":"2024-09-21T09:52:59.993648+0000","last_clean":"2024-09-21T09:52:59.993648+0000","last_became_active":"2024-09-21T09:52:20.369058+0000","last_became_peered":"2024-09-21T09:52:20.369058+0000","last_unstale":"2024-09-21T09:52:59.993648+0000","last_undegraded":"2024-09-21T09:52:59.993648+0000","last_fullsized":"2024-09-21T09:52:59.993648+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:44:59.238491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990037+0000","last_change":"2024-09-21T09:52:20.723224+0000","last_active":"2024-09-21T09:52:59.990037+0000","last_peered":"2024-09-21T09:52:59.990037+0000","last_clean":"2024-09-21T09:52:59.990037+0000","last_became_active":"2024-09-21T09:52:20.722308+0000","last_became_peered":"2024-09-21T09:52:20.722308+0000","last_unstale":"2024-09-21T09:52:59.990037+0000","last_undegraded":"2024-09-21T09:52:59.990037+0000","last_fullsized":"2024-09-21T09:52:59.990037+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:43:23.115705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.993908+0000","last_change":"2024-09-21T09:52:20.371638+0000","last_active":"2024-09-21T09:52:59.993908+0000","last_peered":"2024-09-21T09:52:59.993908+0000","last_clean":"2024-09-21T09:52:59.993908+0000","last_became_active":"2024-09-21T09:52:20.371383+0000","last_became_peered":"2024-09-21T09:52:20.371383+0000","last_unstale":"2024-09-21T09:52:59.993908+0000","last_undegraded":"2024-09-21T09:52:59.993908+0000","last_fullsized":"2024-09-21T09:52:59.993908+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:33:45.384397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994490+0000","last_change":"2024-09-21T09:52:21.386491+0000","last_active":"2024-09-21T09:52:59.994490+0000","last_peered":"2024-09-21T09:52:59.994490+0000","last_clean":"2024-09-21T09:52:59.994490+0000","last_became_active":"2024-09-21T09:52:21.386086+0000","last_became_peered":"2024-09-21T09:52:21.386086+0000","last_unstale":"2024-09-21T09:52:59.994490+0000","last_undegraded":"2024-09-21T09:52:59.994490+0000","last_fullsized":"2024-09-21T09:52:59.994490+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:16.979789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.994708+0000","last_change":"2024-09-21T09:52:20.375466+0000","last_active":"2024-09-21T09:52:59.994708+0000","last_peered":"2024-09-21T09:52:59.994708+0000","last_clean":"2024-09-21T09:52:59.994708+0000","last_became_active":"2024-09-21T09:52:20.374169+0000","last_became_peered":"2024-09-21T09:52:20.374169+0000","last_unstale":"2024-09-21T09:52:59.994708+0000","last_undegraded":"2024-09-21T09:52:59.994708+0000","last_fullsized":"2024-09-21T09:52:59.994708+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:13:29.821368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":23,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:53:01.596702+0000","last_change":"2024-09-21T09:53:01.596702+0000","last_active":"2024-09-21T09:53:01.596702+0000","last_peered":"2024-09-21T09:53:01.596702+0000","last_clean":"2024-09-21T09:53:01.596702+0000","last_became_active":"2024-09-21T09:53:01.596342+0000","last_became_peered":"2024-09-21T09:53:01.596342+0000","last_unstale":"2024-09-21T09:53:01.596702+0000","last_undegraded":"2024-09-21T09:53:01.596702+0000","last_fullsized":"2024-09-21T09:53:01.596702+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:17:48.203293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":18,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:52:59.990681+0000","last_change":"2024-09-21T09:52:20.375908+0000","last_active":"2024-09-21T09:52:59.990681+0000","last_peered":"2024-09-21T09:52:59.990681+0000","last_clean":"2024-09-21T09:52:59.990681+0000","last_became_active":"2024-09-21T09:52:20.375644+0000","last_became_peered":"2024-09-21T09:52:20.375644+0000","last_unstale":"2024-09-21T09:52:59.990681+0000","last_undegraded":"2024-09-21T09:52:59.990681+0000","last_fullsized":"2024-09-21T09:52:59.990681+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:39:22.510856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2338,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397542,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298620,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446532,"statfs":{"total":95995035648,"available":95689248768,"internally_reserved":0,"allocated":1294336,"data_stored":888651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67200000000000004,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.374}]},{"osd":1,"last update":"Sat Sep 21 09:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.54700000000000004,"15min":0.55500000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.435},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.55600000000000005,"15min":0.56200000000000006},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.71599999999999997,"5min":0.745,"15min":0.745},"last":0.41899999999999998}]},{"osd":2,"last update":"Sat Sep 21 09:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52800000000000002,"15min":0.53200000000000003},"min":{"1min":0.39500000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.54300000000000004}]},{"osd":3,"last update":"Sat Sep 21 09:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.54200000000000004,"15min":0.54600000000000004},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56499999999999995,"15min":0.56699999999999995},"min":{"1min":0.442,"5min":0.371,"15min":0.371},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.56299999999999994}]},{"osd":4,"last update":"Sat Sep 21 09:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.40300000000000002,"15min":0.40100000000000002},"min":{"1min":0.30399999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.58199999999999996,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.46600000000000003}]},{"osd":5,"last update":"Sat Sep 21 09:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.42599999999999999,"15min":0.41899999999999998},"min":{"1min":0.32600000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.46700000000000003,"15min":0.46200000000000002},"min":{"1min":0.32300000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.63,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.48599999999999999}]},{"osd":6,"last update":"Sat Sep 21 09:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.47899999999999998,"15min":0.47399999999999998},"min":{"1min":0.36499999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47699999999999998,"15min":0.47499999999999998},"min":{"1min":0.33000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.5}]}]},{"osd":6,"up_from":43,"seq":184683593770,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298612,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446540,"statfs":{"total":95995035648,"available":95689256960,"internally_reserved":0,"allocated":1286144,"data_stored":888524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.48799999999999999,"15min":0.48499999999999999},"min":{"1min":0.32500000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.623,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48599999999999999,"15min":0.48399999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69199999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.45200000000000001}]},{"osd":1,"last update":"Sat Sep 21 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51000000000000001,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.58299999999999996,"15min":0.52000000000000002},"min":{"1min":0.372,"5min":0.309,"15min":0.309},"max":{"1min":8.8780000000000001,"5min":8.8780000000000001,"15min":8.8780000000000001},"last":0.63800000000000001}]},{"osd":2,"last update":"Sat Sep 21 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.50600000000000001,"15min":0.501},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.501,"15min":0.498},"min":{"1min":0.432,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.504}]},{"osd":3,"last update":"Sat Sep 21 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.502,"15min":0.495},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.64200000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52000000000000002,"15min":0.50800000000000001},"min":{"1min":0.42499999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.61499999999999999}]},{"osd":4,"last update":"Sat Sep 21 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.40000000000000002,"15min":0.38600000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.57399999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.433,"15min":0.42099999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.62,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.59799999999999998}]},{"osd":5,"last update":"Sat Sep 21 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.41499999999999998,"15min":0.40100000000000002},"min":{"1min":0.29199999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.42299999999999999,"15min":0.39500000000000002},"min":{"1min":0.35099999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.65200000000000002}]},{"osd":7,"last update":"Sat Sep 21 09:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.33700000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.497,"5min":0.434,"15min":0.42399999999999999},"min":{"1min":0.31900000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.47499999999999998}]}]},{"osd":5,"up_from":37,"seq":158913790000,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298024,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447128,"statfs":{"total":95995035648,"available":95689859072,"internally_reserved":0,"allocated":679936,"data_stored":296859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:52:41 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52900000000000003,"15min":0.51400000000000001},"min":{"1min":0.28599999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.68999999999999995,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50800000000000001,"15min":0.505},"min":{"1min":0.35999999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.67100000000000004,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.53400000000000003}]},{"osd":1,"last update":"Sat Sep 21 09:52:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54000000000000004,"15min":0.53200000000000003},"min":{"1min":0.33700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71799999999999997,"5min":0.748,"15min":0.748},"last":0.621},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51800000000000002,"15min":0.51200000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.495}]},{"osd":2,"last update":"Sat Sep 21 09:52:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51600000000000001,"15min":0.51100000000000001},"min":{"1min":0.41599999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.67900000000000005,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54200000000000004,"15min":0.52800000000000002},"min":{"1min":0.374,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.71999999999999997,"5min":0.747,"15min":0.747},"last":11.885999999999999}]},{"osd":3,"last update":"Sat Sep 21 09:52:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.48899999999999999,"15min":0.47499999999999998},"min":{"1min":0.38,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65900000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":11.923999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56100000000000005,"15min":0.55300000000000005},"min":{"1min":0.39900000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.73899999999999999,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.58299999999999996}]},{"osd":4,"last update":"Sat Sep 21 09:52:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.39400000000000002,"15min":0.36799999999999999},"min":{"1min":0.32900000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.40500000000000003,"15min":0.376},"min":{"1min":0.317,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.59999999999999998}]},{"osd":6,"last update":"Sat Sep 21 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.41799999999999998,"15min":0.39400000000000002},"min":{"1min":0.29299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.625,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.47299999999999998,"15min":0.439},"min":{"1min":0.30199999999999999,"5min":0.252,"15min":0.252},"max":{"1min":0.69799999999999995,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.55100000000000005}]},{"osd":7,"last update":"Sat Sep 21 09:52:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.52300000000000002,"15min":0.51600000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.48399999999999999,"15min":0.47799999999999998},"min":{"1min":0.34000000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.46899999999999997}]}]},{"osd":4,"up_from":31,"seq":133143986229,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298036,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447116,"statfs":{"total":95995035648,"available":95689846784,"internally_reserved":0,"allocated":696320,"data_stored":298213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:52:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.52000000000000002,"15min":0.52800000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.66100000000000003,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.38,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67900000000000005,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.54400000000000004}]},{"osd":1,"last update":"Sat Sep 21 09:52:22 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53000000000000003,"15min":0.53700000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.72699999999999998,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.54000000000000004,"15min":0.54400000000000004},"min":{"1min":0.373,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.70599999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.63100000000000001}]},{"osd":2,"last update":"Sat Sep 21 09:52:22 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.54600000000000004,"15min":0.56000000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.623,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.65400000000000003,"5min":0.879,"15min":0.879},"last":0.48999999999999999}]},{"osd":3,"last update":"Sat Sep 21 09:52:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53500000000000003},"min":{"1min":0.379,"5min":0.375,"15min":0.375},"max":{"1min":0.72999999999999998,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.53200000000000003,"15min":0.53500000000000003},"min":{"1min":0.39200000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.68100000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.59099999999999997}]},{"osd":5,"last update":"Sat Sep 21 09:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.39400000000000002,"15min":0.38100000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.621,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.39900000000000002,"15min":0.39300000000000002},"min":{"1min":0.27800000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.65900000000000003,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.41999999999999998}]},{"osd":6,"last update":"Sat Sep 21 09:53:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.436,"15min":0.41999999999999998},"min":{"1min":0.31,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.432,"15min":0.41799999999999998},"min":{"1min":0.32200000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.64000000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.46000000000000002}]},{"osd":7,"last update":"Sat Sep 21 09:52:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.5,"15min":0.502},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68899999999999995,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.61399999999999999,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.56399999999999995}]}]},{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298020,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447132,"statfs":{"total":95995035648,"available":95689863168,"internally_reserved":0,"allocated":675840,"data_stored":296605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35999999999999999,"15min":0.35699999999999998},"min":{"1min":0.24299999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.48899999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.318},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38800000000000001,"15min":0.39400000000000002},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.498,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.32800000000000001}]},{"osd":1,"last update":"Sat Sep 21 09:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.35299999999999998,"15min":0.35499999999999998},"min":{"1min":0.24199999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.44700000000000001,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36599999999999999,"15min":0.36799999999999999},"min":{"1min":0.29599999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.51100000000000001,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.28000000000000003}]},{"osd":2,"last update":"Sat Sep 21 09:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.41099999999999998,"15min":0.40699999999999997},"min":{"1min":0.29799999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.56200000000000006,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.377,"5min":0.40200000000000002,"15min":0.41499999999999998},"min":{"1min":0.23000000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.53800000000000003,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.307}]},{"osd":4,"last update":"Sat Sep 21 09:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.55200000000000005,"15min":0.54600000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.70099999999999996,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.53100000000000003,"15min":0.51600000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.38900000000000001}]},{"osd":5,"last update":"Sat Sep 21 09:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.54000000000000004,"15min":0.52000000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.75600000000000001,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.378},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55200000000000005,"15min":0.54600000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70999999999999996,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.53900000000000003}]},{"osd":6,"last update":"Sat Sep 21 09:53:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.69299999999999995,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55900000000000005,"15min":0.55300000000000005},"min":{"1min":0.443,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.746,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.52400000000000002}]},{"osd":7,"last update":"Sat Sep 21 09:52:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61399999999999999,"15min":0.61299999999999999},"min":{"1min":0.49099999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.78100000000000003,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.436}]}]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298016,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447136,"statfs":{"total":95995035648,"available":95689867264,"internally_reserved":0,"allocated":679936,"data_stored":297119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:52:43 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.34599999999999997,"15min":0.34599999999999997},"min":{"1min":0.245,"5min":0.22,"15min":0.22},"max":{"1min":0.41599999999999998,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.34899999999999998,"15min":0.34100000000000003},"min":{"1min":0.26300000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.41099999999999998,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.25}]},{"osd":1,"last update":"Sat Sep 21 09:52:43 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.38700000000000001,"15min":0.38800000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.47999999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.39900000000000002,"15min":0.40100000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.45900000000000002,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.32500000000000001}]},{"osd":3,"last update":"Sat Sep 21 09:53:03 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.38400000000000001,"15min":0.39000000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.495,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.312},{"interface":"front","average":{"1min":0.38,"5min":0.40000000000000002,"15min":0.40899999999999997},"min":{"1min":0.27300000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.56299999999999994,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.33600000000000002}]},{"osd":4,"last update":"Sat Sep 21 09:52:20 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.53400000000000003,"15min":0.53600000000000003},"min":{"1min":0.39000000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.70899999999999996,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42599999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73799999999999999,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.60899999999999999}]},{"osd":5,"last update":"Sat Sep 21 09:52:51 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.55300000000000005,"15min":0.56399999999999995},"min":{"1min":0.38300000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.69899999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.624},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56100000000000005,"15min":0.57599999999999996},"min":{"1min":0.39400000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.66500000000000004,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.71799999999999997}]},{"osd":6,"last update":"Sat Sep 21 09:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.69399999999999995,"15min":0.624},"min":{"1min":0.40500000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.76700000000000002,"5min":10.569000000000001,"15min":10.569000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56299999999999994,"15min":0.57699999999999996},"min":{"1min":0.373,"5min":0.371,"15min":0.371},"max":{"1min":0.68000000000000005,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.68000000000000005}]},{"osd":7,"last update":"Sat Sep 21 09:52:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.58699999999999997,"15min":0.59099999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.75900000000000001,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.41399999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77500000000000002,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.79400000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298004,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447148,"statfs":{"total":95995035648,"available":95689879552,"internally_reserved":0,"allocated":667648,"data_stored":295904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34200000000000003,"15min":0.32800000000000001},"min":{"1min":0.20499999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.48999999999999999,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.224},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.35399999999999998,"15min":0.32800000000000001},"min":{"1min":0.25700000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.25700000000000001}]},{"osd":2,"last update":"Sat Sep 21 09:52:45 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39300000000000002,"15min":0.39100000000000001},"min":{"1min":0.28899999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.375,"15min":0.375},"min":{"1min":0.24099999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.55500000000000005,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.29899999999999999}]},{"osd":3,"last update":"Sat Sep 21 09:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.41699999999999998,"15min":0.42999999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.625,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.375,"5min":0.41999999999999998,"15min":0.43099999999999999},"min":{"1min":0.27600000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.59499999999999997,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.27600000000000002}]},{"osd":4,"last update":"Sat Sep 21 09:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.59699999999999998,"15min":0.57899999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.65400000000000003,"15min":0.58999999999999997},"min":{"1min":0.38600000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73299999999999998,"5min":8.4100000000000001,"15min":8.4100000000000001},"last":0.61699999999999999}]},{"osd":5,"last update":"Sat Sep 21 09:52:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":0.67100000000000004,"15min":0.61099999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":9.3179999999999996,"5min":9.3179999999999996,"15min":9.3179999999999996},"last":0.629},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.64400000000000002}]},{"osd":6,"last update":"Sat Sep 21 09:53:05 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58399999999999996,"15min":0.57999999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.83399999999999996,"15min":0.95399999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.78100000000000003,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.59599999999999997}]},{"osd":7,"last update":"Sat Sep 21 09:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.59599999999999997,"15min":0.59999999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76700000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.496},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.44700000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.63500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298600,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446552,"statfs":{"total":95995035648,"available":95689269248,"internally_reserved":0,"allocated":1273856,"data_stored":887475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 09:53:15 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34300000000000003,"15min":0.29899999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.46700000000000003,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.221},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36499999999999999,"15min":0.32800000000000001},"min":{"1min":0.23000000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.54000000000000004,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.30399999999999999}]},{"osd":2,"last update":"Sat Sep 21 09:52:38 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.34499999999999997,"15min":0.34300000000000003},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.46300000000000002,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.377,"15min":0.36899999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.55700000000000005,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.27500000000000002}]},{"osd":3,"last update":"Sat Sep 21 09:53:10 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.27300000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.58699999999999997,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.315},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.38700000000000001,"15min":0.38600000000000001},"min":{"1min":0.26700000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.51500000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.26200000000000001}]},{"osd":4,"last update":"Sat Sep 21 09:52:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.999,"15min":1.204},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.73899999999999999,"5min":15.444000000000001,"15min":15.444000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.441}]},{"osd":5,"last update":"Sat Sep 21 09:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.56299999999999994,"15min":0.55500000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.56100000000000005,"15min":0.55200000000000005},"min":{"1min":0.375,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.919,"5min":1.919,"15min":1.919},"last":0.372}]},{"osd":6,"last update":"Sat Sep 21 09:53:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.57199999999999995,"15min":0.56699999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55600000000000005,"15min":0.55000000000000004},"min":{"1min":0.40899999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.34499999999999997}]},{"osd":7,"last update":"Sat Sep 21 09:52:32 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.56000000000000005,"15min":0.54600000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.57599999999999996,"15min":0.57799999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.73099999999999998,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.49399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T09:53:24.210 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T09:53:24.210 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T09:53:24.210 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-21T09:53:24.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph health --format=json 2024-09-21T09:53:24.455 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:25.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:25 smithi098 ceph-mon[29239]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:53:25.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:25 smithi098 ceph-mon[29239]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:25.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:25 smithi098 ceph-mon[39960]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:53:25.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:25 smithi098 ceph-mon[39960]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:25.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:25 smithi196 ceph-mon[32670]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:53:25.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:25 smithi196 ceph-mon[32670]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:26.414 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T09:53:26.414 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T09:53:26.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:53:26 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:53:26] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:53:27.012 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-21T09:53:27.012 INFO:teuthology.run_tasks:Running task print... 2024-09-21T09:53:27.021 INFO:teuthology.task.print:**** upgrading first half of cluster, with stress **** 2024-09-21T09:53:27.021 INFO:teuthology.run_tasks:Running task parallel... 2024-09-21T09:53:27.029 INFO:teuthology.task.parallel:starting parallel... 2024-09-21T09:53:27.029 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-21T09:53:27.029 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-09-21T09:53:27.034 INFO:tasks.workunit:Pulling workunits from ref quincy 2024-09-21T09:53:27.034 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-21T09:53:27.034 DEBUG:teuthology.orchestra.run.smithi098:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-21T09:53:27.035 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-21T09:53:27.035 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-09-21T09:53:27.035 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-09-21T09:53:27.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph config set mgr mgr/cephadm/daemon_cache_timeout 60' 2024-09-21T09:53:27.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T09:53:27.052 INFO:teuthology.orchestra.run.smithi098.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-21T09:53:27.052 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-21T09:53:27.110 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-21T09:53:27.110 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-21T09:53:27.165 INFO:tasks.workunit:timeout=3h 2024-09-21T09:53:27.165 INFO:tasks.workunit:cleanup=True 2024-09-21T09:53:27.165 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch quincy https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2024-09-21T09:53:27.222 INFO:tasks.workunit.client.0.smithi098.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-21T09:53:27.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:27 smithi098 ceph-mon[29239]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:27.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:27 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/4183259084' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:53:27.242 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:27 smithi098 ceph-mon[39960]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:27.242 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:27 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/4183259084' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:53:27.290 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:27 smithi196 ceph-mon[32670]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:27 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/4183259084' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:53:29.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:29 smithi098 ceph-mon[29239]: pgmap v91: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:29.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:53:29 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:53:29] ENGINE Bus STOPPING 2024-09-21T09:53:29.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:29 smithi098 ceph-mon[39960]: pgmap v91: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:29.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:29 smithi196 ceph-mon[32670]: pgmap v91: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:29.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:53:29 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:53:29] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:53:29.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:53:29 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:53:29] ENGINE Bus STOPPED 2024-09-21T09:53:29.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:53:29 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:53:29] ENGINE Bus STARTING 2024-09-21T09:53:29.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:53:29 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:53:29] ENGINE Serving on http://:::9283 2024-09-21T09:53:29.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:53:29 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:53:29] ENGINE Bus STARTED 2024-09-21T09:53:29.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph config set global log_to_journald false --force' 2024-09-21T09:53:30.273 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:30.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:30 smithi098 ceph-mon[39960]: from='client.? ' entity='client.admin' 2024-09-21T09:53:30.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:30 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:30.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:30 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:30.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:30 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:30.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:30 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:53:30.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:30 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:53:30.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:30 smithi098 ceph-mon[29239]: from='client.? ' entity='client.admin' 2024-09-21T09:53:30.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:30 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:30.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:30 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:30.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:30 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:30.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:30 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:53:30.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:30 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:53:30.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:30 smithi196 ceph-mon[32670]: from='client.? ' entity='client.admin' 2024-09-21T09:53:30.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:30 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:30.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:30 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:30 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:30 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:53:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:30 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:53:31.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:53:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:53:30] "GET /metrics HTTP/1.1" 200 239620 "" "Prometheus/2.43.0" 2024-09-21T09:53:31.580 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:31 smithi098 ceph-mon[29239]: pgmap v92: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:31.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:31 smithi098 ceph-mon[39960]: pgmap v92: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:31.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:31 smithi196 ceph-mon[32670]: pgmap v92: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:33.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:33 smithi098 ceph-mon[29239]: pgmap v93: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:33.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:33 smithi098 ceph-mon[39960]: pgmap v93: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:33.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-09-21T09:53:33.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:33 smithi196 ceph-mon[32670]: pgmap v93: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:33.739 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-21T09:53:33.742 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:33.760 INFO:tasks.workunit.client.0.smithi098.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-21T09:53:33.762 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T09:53:33.762 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-21T09:53:35.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:35 smithi098 ceph-mon[29239]: pgmap v94: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:35.506 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:35 smithi098 ceph-mon[39960]: pgmap v94: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:35.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:35 smithi196 ceph-mon[32670]: pgmap v94: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:36.089 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-21T09:53:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:53:36.173 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-21T09:53:36.215 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T09:53:36.217 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T09:53:36.217 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-21T09:53:36.258 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T09:53:36.261 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-21T09:53:36.261 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-21T09:53:36.318 INFO:tasks.workunit:Running workunits matching rbd/import_export.sh on client.0... 2024-09-21T09:53:36.319 INFO:tasks.workunit:Running workunit rbd/import_export.sh... 2024-09-21T09:53:36.320 DEBUG:teuthology.orchestra.run.smithi098:workunit test rbd/import_export.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_CREATE_ARGS=--new-format adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/import_export.sh 2024-09-21T09:53:36.383 INFO:tasks.workunit.client.0.smithi098.stderr:+ export RBD_FORCE_ALLOW_V1=1 2024-09-21T09:53:36.384 INFO:tasks.workunit.client.0.smithi098.stderr:+ RBD_FORCE_ALLOW_V1=1 2024-09-21T09:53:36.384 INFO:tasks.workunit.client.0.smithi098.stderr:+ TMPDIR=/tmp/rbd_import_export_79553 2024-09-21T09:53:36.384 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm -rf /tmp/rbd_import_export_79553 2024-09-21T09:53:36.384 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir /tmp/rbd_import_export_79553 2024-09-21T09:53:36.385 INFO:tasks.workunit.client.0.smithi098.stderr:+ trap 'rm -rf /tmp/rbd_import_export_79553' INT TERM EXIT 2024-09-21T09:53:36.385 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir foo.79553 2024-09-21T09:53:36.386 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd import foo.79553 foo.dir 2024-09-21T09:53:36.475 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: cannot import a directory 2024-09-21T09:53:36.475 INFO:tasks.workunit.client.0.smithi098.stderr: Importing image: 0% complete...failed. 2024-09-21T09:53:36.475 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: import failed: (21) Is a directory 2024-09-21T09:53:36.478 INFO:tasks.workunit.client.0.smithi098.stderr:+ true 2024-09-21T09:53:36.478 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir foo.79553 2024-09-21T09:53:36.482 INFO:tasks.workunit.client.0.smithi098.stderr:+ dd if=/bin/sh of=/tmp/rbd_import_export_79553/img bs=1k count=1 seek=10 2024-09-21T09:53:36.483 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records in 2024-09-21T09:53:36.484 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records out 2024-09-21T09:53:36.484 INFO:tasks.workunit.client.0.smithi098.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 4.9553e-05 s, 20.7 MB/s 2024-09-21T09:53:36.484 INFO:tasks.workunit.client.0.smithi098.stderr:+ dd if=/bin/dd of=/tmp/rbd_import_export_79553/img bs=1k count=10 seek=100 2024-09-21T09:53:36.484 INFO:tasks.workunit.client.0.smithi098.stderr:10+0 records in 2024-09-21T09:53:36.484 INFO:tasks.workunit.client.0.smithi098.stderr:10+0 records out 2024-09-21T09:53:36.484 INFO:tasks.workunit.client.0.smithi098.stderr:10240 bytes (10 kB, 10 KiB) copied, 4.9706e-05 s, 206 MB/s 2024-09-21T09:53:36.484 INFO:tasks.workunit.client.0.smithi098.stderr:+ dd if=/bin/rm of=/tmp/rbd_import_export_79553/img bs=1k count=100 seek=1000 2024-09-21T09:53:36.485 INFO:tasks.workunit.client.0.smithi098.stderr:60+1 records in 2024-09-21T09:53:36.485 INFO:tasks.workunit.client.0.smithi098.stderr:60+1 records out 2024-09-21T09:53:36.485 INFO:tasks.workunit.client.0.smithi098.stderr:61640 bytes (62 kB, 60 KiB) copied, 0.000147685 s, 417 MB/s 2024-09-21T09:53:36.485 INFO:tasks.workunit.client.0.smithi098.stderr:+ dd if=/bin/ls of=/tmp/rbd_import_export_79553/img bs=1k seek=10000 2024-09-21T09:53:36.486 INFO:tasks.workunit.client.0.smithi098.stderr:137+1 records in 2024-09-21T09:53:36.487 INFO:tasks.workunit.client.0.smithi098.stderr:137+1 records out 2024-09-21T09:53:36.487 INFO:tasks.workunit.client.0.smithi098.stderr:140928 bytes (141 kB, 138 KiB) copied, 0.000579601 s, 243 MB/s 2024-09-21T09:53:36.487 INFO:tasks.workunit.client.0.smithi098.stderr:+ dd if=/bin/ln of=/tmp/rbd_import_export_79553/img bs=1k seek=100000 2024-09-21T09:53:36.487 INFO:tasks.workunit.client.0.smithi098.stderr:60+1 records in 2024-09-21T09:53:36.487 INFO:tasks.workunit.client.0.smithi098.stderr:60+1 records out 2024-09-21T09:53:36.488 INFO:tasks.workunit.client.0.smithi098.stderr:61592 bytes (62 kB, 60 KiB) copied, 0.000156253 s, 394 MB/s 2024-09-21T09:53:36.488 INFO:tasks.workunit.client.0.smithi098.stderr:+ dd if=/bin/grep of=/tmp/rbd_import_export_79553/img bs=1k seek=1000000 2024-09-21T09:53:36.488 INFO:tasks.workunit.client.0.smithi098.stderr:154+1 records in 2024-09-21T09:53:36.488 INFO:tasks.workunit.client.0.smithi098.stderr:154+1 records out 2024-09-21T09:53:36.489 INFO:tasks.workunit.client.0.smithi098.stderr:158176 bytes (158 kB, 154 KiB) copied, 0.000362343 s, 437 MB/s 2024-09-21T09:53:36.489 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd rm testimg 2024-09-21T09:53:36.528 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-21T09:53:36.528 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: delete error: (2) No such file or directory 2024-09-21T09:53:36.531 INFO:tasks.workunit.client.0.smithi098.stderr:+ true 2024-09-21T09:53:36.531 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd import --new-format /tmp/rbd_import_export_79553/img testimg 2024-09-21T09:53:36.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:53:36 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:53:36] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:53:37.083 INFO:tasks.workunit.client.0.smithi098.stderr: Importing image: 1% complete... Importing image: 2% complete... Importing image: 3% complete... Importing image: 4% complete... Importing image: 5% complete... Importing image: 6% complete... Importing image: 7% complete... Importing image: 8% complete... Importing image: 9% complete... Importing image: 10% complete... Importing image: 11% complete... Importing image: 12% complete... Importing image: 13% complete... Importing image: 14% complete... Importing image: 15% complete... Importing image: 16% complete... Importing image: 17% complete... Importing image: 18% complete... Importing image: 19% complete... Importing image: 20% complete... Importing image: 21% complete... Importing image: 22% complete... Importing image: 23% complete... Importing image: 24% complete... Importing image: 25% complete... Importing image: 26% complete... Importing image: 27% complete... Importing image: 28% complete... Importing image: 29% complete... Importing image: 30% complete... Importing image: 31% complete... Importing image: 32% complete... Importing image: 33% complete... Importing image: 34% complete... Importing image: 35% complete... Importing image: 36% complete... Importing image: 37% complete... Importing image: 38% complete... Importing image: 39% complete... Importing image: 40% complete... Importing image: 41% complete... Importing image: 42% complete... Importing image: 43% complete... Importing image: 44% complete... Importing image: 45% complete... Importing image: 46% complete... Importing image: 47% complete... Importing image: 48% complete... Importing image: 49% complete... Importing image: 50% complete... Importing image: 51% complete... Importing image: 52% complete... Importing image: 53% complete... Importing image: 54% complete... Importing image: 55% complete... Importing image: 56% complete... Importing image: 57% complete... Importing image: 58% complete... Importing image: 59% complete... Importing image: 60% complete... Importing image: 61% complete... Importing image: 62% complete... Importing image: 63% complete... Importing image: 64% complete... Importing image: 65% complete... Importing image: 66% complete... Importing image: 67% complete... Importing image: 68% complete... Importing image: 69% complete... Importing image: 70% complete... Importing image: 71% complete... Importing image: 72% complete... Importing image: 73% complete... Importing image: 74% complete... Importing image: 75% complete... Importing image: 76% complete... Importing image: 77% complete... Importing image: 78% complete... Importing image: 79% complete... Importing image: 80% complete... Importing image: 81% complete... Importing image: 82% complete... Importing image: 83% complete... Importing image: 84% complete... Importing image: 85% complete... Importing image: 86% complete... Importing image: 87% complete... Importing image: 88% complete... Importing image: 89% complete... Importing image: 90% complete... Importing image: 91% complete... Importing image: 92% complete... Importing image: 93% complete... Importing image: 94% complete... Importing image: 95% complete... Importing image: 96% complete... Importing image: 97% complete... Importing image: 98% complete... Importing image: 99% complete... Importing image: 100% complete...done. 2024-09-21T09:53:37.083 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-21T09:53:37.086 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd export testimg /tmp/rbd_import_export_79553/img2 2024-09-21T09:53:37.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch ps' 2024-09-21T09:53:37.366 INFO:tasks.workunit.client.0.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-21T09:53:37.371 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd export testimg - 2024-09-21T09:53:37.518 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:37 smithi098 ceph-mon[29239]: pgmap v95: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:37 smithi098 ceph-mon[29239]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:37 smithi098 ceph-mon[29239]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:53:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:37 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:53:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:37 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:37 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:37 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:37 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:53:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:37 smithi098 ceph-mon[29239]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:53:37.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:37 smithi098 ceph-mon[39960]: pgmap v95: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:37.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:37 smithi098 ceph-mon[39960]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:37.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:37 smithi098 ceph-mon[39960]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:53:37.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:37 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:53:37.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:37 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:37.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:37 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:37.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:37 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:37.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:37 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:53:37.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:37 smithi098 ceph-mon[39960]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:53:37.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:37 smithi196 ceph-mon[32670]: pgmap v95: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:37.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:37 smithi196 ceph-mon[32670]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:37 smithi196 ceph-mon[32670]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:53:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:37 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:53:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:37 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:37 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:37 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:37 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:53:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:37 smithi196 ceph-mon[32670]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:53:38.123 INFO:tasks.workunit.client.0.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-21T09:53:38.126 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd rm testimg 2024-09-21T09:53:38.216 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-21T09:53:38.219 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmp /tmp/rbd_import_export_79553/img /tmp/rbd_import_export_79553/img2 2024-09-21T09:53:38.808 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmp /tmp/rbd_import_export_79553/img /tmp/rbd_import_export_79553/img3 2024-09-21T09:53:39.248 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm /tmp/rbd_import_export_79553/img2 /tmp/rbd_import_export_79553/img3 2024-09-21T09:53:39.299 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:39 smithi098 ceph-mon[29239]: pgmap v96: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:39.299 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:39 smithi098 ceph-mon[39960]: pgmap v96: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:39.443 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd import --new-format - testimg 2024-09-21T09:53:39.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:39 smithi196 ceph-mon[32670]: pgmap v96: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:39.639 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T09:53:39.639 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (86s) 56s ago 2m 15.8M - 0.25.0 c8568f914cd2 9ae44cc24819 2024-09-21T09:53:39.639 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi196 *:3000 running (2m) 61s ago 2m 82.8M - 9.4.7 954c08fa6188 be7bf40319e5 2024-09-21T09:53:39.639 INFO:teuthology.orchestra.run.smithi098.stdout:mds.foo.smithi098.stcwsu smithi098 running (72s) 56s ago 72s 15.4M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be d8393a5abb20 2024-09-21T09:53:39.639 INFO:teuthology.orchestra.run.smithi098.stdout:mds.foo.smithi196.ugblah smithi196 running (70s) 61s ago 70s 14.4M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 05a68593bc2b 2024-09-21T09:53:39.639 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi196 *:8443 running (6m) 61s ago 6m 399M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 3ed5ab6c3c4e 2024-09-21T09:53:39.639 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:9283,8765 running (7m) 56s ago 7m 443M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 93d2672b85f8 2024-09-21T09:53:39.639 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (8m) 56s ago 8m 46.7M 2048M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be f501577d66cb 2024-09-21T09:53:39.639 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi196 running (6m) 61s ago 6m 38.9M 2048M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 4f1a8a66c2c8 2024-09-21T09:53:39.639 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (6m) 56s ago 6m 41.2M 2048M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 54e9dc041848 2024-09-21T09:53:39.640 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (117s) 56s ago 116s 15.8M - 1.5.0 0da6a335fe13 20cd79611b03 2024-09-21T09:53:39.640 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi196 *:9100 running (113s) 61s ago 113s 7784k - 1.5.0 0da6a335fe13 5f6e262b5785 2024-09-21T09:53:39.640 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (6m) 56s ago 6m 42.0M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be b4d47286955f 2024-09-21T09:53:39.640 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (5m) 56s ago 5m 42.4M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 850efcb386e6 2024-09-21T09:53:39.640 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (5m) 56s ago 5m 42.6M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be f19a948d9994 2024-09-21T09:53:39.640 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (4m) 56s ago 4m 41.0M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 68f6e62795e7 2024-09-21T09:53:39.640 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi196 running (4m) 61s ago 4m 41.2M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 1693914f1a8b 2024-09-21T09:53:39.640 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi196 running (4m) 61s ago 4m 40.9M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be cb8e4afb2db7 2024-09-21T09:53:39.640 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi196 running (3m) 61s ago 3m 40.7M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 8ba9647073b2 2024-09-21T09:53:39.640 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi196 running (3m) 61s ago 3m 41.2M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 540c98a3c436 2024-09-21T09:53:39.640 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi196 *:9095 running (81s) 61s ago 2m 35.4M - 2.43.0 a07b618ecd1d 93f520945f7b 2024-09-21T09:53:39.880 INFO:tasks.workunit.client.0.smithi098.stderr: Importing image: 100% complete...done. 2024-09-21T09:53:39.881 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-21T09:53:39.885 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd export testimg /tmp/rbd_import_export_79553/img2 2024-09-21T09:53:40.161 INFO:tasks.workunit.client.0.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-21T09:53:40.164 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd export testimg - 2024-09-21T09:53:40.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'echo wait for minority of mons to upgrade' 2024-09-21T09:53:40.737 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:40.975 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:53:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:53:40] "GET /metrics HTTP/1.1" 200 239620 "" "Prometheus/2.43.0" 2024-09-21T09:53:40.992 INFO:tasks.workunit.client.0.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-21T09:53:40.996 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd rm testimg 2024-09-21T09:53:41.082 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-21T09:53:41.085 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmp /tmp/rbd_import_export_79553/img /tmp/rbd_import_export_79553/img2 2024-09-21T09:53:41.235 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:41 smithi098 ceph-mon[29239]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:41.235 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:41 smithi098 ceph-mon[29239]: pgmap v97: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:41.236 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:41 smithi098 ceph-mon[39960]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:41.236 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:41 smithi098 ceph-mon[39960]: pgmap v97: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:41.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:41 smithi196 ceph-mon[32670]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:41.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:41 smithi196 ceph-mon[32670]: pgmap v97: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:41.680 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmp /tmp/rbd_import_export_79553/img /tmp/rbd_import_export_79553/img3 2024-09-21T09:53:42.117 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm /tmp/rbd_import_export_79553/img /tmp/rbd_import_export_79553/img2 /tmp/rbd_import_export_79553/img3 2024-09-21T09:53:42.363 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd help export 2024-09-21T09:53:42.364 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep -q export-format 2024-09-21T09:53:42.378 INFO:tasks.workunit.client.0.smithi098.stderr:+ dd if=/bin/dd of=/tmp/rbd_import_export_79553/img bs=1k count=10 seek=100 2024-09-21T09:53:42.380 INFO:tasks.workunit.client.0.smithi098.stderr:10+0 records in 2024-09-21T09:53:42.380 INFO:tasks.workunit.client.0.smithi098.stderr:10+0 records out 2024-09-21T09:53:42.380 INFO:tasks.workunit.client.0.smithi098.stderr:10240 bytes (10 kB, 10 KiB) copied, 5.8871e-05 s, 174 MB/s 2024-09-21T09:53:42.380 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd import --new-format /tmp/rbd_import_export_79553/img testimg 2024-09-21T09:53:42.442 INFO:tasks.workunit.client.0.smithi098.stderr: Importing image: 100% complete...done. 2024-09-21T09:53:42.443 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-21T09:53:42.446 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd snap create testimg@snap 2024-09-21T09:53:42.961 INFO:teuthology.orchestra.run.smithi098.stdout:wait for minority of mons to upgrade 2024-09-21T09:53:43.176 INFO:tasks.workunit.client.0.smithi098.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-21T09:53:43.182 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd image-meta set testimg key1 value1 2024-09-21T09:53:43.235 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd image-meta list testimg 2024-09-21T09:53:43.276 INFO:tasks.workunit.client.0.smithi098.stderr:+ IMAGEMETA_BEFORE='There is 1 metadatum on this image: 2024-09-21T09:53:43.276 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-09-21T09:53:43.276 INFO:tasks.workunit.client.0.smithi098.stderr:Key Value 2024-09-21T09:53:43.277 INFO:tasks.workunit.client.0.smithi098.stderr:key1 value1' 2024-09-21T09:53:43.277 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_79553/img_v2 2024-09-21T09:53:43.325 INFO:tasks.workunit.client.0.smithi098.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2024-09-21T09:53:43.329 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_79553/img_v2 testimg_import 2024-09-21T09:53:43.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:43 smithi098 ceph-mon[29239]: pgmap v98: 73 pgs: 73 active+clean; 5.9 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 172 KiB/s rd, 77 KiB/s wr, 54 op/s 2024-09-21T09:53:43.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:43 smithi098 ceph-mon[39960]: pgmap v98: 73 pgs: 73 active+clean; 5.9 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 172 KiB/s rd, 77 KiB/s wr, 54 op/s 2024-09-21T09:53:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:43 smithi196 ceph-mon[32670]: pgmap v98: 73 pgs: 73 active+clean; 5.9 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 172 KiB/s rd, 77 KiB/s wr, 54 op/s 2024-09-21T09:53:43.793 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'while ! ceph mon versions | grep $sha1 && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-21T09:53:44.109 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:44.186 INFO:tasks.workunit.client.0.smithi098.stderr: Importing image: 50% complete... Importing image: 100% complete... Importing image: 100% complete...done. 2024-09-21T09:53:44.189 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd info testimg_import 2024-09-21T09:53:44.225 INFO:tasks.workunit.client.0.smithi098.stdout:rbd image 'testimg_import': 2024-09-21T09:53:44.225 INFO:tasks.workunit.client.0.smithi098.stdout: size 110 KiB in 1 objects 2024-09-21T09:53:44.225 INFO:tasks.workunit.client.0.smithi098.stdout: order 22 (4 MiB objects) 2024-09-21T09:53:44.226 INFO:tasks.workunit.client.0.smithi098.stdout: snapshot_count: 1 2024-09-21T09:53:44.226 INFO:tasks.workunit.client.0.smithi098.stdout: id: 3a35fc1cc737 2024-09-21T09:53:44.226 INFO:tasks.workunit.client.0.smithi098.stdout: block_name_prefix: rbd_data.3a35fc1cc737 2024-09-21T09:53:44.226 INFO:tasks.workunit.client.0.smithi098.stdout: format: 2 2024-09-21T09:53:44.226 INFO:tasks.workunit.client.0.smithi098.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2024-09-21T09:53:44.226 INFO:tasks.workunit.client.0.smithi098.stdout: op_features: 2024-09-21T09:53:44.226 INFO:tasks.workunit.client.0.smithi098.stdout: flags: 2024-09-21T09:53:44.226 INFO:tasks.workunit.client.0.smithi098.stdout: create_timestamp: Sat Sep 21 09:53:43 2024 2024-09-21T09:53:44.226 INFO:tasks.workunit.client.0.smithi098.stdout: access_timestamp: Sat Sep 21 09:53:43 2024 2024-09-21T09:53:44.226 INFO:tasks.workunit.client.0.smithi098.stdout: modify_timestamp: Sat Sep 21 09:53:43 2024 2024-09-21T09:53:44.228 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd info testimg_import@snap 2024-09-21T09:53:44.267 INFO:tasks.workunit.client.0.smithi098.stdout:rbd image 'testimg_import': 2024-09-21T09:53:44.267 INFO:tasks.workunit.client.0.smithi098.stdout: size 110 KiB in 1 objects 2024-09-21T09:53:44.267 INFO:tasks.workunit.client.0.smithi098.stdout: order 22 (4 MiB objects) 2024-09-21T09:53:44.267 INFO:tasks.workunit.client.0.smithi098.stdout: snapshot_count: 1 2024-09-21T09:53:44.267 INFO:tasks.workunit.client.0.smithi098.stdout: id: 3a35fc1cc737 2024-09-21T09:53:44.267 INFO:tasks.workunit.client.0.smithi098.stdout: block_name_prefix: rbd_data.3a35fc1cc737 2024-09-21T09:53:44.267 INFO:tasks.workunit.client.0.smithi098.stdout: format: 2 2024-09-21T09:53:44.268 INFO:tasks.workunit.client.0.smithi098.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2024-09-21T09:53:44.268 INFO:tasks.workunit.client.0.smithi098.stdout: op_features: 2024-09-21T09:53:44.268 INFO:tasks.workunit.client.0.smithi098.stdout: flags: 2024-09-21T09:53:44.268 INFO:tasks.workunit.client.0.smithi098.stdout: create_timestamp: Sat Sep 21 09:53:43 2024 2024-09-21T09:53:44.268 INFO:tasks.workunit.client.0.smithi098.stdout: access_timestamp: Sat Sep 21 09:53:43 2024 2024-09-21T09:53:44.268 INFO:tasks.workunit.client.0.smithi098.stdout: modify_timestamp: Sat Sep 21 09:53:43 2024 2024-09-21T09:53:44.268 INFO:tasks.workunit.client.0.smithi098.stdout: protected: False 2024-09-21T09:53:44.270 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd image-meta list testimg_import 2024-09-21T09:53:44.311 INFO:tasks.workunit.client.0.smithi098.stderr:+ IMAGEMETA_AFTER='There is 1 metadatum on this image: 2024-09-21T09:53:44.311 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-09-21T09:53:44.311 INFO:tasks.workunit.client.0.smithi098.stderr:Key Value 2024-09-21T09:53:44.312 INFO:tasks.workunit.client.0.smithi098.stderr:key1 value1' 2024-09-21T09:53:44.312 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 'There is 1 metadatum on this image: 2024-09-21T09:53:44.312 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-09-21T09:53:44.312 INFO:tasks.workunit.client.0.smithi098.stderr:Key Value 2024-09-21T09:53:44.312 INFO:tasks.workunit.client.0.smithi098.stderr:key1 value1' = 'There is 1 metadatum on this image: 2024-09-21T09:53:44.312 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-09-21T09:53:44.312 INFO:tasks.workunit.client.0.smithi098.stderr:Key Value 2024-09-21T09:53:44.312 INFO:tasks.workunit.client.0.smithi098.stderr:key1 value1' ']' 2024-09-21T09:53:44.312 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd export testimg_import /tmp/rbd_import_export_79553/img_import 2024-09-21T09:53:44.348 INFO:tasks.workunit.client.0.smithi098.stderr: Exporting image: 100% complete...done. 2024-09-21T09:53:44.351 INFO:tasks.workunit.client.0.smithi098.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_79553/img /tmp/rbd_import_export_79553/img_import 2024-09-21T09:53:44.351 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmp -l /tmp/rbd_import_export_79553/img /tmp/rbd_import_export_79553/img_import 2024-09-21T09:53:44.352 INFO:tasks.workunit.client.0.smithi098.stderr:++ stat /tmp/rbd_import_export_79553/img --format %b 2024-09-21T09:53:44.353 INFO:tasks.workunit.client.0.smithi098.stderr:+ origsize=24 2024-09-21T09:53:44.353 INFO:tasks.workunit.client.0.smithi098.stderr:++ stat /tmp/rbd_import_export_79553/img_import --format %b 2024-09-21T09:53:44.354 INFO:tasks.workunit.client.0.smithi098.stderr:+ exportsize=224 2024-09-21T09:53:44.354 INFO:tasks.workunit.client.0.smithi098.stderr:+ difference=200 2024-09-21T09:53:44.354 INFO:tasks.workunit.client.0.smithi098.stderr:+ difference=200 2024-09-21T09:53:44.355 INFO:tasks.workunit.client.0.smithi098.stderr:+ test 200 -ge 0 -a 200 -lt 4096 2024-09-21T09:53:44.355 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd export testimg@snap /tmp/rbd_import_export_79553/img_snap 2024-09-21T09:53:44.393 INFO:tasks.workunit.client.0.smithi098.stderr: Exporting image: 100% complete...done. 2024-09-21T09:53:44.396 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd export testimg_import@snap /tmp/rbd_import_export_79553/img_snap_import 2024-09-21T09:53:44.435 INFO:tasks.workunit.client.0.smithi098.stderr: Exporting image: 100% complete...done. 2024-09-21T09:53:44.438 INFO:tasks.workunit.client.0.smithi098.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_79553/img_snap /tmp/rbd_import_export_79553/img_snap_import 2024-09-21T09:53:44.438 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmp -l /tmp/rbd_import_export_79553/img_snap /tmp/rbd_import_export_79553/img_snap_import 2024-09-21T09:53:44.439 INFO:tasks.workunit.client.0.smithi098.stderr:++ stat /tmp/rbd_import_export_79553/img_snap --format %b 2024-09-21T09:53:44.440 INFO:tasks.workunit.client.0.smithi098.stderr:+ origsize=224 2024-09-21T09:53:44.440 INFO:tasks.workunit.client.0.smithi098.stderr:++ stat /tmp/rbd_import_export_79553/img_snap_import --format %b 2024-09-21T09:53:44.441 INFO:tasks.workunit.client.0.smithi098.stderr:+ exportsize=224 2024-09-21T09:53:44.441 INFO:tasks.workunit.client.0.smithi098.stderr:+ difference=0 2024-09-21T09:53:44.441 INFO:tasks.workunit.client.0.smithi098.stderr:+ difference=0 2024-09-21T09:53:44.441 INFO:tasks.workunit.client.0.smithi098.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2024-09-21T09:53:44.441 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm /tmp/rbd_import_export_79553/img_v2 2024-09-21T09:53:44.442 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm /tmp/rbd_import_export_79553/img_import 2024-09-21T09:53:44.443 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm /tmp/rbd_import_export_79553/img_snap 2024-09-21T09:53:44.444 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm /tmp/rbd_import_export_79553/img_snap_import 2024-09-21T09:53:44.444 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd snap rm testimg_import@snap 2024-09-21T09:53:44.489 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:44 smithi098 ceph-mon[39960]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T09:53:44.489 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:44 smithi098 ceph-mon[29239]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T09:53:44.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:44 smithi196 ceph-mon[32670]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T09:53:45.185 INFO:tasks.workunit.client.0.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-21T09:53:45.194 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd remove testimg_import 2024-09-21T09:53:45.283 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 100% complete...done. 2024-09-21T09:53:45.285 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd snap rm testimg@snap 2024-09-21T09:53:45.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:45 smithi098 ceph-mon[29239]: pgmap v100: 73 pgs: 73 active+clean; 5.9 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 206 KiB/s rd, 93 KiB/s wr, 65 op/s 2024-09-21T09:53:45.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:45 smithi098 ceph-mon[29239]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T09:53:45.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:45 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:53:45.542 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:45 smithi098 ceph-mon[39960]: pgmap v100: 73 pgs: 73 active+clean; 5.9 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 206 KiB/s rd, 93 KiB/s wr, 65 op/s 2024-09-21T09:53:45.542 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:45 smithi098 ceph-mon[39960]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T09:53:45.543 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:45 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:53:45.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:45 smithi196 ceph-mon[32670]: pgmap v100: 73 pgs: 73 active+clean; 5.9 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 206 KiB/s rd, 93 KiB/s wr, 65 op/s 2024-09-21T09:53:45.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:45 smithi196 ceph-mon[32670]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T09:53:45.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:45 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:53:46.198 INFO:tasks.workunit.client.0.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-21T09:53:46.205 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd rm testimg 2024-09-21T09:53:46.300 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 100% complete...done. 2024-09-21T09:53:46.303 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd import --order 20 /tmp/rbd_import_export_79553/img testimg 2024-09-21T09:53:46.362 INFO:tasks.workunit.client.0.smithi098.stderr: Importing image: 100% complete...done. 2024-09-21T09:53:46.362 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: --order is deprecated, use --object-size 2024-09-21T09:53:46.365 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_79553/img_v2 2024-09-21T09:53:46.405 INFO:tasks.workunit.client.0.smithi098.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2024-09-21T09:53:46.409 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_79553/img_v2 testimg_import 2024-09-21T09:53:46.471 INFO:tasks.workunit.client.0.smithi098.stderr: Importing image: 100% complete... Importing image: 100% complete...done. 2024-09-21T09:53:46.475 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd info testimg_import 2024-09-21T09:53:46.475 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep order 2024-09-21T09:53:46.475 INFO:tasks.workunit.client.0.smithi098.stderr:+ awk '{print $2}' 2024-09-21T09:53:46.475 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep 20 2024-09-21T09:53:46.515 INFO:tasks.workunit.client.0.smithi098.stdout:20 2024-09-21T09:53:46.516 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm /tmp/rbd_import_export_79553/img_v2 2024-09-21T09:53:46.516 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd remove testimg_import 2024-09-21T09:53:46.553 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:46 smithi196 ceph-mon[32670]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T09:53:46.565 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:53:46.566 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:46 smithi098 ceph-mon[29239]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T09:53:46.566 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:46 smithi098 ceph-mon[39960]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T09:53:46.604 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 100% complete...done. 2024-09-21T09:53:46.607 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd remove testimg 2024-09-21T09:53:46.694 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 100% complete...done. 2024-09-21T09:53:46.699 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd import --image-feature layering /tmp/rbd_import_export_79553/img testimg 2024-09-21T09:53:46.748 INFO:tasks.workunit.client.0.smithi098.stderr: Importing image: 100% complete...done. 2024-09-21T09:53:46.751 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd info testimg 2024-09-21T09:53:46.752 INFO:tasks.workunit.client.0.smithi098.stderr:++ grep features 2024-09-21T09:53:46.791 INFO:tasks.workunit.client.0.smithi098.stderr:+ FEATURES_BEFORE=' features: layering 2024-09-21T09:53:46.791 INFO:tasks.workunit.client.0.smithi098.stderr: op_features: ' 2024-09-21T09:53:46.791 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_79553/img_v2 2024-09-21T09:53:46.831 INFO:tasks.workunit.client.0.smithi098.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2024-09-21T09:53:46.834 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_79553/img_v2 testimg_import 2024-09-21T09:53:46.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:53:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:53:46] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:53:46.889 INFO:tasks.workunit.client.0.smithi098.stderr: Importing image: 100% complete... Importing image: 100% complete...done. 2024-09-21T09:53:46.893 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd info testimg_import 2024-09-21T09:53:46.893 INFO:tasks.workunit.client.0.smithi098.stderr:++ grep features 2024-09-21T09:53:46.932 INFO:tasks.workunit.client.0.smithi098.stderr:+ FEATURES_AFTER=' features: layering 2024-09-21T09:53:46.932 INFO:tasks.workunit.client.0.smithi098.stderr: op_features: ' 2024-09-21T09:53:46.932 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' ' features: layering 2024-09-21T09:53:46.932 INFO:tasks.workunit.client.0.smithi098.stderr: op_features: ' '!=' ' features: layering 2024-09-21T09:53:46.932 INFO:tasks.workunit.client.0.smithi098.stderr: op_features: ' ']' 2024-09-21T09:53:46.932 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm /tmp/rbd_import_export_79553/img_v2 2024-09-21T09:53:46.932 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd remove testimg_import 2024-09-21T09:53:46.999 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 100% complete...done. 2024-09-21T09:53:47.002 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd remove testimg 2024-09-21T09:53:47.072 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 100% complete...done. 2024-09-21T09:53:47.075 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd import --stripe-count 1000 --stripe-unit 4096 /tmp/rbd_import_export_79553/img testimg 2024-09-21T09:53:47.138 INFO:tasks.workunit.client.0.smithi098.stderr: Importing image: 100% complete...done. 2024-09-21T09:53:47.141 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_79553/img_v2 2024-09-21T09:53:47.182 INFO:tasks.workunit.client.0.smithi098.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2024-09-21T09:53:47.185 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_79553/img_v2 testimg_import 2024-09-21T09:53:47.249 INFO:tasks.workunit.client.0.smithi098.stderr: Importing image: 100% complete... Importing image: 100% complete...done. 2024-09-21T09:53:47.252 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd info testimg_import 2024-09-21T09:53:47.252 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep 'stripe unit' 2024-09-21T09:53:47.252 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep -Ei '(4 KiB|4096)' 2024-09-21T09:53:47.291 INFO:tasks.workunit.client.0.smithi098.stdout: stripe unit: 4 KiB 2024-09-21T09:53:47.292 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd info testimg_import 2024-09-21T09:53:47.292 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep 'stripe count' 2024-09-21T09:53:47.292 INFO:tasks.workunit.client.0.smithi098.stderr:+ awk '{print $3}' 2024-09-21T09:53:47.292 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep 1000 2024-09-21T09:53:47.330 INFO:tasks.workunit.client.0.smithi098.stdout:1000 2024-09-21T09:53:47.330 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm /tmp/rbd_import_export_79553/img_v2 2024-09-21T09:53:47.330 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd remove testimg_import 2024-09-21T09:53:47.416 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 3% complete... Removing image: 7% complete... Removing image: 10% complete... Removing image: 14% complete... Removing image: 17% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 32% complete... Removing image: 35% complete... Removing image: 39% complete... Removing image: 42% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 57% complete... Removing image: 60% complete... Removing image: 64% complete... Removing image: 67% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 82% complete... Removing image: 85% complete... Removing image: 89% complete... Removing image: 92% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-21T09:53:47.419 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd remove testimg 2024-09-21T09:53:47.505 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 3% complete... Removing image: 7% complete... Removing image: 10% complete... Removing image: 14% complete... Removing image: 17% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 32% complete... Removing image: 35% complete... Removing image: 39% complete... Removing image: 42% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 57% complete... Removing image: 60% complete... Removing image: 64% complete... Removing image: 67% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 82% complete... Removing image: 85% complete... Removing image: 89% complete... Removing image: 92% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-21T09:53:47.507 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd import --image-format=2 /tmp/rbd_import_export_79553/img testimg 2024-09-21T09:53:47.564 INFO:tasks.workunit.client.0.smithi098.stderr: Importing image: 100% complete...done. 2024-09-21T09:53:47.567 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd snap create testimg@snap1 2024-09-21T09:53:47.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:47 smithi098 ceph-mon[29239]: pgmap v103: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 502 KiB/s rd, 166 KiB/s wr, 279 op/s 2024-09-21T09:53:47.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:47 smithi098 ceph-mon[29239]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T09:53:47.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:47 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3804129606' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:53:47.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:47 smithi098 ceph-mon[39960]: pgmap v103: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 502 KiB/s rd, 166 KiB/s wr, 279 op/s 2024-09-21T09:53:47.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:47 smithi098 ceph-mon[39960]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T09:53:47.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:47 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3804129606' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:53:47.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:47 smithi196 ceph-mon[32670]: pgmap v103: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 502 KiB/s rd, 166 KiB/s wr, 279 op/s 2024-09-21T09:53:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:47 smithi196 ceph-mon[32670]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T09:53:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:47 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3804129606' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:53:48.208 INFO:tasks.workunit.client.0.smithi098.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-21T09:53:48.214 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd snap create testimg@snap2 2024-09-21T09:53:48.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:48 smithi098 ceph-mon[29239]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:48.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:48 smithi098 ceph-mon[29239]: from='client.24899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:48.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:48 smithi098 ceph-mon[39960]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:48.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:48 smithi098 ceph-mon[39960]: from='client.24899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:48.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:48 smithi196 ceph-mon[32670]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:48.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:48 smithi196 ceph-mon[32670]: from='client.24899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:48.789 INFO:tasks.workunit.client.0.smithi098.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-21T09:53:48.796 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd snap protect testimg@snap2 2024-09-21T09:53:48.839 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_79553/snap_protect 2024-09-21T09:53:48.882 INFO:tasks.workunit.client.0.smithi098.stderr: Exporting image: 33% complete... Exporting image: 100% complete... Exporting image: 100% complete...done. 2024-09-21T09:53:48.886 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_79553/snap_protect testimg_import 2024-09-21T09:53:49.566 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:53:49.566 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:49 smithi098 ceph-mon[29239]: pgmap v105: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 KiB/s rd, 14 KiB/s wr, 218 op/s 2024-09-21T09:53:49.566 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:49 smithi098 ceph-mon[29239]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T09:53:49.566 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:49 smithi098 ceph-mon[29239]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T09:53:49.567 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:49 smithi098 ceph-mon[39960]: pgmap v105: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 KiB/s rd, 14 KiB/s wr, 218 op/s 2024-09-21T09:53:49.567 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:49 smithi098 ceph-mon[39960]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T09:53:49.567 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:49 smithi098 ceph-mon[39960]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T09:53:49.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:49 smithi196 ceph-mon[32670]: pgmap v105: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 KiB/s rd, 14 KiB/s wr, 218 op/s 2024-09-21T09:53:49.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:49 smithi196 ceph-mon[32670]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T09:53:49.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:49 smithi196 ceph-mon[32670]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T09:53:50.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:50 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/1175883602' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:53:50.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:50 smithi098 ceph-mon[29239]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T09:53:50.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:50 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/1175883602' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:53:50.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:50 smithi098 ceph-mon[39960]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T09:53:50.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:50 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/1175883602' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:53:50.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:50 smithi196 ceph-mon[32670]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T09:53:50.812 INFO:tasks.workunit.client.0.smithi098.stderr: Importing image: 33% complete... Importing image: 66% complete... Importing image: 100% complete... Importing image: 100% complete...done. 2024-09-21T09:53:50.815 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd info testimg_import@snap1 2024-09-21T09:53:50.815 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep 'protected: False' 2024-09-21T09:53:50.855 INFO:tasks.workunit.client.0.smithi098.stdout: protected: False 2024-09-21T09:53:50.856 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd info testimg_import@snap2 2024-09-21T09:53:50.856 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep 'protected: True' 2024-09-21T09:53:50.895 INFO:tasks.workunit.client.0.smithi098.stdout: protected: True 2024-09-21T09:53:50.895 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm /tmp/rbd_import_export_79553/snap_protect 2024-09-21T09:53:50.896 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd snap unprotect testimg@snap2 2024-09-21T09:53:50.943 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd snap unprotect testimg_import@snap2 2024-09-21T09:53:50.988 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd snap purge testimg 2024-09-21T09:53:51.055 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:53:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:53:50] "GET /metrics HTTP/1.1" 200 240038 "" "Prometheus/2.43.0" 2024-09-21T09:53:51.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:51 smithi098 ceph-mon[29239]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:51.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:51 smithi098 ceph-mon[29239]: pgmap v109: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:51.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:51 smithi098 ceph-mon[29239]: from='client.25037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:51.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:51 smithi098 ceph-mon[29239]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T09:53:51.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:51 smithi098 ceph-mon[39960]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:51.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:51 smithi098 ceph-mon[39960]: pgmap v109: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:51.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:51 smithi098 ceph-mon[39960]: from='client.25037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:51.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:51 smithi098 ceph-mon[39960]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T09:53:51.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:51 smithi196 ceph-mon[32670]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:51.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:51 smithi196 ceph-mon[32670]: pgmap v109: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:53:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:51 smithi196 ceph-mon[32670]: from='client.25037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:51 smithi196 ceph-mon[32670]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T09:53:52.512 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:53:52.800 INFO:tasks.workunit.client.0.smithi098.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-21T09:53:52.809 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd snap purge testimg_import 2024-09-21T09:53:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:52 smithi098 ceph-mon[29239]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T09:53:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:52 smithi098 ceph-mon[29239]: pgmap v112: 73 pgs: 73 active+clean; 799 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 438 KiB/s rd, 54 KiB/s wr, 587 op/s 2024-09-21T09:53:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:52 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3100889828' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:53:53.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:52 smithi098 ceph-mon[39960]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T09:53:53.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:52 smithi098 ceph-mon[39960]: pgmap v112: 73 pgs: 73 active+clean; 799 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 438 KiB/s rd, 54 KiB/s wr, 587 op/s 2024-09-21T09:53:53.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:52 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3100889828' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:53:53.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:52 smithi196 ceph-mon[32670]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T09:53:53.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:52 smithi196 ceph-mon[32670]: pgmap v112: 73 pgs: 73 active+clean; 799 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 438 KiB/s rd, 54 KiB/s wr, 587 op/s 2024-09-21T09:53:53.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:52 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3100889828' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:53:54.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:53 smithi098 ceph-mon[29239]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:54.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:53 smithi098 ceph-mon[29239]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T09:53:54.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:53 smithi098 ceph-mon[29239]: from='client.25085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:54.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:53 smithi098 ceph-mon[29239]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T09:53:54.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:53 smithi098 ceph-mon[39960]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:54.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:53 smithi098 ceph-mon[39960]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T09:53:54.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:53 smithi098 ceph-mon[39960]: from='client.25085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:54.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:53 smithi098 ceph-mon[39960]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T09:53:54.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:53 smithi196 ceph-mon[32670]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:54.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:53 smithi196 ceph-mon[32670]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T09:53:54.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:53 smithi196 ceph-mon[32670]: from='client.25085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:53 smithi196 ceph-mon[32670]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T09:53:54.805 INFO:tasks.workunit.client.0.smithi098.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-21T09:53:54.814 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd remove testimg 2024-09-21T09:53:54.906 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 100% complete...done. 2024-09-21T09:53:54.910 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd remove testimg_import 2024-09-21T09:53:55.021 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 100% complete...done. 2024-09-21T09:53:55.026 INFO:tasks.workunit.client.0.smithi098.stderr:+ tiered=0 2024-09-21T09:53:55.026 INFO:tasks.workunit.client.0.smithi098.stderr:+ ceph osd dump 2024-09-21T09:53:55.026 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep '^pool' 2024-09-21T09:53:55.026 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep ''\''rbd'\''' 2024-09-21T09:53:55.026 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep tier 2024-09-21T09:53:55.096 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:54 smithi098 ceph-mon[29239]: pgmap v115: 73 pgs: 73 active+clean; 799 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 401 KiB/s rd, 50 KiB/s wr, 538 op/s 2024-09-21T09:53:55.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:54 smithi098 ceph-mon[39960]: pgmap v115: 73 pgs: 73 active+clean; 799 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 401 KiB/s rd, 50 KiB/s wr, 538 op/s 2024-09-21T09:53:55.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:54 smithi196 ceph-mon[32670]: pgmap v115: 73 pgs: 73 active+clean; 799 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 401 KiB/s rd, 50 KiB/s wr, 538 op/s 2024-09-21T09:53:55.433 INFO:tasks.workunit.client.0.smithi098.stderr:+ dd if=/dev/urandom bs=1M seek=1 count=1 of=/tmp/rbd_import_export_79553/sparse1 2024-09-21T09:53:55.438 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records in 2024-09-21T09:53:55.438 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records out 2024-09-21T09:53:55.438 INFO:tasks.workunit.client.0.smithi098.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00366893 s, 286 MB/s 2024-09-21T09:53:55.438 INFO:tasks.workunit.client.0.smithi098.stderr:+ dd if=/dev/urandom bs=1M count=1 of=/tmp/rbd_import_export_79553/sparse2 2024-09-21T09:53:55.442 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records in 2024-09-21T09:53:55.442 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records out 2024-09-21T09:53:55.443 INFO:tasks.workunit.client.0.smithi098.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00369109 s, 284 MB/s 2024-09-21T09:53:55.443 INFO:tasks.workunit.client.0.smithi098.stderr:+ truncate /tmp/rbd_import_export_79553/sparse2 -s 2M 2024-09-21T09:53:55.454 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd rm sparse1 2024-09-21T09:53:55.496 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-21T09:53:55.496 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: delete error: (2) No such file or directory 2024-09-21T09:53:55.499 INFO:tasks.workunit.client.0.smithi098.stderr:+ true 2024-09-21T09:53:55.500 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd import --new-format --order 20 /tmp/rbd_import_export_79553/sparse1 2024-09-21T09:53:55.518 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:53:55.565 INFO:tasks.workunit.client.0.smithi098.stderr: Importing image: 50% complete... Importing image: 100% complete...done. 2024-09-21T09:53:55.566 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-21T09:53:55.566 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: --order is deprecated, use --object-size 2024-09-21T09:53:55.569 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd ls -l 2024-09-21T09:53:55.570 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep sparse1 2024-09-21T09:53:55.570 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep -Ei '(2 MiB|2048k)' 2024-09-21T09:53:55.609 INFO:tasks.workunit.client.0.smithi098.stdout:sparse1 2 MiB 2 2024-09-21T09:53:55.610 INFO:tasks.workunit.client.0.smithi098.stderr:++ objects sparse1 2024-09-21T09:53:55.610 INFO:tasks.workunit.client.0.smithi098.stderr:++ image=sparse1 2024-09-21T09:53:55.610 INFO:tasks.workunit.client.0.smithi098.stderr:+++ rbd info sparse1 2024-09-21T09:53:55.610 INFO:tasks.workunit.client.0.smithi098.stderr:+++ grep block_name_prefix 2024-09-21T09:53:55.610 INFO:tasks.workunit.client.0.smithi098.stderr:+++ awk '{print $NF;}' 2024-09-21T09:53:55.649 INFO:tasks.workunit.client.0.smithi098.stderr:++ prefix=rbd_data.3b9174659c80 2024-09-21T09:53:55.649 INFO:tasks.workunit.client.0.smithi098.stderr:+++ grep rbd_data.3b9174659c80 2024-09-21T09:53:55.650 INFO:tasks.workunit.client.0.smithi098.stderr:++++ get_image_data_pool sparse1 2024-09-21T09:53:55.650 INFO:tasks.workunit.client.0.smithi098.stderr:++++ image=sparse1 2024-09-21T09:53:55.650 INFO:tasks.workunit.client.0.smithi098.stderr:+++ sed -e 's/rbd_data.3b9174659c80\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-09-21T09:53:55.650 INFO:tasks.workunit.client.0.smithi098.stderr:+++ sort -u 2024-09-21T09:53:55.650 INFO:tasks.workunit.client.0.smithi098.stderr:+++++ rbd info sparse1 2024-09-21T09:53:55.650 INFO:tasks.workunit.client.0.smithi098.stderr:+++++ grep 'data_pool: ' 2024-09-21T09:53:55.651 INFO:tasks.workunit.client.0.smithi098.stderr:+++++ awk -F: '{ print $NF }' 2024-09-21T09:53:55.689 INFO:tasks.workunit.client.0.smithi098.stderr:++++ data_pool= 2024-09-21T09:53:55.690 INFO:tasks.workunit.client.0.smithi098.stderr:++++ '[' -z ']' 2024-09-21T09:53:55.690 INFO:tasks.workunit.client.0.smithi098.stderr:++++ data_pool=rbd 2024-09-21T09:53:55.690 INFO:tasks.workunit.client.0.smithi098.stderr:++++ echo rbd 2024-09-21T09:53:55.690 INFO:tasks.workunit.client.0.smithi098.stderr:+++ rados ls -p rbd 2024-09-21T09:53:55.727 INFO:tasks.workunit.client.0.smithi098.stderr:++ objects=1 2024-09-21T09:53:55.727 INFO:tasks.workunit.client.0.smithi098.stderr:++ echo 1 2024-09-21T09:53:55.727 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 0 -eq 1 -o 1 = 1 ']' 2024-09-21T09:53:55.728 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd export sparse1 /tmp/rbd_import_export_79553/sparse1.out 2024-09-21T09:53:55.767 INFO:tasks.workunit.client.0.smithi098.stderr: Exporting image: 50% complete... Exporting image: 100% complete...done. 2024-09-21T09:53:55.771 INFO:tasks.workunit.client.0.smithi098.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_79553/sparse1 /tmp/rbd_import_export_79553/sparse1.out 2024-09-21T09:53:55.771 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmp -l /tmp/rbd_import_export_79553/sparse1 /tmp/rbd_import_export_79553/sparse1.out 2024-09-21T09:53:55.773 INFO:tasks.workunit.client.0.smithi098.stderr:++ stat /tmp/rbd_import_export_79553/sparse1 --format %b 2024-09-21T09:53:55.774 INFO:tasks.workunit.client.0.smithi098.stderr:+ origsize=2048 2024-09-21T09:53:55.774 INFO:tasks.workunit.client.0.smithi098.stderr:++ stat /tmp/rbd_import_export_79553/sparse1.out --format %b 2024-09-21T09:53:55.775 INFO:tasks.workunit.client.0.smithi098.stderr:+ exportsize=2048 2024-09-21T09:53:55.775 INFO:tasks.workunit.client.0.smithi098.stderr:+ difference=0 2024-09-21T09:53:55.776 INFO:tasks.workunit.client.0.smithi098.stderr:+ difference=0 2024-09-21T09:53:55.776 INFO:tasks.workunit.client.0.smithi098.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2024-09-21T09:53:55.776 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm /tmp/rbd_import_export_79553/sparse1.out 2024-09-21T09:53:55.776 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd rm sparse1 2024-09-21T09:53:55.872 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 50% complete... Removing image: 100% complete...done. 2024-09-21T09:53:55.875 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd rm sparse2 2024-09-21T09:53:55.916 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-21T09:53:55.916 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: delete error: (2) No such file or directory 2024-09-21T09:53:55.919 INFO:tasks.workunit.client.0.smithi098.stderr:+ true 2024-09-21T09:53:55.919 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd import --new-format --order 20 /tmp/rbd_import_export_79553/sparse2 2024-09-21T09:53:55.987 INFO:tasks.workunit.client.0.smithi098.stderr: Importing image: 50% complete... Importing image: 100% complete...done. 2024-09-21T09:53:55.987 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-21T09:53:55.987 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: --order is deprecated, use --object-size 2024-09-21T09:53:55.990 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd ls -l 2024-09-21T09:53:55.991 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep -Ei '(2 MiB|2048k)' 2024-09-21T09:53:55.991 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep sparse2 2024-09-21T09:53:56.030 INFO:tasks.workunit.client.0.smithi098.stdout:sparse2 2 MiB 2 2024-09-21T09:53:56.031 INFO:tasks.workunit.client.0.smithi098.stderr:++ objects sparse2 2024-09-21T09:53:56.031 INFO:tasks.workunit.client.0.smithi098.stderr:++ image=sparse2 2024-09-21T09:53:56.031 INFO:tasks.workunit.client.0.smithi098.stderr:+++ rbd info sparse2 2024-09-21T09:53:56.032 INFO:tasks.workunit.client.0.smithi098.stderr:+++ grep block_name_prefix 2024-09-21T09:53:56.032 INFO:tasks.workunit.client.0.smithi098.stderr:+++ awk '{print $NF;}' 2024-09-21T09:53:56.070 INFO:tasks.workunit.client.0.smithi098.stderr:++ prefix=rbd_data.3bcab1f11752 2024-09-21T09:53:56.071 INFO:tasks.workunit.client.0.smithi098.stderr:+++ grep rbd_data.3bcab1f11752 2024-09-21T09:53:56.071 INFO:tasks.workunit.client.0.smithi098.stderr:++++ get_image_data_pool sparse2 2024-09-21T09:53:56.071 INFO:tasks.workunit.client.0.smithi098.stderr:+++ sed -e 's/rbd_data.3bcab1f11752\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-09-21T09:53:56.072 INFO:tasks.workunit.client.0.smithi098.stderr:++++ image=sparse2 2024-09-21T09:53:56.072 INFO:tasks.workunit.client.0.smithi098.stderr:+++ sort -u 2024-09-21T09:53:56.072 INFO:tasks.workunit.client.0.smithi098.stderr:+++++ rbd info sparse2 2024-09-21T09:53:56.072 INFO:tasks.workunit.client.0.smithi098.stderr:+++++ grep 'data_pool: ' 2024-09-21T09:53:56.072 INFO:tasks.workunit.client.0.smithi098.stderr:+++++ awk -F: '{ print $NF }' 2024-09-21T09:53:56.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:55 smithi098 ceph-mon[29239]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T09:53:56.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:55 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/371720607' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:53:56.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:55 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/285169316' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-21T09:53:56.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:55 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/285169316' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-21T09:53:56.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:55 smithi098 ceph-mon[39960]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T09:53:56.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:55 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/371720607' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:53:56.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:55 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/285169316' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-21T09:53:56.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:55 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/285169316' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-21T09:53:56.112 INFO:tasks.workunit.client.0.smithi098.stderr:++++ data_pool= 2024-09-21T09:53:56.112 INFO:tasks.workunit.client.0.smithi098.stderr:++++ '[' -z ']' 2024-09-21T09:53:56.112 INFO:tasks.workunit.client.0.smithi098.stderr:++++ data_pool=rbd 2024-09-21T09:53:56.112 INFO:tasks.workunit.client.0.smithi098.stderr:++++ echo rbd 2024-09-21T09:53:56.112 INFO:tasks.workunit.client.0.smithi098.stderr:+++ rados ls -p rbd 2024-09-21T09:53:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:55 smithi196 ceph-mon[32670]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T09:53:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:55 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/371720607' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:53:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:55 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/285169316' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-21T09:53:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:55 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/285169316' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-21T09:53:56.147 INFO:tasks.workunit.client.0.smithi098.stderr:++ objects=0 2024-09-21T09:53:56.147 INFO:tasks.workunit.client.0.smithi098.stderr:++ echo 0 2024-09-21T09:53:56.147 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 0 -eq 1 -o 0 = 0 ']' 2024-09-21T09:53:56.147 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd export sparse2 /tmp/rbd_import_export_79553/sparse2.out 2024-09-21T09:53:56.189 INFO:tasks.workunit.client.0.smithi098.stderr: Exporting image: 50% complete... Exporting image: 100% complete...done. 2024-09-21T09:53:56.192 INFO:tasks.workunit.client.0.smithi098.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_79553/sparse2 /tmp/rbd_import_export_79553/sparse2.out 2024-09-21T09:53:56.193 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmp -l /tmp/rbd_import_export_79553/sparse2 /tmp/rbd_import_export_79553/sparse2.out 2024-09-21T09:53:56.195 INFO:tasks.workunit.client.0.smithi098.stderr:++ stat /tmp/rbd_import_export_79553/sparse2 --format %b 2024-09-21T09:53:56.196 INFO:tasks.workunit.client.0.smithi098.stderr:+ origsize=2048 2024-09-21T09:53:56.196 INFO:tasks.workunit.client.0.smithi098.stderr:++ stat /tmp/rbd_import_export_79553/sparse2.out --format %b 2024-09-21T09:53:56.197 INFO:tasks.workunit.client.0.smithi098.stderr:+ exportsize=2048 2024-09-21T09:53:56.197 INFO:tasks.workunit.client.0.smithi098.stderr:+ difference=0 2024-09-21T09:53:56.197 INFO:tasks.workunit.client.0.smithi098.stderr:+ difference=0 2024-09-21T09:53:56.197 INFO:tasks.workunit.client.0.smithi098.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2024-09-21T09:53:56.197 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm /tmp/rbd_import_export_79553/sparse2.out 2024-09-21T09:53:56.198 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd rm sparse2 2024-09-21T09:53:56.296 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 50% complete... Removing image: 100% complete...done. 2024-09-21T09:53:56.300 INFO:tasks.workunit.client.0.smithi098.stderr:+ truncate /tmp/rbd_import_export_79553/sparse1 -s 10M 2024-09-21T09:53:56.301 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd import --new-format --order 20 - sparse1 2024-09-21T09:53:56.399 INFO:tasks.workunit.client.0.smithi098.stderr: Importing image: 100% complete...done. 2024-09-21T09:53:56.399 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-21T09:53:56.399 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: --order is deprecated, use --object-size 2024-09-21T09:53:56.403 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd ls -l 2024-09-21T09:53:56.403 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep sparse1 2024-09-21T09:53:56.403 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep -Ei '(10 MiB|10240k)' 2024-09-21T09:53:56.443 INFO:tasks.workunit.client.0.smithi098.stdout:sparse1 10 MiB 2 2024-09-21T09:53:56.443 INFO:tasks.workunit.client.0.smithi098.stderr:++ objects sparse1 2024-09-21T09:53:56.443 INFO:tasks.workunit.client.0.smithi098.stderr:++ image=sparse1 2024-09-21T09:53:56.444 INFO:tasks.workunit.client.0.smithi098.stderr:+++ rbd info sparse1 2024-09-21T09:53:56.444 INFO:tasks.workunit.client.0.smithi098.stderr:+++ grep block_name_prefix 2024-09-21T09:53:56.444 INFO:tasks.workunit.client.0.smithi098.stderr:+++ awk '{print $NF;}' 2024-09-21T09:53:56.483 INFO:tasks.workunit.client.0.smithi098.stderr:++ prefix=rbd_data.3bf79be75e92 2024-09-21T09:53:56.484 INFO:tasks.workunit.client.0.smithi098.stderr:+++ grep rbd_data.3bf79be75e92 2024-09-21T09:53:56.484 INFO:tasks.workunit.client.0.smithi098.stderr:++++ get_image_data_pool sparse1 2024-09-21T09:53:56.484 INFO:tasks.workunit.client.0.smithi098.stderr:+++ sed -e 's/rbd_data.3bf79be75e92\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-09-21T09:53:56.484 INFO:tasks.workunit.client.0.smithi098.stderr:++++ image=sparse1 2024-09-21T09:53:56.484 INFO:tasks.workunit.client.0.smithi098.stderr:+++ sort -u 2024-09-21T09:53:56.485 INFO:tasks.workunit.client.0.smithi098.stderr:+++++ rbd info sparse1 2024-09-21T09:53:56.485 INFO:tasks.workunit.client.0.smithi098.stderr:+++++ grep 'data_pool: ' 2024-09-21T09:53:56.485 INFO:tasks.workunit.client.0.smithi098.stderr:+++++ awk -F: '{ print $NF }' 2024-09-21T09:53:56.536 INFO:tasks.workunit.client.0.smithi098.stderr:++++ data_pool= 2024-09-21T09:53:56.536 INFO:tasks.workunit.client.0.smithi098.stderr:++++ '[' -z ']' 2024-09-21T09:53:56.536 INFO:tasks.workunit.client.0.smithi098.stderr:++++ data_pool=rbd 2024-09-21T09:53:56.536 INFO:tasks.workunit.client.0.smithi098.stderr:++++ echo rbd 2024-09-21T09:53:56.537 INFO:tasks.workunit.client.0.smithi098.stderr:+++ rados ls -p rbd 2024-09-21T09:53:56.573 INFO:tasks.workunit.client.0.smithi098.stderr:++ objects=1 2024-09-21T09:53:56.573 INFO:tasks.workunit.client.0.smithi098.stderr:++ echo 1 2024-09-21T09:53:56.573 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 0 -eq 1 -o 1 = 1 ']' 2024-09-21T09:53:56.573 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd export sparse1 /tmp/rbd_import_export_79553/sparse1.out 2024-09-21T09:53:56.614 INFO:tasks.workunit.client.0.smithi098.stderr: Exporting image: 10% complete... Exporting image: 20% complete... Exporting image: 30% complete... Exporting image: 40% complete... Exporting image: 50% complete... Exporting image: 60% complete... Exporting image: 70% complete... Exporting image: 80% complete... Exporting image: 90% complete... Exporting image: 100% complete...done. 2024-09-21T09:53:56.618 INFO:tasks.workunit.client.0.smithi098.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_79553/sparse1 /tmp/rbd_import_export_79553/sparse1.out 2024-09-21T09:53:56.618 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmp -l /tmp/rbd_import_export_79553/sparse1 /tmp/rbd_import_export_79553/sparse1.out 2024-09-21T09:53:56.625 INFO:tasks.workunit.client.0.smithi098.stderr:++ stat /tmp/rbd_import_export_79553/sparse1 --format %b 2024-09-21T09:53:56.627 INFO:tasks.workunit.client.0.smithi098.stderr:+ origsize=2048 2024-09-21T09:53:56.627 INFO:tasks.workunit.client.0.smithi098.stderr:++ stat /tmp/rbd_import_export_79553/sparse1.out --format %b 2024-09-21T09:53:56.628 INFO:tasks.workunit.client.0.smithi098.stderr:+ exportsize=2048 2024-09-21T09:53:56.628 INFO:tasks.workunit.client.0.smithi098.stderr:+ difference=0 2024-09-21T09:53:56.628 INFO:tasks.workunit.client.0.smithi098.stderr:+ difference=0 2024-09-21T09:53:56.628 INFO:tasks.workunit.client.0.smithi098.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2024-09-21T09:53:56.628 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm /tmp/rbd_import_export_79553/sparse1.out 2024-09-21T09:53:56.629 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd rm sparse1 2024-09-21T09:53:56.721 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 10% complete... Removing image: 20% complete... Removing image: 30% complete... Removing image: 40% complete... Removing image: 50% complete... Removing image: 60% complete... Removing image: 70% complete... Removing image: 80% complete... Removing image: 90% complete... Removing image: 100% complete...done. 2024-09-21T09:53:56.725 INFO:tasks.workunit.client.0.smithi098.stderr:+ dd if=/dev/urandom bs=2M count=1 of=/tmp/rbd_import_export_79553/sparse2 oflag=append conv=notrunc 2024-09-21T09:53:56.734 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records in 2024-09-21T09:53:56.734 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records out 2024-09-21T09:53:56.734 INFO:tasks.workunit.client.0.smithi098.stderr:2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.00715825 s, 293 MB/s 2024-09-21T09:53:56.734 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd import --new-format --order 20 - sparse2 2024-09-21T09:53:56.813 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:53:56 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:53:56] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:53:56.816 INFO:tasks.workunit.client.0.smithi098.stderr: Importing image: 100% complete...done. 2024-09-21T09:53:56.817 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-21T09:53:56.817 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: --order is deprecated, use --object-size 2024-09-21T09:53:56.819 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd ls -l 2024-09-21T09:53:56.820 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep sparse2 2024-09-21T09:53:56.820 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep -Ei '(4 MiB|4096k)' 2024-09-21T09:53:56.859 INFO:tasks.workunit.client.0.smithi098.stdout:sparse2 4 MiB 2 2024-09-21T09:53:56.860 INFO:tasks.workunit.client.0.smithi098.stderr:++ objects sparse2 2024-09-21T09:53:56.860 INFO:tasks.workunit.client.0.smithi098.stderr:++ image=sparse2 2024-09-21T09:53:56.860 INFO:tasks.workunit.client.0.smithi098.stderr:+++ rbd info sparse2 2024-09-21T09:53:56.860 INFO:tasks.workunit.client.0.smithi098.stderr:+++ grep block_name_prefix 2024-09-21T09:53:56.860 INFO:tasks.workunit.client.0.smithi098.stderr:+++ awk '{print $NF;}' 2024-09-21T09:53:56.901 INFO:tasks.workunit.client.0.smithi098.stderr:++ prefix=rbd_data.3c2467029cfa 2024-09-21T09:53:56.902 INFO:tasks.workunit.client.0.smithi098.stderr:+++ grep rbd_data.3c2467029cfa 2024-09-21T09:53:56.902 INFO:tasks.workunit.client.0.smithi098.stderr:++++ get_image_data_pool sparse2 2024-09-21T09:53:56.902 INFO:tasks.workunit.client.0.smithi098.stderr:+++ sed -e 's/rbd_data.3c2467029cfa\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-09-21T09:53:56.902 INFO:tasks.workunit.client.0.smithi098.stderr:++++ image=sparse2 2024-09-21T09:53:56.902 INFO:tasks.workunit.client.0.smithi098.stderr:+++ sort -u 2024-09-21T09:53:56.902 INFO:tasks.workunit.client.0.smithi098.stderr:+++++ rbd info sparse2 2024-09-21T09:53:56.903 INFO:tasks.workunit.client.0.smithi098.stderr:+++++ grep 'data_pool: ' 2024-09-21T09:53:56.903 INFO:tasks.workunit.client.0.smithi098.stderr:+++++ awk -F: '{ print $NF }' 2024-09-21T09:53:56.941 INFO:tasks.workunit.client.0.smithi098.stderr:++++ data_pool= 2024-09-21T09:53:56.942 INFO:tasks.workunit.client.0.smithi098.stderr:++++ '[' -z ']' 2024-09-21T09:53:56.942 INFO:tasks.workunit.client.0.smithi098.stderr:++++ data_pool=rbd 2024-09-21T09:53:56.942 INFO:tasks.workunit.client.0.smithi098.stderr:++++ echo rbd 2024-09-21T09:53:56.942 INFO:tasks.workunit.client.0.smithi098.stderr:+++ rados ls -p rbd 2024-09-21T09:53:56.978 INFO:tasks.workunit.client.0.smithi098.stderr:++ objects='0 2024-09-21T09:53:56.978 INFO:tasks.workunit.client.0.smithi098.stderr:2 2024-09-21T09:53:56.978 INFO:tasks.workunit.client.0.smithi098.stderr:3' 2024-09-21T09:53:56.978 INFO:tasks.workunit.client.0.smithi098.stderr:++ echo 0 2 3 2024-09-21T09:53:56.978 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 0 -eq 1 -o '0 2 3' = '0 2 3' ']' 2024-09-21T09:53:56.978 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd export sparse2 /tmp/rbd_import_export_79553/sparse2.out 2024-09-21T09:53:57.020 INFO:tasks.workunit.client.0.smithi098.stderr: Exporting image: 25% complete... Exporting image: 50% complete... Exporting image: 75% complete... Exporting image: 100% complete...done. 2024-09-21T09:53:57.024 INFO:tasks.workunit.client.0.smithi098.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_79553/sparse2 /tmp/rbd_import_export_79553/sparse2.out 2024-09-21T09:53:57.024 INFO:tasks.workunit.client.0.smithi098.stderr:+ cmp -l /tmp/rbd_import_export_79553/sparse2 /tmp/rbd_import_export_79553/sparse2.out 2024-09-21T09:53:57.027 INFO:tasks.workunit.client.0.smithi098.stderr:++ stat /tmp/rbd_import_export_79553/sparse2 --format %b 2024-09-21T09:53:57.028 INFO:tasks.workunit.client.0.smithi098.stderr:+ origsize=6144 2024-09-21T09:53:57.029 INFO:tasks.workunit.client.0.smithi098.stderr:++ stat /tmp/rbd_import_export_79553/sparse2.out --format %b 2024-09-21T09:53:57.029 INFO:tasks.workunit.client.0.smithi098.stderr:+ exportsize=6144 2024-09-21T09:53:57.029 INFO:tasks.workunit.client.0.smithi098.stderr:+ difference=0 2024-09-21T09:53:57.029 INFO:tasks.workunit.client.0.smithi098.stderr:+ difference=0 2024-09-21T09:53:57.030 INFO:tasks.workunit.client.0.smithi098.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2024-09-21T09:53:57.030 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm /tmp/rbd_import_export_79553/sparse2.out 2024-09-21T09:53:57.030 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd rm sparse2 2024-09-21T09:53:57.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:56 smithi098 ceph-mon[29239]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:57.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:56 smithi098 ceph-mon[29239]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:57.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:56 smithi098 ceph-mon[29239]: pgmap v117: 73 pgs: 73 active+clean; 1.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 433 KiB/s rd, 263 KiB/s wr, 243 op/s 2024-09-21T09:53:57.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:56 smithi098 ceph-mon[39960]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:57.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:56 smithi098 ceph-mon[39960]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:57.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:56 smithi098 ceph-mon[39960]: pgmap v117: 73 pgs: 73 active+clean; 1.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 433 KiB/s rd, 263 KiB/s wr, 243 op/s 2024-09-21T09:53:57.116 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 25% complete... Removing image: 50% complete... Removing image: 75% complete... Removing image: 100% complete...done. 2024-09-21T09:53:57.120 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo 'partially-sparse file imports to partially-sparse image' 2024-09-21T09:53:57.120 INFO:tasks.workunit.client.0.smithi098.stdout:partially-sparse file imports to partially-sparse image 2024-09-21T09:53:57.120 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd import --new-format --order 20 /tmp/rbd_import_export_79553/sparse1 sparse 2024-09-21T09:53:57.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:56 smithi196 ceph-mon[32670]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:57.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:56 smithi196 ceph-mon[32670]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:57.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:56 smithi196 ceph-mon[32670]: pgmap v117: 73 pgs: 73 active+clean; 1.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 433 KiB/s rd, 263 KiB/s wr, 243 op/s 2024-09-21T09:53:57.186 INFO:tasks.workunit.client.0.smithi098.stderr: Importing image: 10% complete... Importing image: 20% complete... Importing image: 30% complete... Importing image: 40% complete... Importing image: 50% complete... Importing image: 60% complete... Importing image: 70% complete... Importing image: 80% complete... Importing image: 90% complete... Importing image: 100% complete...done. 2024-09-21T09:53:57.187 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-21T09:53:57.187 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: --order is deprecated, use --object-size 2024-09-21T09:53:57.190 INFO:tasks.workunit.client.0.smithi098.stderr:++ objects sparse 2024-09-21T09:53:57.190 INFO:tasks.workunit.client.0.smithi098.stderr:++ image=sparse 2024-09-21T09:53:57.191 INFO:tasks.workunit.client.0.smithi098.stderr:+++ rbd info sparse 2024-09-21T09:53:57.191 INFO:tasks.workunit.client.0.smithi098.stderr:+++ grep block_name_prefix 2024-09-21T09:53:57.191 INFO:tasks.workunit.client.0.smithi098.stderr:+++ awk '{print $NF;}' 2024-09-21T09:53:57.230 INFO:tasks.workunit.client.0.smithi098.stderr:++ prefix=rbd_data.3c51100bdf3d 2024-09-21T09:53:57.231 INFO:tasks.workunit.client.0.smithi098.stderr:+++ grep rbd_data.3c51100bdf3d 2024-09-21T09:53:57.231 INFO:tasks.workunit.client.0.smithi098.stderr:++++ get_image_data_pool sparse 2024-09-21T09:53:57.231 INFO:tasks.workunit.client.0.smithi098.stderr:++++ image=sparse 2024-09-21T09:53:57.231 INFO:tasks.workunit.client.0.smithi098.stderr:+++ sed -e 's/rbd_data.3c51100bdf3d\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-09-21T09:53:57.231 INFO:tasks.workunit.client.0.smithi098.stderr:+++ sort -u 2024-09-21T09:53:57.231 INFO:tasks.workunit.client.0.smithi098.stderr:+++++ rbd info sparse 2024-09-21T09:53:57.232 INFO:tasks.workunit.client.0.smithi098.stderr:+++++ grep 'data_pool: ' 2024-09-21T09:53:57.232 INFO:tasks.workunit.client.0.smithi098.stderr:+++++ awk -F: '{ print $NF }' 2024-09-21T09:53:57.272 INFO:tasks.workunit.client.0.smithi098.stderr:++++ data_pool= 2024-09-21T09:53:57.272 INFO:tasks.workunit.client.0.smithi098.stderr:++++ '[' -z ']' 2024-09-21T09:53:57.272 INFO:tasks.workunit.client.0.smithi098.stderr:++++ data_pool=rbd 2024-09-21T09:53:57.272 INFO:tasks.workunit.client.0.smithi098.stderr:++++ echo rbd 2024-09-21T09:53:57.272 INFO:tasks.workunit.client.0.smithi098.stderr:+++ rados ls -p rbd 2024-09-21T09:53:57.313 INFO:tasks.workunit.client.0.smithi098.stderr:++ objects=1 2024-09-21T09:53:57.313 INFO:tasks.workunit.client.0.smithi098.stderr:++ echo 1 2024-09-21T09:53:57.313 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 0 -eq 1 -o 1 = 1 ']' 2024-09-21T09:53:57.313 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd rm sparse 2024-09-21T09:53:57.395 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 10% complete... Removing image: 20% complete... Removing image: 30% complete... Removing image: 40% complete... Removing image: 50% complete... Removing image: 60% complete... Removing image: 70% complete... Removing image: 80% complete... Removing image: 90% complete... Removing image: 100% complete...done. 2024-09-21T09:53:57.399 INFO:tasks.workunit.client.0.smithi098.stdout:zeros import through stdin to sparse image 2024-09-21T09:53:57.399 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo 'zeros import through stdin to sparse image' 2024-09-21T09:53:57.399 INFO:tasks.workunit.client.0.smithi098.stderr:+ dd if=/dev/zero bs=1M count=4 2024-09-21T09:53:57.400 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd import --new-format - sparse 2024-09-21T09:53:57.450 INFO:tasks.workunit.client.0.smithi098.stderr:4+0 records in 2024-09-21T09:53:57.450 INFO:tasks.workunit.client.0.smithi098.stderr:4+0 records out 2024-09-21T09:53:57.450 INFO:tasks.workunit.client.0.smithi098.stderr:4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0498005 s, 84.2 MB/s 2024-09-21T09:53:57.459 INFO:tasks.workunit.client.0.smithi098.stderr: Importing image: 100% complete...done. 2024-09-21T09:53:57.460 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-21T09:53:57.463 INFO:tasks.workunit.client.0.smithi098.stderr:++ objects sparse 2024-09-21T09:53:57.463 INFO:tasks.workunit.client.0.smithi098.stderr:++ image=sparse 2024-09-21T09:53:57.463 INFO:tasks.workunit.client.0.smithi098.stderr:+++ rbd info sparse 2024-09-21T09:53:57.463 INFO:tasks.workunit.client.0.smithi098.stderr:+++ grep block_name_prefix 2024-09-21T09:53:57.464 INFO:tasks.workunit.client.0.smithi098.stderr:+++ awk '{print $NF;}' 2024-09-21T09:53:57.503 INFO:tasks.workunit.client.0.smithi098.stderr:++ prefix=rbd_data.3c72993f8c75 2024-09-21T09:53:57.503 INFO:tasks.workunit.client.0.smithi098.stderr:+++ grep rbd_data.3c72993f8c75 2024-09-21T09:53:57.503 INFO:tasks.workunit.client.0.smithi098.stderr:+++ sed -e 's/rbd_data.3c72993f8c75\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-09-21T09:53:57.503 INFO:tasks.workunit.client.0.smithi098.stderr:++++ get_image_data_pool sparse 2024-09-21T09:53:57.503 INFO:tasks.workunit.client.0.smithi098.stderr:++++ image=sparse 2024-09-21T09:53:57.503 INFO:tasks.workunit.client.0.smithi098.stderr:+++ sort -u 2024-09-21T09:53:57.504 INFO:tasks.workunit.client.0.smithi098.stderr:+++++ rbd info sparse 2024-09-21T09:53:57.504 INFO:tasks.workunit.client.0.smithi098.stderr:+++++ grep 'data_pool: ' 2024-09-21T09:53:57.504 INFO:tasks.workunit.client.0.smithi098.stderr:+++++ awk -F: '{ print $NF }' 2024-09-21T09:53:57.542 INFO:tasks.workunit.client.0.smithi098.stderr:++++ data_pool= 2024-09-21T09:53:57.543 INFO:tasks.workunit.client.0.smithi098.stderr:++++ '[' -z ']' 2024-09-21T09:53:57.543 INFO:tasks.workunit.client.0.smithi098.stderr:++++ data_pool=rbd 2024-09-21T09:53:57.543 INFO:tasks.workunit.client.0.smithi098.stderr:++++ echo rbd 2024-09-21T09:53:57.543 INFO:tasks.workunit.client.0.smithi098.stderr:+++ rados ls -p rbd 2024-09-21T09:53:57.576 INFO:tasks.workunit.client.0.smithi098.stderr:++ objects= 2024-09-21T09:53:57.577 INFO:tasks.workunit.client.0.smithi098.stderr:++ echo 2024-09-21T09:53:57.577 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 0 -eq 1 -o '' = '' ']' 2024-09-21T09:53:57.577 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd rm sparse 2024-09-21T09:53:57.657 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 100% complete...done. 2024-09-21T09:53:57.660 INFO:tasks.workunit.client.0.smithi098.stdout:zeros export to sparse file 2024-09-21T09:53:57.660 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo 'zeros export to sparse file' 2024-09-21T09:53:57.660 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd create --new-format sparse --size 4 2024-09-21T09:53:57.698 INFO:tasks.workunit.client.0.smithi098.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-21T09:53:57.701 INFO:tasks.workunit.client.0.smithi098.stderr:++ rbd info sparse 2024-09-21T09:53:57.701 INFO:tasks.workunit.client.0.smithi098.stderr:++ grep block_name_prefix 2024-09-21T09:53:57.701 INFO:tasks.workunit.client.0.smithi098.stderr:++ awk '{print $NF;}' 2024-09-21T09:53:57.741 INFO:tasks.workunit.client.0.smithi098.stderr:+ prefix=rbd_data.3c937480beb 2024-09-21T09:53:57.741 INFO:tasks.workunit.client.0.smithi098.stderr:+ dd if=/dev/zero bs=4M count=1 2024-09-21T09:53:57.741 INFO:tasks.workunit.client.0.smithi098.stderr:++ get_image_data_pool sparse 2024-09-21T09:53:57.741 INFO:tasks.workunit.client.0.smithi098.stderr:++ image=sparse 2024-09-21T09:53:57.742 INFO:tasks.workunit.client.0.smithi098.stderr:+++ rbd info sparse 2024-09-21T09:53:57.742 INFO:tasks.workunit.client.0.smithi098.stderr:+++ grep 'data_pool: ' 2024-09-21T09:53:57.742 INFO:tasks.workunit.client.0.smithi098.stderr:+++ awk -F: '{ print $NF }' 2024-09-21T09:53:57.779 INFO:tasks.workunit.client.0.smithi098.stderr:++ data_pool= 2024-09-21T09:53:57.780 INFO:tasks.workunit.client.0.smithi098.stderr:++ '[' -z ']' 2024-09-21T09:53:57.780 INFO:tasks.workunit.client.0.smithi098.stderr:++ data_pool=rbd 2024-09-21T09:53:57.780 INFO:tasks.workunit.client.0.smithi098.stderr:++ echo rbd 2024-09-21T09:53:57.780 INFO:tasks.workunit.client.0.smithi098.stderr:+ rados -p rbd put rbd_data.3c937480beb.000000000000 - 2024-09-21T09:53:57.810 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records in 2024-09-21T09:53:57.810 INFO:tasks.workunit.client.0.smithi098.stderr:1+0 records out 2024-09-21T09:53:57.810 INFO:tasks.workunit.client.0.smithi098.stderr:4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0689232 s, 60.9 MB/s 2024-09-21T09:53:57.829 INFO:tasks.workunit.client.0.smithi098.stderr:++ objects sparse 2024-09-21T09:53:57.830 INFO:tasks.workunit.client.0.smithi098.stderr:++ image=sparse 2024-09-21T09:53:57.830 INFO:tasks.workunit.client.0.smithi098.stderr:+++ rbd info sparse 2024-09-21T09:53:57.830 INFO:tasks.workunit.client.0.smithi098.stderr:+++ grep block_name_prefix 2024-09-21T09:53:57.830 INFO:tasks.workunit.client.0.smithi098.stderr:+++ awk '{print $NF;}' 2024-09-21T09:53:57.867 INFO:tasks.workunit.client.0.smithi098.stderr:++ prefix=rbd_data.3c937480beb 2024-09-21T09:53:57.868 INFO:tasks.workunit.client.0.smithi098.stderr:+++ grep rbd_data.3c937480beb 2024-09-21T09:53:57.868 INFO:tasks.workunit.client.0.smithi098.stderr:++++ get_image_data_pool sparse 2024-09-21T09:53:57.868 INFO:tasks.workunit.client.0.smithi098.stderr:+++ sed -e 's/rbd_data.3c937480beb\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-09-21T09:53:57.868 INFO:tasks.workunit.client.0.smithi098.stderr:++++ image=sparse 2024-09-21T09:53:57.868 INFO:tasks.workunit.client.0.smithi098.stderr:+++ sort -u 2024-09-21T09:53:57.869 INFO:tasks.workunit.client.0.smithi098.stderr:+++++ rbd info sparse 2024-09-21T09:53:57.869 INFO:tasks.workunit.client.0.smithi098.stderr:+++++ grep 'data_pool: ' 2024-09-21T09:53:57.869 INFO:tasks.workunit.client.0.smithi098.stderr:+++++ awk -F: '{ print $NF }' 2024-09-21T09:53:57.920 INFO:tasks.workunit.client.0.smithi098.stderr:++++ data_pool= 2024-09-21T09:53:57.920 INFO:tasks.workunit.client.0.smithi098.stderr:++++ '[' -z ']' 2024-09-21T09:53:57.920 INFO:tasks.workunit.client.0.smithi098.stderr:++++ data_pool=rbd 2024-09-21T09:53:57.920 INFO:tasks.workunit.client.0.smithi098.stderr:++++ echo rbd 2024-09-21T09:53:57.920 INFO:tasks.workunit.client.0.smithi098.stderr:+++ rados ls -p rbd 2024-09-21T09:53:57.959 INFO:tasks.workunit.client.0.smithi098.stderr:++ objects=0 2024-09-21T09:53:57.959 INFO:tasks.workunit.client.0.smithi098.stderr:++ echo 0 2024-09-21T09:53:57.959 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 0 -eq 1 -o 0 = 0 ']' 2024-09-21T09:53:57.959 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm /tmp/rbd_import_export_79553/sparse 2024-09-21T09:53:57.960 INFO:tasks.workunit.client.0.smithi098.stderr:rm: cannot remove '/tmp/rbd_import_export_79553/sparse': No such file or directory 2024-09-21T09:53:57.960 INFO:tasks.workunit.client.0.smithi098.stderr:+ true 2024-09-21T09:53:57.960 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd export sparse /tmp/rbd_import_export_79553/sparse 2024-09-21T09:53:57.998 INFO:tasks.workunit.client.0.smithi098.stderr: Exporting image: 100% complete...done. 2024-09-21T09:53:58.002 INFO:tasks.workunit.client.0.smithi098.stderr:++ stat /tmp/rbd_import_export_79553/sparse --format=%b 2024-09-21T09:53:58.003 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 0 = 0 ']' 2024-09-21T09:53:58.003 INFO:tasks.workunit.client.0.smithi098.stderr:+ rbd rm sparse 2024-09-21T09:53:58.080 INFO:tasks.workunit.client.0.smithi098.stderr: Removing image: 100% complete...done. 2024-09-21T09:53:58.083 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm /tmp/rbd_import_export_79553/sparse /tmp/rbd_import_export_79553/sparse1 /tmp/rbd_import_export_79553/sparse2 /tmp/rbd_import_export_79553/sparse3 2024-09-21T09:53:58.085 INFO:tasks.workunit.client.0.smithi098.stderr:rm: cannot remove '/tmp/rbd_import_export_79553/sparse3': No such file or directory 2024-09-21T09:53:58.085 INFO:tasks.workunit.client.0.smithi098.stderr:+ true 2024-09-21T09:53:58.085 INFO:tasks.workunit.client.0.smithi098.stdout:OK 2024-09-21T09:53:58.087 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo OK 2024-09-21T09:53:58.087 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm -rf /tmp/rbd_import_export_79553 2024-09-21T09:53:58.087 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-21T09:53:58.087 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-21T09:53:58.115 INFO:tasks.workunit:Stopping ['rbd/import_export.sh'] on client.0... 2024-09-21T09:53:58.115 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-21T09:53:58.415 DEBUG:teuthology.parallel:result is None 2024-09-21T09:53:58.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T09:53:58.445 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T09:53:58.446 DEBUG:teuthology.orchestra.run.smithi098:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-21T09:53:58.503 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T09:53:58.503 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-21T09:53:58.503 INFO:teuthology.task.print:**** done rbd/import_export.sh 5-workload 2024-09-21T09:53:58.503 DEBUG:teuthology.parallel:result is None 2024-09-21T09:53:58.552 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:53:59.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:58 smithi098 ceph-mon[29239]: pgmap v118: 73 pgs: 73 active+clean; 1.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 293 KiB/s rd, 178 KiB/s wr, 164 op/s 2024-09-21T09:53:59.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:58 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2715953692' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:53:59.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:58 smithi098 ceph-mon[29239]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T09:53:59.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:58 smithi098 ceph-mon[39960]: pgmap v118: 73 pgs: 73 active+clean; 1.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 293 KiB/s rd, 178 KiB/s wr, 164 op/s 2024-09-21T09:53:59.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:58 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2715953692' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:53:59.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:58 smithi098 ceph-mon[39960]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T09:53:59.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:58 smithi196 ceph-mon[32670]: pgmap v118: 73 pgs: 73 active+clean; 1.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 293 KiB/s rd, 178 KiB/s wr, 164 op/s 2024-09-21T09:53:59.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:58 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2715953692' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:53:59.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:58 smithi196 ceph-mon[32670]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T09:54:00.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:59 smithi098 ceph-mon[29239]: from='client.25451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:00.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:59 smithi098 ceph-mon[29239]: from='client.25457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:00.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:53:59 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:00.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:59 smithi098 ceph-mon[39960]: from='client.25451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:00.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:59 smithi098 ceph-mon[39960]: from='client.25457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:00.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:53:59 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:00.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:59 smithi196 ceph-mon[32670]: from='client.25451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:00.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:59 smithi196 ceph-mon[32670]: from='client.25457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:53:59 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:01.096 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:00 smithi098 ceph-mon[29239]: pgmap v120: 73 pgs: 73 active+clean; 1.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 290 KiB/s rd, 176 KiB/s wr, 162 op/s 2024-09-21T09:54:01.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:54:00] "GET /metrics HTTP/1.1" 200 240338 "" "Prometheus/2.43.0" 2024-09-21T09:54:01.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:00 smithi098 ceph-mon[39960]: pgmap v120: 73 pgs: 73 active+clean; 1.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 290 KiB/s rd, 176 KiB/s wr, 162 op/s 2024-09-21T09:54:01.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:00 smithi196 ceph-mon[32670]: pgmap v120: 73 pgs: 73 active+clean; 1.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 290 KiB/s rd, 176 KiB/s wr, 162 op/s 2024-09-21T09:54:01.666 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:54:01.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:01 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/565852142' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:01.925 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:01 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/565852142' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:01 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/565852142' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:02 smithi098 ceph-mon[29239]: from='client.15591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:02 smithi098 ceph-mon[29239]: pgmap v121: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 376 op/s 2024-09-21T09:54:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:02 smithi098 ceph-mon[29239]: from='client.25475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:03.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:02 smithi098 ceph-mon[39960]: from='client.15591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:03.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:02 smithi098 ceph-mon[39960]: pgmap v121: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 376 op/s 2024-09-21T09:54:03.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:02 smithi098 ceph-mon[39960]: from='client.25475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:03.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:02 smithi196 ceph-mon[32670]: from='client.15591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:03.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:02 smithi196 ceph-mon[32670]: pgmap v121: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 376 op/s 2024-09-21T09:54:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:02 smithi196 ceph-mon[32670]: from='client.25475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:04.601 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:54:05.191 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:04 smithi196 ceph-mon[32670]: pgmap v122: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 926 KiB/s rd, 1.2 MiB/s wr, 332 op/s 2024-09-21T09:54:05.191 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:04 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3346965982' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:05.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:04 smithi098 ceph-mon[29239]: pgmap v122: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 926 KiB/s rd, 1.2 MiB/s wr, 332 op/s 2024-09-21T09:54:05.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:04 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3346965982' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:05.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:04 smithi098 ceph-mon[39960]: pgmap v122: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 926 KiB/s rd, 1.2 MiB/s wr, 332 op/s 2024-09-21T09:54:05.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:04 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3346965982' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:06.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:05 smithi098 ceph-mon[29239]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:06.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:05 smithi098 ceph-mon[29239]: from='client.25493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:06.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:05 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:06.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:05 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:06.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:05 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:06.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:05 smithi098 ceph-mon[39960]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:06.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:05 smithi098 ceph-mon[39960]: from='client.25493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:06.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:05 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:06.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:05 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:06.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:05 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:05 smithi196 ceph-mon[32670]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:05 smithi196 ceph-mon[32670]: from='client.25493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:05 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:05 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:05 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:06.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:06 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:54:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:54:07.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:06 smithi098 ceph-mon[29239]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (unknown) 2024-09-21T09:54:07.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:06 smithi098 ceph-mon[29239]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T09:54:07.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:06 smithi098 ceph-mon[29239]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-21T09:54:07.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:06 smithi098 ceph-mon[29239]: pgmap v123: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 663 KiB/s rd, 1.0 MiB/s wr, 202 op/s 2024-09-21T09:54:07.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:06 smithi098 ceph-mon[29239]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 on smithi196 2024-09-21T09:54:07.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:06 smithi098 ceph-mon[39960]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (unknown) 2024-09-21T09:54:07.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:06 smithi098 ceph-mon[39960]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T09:54:07.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:06 smithi098 ceph-mon[39960]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-21T09:54:07.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:06 smithi098 ceph-mon[39960]: pgmap v123: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 663 KiB/s rd, 1.0 MiB/s wr, 202 op/s 2024-09-21T09:54:07.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:06 smithi098 ceph-mon[39960]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 on smithi196 2024-09-21T09:54:07.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:06 smithi196 ceph-mon[32670]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (unknown) 2024-09-21T09:54:07.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:06 smithi196 ceph-mon[32670]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T09:54:07.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:06 smithi196 ceph-mon[32670]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-21T09:54:07.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:06 smithi196 ceph-mon[32670]: pgmap v123: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 663 KiB/s rd, 1.0 MiB/s wr, 202 op/s 2024-09-21T09:54:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:06 smithi196 ceph-mon[32670]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 on smithi196 2024-09-21T09:54:07.449 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:54:08.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:07 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/492741856' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:08.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:07 smithi098 ceph-mon[29239]: from='client.15627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:08.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:07 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/492741856' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:08.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:07 smithi098 ceph-mon[39960]: from='client.15627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:07 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/492741856' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:07 smithi196 ceph-mon[32670]: from='client.15627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:09.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:08 smithi098 ceph-mon[29239]: from='client.15633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:09.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:08 smithi098 ceph-mon[29239]: pgmap v124: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 663 KiB/s rd, 1.0 MiB/s wr, 202 op/s 2024-09-21T09:54:09.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:08 smithi098 ceph-mon[39960]: from='client.15633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:09.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:08 smithi098 ceph-mon[39960]: pgmap v124: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 663 KiB/s rd, 1.0 MiB/s wr, 202 op/s 2024-09-21T09:54:09.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:08 smithi196 ceph-mon[32670]: from='client.15633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:09.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:08 smithi196 ceph-mon[32670]: pgmap v124: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 663 KiB/s rd, 1.0 MiB/s wr, 202 op/s 2024-09-21T09:54:10.282 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:54:10.939 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:54:10] "GET /metrics HTTP/1.1" 200 240338 "" "Prometheus/2.43.0" 2024-09-21T09:54:11.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:10 smithi098 ceph-mon[29239]: pgmap v125: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 598 KiB/s rd, 932 KiB/s wr, 182 op/s 2024-09-21T09:54:11.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:10 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3035668734' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:11.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:10 smithi098 ceph-mon[29239]: from='client.25523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:11.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:10 smithi098 ceph-mon[39960]: pgmap v125: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 598 KiB/s rd, 932 KiB/s wr, 182 op/s 2024-09-21T09:54:11.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:10 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3035668734' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:11.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:10 smithi098 ceph-mon[39960]: from='client.25523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:11.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:10 smithi196 ceph-mon[32670]: pgmap v125: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 598 KiB/s rd, 932 KiB/s wr, 182 op/s 2024-09-21T09:54:11.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:10 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3035668734' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:11.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:10 smithi196 ceph-mon[32670]: from='client.25523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:12.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:11 smithi098 ceph-mon[29239]: from='client.15651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:12.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:11 smithi098 ceph-mon[39960]: from='client.15651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:12.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:11 smithi196 ceph-mon[32670]: from='client.15651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:13.119 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:54:13.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:12 smithi098 ceph-mon[29239]: pgmap v126: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 552 KiB/s rd, 861 KiB/s wr, 168 op/s 2024-09-21T09:54:13.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:12 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/782717755' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:13.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:12 smithi098 ceph-mon[39960]: pgmap v126: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 552 KiB/s rd, 861 KiB/s wr, 168 op/s 2024-09-21T09:54:13.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:12 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/782717755' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:13.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:12 smithi196 ceph-mon[32670]: pgmap v126: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 552 KiB/s rd, 861 KiB/s wr, 168 op/s 2024-09-21T09:54:13.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:12 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/782717755' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:14.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:13 smithi098 ceph-mon[29239]: from='client.25541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:14.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:13 smithi098 ceph-mon[29239]: from='client.25547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:14.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:13 smithi098 ceph-mon[39960]: from='client.25541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:14.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:13 smithi098 ceph-mon[39960]: from='client.25547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:14.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:13 smithi196 ceph-mon[32670]: from='client.25541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:13 smithi196 ceph-mon[32670]: from='client.25547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:15.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:14 smithi098 ceph-mon[29239]: pgmap v127: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:15.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:14 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:15.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:14 smithi098 ceph-mon[39960]: pgmap v127: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:15.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:14 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:15.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:14 smithi196 ceph-mon[32670]: pgmap v127: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:15.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:14 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:15.969 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:54:16.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:15 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3469648894' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:16.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:15 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3469648894' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:16.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:15 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3469648894' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:16.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:16 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:54:16] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:54:17.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:16 smithi098 ceph-mon[29239]: pgmap v128: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:17.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:16 smithi098 ceph-mon[29239]: from='client.15681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:17.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:16 smithi098 ceph-mon[29239]: from='client.15687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:17.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:16 smithi098 ceph-mon[39960]: pgmap v128: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:17.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:16 smithi098 ceph-mon[39960]: from='client.15681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:17.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:16 smithi098 ceph-mon[39960]: from='client.15687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:17.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:16 smithi196 ceph-mon[32670]: pgmap v128: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:17.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:16 smithi196 ceph-mon[32670]: from='client.15681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:17.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:16 smithi196 ceph-mon[32670]: from='client.15687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:18.802 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:54:19.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:18 smithi098 ceph-mon[29239]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:19.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:18 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/1137407258' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:19.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:18 smithi098 ceph-mon[39960]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:19.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:18 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/1137407258' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:19.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:18 smithi196 ceph-mon[32670]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:19.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:18 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/1137407258' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:20.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:19 smithi098 ceph-mon[29239]: from='client.15699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:20.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:19 smithi098 ceph-mon[29239]: from='client.15705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:20.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:19 smithi098 ceph-mon[39960]: from='client.15699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:20.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:19 smithi098 ceph-mon[39960]: from='client.15705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:20.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:19 smithi196 ceph-mon[32670]: from='client.15699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:20.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:19 smithi196 ceph-mon[32670]: from='client.15705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:20.972 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:54:20] "GET /metrics HTTP/1.1" 200 240329 "" "Prometheus/2.43.0" 2024-09-21T09:54:21.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:20 smithi098 ceph-mon[29239]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:21.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:20 smithi098 ceph-mon[39960]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:21.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:20 smithi196 ceph-mon[32670]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:21.634 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:54:22.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:21 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/80568276' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:22.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:21 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/80568276' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:22.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:21 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/80568276' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:23.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:22 smithi098 ceph-mon[29239]: from='client.15717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:23.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:22 smithi098 ceph-mon[29239]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:23.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:22 smithi098 ceph-mon[29239]: from='client.25601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:23.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:22 smithi098 ceph-mon[39960]: from='client.15717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:23.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:22 smithi098 ceph-mon[39960]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:23.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:22 smithi098 ceph-mon[39960]: from='client.25601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:23.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:22 smithi196 ceph-mon[32670]: from='client.15717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:23.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:22 smithi196 ceph-mon[32670]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:23.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:22 smithi196 ceph-mon[32670]: from='client.25601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:24.469 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:54:25.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:24 smithi098 ceph-mon[29239]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:25.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:24 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2566915190' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:25.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:24 smithi098 ceph-mon[29239]: from='client.15735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:25.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:24 smithi098 ceph-mon[39960]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:25.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:24 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2566915190' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:25.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:24 smithi098 ceph-mon[39960]: from='client.15735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:25.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:24 smithi196 ceph-mon[32670]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:25.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:24 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2566915190' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:25.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:24 smithi196 ceph-mon[32670]: from='client.15735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:26.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:25 smithi098 ceph-mon[29239]: from='client.15741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:26.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:25 smithi098 ceph-mon[39960]: from='client.15741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:26.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:25 smithi196 ceph-mon[32670]: from='client.15741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:26.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:26 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:54:26] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:54:27.289 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:26 smithi098 ceph-mon[29239]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:27.290 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:54:27.290 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:26 smithi098 ceph-mon[39960]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:26 smithi196 ceph-mon[32670]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:28.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:28 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2636222084' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:28.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:28 smithi098 ceph-mon[29239]: from='client.15753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:28.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:28 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2636222084' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:28.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:28 smithi098 ceph-mon[39960]: from='client.15753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:28.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:28 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2636222084' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:28.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:28 smithi196 ceph-mon[32670]: from='client.15753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:29.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:29 smithi098 ceph-mon[29239]: from='client.15759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:29.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:29 smithi098 ceph-mon[29239]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:29.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:29 smithi098 ceph-mon[39960]: from='client.15759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:29.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:29 smithi098 ceph-mon[39960]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:29.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:29 smithi196 ceph-mon[32670]: from='client.15759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:29.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:29 smithi196 ceph-mon[32670]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:30.122 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:54:30.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:30 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3023529423' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:30.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:30 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:30.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:30 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3023529423' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:30.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:30 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:30.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:30 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3023529423' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:30.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:30 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:31.009 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:54:30] "GET /metrics HTTP/1.1" 200 240275 "" "Prometheus/2.43.0" 2024-09-21T09:54:31.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:31 smithi098 ceph-mon[29239]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:31.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:31 smithi098 ceph-mon[29239]: from='client.15771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:31.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:31 smithi098 ceph-mon[29239]: from='client.25655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:31.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:31 smithi098 ceph-mon[39960]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:31.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:31 smithi098 ceph-mon[39960]: from='client.15771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:31.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:31 smithi098 ceph-mon[39960]: from='client.25655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:31.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:31 smithi196 ceph-mon[32670]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:31.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:31 smithi196 ceph-mon[32670]: from='client.15771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:31.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:31 smithi196 ceph-mon[32670]: from='client.25655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:32.948 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:54:33.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:33 smithi098 ceph-mon[29239]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:33.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:33 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3870822132' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:33.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:33 smithi098 ceph-mon[39960]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:33.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:33 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3870822132' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:33.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:33 smithi196 ceph-mon[32670]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:33.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:33 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3870822132' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:34.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:34 smithi098 ceph-mon[29239]: from='client.15789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:34.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:34 smithi098 ceph-mon[29239]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:34.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:34 smithi098 ceph-mon[39960]: from='client.15789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:34.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:34 smithi098 ceph-mon[39960]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:34 smithi196 ceph-mon[32670]: from='client.15789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:34.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:34 smithi196 ceph-mon[32670]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:35.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:35 smithi196 ceph-mon[32670]: pgmap v137: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:35.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:35 smithi098 ceph-mon[29239]: pgmap v137: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:35.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:35 smithi098 ceph-mon[39960]: pgmap v137: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:35.784 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:54:36.068 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:36 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2951258158' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:36.068 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:36 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2951258158' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:36.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:36 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2951258158' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:36.805 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:36 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[34502]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:54:36] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:54:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:37 smithi098 ceph-mon[29239]: from='client.25685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:37 smithi098 ceph-mon[29239]: pgmap v138: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:37 smithi098 ceph-mon[29239]: from='client.15813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:37 smithi098 ceph-mon[29239]: Upgrade: Updating mgr.x 2024-09-21T09:54:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:37 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:37 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:54:37.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:37 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:54:37.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:37 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:37.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:37 smithi098 ceph-mon[29239]: Deploying daemon mgr.x on smithi196 2024-09-21T09:54:37.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:37 smithi098 ceph-mon[39960]: from='client.25685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:37.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:37 smithi098 ceph-mon[39960]: pgmap v138: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:37.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:37 smithi098 ceph-mon[39960]: from='client.15813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:37.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:37 smithi098 ceph-mon[39960]: Upgrade: Updating mgr.x 2024-09-21T09:54:37.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:37 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:37.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:37 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:54:37.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:37 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:54:37.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:37 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:37.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:37 smithi098 ceph-mon[39960]: Deploying daemon mgr.x on smithi196 2024-09-21T09:54:37.629 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:37 smithi196 ceph-mon[32670]: from='client.25685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:37.629 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:37 smithi196 ceph-mon[32670]: pgmap v138: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:37.629 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:37 smithi196 ceph-mon[32670]: from='client.15813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:37.629 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:37 smithi196 ceph-mon[32670]: Upgrade: Updating mgr.x 2024-09-21T09:54:37.629 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:37 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:37.629 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:37 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:54:37.630 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:37 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:54:37.630 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:37 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:37.630 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:37 smithi196 ceph-mon[32670]: Deploying daemon mgr.x on smithi196 2024-09-21T09:54:37.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:37 smithi196 systemd[1]: Stopping Ceph mgr.x for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:54:38.388 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:38 smithi196 podman[65099]: 2024-09-21 09:54:38.085257135 +0000 UTC m=+0.113437457 container died 3ed5ab6c3c4eeed727de2b69e44514ecf0a4dae86c6e156a90f545988431b047 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x, RELEASE=quincy-fbd1a6a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-21T09:54:38.620 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:54:38.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:38 smithi196 podman[65099]: 2024-09-21 09:54:38.5148626 +0000 UTC m=+0.543042921 container cleanup 3ed5ab6c3c4eeed727de2b69e44514ecf0a4dae86c6e156a90f545988431b047 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=quincy-fbd1a6a) 2024-09-21T09:54:38.889 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:38 smithi196 bash[65099]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x 2024-09-21T09:54:38.889 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:38 smithi196 podman[65112]: 2024-09-21 09:54:38.708243924 +0000 UTC m=+0.617593013 container remove 3ed5ab6c3c4eeed727de2b69e44514ecf0a4dae86c6e156a90f545988431b047 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-21T09:54:38.889 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:38 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-21T09:54:39.341 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:38 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@mgr.x.service: Failed with result 'exit-code'. 2024-09-21T09:54:39.341 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:38 smithi196 systemd[1]: Stopped Ceph mgr.x for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:54:39.341 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:38 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@mgr.x.service: Consumed 10.098s CPU time. 2024-09-21T09:54:39.341 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:39 smithi196 systemd[1]: Starting Ceph mgr.x for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:54:39.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:39 smithi098 ceph-mon[29239]: pgmap v139: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:39.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:39 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2500787784' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:39.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:39 smithi098 ceph-mon[39960]: pgmap v139: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:39.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:39 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2500787784' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:39.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:39 smithi196 ceph-mon[32670]: pgmap v139: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:39.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:39 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2500787784' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:39.605 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:39 smithi196 podman[65241]: 2024-09-21 09:54:39.242271769 +0000 UTC m=+0.015754263 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:54:39.605 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:39 smithi196 podman[65241]: 2024-09-21 09:54:39.36379467 +0000 UTC m=+0.137277152 container create b52583e0c622047094947c3de82a6132a2912397cc722cad5d90c0bbe956c0d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-21T09:54:39.605 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:39 smithi196 podman[65241]: 2024-09-21 09:54:39.603847572 +0000 UTC m=+0.377330061 container init b52583e0c622047094947c3de82a6132a2912397cc722cad5d90c0bbe956c0d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-21T09:54:39.891 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:39 smithi196 podman[65241]: 2024-09-21 09:54:39.606877235 +0000 UTC m=+0.380359724 container start b52583e0c622047094947c3de82a6132a2912397cc722cad5d90c0bbe956c0d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-21T09:54:39.891 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:39 smithi196 ceph-mgr[65260]: -- 172.21.15.196:0/2065296312 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557ddb7ff0e0 con 0x557ddb7dc800 2024-09-21T09:54:39.891 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:39 smithi196 bash[65241]: b52583e0c622047094947c3de82a6132a2912397cc722cad5d90c0bbe956c0d9 2024-09-21T09:54:39.891 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:39 smithi196 systemd[1]: Started Ceph mgr.x for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:54:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:40 smithi098 ceph-mon[29239]: from='client.25703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:40 smithi098 ceph-mon[29239]: from='client.25709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:40 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:40 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:40 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:40.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:40 smithi098 ceph-mon[39960]: from='client.25703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:40.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:40 smithi098 ceph-mon[39960]: from='client.25709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:40.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:40 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:40.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:40 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:40.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:40 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:40.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:40 smithi196 ceph-mon[32670]: from='client.25703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:40.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:40 smithi196 ceph-mon[32670]: from='client.25709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:40 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:40 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:40 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:41.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:54:40] "GET /metrics HTTP/1.1" 200 240275 "" "Prometheus/2.43.0" 2024-09-21T09:54:41.481 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:54:41.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:41 smithi098 ceph-mon[29239]: pgmap v140: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:41.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:41 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3366762444' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:41.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:41 smithi098 ceph-mon[39960]: pgmap v140: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:41.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:41 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3366762444' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:41.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:41 smithi196 ceph-mon[32670]: pgmap v140: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:41.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:41 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3366762444' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:42.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:42 smithi196 ceph-mon[32670]: from='client.15843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:42.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:42 smithi098 ceph-mon[29239]: from='client.15843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:42.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:42 smithi098 ceph-mon[39960]: from='client.15843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:43.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:43 smithi196 ceph-mon[32670]: from='client.15849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:43.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:43 smithi196 ceph-mon[32670]: pgmap v141: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:43.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:43 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:43.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:43 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:43.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:43 smithi098 ceph-mon[29239]: from='client.15849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:43.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:43 smithi098 ceph-mon[29239]: pgmap v141: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:43.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:43 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:43.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:43 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:43.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:43 smithi098 ceph-mon[39960]: from='client.15849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:43.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:43 smithi098 ceph-mon[39960]: pgmap v141: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:43.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:43 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:43.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:43 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:44.326 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:54:45.139 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:44 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: [21/Sep/2024:09:54:44] ENGINE Bus STARTING 2024-09-21T09:54:45.139 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:44 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: CherryPy Checker: 2024-09-21T09:54:45.139 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:44 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: The Application mounted at '' has an empty config. 2024-09-21T09:54:45.139 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:44 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: 2024-09-21T09:54:45.139 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:44 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: [21/Sep/2024:09:54:44] ENGINE Serving on http://:::9283 2024-09-21T09:54:45.139 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:44 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: [21/Sep/2024:09:54:44] ENGINE Bus STARTED 2024-09-21T09:54:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:44 smithi196 ceph-mon[32670]: pgmap v142: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:44 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:44 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:44 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3529665129' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:44 smithi196 ceph-mon[32670]: from='client.25739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:44 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:44 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:44 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:45.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:44 smithi098 ceph-mon[39960]: pgmap v142: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:45.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:44 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:45.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:44 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:45.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:44 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3529665129' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:45.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:44 smithi098 ceph-mon[39960]: from='client.25739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:45.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:44 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:45.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:44 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:45.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:44 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:45.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:44 smithi098 ceph-mon[29239]: pgmap v142: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:45.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:44 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:45.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:44 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:45.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:44 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3529665129' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:45.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:44 smithi098 ceph-mon[29239]: from='client.25739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:45.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:44 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:45.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:44 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:45.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:44 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:46.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:45 smithi196 ceph-mon[32670]: from='client.25745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:45 smithi196 ceph-mon[32670]: Standby manager daemon x restarted 2024-09-21T09:54:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:45 smithi196 ceph-mon[32670]: Standby manager daemon x started 2024-09-21T09:54:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:45 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:54:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:45 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:54:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:45 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:54:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:45 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:54:46.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:45 smithi098 ceph-mon[29239]: from='client.25745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:46.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:45 smithi098 ceph-mon[29239]: Standby manager daemon x restarted 2024-09-21T09:54:46.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:45 smithi098 ceph-mon[29239]: Standby manager daemon x started 2024-09-21T09:54:46.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:45 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:54:46.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:45 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:54:46.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:45 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:54:46.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:45 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:54:46.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:45 smithi098 ceph-mon[39960]: from='client.25745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:46.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:45 smithi098 ceph-mon[39960]: Standby manager daemon x restarted 2024-09-21T09:54:46.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:45 smithi098 ceph-mon[39960]: Standby manager daemon x started 2024-09-21T09:54:46.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:45 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:54:46.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:45 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:54:46.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:45 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:54:46.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:45 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:54:46.810 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:54:46] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:54:47.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:46 smithi196 ceph-mon[32670]: pgmap v143: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:47.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:46 smithi196 ceph-mon[32670]: mgrmap e21: y(active, since 3m), standbys: x 2024-09-21T09:54:47.158 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:54:47.158 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:46 smithi098 ceph-mon[29239]: pgmap v143: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:47.158 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:46 smithi098 ceph-mon[29239]: mgrmap e21: y(active, since 3m), standbys: x 2024-09-21T09:54:47.158 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:46 smithi098 ceph-mon[39960]: pgmap v143: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:47.158 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:46 smithi098 ceph-mon[39960]: mgrmap e21: y(active, since 3m), standbys: x 2024-09-21T09:54:47.932 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:47 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2323488274' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:47.932 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:47 smithi196 ceph-mon[32670]: from='client.25757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:47.932 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:47 smithi196 ceph-mon[32670]: from='client.15885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:48.215 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:47 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2323488274' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:48.215 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:47 smithi098 ceph-mon[29239]: from='client.25757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:48.215 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:47 smithi098 ceph-mon[29239]: from='client.15885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:48.215 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:47 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2323488274' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:48.215 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:47 smithi098 ceph-mon[39960]: from='client.25757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:48.216 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:47 smithi098 ceph-mon[39960]: from='client.15885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:48.216 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:48.078+0000 7fec757ad640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-21T09:54:48.216 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ignoring --setuser ceph since I am not root 2024-09-21T09:54:48.216 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ignoring --setgroup ceph since I am not root 2024-09-21T09:54:48.216 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mgr[29471]: -- 172.21.15.98:0/3643511980 <== mon.2 v2:172.21.15.98:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5586562fc340 con 0x558656308c00 2024-09-21T09:54:48.216 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:48.212+0000 7fcee3cf9ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:54:48.388 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: [21/Sep/2024:09:54:48] ENGINE Bus STOPPING 2024-09-21T09:54:48.592 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:48.267+0000 7fcee3cf9ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:54:48.593 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:48.331+0000 7fcee3cf9ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:54:48.593 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:48.454+0000 7fcee3cf9ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:54:48.593 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:48.590+0000 7fcee3cf9ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:54:48.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: [21/Sep/2024:09:54:48] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:54:48.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: [21/Sep/2024:09:54:48] ENGINE Bus STOPPED 2024-09-21T09:54:49.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:48.841+0000 7fcee3cf9ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:54:49.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:48.901+0000 7fcee3cf9ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:54:49.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:49 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:49.074+0000 7fcee3cf9ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:54:49.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:49.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:49.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:49.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:49.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:49.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:49.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T09:54:49.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: Activating manager daemon x 2024-09-21T09:54:49.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:54:49.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:54:49.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:54:49.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi098.stcwsu"}]: dispatch 2024-09-21T09:54:49.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T09:54:49.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi196.ugblah"}]: dispatch 2024-09-21T09:54:49.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-21T09:54:49.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: mgrmap e22: x(active, starting, since 0.0103725s) 2024-09-21T09:54:49.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:54:49.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:54:49.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:54:49.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:54:49.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:54:49.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:54:49.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:54:49.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:54:49.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:54:49.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:54:49.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:54:49.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:54:49.100 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[39960]: Manager daemon x is now available 2024-09-21T09:54:49.100 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:49.100 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:49.100 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:49.100 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:49.100 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:49.100 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:49.100 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T09:54:49.100 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: Activating manager daemon x 2024-09-21T09:54:49.100 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:54:49.100 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:54:49.100 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:54:49.100 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi098.stcwsu"}]: dispatch 2024-09-21T09:54:49.101 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T09:54:49.101 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi196.ugblah"}]: dispatch 2024-09-21T09:54:49.101 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-21T09:54:49.101 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: mgrmap e22: x(active, starting, since 0.0103725s) 2024-09-21T09:54:49.101 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:54:49.101 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:54:49.101 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:54:49.101 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:54:49.101 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:54:49.101 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:54:49.101 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:54:49.101 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:54:49.101 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:54:49.101 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:54:49.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:54:49.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:54:49.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:48 smithi098 ceph-mon[29239]: Manager daemon x is now available 2024-09-21T09:54:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' 2024-09-21T09:54:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T09:54:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: Activating manager daemon x 2024-09-21T09:54:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:54:49.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:54:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:54:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi098.stcwsu"}]: dispatch 2024-09-21T09:54:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T09:54:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi196.ugblah"}]: dispatch 2024-09-21T09:54:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.24323 172.21.15.98:0/574551412' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-21T09:54:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: mgrmap e22: x(active, starting, since 0.0103725s) 2024-09-21T09:54:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:54:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:54:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:54:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:54:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:54:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:54:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:54:49.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:54:49.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:54:49.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:54:49.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:54:49.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:54:49.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:48 smithi196 ceph-mon[32670]: Manager daemon x is now available 2024-09-21T09:54:49.288 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:49 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: [21/Sep/2024:09:54:49] ENGINE Bus STARTING 2024-09-21T09:54:49.288 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:49 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: 2024-09-21T09:54:49.107+0000 7fd451438640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:54:49.288 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:49 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: 2024-09-21T09:54:49.107+0000 7fd451438640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:54:49.288 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:49 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: 2024-09-21T09:54:49.107+0000 7fd451438640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:54:49.288 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:49 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: 2024-09-21T09:54:49.107+0000 7fd451438640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:54:49.288 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:49 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: 2024-09-21T09:54:49.107+0000 7fd451438640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:54:49.288 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:49 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: 2024-09-21T09:54:49.107+0000 7fd454c3f640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:54:49.288 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:49 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: 2024-09-21T09:54:49.107+0000 7fd454c3f640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:54:49.289 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:49 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: 2024-09-21T09:54:49.107+0000 7fd454c3f640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:54:49.289 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:49 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: 2024-09-21T09:54:49.107+0000 7fd454c3f640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:54:49.289 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:49 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: 2024-09-21T09:54:49.107+0000 7fd454c3f640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:54:49.289 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:49 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: [21/Sep/2024:09:54:49] ENGINE Serving on http://:::9283 2024-09-21T09:54:49.289 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:49 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: [21/Sep/2024:09:54:49] ENGINE Bus STARTED 2024-09-21T09:54:50.082 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:54:50.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:49 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:49.962+0000 7fcee3cf9ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:54:50.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:50.031+0000 7fcee3cf9ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:54:50.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:50.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:50.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[29239]: Checking for cert/key for grafana.a 2024-09-21T09:54:50.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[29239]: Migrating grafana.a cert to cert store 2024-09-21T09:54:50.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:50.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[29239]: Migrating grafana.a key to cert store 2024-09-21T09:54:50.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:50.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:50.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:50.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:50.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:54:50.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:54:50.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[29239]: mgrmap e23: x(active, since 1.01051s) 2024-09-21T09:54:50.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T09:54:50.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T09:54:50.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[29239]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:50.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[29239]: Deploying cephadm binary to smithi098 2024-09-21T09:54:50.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2063690611' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:50.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:50.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:50.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[39960]: Checking for cert/key for grafana.a 2024-09-21T09:54:50.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[39960]: Migrating grafana.a cert to cert store 2024-09-21T09:54:50.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:50.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[39960]: Migrating grafana.a key to cert store 2024-09-21T09:54:50.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:50.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:50.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:50.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:50.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:54:50.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:54:50.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[39960]: mgrmap e23: x(active, since 1.01051s) 2024-09-21T09:54:50.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T09:54:50.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T09:54:50.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[39960]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:50.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[39960]: Deploying cephadm binary to smithi098 2024-09-21T09:54:50.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2063690611' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:50.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-mon[32670]: Checking for cert/key for grafana.a 2024-09-21T09:54:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-mon[32670]: Migrating grafana.a cert to cert store 2024-09-21T09:54:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-mon[32670]: Migrating grafana.a key to cert store 2024-09-21T09:54:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:54:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:54:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:54:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-mon[32670]: mgrmap e23: x(active, since 1.01051s) 2024-09-21T09:54:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T09:54:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T09:54:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-mon[32670]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-mon[32670]: Deploying cephadm binary to smithi098 2024-09-21T09:54:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2063690611' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:50.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: 2024-09-21T09:54:50.214+0000 7fd43d997640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:09:54:50] ENGINE Error in HTTPServer.serve 2024-09-21T09:54:50.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: Traceback (most recent call last): 2024-09-21T09:54:50.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:54:50.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: self._connections.run(self.expiration_interval) 2024-09-21T09:54:50.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:54:50.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: self._run(expiration_interval) 2024-09-21T09:54:50.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:54:50.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:54:50.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:54:50.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:54:50.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:54:50.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: s = self.context.wrap_socket( 2024-09-21T09:54:50.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:54:50.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: return self.sslsocket_class._create( 2024-09-21T09:54:50.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:54:50.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: self.do_handshake() 2024-09-21T09:54:50.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:54:50.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: self._sslobj.do_handshake() 2024-09-21T09:54:50.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:54:50.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: 2024-09-21T09:54:51.039 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:50.781+0000 7fcee3cf9ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:54:51.039 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:50.898+0000 7fcee3cf9ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:54:51.039 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:51.022+0000 7fcee3cf9ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:54:51.291 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:51.085+0000 7fcee3cf9ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:54:51.292 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:51.212+0000 7fcee3cf9ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:54:51.292 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:51.289+0000 7fcee3cf9ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:54:51.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-mon[29239]: [21/Sep/2024:09:54:50] ENGINE Bus STARTING 2024-09-21T09:54:51.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-mon[29239]: from='client.25805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:51.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-mon[29239]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:51.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-mon[29239]: [21/Sep/2024:09:54:50] ENGINE Serving on https://172.21.15.196:7150 2024-09-21T09:54:51.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-mon[29239]: [21/Sep/2024:09:54:50] ENGINE Error in HTTPServer.serve 2024-09-21T09:54:51.292 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-09-21T09:54:51.293 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:54:51.293 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:54:51.293 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:54:51.293 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._run(expiration_interval) 2024-09-21T09:54:51.293 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:54:51.293 INFO:journalctl@ceph.mon.a.smithi098.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:54:51.293 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:54:51.293 INFO:journalctl@ceph.mon.a.smithi098.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:54:51.293 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:54:51.293 INFO:journalctl@ceph.mon.a.smithi098.stdout: s = self.context.wrap_socket( 2024-09-21T09:54:51.293 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:54:51.293 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.sslsocket_class._create( 2024-09-21T09:54:51.293 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:54:51.293 INFO:journalctl@ceph.mon.a.smithi098.stdout: self.do_handshake() 2024-09-21T09:54:51.294 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:54:51.294 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._sslobj.do_handshake() 2024-09-21T09:54:51.294 INFO:journalctl@ceph.mon.a.smithi098.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:54:51.294 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-mon[29239]: Deploying cephadm binary to smithi196 2024-09-21T09:54:51.294 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-mon[29239]: [21/Sep/2024:09:54:50] ENGINE Serving on http://172.21.15.196:8765 2024-09-21T09:54:51.294 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-mon[29239]: [21/Sep/2024:09:54:50] ENGINE Bus STARTED 2024-09-21T09:54:51.294 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-mon[29239]: from='client.15933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:51.294 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-mon[39960]: [21/Sep/2024:09:54:50] ENGINE Bus STARTING 2024-09-21T09:54:51.294 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-mon[39960]: from='client.25805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:51.294 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-mon[39960]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:51.294 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-mon[39960]: [21/Sep/2024:09:54:50] ENGINE Serving on https://172.21.15.196:7150 2024-09-21T09:54:51.294 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-mon[39960]: [21/Sep/2024:09:54:50] ENGINE Error in HTTPServer.serve 2024-09-21T09:54:51.295 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-09-21T09:54:51.295 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:54:51.295 INFO:journalctl@ceph.mon.c.smithi098.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:54:51.295 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:54:51.295 INFO:journalctl@ceph.mon.c.smithi098.stdout: self._run(expiration_interval) 2024-09-21T09:54:51.295 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:54:51.295 INFO:journalctl@ceph.mon.c.smithi098.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:54:51.295 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:54:51.295 INFO:journalctl@ceph.mon.c.smithi098.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:54:51.295 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:54:51.295 INFO:journalctl@ceph.mon.c.smithi098.stdout: s = self.context.wrap_socket( 2024-09-21T09:54:51.295 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:54:51.295 INFO:journalctl@ceph.mon.c.smithi098.stdout: return self.sslsocket_class._create( 2024-09-21T09:54:51.295 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:54:51.296 INFO:journalctl@ceph.mon.c.smithi098.stdout: self.do_handshake() 2024-09-21T09:54:51.296 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:54:51.296 INFO:journalctl@ceph.mon.c.smithi098.stdout: self._sslobj.do_handshake() 2024-09-21T09:54:51.296 INFO:journalctl@ceph.mon.c.smithi098.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:54:51.296 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-mon[39960]: Deploying cephadm binary to smithi196 2024-09-21T09:54:51.296 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-mon[39960]: [21/Sep/2024:09:54:50] ENGINE Serving on http://172.21.15.196:8765 2024-09-21T09:54:51.296 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-mon[39960]: [21/Sep/2024:09:54:50] ENGINE Bus STARTED 2024-09-21T09:54:51.296 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-mon[39960]: from='client.15933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:51.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:51 smithi196 ceph-mon[32670]: [21/Sep/2024:09:54:50] ENGINE Bus STARTING 2024-09-21T09:54:51.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:51 smithi196 ceph-mon[32670]: from='client.25805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:51 smithi196 ceph-mon[32670]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:51 smithi196 ceph-mon[32670]: [21/Sep/2024:09:54:50] ENGINE Serving on https://172.21.15.196:7150 2024-09-21T09:54:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:51 smithi196 ceph-mon[32670]: [21/Sep/2024:09:54:50] ENGINE Error in HTTPServer.serve 2024-09-21T09:54:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-21T09:54:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:54:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:54:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:54:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._run(expiration_interval) 2024-09-21T09:54:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:54:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:54:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:54:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:54:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:54:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: s = self.context.wrap_socket( 2024-09-21T09:54:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:54:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.sslsocket_class._create( 2024-09-21T09:54:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:54:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: self.do_handshake() 2024-09-21T09:54:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:54:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._sslobj.do_handshake() 2024-09-21T09:54:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:54:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:51 smithi196 ceph-mon[32670]: Deploying cephadm binary to smithi196 2024-09-21T09:54:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:51 smithi196 ceph-mon[32670]: [21/Sep/2024:09:54:50] ENGINE Serving on http://172.21.15.196:8765 2024-09-21T09:54:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:51 smithi196 ceph-mon[32670]: [21/Sep/2024:09:54:50] ENGINE Bus STARTED 2024-09-21T09:54:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:51 smithi196 ceph-mon[32670]: from='client.15933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:51.585 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:51.358+0000 7fcee3cf9ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:54:51.586 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:51.511+0000 7fcee3cf9ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:54:51.586 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:51.583+0000 7fcee3cf9ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:54:51.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:51.648+0000 7fcee3cf9ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:54:51.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:51.729+0000 7fcee3cf9ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:54:51.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:51.793+0000 7fcee3cf9ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:54:52.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:52 smithi196 ceph-mon[32670]: mgrmap e24: x(active, since 3s) 2024-09-21T09:54:52.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:52 smithi098 ceph-mon[29239]: mgrmap e24: x(active, since 3s) 2024-09-21T09:54:52.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:51 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:51.852+0000 7fcee3cf9ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:54:52.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:52 smithi098 ceph-mon[39960]: mgrmap e24: x(active, since 3s) 2024-09-21T09:54:52.740 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:52 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:52.628+0000 7fcee3cf9ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:54:52.740 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:52 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:54:52] ENGINE Bus STARTING 2024-09-21T09:54:52.740 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:52 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: CherryPy Checker: 2024-09-21T09:54:52.740 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:52 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: The Application mounted at '' has an empty config. 2024-09-21T09:54:52.740 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:52 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: 2024-09-21T09:54:52.741 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:52 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:54:52] ENGINE Serving on http://:::9283 2024-09-21T09:54:52.741 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:54:52 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: [21/Sep/2024:09:54:52] ENGINE Bus STARTED 2024-09-21T09:54:53.028 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:54:53.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:53 smithi098 ceph-mon[29239]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:53.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:53 smithi098 ceph-mon[29239]: Standby manager daemon y started 2024-09-21T09:54:53.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:53 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.98:0/505267747' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T09:54:53.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:53 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.98:0/505267747' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:54:53.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:53 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.98:0/505267747' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T09:54:53.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:53 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.98:0/505267747' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:54:53.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:53 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2341043194' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:53.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:53 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:53.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:53 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:53.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:53 smithi098 ceph-mon[39960]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:53.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:53 smithi098 ceph-mon[39960]: Standby manager daemon y started 2024-09-21T09:54:53.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:53 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.98:0/505267747' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T09:54:53.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:53 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.98:0/505267747' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:54:53.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:53 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.98:0/505267747' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T09:54:53.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:53 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.98:0/505267747' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:54:53.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:53 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2341043194' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:53.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:53 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:53.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:53 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:53.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:53 smithi196 ceph-mon[32670]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:54:53.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:53 smithi196 ceph-mon[32670]: Standby manager daemon y started 2024-09-21T09:54:53.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:53 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.98:0/505267747' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T09:54:53.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:53 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.98:0/505267747' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:54:53.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:53 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.98:0/505267747' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T09:54:53.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:53 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.98:0/505267747' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:54:53.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:53 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2341043194' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:53.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:53 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:53.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:53 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:54.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[29239]: from='client.15951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:54.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[29239]: mgrmap e25: x(active, since 5s), standbys: y 2024-09-21T09:54:54.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:54:54.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[29239]: from='client.25835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:54.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:54.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:54.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:54.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[39960]: from='client.15951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:54.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[39960]: mgrmap e25: x(active, since 5s), standbys: y 2024-09-21T09:54:54.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:54:54.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[39960]: from='client.25835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:54.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:54.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:54.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:54 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:54.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:54 smithi196 ceph-mon[32670]: from='client.15951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:54 smithi196 ceph-mon[32670]: mgrmap e25: x(active, since 5s), standbys: y 2024-09-21T09:54:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:54 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:54:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:54 smithi196 ceph-mon[32670]: from='client.25835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:54 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:54 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:54 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:54 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:54 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:54 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:54 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:54 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:54 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:54 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:54 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:55.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:55 smithi098 ceph-mon[29239]: Adjusting osd_memory_target on smithi098 to 2480M 2024-09-21T09:54:55.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:55 smithi098 ceph-mon[29239]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 616 B/s wr, 0 op/s 2024-09-21T09:54:55.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:55 smithi098 ceph-mon[39960]: Adjusting osd_memory_target on smithi098 to 2480M 2024-09-21T09:54:55.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:55 smithi098 ceph-mon[39960]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 616 B/s wr, 0 op/s 2024-09-21T09:54:55.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:55 smithi196 ceph-mon[32670]: Adjusting osd_memory_target on smithi098 to 2480M 2024-09-21T09:54:55.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:55 smithi196 ceph-mon[32670]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 616 B/s wr, 0 op/s 2024-09-21T09:54:55.961 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:54:56.554 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:56 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:56.554 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:56 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:56.554 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:56 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2397974086' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:56.554 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:56 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:56.554 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:56 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:56.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:56 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:56.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:56 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:56.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:56 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2397974086' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:56.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:56 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:56.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:56 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:56 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:56 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:56.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:56 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2397974086' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:56.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:56 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:56.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:56 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:54:56.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:54:56 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:54:56] "GET /metrics HTTP/1.1" 200 30959 "" "Prometheus/2.43.0" 2024-09-21T09:54:57.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:57 smithi196 ceph-mon[32670]: from='client.25847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:57.531 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:57 smithi196 ceph-mon[32670]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 439 B/s wr, 10 op/s 2024-09-21T09:54:57.531 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:57 smithi196 ceph-mon[32670]: from='client.15975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:57.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:57 smithi098 ceph-mon[29239]: from='client.25847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:57.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:57 smithi098 ceph-mon[29239]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 439 B/s wr, 10 op/s 2024-09-21T09:54:57.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:57 smithi098 ceph-mon[29239]: from='client.15975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:57.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:57 smithi098 ceph-mon[39960]: from='client.25847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:57.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:57 smithi098 ceph-mon[39960]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 439 B/s wr, 10 op/s 2024-09-21T09:54:57.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:57 smithi098 ceph-mon[39960]: from='client.15975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:58.863 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:54:59.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:59 smithi196 ceph-mon[32670]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 341 B/s wr, 8 op/s 2024-09-21T09:54:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:54:59 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2354865722' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:59.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:59 smithi098 ceph-mon[29239]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 341 B/s wr, 8 op/s 2024-09-21T09:54:59.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:54:59 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2354865722' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:54:59.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:59 smithi098 ceph-mon[39960]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 341 B/s wr, 8 op/s 2024-09-21T09:54:59.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:54:59 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2354865722' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:00.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:00 smithi196 ceph-mon[32670]: from='client.15987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:00.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:00 smithi196 ceph-mon[32670]: from='client.15993 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:00.672 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:00 smithi098 ceph-mon[29239]: from='client.15987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:00.672 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:00 smithi098 ceph-mon[29239]: from='client.15993 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:00.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:00 smithi098 ceph-mon[39960]: from='client.15987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:00.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:00 smithi098 ceph-mon[39960]: from='client.15993 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:01.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:55:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:55:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:01 smithi196 ceph-mon[32670]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 279 B/s wr, 6 op/s 2024-09-21T09:55:01.777 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:55:01.778 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:01 smithi098 ceph-mon[29239]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 279 B/s wr, 6 op/s 2024-09-21T09:55:01.778 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:01 smithi098 ceph-mon[39960]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 279 B/s wr, 6 op/s 2024-09-21T09:55:02.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:02 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3701594216' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:02.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:02 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3701594216' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:02.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:02 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3701594216' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:03.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:03 smithi098 ceph-mon[29239]: from='client.25883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:03.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:03 smithi098 ceph-mon[29239]: from='client.16011 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:03.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:03 smithi098 ceph-mon[29239]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-21T09:55:03.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:03 smithi098 ceph-mon[39960]: from='client.25883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:03.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:03 smithi098 ceph-mon[39960]: from='client.16011 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:03.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:03 smithi098 ceph-mon[39960]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-21T09:55:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:03 smithi196 ceph-mon[32670]: from='client.25883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:03 smithi196 ceph-mon[32670]: from='client.16011 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:03 smithi196 ceph-mon[32670]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-21T09:55:04.691 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:55:04.692 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:04 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:04.692 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:04 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:04.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:04 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:05.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:05 smithi196 ceph-mon[32670]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-21T09:55:05.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:05 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3308233883' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:05.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:05 smithi098 ceph-mon[29239]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-21T09:55:05.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:05 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3308233883' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:05.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:05 smithi098 ceph-mon[39960]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-21T09:55:05.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:05 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3308233883' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:06.638 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:55:06 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:55:06] "GET /metrics HTTP/1.1" 200 33860 "" "Prometheus/2.43.0" 2024-09-21T09:55:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:06 smithi196 ceph-mon[32670]: from='client.25901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:06 smithi196 ceph-mon[32670]: from='client.16029 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:06 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:06 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:06 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:06 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:06 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:06 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:06 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:06 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:06 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:06 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:06 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:06 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:06 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:06.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[29239]: from='client.25901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:06.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[29239]: from='client.16029 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:06.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:06.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:06.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:06.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:06.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:06.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[39960]: from='client.25901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:06.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[39960]: from='client.16029 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:06.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:06.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:06.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:06.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:06.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:06.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:06 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:07.628 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:55:07.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[29239]: Adjusting osd_memory_target on smithi196 to 2480M 2024-09-21T09:55:07.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[29239]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-21T09:55:07.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[29239]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-21T09:55:07.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[29239]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T09:55:07.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[29239]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:55:07.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[29239]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:55:07.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[29239]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:55:07.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[29239]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:55:07.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[29239]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:55:07.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3833946096' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:07.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[39960]: Adjusting osd_memory_target on smithi196 to 2480M 2024-09-21T09:55:07.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[39960]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-21T09:55:07.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[39960]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-21T09:55:07.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[39960]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T09:55:07.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[39960]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:55:07.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[39960]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:55:07.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[39960]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:55:07.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[39960]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:55:07.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[39960]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:55:07.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:07 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3833946096' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:07 smithi196 ceph-mon[32670]: Adjusting osd_memory_target on smithi196 to 2480M 2024-09-21T09:55:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:07 smithi196 ceph-mon[32670]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-21T09:55:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:07 smithi196 ceph-mon[32670]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-21T09:55:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:07 smithi196 ceph-mon[32670]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T09:55:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:07 smithi196 ceph-mon[32670]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:55:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:07 smithi196 ceph-mon[32670]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:55:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:07 smithi196 ceph-mon[32670]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:55:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:07 smithi196 ceph-mon[32670]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:55:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:07 smithi196 ceph-mon[32670]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:55:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:07 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:07 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:07 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:07 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:07 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:07 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:07 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:07 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:07 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:07 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3833946096' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:08.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:08 smithi098 ceph-mon[29239]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:55:08.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:08 smithi098 ceph-mon[29239]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T09:55:08.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:08 smithi098 ceph-mon[29239]: Deploying daemon alertmanager.a on smithi098 2024-09-21T09:55:08.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:08 smithi098 ceph-mon[29239]: from='client.16041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:08.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:08 smithi098 ceph-mon[39960]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:55:08.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:08 smithi098 ceph-mon[39960]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T09:55:08.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:08 smithi098 ceph-mon[39960]: Deploying daemon alertmanager.a on smithi098 2024-09-21T09:55:08.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:08 smithi098 ceph-mon[39960]: from='client.16041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:08 smithi196 ceph-mon[32670]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:55:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:08 smithi196 ceph-mon[32670]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T09:55:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:08 smithi196 ceph-mon[32670]: Deploying daemon alertmanager.a on smithi098 2024-09-21T09:55:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:08 smithi196 ceph-mon[32670]: from='client.16041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:09.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:09 smithi098 ceph-mon[29239]: from='client.16047 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:09.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:09 smithi098 ceph-mon[29239]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:09.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:09 smithi098 ceph-mon[39960]: from='client.16047 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:09.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:09 smithi098 ceph-mon[39960]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:09.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:09 smithi196 ceph-mon[32670]: from='client.16047 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:09.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:09 smithi196 ceph-mon[32670]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:10.589 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:55:10.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:10 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/696116558' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:10.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:55:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:55:10.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:10 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/696116558' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:10 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/696116558' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:11.715 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:11 smithi098 ceph-mon[29239]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:11.715 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:11 smithi098 ceph-mon[29239]: from='client.25937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:11.715 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:11 smithi098 ceph-mon[39960]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:11.715 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:11 smithi098 ceph-mon[39960]: from='client.25937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:11.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:11 smithi196 ceph-mon[32670]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:11.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:11 smithi196 ceph-mon[32670]: from='client.25937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:12.804 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:12 smithi098 ceph-mon[29239]: from='client.16065 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:12.804 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:12 smithi098 ceph-mon[39960]: from='client.16065 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:12 smithi196 ceph-mon[32670]: from='client.16065 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:13.518 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:55:13.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:13 smithi098 ceph-mon[29239]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:13.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:13 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/1941264527' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:13.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:13 smithi098 ceph-mon[39960]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:13.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:13 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/1941264527' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:13.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:13 smithi196 ceph-mon[32670]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:13 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/1941264527' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:14.483 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:14 smithi098 systemd[1]: Stopping Ceph alertmanager.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:55:14.483 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:14 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[72927]: ts=2024-09-21T09:55:14.440Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-21T09:55:14.785 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:14 smithi098 ceph-mon[29239]: from='client.16077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:14.785 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:14 smithi098 ceph-mon[39960]: from='client.16077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:14.785 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:14 smithi098 podman[86743]: 2024-09-21 09:55:14.48158746 +0000 UTC m=+0.120284019 container died 9ae44cc24819ab5583009a5147d8f3dbada4d905d3c96cc4700032b023c48a7c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:55:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:14 smithi196 ceph-mon[32670]: from='client.16077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:15.097 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:14 smithi098 podman[86743]: 2024-09-21 09:55:14.857557921 +0000 UTC m=+0.496254480 container cleanup 9ae44cc24819ab5583009a5147d8f3dbada4d905d3c96cc4700032b023c48a7c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:55:15.097 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:14 smithi098 bash[86743]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a 2024-09-21T09:55:15.426 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:15 smithi098 podman[86755]: 2024-09-21 09:55:15.133960247 +0000 UTC m=+0.644794987 container remove 9ae44cc24819ab5583009a5147d8f3dbada4d905d3c96cc4700032b023c48a7c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:55:15.426 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:15 smithi098 podman[86755]: 2024-09-21 09:55:15.21802522 +0000 UTC m=+0.728859887 volume remove 58ec910acbe9cbe9c8e97e60a1b9a7b61b7b87b75871d370cbf4cad9a4e68ae2 2024-09-21T09:55:15.845 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:15 smithi098 ceph-mon[29239]: from='client.25961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:15.846 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:15 smithi098 ceph-mon[29239]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:15.846 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:15 smithi098 ceph-mon[39960]: from='client.25961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:15.846 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:15 smithi098 ceph-mon[39960]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:15.847 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:15 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-21T09:55:15.847 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:15 smithi098 systemd[1]: Stopped Ceph alertmanager.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:55:15.847 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:15 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@alertmanager.a.service: Consumed 1.022s CPU time. 2024-09-21T09:55:15.847 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:15 smithi098 systemd[1]: Starting Ceph alertmanager.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:55:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:15 smithi196 ceph-mon[32670]: from='client.25961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:15 smithi196 ceph-mon[32670]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:16.097 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:15 smithi098 podman[86882]: 2024-09-21 09:55:15.84407337 +0000 UTC m=+0.108325961 volume create 9416b1407029b682e0f8d9e2952c1ded7b0e30de155db3f7c555a050715d560b 2024-09-21T09:55:16.097 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:15 smithi098 podman[86882]: 2024-09-21 09:55:15.751429671 +0000 UTC m=+0.015682274 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-21T09:55:16.097 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:15 smithi098 podman[86882]: 2024-09-21 09:55:15.936153815 +0000 UTC m=+0.200406404 container create 72b57f9247443847d472287276d478b2da08bc1bf69a8c0d280cf384956c539e (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:55:16.448 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:55:16.448 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:16 smithi098 podman[86882]: 2024-09-21 09:55:16.23926731 +0000 UTC m=+0.503519906 container init 72b57f9247443847d472287276d478b2da08bc1bf69a8c0d280cf384956c539e (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:55:16.449 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:16 smithi098 podman[86882]: 2024-09-21 09:55:16.242189662 +0000 UTC m=+0.506442261 container start 72b57f9247443847d472287276d478b2da08bc1bf69a8c0d280cf384956c539e (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:55:16.449 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:16 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[86918]: ts=2024-09-21T09:55:16.252Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-21T09:55:16.449 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:16 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[86918]: ts=2024-09-21T09:55:16.252Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-21T09:55:16.449 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:16 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[86918]: ts=2024-09-21T09:55:16.253Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.98 port=9094 2024-09-21T09:55:16.449 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:16 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[86918]: ts=2024-09-21T09:55:16.254Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-21T09:55:16.449 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:16 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[86918]: ts=2024-09-21T09:55:16.276Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T09:55:16.449 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:16 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[86918]: ts=2024-09-21T09:55:16.276Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T09:55:16.449 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:16 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[86918]: ts=2024-09-21T09:55:16.278Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-21T09:55:16.449 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:16 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[86918]: ts=2024-09-21T09:55:16.278Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-21T09:55:16.449 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:16 smithi098 bash[86882]: 72b57f9247443847d472287276d478b2da08bc1bf69a8c0d280cf384956c539e 2024-09-21T09:55:16.449 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:16 smithi098 systemd[1]: Started Ceph alertmanager.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:55:16.811 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:16 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3221147563' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:16.811 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:16 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:16.811 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:16 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:16.811 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:55:16 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:55:16] "GET /metrics HTTP/1.1" 200 33860 "" "Prometheus/2.43.0" 2024-09-21T09:55:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:16 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3221147563' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:16 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:16 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:16.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:16 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3221147563' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:16.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:16 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:16.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:16 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:17.347 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:17 smithi098 systemd[1]: Stopping Ceph node-exporter.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:55:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:17 smithi196 ceph-mon[32670]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:17 smithi196 ceph-mon[32670]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-21T09:55:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:17 smithi196 ceph-mon[32670]: Deploying daemon node-exporter.a on smithi098 2024-09-21T09:55:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:17 smithi196 ceph-mon[32670]: from='client.16095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:17.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:17 smithi098 ceph-mon[29239]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:17.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:17 smithi098 ceph-mon[29239]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-21T09:55:17.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:17 smithi098 ceph-mon[29239]: Deploying daemon node-exporter.a on smithi098 2024-09-21T09:55:17.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:17 smithi098 ceph-mon[29239]: from='client.16095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:17.782 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:17 smithi098 ceph-mon[39960]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:17.783 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:17 smithi098 ceph-mon[39960]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-21T09:55:17.783 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:17 smithi098 ceph-mon[39960]: Deploying daemon node-exporter.a on smithi098 2024-09-21T09:55:17.783 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:17 smithi098 ceph-mon[39960]: from='client.16095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:17.783 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:17 smithi098 podman[87101]: 2024-09-21 09:55:17.481645276 +0000 UTC m=+0.130817193 container died 20cd79611b0362b56c8bad8dcc0a64480eb00efa8643cefddacd10040739d2be (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:55:18.041 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:17 smithi098 podman[87101]: 2024-09-21 09:55:17.871817146 +0000 UTC m=+0.520989060 container cleanup 20cd79611b0362b56c8bad8dcc0a64480eb00efa8643cefddacd10040739d2be (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:55:18.041 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:17 smithi098 bash[87101]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a 2024-09-21T09:55:18.302 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:18 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[86918]: ts=2024-09-21T09:55:18.254Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000236841s 2024-09-21T09:55:18.302 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:18 smithi098 podman[87115]: 2024-09-21 09:55:18.039539646 +0000 UTC m=+0.561050916 container remove 20cd79611b0362b56c8bad8dcc0a64480eb00efa8643cefddacd10040739d2be (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:55:18.303 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:18 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-21T09:55:18.303 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:18 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-21T09:55:18.303 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:18 smithi098 systemd[1]: Stopped Ceph node-exporter.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:55:18.303 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:18 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@node-exporter.a.service: Consumed 2.104s CPU time. 2024-09-21T09:55:18.597 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:18 smithi098 systemd[1]: Starting Ceph node-exporter.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:55:18.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:18 smithi196 ceph-mon[32670]: from='client.16101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:19.039 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:18 smithi098 ceph-mon[29239]: from='client.16101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:19.039 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:18 smithi098 ceph-mon[39960]: from='client.16101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:19.039 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:18 smithi098 bash[87250]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-21T09:55:19.039 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:19 smithi098 bash[87250]: Getting image source signatures 2024-09-21T09:55:19.040 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:19 smithi098 bash[87250]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-21T09:55:19.040 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:19 smithi098 bash[87250]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-21T09:55:19.040 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:19 smithi098 bash[87250]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-21T09:55:19.391 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:55:19.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:19 smithi098 ceph-mon[29239]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:19.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:19 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:19.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:19 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2710985105' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:19.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:19 smithi098 ceph-mon[39960]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:19.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:19 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:19.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:19 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2710985105' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:19.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:19 smithi196 ceph-mon[32670]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:19.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:19 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:19 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2710985105' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:20.853 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:20 smithi098 ceph-mon[29239]: from='client.16113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:20.853 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:55:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:55:20.853 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:20 smithi098 ceph-mon[39960]: from='client.16113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:20.853 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:20 smithi098 bash[87250]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-21T09:55:20.854 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:20 smithi098 bash[87250]: Writing manifest to image destination 2024-09-21T09:55:20.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:20 smithi196 ceph-mon[32670]: from='client.16113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:21.251 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:20 smithi098 podman[87250]: 2024-09-21 09:55:20.943419794 +0000 UTC m=+2.366943298 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-21T09:55:21.251 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 podman[87250]: 2024-09-21 09:55:21.033763846 +0000 UTC m=+2.457287346 container create d587eb6669b7b053736a42d8ce28947df2b55e96c9c45bb1d3c1eecf9d6829c9 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:55:21.251 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 podman[87250]: 2024-09-21 09:55:21.249438473 +0000 UTC m=+2.672961973 container init d587eb6669b7b053736a42d8ce28947df2b55e96c9c45bb1d3c1eecf9d6829c9 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:55:21.597 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 podman[87250]: 2024-09-21 09:55:21.252293005 +0000 UTC m=+2.675816506 container start d587eb6669b7b053736a42d8ce28947df2b55e96c9c45bb1d3c1eecf9d6829c9 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:55:21.597 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.255Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-21T09:55:21.598 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.255Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-21T09:55:21.598 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.256Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T09:55:21.598 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.256Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T09:55:21.598 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.256Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T09:55:21.598 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T09:55:21.598 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T09:55:21.598 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T09:55:21.598 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T09:55:21.598 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T09:55:21.598 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T09:55:21.598 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T09:55:21.598 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T09:55:21.598 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T09:55:21.599 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T09:55:21.599 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T09:55:21.599 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T09:55:21.599 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T09:55:21.599 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T09:55:21.599 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T09:55:21.600 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T09:55:21.600 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T09:55:21.600 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T09:55:21.600 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T09:55:21.600 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T09:55:21.600 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T09:55:21.600 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T09:55:21.600 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T09:55:21.600 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T09:55:21.600 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T09:55:21.600 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T09:55:21.600 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T09:55:21.600 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T09:55:21.601 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T09:55:21.601 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T09:55:21.601 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T09:55:21.601 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T09:55:21.601 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T09:55:21.601 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T09:55:21.601 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T09:55:21.601 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T09:55:21.601 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T09:55:21.601 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T09:55:21.601 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T09:55:21.601 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T09:55:21.601 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T09:55:21.601 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T09:55:21.602 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T09:55:21.602 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T09:55:21.602 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T09:55:21.602 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.257Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T09:55:21.602 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.258Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-21T09:55:21.602 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a[87395]: ts=2024-09-21T09:55:21.258Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T09:55:21.602 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 bash[87250]: d587eb6669b7b053736a42d8ce28947df2b55e96c9c45bb1d3c1eecf9d6829c9 2024-09-21T09:55:21.602 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 09:55:21 smithi098 systemd[1]: Started Ceph node-exporter.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:55:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:21 smithi196 ceph-mon[32670]: from='client.25997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:21 smithi196 ceph-mon[32670]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:21 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:21 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:22.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-mon[29239]: from='client.25997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:22.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-mon[29239]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:22.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:22.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:22.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-mon[39960]: from='client.25997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:22.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-mon[39960]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:22.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:22.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:21 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:22.323 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:55:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:22 smithi098 ceph-mon[29239]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:55:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:22 smithi098 ceph-mon[29239]: Deploying daemon prometheus.a on smithi196 2024-09-21T09:55:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:22 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3240549321' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:23.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:22 smithi098 ceph-mon[39960]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:55:23.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:22 smithi098 ceph-mon[39960]: Deploying daemon prometheus.a on smithi196 2024-09-21T09:55:23.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:22 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3240549321' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:23.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:22 smithi196 ceph-mon[32670]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:55:23.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:22 smithi196 ceph-mon[32670]: Deploying daemon prometheus.a on smithi196 2024-09-21T09:55:23.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:22 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3240549321' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:24.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:23 smithi098 ceph-mon[29239]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:24.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:23 smithi098 ceph-mon[29239]: from='client.16131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:24.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:23 smithi098 ceph-mon[29239]: from='client.16137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:24.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:23 smithi098 ceph-mon[39960]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:24.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:23 smithi098 ceph-mon[39960]: from='client.16131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:24.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:23 smithi098 ceph-mon[39960]: from='client.16137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:24.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:23 smithi196 ceph-mon[32670]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:24.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:23 smithi196 ceph-mon[32670]: from='client.16131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:24.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:23 smithi196 ceph-mon[32670]: from='client.16137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:25.227 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:55:26.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:25 smithi098 ceph-mon[29239]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:26.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:25 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/1383194550' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:26.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:25 smithi098 ceph-mon[39960]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:26.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:25 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/1383194550' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:26.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:25 smithi196 ceph-mon[32670]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:26.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:25 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/1383194550' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:26.597 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 09:55:26 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[86918]: ts=2024-09-21T09:55:26.257Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002806563s 2024-09-21T09:55:26.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:55:26 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:55:26] "GET /metrics HTTP/1.1" 200 33861 "" "Prometheus/2.43.0" 2024-09-21T09:55:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:26 smithi196 ceph-mon[32670]: from='client.16149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:26 smithi196 ceph-mon[32670]: from='client.16155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:26 smithi196 ceph-mon[32670]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:26 smithi098 ceph-mon[29239]: from='client.16149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:26 smithi098 ceph-mon[29239]: from='client.16155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:26 smithi098 ceph-mon[29239]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:27.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:26 smithi098 ceph-mon[39960]: from='client.16149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:27.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:26 smithi098 ceph-mon[39960]: from='client.16155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:27.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:26 smithi098 ceph-mon[39960]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:28.131 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:55:28.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:27 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3835911428' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:28.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:27 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3835911428' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:28.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:27 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3835911428' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:29.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:28 smithi098 ceph-mon[29239]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:29.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:28 smithi098 ceph-mon[29239]: from='client.26045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:29.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:28 smithi098 ceph-mon[29239]: from='client.16173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:29.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:28 smithi098 ceph-mon[39960]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:29.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:28 smithi098 ceph-mon[39960]: from='client.26045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:29.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:28 smithi098 ceph-mon[39960]: from='client.16173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:29.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:28 smithi196 ceph-mon[32670]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:29.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:28 smithi196 ceph-mon[32670]: from='client.26045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:29.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:28 smithi196 ceph-mon[32670]: from='client.16173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:31.043 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:55:31.043 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[29467]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:55:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:55:31.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:31 smithi098 ceph-mon[29239]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:31.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:31 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/1920679419' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:31.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:31 smithi098 ceph-mon[39960]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:31.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:31 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/1920679419' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:31 smithi196 ceph-mon[32670]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:31 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/1920679419' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:32.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:32 smithi196 ceph-mon[32670]: from='client.26063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:32.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:32 smithi196 ceph-mon[32670]: from='client.16191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:32.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:32 smithi098 ceph-mon[29239]: from='client.26063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:32.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:32 smithi098 ceph-mon[29239]: from='client.16191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:32.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:32 smithi098 ceph-mon[39960]: from='client.26063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:32.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:32 smithi098 ceph-mon[39960]: from='client.16191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:32.668 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:32 smithi196 systemd[1]: Stopping Ceph prometheus.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:55:32.668 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:32 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:55:32.654Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T09:55:32.668 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:32 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:55:32.655Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-21T09:55:32.668 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:32 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:55:32.655Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-21T09:55:32.668 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:32 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:55:32.655Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T09:55:32.668 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:32 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:55:32.655Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-21T09:55:32.668 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:32 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:55:32.655Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T09:55:32.668 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:32 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:55:32.655Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-21T09:55:32.668 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:32 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:55:32.655Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-21T09:55:32.668 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:32 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:55:32.655Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-21T09:55:32.668 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:32 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:55:32.655Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T09:55:32.668 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:32 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:55:32.655Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-21T09:55:32.669 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:32 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[62497]: ts=2024-09-21T09:55:32.655Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-21T09:55:32.669 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:32 smithi196 podman[73006]: 2024-09-21 09:55:32.665254579 +0000 UTC m=+0.111403584 container died 93f520945f7b39a8f14a902593e50f8177aeb3dc71fdc6dce5ae7f8c5cebc91b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:55:33.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:33 smithi098 ceph-mon[29239]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:33.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:33 smithi098 ceph-mon[39960]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:33.633 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:33 smithi196 ceph-mon[32670]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:33.954 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:55:34.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:34 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3878825900' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:34.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:34 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:34.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:34 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3878825900' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:34.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:34 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:34.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:34 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3878825900' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:34.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:34 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:34.888 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:34 smithi196 podman[73006]: 2024-09-21 09:55:34.597785984 +0000 UTC m=+2.043934986 container cleanup 93f520945f7b39a8f14a902593e50f8177aeb3dc71fdc6dce5ae7f8c5cebc91b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:55:34.889 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:34 smithi196 bash[73006]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T09:55:34.889 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:34 smithi196 podman[73018]: 2024-09-21 09:55:34.782198081 +0000 UTC m=+2.119071618 container remove 93f520945f7b39a8f14a902593e50f8177aeb3dc71fdc6dce5ae7f8c5cebc91b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:55:35.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-mon[32670]: from='client.16203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:35.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-mon[32670]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:35.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-mon[32670]: from='client.16209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:35.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T09:55:35.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 systemd[1]: Stopped Ceph prometheus.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:55:35.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Consumed 1.610s CPU time. 2024-09-21T09:55:35.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 systemd[1]: Starting Ceph prometheus.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:55:35.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:35 smithi098 ceph-mon[29239]: from='client.16203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:35.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:35 smithi098 ceph-mon[29239]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:35.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:35 smithi098 ceph-mon[29239]: from='client.16209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:35.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:35 smithi098 ceph-mon[39960]: from='client.16203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:35.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:35 smithi098 ceph-mon[39960]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:35.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:35 smithi098 ceph-mon[39960]: from='client.16209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:35.686 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 podman[73149]: 2024-09-21 09:55:35.338246934 +0000 UTC m=+0.016572818 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-21T09:55:35.687 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 podman[73149]: 2024-09-21 09:55:35.441612057 +0000 UTC m=+0.119937921 container create dddd236c92a2351533ccdf51eeb916bbab88fe21e08af824e58d11abf9c2c20e (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:55:36.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 podman[73149]: 2024-09-21 09:55:35.685342489 +0000 UTC m=+0.363668382 container init dddd236c92a2351533ccdf51eeb916bbab88fe21e08af824e58d11abf9c2c20e (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:55:36.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 podman[73149]: 2024-09-21 09:55:35.690343118 +0000 UTC m=+0.368668991 container start dddd236c92a2351533ccdf51eeb916bbab88fe21e08af824e58d11abf9c2c20e (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:55:36.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:35.710Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-21T09:55:36.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:35.710Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-21T09:55:36.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:35.710Z caller=main.go:623 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi196 (none))" 2024-09-21T09:55:36.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:35.710Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T09:55:36.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:35.710Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T09:55:36.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:35.712Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T09:55:36.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:35.713Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-21T09:55:36.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:35.714Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T09:55:36.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:35.714Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T09:55:36.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:35.716Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T09:55:36.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:35.716Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.532µs 2024-09-21T09:55:36.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:35.716Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T09:55:36.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:35.716Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-09-21T09:55:36.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:35.728Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-09-21T09:55:36.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:35.728Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-09-21T09:55:36.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:35.728Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=28.854µs wal_replay_duration=12.37159ms wbl_replay_duration=108ns total_replay_duration=12.435196ms 2024-09-21T09:55:36.141 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:35.731Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T09:55:36.141 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:35.731Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-21T09:55:36.141 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:35.731Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T09:55:36.141 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:35.747Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.037255ms db_storage=891ns remote_storage=1.189µs web_handler=326ns query_engine=592ns scrape=361.095µs scrape_sd=118.519µs notify=14.736µs notify_sd=13.752µs rules=15.087231ms tracing=5.496µs 2024-09-21T09:55:36.141 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:35.747Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-21T09:55:36.141 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:35.747Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T09:55:36.141 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 bash[73149]: dddd236c92a2351533ccdf51eeb916bbab88fe21e08af824e58d11abf9c2c20e 2024-09-21T09:55:36.141 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:35 smithi196 systemd[1]: Started Ceph prometheus.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:55:36.869 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:55:37.109 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:36 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:37.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:36 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:37.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:36 smithi196 ceph-mon[32670]: Reconfiguring grafana.a (dependencies changed)... 2024-09-21T09:55:37.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:36 smithi196 ceph-mon[32670]: Reconfiguring daemon grafana.a on smithi196 2024-09-21T09:55:37.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:36 smithi196 ceph-mon[32670]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:37.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:36 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2102302436' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:37.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:36 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:37.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:36 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:37.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:36 smithi098 ceph-mon[29239]: Reconfiguring grafana.a (dependencies changed)... 2024-09-21T09:55:37.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:36 smithi098 ceph-mon[29239]: Reconfiguring daemon grafana.a on smithi196 2024-09-21T09:55:37.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:36 smithi098 ceph-mon[29239]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:37.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:36 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2102302436' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:37.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:36 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:37.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:36 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:37.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:36 smithi098 ceph-mon[39960]: Reconfiguring grafana.a (dependencies changed)... 2024-09-21T09:55:37.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:36 smithi098 ceph-mon[39960]: Reconfiguring daemon grafana.a on smithi196 2024-09-21T09:55:37.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:36 smithi098 ceph-mon[39960]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:37.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:36 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2102302436' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:37.930 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:37 smithi196 ceph-mon[32670]: from='client.16221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:37.930 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:37 smithi196 ceph-mon[32670]: from='client.26105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:37 smithi098 ceph-mon[29239]: from='client.16221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:37 smithi098 ceph-mon[29239]: from='client.26105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:38.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:37 smithi098 ceph-mon[39960]: from='client.16221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:38.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:37 smithi098 ceph-mon[39960]: from='client.26105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:38.865 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:38 smithi196 systemd[1]: Stopping Ceph grafana.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:55:39.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:38 smithi196 ceph-mon[32670]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:39.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:38 smithi196 podman[73351]: 2024-09-21 09:55:38.896389741 +0000 UTC m=+0.110901730 container died be7bf40319e5f8baccf8af3613c924d045b02c9ada77f4a8fb4a67ca718a0dc7 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, com.redhat.component=ubi8-container, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, version=8.5, build-date=2022-04-29T11:48:11.055897, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, name=ubi8, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.32.0, io.openshift.expose-services=) 2024-09-21T09:55:39.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:38 smithi098 ceph-mon[39960]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:39.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:38 smithi098 ceph-mon[29239]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:39.539 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:39 smithi196 podman[73351]: 2024-09-21 09:55:39.252439467 +0000 UTC m=+0.466951450 container cleanup be7bf40319e5f8baccf8af3613c924d045b02c9ada77f4a8fb4a67ca718a0dc7 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a, release=239.1651231664, io.openshift.expose-services=, vendor=Red Hat, Inc., version=8.5, maintainer=Paul Cuzner , name=ubi8, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, architecture=x86_64, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-09-21T09:55:39.539 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:39 smithi196 bash[73351]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a 2024-09-21T09:55:39.781 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:55:39.867 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:39 smithi196 podman[73364]: 2024-09-21 09:55:39.537486697 +0000 UTC m=+0.642899775 container remove be7bf40319e5f8baccf8af3613c924d045b02c9ada77f4a8fb4a67ca718a0dc7 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a, io.openshift.expose-services=, release=239.1651231664, vcs-type=git, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, architecture=x86_64, build-date=2022-04-29T11:48:11.055897, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, description=Ceph Grafana Container, maintainer=Paul Cuzner , io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-09-21T09:55:39.867 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:39 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-21T09:55:39.867 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:39 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@grafana.a.service: Failed with result 'exit-code'. 2024-09-21T09:55:39.867 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:39 smithi196 systemd[1]: Stopped Ceph grafana.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:55:39.868 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:39 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@grafana.a.service: Consumed 5.249s CPU time. 2024-09-21T09:55:39.868 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:39 smithi196 systemd[1]: Starting Ceph grafana.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:55:40.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:39 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/1684238910' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:40.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:39 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/1684238910' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:40.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:39 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/1684238910' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:40.133 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:39 smithi196 podman[73459]: 2024-09-21 09:55:39.780179761 +0000 UTC m=+0.015423116 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-09-21T09:55:40.133 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:39 smithi196 podman[73459]: 2024-09-21 09:55:39.896763485 +0000 UTC m=+0.132006841 container create 7724661af1a8c1859052fa5a9e7c770b3302a3e3e0403f990a6d8a09a43d4f4d (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a, io.buildah.version=1.32.0, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, name=ubi8, vendor=Red Hat, Inc., vcs-type=git, io.openshift.tags=base rhel8, release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-04-29T11:48:11.055897, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-09-21T09:55:40.388 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 podman[73459]: 2024-09-21 09:55:40.132179818 +0000 UTC m=+0.367423178 container init 7724661af1a8c1859052fa5a9e7c770b3302a3e3e0403f990a6d8a09a43d4f4d (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, release=239.1651231664, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, version=8.5, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=) 2024-09-21T09:55:40.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 podman[73459]: 2024-09-21 09:55:40.135151713 +0000 UTC m=+0.370395078 container start 7724661af1a8c1859052fa5a9e7c770b3302a3e3e0403f990a6d8a09a43d4f4d (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., name=ubi8, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, release=239.1651231664, maintainer=Paul Cuzner , io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-04-29T11:48:11.055897, version=8.5, distribution-scope=public) 2024-09-21T09:55:40.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 bash[73459]: 7724661af1a8c1859052fa5a9e7c770b3302a3e3e0403f990a6d8a09a43d4f4d 2024-09-21T09:55:40.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 systemd[1]: Started Ceph grafana.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:55:40.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=settings t=2024-09-21T09:55:40.29054727Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-21T09:55:40.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=settings t=2024-09-21T09:55:40.290764891Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-21T09:55:40.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=settings t=2024-09-21T09:55:40.290790238Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-21T09:55:40.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=settings t=2024-09-21T09:55:40.290803096Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-21T09:55:40.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=settings t=2024-09-21T09:55:40.290814267Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-21T09:55:40.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=settings t=2024-09-21T09:55:40.290826159Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-21T09:55:40.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=settings t=2024-09-21T09:55:40.29083919Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-21T09:55:40.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=settings t=2024-09-21T09:55:40.290850173Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-21T09:55:40.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=settings t=2024-09-21T09:55:40.290860739Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-21T09:55:40.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=settings t=2024-09-21T09:55:40.290877746Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-21T09:55:40.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=settings t=2024-09-21T09:55:40.290891034Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-21T09:55:40.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=settings t=2024-09-21T09:55:40.290902005Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-21T09:55:40.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=settings t=2024-09-21T09:55:40.290913056Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-21T09:55:40.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=settings t=2024-09-21T09:55:40.290923761Z level=info msg="App mode production" 2024-09-21T09:55:40.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=sqlstore t=2024-09-21T09:55:40.290992045Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-21T09:55:40.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=sqlstore t=2024-09-21T09:55:40.291017325Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-21T09:55:40.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=migrator t=2024-09-21T09:55:40.305667545Z level=info msg="Starting DB migrations" 2024-09-21T09:55:40.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=migrator t=2024-09-21T09:55:40.310223742Z level=info msg="migrations completed" performed=0 skipped=481 duration=439.418µs 2024-09-21T09:55:40.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=sqlstore t=2024-09-21T09:55:40.310609714Z level=info msg="Created default organization" 2024-09-21T09:55:40.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=plugin.loader t=2024-09-21T09:55:40.366612303Z level=info msg="Plugin registered" pluginID=input 2024-09-21T09:55:40.818 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=plugin.loader t=2024-09-21T09:55:40.396681386Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-21T09:55:40.818 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=plugin.loader t=2024-09-21T09:55:40.396806461Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-21T09:55:40.818 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=secrets t=2024-09-21T09:55:40.396928341Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-21T09:55:40.818 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=query_data t=2024-09-21T09:55:40.399639005Z level=info msg="Query Service initialization" 2024-09-21T09:55:40.819 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=live.push_http t=2024-09-21T09:55:40.402248893Z level=info msg="Live Push Gateway initialization" 2024-09-21T09:55:41.132 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:40 smithi196 systemd[1]: Stopping Ceph node-exporter.b for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:55:41.385 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:41 smithi196 podman[73604]: 2024-09-21 09:55:41.13097795 +0000 UTC m=+0.149709131 container died 5f6e262b57853085b863d3a9ffaec4a79a7180f1ea92ca868701f2d59e383909 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:55:41.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:41 smithi196 ceph-mon[32670]: from='client.16239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:41.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:41 smithi196 ceph-mon[32670]: from='client.26123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:41.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:41 smithi196 ceph-mon[32670]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:41.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:41 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:41.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:41 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:41.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:41 smithi196 ceph-mon[32670]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-21T09:55:41.386 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:41 smithi196 ceph-mon[32670]: Deploying daemon node-exporter.b on smithi196 2024-09-21T09:55:41.386 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:41 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=infra.usagestats.collector t=2024-09-21T09:55:41.174516938Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-21T09:55:41.386 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:41 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=provisioning.datasources t=2024-09-21T09:55:41.26132974Z level=info msg="deleted correlations based on configuration" ds_name=Dashboard1 2024-09-21T09:55:41.386 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:41 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=provisioning.datasources t=2024-09-21T09:55:41.261363689Z level=info msg="deleted datasource based on configuration" name=Dashboard1 2024-09-21T09:55:41.386 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:41 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=provisioning.datasources t=2024-09-21T09:55:41.272569583Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-21T09:55:41.386 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:41 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=provisioning.alerting t=2024-09-21T09:55:41.353961897Z level=info msg="starting to provision alerting" 2024-09-21T09:55:41.386 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:41 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=provisioning.alerting t=2024-09-21T09:55:41.353989115Z level=info msg="finished to provision alerting" 2024-09-21T09:55:41.386 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:41 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=grafanaStorageLogger t=2024-09-21T09:55:41.354128615Z level=info msg="storage starting" 2024-09-21T09:55:41.386 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:41 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=ngalert.state.manager t=2024-09-21T09:55:41.354171447Z level=info msg="Warming state cache for startup" 2024-09-21T09:55:41.386 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:41 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=http.server t=2024-09-21T09:55:41.35598537Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-21T09:55:41.387 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:41 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=ngalert.state.manager t=2024-09-21T09:55:41.364319402Z level=info msg="State cache has been initialized" states=0 duration=10.146634ms 2024-09-21T09:55:41.387 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:41 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=ngalert.multiorg.alertmanager t=2024-09-21T09:55:41.364364001Z level=info msg="starting MultiOrg Alertmanager" 2024-09-21T09:55:41.387 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 09:55:41 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=ticker t=2024-09-21T09:55:41.364382071Z level=info msg=starting first_tick=2024-09-21T09:55:50Z 2024-09-21T09:55:41.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:41 smithi098 ceph-mon[39960]: from='client.16239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:41.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:41 smithi098 ceph-mon[39960]: from='client.26123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:41.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:41 smithi098 ceph-mon[39960]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:41.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:41 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:41.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:41 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:41.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:41 smithi098 ceph-mon[39960]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-21T09:55:41.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:41 smithi098 ceph-mon[39960]: Deploying daemon node-exporter.b on smithi196 2024-09-21T09:55:41.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:41 smithi098 ceph-mon[29239]: from='client.16239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:41.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:41 smithi098 ceph-mon[29239]: from='client.26123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:41.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:41 smithi098 ceph-mon[29239]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:41.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:41 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:41.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:41 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:41.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:41 smithi098 ceph-mon[29239]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-21T09:55:41.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:41 smithi098 ceph-mon[29239]: Deploying daemon node-exporter.b on smithi196 2024-09-21T09:55:41.638 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:41 smithi196 podman[73604]: 2024-09-21 09:55:41.562923943 +0000 UTC m=+0.581655124 container cleanup 5f6e262b57853085b863d3a9ffaec4a79a7180f1ea92ca868701f2d59e383909 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:55:41.638 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:41 smithi196 bash[73604]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b 2024-09-21T09:55:41.994 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:41 smithi196 podman[73617]: 2024-09-21 09:55:41.731257735 +0000 UTC m=+0.604017131 container remove 5f6e262b57853085b863d3a9ffaec4a79a7180f1ea92ca868701f2d59e383909 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:55:41.994 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:41 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-21T09:55:41.994 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:41 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-21T09:55:41.994 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:41 smithi196 systemd[1]: Stopped Ceph node-exporter.b for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:55:41.994 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:41 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@node-exporter.b.service: Consumed 2.039s CPU time. 2024-09-21T09:55:42.366 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:42 smithi196 systemd[1]: Starting Ceph node-exporter.b for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:55:42.638 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:42 smithi196 bash[73743]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-21T09:55:42.700 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:55:43.138 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:42 smithi196 bash[73743]: Getting image source signatures 2024-09-21T09:55:43.138 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:42 smithi196 bash[73743]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-21T09:55:43.139 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:42 smithi196 bash[73743]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-21T09:55:43.139 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:42 smithi196 bash[73743]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-21T09:55:43.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:43 smithi098 ceph-mon[29239]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:43.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:43 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2972370575' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:43.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:43 smithi098 ceph-mon[39960]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:43.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:43 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2972370575' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:43.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:43 smithi196 ceph-mon[32670]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:43.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:43 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2972370575' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:44.595 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:44 smithi196 bash[73743]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-21T09:55:44.595 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:44 smithi196 ceph-mon[32670]: from='client.16257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:44.595 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:44 smithi196 ceph-mon[32670]: from='client.16263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:44.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:44 smithi098 ceph-mon[29239]: from='client.16257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:44.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:44 smithi098 ceph-mon[29239]: from='client.16263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:44.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:44 smithi098 ceph-mon[39960]: from='client.16257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:44.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:44 smithi098 ceph-mon[39960]: from='client.16263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:44.888 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:44 smithi196 bash[73743]: Writing manifest to image destination 2024-09-21T09:55:45.148 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:44 smithi196 podman[73743]: 2024-09-21 09:55:44.832664673 +0000 UTC m=+2.561602886 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-21T09:55:45.149 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:44 smithi196 podman[73743]: 2024-09-21 09:55:44.914754495 +0000 UTC m=+2.643692732 container create 7a8564adbc711b7285fb6f712a3c6e0f619405530f16ffc330a0e95546238461 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:55:45.149 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 podman[73743]: 2024-09-21 09:55:45.14160884 +0000 UTC m=+2.870547054 container init 7a8564adbc711b7285fb6f712a3c6e0f619405530f16ffc330a0e95546238461 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:55:45.149 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 podman[73743]: 2024-09-21 09:55:45.145503272 +0000 UTC m=+2.874441474 container start 7a8564adbc711b7285fb6f712a3c6e0f619405530f16ffc330a0e95546238461 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:55:45.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:45 smithi098 ceph-mon[29239]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:45.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:45 smithi098 ceph-mon[39960]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:45.633 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:55:45.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-mon[32670]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:45.639 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.149Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-21T09:55:45.639 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.149Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-21T09:55:45.639 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T09:55:45.639 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T09:55:45.639 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T09:55:45.639 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T09:55:45.639 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T09:55:45.639 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T09:55:45.639 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T09:55:45.639 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T09:55:45.639 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T09:55:45.640 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T09:55:45.640 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T09:55:45.640 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T09:55:45.640 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T09:55:45.640 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T09:55:45.640 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T09:55:45.640 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T09:55:45.640 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T09:55:45.641 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T09:55:45.641 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T09:55:45.641 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T09:55:45.641 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T09:55:45.641 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T09:55:45.641 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T09:55:45.641 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T09:55:45.641 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T09:55:45.641 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T09:55:45.641 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T09:55:45.641 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T09:55:45.641 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T09:55:45.641 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T09:55:45.642 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T09:55:45.642 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T09:55:45.642 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T09:55:45.642 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T09:55:45.642 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T09:55:45.642 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T09:55:45.642 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T09:55:45.642 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T09:55:45.642 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T09:55:45.642 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T09:55:45.642 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T09:55:45.642 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T09:55:45.642 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T09:55:45.643 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T09:55:45.643 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T09:55:45.643 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T09:55:45.643 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T09:55:45.643 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T09:55:45.643 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.150Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T09:55:45.643 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.151Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-21T09:55:45.643 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b[73817]: ts=2024-09-21T09:55:45.151Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T09:55:45.643 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 bash[73743]: 7a8564adbc711b7285fb6f712a3c6e0f619405530f16ffc330a0e95546238461 2024-09-21T09:55:45.643 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 09:55:45 smithi196 systemd[1]: Started Ceph node-exporter.b for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:55:46.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:46.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:46.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/638118334' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:55:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[39960]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:55:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:55:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[39960]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:55:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:55:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[39960]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:55:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[39960]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T09:55:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[39960]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T09:55:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[39960]: from='client.16275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:46.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/638118334' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:55:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[29239]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:55:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:55:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[29239]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:55:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:55:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[29239]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:55:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[29239]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T09:55:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[29239]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T09:55:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:46 smithi098 ceph-mon[29239]: from='client.16275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:46.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:46 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:46.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:46 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:46.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:46 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/638118334' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:46 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:55:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:46 smithi196 ceph-mon[32670]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:55:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:46 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:55:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:46 smithi196 ceph-mon[32670]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:55:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:46 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:55:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:46 smithi196 ceph-mon[32670]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:55:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:46 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:46 smithi196 ceph-mon[32670]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T09:55:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:46 smithi196 ceph-mon[32670]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T09:55:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:46 smithi196 ceph-mon[32670]: from='client.16275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:47.416 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:47 smithi098 ceph-mon[29239]: from='client.16281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:47.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:47 smithi098 ceph-mon[29239]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:47.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:47 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:47.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:47 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:55:47.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:47 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:55:47.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:47 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:55:47.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:47 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:47.417 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:47 smithi098 ceph-mon[39960]: from='client.16281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:47.417 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:47 smithi098 ceph-mon[39960]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:47.417 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:47 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:47.417 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:47 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:55:47.417 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:47 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:55:47.418 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:47 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:55:47.418 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:47 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:47.638 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:55:47 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:55:47] "GET /metrics HTTP/1.1" 200 33859 "" "Prometheus/2.51.0" 2024-09-21T09:55:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:47 smithi196 ceph-mon[32670]: from='client.16281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:47 smithi196 ceph-mon[32670]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:47 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:47 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:55:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:47 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:55:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:47 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:55:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:47 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:48.388 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:48 smithi098 ceph-mon[39960]: Upgrade: Updating mgr.y 2024-09-21T09:55:48.389 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:48 smithi098 ceph-mon[39960]: Deploying daemon mgr.y on smithi098 2024-09-21T09:55:48.389 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:48 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/44186993' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:48.389 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:48 smithi098 ceph-mon[29239]: Upgrade: Updating mgr.y 2024-09-21T09:55:48.389 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:48 smithi098 ceph-mon[29239]: Deploying daemon mgr.y on smithi098 2024-09-21T09:55:48.389 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:48 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/44186993' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:48.611 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:55:48.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:48 smithi196 ceph-mon[32670]: Upgrade: Updating mgr.y 2024-09-21T09:55:48.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:48 smithi196 ceph-mon[32670]: Deploying daemon mgr.y on smithi098 2024-09-21T09:55:48.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:48 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/44186993' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:48.716 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:48 smithi098 systemd[1]: Stopping Ceph mgr.y for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:55:49.001 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:48 smithi098 podman[88341]: 2024-09-21 09:55:48.714212941 +0000 UTC m=+0.159946568 container died 93d2672b85f819d52f6903436147786fb17efe41650ab39c2cfe507b07f15104 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2) 2024-09-21T09:55:49.327 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:49 smithi098 podman[88341]: 2024-09-21 09:55:49.082356842 +0000 UTC m=+0.528090460 container cleanup 93d2672b85f819d52f6903436147786fb17efe41650ab39c2cfe507b07f15104 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-21T09:55:49.327 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:49 smithi098 bash[88341]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y 2024-09-21T09:55:49.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:49 smithi098 ceph-mon[29239]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:49.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:49 smithi098 ceph-mon[29239]: from='client.16293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:49.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:49 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:49.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:49 smithi098 ceph-mon[39960]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:49.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:49 smithi098 ceph-mon[39960]: from='client.16293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:49.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:49 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:49.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:49 smithi098 podman[88377]: 2024-09-21 09:55:49.333767014 +0000 UTC m=+0.616586467 container remove 93d2672b85f819d52f6903436147786fb17efe41650ab39c2cfe507b07f15104 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=quincy-fbd1a6a, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-21T09:55:49.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:49 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-21T09:55:49.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:49 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@mgr.y.service: Failed with result 'exit-code'. 2024-09-21T09:55:49.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:49 smithi098 systemd[1]: Stopped Ceph mgr.y for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:55:49.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:49 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@mgr.y.service: Consumed 40.118s CPU time. 2024-09-21T09:55:49.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:49 smithi196 ceph-mon[32670]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:49.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:49 smithi196 ceph-mon[32670]: from='client.16293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:49 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:50.045 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:49 smithi098 systemd[1]: Starting Ceph mgr.y for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:55:50.045 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:49 smithi098 podman[88508]: 2024-09-21 09:55:49.811141979 +0000 UTC m=+0.015657215 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:55:50.045 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:49 smithi098 podman[88508]: 2024-09-21 09:55:49.920061427 +0000 UTC m=+0.124576656 container create da66347fe5a2e6fa8228f978c8e137dc7d89510aa3e4454193e5aa2dba1f7d49 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-21T09:55:50.332 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:50 smithi098 podman[88508]: 2024-09-21 09:55:50.156405898 +0000 UTC m=+0.360921151 container init da66347fe5a2e6fa8228f978c8e137dc7d89510aa3e4454193e5aa2dba1f7d49 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T09:55:50.332 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:50 smithi098 podman[88508]: 2024-09-21 09:55:50.159498868 +0000 UTC m=+0.364014097 container start da66347fe5a2e6fa8228f978c8e137dc7d89510aa3e4454193e5aa2dba1f7d49 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-21T09:55:50.332 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:50 smithi098 ceph-mgr[88527]: -- 172.21.15.98:0/4269943759 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d16818b0e0 con 0x55d168169000 2024-09-21T09:55:50.332 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:50 smithi098 bash[88508]: da66347fe5a2e6fa8228f978c8e137dc7d89510aa3e4454193e5aa2dba1f7d49 2024-09-21T09:55:50.332 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:50 smithi098 systemd[1]: Started Ceph mgr.y for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:55:50.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:50 smithi098 ceph-mon[29239]: from='client.25771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:50.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:50 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:50.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:50 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:50.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:50 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:50.333 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:50 smithi098 ceph-mon[39960]: from='client.25771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:50.333 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:50 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:50.334 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:50 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:50.334 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:50 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:50.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:50 smithi196 ceph-mon[32670]: from='client.25771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:50.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:50 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:50.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:50 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:50.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:50 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:51.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:51 smithi098 ceph-mon[29239]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:51.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:51 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2565671906' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:51.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:51 smithi098 ceph-mon[39960]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:51.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:51 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2565671906' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:51.617 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:51 smithi196 ceph-mon[32670]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:51.617 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:51 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2565671906' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:51.628 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:55:52.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:52 smithi098 ceph-mon[29239]: from='client.26195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:52.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:52 smithi098 ceph-mon[39960]: from='client.26195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:52.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:52 smithi196 ceph-mon[32670]: from='client.26195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:53.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:53 smithi098 ceph-mon[29239]: from='client.26201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:53.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:53 smithi098 ceph-mon[29239]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:53.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:53 smithi098 ceph-mon[39960]: from='client.26201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:53.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:53 smithi098 ceph-mon[39960]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:53.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:53 smithi196 ceph-mon[32670]: from='client.26201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:53.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:53 smithi196 ceph-mon[32670]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:54.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:54 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:54.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:55:54.666 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:55:55.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:55 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:55.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:55 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:55.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:55 smithi098 ceph-mon[29239]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:55.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:55 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/653600935' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:55.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:55 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:55.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:55 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:55.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:55 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:55.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:55 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:55.320 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:55 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:55.320 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:55 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:55.320 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:55 smithi098 ceph-mon[39960]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:55.320 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:55 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/653600935' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:55.320 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:55 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:55.320 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:55 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:55.320 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:55 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:55.320 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:55 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:55.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:55 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:55.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:55 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:55 smithi196 ceph-mon[32670]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:55 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/653600935' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:55 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:55 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:55 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:55 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:55.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:55 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:09:55:55] ENGINE Bus STARTING 2024-09-21T09:55:55.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:55 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: CherryPy Checker: 2024-09-21T09:55:55.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:55 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: The Application mounted at '' has an empty config. 2024-09-21T09:55:55.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:55 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:55:56.065 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:55 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:09:55:55] ENGINE Serving on http://:::9283 2024-09-21T09:55:56.065 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:55 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:09:55:55] ENGINE Bus STARTED 2024-09-21T09:55:56.065 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:55 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:55.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:55:56.066 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:55 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:55.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:55:56.335 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:56 smithi098 ceph-mon[29239]: from='client.16335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:56.336 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:56 smithi098 ceph-mon[29239]: from='client.16341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:56.336 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:56 smithi098 ceph-mon[29239]: Standby manager daemon y restarted 2024-09-21T09:55:56.336 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:56 smithi098 ceph-mon[29239]: Standby manager daemon y started 2024-09-21T09:55:56.336 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:56 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T09:55:56.336 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:56 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:55:56.336 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:56 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T09:55:56.336 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:56 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:55:56.336 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:56 smithi098 ceph-mon[39960]: from='client.16335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:56.336 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:56 smithi098 ceph-mon[39960]: from='client.16341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:56.336 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:56 smithi098 ceph-mon[39960]: Standby manager daemon y restarted 2024-09-21T09:55:56.336 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:56 smithi098 ceph-mon[39960]: Standby manager daemon y started 2024-09-21T09:55:56.337 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:56 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T09:55:56.337 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:56 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:55:56.337 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:56 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T09:55:56.337 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:56 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:55:56.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:56 smithi196 ceph-mon[32670]: from='client.16335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:56 smithi196 ceph-mon[32670]: from='client.16341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:56 smithi196 ceph-mon[32670]: Standby manager daemon y restarted 2024-09-21T09:55:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:56 smithi196 ceph-mon[32670]: Standby manager daemon y started 2024-09-21T09:55:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:56 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T09:55:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:56 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:55:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:56 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T09:55:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:56 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:55:57.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:57 smithi098 ceph-mon[29239]: mgrmap e26: x(active, since 68s), standbys: y 2024-09-21T09:55:57.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:57 smithi098 ceph-mon[29239]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:57.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:57 smithi098 ceph-mon[39960]: mgrmap e26: x(active, since 68s), standbys: y 2024-09-21T09:55:57.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:57 smithi098 ceph-mon[39960]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:57.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:57 smithi196 ceph-mon[32670]: mgrmap e26: x(active, since 68s), standbys: y 2024-09-21T09:55:57.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:57 smithi196 ceph-mon[32670]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:57.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:55:56 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:55:56.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:55:57.624 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:55:57.888 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:55:57 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:55:57] "GET /metrics HTTP/1.1" 200 33856 "" "Prometheus/2.51.0" 2024-09-21T09:55:58.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/1470767407' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:58.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-mon[29239]: from='client.26231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:58.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:58.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:58.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:58.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:58.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:58.348 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:09:55:58] ENGINE Bus STOPPING 2024-09-21T09:55:58.348 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:09:55:58] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:55:58.348 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:09:55:58] ENGINE Bus STOPPED 2024-09-21T09:55:58.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/1470767407' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:58.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-mon[39960]: from='client.26231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:58.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:58.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:58.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:58.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:58.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:58.388 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:55:58 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: 2024-09-21T09:55:58.124+0000 7fd4e71c5640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-21T09:55:58.389 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:55:58 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: ignoring --setuser ceph since I am not root 2024-09-21T09:55:58.389 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:55:58 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: ignoring --setgroup ceph since I am not root 2024-09-21T09:55:58.389 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:55:58 smithi196 ceph-mgr[65260]: -- 172.21.15.196:0/2002736072 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fdf90a10e0 con 0x55fdf907e800 2024-09-21T09:55:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:58 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/1470767407' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:55:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:58 smithi196 ceph-mon[32670]: from='client.26231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:58 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:58 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:58 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:58 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:58 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' 2024-09-21T09:55:58.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:55:58.563+0000 7f0c4bfd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:55:58.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:55:58.564+0000 7f0c4bfd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:55:58.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:55:58.564+0000 7f0c4bfd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:55:58.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:55:58.564+0000 7f0c4bfd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:55:58.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:55:58.564+0000 7f0c4bfd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:55:58.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:55:58.565+0000 7f0c48fcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:55:58.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:55:58.565+0000 7f0c48fcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:55:58.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:55:58.565+0000 7f0c48fcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:55:58.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:55:58.565+0000 7f0c48fcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:55:58.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:55:58.565+0000 7f0c48fcb640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:55:59.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:09:55:59] ENGINE Bus STARTING 2024-09-21T09:55:59.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:09:55:59] ENGINE Serving on http://:::9283 2024-09-21T09:55:59.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:09:55:59] ENGINE Bus STARTED 2024-09-21T09:55:59.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:59.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T09:55:59.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T09:55:59.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: Activating manager daemon y 2024-09-21T09:55:59.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T09:55:59.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.25408 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-21T09:55:59.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: mgrmap e27: y(active, starting, since 0.005931s) 2024-09-21T09:55:59.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:55:59.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:55:59.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:55:59.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi098.stcwsu"}]: dispatch 2024-09-21T09:55:59.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi196.ugblah"}]: dispatch 2024-09-21T09:55:59.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:55:59.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:55:59.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:55:59.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:55:59.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:55:59.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:55:59.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:55:59.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:55:59.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:55:59.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:55:59.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:55:59.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:55:59.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: Manager daemon y is now available 2024-09-21T09:55:59.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:59.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:59.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:55:59.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:59.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T09:55:59.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T09:55:59.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: Activating manager daemon y 2024-09-21T09:55:59.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T09:55:59.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.25408 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-21T09:55:59.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: mgrmap e27: y(active, starting, since 0.005931s) 2024-09-21T09:55:59.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:55:59.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:55:59.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:55:59.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi098.stcwsu"}]: dispatch 2024-09-21T09:55:59.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi196.ugblah"}]: dispatch 2024-09-21T09:55:59.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:55:59.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:55:59.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:55:59.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:55:59.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:55:59.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:55:59.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:55:59.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:55:59.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:55:59.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:55:59.352 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:55:59.352 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:55:59.352 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: Manager daemon y is now available 2024-09-21T09:55:59.352 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:59.352 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:59.352 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:55:59.352 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:55:59.352 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:55:59.352 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:55:59.352 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:55:59.353 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:55:59.353 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:55:59.353 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:55:59.353 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:55:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.25408 172.21.15.196:0/2411710449' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T09:55:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T09:55:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: Activating manager daemon y 2024-09-21T09:55:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T09:55:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.25408 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-21T09:55:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: mgrmap e27: y(active, starting, since 0.005931s) 2024-09-21T09:55:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:55:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:55:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:55:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi098.stcwsu"}]: dispatch 2024-09-21T09:55:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi196.ugblah"}]: dispatch 2024-09-21T09:55:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:55:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:55:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:55:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:55:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:55:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:55:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:55:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:55:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:55:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:55:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:55:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:55:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: Manager daemon y is now available 2024-09-21T09:55:59.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:59.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:59.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:55:59.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:55:59.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:55:59.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:55:59.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:55:59 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:55:59.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:55:59.629+0000 7f0c2ed5c640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:09:55:59] ENGINE Error in HTTPServer.serve 2024-09-21T09:55:59.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: Traceback (most recent call last): 2024-09-21T09:55:59.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:55:59.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: self._connections.run(self.expiration_interval) 2024-09-21T09:55:59.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:55:59.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: self._run(expiration_interval) 2024-09-21T09:55:59.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:55:59.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:55:59.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:55:59.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:55:59.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:55:59.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: s = self.context.wrap_socket( 2024-09-21T09:55:59.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:55:59.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: return self.sslsocket_class._create( 2024-09-21T09:55:59.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:55:59.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: self.do_handshake() 2024-09-21T09:55:59.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:55:59.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: self._sslobj.do_handshake() 2024-09-21T09:55:59.849 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:55:59.849 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:55:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:56:00.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:00 smithi196 ceph-mon[32670]: mgrmap e28: y(active, since 1.01022s) 2024-09-21T09:56:00.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:00 smithi196 ceph-mon[32670]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:56:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:00 smithi196 ceph-mon[32670]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:00.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:56:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:00.560 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:56:00.561 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:00 smithi098 ceph-mon[29239]: mgrmap e28: y(active, since 1.01022s) 2024-09-21T09:56:00.561 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:00 smithi098 ceph-mon[29239]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:56:00.561 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:00 smithi098 ceph-mon[29239]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:00.561 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:00 smithi098 ceph-mon[39960]: mgrmap e28: y(active, since 1.01022s) 2024-09-21T09:56:00.561 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:00 smithi098 ceph-mon[39960]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:56:00.561 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:00 smithi098 ceph-mon[39960]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:01.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:01 smithi098 ceph-mon[29239]: [21/Sep/2024:09:55:59] ENGINE Bus STARTING 2024-09-21T09:56:01.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:01 smithi098 ceph-mon[29239]: [21/Sep/2024:09:55:59] ENGINE Serving on https://172.21.15.98:7150 2024-09-21T09:56:01.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:01 smithi098 ceph-mon[29239]: [21/Sep/2024:09:55:59] ENGINE Error in HTTPServer.serve 2024-09-21T09:56:01.347 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-09-21T09:56:01.347 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:56:01.347 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:56:01.347 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:56:01.347 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._run(expiration_interval) 2024-09-21T09:56:01.347 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:56:01.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:56:01.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:56:01.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:56:01.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:56:01.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: s = self.context.wrap_socket( 2024-09-21T09:56:01.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:56:01.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.sslsocket_class._create( 2024-09-21T09:56:01.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:56:01.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: self.do_handshake() 2024-09-21T09:56:01.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:56:01.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._sslobj.do_handshake() 2024-09-21T09:56:01.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:56:01.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:01 smithi098 ceph-mon[29239]: [21/Sep/2024:09:55:59] ENGINE Serving on http://172.21.15.98:8765 2024-09-21T09:56:01.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:01 smithi098 ceph-mon[29239]: [21/Sep/2024:09:55:59] ENGINE Bus STARTED 2024-09-21T09:56:01.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:01 smithi098 ceph-mon[29239]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:01.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:01 smithi098 ceph-mon[29239]: mgrmap e29: y(active, since 2s) 2024-09-21T09:56:01.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:01 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3524074135' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:01.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:01 smithi098 ceph-mon[39960]: [21/Sep/2024:09:55:59] ENGINE Bus STARTING 2024-09-21T09:56:01.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:01 smithi098 ceph-mon[39960]: [21/Sep/2024:09:55:59] ENGINE Serving on https://172.21.15.98:7150 2024-09-21T09:56:01.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:01 smithi098 ceph-mon[39960]: [21/Sep/2024:09:55:59] ENGINE Error in HTTPServer.serve 2024-09-21T09:56:01.349 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-09-21T09:56:01.349 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:56:01.349 INFO:journalctl@ceph.mon.c.smithi098.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:56:01.349 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:56:01.350 INFO:journalctl@ceph.mon.c.smithi098.stdout: self._run(expiration_interval) 2024-09-21T09:56:01.350 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:56:01.350 INFO:journalctl@ceph.mon.c.smithi098.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:56:01.350 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:56:01.350 INFO:journalctl@ceph.mon.c.smithi098.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:56:01.350 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:56:01.350 INFO:journalctl@ceph.mon.c.smithi098.stdout: s = self.context.wrap_socket( 2024-09-21T09:56:01.350 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:56:01.350 INFO:journalctl@ceph.mon.c.smithi098.stdout: return self.sslsocket_class._create( 2024-09-21T09:56:01.350 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:56:01.350 INFO:journalctl@ceph.mon.c.smithi098.stdout: self.do_handshake() 2024-09-21T09:56:01.350 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:56:01.350 INFO:journalctl@ceph.mon.c.smithi098.stdout: self._sslobj.do_handshake() 2024-09-21T09:56:01.351 INFO:journalctl@ceph.mon.c.smithi098.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:56:01.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:01 smithi098 ceph-mon[39960]: [21/Sep/2024:09:55:59] ENGINE Serving on http://172.21.15.98:8765 2024-09-21T09:56:01.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:01 smithi098 ceph-mon[39960]: [21/Sep/2024:09:55:59] ENGINE Bus STARTED 2024-09-21T09:56:01.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:01 smithi098 ceph-mon[39960]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:01.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:01 smithi098 ceph-mon[39960]: mgrmap e29: y(active, since 2s) 2024-09-21T09:56:01.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:01 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3524074135' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:01.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:01 smithi196 ceph-mon[32670]: [21/Sep/2024:09:55:59] ENGINE Bus STARTING 2024-09-21T09:56:01.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:01 smithi196 ceph-mon[32670]: [21/Sep/2024:09:55:59] ENGINE Serving on https://172.21.15.98:7150 2024-09-21T09:56:01.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:01 smithi196 ceph-mon[32670]: [21/Sep/2024:09:55:59] ENGINE Error in HTTPServer.serve 2024-09-21T09:56:01.388 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-21T09:56:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:56:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:56:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:56:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._run(expiration_interval) 2024-09-21T09:56:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:56:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:56:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:56:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:56:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:56:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: s = self.context.wrap_socket( 2024-09-21T09:56:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:56:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.sslsocket_class._create( 2024-09-21T09:56:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:56:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: self.do_handshake() 2024-09-21T09:56:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:56:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._sslobj.do_handshake() 2024-09-21T09:56:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:56:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:01 smithi196 ceph-mon[32670]: [21/Sep/2024:09:55:59] ENGINE Serving on http://172.21.15.98:8765 2024-09-21T09:56:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:01 smithi196 ceph-mon[32670]: [21/Sep/2024:09:55:59] ENGINE Bus STARTED 2024-09-21T09:56:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:01 smithi196 ceph-mon[32670]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:01 smithi196 ceph-mon[32670]: mgrmap e29: y(active, since 2s) 2024-09-21T09:56:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:01 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3524074135' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:02.509 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:02 smithi196 ceph-mon[32670]: from='client.26285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:02.509 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:02 smithi196 ceph-mon[32670]: from='client.16416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:02.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:02 smithi098 ceph-mon[29239]: from='client.26285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:02.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:02 smithi098 ceph-mon[29239]: from='client.16416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:02.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:02 smithi098 ceph-mon[39960]: from='client.26285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:02.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:02 smithi098 ceph-mon[39960]: from='client.16416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:03.487 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:03 smithi098 ceph-mon[29239]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:03.488 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:03 smithi098 ceph-mon[29239]: mgrmap e30: y(active, since 4s) 2024-09-21T09:56:03.488 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:03 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:03.488 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:03 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:03.488 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:03 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:03.488 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:03 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:03.488 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:03 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:03.488 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:03 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:03.488 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:03 smithi098 ceph-mon[39960]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:03.488 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:03 smithi098 ceph-mon[39960]: mgrmap e30: y(active, since 4s) 2024-09-21T09:56:03.488 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:03 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:03.489 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:03 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:03.489 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:03 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:03.489 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:03 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:03.489 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:03 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:03.489 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:03 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:03.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:03 smithi196 ceph-mon[32670]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:03.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:03 smithi196 ceph-mon[32670]: mgrmap e30: y(active, since 4s) 2024-09-21T09:56:03.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:03 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:03.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:03 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:03.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:03 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:03.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:03 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:03.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:03 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:03.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:03 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:03.490 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:03 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: [21/Sep/2024:09:56:03] ENGINE Bus STARTING 2024-09-21T09:56:03.490 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:03 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: CherryPy Checker: 2024-09-21T09:56:03.490 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:03 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: The Application mounted at '' has an empty config. 2024-09-21T09:56:03.490 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:03 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: 2024-09-21T09:56:03.490 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:03 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: [21/Sep/2024:09:56:03] ENGINE Serving on http://:::9283 2024-09-21T09:56:03.490 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:03 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: [21/Sep/2024:09:56:03] ENGINE Bus STARTED 2024-09-21T09:56:03.534 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:56:04.485 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:04 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2202280807' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:04.485 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:04 smithi098 ceph-mon[29239]: Standby manager daemon x started 2024-09-21T09:56:04.486 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:04 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.196:0/1430090989' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:56:04.486 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:04 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.196:0/1430090989' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:56:04.486 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:04 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.196:0/1430090989' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:56:04.486 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:04 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.196:0/1430090989' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:56:04.486 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:04 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:04.486 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:04 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:04.486 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:04 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2202280807' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:04.486 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:04 smithi098 ceph-mon[39960]: Standby manager daemon x started 2024-09-21T09:56:04.486 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:04 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/1430090989' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:56:04.486 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:04 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/1430090989' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:56:04.486 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:04 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/1430090989' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:56:04.486 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:04 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/1430090989' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:56:04.487 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:04 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:04.487 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:04 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:04.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:04 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:56:04.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:04 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2202280807' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:04 smithi196 ceph-mon[32670]: Standby manager daemon x started 2024-09-21T09:56:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:04 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/1430090989' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:56:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:04 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/1430090989' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:56:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:04 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/1430090989' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:56:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:04 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/1430090989' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:56:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:04 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:04 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:05.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:05 smithi098 ceph-mon[29239]: from='client.26309 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:05.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:05 smithi098 ceph-mon[29239]: from='client.26315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:05.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:05 smithi098 ceph-mon[29239]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:05.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:05 smithi098 ceph-mon[29239]: mgrmap e31: y(active, since 6s), standbys: x 2024-09-21T09:56:05.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:05 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:56:05.413 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:05 smithi098 ceph-mon[39960]: from='client.26309 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:05.413 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:05 smithi098 ceph-mon[39960]: from='client.26315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:05.413 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:05 smithi098 ceph-mon[39960]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:05.413 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:05 smithi098 ceph-mon[39960]: mgrmap e31: y(active, since 6s), standbys: x 2024-09-21T09:56:05.413 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:05 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:56:05.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:05 smithi196 ceph-mon[32670]: from='client.26309 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:05.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:05 smithi196 ceph-mon[32670]: from='client.26315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:05.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:05 smithi196 ceph-mon[32670]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:05.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:05 smithi196 ceph-mon[32670]: mgrmap e31: y(active, since 6s), standbys: x 2024-09-21T09:56:05.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:05 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:56:05.888 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:05 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:56:05.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:05.888 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:05 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:56:05.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:06.486 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:56:06.486 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:06 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/898328808' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:06.486 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:06 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/898328808' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:06.570 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:06 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/898328808' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:07.390 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:06 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:56:06.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:07 smithi196 ceph-mon[32670]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T09:56:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:07 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:07 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:07 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:07 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:56:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:07 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:07 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:07.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:07 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:07.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:07 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:07.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:07 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:07 smithi098 ceph-mon[29239]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T09:56:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:07 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:07 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:07 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:07 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:56:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:07 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:07 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:07.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:07 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:07.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:07 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:07.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:07 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:07.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:07 smithi098 ceph-mon[39960]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T09:56:07.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:07 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:07.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:07 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:07.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:07 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:07.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:07 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:56:07.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:07 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:07.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:07 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:07.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:07 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:07.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:07 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:07.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:07 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:07.741 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:07 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[65256]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:56:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.51.0" 2024-09-21T09:56:08.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:08 smithi196 ceph-mon[32670]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-21T09:56:08.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:08 smithi196 ceph-mon[32670]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-21T09:56:08.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:08 smithi196 ceph-mon[32670]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:56:08.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:08 smithi196 ceph-mon[32670]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:56:08.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:08 smithi196 ceph-mon[32670]: from='client.16446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:08.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:08 smithi196 ceph-mon[32670]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:56:08.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:08 smithi196 ceph-mon[32670]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:56:08.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:08 smithi196 ceph-mon[32670]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:56:08.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:08 smithi196 ceph-mon[32670]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:56:08.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:08 smithi196 ceph-mon[32670]: from='client.26333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:08.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:08 smithi196 ceph-mon[32670]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:56:08.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:08 smithi196 ceph-mon[32670]: Reconfiguring daemon prometheus.a on smithi196 2024-09-21T09:56:08.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[29239]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-21T09:56:08.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[29239]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-21T09:56:08.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[29239]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:56:08.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[29239]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:56:08.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[29239]: from='client.16446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:08.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[29239]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:56:08.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[29239]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:56:08.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[29239]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:56:08.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[29239]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:56:08.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[29239]: from='client.26333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:08.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[29239]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:56:08.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[29239]: Reconfiguring daemon prometheus.a on smithi196 2024-09-21T09:56:08.600 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[39960]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-21T09:56:08.601 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[39960]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-21T09:56:08.601 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[39960]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:56:08.601 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[39960]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:56:08.601 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[39960]: from='client.16446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:08.601 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[39960]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:56:08.601 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[39960]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:56:08.601 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[39960]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:56:08.601 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[39960]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:56:08.602 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[39960]: from='client.26333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:08.602 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[39960]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:56:08.602 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:08 smithi098 ceph-mon[39960]: Reconfiguring daemon prometheus.a on smithi196 2024-09-21T09:56:09.420 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:56:09.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:09 smithi098 ceph-mon[29239]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T09:56:09.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:09 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/100587532' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:09.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:09 smithi098 ceph-mon[39960]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T09:56:09.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:09 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/100587532' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:09 smithi196 ceph-mon[32670]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T09:56:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:09 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/100587532' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:10.215 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:10 smithi196 systemd[1]: Stopping Ceph prometheus.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:56:10.628 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:56:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:10.628 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:56:10.309Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T09:56:10.628 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:56:10.309Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-21T09:56:10.628 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:56:10.309Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-21T09:56:10.628 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:56:10.309Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T09:56:10.628 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:56:10.309Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T09:56:10.628 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:56:10.309Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-21T09:56:10.628 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:56:10.309Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-21T09:56:10.628 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:56:10.309Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-21T09:56:10.628 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:56:10.309Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-21T09:56:10.628 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:56:10.310Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T09:56:10.629 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:56:10.310Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-21T09:56:10.629 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[73164]: ts=2024-09-21T09:56:10.310Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-21T09:56:10.629 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:10 smithi196 podman[75663]: 2024-09-21 09:56:10.319460349 +0000 UTC m=+0.125055750 container died dddd236c92a2351533ccdf51eeb916bbab88fe21e08af824e58d11abf9c2c20e (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:56:10.888 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:10 smithi196 podman[75663]: 2024-09-21 09:56:10.717128352 +0000 UTC m=+0.522723742 container cleanup dddd236c92a2351533ccdf51eeb916bbab88fe21e08af824e58d11abf9c2c20e (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:56:10.889 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:10 smithi196 bash[75663]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T09:56:11.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-mon[32670]: from='client.16464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:11.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-mon[32670]: from='client.16470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:11.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-mon[32670]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T09:56:11.296 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:10 smithi196 podman[75675]: 2024-09-21 09:56:10.943582594 +0000 UTC m=+0.625631331 container remove dddd236c92a2351533ccdf51eeb916bbab88fe21e08af824e58d11abf9c2c20e (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:56:11.296 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T09:56:11.296 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 systemd[1]: Stopped Ceph prometheus.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:56:11.296 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Consumed 1.047s CPU time. 2024-09-21T09:56:11.297 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 systemd[1]: Starting Ceph prometheus.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:56:11.547 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 podman[75768]: 2024-09-21 09:56:11.294338651 +0000 UTC m=+0.105531194 container create b30b25036dc770698c359850128bd31911b0a40808e1921ae30d90e2ee267d70 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:56:11.547 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 podman[75768]: 2024-09-21 09:56:11.205547353 +0000 UTC m=+0.016739912 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-21T09:56:11.547 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 podman[75768]: 2024-09-21 09:56:11.504942584 +0000 UTC m=+0.316135137 container init b30b25036dc770698c359850128bd31911b0a40808e1921ae30d90e2ee267d70 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:56:11.547 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 podman[75768]: 2024-09-21 09:56:11.507798851 +0000 UTC m=+0.318991395 container start b30b25036dc770698c359850128bd31911b0a40808e1921ae30d90e2ee267d70 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:56:11.547 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.526Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-21T09:56:11.548 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.526Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-21T09:56:11.548 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.526Z caller=main.go:623 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi196 (none))" 2024-09-21T09:56:11.548 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.526Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T09:56:11.548 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.526Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T09:56:11.548 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.528Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T09:56:11.548 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.528Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-21T09:56:11.548 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.530Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T09:56:11.548 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.530Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T09:56:11.548 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.532Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T09:56:11.548 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.532Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.606µs 2024-09-21T09:56:11.548 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.532Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T09:56:11.548 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.532Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-09-21T09:56:11.548 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.545Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-09-21T09:56:11.558 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:11 smithi098 ceph-mon[29239]: from='client.16464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:11.558 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:11 smithi098 ceph-mon[29239]: from='client.16470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:11.559 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:11 smithi098 ceph-mon[29239]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T09:56:11.559 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:11 smithi098 ceph-mon[39960]: from='client.16464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:11.559 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:11 smithi098 ceph-mon[39960]: from='client.16470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:11.559 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:11 smithi098 ceph-mon[39960]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T09:56:11.890 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.550Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-09-21T09:56:11.890 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.550Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-09-21T09:56:11.890 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.551Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=24.124µs wal_replay_duration=18.602864ms wbl_replay_duration=146ns total_replay_duration=18.644495ms 2024-09-21T09:56:11.890 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.555Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T09:56:11.890 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.555Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-21T09:56:11.890 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.555Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T09:56:11.890 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.570Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.543188ms db_storage=868ns remote_storage=1.011µs web_handler=436ns query_engine=728ns scrape=655.042µs scrape_sd=165.061µs notify=9.705µs notify_sd=7.74µs rules=14.177645ms tracing=5.192µs 2024-09-21T09:56:11.890 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.570Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-21T09:56:11.890 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:11.570Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T09:56:11.890 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 bash[75768]: b30b25036dc770698c359850128bd31911b0a40808e1921ae30d90e2ee267d70 2024-09-21T09:56:11.891 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:11 smithi196 systemd[1]: Started Ceph prometheus.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:56:12.317 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:56:13.021 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:12 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:13.021 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:12 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:13.021 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:12 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:56:13.022 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:12 smithi196 ceph-mon[32670]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:56:13.022 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:12 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:13.022 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:12 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3848338289' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:13.022 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:12 smithi196 ceph-mon[32670]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T09:56:13.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:12 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:13.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:12 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:13.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:12 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:56:13.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:12 smithi098 ceph-mon[29239]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:56:13.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:12 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:13.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:12 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3848338289' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:13.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:12 smithi098 ceph-mon[29239]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T09:56:13.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:12 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:13.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:12 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:13.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:12 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:56:13.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:12 smithi098 ceph-mon[39960]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:56:13.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:12 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:13.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:12 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3848338289' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:13.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:12 smithi098 ceph-mon[39960]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T09:56:14.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:13 smithi098 ceph-mon[29239]: from='client.26363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:14.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:13 smithi098 ceph-mon[29239]: from='client.26369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:14.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:13 smithi098 ceph-mon[29239]: Upgrade: Updating mgr.x 2024-09-21T09:56:14.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:13 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:56:14.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:13 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:56:14.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:13 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:56:14.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:13 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:14.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:13 smithi098 ceph-mon[29239]: Deploying daemon mgr.x on smithi196 2024-09-21T09:56:14.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:13 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:14.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:13 smithi098 ceph-mon[39960]: from='client.26363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:14.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:13 smithi098 ceph-mon[39960]: from='client.26369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:14.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:13 smithi098 ceph-mon[39960]: Upgrade: Updating mgr.x 2024-09-21T09:56:14.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:13 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:56:14.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:13 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:56:14.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:13 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:56:14.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:13 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:14.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:13 smithi098 ceph-mon[39960]: Deploying daemon mgr.x on smithi196 2024-09-21T09:56:14.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:13 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:14.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:13 smithi196 ceph-mon[32670]: from='client.26363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:14.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:13 smithi196 ceph-mon[32670]: from='client.26369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:14.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:13 smithi196 ceph-mon[32670]: Upgrade: Updating mgr.x 2024-09-21T09:56:14.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:13 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:56:14.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:13 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:56:14.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:13 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:56:14.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:13 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:14.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:13 smithi196 ceph-mon[32670]: Deploying daemon mgr.x on smithi196 2024-09-21T09:56:14.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:13 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:14.941 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:14 smithi196 ceph-mon[32670]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T09:56:14.941 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:14 smithi196 systemd[1]: Stopping Ceph mgr.x for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:56:15.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:14 smithi098 ceph-mon[29239]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T09:56:15.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:14 smithi098 ceph-mon[39960]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T09:56:15.226 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:56:15.260 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:14 smithi196 podman[76041]: 2024-09-21 09:56:14.939365638 +0000 UTC m=+0.144683681 container died b52583e0c622047094947c3de82a6132a2912397cc722cad5d90c0bbe956c0d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=main-62da54b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True) 2024-09-21T09:56:15.260 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:15 smithi196 podman[76041]: 2024-09-21 09:56:15.258711178 +0000 UTC m=+0.464029215 container cleanup b52583e0c622047094947c3de82a6132a2912397cc722cad5d90c0bbe956c0d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD) 2024-09-21T09:56:15.638 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:15 smithi196 bash[76041]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x 2024-09-21T09:56:15.638 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:15 smithi196 podman[76055]: 2024-09-21 09:56:15.468215344 +0000 UTC m=+0.529025635 container remove b52583e0c622047094947c3de82a6132a2912397cc722cad5d90c0bbe956c0d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-21T09:56:15.638 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:15 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-21T09:56:16.091 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:15 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/1800706986' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:16.091 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:15 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@mgr.x.service: Failed with result 'exit-code'. 2024-09-21T09:56:16.091 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:15 smithi196 systemd[1]: Stopped Ceph mgr.x for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:56:16.091 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:15 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@mgr.x.service: Consumed 17.371s CPU time. 2024-09-21T09:56:16.091 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:15 smithi196 systemd[1]: Starting Ceph mgr.x for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:56:16.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:15 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/1800706986' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:16.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:15 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/1800706986' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:16.348 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:16 smithi196 podman[76177]: 2024-09-21 09:56:15.992162062 +0000 UTC m=+0.015904617 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:56:16.348 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:16 smithi196 podman[76177]: 2024-09-21 09:56:16.120491582 +0000 UTC m=+0.144234159 container create d4969d635bcd65334e4e29736a0a044e98c46ee74eabca0cc2298ab14ae641bc (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T09:56:16.348 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:16 smithi196 podman[76177]: 2024-09-21 09:56:16.346608147 +0000 UTC m=+0.370350691 container init d4969d635bcd65334e4e29736a0a044e98c46ee74eabca0cc2298ab14ae641bc (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731) 2024-09-21T09:56:16.638 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:16 smithi196 podman[76177]: 2024-09-21 09:56:16.349848357 +0000 UTC m=+0.373590900 container start d4969d635bcd65334e4e29736a0a044e98c46ee74eabca0cc2298ab14ae641bc (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x, RELEASE=main-62da54b, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-09-21T09:56:16.639 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:16 smithi196 ceph-mgr[76197]: -- 172.21.15.196:0/4044880720 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5624c38f70e0 con 0x5624c38d5400 2024-09-21T09:56:16.639 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:16 smithi196 bash[76177]: d4969d635bcd65334e4e29736a0a044e98c46ee74eabca0cc2298ab14ae641bc 2024-09-21T09:56:16.639 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:16 smithi196 systemd[1]: Started Ceph mgr.x for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:56:16.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:16 smithi196 ceph-mon[32670]: from='client.26381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:16.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:16 smithi196 ceph-mon[32670]: from='client.16506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:16.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:16 smithi196 ceph-mon[32670]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T09:56:16.951 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:16 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:16.951 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:16 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:16.951 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:16 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:17.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:16 smithi098 ceph-mon[29239]: from='client.26381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:17.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:16 smithi098 ceph-mon[29239]: from='client.16506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:17.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:16 smithi098 ceph-mon[29239]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T09:56:17.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:16 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:17.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:16 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:17.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:16 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:17.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:16 smithi098 ceph-mon[39960]: from='client.26381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:17.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:16 smithi098 ceph-mon[39960]: from='client.16506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:17.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:16 smithi098 ceph-mon[39960]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T09:56:17.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:16 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:17.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:16 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:17.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:16 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:17.261 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:16 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:16.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:18.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:17 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:18.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:17 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:18.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:17 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:18.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:17 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:18.128 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:56:18.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:17 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:18.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:17 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:19.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:18 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2350077717' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:19.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:18 smithi098 ceph-mon[29239]: from='client.26402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:19.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:18 smithi098 ceph-mon[29239]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:19.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:18 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2350077717' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:19.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:18 smithi098 ceph-mon[39960]: from='client.26402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:19.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:18 smithi098 ceph-mon[39960]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:19.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:18 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2350077717' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:19.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:18 smithi196 ceph-mon[32670]: from='client.26402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:19.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:18 smithi196 ceph-mon[32670]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:19 smithi196 ceph-mon[32670]: from='client.26408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:20.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:19 smithi098 ceph-mon[29239]: from='client.26408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:20.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:19 smithi098 ceph-mon[39960]: from='client.26408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:20.524 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:20.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:21.047 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:56:21.047 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:56:20] "GET /metrics HTTP/1.1" 200 33863 "" "Prometheus/2.51.0" 2024-09-21T09:56:21.047 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:20 smithi098 ceph-mon[29239]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:21.048 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:20 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/2823167356' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:21.048 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:20 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:21.048 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:20 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:21.048 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:20 smithi098 ceph-mon[39960]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:21.048 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:20 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2823167356' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:21.048 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:20 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:21.048 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:20 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:21.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:20 smithi196 ceph-mon[32670]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:21.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:20 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2823167356' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:21.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:20 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:20 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:21 smithi196 ceph-mon[32670]: from='client.16539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:21 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:21 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:21 smithi196 ceph-mon[32670]: Standby manager daemon x restarted 2024-09-21T09:56:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:21 smithi196 ceph-mon[32670]: Standby manager daemon x started 2024-09-21T09:56:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:21 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/2978217784' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:56:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:21 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/2978217784' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:56:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:21 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/2978217784' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:56:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:21 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/2978217784' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:56:21.889 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:21 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[76193]: [21/Sep/2024:09:56:21] ENGINE Bus STARTING 2024-09-21T09:56:21.890 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:21 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[76193]: CherryPy Checker: 2024-09-21T09:56:21.890 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:21 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[76193]: The Application mounted at '' has an empty config. 2024-09-21T09:56:21.890 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:21 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[76193]: 2024-09-21T09:56:21.890 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:21 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[76193]: [21/Sep/2024:09:56:21] ENGINE Serving on http://:::9283 2024-09-21T09:56:21.890 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:21 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[76193]: [21/Sep/2024:09:56:21] ENGINE Bus STARTED 2024-09-21T09:56:22.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:21 smithi098 ceph-mon[29239]: from='client.16539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:22.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:21 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:22.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:21 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:22.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:21 smithi098 ceph-mon[29239]: Standby manager daemon x restarted 2024-09-21T09:56:22.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:21 smithi098 ceph-mon[29239]: Standby manager daemon x started 2024-09-21T09:56:22.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:21 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.196:0/2978217784' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:56:22.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:21 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.196:0/2978217784' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:56:22.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:21 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.196:0/2978217784' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:56:22.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:21 smithi098 ceph-mon[29239]: from='mgr.? 172.21.15.196:0/2978217784' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:56:22.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:21 smithi098 ceph-mon[39960]: from='client.16539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:22.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:21 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:22.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:21 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:22.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:21 smithi098 ceph-mon[39960]: Standby manager daemon x restarted 2024-09-21T09:56:22.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:21 smithi098 ceph-mon[39960]: Standby manager daemon x started 2024-09-21T09:56:22.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:21 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/2978217784' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:56:22.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:21 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/2978217784' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:56:22.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:21 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/2978217784' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:56:22.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:21 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/2978217784' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:56:23.071 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:22 smithi196 ceph-mon[32670]: from='client.25984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:23.071 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:22 smithi196 ceph-mon[32670]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:23.071 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:22 smithi196 ceph-mon[32670]: mgrmap e32: y(active, since 24s), standbys: x 2024-09-21T09:56:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:22 smithi098 ceph-mon[29239]: from='client.25984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:22 smithi098 ceph-mon[29239]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:22 smithi098 ceph-mon[29239]: mgrmap e32: y(active, since 24s), standbys: x 2024-09-21T09:56:23.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:22 smithi098 ceph-mon[39960]: from='client.25984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:23.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:22 smithi098 ceph-mon[39960]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:23.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:22 smithi098 ceph-mon[39960]: mgrmap e32: y(active, since 24s), standbys: x 2024-09-21T09:56:23.979 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:56:24.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:23 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/1344252653' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:24.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:23 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/1344252653' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:24.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:23 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/1344252653' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:24.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:24 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:24.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:25.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:24 smithi196 ceph-mon[32670]: from='client.26438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:24 smithi196 ceph-mon[32670]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:24 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:24 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:24 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:24 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:56:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:24 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:24 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:24 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:56:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:24 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:24 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-21T09:56:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:24 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-21T09:56:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:24 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-21T09:56:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:24 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-21T09:56:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:24 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-21T09:56:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:24 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-21T09:56:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:24 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T09:56:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:24 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-21T09:56:25.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[29239]: from='client.26438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:25.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[29239]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:25.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:25.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:25.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:25.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:56:25.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:25.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:25.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:56:25.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:25.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-21T09:56:25.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-21T09:56:25.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-21T09:56:25.168 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-21T09:56:25.168 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-21T09:56:25.168 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-21T09:56:25.168 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T09:56:25.168 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-21T09:56:25.168 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[39960]: from='client.26438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:25.168 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[39960]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:25.168 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:25.168 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:25.168 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:25.168 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:56:25.169 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:25.169 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:25.169 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:56:25.169 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:25.169 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-21T09:56:25.169 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-21T09:56:25.169 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-21T09:56:25.169 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-21T09:56:25.169 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-21T09:56:25.169 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-21T09:56:25.169 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T09:56:25.169 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:24 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-21T09:56:26.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:25 smithi098 ceph-mon[29239]: from='client.26444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:26.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:25 smithi098 ceph-mon[29239]: Upgrade: Setting container_image for all mgr 2024-09-21T09:56:26.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:25 smithi098 ceph-mon[29239]: Upgrade: It appears safe to stop mon.a 2024-09-21T09:56:26.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:25 smithi098 ceph-mon[39960]: from='client.26444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:26.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:25 smithi098 ceph-mon[39960]: Upgrade: Setting container_image for all mgr 2024-09-21T09:56:26.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:25 smithi098 ceph-mon[39960]: Upgrade: It appears safe to stop mon.a 2024-09-21T09:56:26.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:25 smithi196 ceph-mon[32670]: from='client.26444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:25 smithi196 ceph-mon[32670]: Upgrade: Setting container_image for all mgr 2024-09-21T09:56:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:25 smithi196 ceph-mon[32670]: Upgrade: It appears safe to stop mon.a 2024-09-21T09:56:26.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:25 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:25.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:26.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:25 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:25.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:26.928 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:56:26.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:26 smithi098 ceph-mon[29239]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:26.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:26 smithi098 ceph-mon[29239]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:26.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:26 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:56:26.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:26 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:56:26.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:26 smithi098 ceph-mon[29239]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:26.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:26 smithi098 ceph-mon[29239]: from='client.? 172.21.15.98:0/3404207759' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:26.929 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:26 smithi098 ceph-mon[39960]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:26.929 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:26 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:26.930 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:26 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:56:26.930 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:26 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:56:26.930 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:26 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:26.930 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:26 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/3404207759' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:26 smithi196 ceph-mon[32670]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:26 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='mgr.y' 2024-09-21T09:56:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:26 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:56:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:26 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:56:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:26 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:26 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/3404207759' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:27.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:26 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:26.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:28.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:27 smithi098 ceph-mon[29239]: Upgrade: Updating mon.a 2024-09-21T09:56:28.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:27 smithi098 ceph-mon[29239]: Deploying daemon mon.a on smithi098 2024-09-21T09:56:28.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:27 smithi098 ceph-mon[29239]: from='client.16575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:28.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:27 smithi098 ceph-mon[39960]: Upgrade: Updating mon.a 2024-09-21T09:56:28.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:27 smithi098 ceph-mon[39960]: Deploying daemon mon.a on smithi098 2024-09-21T09:56:28.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:27 smithi098 ceph-mon[39960]: from='client.16575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:28.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:27 smithi196 ceph-mon[32670]: Upgrade: Updating mon.a 2024-09-21T09:56:28.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:27 smithi196 ceph-mon[32670]: Deploying daemon mon.a on smithi098 2024-09-21T09:56:28.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:27 smithi196 ceph-mon[32670]: from='client.16575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:28.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:28 smithi098 systemd[1]: Stopping Ceph mon.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:56:28.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:28 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a[29235]: 2024-09-21T09:56:28.339+0000 7fc96b89d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T09:56:28.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:28 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a[29235]: 2024-09-21T09:56:28.339+0000 7fc96b89d640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-21T09:56:28.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:28 smithi098 podman[97350]: 2024-09-21 09:56:28.595297529 +0000 UTC m=+0.360781888 container died f501577d66cb7f2c3ae4a9f00ff86f8e580153126e1c088fd4f1612484a2be1f (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=quincy-fbd1a6a, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T09:56:29.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:29 smithi098 podman[97350]: 2024-09-21 09:56:29.03192052 +0000 UTC m=+0.797404875 container cleanup f501577d66cb7f2c3ae4a9f00ff86f8e580153126e1c088fd4f1612484a2be1f (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-21T09:56:29.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:29 smithi098 bash[97350]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a 2024-09-21T09:56:29.626 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:29 smithi098 podman[97362]: 2024-09-21 09:56:29.358952412 +0000 UTC m=+0.763753826 container remove f501577d66cb7f2c3ae4a9f00ff86f8e580153126e1c088fd4f1612484a2be1f (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a, RELEASE=quincy-fbd1a6a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-21T09:56:29.906 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:29 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-21T09:56:29.906 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:29 smithi098 systemd[1]: Stopped Ceph mon.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:56:29.907 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:29 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@mon.a.service: Consumed 12.132s CPU time. 2024-09-21T09:56:29.907 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:29 smithi098 systemd[1]: Starting Ceph mon.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:56:29.907 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:56:30.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 podman[97539]: 2024-09-21 09:56:29.971000995 +0000 UTC m=+0.019859199 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:56:30.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 podman[97539]: 2024-09-21 09:56:30.11043176 +0000 UTC m=+0.159289953 container create 0a2195915f653b39d02facf97187ed42b1411801de2b8544e408d25b7814e465 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-21T09:56:30.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 podman[97539]: 2024-09-21 09:56:30.338007921 +0000 UTC m=+0.386866125 container init 0a2195915f653b39d02facf97187ed42b1411801de2b8544e408d25b7814e465 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-21T09:56:30.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 podman[97539]: 2024-09-21 09:56:30.34118851 +0000 UTC m=+0.390046706 container start 0a2195915f653b39d02facf97187ed42b1411801de2b8544e408d25b7814e465 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-21T09:56:30.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T09:56:30.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev), process ceph-mon, pid 2 2024-09-21T09:56:30.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: pidfile_write: ignore empty --pid-file 2024-09-21T09:56:30.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: load: jerasure load: lrc 2024-09-21T09:56:30.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: RocksDB version: 7.9.2 2024-09-21T09:56:30.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Git sha 0 2024-09-21T09:56:30.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Compile date 2024-09-20 20:32:28 2024-09-21T09:56:30.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: DB SUMMARY 2024-09-21T09:56:30.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: DB Session ID: GGMEFU0RTS87N08D6AI2 2024-09-21T09:56:30.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: CURRENT file: CURRENT 2024-09-21T09:56:30.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T09:56:30.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: MANIFEST file: MANIFEST-000015 size: 972 Bytes 2024-09-21T09:56:30.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000030.sst 2024-09-21T09:56:30.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000028.log size: 3077667 ; 2024-09-21T09:56:30.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.error_if_exists: 0 2024-09-21T09:56:30.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.create_if_missing: 0 2024-09-21T09:56:30.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.paranoid_checks: 1 2024-09-21T09:56:30.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T09:56:30.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T09:56:30.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T09:56:30.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.env: 0x55db1ef5d120 2024-09-21T09:56:30.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T09:56:30.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.info_log: 0x55db214f7700 2024-09-21T09:56:30.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T09:56:30.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.statistics: (nil) 2024-09-21T09:56:30.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.use_fsync: 0 2024-09-21T09:56:30.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_log_file_size: 0 2024-09-21T09:56:30.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T09:56:30.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T09:56:30.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T09:56:30.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T09:56:30.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.allow_fallocate: 1 2024-09-21T09:56:30.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T09:56:30.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T09:56:30.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.use_direct_reads: 0 2024-09-21T09:56:30.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T09:56:30.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T09:56:30.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.db_log_dir: 2024-09-21T09:56:30.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.wal_dir: 2024-09-21T09:56:30.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T09:56:30.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T09:56:30.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T09:56:30.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T09:56:30.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T09:56:30.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T09:56:30.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T09:56:30.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T09:56:30.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.write_buffer_manager: 0x55db214fa1e0 2024-09-21T09:56:30.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T09:56:30.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T09:56:30.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T09:56:30.602 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T09:56:30.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T09:56:30.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T09:56:30.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T09:56:30.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T09:56:30.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.unordered_write: 0 2024-09-21T09:56:30.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T09:56:30.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T09:56:30.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T09:56:30.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T09:56:30.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.row_cache: None 2024-09-21T09:56:30.603 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.wal_filter: None 2024-09-21T09:56:30.604 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T09:56:30.604 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T09:56:30.604 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.two_write_queues: 0 2024-09-21T09:56:30.604 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T09:56:30.604 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.wal_compression: 0 2024-09-21T09:56:30.604 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.atomic_flush: 0 2024-09-21T09:56:30.604 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T09:56:30.604 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T09:56:30.604 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T09:56:30.604 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.log_readahead_size: 0 2024-09-21T09:56:30.604 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T09:56:30.604 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T09:56:30.604 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T09:56:30.605 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T09:56:30.605 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T09:56:30.605 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T09:56:30.605 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T09:56:30.605 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_background_jobs: 2 2024-09-21T09:56:30.605 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_background_compactions: -1 2024-09-21T09:56:30.605 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_subcompactions: 1 2024-09-21T09:56:30.605 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T09:56:30.605 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T09:56:30.605 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T09:56:30.605 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T09:56:30.605 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T09:56:30.605 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T09:56:30.605 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T09:56:30.606 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T09:56:30.606 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_open_files: -1 2024-09-21T09:56:30.606 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T09:56:30.606 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T09:56:30.606 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T09:56:30.606 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T09:56:30.606 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_background_flushes: -1 2024-09-21T09:56:30.606 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Compression algorithms supported: 2024-09-21T09:56:30.606 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: kZSTD supported: 0 2024-09-21T09:56:30.606 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: kXpressCompression supported: 0 2024-09-21T09:56:30.606 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: kBZip2Compression supported: 0 2024-09-21T09:56:30.606 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T09:56:30.606 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: kLZ4Compression supported: 1 2024-09-21T09:56:30.606 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: kZlibCompression supported: 1 2024-09-21T09:56:30.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T09:56:30.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: kSnappyCompression supported: 1 2024-09-21T09:56:30.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T09:56:30.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T09:56:30.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-09-21T09:56:30.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T09:56:30.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T09:56:30.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.merge_operator: 2024-09-21T09:56:30.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compaction_filter: None 2024-09-21T09:56:30.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T09:56:30.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T09:56:30.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T09:56:30.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T09:56:30.607 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55db214f7320) 2024-09-21T09:56:30.608 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks: 1 2024-09-21T09:56:30.608 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T09:56:30.608 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T09:56:30.608 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_top_level_index_and_filter: 1 2024-09-21T09:56:30.608 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_type: 0 2024-09-21T09:56:30.608 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_index_type: 0 2024-09-21T09:56:30.608 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_shortening: 1 2024-09-21T09:56:30.608 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T09:56:30.608 INFO:journalctl@ceph.mon.a.smithi098.stdout: checksum: 4 2024-09-21T09:56:30.608 INFO:journalctl@ceph.mon.a.smithi098.stdout: no_block_cache: 0 2024-09-21T09:56:30.608 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache: 0x55db214eb610 2024-09-21T09:56:30.608 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_name: BinnedLRUCache 2024-09-21T09:56:30.608 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_options: 2024-09-21T09:56:30.608 INFO:journalctl@ceph.mon.a.smithi098.stdout: capacity : 536870912 2024-09-21T09:56:30.609 INFO:journalctl@ceph.mon.a.smithi098.stdout: num_shard_bits : 4 2024-09-21T09:56:30.609 INFO:journalctl@ceph.mon.a.smithi098.stdout: strict_capacity_limit : 0 2024-09-21T09:56:30.609 INFO:journalctl@ceph.mon.a.smithi098.stdout: high_pri_pool_ratio: 0.000 2024-09-21T09:56:30.609 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_compressed: (nil) 2024-09-21T09:56:30.609 INFO:journalctl@ceph.mon.a.smithi098.stdout: persistent_cache: (nil) 2024-09-21T09:56:30.609 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size: 4096 2024-09-21T09:56:30.609 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size_deviation: 10 2024-09-21T09:56:30.609 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_restart_interval: 16 2024-09-21T09:56:30.609 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_block_restart_interval: 1 2024-09-21T09:56:30.609 INFO:journalctl@ceph.mon.a.smithi098.stdout: metadata_block_size: 4096 2024-09-21T09:56:30.609 INFO:journalctl@ceph.mon.a.smithi098.stdout: partition_filters: 0 2024-09-21T09:56:30.609 INFO:journalctl@ceph.mon.a.smithi098.stdout: use_delta_encoding: 1 2024-09-21T09:56:30.609 INFO:journalctl@ceph.mon.a.smithi098.stdout: filter_policy: bloomfilter 2024-09-21T09:56:30.609 INFO:journalctl@ceph.mon.a.smithi098.stdout: whole_key_filtering: 1 2024-09-21T09:56:30.609 INFO:journalctl@ceph.mon.a.smithi098.stdout: verify_compression: 0 2024-09-21T09:56:30.610 INFO:journalctl@ceph.mon.a.smithi098.stdout: read_amp_bytes_per_bit: 0 2024-09-21T09:56:30.610 INFO:journalctl@ceph.mon.a.smithi098.stdout: format_version: 5 2024-09-21T09:56:30.610 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_index_compression: 1 2024-09-21T09:56:30.610 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_align: 0 2024-09-21T09:56:30.610 INFO:journalctl@ceph.mon.a.smithi098.stdout: max_auto_readahead_size: 262144 2024-09-21T09:56:30.610 INFO:journalctl@ceph.mon.a.smithi098.stdout: prepopulate_block_cache: 0 2024-09-21T09:56:30.610 INFO:journalctl@ceph.mon.a.smithi098.stdout: initial_auto_readahead_size: 8192 2024-09-21T09:56:30.610 INFO:journalctl@ceph.mon.a.smithi098.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T09:56:30.610 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T09:56:30.610 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T09:56:30.610 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compression: NoCompression 2024-09-21T09:56:30.610 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T09:56:30.610 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T09:56:30.611 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T09:56:30.611 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.num_levels: 7 2024-09-21T09:56:30.611 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T09:56:30.611 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T09:56:30.611 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T09:56:30.611 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T09:56:30.611 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T09:56:30.611 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T09:56:30.611 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T09:56:30.611 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:56:30.611 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T09:56:30.611 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T09:56:30.611 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T09:56:30.611 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T09:56:30.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T09:56:30.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T09:56:30.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T09:56:30.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T09:56:30.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:56:30.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T09:56:30.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T09:56:30.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T09:56:30.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T09:56:30.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T09:56:30.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T09:56:30.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T09:56:30.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T09:56:30.612 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T09:56:30.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T09:56:30.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T09:56:30.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T09:56:30.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T09:56:30.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T09:56:30.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T09:56:30.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T09:56:30.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T09:56:30.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T09:56:30.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T09:56:30.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T09:56:30.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T09:56:30.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T09:56:30.613 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T09:56:30.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T09:56:30.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T09:56:30.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T09:56:30.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T09:56:30.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T09:56:30.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T09:56:30.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T09:56:30.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T09:56:30.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T09:56:30.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T09:56:30.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T09:56:30.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T09:56:30.614 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T09:56:30.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T09:56:30.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.inplace_update_support: 0 2024-09-21T09:56:30.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T09:56:30.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T09:56:30.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T09:56:30.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T09:56:30.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.bloom_locality: 0 2024-09-21T09:56:30.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.max_successive_merges: 0 2024-09-21T09:56:30.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T09:56:30.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T09:56:30.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T09:56:30.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T09:56:30.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.ttl: 2592000 2024-09-21T09:56:30.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T09:56:30.615 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T09:56:30.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T09:56:30.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.enable_blob_files: false 2024-09-21T09:56:30.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.min_blob_size: 0 2024-09-21T09:56:30.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T09:56:30.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T09:56:30.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T09:56:30.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T09:56:30.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T09:56:30.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T09:56:30.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T09:56:30.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T09:56:30.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-09-21T09:56:30.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T09:56:30.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 10620, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T09:56:30.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-09-21T09:56:30.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c8237901-677b-4a88-bea9-c36a33f86681 2024-09-21T09:56:30.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912590367978, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-09-21T09:56:30.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-09-21T09:56:30.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912590376423, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 2807503, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 10621, "largest_seqno": 11977, "table_properties": {"data_size": 2800538, "index_size": 4320, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1605, "raw_key_size": 14706, "raw_average_key_size": 23, "raw_value_size": 2786944, "raw_average_value_size": 4368, "num_data_blocks": 197, "num_entries": 638, "num_filter_entries": 638, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726912590, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c8237901-677b-4a88-bea9-c36a33f86681", "db_session_id": "GGMEFU0RTS87N08D6AI2", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-09-21T09:56:30.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912590376562, "job": 1, "event": "recovery_finished"} 2024-09-21T09:56:30.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-09-21T09:56:30.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T09:56:30.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T09:56:30.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55db2151ae00 2024-09-21T09:56:30.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: DB pointer 0x55db2161c000 2024-09-21T09:56:30.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T09:56:30.617 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T09:56:30.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** DB Stats ** 2024-09-21T09:56:30.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:56:30.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T09:56:30.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:56:30.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:56:30.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T09:56:30.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:56:30.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:56:30.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-09-21T09:56:30.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-09-21T09:56:30.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T09:56:30.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T09:56:30.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: L0 1/0 2.68 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 657.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-21T09:56:30.618 INFO:journalctl@ceph.mon.a.smithi098.stdout: L6 1/0 6.35 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-21T09:56:30.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: Sum 2/0 9.03 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 657.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-21T09:56:30.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 657.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-21T09:56:30.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-09-21T09:56:30.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-09-21T09:56:30.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T09:56:30.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:56:30.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 657.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-21T09:56:30.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-09-21T09:56:30.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T09:56:30.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-09-21T09:56:30.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:56:30.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-09-21T09:56:30.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:56:30.619 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:56:30.620 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:56:30.620 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:56:30.620 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative compaction: 0.00 GB write, 210.96 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:56:30.620 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval compaction: 0.00 GB write, 210.96 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:56:30.620 INFO:journalctl@ceph.mon.a.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:56:30.620 INFO:journalctl@ceph.mon.a.smithi098.stdout: Block cache BinnedLRUCache@0x55db214eb610#2 capacity: 512.00 MB usage: 6.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-21T09:56:30.620 INFO:journalctl@ceph.mon.a.smithi098.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,4.58 KB,0.000873208%) Misc(1,0.00 KB,0%) 2024-09-21T09:56:30.620 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-09-21T09:56:30.620 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:56:30.620 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: starting mon.a rank 0 at public addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] at bind addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:56:30.620 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: mon.a@-1(???) e3 preinit fsid 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:56:30.620 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: mon.a@-1(???).mds e9 new map 2024-09-21T09:56:30.620 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: mon.a@-1(???).mds e9 print_map 2024-09-21T09:56:30.620 INFO:journalctl@ceph.mon.a.smithi098.stdout: e9 2024-09-21T09:56:30.620 INFO:journalctl@ceph.mon.a.smithi098.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-21T09:56:30.621 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T09:56:30.621 INFO:journalctl@ceph.mon.a.smithi098.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:56:30.621 INFO:journalctl@ceph.mon.a.smithi098.stdout: legacy client fscid: 1 2024-09-21T09:56:30.621 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-09-21T09:56:30.621 INFO:journalctl@ceph.mon.a.smithi098.stdout: Filesystem 'foo' (1) 2024-09-21T09:56:30.621 INFO:journalctl@ceph.mon.a.smithi098.stdout: fs_name foo 2024-09-21T09:56:30.621 INFO:journalctl@ceph.mon.a.smithi098.stdout: epoch 8 2024-09-21T09:56:30.621 INFO:journalctl@ceph.mon.a.smithi098.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-21T09:56:30.621 INFO:journalctl@ceph.mon.a.smithi098.stdout: created 2024-09-21T09:52:20.376206+0000 2024-09-21T09:56:30.621 INFO:journalctl@ceph.mon.a.smithi098.stdout: modified 2024-09-21T09:52:32.894452+0000 2024-09-21T09:56:30.621 INFO:journalctl@ceph.mon.a.smithi098.stdout: tableserver 0 2024-09-21T09:56:30.621 INFO:journalctl@ceph.mon.a.smithi098.stdout: root 0 2024-09-21T09:56:30.621 INFO:journalctl@ceph.mon.a.smithi098.stdout: session_timeout 60 2024-09-21T09:56:30.621 INFO:journalctl@ceph.mon.a.smithi098.stdout: session_autoclose 300 2024-09-21T09:56:30.622 INFO:journalctl@ceph.mon.a.smithi098.stdout: max_file_size 1099511627776 2024-09-21T09:56:30.622 INFO:journalctl@ceph.mon.a.smithi098.stdout: max_xattr_size 65536 2024-09-21T09:56:30.622 INFO:journalctl@ceph.mon.a.smithi098.stdout: required_client_features {} 2024-09-21T09:56:30.622 INFO:journalctl@ceph.mon.a.smithi098.stdout: last_failure 0 2024-09-21T09:56:30.622 INFO:journalctl@ceph.mon.a.smithi098.stdout: last_failure_osd_epoch 0 2024-09-21T09:56:30.622 INFO:journalctl@ceph.mon.a.smithi098.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:56:30.622 INFO:journalctl@ceph.mon.a.smithi098.stdout: max_mds 1 2024-09-21T09:56:30.622 INFO:journalctl@ceph.mon.a.smithi098.stdout: in 0 2024-09-21T09:56:30.622 INFO:journalctl@ceph.mon.a.smithi098.stdout: up {0=14640} 2024-09-21T09:56:30.622 INFO:journalctl@ceph.mon.a.smithi098.stdout: failed 2024-09-21T09:56:30.622 INFO:journalctl@ceph.mon.a.smithi098.stdout: damaged 2024-09-21T09:56:30.622 INFO:journalctl@ceph.mon.a.smithi098.stdout: stopped 2024-09-21T09:56:30.622 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_pools [4] 2024-09-21T09:56:30.622 INFO:journalctl@ceph.mon.a.smithi098.stdout: metadata_pool 3 2024-09-21T09:56:30.622 INFO:journalctl@ceph.mon.a.smithi098.stdout: inline_data disabled 2024-09-21T09:56:30.623 INFO:journalctl@ceph.mon.a.smithi098.stdout: balancer 2024-09-21T09:56:30.623 INFO:journalctl@ceph.mon.a.smithi098.stdout: bal_rank_mask -1 2024-09-21T09:56:30.623 INFO:journalctl@ceph.mon.a.smithi098.stdout: standby_count_wanted 1 2024-09-21T09:56:30.623 INFO:journalctl@ceph.mon.a.smithi098.stdout: qdb_cluster leader: 0 members: 2024-09-21T09:56:30.623 INFO:journalctl@ceph.mon.a.smithi098.stdout: [mds.foo.smithi098.stcwsu{0:14640} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.98:6834/2705511271,v1:172.21.15.98:6835/2705511271] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T09:56:30.623 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-09-21T09:56:30.623 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-09-21T09:56:30.623 INFO:journalctl@ceph.mon.a.smithi098.stdout: Standby daemons: 2024-09-21T09:56:30.623 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-09-21T09:56:30.623 INFO:journalctl@ceph.mon.a.smithi098.stdout: [mds.foo.smithi196.ugblah{-1:24481} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.196:6832/3677035285,v1:172.21.15.196:6833/3677035285] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T09:56:30.623 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: mon.a@-1(???).osd e81 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-21T09:56:30.623 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: mon.a@-1(???).osd e81 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T09:56:30.623 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: mon.a@-1(???).osd e81 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T09:56:30.623 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: mon.a@-1(???).osd e81 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T09:56:30.624 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-mon[97580]: mon.a@-1(???).paxosservice(auth 1..18) refresh upgraded, format 0 -> 3 2024-09-21T09:56:30.624 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 bash[97539]: 0a2195915f653b39d02facf97187ed42b1411801de2b8544e408d25b7814e465 2024-09-21T09:56:30.624 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:30 smithi098 systemd[1]: Started Ceph mon.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:56:30.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:31.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:56:30] "GET /metrics HTTP/1.1" 200 33858 "" "Prometheus/2.51.0" 2024-09-21T09:56:32.138 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[76193]: ignoring --setuser ceph since I am not root 2024-09-21T09:56:32.139 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[76193]: ignoring --setgroup ceph since I am not root 2024-09-21T09:56:32.139 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mgr[76197]: -- 172.21.15.196:0/2541627082 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f72bcd50e0 con 0x55f72bcb2800 2024-09-21T09:56:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mon[32670]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2686382721' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mon[32670]: from='client.26474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mon[32670]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mon[32670]: from='client.26480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mon[32670]: mon.a calling monitor election 2024-09-21T09:56:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mon[32670]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:56:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mon[32670]: monmap epoch 3 2024-09-21T09:56:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mon[32670]: fsid 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:56:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mon[32670]: last_changed 2024-09-21T09:47:06.778233+0000 2024-09-21T09:56:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mon[32670]: created 2024-09-21T09:45:30.814529+0000 2024-09-21T09:56:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mon[32670]: min_mon_release 17 (quincy) 2024-09-21T09:56:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mon[32670]: election_strategy: 1 2024-09-21T09:56:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mon[32670]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-21T09:56:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mon[32670]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-21T09:56:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mon[32670]: 2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.c 2024-09-21T09:56:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mon[32670]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 1 up:standby 2024-09-21T09:56:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mon[32670]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T09:56:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mon[32670]: mgrmap e32: y(active, since 33s), standbys: x 2024-09-21T09:56:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mon[32670]: overall HEALTH_OK 2024-09-21T09:56:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mon[32670]: from='mgr.16305 ' entity='' 2024-09-21T09:56:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:31 smithi196 ceph-mon[32670]: mgrmap e33: y(active, since 33s), standbys: x 2024-09-21T09:56:32.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ignoring --setuser ceph since I am not root 2024-09-21T09:56:32.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ignoring --setgroup ceph since I am not root 2024-09-21T09:56:32.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mgr[88527]: -- 172.21.15.98:0/2334229947 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559183add0e0 con 0x559183aba800 2024-09-21T09:56:32.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[97580]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:32.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2686382721' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:32.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[97580]: from='client.26474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:32.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[97580]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:32.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[97580]: from='client.26480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:32.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[97580]: mon.a calling monitor election 2024-09-21T09:56:32.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[97580]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:56:32.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[97580]: monmap epoch 3 2024-09-21T09:56:32.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[97580]: fsid 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:56:32.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[97580]: last_changed 2024-09-21T09:47:06.778233+0000 2024-09-21T09:56:32.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[97580]: created 2024-09-21T09:45:30.814529+0000 2024-09-21T09:56:32.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[97580]: min_mon_release 17 (quincy) 2024-09-21T09:56:32.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[97580]: election_strategy: 1 2024-09-21T09:56:32.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[97580]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-21T09:56:32.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[97580]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-21T09:56:32.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[97580]: 2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.c 2024-09-21T09:56:32.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[97580]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 1 up:standby 2024-09-21T09:56:32.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[97580]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T09:56:32.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[97580]: mgrmap e32: y(active, since 33s), standbys: x 2024-09-21T09:56:32.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[97580]: overall HEALTH_OK 2024-09-21T09:56:32.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[97580]: from='mgr.16305 ' entity='' 2024-09-21T09:56:32.351 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[97580]: mgrmap e33: y(active, since 33s), standbys: x 2024-09-21T09:56:32.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[39960]: from='mgr.16305 172.21.15.98:0/944434276' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:32.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2686382721' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:56:32.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[39960]: from='client.26474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:32.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[39960]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:32.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[39960]: from='client.26480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:32.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[39960]: mon.a calling monitor election 2024-09-21T09:56:32.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[39960]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:56:32.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[39960]: monmap epoch 3 2024-09-21T09:56:32.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[39960]: fsid 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:56:32.351 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[39960]: last_changed 2024-09-21T09:47:06.778233+0000 2024-09-21T09:56:32.352 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[39960]: created 2024-09-21T09:45:30.814529+0000 2024-09-21T09:56:32.352 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[39960]: min_mon_release 17 (quincy) 2024-09-21T09:56:32.352 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[39960]: election_strategy: 1 2024-09-21T09:56:32.352 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[39960]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-21T09:56:32.352 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[39960]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-21T09:56:32.352 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[39960]: 2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.c 2024-09-21T09:56:32.352 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[39960]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 1 up:standby 2024-09-21T09:56:32.352 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[39960]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T09:56:32.352 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[39960]: mgrmap e32: y(active, since 33s), standbys: x 2024-09-21T09:56:32.352 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[39960]: overall HEALTH_OK 2024-09-21T09:56:32.352 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[39960]: from='mgr.16305 ' entity='' 2024-09-21T09:56:32.352 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:31 smithi098 ceph-mon[39960]: mgrmap e33: y(active, since 33s), standbys: x 2024-09-21T09:56:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 1 2024-09-21T09:56:33.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:32 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2863451693' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T09:56:33.294 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:32 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2863451693' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T09:56:33.294 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:32 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2863451693' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T09:56:33.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch ps' 2024-09-21T09:56:33.575 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:34.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:34 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:34.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:36.146 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:35.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:36.146 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:35.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:36 smithi196 ceph-mon[32670]: Standby manager daemon x restarted 2024-09-21T09:56:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:36 smithi196 ceph-mon[32670]: Standby manager daemon x started 2024-09-21T09:56:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:36 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/1418628139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:56:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:36 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/1418628139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:56:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:36 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/1418628139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:56:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:36 smithi196 ceph-mon[32670]: from='mgr.? 172.21.15.196:0/1418628139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:56:37.141 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:36 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[76193]: [21/Sep/2024:09:56:36] ENGINE Bus STARTING 2024-09-21T09:56:37.141 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:36 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[76193]: CherryPy Checker: 2024-09-21T09:56:37.141 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:36 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[76193]: The Application mounted at '' has an empty config. 2024-09-21T09:56:37.141 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:36 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[76193]: 2024-09-21T09:56:37.141 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:36 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[76193]: [21/Sep/2024:09:56:36] ENGINE Serving on http://:::9283 2024-09-21T09:56:37.141 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 09:56:36 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x[76193]: [21/Sep/2024:09:56:36] ENGINE Bus STARTED 2024-09-21T09:56:37.141 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:36 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:36.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:37.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:36 smithi098 ceph-mon[97580]: Standby manager daemon x restarted 2024-09-21T09:56:37.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:36 smithi098 ceph-mon[97580]: Standby manager daemon x started 2024-09-21T09:56:37.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:36 smithi098 ceph-mon[97580]: from='mgr.? 172.21.15.196:0/1418628139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:56:37.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:36 smithi098 ceph-mon[97580]: from='mgr.? 172.21.15.196:0/1418628139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:56:37.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:36 smithi098 ceph-mon[97580]: from='mgr.? 172.21.15.196:0/1418628139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:56:37.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:36 smithi098 ceph-mon[97580]: from='mgr.? 172.21.15.196:0/1418628139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:56:37.255 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:36 smithi098 ceph-mon[39960]: Standby manager daemon x restarted 2024-09-21T09:56:37.255 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:36 smithi098 ceph-mon[39960]: Standby manager daemon x started 2024-09-21T09:56:37.255 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:36 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/1418628139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:56:37.255 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:36 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/1418628139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:56:37.255 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:36 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/1418628139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:56:37.256 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:36 smithi098 ceph-mon[39960]: from='mgr.? 172.21.15.196:0/1418628139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:56:37.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:56:37.250+0000 7f6c05910640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:56:37.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:56:37.251+0000 7f6c05910640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:56:37.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:56:37.251+0000 7f6c05910640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:56:37.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:56:37.251+0000 7f6c05910640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:56:37.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:56:37.251+0000 7f6c05910640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:56:37.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:56:37.251+0000 7f6c07153640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:56:37.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:56:37.251+0000 7f6c07153640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:56:37.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:56:37.251+0000 7f6c07153640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:56:37.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:56:37.251+0000 7f6c07153640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:56:37.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:56:37.251+0000 7f6c07153640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:56:37.928 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T09:56:37.929 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (82s) 35s ago 5m 18.9M - 0.27.0 11f11916f8cd 72b57f924744 2024-09-21T09:56:37.929 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi196 *:3000 running (58s) 17s ago 5m 78.0M - 9.4.7 954c08fa6188 7724661af1a8 2024-09-21T09:56:37.929 INFO:teuthology.orchestra.run.smithi098.stdout:mds.foo.smithi098.stcwsu smithi098 running (4m) 35s ago 4m 17.2M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be d8393a5abb20 2024-09-21T09:56:37.929 INFO:teuthology.orchestra.run.smithi098.stdout:mds.foo.smithi196.ugblah smithi196 running (4m) 17s ago 4m 15.5M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 05a68593bc2b 2024-09-21T09:56:37.929 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi196 *:8443,9283,8765 running (21s) 17s ago 9m 68.8M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d4969d635bcd 2024-09-21T09:56:37.929 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (48s) 35s ago 10m 487M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 da66347fe5a2 2024-09-21T09:56:37.929 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (10m) 35s ago 11m 52.9M 2048M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be f501577d66cb 2024-09-21T09:56:37.929 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi196 running (9m) 17s ago 9m 45.7M 2048M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 4f1a8a66c2c8 2024-09-21T09:56:37.929 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (9m) 35s ago 9m 44.7M 2048M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 54e9dc041848 2024-09-21T09:56:37.929 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (76s) 35s ago 4m 8186k - 1.7.0 72c9c2088986 d587eb6669b7 2024-09-21T09:56:37.929 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi196 *:9100 running (53s) 17s ago 4m 7960k - 1.7.0 72c9c2088986 7a8564adbc71 2024-09-21T09:56:37.930 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (8m) 35s ago 8m 46.9M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be b4d47286955f 2024-09-21T09:56:37.930 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (8m) 35s ago 8m 48.7M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 850efcb386e6 2024-09-21T09:56:37.930 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (8m) 35s ago 8m 48.9M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be f19a948d9994 2024-09-21T09:56:37.930 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (7m) 35s ago 7m 45.0M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 68f6e62795e7 2024-09-21T09:56:37.930 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi196 running (7m) 17s ago 7m 47.8M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 1693914f1a8b 2024-09-21T09:56:37.930 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi196 running (7m) 17s ago 7m 46.0M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be cb8e4afb2db7 2024-09-21T09:56:37.930 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi196 running (6m) 17s ago 6m 45.6M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 8ba9647073b2 2024-09-21T09:56:37.930 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi196 running (6m) 17s ago 6m 49.8M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 540c98a3c436 2024-09-21T09:56:37.930 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi196 *:9095 running (26s) 17s ago 5m 40.2M - 2.51.0 1d3b7f56885b b30b25036dc7 2024-09-21T09:56:38.180 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: mgrmap e34: y(active, since 38s), standbys: x 2024-09-21T09:56:38.180 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: Active manager daemon y restarted 2024-09-21T09:56:38.181 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: Activating manager daemon y 2024-09-21T09:56:38.181 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T09:56:38.181 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: mgrmap e35: y(active, starting, since 0.0083218s), standbys: x 2024-09-21T09:56:38.181 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:56:38.181 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:56:38.181 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:56:38.181 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi098.stcwsu"}]: dispatch 2024-09-21T09:56:38.181 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi196.ugblah"}]: dispatch 2024-09-21T09:56:38.181 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:56:38.182 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:56:38.182 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:56:38.182 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:56:38.182 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:56:38.182 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:56:38.182 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:56:38.182 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:56:38.182 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:56:38.182 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:56:38.182 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:56:38.182 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:56:38.182 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:56:38.183 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: Manager daemon y is now available 2024-09-21T09:56:38.183 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:38.183 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:38.183 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:56:38.183 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:56:38.183 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T09:56:38.183 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:56:38.184 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:37 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T09:56:38.315 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:56:37.913+0000 7f6c37c82640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-21T09:56:38.316 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:09:56:37] ENGINE Bus STARTING 2024-09-21T09:56:38.316 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: CherryPy Checker: 2024-09-21T09:56:38.316 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: The Application mounted at '' has an empty config. 2024-09-21T09:56:38.316 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:56:38.316 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:09:56:38] ENGINE Serving on http://:::9283 2024-09-21T09:56:38.316 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:09:56:38] ENGINE Bus STARTED 2024-09-21T09:56:38.316 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: mgrmap e34: y(active, since 38s), standbys: x 2024-09-21T09:56:38.317 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: Active manager daemon y restarted 2024-09-21T09:56:38.317 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: Activating manager daemon y 2024-09-21T09:56:38.317 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T09:56:38.317 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: mgrmap e35: y(active, starting, since 0.0083218s), standbys: x 2024-09-21T09:56:38.317 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:56:38.317 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:56:38.317 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:56:38.317 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi098.stcwsu"}]: dispatch 2024-09-21T09:56:38.317 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi196.ugblah"}]: dispatch 2024-09-21T09:56:38.317 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:56:38.317 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:56:38.317 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:56:38.317 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:56:38.317 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:56:38.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:56:38.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:56:38.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:56:38.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:56:38.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:56:38.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:56:38.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:56:38.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:56:38.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: Manager daemon y is now available 2024-09-21T09:56:38.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:38.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:38.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:56:38.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:56:38.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T09:56:38.319 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:56:38.319 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T09:56:38.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: mgrmap e34: y(active, since 38s), standbys: x 2024-09-21T09:56:38.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: Active manager daemon y restarted 2024-09-21T09:56:38.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: Activating manager daemon y 2024-09-21T09:56:38.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T09:56:38.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: mgrmap e35: y(active, starting, since 0.0083218s), standbys: x 2024-09-21T09:56:38.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:56:38.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:56:38.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:56:38.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi098.stcwsu"}]: dispatch 2024-09-21T09:56:38.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi196.ugblah"}]: dispatch 2024-09-21T09:56:38.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:56:38.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:56:38.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:56:38.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:56:38.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:56:38.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:56:38.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:56:38.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:56:38.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:56:38.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:56:38.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:56:38.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:56:38.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:56:38.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: Manager daemon y is now available 2024-09-21T09:56:38.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:38.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:38.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:56:38.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:56:38.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T09:56:38.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:56:38.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:37 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T09:56:38.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:56:38.313+0000 7f6beb762640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:09:56:38] ENGINE Error in HTTPServer.serve 2024-09-21T09:56:38.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: Traceback (most recent call last): 2024-09-21T09:56:38.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:56:38.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: self._connections.run(self.expiration_interval) 2024-09-21T09:56:38.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:56:38.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: self._run(expiration_interval) 2024-09-21T09:56:38.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:56:38.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:56:38.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:56:38.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:56:38.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:56:38.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: s = self.context.wrap_socket( 2024-09-21T09:56:38.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:56:38.599 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: return self.sslsocket_class._create( 2024-09-21T09:56:38.599 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:56:38.599 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: self.do_handshake() 2024-09-21T09:56:38.599 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:56:38.599 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: self._sslobj.do_handshake() 2024-09-21T09:56:38.599 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:56:38.599 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:56:38.926 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade pause' 2024-09-21T09:56:39.165 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-mon[97580]: mgrmap e36: y(active, since 1.01436s), standbys: x 2024-09-21T09:56:39.165 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-mon[97580]: [21/Sep/2024:09:56:38] ENGINE Bus STARTING 2024-09-21T09:56:39.165 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-mon[97580]: [21/Sep/2024:09:56:38] ENGINE Serving on https://172.21.15.98:7150 2024-09-21T09:56:39.165 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-mon[97580]: [21/Sep/2024:09:56:38] ENGINE Error in HTTPServer.serve 2024-09-21T09:56:39.165 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-09-21T09:56:39.165 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:56:39.165 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:56:39.165 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:56:39.165 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._run(expiration_interval) 2024-09-21T09:56:39.166 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:56:39.166 INFO:journalctl@ceph.mon.a.smithi098.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:56:39.166 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:56:39.166 INFO:journalctl@ceph.mon.a.smithi098.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:56:39.166 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:56:39.166 INFO:journalctl@ceph.mon.a.smithi098.stdout: s = self.context.wrap_socket( 2024-09-21T09:56:39.166 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:56:39.166 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.sslsocket_class._create( 2024-09-21T09:56:39.166 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:56:39.166 INFO:journalctl@ceph.mon.a.smithi098.stdout: self.do_handshake() 2024-09-21T09:56:39.166 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:56:39.166 INFO:journalctl@ceph.mon.a.smithi098.stdout: self._sslobj.do_handshake() 2024-09-21T09:56:39.166 INFO:journalctl@ceph.mon.a.smithi098.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:56:39.166 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-mon[97580]: [21/Sep/2024:09:56:38] ENGINE Serving on http://172.21.15.98:8765 2024-09-21T09:56:39.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-mon[97580]: [21/Sep/2024:09:56:38] ENGINE Bus STARTED 2024-09-21T09:56:39.167 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-mon[39960]: mgrmap e36: y(active, since 1.01436s), standbys: x 2024-09-21T09:56:39.167 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-mon[39960]: [21/Sep/2024:09:56:38] ENGINE Bus STARTING 2024-09-21T09:56:39.167 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-mon[39960]: [21/Sep/2024:09:56:38] ENGINE Serving on https://172.21.15.98:7150 2024-09-21T09:56:39.167 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-mon[39960]: [21/Sep/2024:09:56:38] ENGINE Error in HTTPServer.serve 2024-09-21T09:56:39.167 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-09-21T09:56:39.167 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:56:39.167 INFO:journalctl@ceph.mon.c.smithi098.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:56:39.167 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:56:39.167 INFO:journalctl@ceph.mon.c.smithi098.stdout: self._run(expiration_interval) 2024-09-21T09:56:39.167 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:56:39.168 INFO:journalctl@ceph.mon.c.smithi098.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:56:39.168 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:56:39.168 INFO:journalctl@ceph.mon.c.smithi098.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:56:39.168 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:56:39.168 INFO:journalctl@ceph.mon.c.smithi098.stdout: s = self.context.wrap_socket( 2024-09-21T09:56:39.168 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:56:39.168 INFO:journalctl@ceph.mon.c.smithi098.stdout: return self.sslsocket_class._create( 2024-09-21T09:56:39.168 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:56:39.168 INFO:journalctl@ceph.mon.c.smithi098.stdout: self.do_handshake() 2024-09-21T09:56:39.168 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:56:39.168 INFO:journalctl@ceph.mon.c.smithi098.stdout: self._sslobj.do_handshake() 2024-09-21T09:56:39.168 INFO:journalctl@ceph.mon.c.smithi098.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:56:39.168 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-mon[39960]: [21/Sep/2024:09:56:38] ENGINE Serving on http://172.21.15.98:8765 2024-09-21T09:56:39.168 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:38 smithi098 ceph-mon[39960]: [21/Sep/2024:09:56:38] ENGINE Bus STARTED 2024-09-21T09:56:39.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:38 smithi196 ceph-mon[32670]: mgrmap e36: y(active, since 1.01436s), standbys: x 2024-09-21T09:56:39.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:38 smithi196 ceph-mon[32670]: [21/Sep/2024:09:56:38] ENGINE Bus STARTING 2024-09-21T09:56:39.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:38 smithi196 ceph-mon[32670]: [21/Sep/2024:09:56:38] ENGINE Serving on https://172.21.15.98:7150 2024-09-21T09:56:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:38 smithi196 ceph-mon[32670]: [21/Sep/2024:09:56:38] ENGINE Error in HTTPServer.serve 2024-09-21T09:56:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-21T09:56:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:56:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:56:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:56:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._run(expiration_interval) 2024-09-21T09:56:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:56:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:56:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:56:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:56:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:56:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: s = self.context.wrap_socket( 2024-09-21T09:56:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:56:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.sslsocket_class._create( 2024-09-21T09:56:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:56:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: self.do_handshake() 2024-09-21T09:56:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:56:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._sslobj.do_handshake() 2024-09-21T09:56:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:56:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:38 smithi196 ceph-mon[32670]: [21/Sep/2024:09:56:38] ENGINE Serving on http://172.21.15.98:8765 2024-09-21T09:56:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:38 smithi196 ceph-mon[32670]: [21/Sep/2024:09:56:38] ENGINE Bus STARTED 2024-09-21T09:56:39.412 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:40.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:39 smithi098 ceph-mon[97580]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:40.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:39 smithi098 ceph-mon[97580]: mgrmap e37: y(active, since 2s), standbys: x 2024-09-21T09:56:40.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:39 smithi098 ceph-mon[39960]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:40.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:39 smithi098 ceph-mon[39960]: mgrmap e37: y(active, since 2s), standbys: x 2024-09-21T09:56:40.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:39 smithi196 ceph-mon[32670]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:40.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:39 smithi196 ceph-mon[32670]: mgrmap e37: y(active, since 2s), standbys: x 2024-09-21T09:56:40.528 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:41.010 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:56:40] "GET /metrics HTTP/1.1" 200 31165 "" "Prometheus/2.51.0" 2024-09-21T09:56:41.925 INFO:teuthology.orchestra.run.smithi098.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:56:42.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:41 smithi098 ceph-mon[97580]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:42.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:41 smithi098 ceph-mon[97580]: mgrmap e38: y(active, since 4s), standbys: x 2024-09-21T09:56:42.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:41 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:42.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:41 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:42.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:41 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:42.033 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:41 smithi098 ceph-mon[39960]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:42.033 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:41 smithi098 ceph-mon[39960]: mgrmap e38: y(active, since 4s), standbys: x 2024-09-21T09:56:42.033 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:41 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:42.033 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:41 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:42.033 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:41 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:42.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:41 smithi196 ceph-mon[32670]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:42.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:41 smithi196 ceph-mon[32670]: mgrmap e38: y(active, since 4s), standbys: x 2024-09-21T09:56:42.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:41 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:41 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:41 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:42.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'sleep 60' 2024-09-21T09:56:43.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:43 smithi098 ceph-mon[39960]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:43.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:43 smithi098 ceph-mon[39960]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:56:43.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:43 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:43.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:43 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:43.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:43 smithi098 ceph-mon[97580]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:43.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:43 smithi098 ceph-mon[97580]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:56:43.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:43 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:43.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:43 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:43.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:43 smithi196 ceph-mon[32670]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:43.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:43 smithi196 ceph-mon[32670]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:56:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:43 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:43 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:44.378 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:44 smithi098 ceph-mon[97580]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:44.379 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:44 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:44.379 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:44 smithi098 ceph-mon[39960]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:44.379 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:44 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:44.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:44 smithi196 ceph-mon[32670]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:44.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:44 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:44.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:44 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:44.146Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:45.372 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:45 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:45.372 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:45 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:45.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:45 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.138 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:45.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:46.138 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:45.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:46.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[39960]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T09:56:46.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:56:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[39960]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-21T09:56:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[39960]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-21T09:56:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[39960]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:56:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[39960]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:56:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[39960]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:56:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[39960]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:56:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[97580]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T09:56:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:56:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[97580]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-21T09:56:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[97580]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-21T09:56:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[97580]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:56:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[97580]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:56:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[97580]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:56:46.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[97580]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:56:46.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:46 smithi196 ceph-mon[32670]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T09:56:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:46 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:46 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:46 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:46 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:46 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:46 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:56:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:46 smithi196 ceph-mon[32670]: Updating smithi098:/etc/ceph/ceph.conf 2024-09-21T09:56:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:46 smithi196 ceph-mon[32670]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-21T09:56:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:46 smithi196 ceph-mon[32670]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:56:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:46 smithi196 ceph-mon[32670]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:56:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:46 smithi196 ceph-mon[32670]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:56:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:46 smithi196 ceph-mon[32670]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:56:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:46 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:46 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:46 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:46 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:46 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:46 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:47.381 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:46.947Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:47.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:47 smithi196 ceph-mon[32670]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:56:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:47 smithi196 ceph-mon[32670]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:56:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:47 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:47 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:47 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:47 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:56:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:47 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:47.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:47 smithi098 ceph-mon[39960]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:56:47.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:47 smithi098 ceph-mon[39960]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:56:47.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:47 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:47.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:47 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:47.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:47 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:47.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:47 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:56:47.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:47 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:47.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:47 smithi098 ceph-mon[97580]: Updating smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:56:47.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:47 smithi098 ceph-mon[97580]: Updating smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:56:47.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:47 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:47.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:47 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:47.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:47 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:47.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:47 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:56:47.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:47 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:56:48.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:48 smithi098 ceph-mon[97580]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T09:56:48.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:48 smithi098 ceph-mon[39960]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T09:56:48.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:48 smithi196 ceph-mon[32670]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T09:56:50.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:50 smithi196 ceph-mon[32670]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T09:56:50.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:50.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:50 smithi098 ceph-mon[97580]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T09:56:50.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:50 smithi098 ceph-mon[39960]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T09:56:51.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:56:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:56:50] "GET /metrics HTTP/1.1" 200 31165 "" "Prometheus/2.51.0" 2024-09-21T09:56:52.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:52 smithi098 ceph-mon[97580]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T09:56:52.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:52.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:52 smithi098 ceph-mon[39960]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T09:56:52.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:52 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:52.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:52 smithi196 ceph-mon[32670]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T09:56:52.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:52 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:54.409 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:54 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:54.146Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:54.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:54 smithi098 ceph-mon[97580]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T09:56:54.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:54 smithi098 ceph-mon[39960]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T09:56:54.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:54 smithi196 ceph-mon[32670]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T09:56:56.138 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:55 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:55.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:56.138 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:55 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:55.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:56.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:56 smithi098 ceph-mon[97580]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T09:56:56.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:56 smithi098 ceph-mon[39960]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T09:56:56.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:56 smithi196 ceph-mon[32670]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T09:56:57.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:56:56 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:56:56.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:58.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:56:58 smithi098 ceph-mon[97580]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:58.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:56:58 smithi098 ceph-mon[39960]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:56:58 smithi196 ceph-mon[32670]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:00.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:00 smithi196 ceph-mon[32670]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:00.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:00.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:00 smithi098 ceph-mon[97580]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:00.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:57:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:57:00] "GET /metrics HTTP/1.1" 200 33853 "" "Prometheus/2.51.0" 2024-09-21T09:57:00.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:00 smithi098 ceph-mon[39960]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:02.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:02 smithi098 ceph-mon[97580]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:02.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:02 smithi098 ceph-mon[39960]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:02.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:02 smithi196 ceph-mon[32670]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:04.423 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:04 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:04.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:04.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:04 smithi098 ceph-mon[97580]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:04.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:04 smithi098 ceph-mon[39960]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:04.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:04 smithi196 ceph-mon[32670]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:06.138 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:05 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:06.138 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:05 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:05.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:06.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:06 smithi098 ceph-mon[97580]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:06.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:06 smithi098 ceph-mon[39960]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:06.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:06 smithi196 ceph-mon[32670]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:07.153 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:06 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:06.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:07.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:07 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:07.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:07.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:07 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:08.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:08 smithi098 ceph-mon[39960]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:08.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:08 smithi098 ceph-mon[97580]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:08 smithi196 ceph-mon[32670]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:10 smithi196 ceph-mon[32670]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:10.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:10.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:10 smithi098 ceph-mon[97580]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:10.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:57:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:57:10] "GET /metrics HTTP/1.1" 200 33852 "" "Prometheus/2.51.0" 2024-09-21T09:57:10.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:10 smithi098 ceph-mon[39960]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:12.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:12 smithi098 ceph-mon[97580]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:12.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:12 smithi098 ceph-mon[39960]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:12 smithi196 ceph-mon[32670]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:14.444 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:14 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:14.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:14.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:14 smithi098 ceph-mon[97580]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:14.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:14 smithi098 ceph-mon[39960]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:14 smithi196 ceph-mon[32670]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:16.138 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:15 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:15.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:16.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:15 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:15.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:16 smithi098 ceph-mon[97580]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:16.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:16 smithi098 ceph-mon[39960]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:16 smithi196 ceph-mon[32670]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:17.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:16 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:16.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:18 smithi098 ceph-mon[97580]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:18.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:18 smithi098 ceph-mon[39960]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:18.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:18 smithi196 ceph-mon[32670]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:20.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:20 smithi196 ceph-mon[32670]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:20.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:20.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:20 smithi098 ceph-mon[97580]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:20.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:57:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:57:20] "GET /metrics HTTP/1.1" 200 33852 "" "Prometheus/2.51.0" 2024-09-21T09:57:20.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:20 smithi098 ceph-mon[39960]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:22 smithi098 ceph-mon[97580]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:22.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:22 smithi098 ceph-mon[39960]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:22.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:22 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:22 smithi196 ceph-mon[32670]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:22 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:24.459 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:24 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:24.146Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:24 smithi098 ceph-mon[97580]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:24.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:24 smithi098 ceph-mon[39960]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:24 smithi196 ceph-mon[32670]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:26.138 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:25 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:25.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:26.138 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:25 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:25.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:26.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:26 smithi098 ceph-mon[97580]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:26.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:26 smithi098 ceph-mon[39960]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:26 smithi196 ceph-mon[32670]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:27.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:26 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:26.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:28.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:28 smithi098 ceph-mon[97580]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:28.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:28 smithi098 ceph-mon[39960]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:28 smithi196 ceph-mon[32670]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:30.467 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:30.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:30 smithi098 ceph-mon[97580]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:30.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:57:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:57:30] "GET /metrics HTTP/1.1" 200 33851 "" "Prometheus/2.51.0" 2024-09-21T09:57:30.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:30 smithi098 ceph-mon[39960]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:30.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:30 smithi196 ceph-mon[32670]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:32.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:32 smithi098 ceph-mon[97580]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:32.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:32 smithi098 ceph-mon[39960]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:32.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:32 smithi196 ceph-mon[32670]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:34.473 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:34 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:34.146Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:34.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:34 smithi098 ceph-mon[97580]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:34.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:34 smithi098 ceph-mon[39960]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:34 smithi196 ceph-mon[32670]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:36.138 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:35.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:36.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:35.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:36.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:36 smithi098 ceph-mon[97580]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:36.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:36 smithi098 ceph-mon[39960]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:36.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:36 smithi196 ceph-mon[32670]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:37.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:36 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:36.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:37.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:37.851 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:37 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:37.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:37 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:38.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:38 smithi098 ceph-mon[97580]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:38.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:38 smithi098 ceph-mon[39960]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:38.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:38 smithi196 ceph-mon[32670]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:40.488 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:40.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:40 smithi098 ceph-mon[97580]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:40.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:57:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:57:40] "GET /metrics HTTP/1.1" 200 33850 "" "Prometheus/2.51.0" 2024-09-21T09:57:40.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:40 smithi098 ceph-mon[39960]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:40.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:40 smithi196 ceph-mon[32670]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:42.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:42 smithi098 ceph-mon[97580]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:42.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:42 smithi098 ceph-mon[39960]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:42.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:42 smithi196 ceph-mon[32670]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:44.492 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:44 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:44.146Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:44.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:44 smithi098 ceph-mon[97580]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:44.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:44 smithi098 ceph-mon[39960]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:44.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:44 smithi196 ceph-mon[32670]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:46.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:45.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:46.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:45.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:46.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade resume' 2024-09-21T09:57:46.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:46 smithi098 ceph-mon[97580]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:46.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:46 smithi098 ceph-mon[39960]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:46 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:46.625 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:46.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:46 smithi196 ceph-mon[32670]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:46 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:47.203 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:46.947Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:48.378 INFO:teuthology.orchestra.run.smithi098.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:57:48.767 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:48 smithi098 ceph-mon[97580]: pgmap v38: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:48.767 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:48 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:57:48.767 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:48 smithi098 ceph-mon[39960]: pgmap v38: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:48.768 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:48 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:57:48.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:48 smithi196 ceph-mon[32670]: pgmap v38: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:48.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:48 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:57:49.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'echo wait for majority of mons to upgrade' 2024-09-21T09:57:49.690 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:49 smithi098 ceph-mon[97580]: from='client.26546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:49.690 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:49 smithi098 ceph-mon[97580]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:57:49.690 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:49 smithi098 ceph-mon[39960]: from='client.26546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:49.691 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:49 smithi098 ceph-mon[39960]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:57:49.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:49 smithi196 ceph-mon[32670]: from='client.26546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:49.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:49 smithi196 ceph-mon[32670]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:57:50.459 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:50.504 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:50.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:50 smithi098 ceph-mon[97580]: pgmap v39: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:50.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:50 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:57:50.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:50 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:57:50.838 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:57:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:57:50] "GET /metrics HTTP/1.1" 200 33850 "" "Prometheus/2.51.0" 2024-09-21T09:57:50.839 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:50 smithi098 ceph-mon[39960]: pgmap v39: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:50.839 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:50 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:57:50.839 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:50 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:57:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:50 smithi196 ceph-mon[32670]: pgmap v39: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:50 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:57:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:50 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:57:52.171 INFO:teuthology.orchestra.run.smithi098.stdout:wait for majority of mons to upgrade 2024-09-21T09:57:52.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:52 smithi098 ceph-mon[97580]: pgmap v40: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:52.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:52.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:52 smithi098 ceph-mon[39960]: pgmap v40: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:52.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:52 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:52.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:52 smithi196 ceph-mon[32670]: pgmap v40: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:52 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:53.069 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'while ! ceph mon versions | grep $sha1 | egrep '"'"': [23]'"'"' && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-21T09:57:53.753 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:54.510 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:54 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:54.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:54.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:54 smithi098 ceph-mon[97580]: pgmap v41: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:54.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:54 smithi098 ceph-mon[39960]: pgmap v41: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:54.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:54 smithi196 ceph-mon[32670]: pgmap v41: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:56.120 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:55 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:55.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:56.120 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:55 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:55.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:56.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:56 smithi098 ceph-mon[97580]: pgmap v42: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:56.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:56 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:57:56.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:56 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:57:56.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:56 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:56.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:56 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:57:56.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:56 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:57:56.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:56 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:56.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:56 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:57:56.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:56 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T09:57:56.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:56 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-21T09:57:56.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:56 smithi098 ceph-mon[97580]: Upgrade: It appears safe to stop mon.c 2024-09-21T09:57:56.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:56 smithi098 ceph-mon[39960]: pgmap v42: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:56.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:56 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:57:56.327 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:56 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:57:56.327 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:56 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:56.327 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:56 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:57:56.327 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:56 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:57:56.327 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:56 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:56.327 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:56 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:57:56.327 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:56 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T09:57:56.327 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:56 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-21T09:57:56.327 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:56 smithi098 ceph-mon[39960]: Upgrade: It appears safe to stop mon.c 2024-09-21T09:57:56.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:56 smithi196 ceph-mon[32670]: pgmap v42: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:56.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:56 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:57:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:56 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:57:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:56 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:56 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:57:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:56 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:57:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:56 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:56 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:57:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:56 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T09:57:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:56 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-21T09:57:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:56 smithi196 ceph-mon[32670]: Upgrade: It appears safe to stop mon.c 2024-09-21T09:57:56.630 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:57:57.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:57 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/2478054258' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:57:57.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:57 smithi196 ceph-mon[32670]: from='client.26558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:57.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:57:56 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:57:56.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:57.586 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:57 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2478054258' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:57:57.586 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:57 smithi098 ceph-mon[97580]: from='client.26558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:57.586 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:57 smithi098 ceph-mon[39960]: from='client.? 172.21.15.98:0/2478054258' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:57:57.586 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:57 smithi098 ceph-mon[39960]: from='client.26558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:58.553 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:58 smithi098 ceph-mon[97580]: from='client.34182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:58.553 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:58 smithi098 ceph-mon[97580]: pgmap v43: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:58.553 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:58 smithi098 ceph-mon[97580]: Upgrade: Updating mon.c 2024-09-21T09:57:58.553 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:58 smithi098 ceph-mon[97580]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:57:58.553 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:58 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:57:58.553 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:58 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:57:58.553 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:58 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:58.553 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:57:58 smithi098 ceph-mon[97580]: Deploying daemon mon.c on smithi098 2024-09-21T09:57:58.554 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:58 smithi098 ceph-mon[39960]: from='client.34182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:58.554 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:58 smithi098 ceph-mon[39960]: pgmap v43: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:58.554 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:58 smithi098 ceph-mon[39960]: Upgrade: Updating mon.c 2024-09-21T09:57:58.554 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:58 smithi098 ceph-mon[39960]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:57:58.554 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:58 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:57:58.554 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:58 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:57:58.554 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:58 smithi098 ceph-mon[39960]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:58.554 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:58 smithi098 ceph-mon[39960]: Deploying daemon mon.c on smithi098 2024-09-21T09:57:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:58 smithi196 ceph-mon[32670]: from='client.34182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:58 smithi196 ceph-mon[32670]: pgmap v43: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:58 smithi196 ceph-mon[32670]: Upgrade: Updating mon.c 2024-09-21T09:57:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:58 smithi196 ceph-mon[32670]: from='mgr.26492 ' entity='mgr.y' 2024-09-21T09:57:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:58 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:57:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:58 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:57:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:58 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:57:58 smithi196 ceph-mon[32670]: Deploying daemon mon.c on smithi098 2024-09-21T09:57:59.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:58 smithi098 systemd[1]: Stopping Ceph mon.c for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:57:59.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-c[39956]: 2024-09-21T09:57:59.088+0000 7ff320a67640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T09:57:59.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-c[39956]: 2024-09-21T09:57:59.088+0000 7ff320a67640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-21T09:57:59.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:59 smithi098 podman[100671]: 2024-09-21 09:57:59.293489799 +0000 UTC m=+0.316266795 container died 54e9dc04184896a78fac2c92458f6a8f3f44f685c6e8183c8cf77dab2abf58a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True) 2024-09-21T09:57:59.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:57:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:09:57:59] ENGINE Bus STOPPING 2024-09-21T09:57:59.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:57:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:09:57:59] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:57:59.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:57:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:09:57:59] ENGINE Bus STOPPED 2024-09-21T09:57:59.347 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:57:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:09:57:59] ENGINE Bus STARTING 2024-09-21T09:57:59.348 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:57:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:09:57:59] ENGINE Serving on http://:::9283 2024-09-21T09:57:59.348 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:57:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:09:57:59] ENGINE Bus STARTED 2024-09-21T09:57:59.579 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:57:59.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:59 smithi098 podman[100671]: 2024-09-21 09:57:59.68171918 +0000 UTC m=+0.704496180 container cleanup 54e9dc04184896a78fac2c92458f6a8f3f44f685c6e8183c8cf77dab2abf58a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-c, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-21T09:57:59.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:59 smithi098 bash[100671]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-c 2024-09-21T09:58:00.334 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:57:59 smithi098 podman[100718]: 2024-09-21 09:57:59.950003779 +0000 UTC m=+0.650178586 container remove 54e9dc04184896a78fac2c92458f6a8f3f44f685c6e8183c8cf77dab2abf58a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-c, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-fbd1a6a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-21T09:58:00.335 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@mon.c.service: Deactivated successfully. 2024-09-21T09:58:00.335 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 systemd[1]: Stopped Ceph mon.c for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:58:00.335 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@mon.c.service: Consumed 7.612s CPU time. 2024-09-21T09:58:00.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 systemd[1]: Starting Ceph mon.c for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:58:00.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 podman[100895]: 2024-09-21 09:58:00.442092061 +0000 UTC m=+0.015973268 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:58:00.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 podman[100895]: 2024-09-21 09:58:00.551507716 +0000 UTC m=+0.125388927 container create ef4452ba8cf09dd0181ce0bcb93eea8c284f12f160a55987aa8a7b3826c8dd5d (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-c, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T09:58:00.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:00.922 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:58:00] "GET /metrics HTTP/1.1" 200 33846 "" "Prometheus/2.51.0" 2024-09-21T09:58:00.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 podman[100895]: 2024-09-21 09:58:00.821080115 +0000 UTC m=+0.394961327 container init ef4452ba8cf09dd0181ce0bcb93eea8c284f12f160a55987aa8a7b3826c8dd5d (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-c, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T09:58:00.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 podman[100895]: 2024-09-21 09:58:00.82457837 +0000 UTC m=+0.398459588 container start ef4452ba8cf09dd0181ce0bcb93eea8c284f12f160a55987aa8a7b3826c8dd5d (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-c, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=main-62da54b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-21T09:58:00.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T09:58:00.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev), process ceph-mon, pid 2 2024-09-21T09:58:00.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: pidfile_write: ignore empty --pid-file 2024-09-21T09:58:00.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: load: jerasure load: lrc 2024-09-21T09:58:00.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: RocksDB version: 7.9.2 2024-09-21T09:58:00.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Git sha 0 2024-09-21T09:58:00.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Compile date 2024-09-20 20:32:28 2024-09-21T09:58:00.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: DB SUMMARY 2024-09-21T09:58:00.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: DB Session ID: G0320HM9A9LPLW78J649 2024-09-21T09:58:00.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: CURRENT file: CURRENT 2024-09-21T09:58:00.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T09:58:00.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1083 Bytes 2024-09-21T09:58:00.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-09-21T09:58:00.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 1605417 ; 2024-09-21T09:58:00.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.error_if_exists: 0 2024-09-21T09:58:00.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.create_if_missing: 0 2024-09-21T09:58:00.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.paranoid_checks: 1 2024-09-21T09:58:00.925 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T09:58:00.925 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T09:58:00.925 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T09:58:00.925 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.env: 0x5601b5777120 2024-09-21T09:58:00.925 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T09:58:00.925 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.info_log: 0x5601b754b700 2024-09-21T09:58:00.925 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T09:58:00.926 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.statistics: (nil) 2024-09-21T09:58:00.926 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.use_fsync: 0 2024-09-21T09:58:00.926 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_log_file_size: 0 2024-09-21T09:58:00.926 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T09:58:00.926 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T09:58:00.927 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T09:58:00.927 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T09:58:00.927 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.allow_fallocate: 1 2024-09-21T09:58:00.927 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T09:58:00.927 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T09:58:00.927 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.use_direct_reads: 0 2024-09-21T09:58:00.927 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T09:58:00.927 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T09:58:00.927 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.db_log_dir: 2024-09-21T09:58:00.927 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.wal_dir: 2024-09-21T09:58:00.927 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T09:58:00.927 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T09:58:00.928 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T09:58:00.928 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T09:58:00.928 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T09:58:00.928 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T09:58:00.928 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T09:58:00.928 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T09:58:00.928 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.write_buffer_manager: 0x5601b754e1e0 2024-09-21T09:58:00.928 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T09:58:00.928 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T09:58:00.928 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T09:58:00.928 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T09:58:00.928 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T09:58:00.929 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T09:58:00.929 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T09:58:00.929 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T09:58:00.929 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.unordered_write: 0 2024-09-21T09:58:00.929 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T09:58:00.929 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T09:58:00.929 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T09:58:00.929 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T09:58:00.929 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.row_cache: None 2024-09-21T09:58:00.929 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.wal_filter: None 2024-09-21T09:58:00.929 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T09:58:00.929 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T09:58:00.929 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.two_write_queues: 0 2024-09-21T09:58:00.929 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T09:58:00.930 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.wal_compression: 0 2024-09-21T09:58:00.930 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.atomic_flush: 0 2024-09-21T09:58:00.930 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T09:58:00.930 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T09:58:00.930 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T09:58:00.930 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.log_readahead_size: 0 2024-09-21T09:58:00.930 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T09:58:00.930 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T09:58:00.930 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T09:58:00.930 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T09:58:00.930 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T09:58:00.930 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T09:58:00.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T09:58:00.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_background_jobs: 2 2024-09-21T09:58:00.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_background_compactions: -1 2024-09-21T09:58:00.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_subcompactions: 1 2024-09-21T09:58:00.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T09:58:00.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T09:58:00.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T09:58:00.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T09:58:00.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T09:58:00.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T09:58:00.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T09:58:00.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T09:58:00.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_open_files: -1 2024-09-21T09:58:00.932 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T09:58:00.932 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T09:58:00.932 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T09:58:00.932 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T09:58:00.932 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_background_flushes: -1 2024-09-21T09:58:00.932 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Compression algorithms supported: 2024-09-21T09:58:00.932 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: kZSTD supported: 0 2024-09-21T09:58:00.932 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: kXpressCompression supported: 0 2024-09-21T09:58:00.932 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: kBZip2Compression supported: 0 2024-09-21T09:58:00.932 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T09:58:00.932 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: kLZ4Compression supported: 1 2024-09-21T09:58:00.932 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: kZlibCompression supported: 1 2024-09-21T09:58:00.933 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T09:58:00.933 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: kSnappyCompression supported: 1 2024-09-21T09:58:00.933 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T09:58:00.933 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T09:58:00.933 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-09-21T09:58:00.933 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T09:58:00.933 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T09:58:00.933 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.merge_operator: 2024-09-21T09:58:00.933 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compaction_filter: None 2024-09-21T09:58:00.933 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T09:58:00.933 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T09:58:00.934 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T09:58:00.934 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T09:58:00.934 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5601b754b320) 2024-09-21T09:58:00.934 INFO:journalctl@ceph.mon.c.smithi098.stdout: cache_index_and_filter_blocks: 1 2024-09-21T09:58:00.934 INFO:journalctl@ceph.mon.c.smithi098.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T09:58:00.934 INFO:journalctl@ceph.mon.c.smithi098.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T09:58:00.934 INFO:journalctl@ceph.mon.c.smithi098.stdout: pin_top_level_index_and_filter: 1 2024-09-21T09:58:00.934 INFO:journalctl@ceph.mon.c.smithi098.stdout: index_type: 0 2024-09-21T09:58:00.934 INFO:journalctl@ceph.mon.c.smithi098.stdout: data_block_index_type: 0 2024-09-21T09:58:00.934 INFO:journalctl@ceph.mon.c.smithi098.stdout: index_shortening: 1 2024-09-21T09:58:00.934 INFO:journalctl@ceph.mon.c.smithi098.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T09:58:00.934 INFO:journalctl@ceph.mon.c.smithi098.stdout: checksum: 4 2024-09-21T09:58:00.934 INFO:journalctl@ceph.mon.c.smithi098.stdout: no_block_cache: 0 2024-09-21T09:58:00.935 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_cache: 0x5601b753f610 2024-09-21T09:58:00.935 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_cache_name: BinnedLRUCache 2024-09-21T09:58:00.935 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_cache_options: 2024-09-21T09:58:00.935 INFO:journalctl@ceph.mon.c.smithi098.stdout: capacity : 536870912 2024-09-21T09:58:00.935 INFO:journalctl@ceph.mon.c.smithi098.stdout: num_shard_bits : 4 2024-09-21T09:58:00.935 INFO:journalctl@ceph.mon.c.smithi098.stdout: strict_capacity_limit : 0 2024-09-21T09:58:00.935 INFO:journalctl@ceph.mon.c.smithi098.stdout: high_pri_pool_ratio: 0.000 2024-09-21T09:58:00.935 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_cache_compressed: (nil) 2024-09-21T09:58:00.935 INFO:journalctl@ceph.mon.c.smithi098.stdout: persistent_cache: (nil) 2024-09-21T09:58:00.935 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_size: 4096 2024-09-21T09:58:00.935 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_size_deviation: 10 2024-09-21T09:58:00.935 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_restart_interval: 16 2024-09-21T09:58:00.937 INFO:journalctl@ceph.mon.c.smithi098.stdout: index_block_restart_interval: 1 2024-09-21T09:58:00.937 INFO:journalctl@ceph.mon.c.smithi098.stdout: metadata_block_size: 4096 2024-09-21T09:58:00.937 INFO:journalctl@ceph.mon.c.smithi098.stdout: partition_filters: 0 2024-09-21T09:58:00.937 INFO:journalctl@ceph.mon.c.smithi098.stdout: use_delta_encoding: 1 2024-09-21T09:58:00.938 INFO:journalctl@ceph.mon.c.smithi098.stdout: filter_policy: bloomfilter 2024-09-21T09:58:00.938 INFO:journalctl@ceph.mon.c.smithi098.stdout: whole_key_filtering: 1 2024-09-21T09:58:00.938 INFO:journalctl@ceph.mon.c.smithi098.stdout: verify_compression: 0 2024-09-21T09:58:00.938 INFO:journalctl@ceph.mon.c.smithi098.stdout: read_amp_bytes_per_bit: 0 2024-09-21T09:58:00.938 INFO:journalctl@ceph.mon.c.smithi098.stdout: format_version: 5 2024-09-21T09:58:00.938 INFO:journalctl@ceph.mon.c.smithi098.stdout: enable_index_compression: 1 2024-09-21T09:58:00.938 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_align: 0 2024-09-21T09:58:00.938 INFO:journalctl@ceph.mon.c.smithi098.stdout: max_auto_readahead_size: 262144 2024-09-21T09:58:00.938 INFO:journalctl@ceph.mon.c.smithi098.stdout: prepopulate_block_cache: 0 2024-09-21T09:58:00.938 INFO:journalctl@ceph.mon.c.smithi098.stdout: initial_auto_readahead_size: 8192 2024-09-21T09:58:00.938 INFO:journalctl@ceph.mon.c.smithi098.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T09:58:00.938 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T09:58:00.938 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T09:58:00.938 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compression: NoCompression 2024-09-21T09:58:00.939 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T09:58:00.939 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T09:58:00.939 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T09:58:00.939 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.num_levels: 7 2024-09-21T09:58:00.939 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T09:58:00.939 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T09:58:00.939 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T09:58:00.939 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T09:58:00.939 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T09:58:00.939 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T09:58:00.939 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T09:58:00.939 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:58:00.939 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T09:58:00.940 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T09:58:00.940 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T09:58:00.940 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T09:58:00.940 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T09:58:00.940 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T09:58:00.940 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T09:58:00.940 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T09:58:00.940 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:58:00.940 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T09:58:00.940 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T09:58:00.940 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T09:58:00.940 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T09:58:00.940 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T09:58:00.941 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T09:58:00.941 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T09:58:00.941 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T09:58:00.941 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T09:58:00.941 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T09:58:00.941 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T09:58:00.941 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T09:58:00.941 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T09:58:00.941 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T09:58:00.941 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T09:58:00.941 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T09:58:00.941 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T09:58:00.942 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T09:58:00.942 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T09:58:00.942 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T09:58:00.942 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T09:58:00.942 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T09:58:00.942 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T09:58:00.942 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T09:58:00.942 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T09:58:00.942 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T09:58:00.942 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T09:58:00.942 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T09:58:00.942 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T09:58:00.942 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T09:58:00.943 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T09:58:00.943 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T09:58:00.943 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T09:58:00.943 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T09:58:00.943 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T09:58:00.943 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T09:58:00.943 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T09:58:00.943 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.inplace_update_support: 0 2024-09-21T09:58:00.943 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T09:58:00.943 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T09:58:00.943 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T09:58:00.943 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T09:58:00.944 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.bloom_locality: 0 2024-09-21T09:58:00.944 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.max_successive_merges: 0 2024-09-21T09:58:00.944 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T09:58:00.944 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T09:58:00.944 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T09:58:00.944 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T09:58:00.944 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.ttl: 2592000 2024-09-21T09:58:00.944 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T09:58:00.944 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T09:58:00.944 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T09:58:00.944 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.enable_blob_files: false 2024-09-21T09:58:00.944 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.min_blob_size: 0 2024-09-21T09:58:00.944 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T09:58:00.944 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T09:58:00.945 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T09:58:00.945 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T09:58:00.945 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T09:58:00.945 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T09:58:00.945 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T09:58:00.945 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T09:58:00.945 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-09-21T09:58:00.945 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T09:58:00.945 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 13046, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T09:58:00.945 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-09-21T09:58:00.945 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 1beb1b77-dbd2-426b-9c50-5de474484cfb 2024-09-21T09:58:00.945 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912680851276, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-09-21T09:58:00.945 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-09-21T09:58:00.946 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912680855921, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 955353, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 13043, "largest_seqno": 13990, "table_properties": {"data_size": 950761, "index_size": 2656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 901, "raw_key_size": 7548, "raw_average_key_size": 21, "raw_value_size": 942969, "raw_average_value_size": 2717, "num_data_blocks": 122, "num_entries": 347, "num_filter_entries": 347, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726912680, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "1beb1b77-dbd2-426b-9c50-5de474484cfb", "db_session_id": "G0320HM9A9LPLW78J649", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-09-21T09:58:00.946 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912680856040, "job": 1, "event": "recovery_finished"} 2024-09-21T09:58:00.947 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-09-21T09:58:00.947 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T09:58:00.947 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T09:58:00.947 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5601b756ee00 2024-09-21T09:58:00.947 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: DB pointer 0x5601b7670000 2024-09-21T09:58:00.947 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T09:58:00.947 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T09:58:00.947 INFO:journalctl@ceph.mon.c.smithi098.stdout: ** DB Stats ** 2024-09-21T09:58:00.947 INFO:journalctl@ceph.mon.c.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:58:00.947 INFO:journalctl@ceph.mon.c.smithi098.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T09:58:00.947 INFO:journalctl@ceph.mon.c.smithi098.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:58:00.947 INFO:journalctl@ceph.mon.c.smithi098.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:58:00.948 INFO:journalctl@ceph.mon.c.smithi098.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T09:58:00.948 INFO:journalctl@ceph.mon.c.smithi098.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:58:00.948 INFO:journalctl@ceph.mon.c.smithi098.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:58:00.948 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-09-21T09:58:00.948 INFO:journalctl@ceph.mon.c.smithi098.stdout: ** Compaction Stats [default] ** 2024-09-21T09:58:00.948 INFO:journalctl@ceph.mon.c.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T09:58:00.948 INFO:journalctl@ceph.mon.c.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T09:58:00.948 INFO:journalctl@ceph.mon.c.smithi098.stdout: L0 1/0 932.96 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 392.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-21T09:58:00.948 INFO:journalctl@ceph.mon.c.smithi098.stdout: L6 1/0 8.90 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-21T09:58:00.948 INFO:journalctl@ceph.mon.c.smithi098.stdout: Sum 2/0 9.81 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 392.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-21T09:58:00.948 INFO:journalctl@ceph.mon.c.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 392.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-21T09:58:00.948 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-09-21T09:58:00.948 INFO:journalctl@ceph.mon.c.smithi098.stdout: ** Compaction Stats [default] ** 2024-09-21T09:58:00.949 INFO:journalctl@ceph.mon.c.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T09:58:00.949 INFO:journalctl@ceph.mon.c.smithi098.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:58:00.949 INFO:journalctl@ceph.mon.c.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 392.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-21T09:58:00.949 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-09-21T09:58:00.949 INFO:journalctl@ceph.mon.c.smithi098.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T09:58:00.949 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-09-21T09:58:00.949 INFO:journalctl@ceph.mon.c.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:58:00.949 INFO:journalctl@ceph.mon.c.smithi098.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-09-21T09:58:00.949 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:58:00.949 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:58:00.949 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:58:00.949 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:58:00.949 INFO:journalctl@ceph.mon.c.smithi098.stdout: Cumulative compaction: 0.00 GB write, 124.09 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:58:00.949 INFO:journalctl@ceph.mon.c.smithi098.stdout: Interval compaction: 0.00 GB write, 124.09 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:58:00.950 INFO:journalctl@ceph.mon.c.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:58:00.950 INFO:journalctl@ceph.mon.c.smithi098.stdout: Block cache BinnedLRUCache@0x5601b753f610#2 capacity: 512.00 MB usage: 3.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-09-21T09:58:00.950 INFO:journalctl@ceph.mon.c.smithi098.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.03 KB,0.000196695%) IndexBlock(1,2.83 KB,0.000539422%) Misc(1,0.00 KB,0%) 2024-09-21T09:58:00.950 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-09-21T09:58:00.950 INFO:journalctl@ceph.mon.c.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:58:00.950 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: starting mon.c rank 2 at public addrs [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] at bind addrs [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:58:00.950 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: mon.c@-1(???) e3 preinit fsid 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:58:00.950 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: mon.c@-1(???).mds e9 new map 2024-09-21T09:58:00.950 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: mon.c@-1(???).mds e9 print_map 2024-09-21T09:58:00.950 INFO:journalctl@ceph.mon.c.smithi098.stdout: e9 2024-09-21T09:58:00.950 INFO:journalctl@ceph.mon.c.smithi098.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-21T09:58:00.950 INFO:journalctl@ceph.mon.c.smithi098.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T09:58:00.950 INFO:journalctl@ceph.mon.c.smithi098.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:58:00.951 INFO:journalctl@ceph.mon.c.smithi098.stdout: legacy client fscid: 1 2024-09-21T09:58:00.951 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-09-21T09:58:00.951 INFO:journalctl@ceph.mon.c.smithi098.stdout: Filesystem 'foo' (1) 2024-09-21T09:58:00.951 INFO:journalctl@ceph.mon.c.smithi098.stdout: fs_name foo 2024-09-21T09:58:00.951 INFO:journalctl@ceph.mon.c.smithi098.stdout: epoch 8 2024-09-21T09:58:00.951 INFO:journalctl@ceph.mon.c.smithi098.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-21T09:58:00.951 INFO:journalctl@ceph.mon.c.smithi098.stdout: created 2024-09-21T09:52:20.376206+0000 2024-09-21T09:58:00.951 INFO:journalctl@ceph.mon.c.smithi098.stdout: modified 2024-09-21T09:52:32.894452+0000 2024-09-21T09:58:00.951 INFO:journalctl@ceph.mon.c.smithi098.stdout: tableserver 0 2024-09-21T09:58:00.951 INFO:journalctl@ceph.mon.c.smithi098.stdout: root 0 2024-09-21T09:58:00.951 INFO:journalctl@ceph.mon.c.smithi098.stdout: session_timeout 60 2024-09-21T09:58:00.951 INFO:journalctl@ceph.mon.c.smithi098.stdout: session_autoclose 300 2024-09-21T09:58:00.951 INFO:journalctl@ceph.mon.c.smithi098.stdout: max_file_size 1099511627776 2024-09-21T09:58:00.951 INFO:journalctl@ceph.mon.c.smithi098.stdout: max_xattr_size 65536 2024-09-21T09:58:00.952 INFO:journalctl@ceph.mon.c.smithi098.stdout: required_client_features {} 2024-09-21T09:58:00.952 INFO:journalctl@ceph.mon.c.smithi098.stdout: last_failure 0 2024-09-21T09:58:00.952 INFO:journalctl@ceph.mon.c.smithi098.stdout: last_failure_osd_epoch 0 2024-09-21T09:58:00.952 INFO:journalctl@ceph.mon.c.smithi098.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:58:00.952 INFO:journalctl@ceph.mon.c.smithi098.stdout: max_mds 1 2024-09-21T09:58:00.952 INFO:journalctl@ceph.mon.c.smithi098.stdout: in 0 2024-09-21T09:58:00.952 INFO:journalctl@ceph.mon.c.smithi098.stdout: up {0=14640} 2024-09-21T09:58:00.952 INFO:journalctl@ceph.mon.c.smithi098.stdout: failed 2024-09-21T09:58:00.952 INFO:journalctl@ceph.mon.c.smithi098.stdout: damaged 2024-09-21T09:58:00.952 INFO:journalctl@ceph.mon.c.smithi098.stdout: stopped 2024-09-21T09:58:00.952 INFO:journalctl@ceph.mon.c.smithi098.stdout: data_pools [4] 2024-09-21T09:58:00.952 INFO:journalctl@ceph.mon.c.smithi098.stdout: metadata_pool 3 2024-09-21T09:58:00.952 INFO:journalctl@ceph.mon.c.smithi098.stdout: inline_data disabled 2024-09-21T09:58:00.952 INFO:journalctl@ceph.mon.c.smithi098.stdout: balancer 2024-09-21T09:58:00.953 INFO:journalctl@ceph.mon.c.smithi098.stdout: bal_rank_mask -1 2024-09-21T09:58:00.953 INFO:journalctl@ceph.mon.c.smithi098.stdout: standby_count_wanted 1 2024-09-21T09:58:00.953 INFO:journalctl@ceph.mon.c.smithi098.stdout: qdb_cluster leader: 0 members: 2024-09-21T09:58:00.953 INFO:journalctl@ceph.mon.c.smithi098.stdout: [mds.foo.smithi098.stcwsu{0:14640} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.98:6834/2705511271,v1:172.21.15.98:6835/2705511271] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T09:58:00.953 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-09-21T09:58:00.953 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-09-21T09:58:00.953 INFO:journalctl@ceph.mon.c.smithi098.stdout: Standby daemons: 2024-09-21T09:58:00.953 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-09-21T09:58:00.953 INFO:journalctl@ceph.mon.c.smithi098.stdout: [mds.foo.smithi196.ugblah{-1:24481} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.196:6832/3677035285,v1:172.21.15.196:6833/3677035285] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T09:58:00.953 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: mon.c@-1(???).osd e82 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-21T09:58:00.953 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: mon.c@-1(???).osd e82 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T09:58:00.953 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: mon.c@-1(???).osd e82 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T09:58:00.953 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: mon.c@-1(???).osd e82 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T09:58:00.953 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 ceph-mon[100914]: mon.c@-1(???).paxosservice(auth 1..20) refresh upgraded, format 0 -> 3 2024-09-21T09:58:00.954 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 bash[100895]: ef4452ba8cf09dd0181ce0bcb93eea8c284f12f160a55987aa8a7b3826c8dd5d 2024-09-21T09:58:00.954 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:00 smithi098 systemd[1]: Started Ceph mon.c for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:58:04.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:04 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:04.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:06.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:05 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:06.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:05 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:05.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:06.961 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T09:58:07.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[100914]: mon.c calling monitor election 2024-09-21T09:58:07.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[100914]: mon.b calling monitor election 2024-09-21T09:58:07.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[100914]: mon.a calling monitor election 2024-09-21T09:58:07.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[100914]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:58:07.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[100914]: monmap epoch 3 2024-09-21T09:58:07.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[100914]: fsid 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:58:07.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[100914]: last_changed 2024-09-21T09:47:06.778233+0000 2024-09-21T09:58:07.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[100914]: created 2024-09-21T09:45:30.814529+0000 2024-09-21T09:58:07.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[100914]: min_mon_release 17 (quincy) 2024-09-21T09:58:07.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[100914]: election_strategy: 1 2024-09-21T09:58:07.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[100914]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-21T09:58:07.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[100914]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-21T09:58:07.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[100914]: 2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.c 2024-09-21T09:58:07.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[100914]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 1 up:standby 2024-09-21T09:58:07.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[100914]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T09:58:07.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[100914]: mgrmap e38: y(active, since 90s), standbys: x 2024-09-21T09:58:07.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[100914]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-21T09:58:07.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T09:58:07.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[100914]: overall HEALTH_OK 2024-09-21T09:58:07.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[97580]: mon.c calling monitor election 2024-09-21T09:58:07.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[97580]: mon.b calling monitor election 2024-09-21T09:58:07.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[97580]: mon.a calling monitor election 2024-09-21T09:58:07.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[97580]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:58:07.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[97580]: monmap epoch 3 2024-09-21T09:58:07.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[97580]: fsid 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:58:07.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[97580]: last_changed 2024-09-21T09:47:06.778233+0000 2024-09-21T09:58:07.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[97580]: created 2024-09-21T09:45:30.814529+0000 2024-09-21T09:58:07.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[97580]: min_mon_release 17 (quincy) 2024-09-21T09:58:07.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[97580]: election_strategy: 1 2024-09-21T09:58:07.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[97580]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-21T09:58:07.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[97580]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-21T09:58:07.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[97580]: 2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.c 2024-09-21T09:58:07.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[97580]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 1 up:standby 2024-09-21T09:58:07.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[97580]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T09:58:07.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[97580]: mgrmap e38: y(active, since 90s), standbys: x 2024-09-21T09:58:07.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[97580]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-21T09:58:07.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T09:58:07.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:07 smithi098 ceph-mon[97580]: overall HEALTH_OK 2024-09-21T09:58:07.352 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:06 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:06.947Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:07.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:07 smithi196 ceph-mon[32670]: mon.c calling monitor election 2024-09-21T09:58:07.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:07 smithi196 ceph-mon[32670]: mon.b calling monitor election 2024-09-21T09:58:07.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:07 smithi196 ceph-mon[32670]: mon.a calling monitor election 2024-09-21T09:58:07.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:07 smithi196 ceph-mon[32670]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:58:07.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:07 smithi196 ceph-mon[32670]: monmap epoch 3 2024-09-21T09:58:07.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:07 smithi196 ceph-mon[32670]: fsid 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:58:07.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:07 smithi196 ceph-mon[32670]: last_changed 2024-09-21T09:47:06.778233+0000 2024-09-21T09:58:07.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:07 smithi196 ceph-mon[32670]: created 2024-09-21T09:45:30.814529+0000 2024-09-21T09:58:07.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:07 smithi196 ceph-mon[32670]: min_mon_release 17 (quincy) 2024-09-21T09:58:07.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:07 smithi196 ceph-mon[32670]: election_strategy: 1 2024-09-21T09:58:07.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:07 smithi196 ceph-mon[32670]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-21T09:58:07.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:07 smithi196 ceph-mon[32670]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-21T09:58:07.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:07 smithi196 ceph-mon[32670]: 2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.c 2024-09-21T09:58:07.354 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:07 smithi196 ceph-mon[32670]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 1 up:standby 2024-09-21T09:58:07.354 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:07 smithi196 ceph-mon[32670]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T09:58:07.354 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:07 smithi196 ceph-mon[32670]: mgrmap e38: y(active, since 90s), standbys: x 2024-09-21T09:58:07.354 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:07 smithi196 ceph-mon[32670]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-21T09:58:07.354 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:07 smithi196 ceph-mon[32670]: Cluster is now healthy 2024-09-21T09:58:07.354 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:07 smithi196 ceph-mon[32670]: overall HEALTH_OK 2024-09-21T09:58:08.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:08 smithi196 ceph-mon[32670]: pgmap v48: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:08.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:08 smithi196 ceph-mon[32670]: from='client.34212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:08 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:08 smithi196 ceph-mon[32670]: from='client.34218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:08.410 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:08 smithi098 ceph-mon[97580]: pgmap v48: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:08.410 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:08 smithi098 ceph-mon[97580]: from='client.34212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:08.410 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:08.410 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:08 smithi098 ceph-mon[97580]: from='client.34218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:08.410 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:08 smithi098 ceph-mon[100914]: pgmap v48: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:08.410 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:08 smithi098 ceph-mon[100914]: from='client.34212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:08.410 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:08.410 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:08 smithi098 ceph-mon[100914]: from='client.34218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:09.630 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T09:58:10.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:10 smithi098 ceph-mon[97580]: pgmap v49: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:10.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:10 smithi098 ceph-mon[97580]: mgrmap e39: y(active, since 92s), standbys: x 2024-09-21T09:58:10.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:10 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:10.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:10 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:10.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:10 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1250835369' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T09:58:10.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:10 smithi098 ceph-mon[100914]: pgmap v49: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:10.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:10 smithi098 ceph-mon[100914]: mgrmap e39: y(active, since 92s), standbys: x 2024-09-21T09:58:10.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:10 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:10.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:10 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:10.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:10 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1250835369' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T09:58:10.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:10 smithi196 ceph-mon[32670]: pgmap v49: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:10 smithi196 ceph-mon[32670]: mgrmap e39: y(active, since 92s), standbys: x 2024-09-21T09:58:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:10 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:10 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:10 smithi196 ceph-mon[32670]: from='client.? 172.21.15.98:0/1250835369' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T09:58:10.402 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch ps' 2024-09-21T09:58:10.660 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:58:10.918 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:58:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:58:10] "GET /metrics HTTP/1.1" 200 33845 "" "Prometheus/2.51.0" 2024-09-21T09:58:10.918 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:58:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:58:10.866+0000 7f6c37c82640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-21T09:58:11.770 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:11 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:11.770 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:11 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:11.770 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:11 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:11.770 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:11 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:11.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:11 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:11.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:11 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:13.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:12 smithi098 ceph-mon[97580]: pgmap v50: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:13.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:12 smithi098 ceph-mon[100914]: pgmap v50: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:13.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:12 smithi196 ceph-mon[32670]: pgmap v50: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:14.265 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T09:58:14.265 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (2m) 4s ago 6m 18.9M - 0.27.0 11f11916f8cd 72b57f924744 2024-09-21T09:58:14.266 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi196 *:3000 running (2m) 23s ago 6m 82.8M - 9.4.7 954c08fa6188 7724661af1a8 2024-09-21T09:58:14.266 INFO:teuthology.orchestra.run.smithi098.stdout:mds.foo.smithi098.stcwsu smithi098 running (5m) 4s ago 5m 18.2M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be d8393a5abb20 2024-09-21T09:58:14.266 INFO:teuthology.orchestra.run.smithi098.stdout:mds.foo.smithi196.ugblah smithi196 running (5m) 23s ago 5m 15.7M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 05a68593bc2b 2024-09-21T09:58:14.266 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi196 *:8443,9283,8765 running (118s) 23s ago 10m 453M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d4969d635bcd 2024-09-21T09:58:14.266 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (2m) 4s ago 12m 508M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 da66347fe5a2 2024-09-21T09:58:14.266 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (104s) 4s ago 12m 40.4M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 0a2195915f65 2024-09-21T09:58:14.266 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi196 running (11m) 23s ago 11m 47.7M 2048M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 4f1a8a66c2c8 2024-09-21T09:58:14.266 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (13s) 4s ago 11m 17.0M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 ef4452ba8cf0 2024-09-21T09:58:14.266 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (2m) 4s ago 6m 10.5M - 1.7.0 72c9c2088986 d587eb6669b7 2024-09-21T09:58:14.266 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi196 *:9100 running (2m) 23s ago 6m 10.4M - 1.7.0 72c9c2088986 7a8564adbc71 2024-09-21T09:58:14.266 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (10m) 4s ago 10m 48.1M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be b4d47286955f 2024-09-21T09:58:14.266 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (10m) 4s ago 10m 49.9M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 850efcb386e6 2024-09-21T09:58:14.266 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (9m) 4s ago 9m 49.8M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be f19a948d9994 2024-09-21T09:58:14.267 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (9m) 4s ago 9m 46.2M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 68f6e62795e7 2024-09-21T09:58:14.267 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi196 running (9m) 23s ago 9m 48.5M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 1693914f1a8b 2024-09-21T09:58:14.267 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi196 running (8m) 23s ago 8m 46.5M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be cb8e4afb2db7 2024-09-21T09:58:14.267 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi196 running (8m) 23s ago 8m 46.4M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 8ba9647073b2 2024-09-21T09:58:14.267 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi196 running (7m) 23s ago 7m 50.2M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 540c98a3c436 2024-09-21T09:58:14.267 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi196 *:9095 running (2m) 23s ago 7m 46.4M - 2.51.0 1d3b7f56885b b30b25036dc7 2024-09-21T09:58:14.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:14 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:14.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:15.137 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:14 smithi098 ceph-mon[97580]: pgmap v51: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:15.137 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:14 smithi098 ceph-mon[100914]: pgmap v51: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:15.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:14 smithi196 ceph-mon[32670]: pgmap v51: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:15.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade pause' 2024-09-21T09:58:15.807 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:58:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:15 smithi196 ceph-mon[32670]: from='client.34230 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:15 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:15 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:15 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:15 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:58:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:15 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:15 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:58:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:15 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:58:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:15 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-21T09:58:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:15 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-21T09:58:16.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:15 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:15.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:16.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:15 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:15.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:16.166 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:15 smithi098 ceph-mon[100914]: from='client.34230 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:16.166 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:15 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:16.167 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:15 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:16.167 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:15 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:16.167 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:15 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:58:16.167 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:15 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:16.167 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:15 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:58:16.167 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:15 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:58:16.167 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:15 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-21T09:58:16.167 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:15 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-21T09:58:16.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:15 smithi098 ceph-mon[97580]: from='client.34230 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:16.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:15 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:16.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:15 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:16.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:15 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:16.168 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:15 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:58:16.168 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:15 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:16.168 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:15 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:58:16.168 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:15 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:58:16.168 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:15 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-21T09:58:16.168 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:15 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-21T09:58:17.096 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:16 smithi196 ceph-mon[32670]: pgmap v52: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:17.097 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:16 smithi196 ceph-mon[32670]: Upgrade: It appears safe to stop mon.b 2024-09-21T09:58:17.097 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:16 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:17.097 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:16 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T09:58:17.097 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:16 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T09:58:17.097 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:16 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:17.097 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:16 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:16.947Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:17.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:16 smithi098 ceph-mon[97580]: pgmap v52: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:17.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:16 smithi098 ceph-mon[97580]: Upgrade: It appears safe to stop mon.b 2024-09-21T09:58:17.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:16 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:17.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:16 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T09:58:17.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:16 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T09:58:17.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:16 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:17.228 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:16 smithi098 ceph-mon[100914]: pgmap v52: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:17.228 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:16 smithi098 ceph-mon[100914]: Upgrade: It appears safe to stop mon.b 2024-09-21T09:58:17.228 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:16 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:17.229 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:16 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T09:58:17.229 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:16 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T09:58:17.229 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:16 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:17.660 INFO:teuthology.orchestra.run.smithi098.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:58:18.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:17 smithi196 ceph-mon[32670]: Upgrade: Updating mon.b 2024-09-21T09:58:18.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:17 smithi196 ceph-mon[32670]: Deploying daemon mon.b on smithi196 2024-09-21T09:58:18.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:17 smithi196 ceph-mon[32670]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:18.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:17 smithi098 ceph-mon[97580]: Upgrade: Updating mon.b 2024-09-21T09:58:18.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:17 smithi098 ceph-mon[97580]: Deploying daemon mon.b on smithi196 2024-09-21T09:58:18.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:17 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:18.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:17 smithi098 ceph-mon[100914]: Upgrade: Updating mon.b 2024-09-21T09:58:18.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:17 smithi098 ceph-mon[100914]: Deploying daemon mon.b on smithi196 2024-09-21T09:58:18.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:17 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:18.413 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'sleep 60' 2024-09-21T09:58:18.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:18 smithi196 systemd[1]: Stopping Ceph mon.b for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:58:18.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-b[32666]: 2024-09-21T09:58:18.569+0000 7f3a4237e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T09:58:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-b[32666]: 2024-09-21T09:58:18.569+0000 7f3a4237e640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-21T09:58:18.658 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:58:19.022 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:18 smithi196 podman[84622]: 2024-09-21 09:58:18.744290086 +0000 UTC m=+0.262731892 container died 4f1a8a66c2c869baf95034670b981005ccdf657396ee0d486e2e892e2b19bac8 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-b, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-21T09:58:19.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:19 smithi196 podman[84622]: 2024-09-21 09:58:19.095171588 +0000 UTC m=+0.613613393 container cleanup 4f1a8a66c2c869baf95034670b981005ccdf657396ee0d486e2e892e2b19bac8 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-b, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=quincy-fbd1a6a, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T09:58:19.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:19 smithi196 bash[84622]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-b 2024-09-21T09:58:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:19 smithi196 podman[84633]: 2024-09-21 09:58:19.379733543 +0000 UTC m=+0.633795685 container remove 4f1a8a66c2c869baf95034670b981005ccdf657396ee0d486e2e892e2b19bac8 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T09:58:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:19 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@mon.b.service: Deactivated successfully. 2024-09-21T09:58:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:19 smithi196 systemd[1]: Stopped Ceph mon.b for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:58:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:19 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@mon.b.service: Consumed 8.355s CPU time. 2024-09-21T09:58:20.023 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:19 smithi196 systemd[1]: Starting Ceph mon.b for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:58:20.023 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:19 smithi196 podman[84765]: 2024-09-21 09:58:19.808960125 +0000 UTC m=+0.016310966 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:58:20.023 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:19 smithi196 podman[84765]: 2024-09-21 09:58:19.914254053 +0000 UTC m=+0.121604886 container create 2776b693f5c027514541e559e3dbc2a7a04cb3bc9f46292910262e9598772bd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-b, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T09:58:20.199 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 podman[84765]: 2024-09-21 09:58:20.167236358 +0000 UTC m=+0.374587197 container init 2776b693f5c027514541e559e3dbc2a7a04cb3bc9f46292910262e9598772bd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-b, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-21T09:58:20.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 podman[84765]: 2024-09-21 09:58:20.170615256 +0000 UTC m=+0.377966087 container start 2776b693f5c027514541e559e3dbc2a7a04cb3bc9f46292910262e9598772bd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-b, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-21T09:58:20.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T09:58:20.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev), process ceph-mon, pid 2 2024-09-21T09:58:20.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: pidfile_write: ignore empty --pid-file 2024-09-21T09:58:20.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: load: jerasure load: lrc 2024-09-21T09:58:20.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: RocksDB version: 7.9.2 2024-09-21T09:58:20.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Git sha 0 2024-09-21T09:58:20.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Compile date 2024-09-20 20:32:28 2024-09-21T09:58:20.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: DB SUMMARY 2024-09-21T09:58:20.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: DB Session ID: 7GO5HAWCI1LCGK18BMBZ 2024-09-21T09:58:20.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: CURRENT file: CURRENT 2024-09-21T09:58:20.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T09:58:20.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1083 Bytes 2024-09-21T09:58:20.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-09-21T09:58:20.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 2439413 ; 2024-09-21T09:58:20.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.error_if_exists: 0 2024-09-21T09:58:20.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.create_if_missing: 0 2024-09-21T09:58:20.202 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.paranoid_checks: 1 2024-09-21T09:58:20.202 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T09:58:20.202 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T09:58:20.202 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T09:58:20.202 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.env: 0x563b6fc7e120 2024-09-21T09:58:20.202 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T09:58:20.202 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.info_log: 0x563b71a6d700 2024-09-21T09:58:20.202 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T09:58:20.202 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.statistics: (nil) 2024-09-21T09:58:20.203 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.use_fsync: 0 2024-09-21T09:58:20.203 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_log_file_size: 0 2024-09-21T09:58:20.203 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T09:58:20.203 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T09:58:20.203 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T09:58:20.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T09:58:20.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.allow_fallocate: 1 2024-09-21T09:58:20.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T09:58:20.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T09:58:20.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.use_direct_reads: 0 2024-09-21T09:58:20.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T09:58:20.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T09:58:20.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.db_log_dir: 2024-09-21T09:58:20.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.wal_dir: 2024-09-21T09:58:20.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T09:58:20.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T09:58:20.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T09:58:20.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T09:58:20.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T09:58:20.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T09:58:20.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T09:58:20.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T09:58:20.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.write_buffer_manager: 0x563b71a701e0 2024-09-21T09:58:20.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T09:58:20.206 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T09:58:20.206 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T09:58:20.206 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T09:58:20.206 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T09:58:20.206 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T09:58:20.206 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T09:58:20.206 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T09:58:20.206 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.unordered_write: 0 2024-09-21T09:58:20.206 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T09:58:20.206 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T09:58:20.206 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T09:58:20.206 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T09:58:20.206 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.row_cache: None 2024-09-21T09:58:20.279 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:20.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.wal_filter: None 2024-09-21T09:58:20.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T09:58:20.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T09:58:20.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.two_write_queues: 0 2024-09-21T09:58:20.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T09:58:20.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.wal_compression: 0 2024-09-21T09:58:20.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.atomic_flush: 0 2024-09-21T09:58:20.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T09:58:20.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T09:58:20.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T09:58:20.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.log_readahead_size: 0 2024-09-21T09:58:20.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T09:58:20.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T09:58:20.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T09:58:20.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T09:58:20.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T09:58:20.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T09:58:20.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T09:58:20.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_background_jobs: 2 2024-09-21T09:58:20.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_background_compactions: -1 2024-09-21T09:58:20.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_subcompactions: 1 2024-09-21T09:58:20.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T09:58:20.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T09:58:20.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T09:58:20.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T09:58:20.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T09:58:20.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T09:58:20.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T09:58:20.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T09:58:20.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_open_files: -1 2024-09-21T09:58:20.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T09:58:20.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T09:58:20.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T09:58:20.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T09:58:20.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_background_flushes: -1 2024-09-21T09:58:20.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Compression algorithms supported: 2024-09-21T09:58:20.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: kZSTD supported: 0 2024-09-21T09:58:20.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: kXpressCompression supported: 0 2024-09-21T09:58:20.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: kBZip2Compression supported: 0 2024-09-21T09:58:20.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T09:58:20.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: kLZ4Compression supported: 1 2024-09-21T09:58:20.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: kZlibCompression supported: 1 2024-09-21T09:58:20.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T09:58:20.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: kSnappyCompression supported: 1 2024-09-21T09:58:20.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T09:58:20.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T09:58:20.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-09-21T09:58:20.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T09:58:20.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T09:58:20.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.merge_operator: 2024-09-21T09:58:20.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compaction_filter: None 2024-09-21T09:58:20.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T09:58:20.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T09:58:20.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T09:58:20.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T09:58:20.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563b71a6d320) 2024-09-21T09:58:20.286 INFO:journalctl@ceph.mon.b.smithi196.stdout: cache_index_and_filter_blocks: 1 2024-09-21T09:58:20.286 INFO:journalctl@ceph.mon.b.smithi196.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T09:58:20.286 INFO:journalctl@ceph.mon.b.smithi196.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T09:58:20.286 INFO:journalctl@ceph.mon.b.smithi196.stdout: pin_top_level_index_and_filter: 1 2024-09-21T09:58:20.286 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_type: 0 2024-09-21T09:58:20.286 INFO:journalctl@ceph.mon.b.smithi196.stdout: data_block_index_type: 0 2024-09-21T09:58:20.286 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_shortening: 1 2024-09-21T09:58:20.286 INFO:journalctl@ceph.mon.b.smithi196.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T09:58:20.286 INFO:journalctl@ceph.mon.b.smithi196.stdout: checksum: 4 2024-09-21T09:58:20.287 INFO:journalctl@ceph.mon.b.smithi196.stdout: no_block_cache: 0 2024-09-21T09:58:20.287 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache: 0x563b71a61610 2024-09-21T09:58:20.287 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_name: BinnedLRUCache 2024-09-21T09:58:20.287 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_options: 2024-09-21T09:58:20.287 INFO:journalctl@ceph.mon.b.smithi196.stdout: capacity : 536870912 2024-09-21T09:58:20.287 INFO:journalctl@ceph.mon.b.smithi196.stdout: num_shard_bits : 4 2024-09-21T09:58:20.287 INFO:journalctl@ceph.mon.b.smithi196.stdout: strict_capacity_limit : 0 2024-09-21T09:58:20.287 INFO:journalctl@ceph.mon.b.smithi196.stdout: high_pri_pool_ratio: 0.000 2024-09-21T09:58:20.287 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_compressed: (nil) 2024-09-21T09:58:20.287 INFO:journalctl@ceph.mon.b.smithi196.stdout: persistent_cache: (nil) 2024-09-21T09:58:20.288 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_size: 4096 2024-09-21T09:58:20.288 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_size_deviation: 10 2024-09-21T09:58:20.288 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_restart_interval: 16 2024-09-21T09:58:20.288 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_block_restart_interval: 1 2024-09-21T09:58:20.288 INFO:journalctl@ceph.mon.b.smithi196.stdout: metadata_block_size: 4096 2024-09-21T09:58:20.288 INFO:journalctl@ceph.mon.b.smithi196.stdout: partition_filters: 0 2024-09-21T09:58:20.288 INFO:journalctl@ceph.mon.b.smithi196.stdout: use_delta_encoding: 1 2024-09-21T09:58:20.288 INFO:journalctl@ceph.mon.b.smithi196.stdout: filter_policy: bloomfilter 2024-09-21T09:58:20.288 INFO:journalctl@ceph.mon.b.smithi196.stdout: whole_key_filtering: 1 2024-09-21T09:58:20.288 INFO:journalctl@ceph.mon.b.smithi196.stdout: verify_compression: 0 2024-09-21T09:58:20.288 INFO:journalctl@ceph.mon.b.smithi196.stdout: read_amp_bytes_per_bit: 0 2024-09-21T09:58:20.288 INFO:journalctl@ceph.mon.b.smithi196.stdout: format_version: 5 2024-09-21T09:58:20.288 INFO:journalctl@ceph.mon.b.smithi196.stdout: enable_index_compression: 1 2024-09-21T09:58:20.289 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_align: 0 2024-09-21T09:58:20.289 INFO:journalctl@ceph.mon.b.smithi196.stdout: max_auto_readahead_size: 262144 2024-09-21T09:58:20.289 INFO:journalctl@ceph.mon.b.smithi196.stdout: prepopulate_block_cache: 0 2024-09-21T09:58:20.289 INFO:journalctl@ceph.mon.b.smithi196.stdout: initial_auto_readahead_size: 8192 2024-09-21T09:58:20.289 INFO:journalctl@ceph.mon.b.smithi196.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T09:58:20.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T09:58:20.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T09:58:20.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compression: NoCompression 2024-09-21T09:58:20.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T09:58:20.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T09:58:20.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T09:58:20.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.num_levels: 7 2024-09-21T09:58:20.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T09:58:20.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T09:58:20.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T09:58:20.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T09:58:20.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T09:58:20.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T09:58:20.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T09:58:20.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:58:20.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T09:58:20.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T09:58:20.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T09:58:20.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T09:58:20.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T09:58:20.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T09:58:20.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T09:58:20.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T09:58:20.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:58:20.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T09:58:20.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T09:58:20.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T09:58:20.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T09:58:20.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T09:58:20.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T09:58:20.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T09:58:20.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T09:58:20.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T09:58:20.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T09:58:20.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T09:58:20.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T09:58:20.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T09:58:20.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T09:58:20.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T09:58:20.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T09:58:20.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T09:58:20.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T09:58:20.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T09:58:20.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T09:58:20.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T09:58:20.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T09:58:20.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T09:58:20.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T09:58:20.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T09:58:20.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T09:58:20.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T09:58:20.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T09:58:20.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T09:58:20.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T09:58:20.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T09:58:20.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T09:58:20.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T09:58:20.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T09:58:20.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T09:58:20.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T09:58:20.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T09:58:20.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.inplace_update_support: 0 2024-09-21T09:58:20.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T09:58:20.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T09:58:20.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T09:58:20.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T09:58:20.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.bloom_locality: 0 2024-09-21T09:58:20.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.max_successive_merges: 0 2024-09-21T09:58:20.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T09:58:20.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T09:58:20.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T09:58:20.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T09:58:20.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.ttl: 2592000 2024-09-21T09:58:20.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T09:58:20.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T09:58:20.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T09:58:20.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.enable_blob_files: false 2024-09-21T09:58:20.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.min_blob_size: 0 2024-09-21T09:58:20.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T09:58:20.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T09:58:20.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T09:58:20.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T09:58:20.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T09:58:20.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T09:58:20.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T09:58:20.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T09:58:20.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-09-21T09:58:20.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T09:58:20.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 13125, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T09:58:20.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-09-21T09:58:20.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b017cf38-cc8c-4ac1-8581-518c0ad88a40 2024-09-21T09:58:20.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912700197634, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-09-21T09:58:20.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-09-21T09:58:20.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912700204456, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 1445808, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 13122, "largest_seqno": 14388, "table_properties": {"data_size": 1440156, "index_size": 3395, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1221, "raw_key_size": 10394, "raw_average_key_size": 22, "raw_value_size": 1429810, "raw_average_value_size": 3042, "num_data_blocks": 157, "num_entries": 470, "num_filter_entries": 470, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726912700, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b017cf38-cc8c-4ac1-8581-518c0ad88a40", "db_session_id": "7GO5HAWCI1LCGK18BMBZ", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-09-21T09:58:20.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912700204570, "job": 1, "event": "recovery_finished"} 2024-09-21T09:58:20.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-09-21T09:58:20.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T09:58:20.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T09:58:20.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563b71a90e00 2024-09-21T09:58:20.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: DB pointer 0x563b71b92000 2024-09-21T09:58:20.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T09:58:20.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T09:58:20.300 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** DB Stats ** 2024-09-21T09:58:20.300 INFO:journalctl@ceph.mon.b.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:58:20.300 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T09:58:20.301 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:58:20.301 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:58:20.301 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T09:58:20.301 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:58:20.301 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:58:20.301 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-21T09:58:20.301 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** Compaction Stats [default] ** 2024-09-21T09:58:20.301 INFO:journalctl@ceph.mon.b.smithi196.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T09:58:20.301 INFO:journalctl@ceph.mon.b.smithi196.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T09:58:20.302 INFO:journalctl@ceph.mon.b.smithi196.stdout: L0 1/0 1.38 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 473.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-21T09:58:20.302 INFO:journalctl@ceph.mon.b.smithi196.stdout: L6 1/0 8.90 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-21T09:58:20.302 INFO:journalctl@ceph.mon.b.smithi196.stdout: Sum 2/0 10.28 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 473.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-21T09:58:20.302 INFO:journalctl@ceph.mon.b.smithi196.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 473.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-21T09:58:20.302 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-21T09:58:20.302 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** Compaction Stats [default] ** 2024-09-21T09:58:20.302 INFO:journalctl@ceph.mon.b.smithi196.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T09:58:20.302 INFO:journalctl@ceph.mon.b.smithi196.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:58:20.303 INFO:journalctl@ceph.mon.b.smithi196.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 473.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-21T09:58:20.303 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-21T09:58:20.303 INFO:journalctl@ceph.mon.b.smithi196.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T09:58:20.303 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-21T09:58:20.303 INFO:journalctl@ceph.mon.b.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:58:20.303 INFO:journalctl@ceph.mon.b.smithi196.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-09-21T09:58:20.303 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:58:20.303 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:58:20.303 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:58:20.304 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:58:20.304 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative compaction: 0.00 GB write, 118.40 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:58:20.304 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval compaction: 0.00 GB write, 118.40 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:58:20.304 INFO:journalctl@ceph.mon.b.smithi196.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:58:20.304 INFO:journalctl@ceph.mon.b.smithi196.stdout: Block cache BinnedLRUCache@0x563b71a61610#2 capacity: 512.00 MB usage: 5.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-09-21T09:58:20.304 INFO:journalctl@ceph.mon.b.smithi196.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.28 KB,0.000244379%) IndexBlock(1,4.08 KB,0.000777841%) Misc(1,0.00 KB,0%) 2024-09-21T09:58:20.304 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-21T09:58:20.304 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:58:20.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: starting mon.b rank 1 at public addrs [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] at bind addrs [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:58:20.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: mon.b@-1(???) e3 preinit fsid 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:58:20.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: mon.b@-1(???).mds e9 new map 2024-09-21T09:58:20.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: mon.b@-1(???).mds e9 print_map 2024-09-21T09:58:20.305 INFO:journalctl@ceph.mon.b.smithi196.stdout: e9 2024-09-21T09:58:20.305 INFO:journalctl@ceph.mon.b.smithi196.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-21T09:58:20.305 INFO:journalctl@ceph.mon.b.smithi196.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T09:58:20.305 INFO:journalctl@ceph.mon.b.smithi196.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:58:20.305 INFO:journalctl@ceph.mon.b.smithi196.stdout: legacy client fscid: 1 2024-09-21T09:58:20.305 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-21T09:58:20.305 INFO:journalctl@ceph.mon.b.smithi196.stdout: Filesystem 'foo' (1) 2024-09-21T09:58:20.305 INFO:journalctl@ceph.mon.b.smithi196.stdout: fs_name foo 2024-09-21T09:58:20.305 INFO:journalctl@ceph.mon.b.smithi196.stdout: epoch 8 2024-09-21T09:58:20.305 INFO:journalctl@ceph.mon.b.smithi196.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-21T09:58:20.306 INFO:journalctl@ceph.mon.b.smithi196.stdout: created 2024-09-21T09:52:20.376206+0000 2024-09-21T09:58:20.306 INFO:journalctl@ceph.mon.b.smithi196.stdout: modified 2024-09-21T09:52:32.894452+0000 2024-09-21T09:58:20.306 INFO:journalctl@ceph.mon.b.smithi196.stdout: tableserver 0 2024-09-21T09:58:20.306 INFO:journalctl@ceph.mon.b.smithi196.stdout: root 0 2024-09-21T09:58:20.306 INFO:journalctl@ceph.mon.b.smithi196.stdout: session_timeout 60 2024-09-21T09:58:20.306 INFO:journalctl@ceph.mon.b.smithi196.stdout: session_autoclose 300 2024-09-21T09:58:20.306 INFO:journalctl@ceph.mon.b.smithi196.stdout: max_file_size 1099511627776 2024-09-21T09:58:20.306 INFO:journalctl@ceph.mon.b.smithi196.stdout: max_xattr_size 65536 2024-09-21T09:58:20.306 INFO:journalctl@ceph.mon.b.smithi196.stdout: required_client_features {} 2024-09-21T09:58:20.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: last_failure 0 2024-09-21T09:58:20.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: last_failure_osd_epoch 0 2024-09-21T09:58:20.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:58:20.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: max_mds 1 2024-09-21T09:58:20.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: in 0 2024-09-21T09:58:20.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: up {0=14640} 2024-09-21T09:58:20.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: failed 2024-09-21T09:58:20.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: damaged 2024-09-21T09:58:20.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: stopped 2024-09-21T09:58:20.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: data_pools [4] 2024-09-21T09:58:20.308 INFO:journalctl@ceph.mon.b.smithi196.stdout: metadata_pool 3 2024-09-21T09:58:20.308 INFO:journalctl@ceph.mon.b.smithi196.stdout: inline_data disabled 2024-09-21T09:58:20.308 INFO:journalctl@ceph.mon.b.smithi196.stdout: balancer 2024-09-21T09:58:20.308 INFO:journalctl@ceph.mon.b.smithi196.stdout: bal_rank_mask -1 2024-09-21T09:58:20.308 INFO:journalctl@ceph.mon.b.smithi196.stdout: standby_count_wanted 1 2024-09-21T09:58:20.308 INFO:journalctl@ceph.mon.b.smithi196.stdout: qdb_cluster leader: 0 members: 2024-09-21T09:58:20.308 INFO:journalctl@ceph.mon.b.smithi196.stdout: [mds.foo.smithi098.stcwsu{0:14640} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.98:6834/2705511271,v1:172.21.15.98:6835/2705511271] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T09:58:20.309 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-21T09:58:20.309 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-21T09:58:20.309 INFO:journalctl@ceph.mon.b.smithi196.stdout: Standby daemons: 2024-09-21T09:58:20.309 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-21T09:58:20.309 INFO:journalctl@ceph.mon.b.smithi196.stdout: [mds.foo.smithi196.ugblah{-1:24481} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.196:6832/3677035285,v1:172.21.15.196:6833/3677035285] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T09:58:20.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: mon.b@-1(???).osd e82 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-21T09:58:20.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: mon.b@-1(???).osd e82 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T09:58:20.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: mon.b@-1(???).osd e82 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T09:58:20.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: mon.b@-1(???).osd e82 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T09:58:20.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 ceph-mon[84784]: mon.b@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2024-09-21T09:58:20.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 bash[84765]: 2776b693f5c027514541e559e3dbc2a7a04cb3bc9f46292910262e9598772bd6 2024-09-21T09:58:20.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:20 smithi196 systemd[1]: Started Ceph mon.b for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:58:21.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:58:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:58:20] "GET /metrics HTTP/1.1" 200 33845 "" "Prometheus/2.51.0" 2024-09-21T09:58:21.864 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:21 smithi098 ceph-mon[97580]: pgmap v53: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:21.864 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:21 smithi098 ceph-mon[97580]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:21.864 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:21 smithi098 ceph-mon[97580]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:58:21.864 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:21 smithi098 ceph-mon[100914]: pgmap v53: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:21.864 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:21 smithi098 ceph-mon[100914]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:21.865 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:21 smithi098 ceph-mon[100914]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:58:21.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:21 smithi196 ceph-mon[84784]: pgmap v53: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:21.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:21 smithi196 ceph-mon[84784]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:21.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:21 smithi196 ceph-mon[84784]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:58:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: pgmap v55: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: mon.b calling monitor election 2024-09-21T09:58:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: mon.c calling monitor election 2024-09-21T09:58:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T09:58:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T09:58:23.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: mon.a calling monitor election 2024-09-21T09:58:23.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:58:23.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T09:58:23.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: monmap epoch 4 2024-09-21T09:58:23.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: fsid 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:58:23.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: last_changed 2024-09-21T09:58:21.841525+0000 2024-09-21T09:58:23.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: created 2024-09-21T09:45:30.814529+0000 2024-09-21T09:58:23.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: min_mon_release 19 (squid) 2024-09-21T09:58:23.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: election_strategy: 1 2024-09-21T09:58:23.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-21T09:58:23.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-21T09:58:23.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: 2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.c 2024-09-21T09:58:23.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 1 up:standby 2024-09-21T09:58:23.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T09:58:23.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: mgrmap e39: y(active, since 104s), standbys: x 2024-09-21T09:58:23.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: overall HEALTH_OK 2024-09-21T09:58:23.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:23.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:23.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:58:23.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:23.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: pgmap v55: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:23.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: mon.b calling monitor election 2024-09-21T09:58:23.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: mon.c calling monitor election 2024-09-21T09:58:23.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T09:58:23.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T09:58:23.100 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: mon.a calling monitor election 2024-09-21T09:58:23.100 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:58:23.100 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T09:58:23.100 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: monmap epoch 4 2024-09-21T09:58:23.100 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: fsid 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:58:23.100 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: last_changed 2024-09-21T09:58:21.841525+0000 2024-09-21T09:58:23.100 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: created 2024-09-21T09:45:30.814529+0000 2024-09-21T09:58:23.100 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: min_mon_release 19 (squid) 2024-09-21T09:58:23.100 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: election_strategy: 1 2024-09-21T09:58:23.100 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-21T09:58:23.100 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-21T09:58:23.100 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: 2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.c 2024-09-21T09:58:23.100 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 1 up:standby 2024-09-21T09:58:23.100 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T09:58:23.101 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: mgrmap e39: y(active, since 104s), standbys: x 2024-09-21T09:58:23.101 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: overall HEALTH_OK 2024-09-21T09:58:23.101 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:23.101 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:23.101 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:58:23.101 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: pgmap v55: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: mon.b calling monitor election 2024-09-21T09:58:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: mon.c calling monitor election 2024-09-21T09:58:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T09:58:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T09:58:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: mon.a calling monitor election 2024-09-21T09:58:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:58:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T09:58:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: monmap epoch 4 2024-09-21T09:58:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: fsid 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T09:58:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: last_changed 2024-09-21T09:58:21.841525+0000 2024-09-21T09:58:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: created 2024-09-21T09:45:30.814529+0000 2024-09-21T09:58:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: min_mon_release 19 (squid) 2024-09-21T09:58:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: election_strategy: 1 2024-09-21T09:58:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-21T09:58:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-21T09:58:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: 2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.c 2024-09-21T09:58:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: fsmap foo:1 {0=foo.smithi098.stcwsu=up:active} 1 up:standby 2024-09-21T09:58:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T09:58:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: mgrmap e39: y(active, since 104s), standbys: x 2024-09-21T09:58:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: overall HEALTH_OK 2024-09-21T09:58:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:58:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:24.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:24 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:24.146Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:25.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:24 smithi196 ceph-mon[84784]: pgmap v56: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:25.220 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:24 smithi098 ceph-mon[97580]: pgmap v56: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:25.220 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:24 smithi098 ceph-mon[100914]: pgmap v56: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:25.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:58:25 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T09:58:25.217+0000 7f6c37c82640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-21T09:58:26.138 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:25 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:25.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:26.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:25 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:25.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:27.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:26 smithi196 ceph-mon[84784]: pgmap v57: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:27.047 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:26 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:27.047 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:26 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:27.047 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:26 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:27.047 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:26 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:27.047 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:26 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:26.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:27.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:26 smithi098 ceph-mon[100914]: pgmap v57: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:27.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:26 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:27.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:26 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:27.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:26 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:27.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:26 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:27.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:26 smithi098 ceph-mon[97580]: pgmap v57: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:27.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:26 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:27.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:26 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:27.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:26 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:27.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:26 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:29.083 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:28 smithi196 ceph-mon[84784]: pgmap v58: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:29.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:28 smithi098 ceph-mon[97580]: pgmap v58: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:29.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:28 smithi098 ceph-mon[100914]: pgmap v58: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:30.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:30 smithi196 ceph-mon[84784]: pgmap v59: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:58:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:30 smithi196 ceph-mon[84784]: Reconfiguring mon.a (monmap changed)... 2024-09-21T09:58:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T09:58:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T09:58:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:30 smithi196 ceph-mon[84784]: Reconfiguring daemon mon.a on smithi098 2024-09-21T09:58:30.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-mon[100914]: pgmap v59: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:30.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:30.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:30.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:30.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:58:30.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:30.660 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-mon[100914]: Reconfiguring mon.a (monmap changed)... 2024-09-21T09:58:30.660 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T09:58:30.660 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T09:58:30.660 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:30.660 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-mon[100914]: Reconfiguring daemon mon.a on smithi098 2024-09-21T09:58:30.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-mon[97580]: pgmap v59: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:30.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:30.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:30.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:30.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:58:30.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:30.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-mon[97580]: Reconfiguring mon.a (monmap changed)... 2024-09-21T09:58:30.661 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T09:58:30.661 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T09:58:30.661 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:30.661 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-mon[97580]: Reconfiguring daemon mon.a on smithi098 2024-09-21T09:58:31.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:58:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:58:30] "GET /metrics HTTP/1.1" 200 33772 "" "Prometheus/2.51.0" 2024-09-21T09:58:32.336 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:31 smithi098 ceph-mon[97580]: pgmap v60: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:32.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:31 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:32.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:31 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:32.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:31 smithi098 ceph-mon[97580]: Reconfiguring mgr.y (monmap changed)... 2024-09-21T09:58:32.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:31 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T09:58:32.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:31 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T09:58:32.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:31 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:32.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:31 smithi098 ceph-mon[97580]: Reconfiguring daemon mgr.y on smithi098 2024-09-21T09:58:32.337 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:31 smithi098 ceph-mon[100914]: pgmap v60: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:32.337 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:31 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:32.337 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:31 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:32.337 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:31 smithi098 ceph-mon[100914]: Reconfiguring mgr.y (monmap changed)... 2024-09-21T09:58:32.338 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:31 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T09:58:32.338 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:31 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T09:58:32.338 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:31 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:32.338 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:31 smithi098 ceph-mon[100914]: Reconfiguring daemon mgr.y on smithi098 2024-09-21T09:58:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:31 smithi196 ceph-mon[84784]: pgmap v60: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:31 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:31 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:31 smithi196 ceph-mon[84784]: Reconfiguring mgr.y (monmap changed)... 2024-09-21T09:58:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:31 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T09:58:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:31 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T09:58:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:31 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:31 smithi196 ceph-mon[84784]: Reconfiguring daemon mgr.y on smithi098 2024-09-21T09:58:33.774 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:33 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:33.774 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:33 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:33.774 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:33 smithi098 ceph-mon[97580]: Reconfiguring mon.c (monmap changed)... 2024-09-21T09:58:33.774 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:33 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T09:58:33.774 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:33 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T09:58:33.774 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:33 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:33.774 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:33 smithi098 ceph-mon[97580]: Reconfiguring daemon mon.c on smithi098 2024-09-21T09:58:33.775 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:33 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:33.775 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:33 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:33.775 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:33 smithi098 ceph-mon[100914]: Reconfiguring mon.c (monmap changed)... 2024-09-21T09:58:33.775 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:33 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T09:58:33.775 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:33 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T09:58:33.775 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:33 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:33.775 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:33 smithi098 ceph-mon[100914]: Reconfiguring daemon mon.c on smithi098 2024-09-21T09:58:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:33 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:33 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:33 smithi196 ceph-mon[84784]: Reconfiguring mon.c (monmap changed)... 2024-09-21T09:58:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:33 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T09:58:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:33 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T09:58:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:33 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:33 smithi196 ceph-mon[84784]: Reconfiguring daemon mon.c on smithi098 2024-09-21T09:58:34.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:34 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:34.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:35.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:34 smithi098 ceph-mon[100914]: pgmap v61: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:35.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:34 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:35.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:34 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:35.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:34 smithi098 ceph-mon[100914]: Reconfiguring osd.0 (monmap changed)... 2024-09-21T09:58:35.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:34 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T09:58:35.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:34 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:35.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:34 smithi098 ceph-mon[100914]: Reconfiguring daemon osd.0 on smithi098 2024-09-21T09:58:35.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:34 smithi098 ceph-mon[97580]: pgmap v61: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:35.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:34 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:35.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:34 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:35.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:34 smithi098 ceph-mon[97580]: Reconfiguring osd.0 (monmap changed)... 2024-09-21T09:58:35.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:34 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T09:58:35.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:34 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:35.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:34 smithi098 ceph-mon[97580]: Reconfiguring daemon osd.0 on smithi098 2024-09-21T09:58:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:34 smithi196 ceph-mon[84784]: pgmap v61: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:34 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:34 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:34 smithi196 ceph-mon[84784]: Reconfiguring osd.0 (monmap changed)... 2024-09-21T09:58:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:34 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T09:58:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:34 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:34 smithi196 ceph-mon[84784]: Reconfiguring daemon osd.0 on smithi098 2024-09-21T09:58:36.041 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:35.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:36.041 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:35.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:36.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:36 smithi098 ceph-mon[97580]: pgmap v62: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:36.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:36 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:36.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:36 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:36.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:36 smithi098 ceph-mon[97580]: Reconfiguring osd.1 (monmap changed)... 2024-09-21T09:58:36.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:36 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T09:58:36.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:36 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:36.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:36 smithi098 ceph-mon[97580]: Reconfiguring daemon osd.1 on smithi098 2024-09-21T09:58:36.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:36 smithi098 ceph-mon[100914]: pgmap v62: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:36.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:36 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:36.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:36 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:36.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:36 smithi098 ceph-mon[100914]: Reconfiguring osd.1 (monmap changed)... 2024-09-21T09:58:36.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:36 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T09:58:36.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:36 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:36.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:36 smithi098 ceph-mon[100914]: Reconfiguring daemon osd.1 on smithi098 2024-09-21T09:58:36.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:36 smithi196 ceph-mon[84784]: pgmap v62: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:36.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:36 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:36 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:36 smithi196 ceph-mon[84784]: Reconfiguring osd.1 (monmap changed)... 2024-09-21T09:58:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:36 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T09:58:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:36 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:36 smithi196 ceph-mon[84784]: Reconfiguring daemon osd.1 on smithi098 2024-09-21T09:58:37.302 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:36 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:36.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:38 smithi098 ceph-mon[97580]: pgmap v63: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:38 smithi098 ceph-mon[97580]: Reconfiguring osd.2 (monmap changed)... 2024-09-21T09:58:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T09:58:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:38.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:38 smithi098 ceph-mon[97580]: Reconfiguring daemon osd.2 on smithi098 2024-09-21T09:58:38.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:38.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:38 smithi098 ceph-mon[100914]: pgmap v63: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:38.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:38.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:38.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:38 smithi098 ceph-mon[100914]: Reconfiguring osd.2 (monmap changed)... 2024-09-21T09:58:38.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T09:58:38.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:38.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:38 smithi098 ceph-mon[100914]: Reconfiguring daemon osd.2 on smithi098 2024-09-21T09:58:38.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:38.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:38 smithi196 ceph-mon[84784]: pgmap v63: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:38 smithi196 ceph-mon[84784]: Reconfiguring osd.2 (monmap changed)... 2024-09-21T09:58:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T09:58:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:38 smithi196 ceph-mon[84784]: Reconfiguring daemon osd.2 on smithi098 2024-09-21T09:58:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:39.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:39 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:39.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:39 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:39.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:39 smithi098 ceph-mon[97580]: Reconfiguring osd.3 (monmap changed)... 2024-09-21T09:58:39.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:39 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T09:58:39.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:39 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:39.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:39 smithi098 ceph-mon[97580]: Reconfiguring daemon osd.3 on smithi098 2024-09-21T09:58:39.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:39 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:39.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:39 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:39.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:39 smithi098 ceph-mon[100914]: Reconfiguring osd.3 (monmap changed)... 2024-09-21T09:58:39.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:39 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T09:58:39.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:39 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:39.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:39 smithi098 ceph-mon[100914]: Reconfiguring daemon osd.3 on smithi098 2024-09-21T09:58:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:39 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:39 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:39 smithi196 ceph-mon[84784]: Reconfiguring osd.3 (monmap changed)... 2024-09-21T09:58:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:39 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T09:58:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:39 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:39 smithi196 ceph-mon[84784]: Reconfiguring daemon osd.3 on smithi098 2024-09-21T09:58:40.564 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:40.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:40 smithi098 ceph-mon[97580]: pgmap v64: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:40.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:40 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:40.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:40 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:40.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:40 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi098.stcwsu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T09:58:40.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:40 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:40.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:40 smithi098 ceph-mon[100914]: pgmap v64: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:40.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:40 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:40.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:40 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:40.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:40 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi098.stcwsu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T09:58:40.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:40 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:40.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:40 smithi196 ceph-mon[84784]: pgmap v64: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:40.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:40 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:40.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:40 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:40 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi098.stcwsu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T09:58:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:40 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:41.064 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:58:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:58:40] "GET /metrics HTTP/1.1" 200 33839 "" "Prometheus/2.51.0" 2024-09-21T09:58:41.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:41 smithi098 ceph-mon[97580]: Reconfiguring mds.foo.smithi098.stcwsu (monmap changed)... 2024-09-21T09:58:41.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:41 smithi098 ceph-mon[97580]: Reconfiguring daemon mds.foo.smithi098.stcwsu on smithi098 2024-09-21T09:58:41.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:41 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:41.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:41 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:41.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:41 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T09:58:41.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:41 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T09:58:41.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:41 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:41.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:41 smithi098 ceph-mon[100914]: Reconfiguring mds.foo.smithi098.stcwsu (monmap changed)... 2024-09-21T09:58:41.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:41 smithi098 ceph-mon[100914]: Reconfiguring daemon mds.foo.smithi098.stcwsu on smithi098 2024-09-21T09:58:41.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:41 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:41.851 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:41 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:41.851 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:41 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T09:58:41.851 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:41 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T09:58:41.851 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:41 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:41.857 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:41 smithi196 ceph-mon[84784]: Reconfiguring mds.foo.smithi098.stcwsu (monmap changed)... 2024-09-21T09:58:41.857 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:41 smithi196 ceph-mon[84784]: Reconfiguring daemon mds.foo.smithi098.stcwsu on smithi098 2024-09-21T09:58:41.857 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:41 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:41.857 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:41 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:41.857 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:41 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T09:58:41.857 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:41 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T09:58:41.857 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:41 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:42.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:42 smithi098 ceph-mon[97580]: pgmap v65: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:42.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:42 smithi098 ceph-mon[97580]: Reconfiguring mon.b (monmap changed)... 2024-09-21T09:58:42.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:42 smithi098 ceph-mon[97580]: Reconfiguring daemon mon.b on smithi196 2024-09-21T09:58:42.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:42 smithi098 ceph-mon[100914]: pgmap v65: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:42.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:42 smithi098 ceph-mon[100914]: Reconfiguring mon.b (monmap changed)... 2024-09-21T09:58:42.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:42 smithi098 ceph-mon[100914]: Reconfiguring daemon mon.b on smithi196 2024-09-21T09:58:42.864 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:42 smithi196 ceph-mon[84784]: pgmap v65: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:42.864 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:42 smithi196 ceph-mon[84784]: Reconfiguring mon.b (monmap changed)... 2024-09-21T09:58:42.864 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:42 smithi196 ceph-mon[84784]: Reconfiguring daemon mon.b on smithi196 2024-09-21T09:58:44.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:43 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:44.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:43 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:44.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:43 smithi098 ceph-mon[97580]: Reconfiguring mgr.x (monmap changed)... 2024-09-21T09:58:44.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:43 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T09:58:44.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:43 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T09:58:44.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:43 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:44.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:43 smithi098 ceph-mon[97580]: Reconfiguring daemon mgr.x on smithi196 2024-09-21T09:58:44.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:43 smithi098 ceph-mon[97580]: pgmap v66: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:44.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:43 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:44.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:43 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:44.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:43 smithi098 ceph-mon[100914]: Reconfiguring mgr.x (monmap changed)... 2024-09-21T09:58:44.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:43 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T09:58:44.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:43 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T09:58:44.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:43 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:44.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:43 smithi098 ceph-mon[100914]: Reconfiguring daemon mgr.x on smithi196 2024-09-21T09:58:44.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:43 smithi098 ceph-mon[100914]: pgmap v66: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:44.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:44 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:44.146Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:43 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:43 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:43 smithi196 ceph-mon[84784]: Reconfiguring mgr.x (monmap changed)... 2024-09-21T09:58:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:43 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T09:58:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:43 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T09:58:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:43 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:43 smithi196 ceph-mon[84784]: Reconfiguring daemon mgr.x on smithi196 2024-09-21T09:58:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:43 smithi196 ceph-mon[84784]: pgmap v66: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:45.509 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:45 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:45.509 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:45 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:45.509 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:45 smithi196 ceph-mon[84784]: Reconfiguring osd.4 (monmap changed)... 2024-09-21T09:58:45.510 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:45 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T09:58:45.510 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:45 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:45.510 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:45 smithi196 ceph-mon[84784]: Reconfiguring daemon osd.4 on smithi196 2024-09-21T09:58:45.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:45 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:45.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:45 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:45.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:45 smithi098 ceph-mon[97580]: Reconfiguring osd.4 (monmap changed)... 2024-09-21T09:58:45.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:45 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T09:58:45.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:45 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:45.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:45 smithi098 ceph-mon[97580]: Reconfiguring daemon osd.4 on smithi196 2024-09-21T09:58:45.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:45 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:45.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:45 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:45.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:45 smithi098 ceph-mon[100914]: Reconfiguring osd.4 (monmap changed)... 2024-09-21T09:58:45.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:45 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T09:58:45.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:45 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:45.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:45 smithi098 ceph-mon[100914]: Reconfiguring daemon osd.4 on smithi196 2024-09-21T09:58:45.888 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:45.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:45.889 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:45.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:46 smithi196 ceph-mon[84784]: pgmap v67: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:46 smithi196 ceph-mon[84784]: Reconfiguring osd.5 (monmap changed)... 2024-09-21T09:58:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T09:58:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:46 smithi196 ceph-mon[84784]: Reconfiguring daemon osd.5 on smithi196 2024-09-21T09:58:47.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:46 smithi098 ceph-mon[97580]: pgmap v67: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:47.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:47.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:47.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:46 smithi098 ceph-mon[97580]: Reconfiguring osd.5 (monmap changed)... 2024-09-21T09:58:47.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T09:58:47.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:47.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:46 smithi098 ceph-mon[97580]: Reconfiguring daemon osd.5 on smithi196 2024-09-21T09:58:47.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:46 smithi098 ceph-mon[100914]: pgmap v67: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:47.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:47.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:47.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:46 smithi098 ceph-mon[100914]: Reconfiguring osd.5 (monmap changed)... 2024-09-21T09:58:47.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T09:58:47.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:47.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:46 smithi098 ceph-mon[100914]: Reconfiguring daemon osd.5 on smithi196 2024-09-21T09:58:47.350 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:46.947Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:48.229 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:48 smithi196 ceph-mon[84784]: pgmap v68: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:48.229 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:48 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:48.230 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:48 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:48.230 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:48 smithi196 ceph-mon[84784]: Reconfiguring osd.6 (monmap changed)... 2024-09-21T09:58:48.230 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:48 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T09:58:48.230 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:48 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:48.230 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:48 smithi196 ceph-mon[84784]: Reconfiguring daemon osd.6 on smithi196 2024-09-21T09:58:48.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:48 smithi098 ceph-mon[100914]: pgmap v68: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:48.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:48 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:48.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:48 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:48.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:48 smithi098 ceph-mon[100914]: Reconfiguring osd.6 (monmap changed)... 2024-09-21T09:58:48.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:48 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T09:58:48.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:48 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:48.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:48 smithi098 ceph-mon[100914]: Reconfiguring daemon osd.6 on smithi196 2024-09-21T09:58:48.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:48 smithi098 ceph-mon[97580]: pgmap v68: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:48.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:48 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:48.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:48 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:48.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:48 smithi098 ceph-mon[97580]: Reconfiguring osd.6 (monmap changed)... 2024-09-21T09:58:48.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:48 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T09:58:48.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:48 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:48.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:48 smithi098 ceph-mon[97580]: Reconfiguring daemon osd.6 on smithi196 2024-09-21T09:58:49.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:49 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:49.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:49 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:49.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:49 smithi098 ceph-mon[100914]: Reconfiguring osd.7 (monmap changed)... 2024-09-21T09:58:49.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:49 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T09:58:49.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:49 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:49.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:49 smithi098 ceph-mon[100914]: Reconfiguring daemon osd.7 on smithi196 2024-09-21T09:58:49.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:49 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:49.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:49 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:49.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:49 smithi098 ceph-mon[97580]: Reconfiguring osd.7 (monmap changed)... 2024-09-21T09:58:49.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:49 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T09:58:49.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:49 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:49.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:49 smithi098 ceph-mon[97580]: Reconfiguring daemon osd.7 on smithi196 2024-09-21T09:58:49.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:49 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:49.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:49 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:49.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:49 smithi196 ceph-mon[84784]: Reconfiguring osd.7 (monmap changed)... 2024-09-21T09:58:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:49 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T09:58:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:49 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:49 smithi196 ceph-mon[84784]: Reconfiguring daemon osd.7 on smithi196 2024-09-21T09:58:50.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:51.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:50 smithi196 ceph-mon[84784]: pgmap v69: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:51.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:50 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:51.020 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:50 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:51.020 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:50 smithi196 ceph-mon[84784]: Reconfiguring mds.foo.smithi196.ugblah (monmap changed)... 2024-09-21T09:58:51.020 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:50 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi196.ugblah", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T09:58:51.020 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:50 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:51.020 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:50 smithi196 ceph-mon[84784]: Reconfiguring daemon mds.foo.smithi196.ugblah on smithi196 2024-09-21T09:58:51.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:58:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:58:50] "GET /metrics HTTP/1.1" 200 33839 "" "Prometheus/2.51.0" 2024-09-21T09:58:51.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:50 smithi098 ceph-mon[97580]: pgmap v69: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:51.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:50 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:51.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:50 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:51.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:50 smithi098 ceph-mon[97580]: Reconfiguring mds.foo.smithi196.ugblah (monmap changed)... 2024-09-21T09:58:51.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:50 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi196.ugblah", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T09:58:51.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:50 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:51.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:50 smithi098 ceph-mon[97580]: Reconfiguring daemon mds.foo.smithi196.ugblah on smithi196 2024-09-21T09:58:51.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:50 smithi098 ceph-mon[100914]: pgmap v69: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:51.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:50 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:51.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:50 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:51.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:50 smithi098 ceph-mon[100914]: Reconfiguring mds.foo.smithi196.ugblah (monmap changed)... 2024-09-21T09:58:51.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:50 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi196.ugblah", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T09:58:51.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:50 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:51.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:50 smithi098 ceph-mon[100914]: Reconfiguring daemon mds.foo.smithi196.ugblah on smithi196 2024-09-21T09:58:52.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:52 smithi098 ceph-mon[100914]: pgmap v70: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:52.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:52.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:52.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:58:52.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:52.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:58:52.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:52.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:52 smithi098 ceph-mon[97580]: pgmap v70: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:52.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:52.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:52.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:58:52.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:52.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:58:52.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:52 smithi196 ceph-mon[84784]: pgmap v70: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:58:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:58:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:58:53.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:53.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:53.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:54.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:54 smithi098 ceph-mon[97580]: pgmap v71: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:54.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:54 smithi098 ceph-mon[100914]: pgmap v71: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:54 smithi196 ceph-mon[84784]: pgmap v71: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:54.390 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:54 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:54.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:56.083 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:55 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:55.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:56.083 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:55 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:55.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:56.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:56 smithi098 ceph-mon[97580]: pgmap v72: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:56.350 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:56 smithi098 ceph-mon[100914]: pgmap v72: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:56.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:56 smithi196 ceph-mon[84784]: pgmap v72: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:57.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:58:56 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:58:56.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:58.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:58:58 smithi098 ceph-mon[97580]: pgmap v73: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:58.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:58:58 smithi098 ceph-mon[100914]: pgmap v73: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:58.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:58:58 smithi196 ceph-mon[84784]: pgmap v73: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:00.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:00 smithi098 ceph-mon[97580]: pgmap v74: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:00.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:00 smithi098 ceph-mon[100914]: pgmap v74: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:00.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:00 smithi196 ceph-mon[84784]: pgmap v74: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:00.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:01.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:59:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:59:00] "GET /metrics HTTP/1.1" 200 33849 "" "Prometheus/2.51.0" 2024-09-21T09:59:02.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:02 smithi098 ceph-mon[97580]: pgmap v75: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:02.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:02 smithi098 ceph-mon[100914]: pgmap v75: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:02.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:02 smithi196 ceph-mon[84784]: pgmap v75: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:04.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:04 smithi098 ceph-mon[97580]: pgmap v76: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:04.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:04 smithi098 ceph-mon[100914]: pgmap v76: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:04.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:04 smithi196 ceph-mon[84784]: pgmap v76: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:04.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:04 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:04.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:06.097 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:05 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:06.097 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:05 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:05.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:06.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:06 smithi098 ceph-mon[97580]: pgmap v77: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:06.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:06 smithi098 ceph-mon[100914]: pgmap v77: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:06.362 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:06 smithi196 ceph-mon[84784]: pgmap v77: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:07.323 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:06 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:06.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:08.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:08 smithi196 ceph-mon[84784]: pgmap v78: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:08.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:08 smithi098 ceph-mon[97580]: pgmap v78: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:08.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:08 smithi098 ceph-mon[100914]: pgmap v78: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:08.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:10.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:10 smithi098 ceph-mon[97580]: pgmap v79: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:10.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:10 smithi098 ceph-mon[100914]: pgmap v79: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:10.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:10 smithi196 ceph-mon[84784]: pgmap v79: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:10.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:10.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:59:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:59:10] "GET /metrics HTTP/1.1" 200 33852 "" "Prometheus/2.51.0" 2024-09-21T09:59:12.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:12 smithi196 ceph-mon[84784]: pgmap v80: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:12.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:12 smithi098 ceph-mon[97580]: pgmap v80: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:12.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:12 smithi098 ceph-mon[100914]: pgmap v80: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:14 smithi196 ceph-mon[84784]: pgmap v81: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:14.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:14 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:14.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:14.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:14 smithi098 ceph-mon[97580]: pgmap v81: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:14.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:14 smithi098 ceph-mon[100914]: pgmap v81: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:16.111 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:15 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:15.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:16.111 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:15 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:15.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:16.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:16 smithi196 ceph-mon[84784]: pgmap v82: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:16.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:16 smithi098 ceph-mon[97580]: pgmap v82: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:16.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:16 smithi098 ceph-mon[100914]: pgmap v82: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:17.138 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:16 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:16.947Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:18.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:18 smithi196 ceph-mon[84784]: pgmap v83: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:18.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:18 smithi098 ceph-mon[97580]: pgmap v83: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:18.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:18 smithi098 ceph-mon[100914]: pgmap v83: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:20.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:20 smithi196 ceph-mon[84784]: pgmap v84: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:20.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:20.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:20.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:20 smithi098 ceph-mon[97580]: pgmap v84: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:20.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:20 smithi098 ceph-mon[100914]: pgmap v84: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:20.995 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:59:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:59:20] "GET /metrics HTTP/1.1" 200 33852 "" "Prometheus/2.51.0" 2024-09-21T09:59:21.023 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade resume' 2024-09-21T09:59:21.275 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:22.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:22 smithi098 ceph-mon[97580]: pgmap v85: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:22.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:22 smithi098 ceph-mon[100914]: pgmap v85: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:22.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:22 smithi196 ceph-mon[84784]: pgmap v85: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:23.035 INFO:teuthology.orchestra.run.smithi098.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:59:23.312 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:23 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:23.312 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:23 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:23.312 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:23 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:59:23.312 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:23 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:23.312 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:23 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:23.312 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:23 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:59:23.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:23 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:23.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:23 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:23 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:59:23.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'echo wait for all mons to upgrade' 2024-09-21T09:59:24.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:24 smithi196 ceph-mon[84784]: pgmap v86: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:24 smithi196 ceph-mon[84784]: from='client.44114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:24 smithi196 ceph-mon[84784]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:59:24.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:24 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:24.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:24.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:24 smithi098 ceph-mon[97580]: pgmap v86: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:24.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:24 smithi098 ceph-mon[97580]: from='client.44114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:24.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:24 smithi098 ceph-mon[97580]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:59:24.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:24 smithi098 ceph-mon[100914]: pgmap v86: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:24.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:24 smithi098 ceph-mon[100914]: from='client.44114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:24.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:24 smithi098 ceph-mon[100914]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:59:24.616 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:26.138 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:25 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:25.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:26.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:25 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:25.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:26.298 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:26 smithi098 ceph-mon[97580]: pgmap v87: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:26.298 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:26 smithi098 ceph-mon[100914]: pgmap v87: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:26.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:26 smithi196 ceph-mon[84784]: pgmap v87: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:26.720 INFO:teuthology.orchestra.run.smithi098.stdout:wait for all mons to upgrade 2024-09-21T09:59:27.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'while ! ceph mon versions | grep $sha1 | grep '"'"': 3'"'"' && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-21T09:59:27.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:26 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:26.947Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:28.242 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:28.367 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:28 smithi098 ceph-mon[97580]: pgmap v88: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:28.368 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:28 smithi098 ceph-mon[100914]: pgmap v88: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:28.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:28 smithi196 ceph-mon[84784]: pgmap v88: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:30.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:30.700 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:59:30] "GET /metrics HTTP/1.1" 200 33849 "" "Prometheus/2.51.0" 2024-09-21T09:59:30.929 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 3 2024-09-21T09:59:30.984 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[100914]: pgmap v89: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:30.984 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:30.984 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:30.984 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:59:30.984 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:59:30.984 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:30.985 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:59:30.985 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:59:30.985 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:59:30.985 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:30.985 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T09:59:30.985 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-21T09:59:30.985 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T09:59:30.985 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-21T09:59:30.986 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T09:59:30.986 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-21T09:59:30.986 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:59:30.986 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:30.986 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-21T09:59:30.986 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[97580]: pgmap v89: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:30.986 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:30.987 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:30.987 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:59:30.987 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:59:30.987 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:30.987 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:59:30.988 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:59:30.988 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:59:30.988 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:30.988 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T09:59:30.988 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-21T09:59:30.988 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T09:59:30.988 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-21T09:59:30.988 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T09:59:30.988 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-21T09:59:30.989 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:59:30.989 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:30.989 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-21T09:59:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:30 smithi196 ceph-mon[84784]: pgmap v89: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:59:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:59:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:59:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:59:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:59:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T09:59:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-21T09:59:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T09:59:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-21T09:59:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T09:59:31.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-21T09:59:31.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:59:31.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:31.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-21T09:59:31.593 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch ps' 2024-09-21T09:59:31.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:31 smithi098 ceph-mon[97580]: Upgrade: Setting container_image for all mon 2024-09-21T09:59:31.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:31 smithi098 ceph-mon[97580]: Upgrade: Setting container_image for all crash 2024-09-21T09:59:31.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:31 smithi098 ceph-mon[97580]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-21T09:59:31.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:31 smithi098 ceph-mon[97580]: Upgrade: osd.0 is safe to restart 2024-09-21T09:59:31.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:31 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2888914506' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T09:59:31.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:31 smithi098 ceph-mon[100914]: Upgrade: Setting container_image for all mon 2024-09-21T09:59:31.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:31 smithi098 ceph-mon[100914]: Upgrade: Setting container_image for all crash 2024-09-21T09:59:31.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:31 smithi098 ceph-mon[100914]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-21T09:59:31.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:31 smithi098 ceph-mon[100914]: Upgrade: osd.0 is safe to restart 2024-09-21T09:59:31.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:31 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2888914506' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T09:59:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:31 smithi196 ceph-mon[84784]: Upgrade: Setting container_image for all mon 2024-09-21T09:59:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:31 smithi196 ceph-mon[84784]: Upgrade: Setting container_image for all crash 2024-09-21T09:59:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:31 smithi196 ceph-mon[84784]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-21T09:59:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:31 smithi196 ceph-mon[84784]: Upgrade: osd.0 is safe to restart 2024-09-21T09:59:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:31 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2888914506' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T09:59:32.454 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:33.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:32 smithi098 ceph-mon[97580]: pgmap v90: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:33.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:32 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:33.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:32 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T09:59:33.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:32 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:59:33.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:32 smithi098 ceph-mon[100914]: pgmap v90: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:33.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:32 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:33.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:32 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T09:59:33.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:32 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:59:33.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:32 smithi196 ceph-mon[84784]: pgmap v90: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:32 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:32 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T09:59:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:32 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:59:33.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:33 smithi098 ceph-mon[97580]: Upgrade: Updating osd.0 2024-09-21T09:59:33.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:33 smithi098 ceph-mon[97580]: Deploying daemon osd.0 on smithi098 2024-09-21T09:59:33.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:33 smithi098 ceph-mon[100914]: Upgrade: Updating osd.0 2024-09-21T09:59:33.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:33 smithi098 ceph-mon[100914]: Deploying daemon osd.0 on smithi098 2024-09-21T09:59:34.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:33 smithi196 ceph-mon[84784]: Upgrade: Updating osd.0 2024-09-21T09:59:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:33 smithi196 ceph-mon[84784]: Deploying daemon osd.0 on smithi098 2024-09-21T09:59:34.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:34 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:34.146Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:34.857 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:34 smithi098 ceph-mon[97580]: pgmap v91: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:34.857 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:34 smithi098 ceph-mon[100914]: pgmap v91: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:35.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:34 smithi196 ceph-mon[84784]: pgmap v91: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:35.410 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T09:59:35.410 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (4m) 5s ago 8m 18.0M - 0.27.0 11f11916f8cd 72b57f924744 2024-09-21T09:59:35.410 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi196 *:3000 running (3m) 69s ago 8m 85.7M - 9.4.7 954c08fa6188 7724661af1a8 2024-09-21T09:59:35.410 INFO:teuthology.orchestra.run.smithi098.stdout:mds.foo.smithi098.stcwsu smithi098 running (7m) 5s ago 7m 18.8M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be d8393a5abb20 2024-09-21T09:59:35.411 INFO:teuthology.orchestra.run.smithi098.stdout:mds.foo.smithi196.ugblah smithi196 running (7m) 69s ago 7m 15.7M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 05a68593bc2b 2024-09-21T09:59:35.411 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi196 *:8443,9283,8765 running (3m) 69s ago 12m 455M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d4969d635bcd 2024-09-21T09:59:35.411 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (3m) 5s ago 13m 512M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 da66347fe5a2 2024-09-21T09:59:35.411 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (3m) 5s ago 14m 44.7M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 0a2195915f65 2024-09-21T09:59:35.411 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi196 running (75s) 69s ago 12m 20.7M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 2776b693f5c0 2024-09-21T09:59:35.411 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (94s) 5s ago 12m 35.0M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 ef4452ba8cf0 2024-09-21T09:59:35.411 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (4m) 5s ago 7m 10.8M - 1.7.0 72c9c2088986 d587eb6669b7 2024-09-21T09:59:35.411 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi196 *:9100 running (3m) 69s ago 7m 11.7M - 1.7.0 72c9c2088986 7a8564adbc71 2024-09-21T09:59:35.411 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (11m) 5s ago 11m 48.7M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be b4d47286955f 2024-09-21T09:59:35.411 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (11m) 5s ago 11m 50.5M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 850efcb386e6 2024-09-21T09:59:35.411 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (11m) 5s ago 11m 50.4M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be f19a948d9994 2024-09-21T09:59:35.412 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (10m) 5s ago 10m 46.8M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 68f6e62795e7 2024-09-21T09:59:35.412 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi196 running (10m) 69s ago 10m 48.9M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 1693914f1a8b 2024-09-21T09:59:35.412 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi196 running (10m) 69s ago 10m 46.7M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be cb8e4afb2db7 2024-09-21T09:59:35.412 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi196 running (9m) 69s ago 9m 46.6M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 8ba9647073b2 2024-09-21T09:59:35.412 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi196 running (9m) 69s ago 9m 50.7M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 540c98a3c436 2024-09-21T09:59:35.412 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi196 *:9095 running (3m) 69s ago 8m 46.6M - 2.51.0 1d3b7f56885b b30b25036dc7 2024-09-21T09:59:36.040 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:35.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:36.040 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:35.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:36.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade pause' 2024-09-21T09:59:36.699 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:37.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:36 smithi098 ceph-mon[100914]: pgmap v92: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:37.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:36 smithi098 ceph-mon[100914]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:37.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:36 smithi098 ceph-mon[97580]: pgmap v92: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:37.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:36 smithi098 ceph-mon[97580]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:37.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:36 smithi196 ceph-mon[84784]: pgmap v92: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:36 smithi196 ceph-mon[84784]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:37.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:36 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:36.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:37.722 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:37 smithi098 systemd[1]: Stopping Ceph osd.0 for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:59:38.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:38.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:38.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:38.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:38 smithi098 ceph-mon[97580]: pgmap v93: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:38.843 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:38 smithi098 ceph-mon[100914]: pgmap v93: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:39.097 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0[43805]: 2024-09-21T09:59:38.946+0000 7fed3815b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T09:59:39.097 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0[43805]: 2024-09-21T09:59:38.946+0000 7fed3815b640 -1 osd.0 82 *** Got signal Terminated *** 2024-09-21T09:59:39.097 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0[43805]: 2024-09-21T09:59:38.946+0000 7fed3815b640 -1 osd.0 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T09:59:39.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:38 smithi196 ceph-mon[84784]: pgmap v93: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:39.361 INFO:teuthology.orchestra.run.smithi098.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:59:40.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:39 smithi098 ceph-mon[97580]: osd.0 marked itself down and dead 2024-09-21T09:59:40.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:39 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:40.043 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:39 smithi098 ceph-mon[100914]: osd.0 marked itself down and dead 2024-09-21T09:59:40.043 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:39 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:40.070 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'sleep 60' 2024-09-21T09:59:40.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:39 smithi196 ceph-mon[84784]: osd.0 marked itself down and dead 2024-09-21T09:59:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:39 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:40.347 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:40 smithi098 podman[107301]: 2024-09-21 09:59:40.066995938 +0000 UTC m=+1.467062371 container died b4d47286955f60a565f64507d71fa29638fc13695661d83ce567d38c8620446e (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-fbd1a6a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-21T09:59:40.597 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:40 smithi098 podman[107301]: 2024-09-21 09:59:40.444564366 +0000 UTC m=+1.844630788 container cleanup b4d47286955f60a565f64507d71fa29638fc13695661d83ce567d38c8620446e (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=quincy-fbd1a6a, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T09:59:40.597 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:40 smithi098 bash[107301]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0 2024-09-21T09:59:40.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:40.749 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:40.923 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:59:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:59:40] "GET /metrics HTTP/1.1" 200 33848 "" "Prometheus/2.51.0" 2024-09-21T09:59:40.923 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:40 smithi098 podman[107361]: 2024-09-21 09:59:40.704267725 +0000 UTC m=+0.636566331 container remove b4d47286955f60a565f64507d71fa29638fc13695661d83ce567d38c8620446e (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=) 2024-09-21T09:59:40.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:40 smithi098 ceph-mon[97580]: pgmap v94: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:40.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:40 smithi098 ceph-mon[97580]: from='client.44132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:40.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:40 smithi098 ceph-mon[97580]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:59:40.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:40 smithi098 ceph-mon[97580]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T09:59:40.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:40 smithi098 ceph-mon[97580]: osdmap e83: 8 total, 7 up, 8 in 2024-09-21T09:59:40.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:40 smithi098 ceph-mon[100914]: pgmap v94: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:40.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:40 smithi098 ceph-mon[100914]: from='client.44132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:40.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:40 smithi098 ceph-mon[100914]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:59:40.925 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:40 smithi098 ceph-mon[100914]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T09:59:40.925 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:40 smithi098 ceph-mon[100914]: osdmap e83: 8 total, 7 up, 8 in 2024-09-21T09:59:41.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:40 smithi196 ceph-mon[84784]: pgmap v94: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:41.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:40 smithi196 ceph-mon[84784]: from='client.44132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:40 smithi196 ceph-mon[84784]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:59:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:40 smithi196 ceph-mon[84784]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T09:59:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:40 smithi196 ceph-mon[84784]: osdmap e83: 8 total, 7 up, 8 in 2024-09-21T09:59:41.597 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:41 smithi098 podman[107450]: 2024-09-21 09:59:41.249922141 +0000 UTC m=+0.492994302 container died 550f29c95e45367f9cad1fd84277dea6fa88b4477fc4d53c428bbfe7628ba4f5 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=friendly_rubin, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T09:59:41.880 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:41 smithi098 ceph-mon[97580]: osdmap e84: 8 total, 7 up, 8 in 2024-09-21T09:59:41.880 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:41 smithi098 ceph-mon[100914]: osdmap e84: 8 total, 7 up, 8 in 2024-09-21T09:59:42.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:41 smithi196 ceph-mon[84784]: osdmap e84: 8 total, 7 up, 8 in 2024-09-21T09:59:42.518 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:42 smithi098 podman[107559]: 2024-09-21 09:59:42.188797893 +0000 UTC m=+0.016906686 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:59:42.820 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:42 smithi098 ceph-mon[97580]: pgmap v97: 73 pgs: 11 stale+active+clean, 62 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:42.821 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:42 smithi098 ceph-mon[100914]: pgmap v97: 73 pgs: 11 stale+active+clean, 62 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:42.821 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:42 smithi098 podman[107559]: 2024-09-21 09:59:42.516275594 +0000 UTC m=+0.344384422 container create bb877392f01dc77e404f669e005b2640083382a68a3668a1c87ebc7cd8d482d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-21T09:59:42.821 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:42 smithi098 podman[107559]: 2024-09-21 09:59:42.727318061 +0000 UTC m=+0.555426853 container init bb877392f01dc77e404f669e005b2640083382a68a3668a1c87ebc7cd8d482d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-21T09:59:42.821 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:42 smithi098 podman[107559]: 2024-09-21 09:59:42.730779015 +0000 UTC m=+0.558887807 container start bb877392f01dc77e404f669e005b2640083382a68a3668a1c87ebc7cd8d482d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-21T09:59:42.821 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:42 smithi098 podman[107559]: 2024-09-21 09:59:42.819228583 +0000 UTC m=+0.647337389 container attach bb877392f01dc77e404f669e005b2640083382a68a3668a1c87ebc7cd8d482d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T09:59:43.097 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:42 smithi098 podman[107559]: 2024-09-21 09:59:42.922599328 +0000 UTC m=+0.750708123 container died bb877392f01dc77e404f669e005b2640083382a68a3668a1c87ebc7cd8d482d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T09:59:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:42 smithi196 ceph-mon[84784]: pgmap v97: 73 pgs: 11 stale+active+clean, 62 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:59:43.791 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:43 smithi098 podman[107587]: 2024-09-21 09:59:43.531613822 +0000 UTC m=+0.603101352 container remove bb877392f01dc77e404f669e005b2640083382a68a3668a1c87ebc7cd8d482d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-21T09:59:43.791 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:43 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.0.service: Deactivated successfully. 2024-09-21T09:59:43.791 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:43 smithi098 systemd[1]: Stopped Ceph osd.0 for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:59:43.792 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:43 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.0.service: Consumed 5.110s CPU time. 2024-09-21T09:59:43.792 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:43 smithi098 systemd[1]: Starting Ceph osd.0 for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:59:44.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:43 smithi098 ceph-mon[97580]: Health check failed: Degraded data redundancy: 13/84 objects degraded (15.476%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T09:59:44.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:43 smithi098 ceph-mon[100914]: Health check failed: Degraded data redundancy: 13/84 objects degraded (15.476%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T09:59:44.098 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:43 smithi098 podman[107687]: 2024-09-21 09:59:43.877918762 +0000 UTC m=+0.016071051 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:59:44.098 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:43 smithi098 podman[107687]: 2024-09-21 09:59:43.983071892 +0000 UTC m=+0.121224167 container create e71be189d0a582cf2da02b0584634a6e005db62985c9ca174e9d88ee735d2416 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-activate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=main-62da54b, org.label-schema.vendor=CentOS) 2024-09-21T09:59:44.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:43 smithi196 ceph-mon[84784]: Health check failed: Degraded data redundancy: 13/84 objects degraded (15.476%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T09:59:44.359 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:44 smithi098 podman[107687]: 2024-09-21 09:59:44.210732574 +0000 UTC m=+0.348884850 container init e71be189d0a582cf2da02b0584634a6e005db62985c9ca174e9d88ee735d2416 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, org.label-schema.schema-version=1.0) 2024-09-21T09:59:44.359 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:44 smithi098 podman[107687]: 2024-09-21 09:59:44.214219202 +0000 UTC m=+0.352371486 container start e71be189d0a582cf2da02b0584634a6e005db62985c9ca174e9d88ee735d2416 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-21T09:59:44.359 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:44 smithi098 podman[107687]: 2024-09-21 09:59:44.286039191 +0000 UTC m=+0.424191479 container attach e71be189d0a582cf2da02b0584634a6e005db62985c9ca174e9d88ee735d2416 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-activate, GIT_BRANCH=HEAD, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-21T09:59:44.359 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:44 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-activate[107702]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T09:59:44.359 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:44 smithi098 bash[107687]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T09:59:44.360 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:44 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-activate[107702]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T09:59:44.360 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:44 smithi098 bash[107687]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T09:59:44.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:44 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:44.146Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:45.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:44 smithi098 ceph-mon[97580]: pgmap v98: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-21T09:59:45.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:44 smithi098 ceph-mon[100914]: pgmap v98: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-21T09:59:45.097 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:44 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-activate[107702]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T09:59:45.097 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:44 smithi098 bash[107687]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T09:59:45.098 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:44 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-activate[107702]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-21T09:59:45.098 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:44 smithi098 bash[107687]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-21T09:59:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:44 smithi196 ceph-mon[84784]: pgmap v98: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-21T09:59:45.416 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:45 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-activate[107702]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-21T09:59:45.416 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:45 smithi098 bash[107687]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-21T09:59:45.416 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:45 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-activate[107702]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-21T09:59:45.416 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:45 smithi098 bash[107687]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-21T09:59:45.416 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:45 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-activate[107702]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-21T09:59:45.416 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:45 smithi098 bash[107687]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-21T09:59:45.417 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:45 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-activate[107702]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T09:59:45.417 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:45 smithi098 bash[107687]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T09:59:45.417 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:45 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-activate[107702]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-21T09:59:45.417 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:45 smithi098 bash[107687]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-21T09:59:45.417 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:45 smithi098 podman[107687]: 2024-09-21 09:59:45.126995789 +0000 UTC m=+1.265148077 container died e71be189d0a582cf2da02b0584634a6e005db62985c9ca174e9d88ee735d2416 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-activate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=main-62da54b, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True) 2024-09-21T09:59:46.020 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:45 smithi098 podman[107827]: 2024-09-21 09:59:45.681814436 +0000 UTC m=+0.544024941 container remove e71be189d0a582cf2da02b0584634a6e005db62985c9ca174e9d88ee735d2416 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-activate, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-21T09:59:46.020 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:45 smithi098 podman[107885]: 2024-09-21 09:59:45.790697327 +0000 UTC m=+0.015498605 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:59:46.020 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:45 smithi098 podman[107885]: 2024-09-21 09:59:45.89102311 +0000 UTC m=+0.115824415 container create 9c8da4094eee81a38b2328339c2d8d60a0a33c15f3e0d571d5680b262ce64507 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-21T09:59:46.138 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:45.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:46.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:45 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:45.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:46.597 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:46 smithi098 podman[107885]: 2024-09-21 09:59:46.160140674 +0000 UTC m=+0.384941947 container init 9c8da4094eee81a38b2328339c2d8d60a0a33c15f3e0d571d5680b262ce64507 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-21T09:59:46.597 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:46 smithi098 podman[107885]: 2024-09-21 09:59:46.163373624 +0000 UTC m=+0.388174886 container start 9c8da4094eee81a38b2328339c2d8d60a0a33c15f3e0d571d5680b262ce64507 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-21T09:59:46.597 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:46 smithi098 ceph-osd[107904]: -- 172.21.15.98:0/3546490260 <== mon.2 v2:172.21.15.98:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e30fa4da40 con 0x55e31081e400 2024-09-21T09:59:46.597 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:46 smithi098 bash[107885]: 9c8da4094eee81a38b2328339c2d8d60a0a33c15f3e0d571d5680b262ce64507 2024-09-21T09:59:46.598 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:46 smithi098 systemd[1]: Started Ceph osd.0 for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:59:47.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:46 smithi098 ceph-mon[97580]: pgmap v99: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-21T09:59:47.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:47.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:47.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:59:47.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:46 smithi098 ceph-mon[100914]: pgmap v99: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-21T09:59:47.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:47.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:47.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:59:47.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:46 smithi196 ceph-mon[84784]: pgmap v99: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-21T09:59:47.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:47.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:47.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:59:47.116 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:46 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:46.947Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:48.069 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:47 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0[107900]: 2024-09-21T09:59:47.798+0000 7fb645f5f640 -1 Falling back to public interface 2024-09-21T09:59:49.074 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:48 smithi196 ceph-mon[84784]: pgmap v100: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-21T09:59:49.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:48 smithi098 ceph-mon[97580]: pgmap v100: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-21T09:59:49.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:48 smithi098 ceph-mon[100914]: pgmap v100: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-21T09:59:50.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:51.015 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 09:59:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:09:59:50] "GET /metrics HTTP/1.1" 200 33848 "" "Prometheus/2.51.0" 2024-09-21T09:59:51.015 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:50 smithi098 ceph-mon[97580]: pgmap v101: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-21T09:59:51.016 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:50 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:51.016 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:50 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:51.016 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:50 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:51.016 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:50 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:51.016 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:50 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:51.016 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:50 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:51.016 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:50 smithi098 ceph-mon[100914]: pgmap v101: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-21T09:59:51.016 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:50 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:51.016 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:50 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:51.016 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:50 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:51.016 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:50 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:51.017 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:50 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:51.017 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:50 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:51.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:50 smithi196 ceph-mon[84784]: pgmap v101: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-21T09:59:51.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:50 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:51.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:50 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:50 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:50 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:50 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:50 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:51.347 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:51 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0[107900]: 2024-09-21T09:59:51.133+0000 7fb645f5f640 -1 osd.0 82 log_to_monitors true 2024-09-21T09:59:52.030 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:51 smithi098 ceph-mon[97580]: from='osd.0 [v2:172.21.15.98:6802/1869651717,v1:172.21.15.98:6803/1869651717]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T09:59:52.030 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:51 smithi098 ceph-mon[100914]: from='osd.0 [v2:172.21.15.98:6802/1869651717,v1:172.21.15.98:6803/1869651717]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T09:59:52.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:51 smithi196 ceph-mon[84784]: from='osd.0 [v2:172.21.15.98:6802/1869651717,v1:172.21.15.98:6803/1869651717]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T09:59:52.315 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 09:59:52 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0[107900]: 2024-09-21T09:59:52.257+0000 7fb639d01640 -1 osd.0 82 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:59:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:52 smithi098 ceph-mon[97580]: pgmap v102: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-21T09:59:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:52 smithi098 ceph-mon[97580]: from='osd.0 [v2:172.21.15.98:6802/1869651717,v1:172.21.15.98:6803/1869651717]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T09:59:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:52 smithi098 ceph-mon[97580]: osdmap e85: 8 total, 7 up, 8 in 2024-09-21T09:59:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:52 smithi098 ceph-mon[97580]: from='osd.0 [v2:172.21.15.98:6802/1869651717,v1:172.21.15.98:6803/1869651717]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]} : dispatch 2024-09-21T09:59:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:52 smithi098 ceph-mon[97580]: from='osd.0 [v2:172.21.15.98:6802/1869651717,v1:172.21.15.98:6803/1869651717]' entity='osd.0' 2024-09-21T09:59:53.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:52 smithi098 ceph-mon[100914]: pgmap v102: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-21T09:59:53.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:52 smithi098 ceph-mon[100914]: from='osd.0 [v2:172.21.15.98:6802/1869651717,v1:172.21.15.98:6803/1869651717]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T09:59:53.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:52 smithi098 ceph-mon[100914]: osdmap e85: 8 total, 7 up, 8 in 2024-09-21T09:59:53.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:52 smithi098 ceph-mon[100914]: from='osd.0 [v2:172.21.15.98:6802/1869651717,v1:172.21.15.98:6803/1869651717]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]} : dispatch 2024-09-21T09:59:53.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:53.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:53.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:52 smithi098 ceph-mon[100914]: from='osd.0 [v2:172.21.15.98:6802/1869651717,v1:172.21.15.98:6803/1869651717]' entity='osd.0' 2024-09-21T09:59:53.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:52 smithi196 ceph-mon[84784]: pgmap v102: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-21T09:59:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:52 smithi196 ceph-mon[84784]: from='osd.0 [v2:172.21.15.98:6802/1869651717,v1:172.21.15.98:6803/1869651717]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T09:59:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:52 smithi196 ceph-mon[84784]: osdmap e85: 8 total, 7 up, 8 in 2024-09-21T09:59:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:52 smithi196 ceph-mon[84784]: from='osd.0 [v2:172.21.15.98:6802/1869651717,v1:172.21.15.98:6803/1869651717]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]} : dispatch 2024-09-21T09:59:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:52 smithi196 ceph-mon[84784]: from='osd.0 [v2:172.21.15.98:6802/1869651717,v1:172.21.15.98:6803/1869651717]' entity='osd.0' 2024-09-21T09:59:54.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[97580]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T09:59:54.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[97580]: osd.0 [v2:172.21.15.98:6802/1869651717,v1:172.21.15.98:6803/1869651717] boot 2024-09-21T09:59:54.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[97580]: osdmap e86: 8 total, 8 up, 8 in 2024-09-21T09:59:54.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T09:59:54.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:54.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:54.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:59:54.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:59:54.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:54.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:59:54.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:59:54.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:59:54.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:54.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[100914]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T09:59:54.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[100914]: osd.0 [v2:172.21.15.98:6802/1869651717,v1:172.21.15.98:6803/1869651717] boot 2024-09-21T09:59:54.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[100914]: osdmap e86: 8 total, 8 up, 8 in 2024-09-21T09:59:54.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T09:59:54.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:54.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:54.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:59:54.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:59:54.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:54.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:59:54.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:59:54.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:59:54.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:54.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:53 smithi196 ceph-mon[84784]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T09:59:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:53 smithi196 ceph-mon[84784]: osd.0 [v2:172.21.15.98:6802/1869651717,v1:172.21.15.98:6803/1869651717] boot 2024-09-21T09:59:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:53 smithi196 ceph-mon[84784]: osdmap e86: 8 total, 8 up, 8 in 2024-09-21T09:59:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T09:59:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:59:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:59:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:59:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:59:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:59:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T09:59:54.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:54 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:54.146Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:55.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:54 smithi098 ceph-mon[97580]: pgmap v104: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-21T09:59:55.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:54 smithi098 ceph-mon[97580]: osdmap e87: 8 total, 8 up, 8 in 2024-09-21T09:59:55.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:54 smithi098 ceph-mon[100914]: pgmap v104: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-21T09:59:55.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:54 smithi098 ceph-mon[100914]: osdmap e87: 8 total, 8 up, 8 in 2024-09-21T09:59:55.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:54 smithi196 ceph-mon[84784]: pgmap v104: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-21T09:59:55.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:54 smithi196 ceph-mon[84784]: osdmap e87: 8 total, 8 up, 8 in 2024-09-21T09:59:56.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:55 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:55.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:56.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:55 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:55.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:57.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:56 smithi098 ceph-mon[97580]: pgmap v107: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-21T09:59:57.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:56 smithi098 ceph-mon[100914]: pgmap v107: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-21T09:59:57.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:56 smithi196 ceph-mon[84784]: pgmap v107: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-21T09:59:57.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 09:59:56 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T09:59:56.947Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:58.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:57 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 10/84 objects degraded (11.905%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:59:58.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:57 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 10/84 objects degraded (11.905%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:59:58.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:57 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 10/84 objects degraded (11.905%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:59:59.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:58 smithi196 ceph-mon[84784]: pgmap v108: 73 pgs: 9 active+undersized, 6 active+undersized+degraded, 58 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 10/84 objects degraded (11.905%) 2024-09-21T09:59:59.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:58 smithi098 ceph-mon[97580]: pgmap v108: 73 pgs: 9 active+undersized, 6 active+undersized+degraded, 58 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 10/84 objects degraded (11.905%) 2024-09-21T09:59:59.249 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:58 smithi098 ceph-mon[100914]: pgmap v108: 73 pgs: 9 active+undersized, 6 active+undersized+degraded, 58 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 10/84 objects degraded (11.905%) 2024-09-21T10:00:00.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:59 smithi196 ceph-mon[84784]: pgmap v109: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:00.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:59 smithi196 ceph-mon[84784]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/84 objects degraded (11.905%), 6 pgs degraded) 2024-09-21T10:00:00.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 09:59:59 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:00:00.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:59 smithi098 ceph-mon[97580]: pgmap v109: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:00.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:59 smithi098 ceph-mon[97580]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/84 objects degraded (11.905%), 6 pgs degraded) 2024-09-21T10:00:00.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 09:59:59 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:00:00.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:59 smithi098 ceph-mon[100914]: pgmap v109: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:00.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:59 smithi098 ceph-mon[100914]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/84 objects degraded (11.905%), 6 pgs degraded) 2024-09-21T10:00:00.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 09:59:59 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:00:00.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:00:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:00:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:00.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:00:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:00:00] "GET /metrics HTTP/1.1" 200 33833 "" "Prometheus/2.51.0" 2024-09-21T10:00:01.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:00 smithi098 ceph-mon[97580]: overall HEALTH_OK 2024-09-21T10:00:01.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:00 smithi098 ceph-mon[100914]: overall HEALTH_OK 2024-09-21T10:00:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:00 smithi196 ceph-mon[84784]: overall HEALTH_OK 2024-09-21T10:00:02.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:01 smithi098 ceph-mon[97580]: pgmap v110: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:02.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:01 smithi098 ceph-mon[100914]: pgmap v110: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:02.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:01 smithi196 ceph-mon[84784]: pgmap v110: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:04.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:04 smithi098 ceph-mon[97580]: pgmap v111: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:04.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:04 smithi098 ceph-mon[100914]: pgmap v111: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:04.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:04 smithi196 ceph-mon[84784]: pgmap v111: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:04.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:00:04 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:00:04.146Z caller=alerting.go:391 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-09-21T10:00:04.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:00:04 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:00:04.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:06.016 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:00:05 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:00:05.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:06.016 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:00:05 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:00:05.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:06.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:06 smithi098 ceph-mon[97580]: pgmap v112: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:06.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:06 smithi098 ceph-mon[100914]: pgmap v112: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:06.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:06 smithi196 ceph-mon[84784]: pgmap v112: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:07.287 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:00:06 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:00:06.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:08 smithi098 ceph-mon[97580]: pgmap v113: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:00:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:00:08.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:08 smithi098 ceph-mon[100914]: pgmap v113: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:08.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:00:08.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:08.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:00:08.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:08 smithi196 ceph-mon[84784]: pgmap v113: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:08.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:00:08.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:08.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:00:10.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:10 smithi098 ceph-mon[97580]: pgmap v114: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:10.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:10 smithi098 ceph-mon[100914]: pgmap v114: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:10.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:10 smithi196 ceph-mon[84784]: pgmap v114: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:10.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:00:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:00:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:10.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:00:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:00:10] "GET /metrics HTTP/1.1" 200 33979 "" "Prometheus/2.51.0" 2024-09-21T10:00:12.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:12 smithi098 ceph-mon[97580]: pgmap v115: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:12.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:12 smithi098 ceph-mon[100914]: pgmap v115: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:12.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:12 smithi196 ceph-mon[84784]: pgmap v115: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:14.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:14 smithi098 ceph-mon[97580]: pgmap v116: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:14.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:14 smithi098 ceph-mon[100914]: pgmap v116: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:14.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:14 smithi196 ceph-mon[84784]: pgmap v116: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:14.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:00:14 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:00:14.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.1\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.1\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.1\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:16.138 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:00:15 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:00:15.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:16.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:00:15 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:00:15.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:16.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:16 smithi098 ceph-mon[97580]: pgmap v117: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:16.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:16 smithi098 ceph-mon[100914]: pgmap v117: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:16.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:16 smithi196 ceph-mon[84784]: pgmap v117: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:17.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:00:16 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:00:16.947Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:18.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:18 smithi098 ceph-mon[97580]: pgmap v118: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:18.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:18 smithi098 ceph-mon[100914]: pgmap v118: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:18.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:18 smithi196 ceph-mon[84784]: pgmap v118: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:20.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:20 smithi098 ceph-mon[97580]: pgmap v119: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:20.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:20 smithi098 ceph-mon[100914]: pgmap v119: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:20 smithi196 ceph-mon[84784]: pgmap v119: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:20.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:00:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:00:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:21.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:00:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:00:20] "GET /metrics HTTP/1.1" 200 33979 "" "Prometheus/2.51.0" 2024-09-21T10:00:22.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:22 smithi098 ceph-mon[97580]: pgmap v120: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:22.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:22.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:22 smithi098 ceph-mon[100914]: pgmap v120: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:22.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:22.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:22 smithi196 ceph-mon[84784]: pgmap v120: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:24.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:24 smithi098 ceph-mon[97580]: pgmap v121: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:24.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:24 smithi098 ceph-mon[100914]: pgmap v121: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:24.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:24 smithi196 ceph-mon[84784]: pgmap v121: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:24.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:00:24 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:00:24.146Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.1\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.1\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.1\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:26.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:00:25 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:00:25.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:26.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:00:25 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:00:25.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:26.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:26 smithi098 ceph-mon[97580]: pgmap v122: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:26.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:26 smithi098 ceph-mon[100914]: pgmap v122: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:26.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:26 smithi196 ceph-mon[84784]: pgmap v122: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:28.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:28 smithi098 ceph-mon[97580]: pgmap v123: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:28.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:28 smithi098 ceph-mon[100914]: pgmap v123: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:28.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:28 smithi196 ceph-mon[84784]: pgmap v123: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:30.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:30 smithi098 ceph-mon[97580]: pgmap v124: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:30.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:30 smithi098 ceph-mon[100914]: pgmap v124: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:30 smithi196 ceph-mon[84784]: pgmap v124: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:30.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:00:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:00:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:31.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:00:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:00:30] "GET /metrics HTTP/1.1" 200 33907 "" "Prometheus/2.51.0" 2024-09-21T10:00:32.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:32 smithi098 ceph-mon[97580]: pgmap v125: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:32.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:32 smithi098 ceph-mon[100914]: pgmap v125: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:32.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:32 smithi196 ceph-mon[84784]: pgmap v125: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:34.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:34 smithi098 ceph-mon[97580]: pgmap v126: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:34.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:34 smithi098 ceph-mon[100914]: pgmap v126: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:34.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:34 smithi196 ceph-mon[84784]: pgmap v126: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:36.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:36 smithi098 ceph-mon[97580]: pgmap v127: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:36.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:36 smithi098 ceph-mon[100914]: pgmap v127: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:36.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:36 smithi196 ceph-mon[84784]: pgmap v127: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:37.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:37.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:38.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:38 smithi098 ceph-mon[97580]: pgmap v128: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:38.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:38 smithi098 ceph-mon[100914]: pgmap v128: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:38.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:38 smithi196 ceph-mon[84784]: pgmap v128: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:40 smithi098 ceph-mon[97580]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:40.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:40 smithi098 ceph-mon[100914]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:40.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:40 smithi196 ceph-mon[84784]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:40.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:00:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:00:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:41.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:00:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:00:40] "GET /metrics HTTP/1.1" 200 33973 "" "Prometheus/2.51.0" 2024-09-21T10:00:42.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:42 smithi098 ceph-mon[97580]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:42.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:42 smithi098 ceph-mon[100914]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:42.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:42 smithi196 ceph-mon[84784]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:43.003 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade resume' 2024-09-21T10:00:43.254 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:44.548 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:44 smithi098 ceph-mon[97580]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:44.548 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:44 smithi098 ceph-mon[100914]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:44.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:44 smithi196 ceph-mon[84784]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:45.232 INFO:teuthology.orchestra.run.smithi098.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:00:45.888 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'echo wait for half of osds to upgrade' 2024-09-21T10:00:46.439 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[97580]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:46.439 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[97580]: from='client.44144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:46.439 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[97580]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:00:46.439 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:00:46.439 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:00:46.439 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:00:46.439 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:00:46.440 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:00:46.440 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:00:46.440 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:00:46.440 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:00:46.440 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:00:46.440 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-21T10:00:46.440 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[97580]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-21T10:00:46.440 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[97580]: Upgrade: osd.1 is safe to restart 2024-09-21T10:00:46.440 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[100914]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:46.440 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[100914]: from='client.44144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:46.441 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[100914]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:00:46.441 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:00:46.441 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:00:46.441 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:00:46.441 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:00:46.441 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:00:46.441 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:00:46.441 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:00:46.441 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:00:46.441 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:00:46.441 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-21T10:00:46.441 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[100914]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-21T10:00:46.441 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:46 smithi098 ceph-mon[100914]: Upgrade: osd.1 is safe to restart 2024-09-21T10:00:46.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:46 smithi196 ceph-mon[84784]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:46 smithi196 ceph-mon[84784]: from='client.44144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:46 smithi196 ceph-mon[84784]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:00:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:00:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:00:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:00:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:00:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:00:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:00:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:00:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:00:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:00:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-21T10:00:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:46 smithi196 ceph-mon[84784]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-21T10:00:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:46 smithi196 ceph-mon[84784]: Upgrade: osd.1 is safe to restart 2024-09-21T10:00:47.209 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:48.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:48 smithi098 ceph-mon[97580]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:48.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:48 smithi098 ceph-mon[97580]: Upgrade: Updating osd.1 2024-09-21T10:00:48.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:48 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:00:48.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:48 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T10:00:48.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:48 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:00:48.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:48 smithi098 ceph-mon[97580]: Deploying daemon osd.1 on smithi098 2024-09-21T10:00:48.282 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:48 smithi098 ceph-mon[100914]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:48.282 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:48 smithi098 ceph-mon[100914]: Upgrade: Updating osd.1 2024-09-21T10:00:48.283 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:48 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:00:48.283 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:48 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T10:00:48.283 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:48 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:00:48.283 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:48 smithi098 ceph-mon[100914]: Deploying daemon osd.1 on smithi098 2024-09-21T10:00:48.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:48 smithi196 ceph-mon[84784]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:48 smithi196 ceph-mon[84784]: Upgrade: Updating osd.1 2024-09-21T10:00:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:48 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:00:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:48 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T10:00:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:48 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:00:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:48 smithi196 ceph-mon[84784]: Deploying daemon osd.1 on smithi098 2024-09-21T10:00:49.522 INFO:teuthology.orchestra.run.smithi098.stdout:wait for half of osds to upgrade 2024-09-21T10:00:50.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:50 smithi098 ceph-mon[97580]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:50.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:50 smithi098 ceph-mon[100914]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:50.596 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'while ! ceph osd versions | grep $sha1 | egrep '"'"': [45678]'"'"' && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-21T10:00:50.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:50 smithi196 ceph-mon[84784]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:50.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:00:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:00:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:50.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:00:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:00:50] "GET /metrics HTTP/1.1" 200 33973 "" "Prometheus/2.51.0" 2024-09-21T10:00:51.113 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:52.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:52 smithi098 ceph-mon[97580]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:52.284 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:52 smithi098 ceph-mon[100914]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:52.597 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:52 smithi098 systemd[1]: Stopping Ceph osd.1 for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:00:52.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:52 smithi196 ceph-mon[84784]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:53.530 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:53.530 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:53.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:53.847 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:00:53.847 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:53 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1[49687]: 2024-09-21T10:00:53.527+0000 7f75784a2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:00:53.847 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:53 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1[49687]: 2024-09-21T10:00:53.527+0000 7f75784a2640 -1 osd.1 87 *** Got signal Terminated *** 2024-09-21T10:00:53.847 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:53 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1[49687]: 2024-09-21T10:00:53.527+0000 7f75784a2640 -1 osd.1 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:00:54.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:54 smithi098 ceph-mon[97580]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:54.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:54 smithi098 ceph-mon[97580]: osd.1 marked itself down and dead 2024-09-21T10:00:54.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:54 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1507833002' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:00:54.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:54 smithi098 ceph-mon[97580]: from='client.44156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:54.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:54 smithi098 ceph-mon[100914]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:54.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:54 smithi098 ceph-mon[100914]: osd.1 marked itself down and dead 2024-09-21T10:00:54.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:54 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1507833002' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:00:54.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:54 smithi098 ceph-mon[100914]: from='client.44156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:54 smithi196 ceph-mon[84784]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:54 smithi196 ceph-mon[84784]: osd.1 marked itself down and dead 2024-09-21T10:00:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:54 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1507833002' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:00:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:54 smithi196 ceph-mon[84784]: from='client.44156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:55.005 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:54 smithi098 podman[112344]: 2024-09-21 10:00:54.707909664 +0000 UTC m=+1.285113082 container died 850efcb386e62215d06862500e4cf43b94818b84fab912b5108c3362780b9fca (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=quincy-fbd1a6a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-21T10:00:55.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:55 smithi098 ceph-mon[100914]: from='client.44162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:55.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:55 smithi098 ceph-mon[100914]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:00:55.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:55 smithi098 ceph-mon[100914]: osdmap e88: 8 total, 7 up, 8 in 2024-09-21T10:00:55.347 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:55 smithi098 podman[112344]: 2024-09-21 10:00:55.194370767 +0000 UTC m=+1.771574184 container cleanup 850efcb386e62215d06862500e4cf43b94818b84fab912b5108c3362780b9fca (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-21T10:00:55.348 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:55 smithi098 bash[112344]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1 2024-09-21T10:00:55.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:55 smithi098 ceph-mon[97580]: from='client.44162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:55.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:55 smithi098 ceph-mon[97580]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:00:55.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:55 smithi098 ceph-mon[97580]: osdmap e88: 8 total, 7 up, 8 in 2024-09-21T10:00:55.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:55 smithi196 ceph-mon[84784]: from='client.44162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:55.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:55 smithi196 ceph-mon[84784]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:00:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:55 smithi196 ceph-mon[84784]: osdmap e88: 8 total, 7 up, 8 in 2024-09-21T10:00:55.801 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:55 smithi098 podman[112410]: 2024-09-21 10:00:55.479399739 +0000 UTC m=+0.772220124 container remove 850efcb386e62215d06862500e4cf43b94818b84fab912b5108c3362780b9fca (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=quincy-fbd1a6a, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:00:56.097 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:55 smithi098 podman[112500]: 2024-09-21 10:00:55.702473681 +0000 UTC m=+0.016325667 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:00:56.097 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:55 smithi098 podman[112500]: 2024-09-21 10:00:55.822087279 +0000 UTC m=+0.135939252 container create 17f9e66458f2037268dd82977f25f454694995823b36fa9ad0d3fa930778b065 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-21T10:00:56.097 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:56 smithi098 podman[112500]: 2024-09-21 10:00:56.042097452 +0000 UTC m=+0.355949438 container init 17f9e66458f2037268dd82977f25f454694995823b36fa9ad0d3fa930778b065 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-21T10:00:56.097 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:56 smithi098 podman[112500]: 2024-09-21 10:00:56.045502078 +0000 UTC m=+0.359354055 container start 17f9e66458f2037268dd82977f25f454694995823b36fa9ad0d3fa930778b065 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-21T10:00:56.428 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:56 smithi098 ceph-mon[97580]: pgmap v138: 73 pgs: 8 stale+active+clean, 65 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:56.428 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:56 smithi098 ceph-mon[97580]: osdmap e89: 8 total, 7 up, 8 in 2024-09-21T10:00:56.429 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:56 smithi098 ceph-mon[100914]: pgmap v138: 73 pgs: 8 stale+active+clean, 65 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:56.429 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:56 smithi098 ceph-mon[100914]: osdmap e89: 8 total, 7 up, 8 in 2024-09-21T10:00:56.429 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:56 smithi098 podman[112500]: 2024-09-21 10:00:56.133489291 +0000 UTC m=+0.447341281 container attach 17f9e66458f2037268dd82977f25f454694995823b36fa9ad0d3fa930778b065 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:00:56.429 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:56 smithi098 podman[112500]: 2024-09-21 10:00:56.238547874 +0000 UTC m=+0.552399858 container died 17f9e66458f2037268dd82977f25f454694995823b36fa9ad0d3fa930778b065 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-21T10:00:56.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:56 smithi196 ceph-mon[84784]: pgmap v138: 73 pgs: 8 stale+active+clean, 65 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:00:56.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:56 smithi196 ceph-mon[84784]: osdmap e89: 8 total, 7 up, 8 in 2024-09-21T10:00:56.714 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:00:57.070 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:56 smithi098 podman[112544]: 2024-09-21 10:00:56.837665944 +0000 UTC m=+0.592906642 container remove 17f9e66458f2037268dd82977f25f454694995823b36fa9ad0d3fa930778b065 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-21T10:00:57.071 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:56 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.1.service: Deactivated successfully. 2024-09-21T10:00:57.071 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:56 smithi098 systemd[1]: Stopped Ceph osd.1 for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:00:57.071 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:56 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.1.service: Consumed 6.135s CPU time. 2024-09-21T10:00:57.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:57 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1384513084' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:00:57.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:57 smithi098 ceph-mon[97580]: from='client.44174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:57.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:57 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1384513084' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:00:57.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:57 smithi098 ceph-mon[100914]: from='client.44174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:57.348 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:57 smithi098 systemd[1]: Starting Ceph osd.1 for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:00:57.348 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:57 smithi098 podman[112693]: 2024-09-21 10:00:57.180693556 +0000 UTC m=+0.016461706 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:00:57.348 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:57 smithi098 podman[112693]: 2024-09-21 10:00:57.297318593 +0000 UTC m=+0.133086785 container create 15d5c94f619a7065377ebc6fb1615ec904d1e3914fe7e562ca44fc67183239c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-activate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, RELEASE=main-62da54b, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-21T10:00:57.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:57 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1384513084' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:00:57.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:57 smithi196 ceph-mon[84784]: from='client.44174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:57.684 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:57 smithi098 podman[112693]: 2024-09-21 10:00:57.533512407 +0000 UTC m=+0.369280571 container init 15d5c94f619a7065377ebc6fb1615ec904d1e3914fe7e562ca44fc67183239c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-activate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-21T10:00:57.684 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:57 smithi098 podman[112693]: 2024-09-21 10:00:57.537952541 +0000 UTC m=+0.373720694 container start 15d5c94f619a7065377ebc6fb1615ec904d1e3914fe7e562ca44fc67183239c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-activate, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T10:00:57.684 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:57 smithi098 podman[112693]: 2024-09-21 10:00:57.625159674 +0000 UTC m=+0.460927839 container attach 15d5c94f619a7065377ebc6fb1615ec904d1e3914fe7e562ca44fc67183239c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, RELEASE=main-62da54b, org.label-schema.license=GPLv2) 2024-09-21T10:00:57.684 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:57 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-activate[112709]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:00:57.684 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:57 smithi098 bash[112693]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:00:57.685 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:57 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-activate[112709]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:00:57.685 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:57 smithi098 bash[112693]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:00:58.426 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:58 smithi098 ceph-mon[97580]: pgmap v140: 73 pgs: 2 active+undersized, 7 stale+active+clean, 2 active+undersized+degraded, 62 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2/84 objects degraded (2.381%) 2024-09-21T10:00:58.426 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:58 smithi098 ceph-mon[97580]: from='client.44180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:58.426 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:00:58 smithi098 ceph-mon[97580]: Health check failed: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:00:58.426 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:58 smithi098 ceph-mon[100914]: pgmap v140: 73 pgs: 2 active+undersized, 7 stale+active+clean, 2 active+undersized+degraded, 62 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2/84 objects degraded (2.381%) 2024-09-21T10:00:58.426 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:58 smithi098 ceph-mon[100914]: from='client.44180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:58.426 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:00:58 smithi098 ceph-mon[100914]: Health check failed: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:00:58.427 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-activate[112709]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T10:00:58.427 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:58 smithi098 bash[112693]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T10:00:58.427 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-activate[112709]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-21T10:00:58.427 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:58 smithi098 bash[112693]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-21T10:00:58.427 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-activate[112709]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-21T10:00:58.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:58 smithi196 ceph-mon[84784]: pgmap v140: 73 pgs: 2 active+undersized, 7 stale+active+clean, 2 active+undersized+degraded, 62 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2/84 objects degraded (2.381%) 2024-09-21T10:00:58.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:58 smithi196 ceph-mon[84784]: from='client.44180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:58.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:00:58 smithi196 ceph-mon[84784]: Health check failed: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:00:58.796 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:58 smithi098 bash[112693]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-21T10:00:58.796 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-activate[112709]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-21T10:00:58.796 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:58 smithi098 bash[112693]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-21T10:00:58.797 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-activate[112709]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-21T10:00:58.797 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:58 smithi098 bash[112693]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-21T10:00:58.797 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-activate[112709]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T10:00:58.797 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:58 smithi098 bash[112693]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T10:00:58.797 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-activate[112709]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-21T10:00:58.797 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:58 smithi098 bash[112693]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-21T10:00:58.797 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:58 smithi098 podman[112693]: 2024-09-21 10:00:58.452871916 +0000 UTC m=+1.288640074 container died 15d5c94f619a7065377ebc6fb1615ec904d1e3914fe7e562ca44fc67183239c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-activate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-62da54b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-21T10:00:59.517 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:59 smithi098 podman[112850]: 2024-09-21 10:00:59.179796843 +0000 UTC m=+0.720816036 container remove 15d5c94f619a7065377ebc6fb1615ec904d1e3914fe7e562ca44fc67183239c0 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-activate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True) 2024-09-21T10:00:59.517 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:59 smithi098 podman[112930]: 2024-09-21 10:00:59.288626271 +0000 UTC m=+0.016195441 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:00:59.517 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:59 smithi098 podman[112930]: 2024-09-21 10:00:59.414478109 +0000 UTC m=+0.142047278 container create c84cfb0f2fd7a854e6c42a645b8b3b6c4ad3b416379d92322e82c24bc587f4ea (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-62da54b) 2024-09-21T10:00:59.604 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:00:59.847 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:59 smithi098 podman[112930]: 2024-09-21 10:00:59.616872315 +0000 UTC m=+0.344441480 container init c84cfb0f2fd7a854e6c42a645b8b3b6c4ad3b416379d92322e82c24bc587f4ea (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-21T10:00:59.847 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:59 smithi098 podman[112930]: 2024-09-21 10:00:59.620096504 +0000 UTC m=+0.347665673 container start c84cfb0f2fd7a854e6c42a645b8b3b6c4ad3b416379d92322e82c24bc587f4ea (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:00:59.847 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:59 smithi098 ceph-osd[112978]: -- 172.21.15.98:0/135262431 <== mon.2 v2:172.21.15.98:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dbc2d6fa40 con 0x55dbc3b3e000 2024-09-21T10:00:59.847 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:59 smithi098 bash[112930]: c84cfb0f2fd7a854e6c42a645b8b3b6c4ad3b416379d92322e82c24bc587f4ea 2024-09-21T10:00:59.847 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:00:59 smithi098 systemd[1]: Started Ceph osd.1 for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:01:00.463 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:00 smithi098 ceph-mon[97580]: pgmap v141: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-21T10:01:00.463 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:00 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2082588312' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:00.463 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:00 smithi098 ceph-mon[97580]: from='client.44192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:00.463 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:00 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:00.464 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:00 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:00.464 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:00 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:01:00.464 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:00 smithi098 ceph-mon[97580]: from='client.34329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:00.464 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:00 smithi098 ceph-mon[100914]: pgmap v141: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-21T10:01:00.464 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:00 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2082588312' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:00.464 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:00 smithi098 ceph-mon[100914]: from='client.44192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:00.464 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:00 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:00.464 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:00 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:00.464 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:00 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:01:00.464 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:00 smithi098 ceph-mon[100914]: from='client.34329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:00.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:00 smithi196 ceph-mon[84784]: pgmap v141: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-21T10:01:00.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:00 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2082588312' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:00.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:00 smithi196 ceph-mon[84784]: from='client.44192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:00.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:00 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:00.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:00 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:00.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:00 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:01:00.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:00 smithi196 ceph-mon[84784]: from='client.34329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:00.574 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:01:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:01:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:00.723 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:01:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:01:00] "GET /metrics HTTP/1.1" 200 33975 "" "Prometheus/2.51.0" 2024-09-21T10:01:01.295 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:01:01 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1[112969]: 2024-09-21T10:01:01.035+0000 7f0efe3e1640 -1 Falling back to public interface 2024-09-21T10:01:02.365 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:02 smithi098 ceph-mon[97580]: pgmap v142: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-21T10:01:02.365 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:02 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3213982725' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:02.365 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:02 smithi098 ceph-mon[100914]: pgmap v142: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-21T10:01:02.365 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:02 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3213982725' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:02.491 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:01:02.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:02 smithi196 ceph-mon[84784]: pgmap v142: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-21T10:01:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:02 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3213982725' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:03.504 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:03 smithi196 ceph-mon[84784]: from='client.34341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:03.505 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:03 smithi196 ceph-mon[84784]: from='client.44219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:03.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:03 smithi098 ceph-mon[97580]: from='client.34341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:03.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:03 smithi098 ceph-mon[97580]: from='client.44219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:03.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:03 smithi098 ceph-mon[100914]: from='client.34341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:03.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:03 smithi098 ceph-mon[100914]: from='client.44219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:04.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:04 smithi098 ceph-mon[97580]: pgmap v143: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-21T10:01:04.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:04 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:04.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:04 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:04.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:04 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:04.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:04 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:04.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:04 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:04.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:04 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:04.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:04 smithi098 ceph-mon[100914]: pgmap v143: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-21T10:01:04.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:04 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:04.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:04 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:04.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:04 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:04.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:04 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:04.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:04 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:04.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:04 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:04 smithi196 ceph-mon[84784]: pgmap v143: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-21T10:01:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:04 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:04 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:04 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:04 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:04 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:04 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:05.097 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:01:04 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1[112969]: 2024-09-21T10:01:04.601+0000 7f0efe3e1640 -1 osd.1 87 log_to_monitors true 2024-09-21T10:01:05.351 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:01:05.720 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:01:05 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1[112969]: 2024-09-21T10:01:05.428+0000 7f0ef618b640 -1 osd.1 87 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T10:01:05.720 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:05 smithi098 ceph-mon[97580]: from='osd.1 [v2:172.21.15.98:6810/4249036577,v1:172.21.15.98:6811/4249036577]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T10:01:05.720 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:05 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3869890998' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:05.720 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:05 smithi098 ceph-mon[100914]: from='osd.1 [v2:172.21.15.98:6810/4249036577,v1:172.21.15.98:6811/4249036577]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T10:01:05.720 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:05 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3869890998' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:05.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:05 smithi196 ceph-mon[84784]: from='osd.1 [v2:172.21.15.98:6810/4249036577,v1:172.21.15.98:6811/4249036577]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T10:01:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:05 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3869890998' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:06.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:06 smithi098 ceph-mon[97580]: pgmap v144: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-21T10:01:06.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:06 smithi098 ceph-mon[97580]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:06.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:06 smithi098 ceph-mon[97580]: from='osd.1 [v2:172.21.15.98:6810/4249036577,v1:172.21.15.98:6811/4249036577]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T10:01:06.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:06 smithi098 ceph-mon[97580]: osdmap e90: 8 total, 7 up, 8 in 2024-09-21T10:01:06.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:06 smithi098 ceph-mon[97580]: from='osd.1 [v2:172.21.15.98:6810/4249036577,v1:172.21.15.98:6811/4249036577]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]} : dispatch 2024-09-21T10:01:06.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:06 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 5/84 objects degraded (5.952%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:06.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:06 smithi098 ceph-mon[97580]: from='client.34368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:06.772 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:06 smithi098 ceph-mon[100914]: pgmap v144: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-21T10:01:06.772 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:06 smithi098 ceph-mon[100914]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:06.772 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:06 smithi098 ceph-mon[100914]: from='osd.1 [v2:172.21.15.98:6810/4249036577,v1:172.21.15.98:6811/4249036577]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T10:01:06.772 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:06 smithi098 ceph-mon[100914]: osdmap e90: 8 total, 7 up, 8 in 2024-09-21T10:01:06.772 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:06 smithi098 ceph-mon[100914]: from='osd.1 [v2:172.21.15.98:6810/4249036577,v1:172.21.15.98:6811/4249036577]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]} : dispatch 2024-09-21T10:01:06.772 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:06 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 5/84 objects degraded (5.952%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:06.772 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:06 smithi098 ceph-mon[100914]: from='client.34368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:06.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:06 smithi196 ceph-mon[84784]: pgmap v144: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-21T10:01:06.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:06 smithi196 ceph-mon[84784]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:06.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:06 smithi196 ceph-mon[84784]: from='osd.1 [v2:172.21.15.98:6810/4249036577,v1:172.21.15.98:6811/4249036577]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T10:01:06.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:06 smithi196 ceph-mon[84784]: osdmap e90: 8 total, 7 up, 8 in 2024-09-21T10:01:06.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:06 smithi196 ceph-mon[84784]: from='osd.1 [v2:172.21.15.98:6810/4249036577,v1:172.21.15.98:6811/4249036577]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]} : dispatch 2024-09-21T10:01:06.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:06 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 5/84 objects degraded (5.952%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:06.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:06 smithi196 ceph-mon[84784]: from='client.34368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:07.520 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:07 smithi098 ceph-mon[100914]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:01:07.520 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:07 smithi098 ceph-mon[100914]: osd.1 [v2:172.21.15.98:6810/4249036577,v1:172.21.15.98:6811/4249036577] boot 2024-09-21T10:01:07.520 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:07 smithi098 ceph-mon[100914]: osdmap e91: 8 total, 8 up, 8 in 2024-09-21T10:01:07.520 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T10:01:07.520 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:07.521 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:07.521 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:07.521 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:07 smithi098 ceph-mon[100914]: osdmap e92: 8 total, 8 up, 8 in 2024-09-21T10:01:07.521 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:07 smithi098 ceph-mon[97580]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:01:07.521 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:07 smithi098 ceph-mon[97580]: osd.1 [v2:172.21.15.98:6810/4249036577,v1:172.21.15.98:6811/4249036577] boot 2024-09-21T10:01:07.521 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:07 smithi098 ceph-mon[97580]: osdmap e91: 8 total, 8 up, 8 in 2024-09-21T10:01:07.521 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T10:01:07.521 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:07.522 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:07.522 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:07.522 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:07 smithi098 ceph-mon[97580]: osdmap e92: 8 total, 8 up, 8 in 2024-09-21T10:01:07.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:01:07 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T10:01:07.646+0000 7f6c37c82640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-21T10:01:07.851 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:07 smithi196 ceph-mon[84784]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:01:07.851 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:07 smithi196 ceph-mon[84784]: osd.1 [v2:172.21.15.98:6810/4249036577,v1:172.21.15.98:6811/4249036577] boot 2024-09-21T10:01:07.851 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:07 smithi196 ceph-mon[84784]: osdmap e91: 8 total, 8 up, 8 in 2024-09-21T10:01:07.851 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T10:01:07.851 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:07.852 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:07.852 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:07.852 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:07 smithi196 ceph-mon[84784]: osdmap e92: 8 total, 8 up, 8 in 2024-09-21T10:01:08.209 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:01:08.812 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:08 smithi196 ceph-mon[84784]: pgmap v147: 73 pgs: 3 peering, 25 active+undersized, 4 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-21T10:01:08.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:08.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:08.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:01:08.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:01:08.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:08.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:01:08.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:01:08.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:01:08.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:01:08.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:01:08.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:08 smithi196 ceph-mon[84784]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:01:08.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:08 smithi196 ceph-mon[84784]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-21T10:01:08.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:08 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/985951533' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:08.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[100914]: pgmap v147: 73 pgs: 3 peering, 25 active+undersized, 4 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-21T10:01:08.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:08.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:08.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:01:08.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:01:08.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:08.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:01:08.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:01:08.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:01:08.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:01:08.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:01:08.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[100914]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:01:08.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[100914]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-21T10:01:08.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/985951533' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:08.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[97580]: pgmap v147: 73 pgs: 3 peering, 25 active+undersized, 4 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-21T10:01:08.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:08.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:08.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:01:08.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:01:08.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:08.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:01:08.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:01:08.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:01:08.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:01:08.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:01:08.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[97580]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:01:08.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[97580]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-21T10:01:08.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:08 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/985951533' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:09.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:09 smithi098 ceph-mon[97580]: from='client.44252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:09.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:09 smithi098 ceph-mon[97580]: from='client.34386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:09.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:09 smithi098 ceph-mon[100914]: from='client.44252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:09.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:09 smithi098 ceph-mon[100914]: from='client.34386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:09.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:09 smithi196 ceph-mon[84784]: from='client.44252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:09.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:09 smithi196 ceph-mon[84784]: from='client.34386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:10.560 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:01:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:01:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:10.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:10 smithi098 ceph-mon[97580]: pgmap v149: 73 pgs: 3 peering, 7 active+undersized, 1 active+undersized+degraded, 62 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1/84 objects degraded (1.190%) 2024-09-21T10:01:10.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:10 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 1/84 objects degraded (1.190%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:01:10.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:01:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:01:10] "GET /metrics HTTP/1.1" 200 33825 "" "Prometheus/2.51.0" 2024-09-21T10:01:10.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:10 smithi098 ceph-mon[100914]: pgmap v149: 73 pgs: 3 peering, 7 active+undersized, 1 active+undersized+degraded, 62 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1/84 objects degraded (1.190%) 2024-09-21T10:01:10.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:10 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 1/84 objects degraded (1.190%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:01:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:10 smithi196 ceph-mon[84784]: pgmap v149: 73 pgs: 3 peering, 7 active+undersized, 1 active+undersized+degraded, 62 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1/84 objects degraded (1.190%) 2024-09-21T10:01:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:10 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 1/84 objects degraded (1.190%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:01:11.068 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:01:11.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:11 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/143142740' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:11.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:11 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/143142740' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:11.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:11 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/143142740' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:12.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:12 smithi098 ceph-mon[97580]: pgmap v150: 73 pgs: 3 peering, 70 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:12.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:12 smithi098 ceph-mon[97580]: from='client.34398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:12.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:12 smithi098 ceph-mon[97580]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:12.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:12 smithi098 ceph-mon[97580]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/84 objects degraded (1.190%), 1 pg degraded) 2024-09-21T10:01:12.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:12 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:01:12.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:12 smithi098 ceph-mon[100914]: pgmap v150: 73 pgs: 3 peering, 70 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:12.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:12 smithi098 ceph-mon[100914]: from='client.34398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:12.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:12 smithi098 ceph-mon[100914]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:12.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:12 smithi098 ceph-mon[100914]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/84 objects degraded (1.190%), 1 pg degraded) 2024-09-21T10:01:12.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:12 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:01:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:12 smithi196 ceph-mon[84784]: pgmap v150: 73 pgs: 3 peering, 70 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:12 smithi196 ceph-mon[84784]: from='client.34398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:12 smithi196 ceph-mon[84784]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:12 smithi196 ceph-mon[84784]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/84 objects degraded (1.190%), 1 pg degraded) 2024-09-21T10:01:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:12 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:01:13.926 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:01:14.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:14 smithi098 ceph-mon[97580]: pgmap v151: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:14.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:14 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2507762114' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:14.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:14 smithi098 ceph-mon[97580]: from='client.44288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:14.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:14 smithi098 ceph-mon[100914]: pgmap v151: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:14.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:14 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2507762114' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:14.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:14 smithi098 ceph-mon[100914]: from='client.44288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:14 smithi196 ceph-mon[84784]: pgmap v151: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:14 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2507762114' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:14 smithi196 ceph-mon[84784]: from='client.44288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:15.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:15 smithi098 ceph-mon[97580]: from='client.34422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:15.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:15 smithi098 ceph-mon[100914]: from='client.34422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:15.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:15 smithi196 ceph-mon[84784]: from='client.34422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:16.783 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:01:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:16 smithi098 ceph-mon[97580]: pgmap v152: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:16 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1565570967' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:16.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:16 smithi098 ceph-mon[100914]: pgmap v152: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:16.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:16 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1565570967' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:16 smithi196 ceph-mon[84784]: pgmap v152: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:16 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1565570967' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:17.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:17 smithi098 ceph-mon[97580]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:17.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:17 smithi098 ceph-mon[100914]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:17.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:17 smithi196 ceph-mon[84784]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:18 smithi098 ceph-mon[97580]: pgmap v153: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:18 smithi098 ceph-mon[97580]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:18.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:18 smithi098 ceph-mon[100914]: pgmap v153: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:18.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:18 smithi098 ceph-mon[100914]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:18.886 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:18 smithi196 ceph-mon[84784]: pgmap v153: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:18.886 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:18 smithi196 ceph-mon[84784]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:19.635 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:01:19.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:19 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3527673544' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:19.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:19 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3527673544' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:19.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:19 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3527673544' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:20.592 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:01:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:01:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:20.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:20 smithi098 ceph-mon[97580]: pgmap v154: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:20.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:20 smithi098 ceph-mon[97580]: from='client.44324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:20.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:20 smithi098 ceph-mon[97580]: from='client.44330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:20.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:01:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:01:20] "GET /metrics HTTP/1.1" 200 33825 "" "Prometheus/2.51.0" 2024-09-21T10:01:20.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:20 smithi098 ceph-mon[100914]: pgmap v154: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:20.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:20 smithi098 ceph-mon[100914]: from='client.44324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:20.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:20 smithi098 ceph-mon[100914]: from='client.44330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:20.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:20 smithi196 ceph-mon[84784]: pgmap v154: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:20.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:20 smithi196 ceph-mon[84784]: from='client.44324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:20.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:20 smithi196 ceph-mon[84784]: from='client.44330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:22.496 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:01:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:22 smithi098 ceph-mon[97580]: pgmap v155: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:22 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1415510004' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:22.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:22 smithi098 ceph-mon[100914]: pgmap v155: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:22.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:22 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1415510004' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:22.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:22.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:22.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:22 smithi196 ceph-mon[84784]: pgmap v155: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:22 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1415510004' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:23.886 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:23 smithi098 ceph-mon[97580]: from='client.44342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:23.886 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:23 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:01:23.886 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:23 smithi098 ceph-mon[97580]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:01:23.886 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:23 smithi098 ceph-mon[97580]: Upgrade: osd.2 is safe to restart 2024-09-21T10:01:23.886 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:23 smithi098 ceph-mon[97580]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:23.886 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:23 smithi098 ceph-mon[100914]: from='client.44342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:23.887 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:23 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:01:23.887 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:23 smithi098 ceph-mon[100914]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:01:23.887 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:23 smithi098 ceph-mon[100914]: Upgrade: osd.2 is safe to restart 2024-09-21T10:01:23.887 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:23 smithi098 ceph-mon[100914]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:23.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:23 smithi196 ceph-mon[84784]: from='client.44342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:23.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:23 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:01:23.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:23 smithi196 ceph-mon[84784]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:01:23.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:23 smithi196 ceph-mon[84784]: Upgrade: osd.2 is safe to restart 2024-09-21T10:01:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:23 smithi196 ceph-mon[84784]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:25.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:24 smithi098 ceph-mon[97580]: pgmap v156: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:25.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:24 smithi098 ceph-mon[97580]: Upgrade: Updating osd.2 2024-09-21T10:01:25.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:24 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:25.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:24 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T10:01:25.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:24 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:01:25.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:24 smithi098 ceph-mon[97580]: Deploying daemon osd.2 on smithi098 2024-09-21T10:01:25.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:24 smithi098 ceph-mon[100914]: pgmap v156: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:25.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:24 smithi098 ceph-mon[100914]: Upgrade: Updating osd.2 2024-09-21T10:01:25.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:24 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:25.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:24 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T10:01:25.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:24 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:01:25.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:24 smithi098 ceph-mon[100914]: Deploying daemon osd.2 on smithi098 2024-09-21T10:01:25.367 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:01:25.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:24 smithi196 ceph-mon[84784]: pgmap v156: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:24 smithi196 ceph-mon[84784]: Upgrade: Updating osd.2 2024-09-21T10:01:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:24 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:24 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T10:01:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:24 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:01:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:24 smithi196 ceph-mon[84784]: Deploying daemon osd.2 on smithi098 2024-09-21T10:01:26.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:25 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2097530554' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:26.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:25 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2097530554' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:26.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:25 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2097530554' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:27.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:26 smithi098 ceph-mon[97580]: pgmap v157: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:27.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:26 smithi098 ceph-mon[97580]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:27.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:26 smithi098 ceph-mon[97580]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:27.033 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:26 smithi098 ceph-mon[100914]: pgmap v157: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:27.033 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:26 smithi098 ceph-mon[100914]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:27.033 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:26 smithi098 ceph-mon[100914]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:27.034 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:26 smithi098 systemd[1]: Stopping Ceph osd.2 for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:01:27.347 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:27 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2[55266]: 2024-09-21T10:01:27.046+0000 7fe14083c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:01:27.347 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:27 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2[55266]: 2024-09-21T10:01:27.046+0000 7fe14083c640 -1 osd.2 92 *** Got signal Terminated *** 2024-09-21T10:01:27.347 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:27 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2[55266]: 2024-09-21T10:01:27.046+0000 7fe14083c640 -1 osd.2 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:01:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:26 smithi196 ceph-mon[84784]: pgmap v157: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:26 smithi196 ceph-mon[84784]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:26 smithi196 ceph-mon[84784]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:28.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:27 smithi098 ceph-mon[97580]: osd.2 marked itself down and dead 2024-09-21T10:01:28.245 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:27 smithi098 ceph-mon[100914]: osd.2 marked itself down and dead 2024-09-21T10:01:28.245 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:01:28.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:27 smithi196 ceph-mon[84784]: osd.2 marked itself down and dead 2024-09-21T10:01:28.597 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:28 smithi098 podman[117609]: 2024-09-21 10:01:28.41937208 +0000 UTC m=+1.492827904 container died f19a948d99949763ab2a491c216e751fbec941581d14ef67c7f82cf2d1886252 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=quincy-fbd1a6a, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-21T10:01:29.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:28 smithi098 ceph-mon[97580]: pgmap v158: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:29.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:28 smithi098 ceph-mon[97580]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:01:29.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:28 smithi098 ceph-mon[97580]: osdmap e93: 8 total, 7 up, 8 in 2024-09-21T10:01:29.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:28 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2669773042' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:29.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:28 smithi098 ceph-mon[100914]: pgmap v158: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:29.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:28 smithi098 ceph-mon[100914]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:01:29.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:28 smithi098 ceph-mon[100914]: osdmap e93: 8 total, 7 up, 8 in 2024-09-21T10:01:29.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:28 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2669773042' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:29.098 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:28 smithi098 podman[117609]: 2024-09-21 10:01:28.924239694 +0000 UTC m=+1.997695519 container cleanup f19a948d99949763ab2a491c216e751fbec941581d14ef67c7f82cf2d1886252 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=quincy-fbd1a6a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-21T10:01:29.098 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:28 smithi098 bash[117609]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2 2024-09-21T10:01:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:28 smithi196 ceph-mon[84784]: pgmap v158: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:28 smithi196 ceph-mon[84784]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:01:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:28 smithi196 ceph-mon[84784]: osdmap e93: 8 total, 7 up, 8 in 2024-09-21T10:01:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:28 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2669773042' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:29.532 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:29 smithi098 podman[117707]: 2024-09-21 10:01:29.192973363 +0000 UTC m=+0.765221504 container remove f19a948d99949763ab2a491c216e751fbec941581d14ef67c7f82cf2d1886252 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=quincy-fbd1a6a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-21T10:01:29.842 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:29 smithi098 podman[117800]: 2024-09-21 10:01:29.433687303 +0000 UTC m=+0.015689428 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:01:29.842 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:29 smithi098 podman[117800]: 2024-09-21 10:01:29.53555901 +0000 UTC m=+0.117561166 container create df89741c9685ca33f788e6195e5e854a3d9349ab12f66a582fb78a46f0e6910f (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-21T10:01:29.842 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:29 smithi098 podman[117800]: 2024-09-21 10:01:29.763435895 +0000 UTC m=+0.345438014 container init df89741c9685ca33f788e6195e5e854a3d9349ab12f66a582fb78a46f0e6910f (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-09-21T10:01:29.842 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:29 smithi098 podman[117800]: 2024-09-21 10:01:29.766772657 +0000 UTC m=+0.348774772 container start df89741c9685ca33f788e6195e5e854a3d9349ab12f66a582fb78a46f0e6910f (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731) 2024-09-21T10:01:30.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:29 smithi098 ceph-mon[97580]: from='client.44378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:30.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:29 smithi098 ceph-mon[97580]: from='client.44384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:30.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:29 smithi098 ceph-mon[97580]: osdmap e94: 8 total, 7 up, 8 in 2024-09-21T10:01:30.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:29 smithi098 ceph-mon[100914]: from='client.44378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:30.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:29 smithi098 ceph-mon[100914]: from='client.44384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:30.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:29 smithi098 ceph-mon[100914]: osdmap e94: 8 total, 7 up, 8 in 2024-09-21T10:01:30.098 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:29 smithi098 podman[117800]: 2024-09-21 10:01:29.838160816 +0000 UTC m=+0.420162933 container attach df89741c9685ca33f788e6195e5e854a3d9349ab12f66a582fb78a46f0e6910f (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=) 2024-09-21T10:01:30.098 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:29 smithi098 podman[117800]: 2024-09-21 10:01:29.96154355 +0000 UTC m=+0.543545664 container died df89741c9685ca33f788e6195e5e854a3d9349ab12f66a582fb78a46f0e6910f (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True) 2024-09-21T10:01:30.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:29 smithi196 ceph-mon[84784]: from='client.44378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:30.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:29 smithi196 ceph-mon[84784]: from='client.44384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:30.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:29 smithi196 ceph-mon[84784]: osdmap e94: 8 total, 7 up, 8 in 2024-09-21T10:01:30.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:01:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:01:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:30.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:01:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:01:30] "GET /metrics HTTP/1.1" 200 33971 "" "Prometheus/2.51.0" 2024-09-21T10:01:30.847 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:30 smithi098 podman[117823]: 2024-09-21 10:01:30.575777153 +0000 UTC m=+0.607843328 container remove df89741c9685ca33f788e6195e5e854a3d9349ab12f66a582fb78a46f0e6910f (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=main-62da54b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-21T10:01:30.847 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:30 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.2.service: Deactivated successfully. 2024-09-21T10:01:30.847 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:30 smithi098 systemd[1]: Stopped Ceph osd.2 for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:01:30.847 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:30 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.2.service: Consumed 5.422s CPU time. 2024-09-21T10:01:30.848 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:30 smithi098 systemd[1]: Starting Ceph osd.2 for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:01:31.129 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:01:31.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:30 smithi098 ceph-mon[97580]: pgmap v161: 73 pgs: 13 peering, 1 stale+active+clean, 59 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:31.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:30 smithi098 ceph-mon[97580]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:01:31.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:30 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/321865294' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:31.248 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:30 smithi098 ceph-mon[100914]: pgmap v161: 73 pgs: 13 peering, 1 stale+active+clean, 59 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:31.249 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:30 smithi098 ceph-mon[100914]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:01:31.249 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:30 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/321865294' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:31.249 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:31 smithi098 podman[117948]: 2024-09-21 10:01:30.92032242 +0000 UTC m=+0.017223047 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:01:31.249 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:31 smithi098 podman[117948]: 2024-09-21 10:01:31.043793277 +0000 UTC m=+0.140693890 container create 92e410e30a9ce9f701f718dc26ad259ed614c4ab41cbf61f1f33d75dfaa806ae (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-activate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-21T10:01:31.249 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:31 smithi098 podman[117948]: 2024-09-21 10:01:31.246639845 +0000 UTC m=+0.343540469 container init 92e410e30a9ce9f701f718dc26ad259ed614c4ab41cbf61f1f33d75dfaa806ae (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=main-62da54b, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:01:31.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:30 smithi196 ceph-mon[84784]: pgmap v161: 73 pgs: 13 peering, 1 stale+active+clean, 59 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:30 smithi196 ceph-mon[84784]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:01:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:30 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/321865294' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:31.597 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:31 smithi098 podman[117948]: 2024-09-21 10:01:31.250642898 +0000 UTC m=+0.347543516 container start 92e410e30a9ce9f701f718dc26ad259ed614c4ab41cbf61f1f33d75dfaa806ae (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-activate, org.label-schema.license=GPLv2, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-21T10:01:31.597 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:31 smithi098 podman[117948]: 2024-09-21 10:01:31.354832664 +0000 UTC m=+0.451733281 container attach 92e410e30a9ce9f701f718dc26ad259ed614c4ab41cbf61f1f33d75dfaa806ae (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-activate, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-21T10:01:31.597 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:31 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-activate[117984]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:01:31.598 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:31 smithi098 bash[117948]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:01:31.598 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:31 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-activate[117984]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:01:31.598 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:31 smithi098 bash[117948]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:01:32.140 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:31 smithi098 ceph-mon[97580]: pgmap v162: 73 pgs: 4 active+undersized, 13 peering, 56 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:32.140 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:31 smithi098 ceph-mon[97580]: from='client.44396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:32.140 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:31 smithi098 ceph-mon[97580]: from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:32.141 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:31 smithi098 ceph-mon[100914]: pgmap v162: 73 pgs: 4 active+undersized, 13 peering, 56 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:32.141 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:31 smithi098 ceph-mon[100914]: from='client.44396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:32.141 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:31 smithi098 ceph-mon[100914]: from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:32.141 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:31 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-activate[117984]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T10:01:32.141 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:31 smithi098 bash[117948]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T10:01:32.141 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:31 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-activate[117984]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-21T10:01:32.141 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:31 smithi098 bash[117948]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-21T10:01:32.141 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:32 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-activate[117984]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-21T10:01:32.141 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:32 smithi098 bash[117948]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-21T10:01:32.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:31 smithi196 ceph-mon[84784]: pgmap v162: 73 pgs: 4 active+undersized, 13 peering, 56 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:01:32.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:31 smithi196 ceph-mon[84784]: from='client.44396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:31 smithi196 ceph-mon[84784]: from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:32.510 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:32 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-activate[117984]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-21T10:01:32.510 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:32 smithi098 bash[117948]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-21T10:01:32.510 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:32 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-activate[117984]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-21T10:01:32.510 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:32 smithi098 bash[117948]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-21T10:01:32.510 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:32 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-activate[117984]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T10:01:32.510 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:32 smithi098 bash[117948]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T10:01:32.511 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:32 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-activate[117984]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-21T10:01:32.511 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:32 smithi098 bash[117948]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-21T10:01:32.511 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:32 smithi098 podman[117948]: 2024-09-21 10:01:32.16678058 +0000 UTC m=+1.263681202 container died 92e410e30a9ce9f701f718dc26ad259ed614c4ab41cbf61f1f33d75dfaa806ae (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b) 2024-09-21T10:01:33.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:33 smithi098 ceph-mon[97580]: Health check failed: Degraded data redundancy: 6/84 objects degraded (7.143%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:33.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:33 smithi098 ceph-mon[97580]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T10:01:33.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:33 smithi098 ceph-mon[100914]: Health check failed: Degraded data redundancy: 6/84 objects degraded (7.143%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:33.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:33 smithi098 ceph-mon[100914]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T10:01:33.098 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:32 smithi098 podman[118148]: 2024-09-21 10:01:32.842702981 +0000 UTC m=+0.665916770 container remove 92e410e30a9ce9f701f718dc26ad259ed614c4ab41cbf61f1f33d75dfaa806ae (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-activate, org.label-schema.build-date=20240731, RELEASE=main-62da54b, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T10:01:33.098 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:33 smithi098 podman[118205]: 2024-09-21 10:01:32.950728104 +0000 UTC m=+0.016103907 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:01:33.098 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:33 smithi098 podman[118205]: 2024-09-21 10:01:33.077571763 +0000 UTC m=+0.142947589 container create 47ba7450689130a5ab1b9a26257396d4534bb1fe1977d052cc962639ed7b3df3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, ceph=True) 2024-09-21T10:01:33.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:33 smithi196 ceph-mon[84784]: Health check failed: Degraded data redundancy: 6/84 objects degraded (7.143%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:33 smithi196 ceph-mon[84784]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T10:01:33.597 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:33 smithi098 podman[118205]: 2024-09-21 10:01:33.296575497 +0000 UTC m=+0.361951295 container init 47ba7450689130a5ab1b9a26257396d4534bb1fe1977d052cc962639ed7b3df3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, ceph=True, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-21T10:01:33.597 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:33 smithi098 podman[118205]: 2024-09-21 10:01:33.299745686 +0000 UTC m=+0.365121467 container start 47ba7450689130a5ab1b9a26257396d4534bb1fe1977d052cc962639ed7b3df3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, org.label-schema.vendor=CentOS) 2024-09-21T10:01:33.597 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:33 smithi098 ceph-osd[118225]: -- 172.21.15.98:0/3578654662 <== mon.2 v2:172.21.15.98:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5559066dfa40 con 0x5559066bdc00 2024-09-21T10:01:33.597 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:33 smithi098 bash[118205]: 47ba7450689130a5ab1b9a26257396d4534bb1fe1977d052cc962639ed7b3df3 2024-09-21T10:01:33.597 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:33 smithi098 systemd[1]: Started Ceph osd.2 for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:01:34.018 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:01:34.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:34 smithi098 ceph-mon[100914]: pgmap v163: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-21T10:01:34.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:34 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:34.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:34 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:34.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:34 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:01:34.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:34 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/225286319' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:34.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:34 smithi098 ceph-mon[97580]: pgmap v163: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-21T10:01:34.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:34 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:34.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:34 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:34.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:34 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:01:34.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:34 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/225286319' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:34 smithi196 ceph-mon[84784]: pgmap v163: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-21T10:01:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:34 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:34 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:34 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:01:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:34 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/225286319' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:35.189 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:34 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2[118221]: 2024-09-21T10:01:34.920+0000 7fa2821e1640 -1 Falling back to public interface 2024-09-21T10:01:35.799 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:35 smithi098 ceph-mon[97580]: from='client.34545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:35.799 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:35 smithi098 ceph-mon[97580]: from='client.44423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:35.800 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:35 smithi098 ceph-mon[100914]: from='client.34545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:35.800 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:35 smithi098 ceph-mon[100914]: from='client.44423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:35.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:35 smithi196 ceph-mon[84784]: from='client.34545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:35.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:35 smithi196 ceph-mon[84784]: from='client.44423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:36.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:36 smithi196 ceph-mon[84784]: pgmap v164: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-21T10:01:36.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:36 smithi098 ceph-mon[97580]: pgmap v164: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-21T10:01:36.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:36 smithi098 ceph-mon[100914]: pgmap v164: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-21T10:01:36.891 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:01:37.798 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:37 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4051378028' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:37.798 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:37.798 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:37.798 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:37 smithi098 ceph-mon[97580]: from='client.44435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:37.798 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:37.799 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:37.799 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:37.799 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:37.799 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:37 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4051378028' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:37.799 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:37.799 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:37.799 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:37 smithi098 ceph-mon[100914]: from='client.44435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:37.799 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:37.799 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:37.799 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:37.799 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:37.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:37 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4051378028' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:37 smithi196 ceph-mon[84784]: from='client.44435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:38.802 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2[118221]: 2024-09-21T10:01:38.456+0000 7fa2821e1640 -1 osd.2 92 log_to_monitors true 2024-09-21T10:01:39.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:38 smithi098 ceph-mon[97580]: pgmap v165: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-21T10:01:39.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:38 smithi098 ceph-mon[97580]: from='client.34569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:39.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:38 smithi098 ceph-mon[97580]: from='osd.2 [v2:172.21.15.98:6818/1274766038,v1:172.21.15.98:6819/1274766038]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T10:01:39.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:38 smithi098 ceph-mon[100914]: pgmap v165: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-21T10:01:39.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:38 smithi098 ceph-mon[100914]: from='client.34569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:39.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:38 smithi098 ceph-mon[100914]: from='osd.2 [v2:172.21.15.98:6818/1274766038,v1:172.21.15.98:6819/1274766038]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T10:01:39.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:38 smithi196 ceph-mon[84784]: pgmap v165: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-21T10:01:39.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:38 smithi196 ceph-mon[84784]: from='client.34569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:38 smithi196 ceph-mon[84784]: from='osd.2 [v2:172.21.15.98:6818/1274766038,v1:172.21.15.98:6819/1274766038]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T10:01:39.469 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:01:39 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2[118221]: 2024-09-21T10:01:39.364+0000 7fa275f83640 -1 osd.2 92 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T10:01:39.794 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:01:40.069 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:39 smithi098 ceph-mon[97580]: from='osd.2 [v2:172.21.15.98:6818/1274766038,v1:172.21.15.98:6819/1274766038]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T10:01:40.070 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:39 smithi098 ceph-mon[97580]: osdmap e95: 8 total, 7 up, 8 in 2024-09-21T10:01:40.070 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:39 smithi098 ceph-mon[97580]: from='osd.2 [v2:172.21.15.98:6818/1274766038,v1:172.21.15.98:6819/1274766038]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]} : dispatch 2024-09-21T10:01:40.070 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:39 smithi098 ceph-mon[97580]: from='osd.2 [v2:172.21.15.98:6818/1274766038,v1:172.21.15.98:6819/1274766038]' entity='osd.2' 2024-09-21T10:01:40.070 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:39 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3979018012' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:40.070 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:39 smithi098 ceph-mon[100914]: from='osd.2 [v2:172.21.15.98:6818/1274766038,v1:172.21.15.98:6819/1274766038]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T10:01:40.070 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:39 smithi098 ceph-mon[100914]: osdmap e95: 8 total, 7 up, 8 in 2024-09-21T10:01:40.070 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:39 smithi098 ceph-mon[100914]: from='osd.2 [v2:172.21.15.98:6818/1274766038,v1:172.21.15.98:6819/1274766038]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]} : dispatch 2024-09-21T10:01:40.070 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:39 smithi098 ceph-mon[100914]: from='osd.2 [v2:172.21.15.98:6818/1274766038,v1:172.21.15.98:6819/1274766038]' entity='osd.2' 2024-09-21T10:01:40.071 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:39 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3979018012' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:39 smithi196 ceph-mon[84784]: from='osd.2 [v2:172.21.15.98:6818/1274766038,v1:172.21.15.98:6819/1274766038]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T10:01:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:39 smithi196 ceph-mon[84784]: osdmap e95: 8 total, 7 up, 8 in 2024-09-21T10:01:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:39 smithi196 ceph-mon[84784]: from='osd.2 [v2:172.21.15.98:6818/1274766038,v1:172.21.15.98:6819/1274766038]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]} : dispatch 2024-09-21T10:01:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:39 smithi196 ceph-mon[84784]: from='osd.2 [v2:172.21.15.98:6818/1274766038,v1:172.21.15.98:6819/1274766038]' entity='osd.2' 2024-09-21T10:01:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:39 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3979018012' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:40.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:01:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:01:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:40.921 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:01:40] "GET /metrics HTTP/1.1" 200 33968 "" "Prometheus/2.51.0" 2024-09-21T10:01:40.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-mon[100914]: pgmap v167: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-21T10:01:40.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-mon[100914]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:40.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-mon[100914]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:01:40.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-mon[100914]: osd.2 [v2:172.21.15.98:6818/1274766038,v1:172.21.15.98:6819/1274766038] boot 2024-09-21T10:01:40.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-mon[100914]: osdmap e96: 8 total, 8 up, 8 in 2024-09-21T10:01:40.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T10:01:40.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:40.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:40.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:01:40.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:01:40.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:40.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-mon[97580]: pgmap v167: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-21T10:01:40.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-mon[97580]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:40.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-mon[97580]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:01:40.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-mon[97580]: osd.2 [v2:172.21.15.98:6818/1274766038,v1:172.21.15.98:6819/1274766038] boot 2024-09-21T10:01:40.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-mon[97580]: osdmap e96: 8 total, 8 up, 8 in 2024-09-21T10:01:40.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T10:01:40.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:40.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:40.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:01:40.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:01:40.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:40 smithi196 ceph-mon[84784]: pgmap v167: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-21T10:01:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:40 smithi196 ceph-mon[84784]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:40 smithi196 ceph-mon[84784]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:01:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:40 smithi196 ceph-mon[84784]: osd.2 [v2:172.21.15.98:6818/1274766038,v1:172.21.15.98:6819/1274766038] boot 2024-09-21T10:01:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:40 smithi196 ceph-mon[84784]: osdmap e96: 8 total, 8 up, 8 in 2024-09-21T10:01:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:40 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T10:01:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:40 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:40 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:40 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:01:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:40 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:01:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:40 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:41.191 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:01:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T10:01:40.918+0000 7f6c37c82640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-21T10:01:42.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:41 smithi196 ceph-mon[84784]: from='client.44462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:41 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:01:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:41 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:01:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:41 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:01:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:41 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:01:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:41 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T10:01:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:41 smithi196 ceph-mon[84784]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T10:01:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:41 smithi196 ceph-mon[84784]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-21T10:01:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:41 smithi196 ceph-mon[84784]: osdmap e97: 8 total, 8 up, 8 in 2024-09-21T10:01:42.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:41 smithi098 ceph-mon[97580]: from='client.44462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:42.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:41 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:01:42.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:41 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:01:42.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:41 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:01:42.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:41 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:01:42.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:41 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T10:01:42.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:41 smithi098 ceph-mon[97580]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T10:01:42.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:41 smithi098 ceph-mon[97580]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-21T10:01:42.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:41 smithi098 ceph-mon[97580]: osdmap e97: 8 total, 8 up, 8 in 2024-09-21T10:01:42.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:41 smithi098 ceph-mon[100914]: from='client.44462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:42.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:41 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:01:42.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:41 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:01:42.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:41 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:01:42.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:41 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:01:42.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:41 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T10:01:42.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:41 smithi098 ceph-mon[100914]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T10:01:42.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:41 smithi098 ceph-mon[100914]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-21T10:01:42.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:41 smithi098 ceph-mon[100914]: osdmap e97: 8 total, 8 up, 8 in 2024-09-21T10:01:42.686 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:01:43.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:42 smithi098 ceph-mon[97580]: pgmap v169: 73 pgs: 3 peering, 13 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-21T10:01:43.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:42 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:43.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:42 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3096579351' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:43.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:42 smithi098 ceph-mon[100914]: pgmap v169: 73 pgs: 3 peering, 13 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-21T10:01:43.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:42 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:43.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:42 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3096579351' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:42 smithi196 ceph-mon[84784]: pgmap v169: 73 pgs: 3 peering, 13 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-21T10:01:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:42 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:42 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3096579351' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:44.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:43 smithi196 ceph-mon[84784]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:44.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:43 smithi098 ceph-mon[97580]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:44.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:43 smithi098 ceph-mon[100914]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:45.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:44 smithi098 ceph-mon[97580]: from='client.34608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:45.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:44 smithi098 ceph-mon[97580]: pgmap v171: 73 pgs: 3 peering, 3 active+undersized, 67 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:45.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:44 smithi098 ceph-mon[97580]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/84 objects degraded (7.143%), 3 pgs degraded) 2024-09-21T10:01:45.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:44 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:01:45.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:44 smithi098 ceph-mon[100914]: from='client.34608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:45.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:44 smithi098 ceph-mon[100914]: pgmap v171: 73 pgs: 3 peering, 3 active+undersized, 67 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:45.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:44 smithi098 ceph-mon[100914]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/84 objects degraded (7.143%), 3 pgs degraded) 2024-09-21T10:01:45.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:44 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:01:45.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:44 smithi196 ceph-mon[84784]: from='client.34608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:45.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:44 smithi196 ceph-mon[84784]: pgmap v171: 73 pgs: 3 peering, 3 active+undersized, 67 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:45.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:44 smithi196 ceph-mon[84784]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/84 objects degraded (7.143%), 3 pgs degraded) 2024-09-21T10:01:45.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:44 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:01:45.541 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:01:46.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:45 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2084698095' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:46.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:45 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2084698095' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:46.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:45 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2084698095' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:47.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:46 smithi098 ceph-mon[97580]: pgmap v172: 73 pgs: 3 peering, 70 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:47.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:46 smithi098 ceph-mon[97580]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:47.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:46 smithi098 ceph-mon[97580]: from='client.44498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:47.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:46 smithi098 ceph-mon[100914]: pgmap v172: 73 pgs: 3 peering, 70 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:47.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:46 smithi098 ceph-mon[100914]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:47.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:46 smithi098 ceph-mon[100914]: from='client.44498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:46 smithi196 ceph-mon[84784]: pgmap v172: 73 pgs: 3 peering, 70 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:46 smithi196 ceph-mon[84784]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:46 smithi196 ceph-mon[84784]: from='client.44498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:48.392 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:01:49.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:48 smithi098 ceph-mon[97580]: pgmap v173: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:49.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:48 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1514845281' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:49.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:48 smithi098 ceph-mon[100914]: pgmap v173: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:49.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:48 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1514845281' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:49.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:48 smithi196 ceph-mon[84784]: pgmap v173: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:49.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:48 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1514845281' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:50.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:49 smithi196 ceph-mon[84784]: from='client.44510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:50.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:49 smithi196 ceph-mon[84784]: from='client.44516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:50.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:49 smithi098 ceph-mon[97580]: from='client.44510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:50.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:49 smithi098 ceph-mon[97580]: from='client.44516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:50.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:49 smithi098 ceph-mon[100914]: from='client.44510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:50.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:49 smithi098 ceph-mon[100914]: from='client.44516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:50.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:01:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:01:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:50.909 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:01:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:01:50] "GET /metrics HTTP/1.1" 200 33968 "" "Prometheus/2.51.0" 2024-09-21T10:01:51.241 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:01:51.241 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:50 smithi098 ceph-mon[100914]: pgmap v174: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:51.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:50 smithi098 ceph-mon[97580]: pgmap v174: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:51.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:50 smithi196 ceph-mon[84784]: pgmap v174: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:52.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:51 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/442488664' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:52.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:51 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/442488664' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:52.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:51 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/442488664' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:53.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:52 smithi098 ceph-mon[97580]: pgmap v175: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:53.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:52 smithi098 ceph-mon[97580]: from='client.44528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:53.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:52 smithi098 ceph-mon[97580]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:53.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:53.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:53.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:53.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:52 smithi098 ceph-mon[100914]: pgmap v175: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:53.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:52 smithi098 ceph-mon[100914]: from='client.44528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:53.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:52 smithi098 ceph-mon[100914]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:53.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:53.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:53.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:53.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:52 smithi196 ceph-mon[84784]: pgmap v175: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:53.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:52 smithi196 ceph-mon[84784]: from='client.44528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:52 smithi196 ceph-mon[84784]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:54.102 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:01:54.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:53 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3235382059' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:54.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:53 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3235382059' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:53 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3235382059' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:55.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:54 smithi098 ceph-mon[97580]: pgmap v176: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:55.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:54 smithi098 ceph-mon[100914]: pgmap v176: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:54 smithi196 ceph-mon[84784]: pgmap v176: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:56.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:55 smithi098 ceph-mon[97580]: from='client.44546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:56.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:55 smithi098 ceph-mon[97580]: from='client.44552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:56.291 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:55 smithi098 ceph-mon[100914]: from='client.44546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:56.291 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:55 smithi098 ceph-mon[100914]: from='client.44552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:56.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:55 smithi196 ceph-mon[84784]: from='client.44546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:56.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:55 smithi196 ceph-mon[84784]: from='client.44552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:56.961 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:01:57.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:56 smithi098 ceph-mon[97580]: pgmap v177: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:57.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:56 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T10:01:57.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:56 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1163102764' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:57.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:56 smithi098 ceph-mon[100914]: pgmap v177: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:57.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:56 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T10:01:57.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:56 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1163102764' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:57.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:56 smithi196 ceph-mon[84784]: pgmap v177: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:57.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:56 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T10:01:57.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:56 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1163102764' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:01:58.070 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:57 smithi098 ceph-mon[97580]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T10:01:58.070 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:57 smithi098 ceph-mon[97580]: Upgrade: osd.3 is safe to restart 2024-09-21T10:01:58.070 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:57 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:58.070 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:57 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T10:01:58.070 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:57 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:01:58.070 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:57 smithi098 ceph-mon[100914]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T10:01:58.070 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:57 smithi098 ceph-mon[100914]: Upgrade: osd.3 is safe to restart 2024-09-21T10:01:58.071 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:57 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:58.071 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:57 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T10:01:58.071 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:57 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:01:58.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:57 smithi196 ceph-mon[84784]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T10:01:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:57 smithi196 ceph-mon[84784]: Upgrade: osd.3 is safe to restart 2024-09-21T10:01:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:57 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:01:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:57 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T10:01:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:57 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:01:59.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:58 smithi098 ceph-mon[97580]: from='client.44564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:59.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:58 smithi098 ceph-mon[97580]: pgmap v178: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:59.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:58 smithi098 ceph-mon[97580]: from='client.34698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:59.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:58 smithi098 ceph-mon[97580]: Upgrade: Updating osd.3 2024-09-21T10:01:59.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:58 smithi098 ceph-mon[97580]: Deploying daemon osd.3 on smithi098 2024-09-21T10:01:59.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:58 smithi098 ceph-mon[100914]: from='client.44564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:59.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:58 smithi098 ceph-mon[100914]: pgmap v178: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:59.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:58 smithi098 ceph-mon[100914]: from='client.34698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:59.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:58 smithi098 ceph-mon[100914]: Upgrade: Updating osd.3 2024-09-21T10:01:59.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:58 smithi098 ceph-mon[100914]: Deploying daemon osd.3 on smithi098 2024-09-21T10:01:59.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:58 smithi196 ceph-mon[84784]: from='client.44564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:58 smithi196 ceph-mon[84784]: pgmap v178: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:01:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:58 smithi196 ceph-mon[84784]: from='client.34698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:58 smithi196 ceph-mon[84784]: Upgrade: Updating osd.3 2024-09-21T10:01:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:58 smithi196 ceph-mon[84784]: Deploying daemon osd.3 on smithi098 2024-09-21T10:01:59.831 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:02:00.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:59 smithi196 ceph-mon[84784]: pgmap v179: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:00.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:59 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4170074594' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:02:00.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:01:59 smithi196 ceph-mon[84784]: from='client.34710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:00.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:59 smithi098 ceph-mon[97580]: pgmap v179: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:00.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:59 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4170074594' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:02:00.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:01:59 smithi098 ceph-mon[97580]: from='client.34710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:00.269 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:59 smithi098 ceph-mon[100914]: pgmap v179: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:00.269 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:59 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4170074594' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:02:00.269 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:01:59 smithi098 ceph-mon[100914]: from='client.34710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:00.554 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:00 smithi098 systemd[1]: Stopping Ceph osd.3 for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:02:00.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:02:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:02:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:00.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:02:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:02:00] "GET /metrics HTTP/1.1" 200 33967 "" "Prometheus/2.51.0" 2024-09-21T10:02:00.847 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3[60910]: 2024-09-21T10:02:00.552+0000 7f75a7e1e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:02:00.847 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3[60910]: 2024-09-21T10:02:00.552+0000 7f75a7e1e640 -1 osd.3 97 *** Got signal Terminated *** 2024-09-21T10:02:00.847 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3[60910]: 2024-09-21T10:02:00.552+0000 7f75a7e1e640 -1 osd.3 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:02:01.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:00 smithi098 ceph-mon[97580]: from='client.44588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:01.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:00 smithi098 ceph-mon[97580]: osd.3 marked itself down and dead 2024-09-21T10:02:01.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:00 smithi098 ceph-mon[100914]: from='client.44588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:01.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:00 smithi098 ceph-mon[100914]: osd.3 marked itself down and dead 2024-09-21T10:02:01.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:00 smithi196 ceph-mon[84784]: from='client.44588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:01.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:00 smithi196 ceph-mon[84784]: osd.3 marked itself down and dead 2024-09-21T10:02:01.663 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:01 smithi098 podman[123048]: 2024-09-21 10:02:01.378531437 +0000 UTC m=+0.962127151 container died 68f6e62795e765183e7b96f24b89e479a67a700b0b8562c19fe0d0541f579e96 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-21T10:02:01.950 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:01 smithi098 podman[123048]: 2024-09-21 10:02:01.751904276 +0000 UTC m=+1.335499989 container cleanup 68f6e62795e765183e7b96f24b89e479a67a700b0b8562c19fe0d0541f579e96 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-fbd1a6a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-21T10:02:01.951 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:01 smithi098 bash[123048]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3 2024-09-21T10:02:02.347 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:02 smithi098 podman[123063]: 2024-09-21 10:02:02.045216887 +0000 UTC m=+0.664044932 container remove 68f6e62795e765183e7b96f24b89e479a67a700b0b8562c19fe0d0541f579e96 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a) 2024-09-21T10:02:02.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:01 smithi098 ceph-mon[97580]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:02:02.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:01 smithi098 ceph-mon[97580]: osdmap e98: 8 total, 7 up, 8 in 2024-09-21T10:02:02.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:01 smithi098 ceph-mon[97580]: pgmap v181: 73 pgs: 13 stale+active+clean, 60 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:02.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:01 smithi098 ceph-mon[100914]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:02:02.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:01 smithi098 ceph-mon[100914]: osdmap e98: 8 total, 7 up, 8 in 2024-09-21T10:02:02.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:01 smithi098 ceph-mon[100914]: pgmap v181: 73 pgs: 13 stale+active+clean, 60 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:02.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:01 smithi196 ceph-mon[84784]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:02:02.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:01 smithi196 ceph-mon[84784]: osdmap e98: 8 total, 7 up, 8 in 2024-09-21T10:02:02.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:01 smithi196 ceph-mon[84784]: pgmap v181: 73 pgs: 13 stale+active+clean, 60 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:02.635 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:02 smithi098 podman[123154]: 2024-09-21 10:02:02.266679316 +0000 UTC m=+0.017541686 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:02:02.635 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:02 smithi098 podman[123154]: 2024-09-21 10:02:02.388452745 +0000 UTC m=+0.139315116 container create 716d96803b18b36c9811842a1161bd4d7b252d966fe972644676d74eb4a5bf33 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-21T10:02:02.635 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:02 smithi098 podman[123154]: 2024-09-21 10:02:02.632916896 +0000 UTC m=+0.383779270 container init 716d96803b18b36c9811842a1161bd4d7b252d966fe972644676d74eb4a5bf33 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux ) 2024-09-21T10:02:02.756 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:02:02.959 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:02 smithi098 podman[123154]: 2024-09-21 10:02:02.636842413 +0000 UTC m=+0.387704777 container start 716d96803b18b36c9811842a1161bd4d7b252d966fe972644676d74eb4a5bf33 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-21T10:02:02.959 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:02 smithi098 podman[123154]: 2024-09-21 10:02:02.716280343 +0000 UTC m=+0.467142715 container attach 716d96803b18b36c9811842a1161bd4d7b252d966fe972644676d74eb4a5bf33 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-21T10:02:02.959 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:02 smithi098 podman[123154]: 2024-09-21 10:02:02.846866016 +0000 UTC m=+0.597728380 container died 716d96803b18b36c9811842a1161bd4d7b252d966fe972644676d74eb4a5bf33 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, ceph=True) 2024-09-21T10:02:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:02 smithi098 ceph-mon[97580]: osdmap e99: 8 total, 7 up, 8 in 2024-09-21T10:02:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:02 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2608497502' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:02:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:02 smithi098 ceph-mon[97580]: from='client.34728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:03.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:02 smithi098 ceph-mon[100914]: osdmap e99: 8 total, 7 up, 8 in 2024-09-21T10:02:03.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:02 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2608497502' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:02:03.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:02 smithi098 ceph-mon[100914]: from='client.34728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:03.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:02 smithi196 ceph-mon[84784]: osdmap e99: 8 total, 7 up, 8 in 2024-09-21T10:02:03.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:02 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2608497502' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:02:03.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:02 smithi196 ceph-mon[84784]: from='client.34728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:03.847 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:03 smithi098 podman[123243]: 2024-09-21 10:02:03.470618857 +0000 UTC m=+0.617793666 container remove 716d96803b18b36c9811842a1161bd4d7b252d966fe972644676d74eb4a5bf33 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_CLEAN=True, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-21T10:02:03.847 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:03 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.3.service: Deactivated successfully. 2024-09-21T10:02:03.847 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:03 smithi098 systemd[1]: Stopped Ceph osd.3 for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:02:03.847 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:03 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.3.service: Consumed 4.890s CPU time. 2024-09-21T10:02:03.847 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:03 smithi098 systemd[1]: Starting Ceph osd.3 for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:02:04.235 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:03 smithi098 ceph-mon[97580]: pgmap v183: 73 pgs: 13 active+undersized, 7 peering, 5 stale+active+clean, 2 active+undersized+degraded, 46 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3/84 objects degraded (3.571%) 2024-09-21T10:02:04.235 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:03 smithi098 ceph-mon[97580]: from='client.44606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:04.235 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:03 smithi098 ceph-mon[100914]: pgmap v183: 73 pgs: 13 active+undersized, 7 peering, 5 stale+active+clean, 2 active+undersized+degraded, 46 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3/84 objects degraded (3.571%) 2024-09-21T10:02:04.235 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:03 smithi098 ceph-mon[100914]: from='client.44606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:04.235 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:03 smithi098 podman[123341]: 2024-09-21 10:02:03.807724006 +0000 UTC m=+0.015620662 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:02:04.236 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:03 smithi098 podman[123341]: 2024-09-21 10:02:03.905259186 +0000 UTC m=+0.113155827 container create cf55b2ce93b6d7b668737eaf4f5e0a9dccca34a629a895a91210bf56128712ba (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-activate, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-21T10:02:04.236 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:04 smithi098 podman[123341]: 2024-09-21 10:02:04.133310046 +0000 UTC m=+0.341206695 container init cf55b2ce93b6d7b668737eaf4f5e0a9dccca34a629a895a91210bf56128712ba (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-21T10:02:04.236 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:04 smithi098 podman[123341]: 2024-09-21 10:02:04.137024462 +0000 UTC m=+0.344921109 container start cf55b2ce93b6d7b668737eaf4f5e0a9dccca34a629a895a91210bf56128712ba (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-21T10:02:04.236 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:04 smithi098 podman[123341]: 2024-09-21 10:02:04.23318733 +0000 UTC m=+0.441083973 container attach cf55b2ce93b6d7b668737eaf4f5e0a9dccca34a629a895a91210bf56128712ba (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-activate, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-62da54b, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:02:04.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:03 smithi196 ceph-mon[84784]: pgmap v183: 73 pgs: 13 active+undersized, 7 peering, 5 stale+active+clean, 2 active+undersized+degraded, 46 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3/84 objects degraded (3.571%) 2024-09-21T10:02:04.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:03 smithi196 ceph-mon[84784]: from='client.44606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:04.597 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:04 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-activate[123356]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:02:04.597 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:04 smithi098 bash[123341]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:02:04.597 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:04 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-activate[123356]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:02:04.597 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:04 smithi098 bash[123341]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:02:05.025 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:04 smithi098 ceph-mon[97580]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:02:05.025 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:04 smithi098 ceph-mon[97580]: Health check failed: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:02:05.025 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:04 smithi098 ceph-mon[100914]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:02:05.025 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:04 smithi098 ceph-mon[100914]: Health check failed: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:02:05.026 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:04 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-activate[123356]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T10:02:05.026 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:04 smithi098 bash[123341]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T10:02:05.026 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:04 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-activate[123356]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-21T10:02:05.026 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:04 smithi098 bash[123341]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-21T10:02:05.026 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:05 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-activate[123356]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-21T10:02:05.026 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:05 smithi098 bash[123341]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-21T10:02:05.347 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:05 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-activate[123356]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-21T10:02:05.347 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:05 smithi098 bash[123341]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-21T10:02:05.347 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:05 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-activate[123356]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-21T10:02:05.347 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:05 smithi098 bash[123341]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-21T10:02:05.347 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:05 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-activate[123356]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T10:02:05.347 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:05 smithi098 bash[123341]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T10:02:05.347 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:05 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-activate[123356]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-21T10:02:05.348 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:05 smithi098 bash[123341]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-21T10:02:05.348 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:05 smithi098 podman[123341]: 2024-09-21 10:02:05.051697553 +0000 UTC m=+1.259594202 container died cf55b2ce93b6d7b668737eaf4f5e0a9dccca34a629a895a91210bf56128712ba (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True) 2024-09-21T10:02:05.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:04 smithi196 ceph-mon[84784]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:02:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:04 smithi196 ceph-mon[84784]: Health check failed: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:02:05.628 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:02:06.070 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:05 smithi098 ceph-mon[97580]: pgmap v184: 73 pgs: 16 active+undersized, 7 peering, 4 stale+active+clean, 2 active+undersized+degraded, 44 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3/84 objects degraded (3.571%) 2024-09-21T10:02:06.070 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:05 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4046161853' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:02:06.070 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:05 smithi098 ceph-mon[97580]: from='client.34746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:06.071 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:05 smithi098 ceph-mon[97580]: from='client.44624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:06.071 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:05 smithi098 ceph-mon[100914]: pgmap v184: 73 pgs: 16 active+undersized, 7 peering, 4 stale+active+clean, 2 active+undersized+degraded, 44 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3/84 objects degraded (3.571%) 2024-09-21T10:02:06.071 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:05 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4046161853' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:02:06.071 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:05 smithi098 ceph-mon[100914]: from='client.34746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:06.071 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:05 smithi098 ceph-mon[100914]: from='client.44624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:06.071 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:05 smithi098 podman[123487]: 2024-09-21 10:02:05.721373747 +0000 UTC m=+0.663680445 container remove cf55b2ce93b6d7b668737eaf4f5e0a9dccca34a629a895a91210bf56128712ba (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-activate, org.label-schema.license=GPLv2, RELEASE=main-62da54b, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-21T10:02:06.071 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:05 smithi098 podman[123618]: 2024-09-21 10:02:05.831341071 +0000 UTC m=+0.017960571 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:02:06.071 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:05 smithi098 podman[123618]: 2024-09-21 10:02:05.947549616 +0000 UTC m=+0.134169139 container create 98d8757e4343a5166a83b992ad05a410f9a789e371b4490f417df2338060e6b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-21T10:02:06.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:05 smithi196 ceph-mon[84784]: pgmap v184: 73 pgs: 16 active+undersized, 7 peering, 4 stale+active+clean, 2 active+undersized+degraded, 44 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3/84 objects degraded (3.571%) 2024-09-21T10:02:06.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:05 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4046161853' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:02:06.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:05 smithi196 ceph-mon[84784]: from='client.34746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:06.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:05 smithi196 ceph-mon[84784]: from='client.44624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:06.347 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:06 smithi098 podman[123618]: 2024-09-21 10:02:06.175199785 +0000 UTC m=+0.361819275 container init 98d8757e4343a5166a83b992ad05a410f9a789e371b4490f417df2338060e6b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731) 2024-09-21T10:02:06.347 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:06 smithi098 podman[123618]: 2024-09-21 10:02:06.178313323 +0000 UTC m=+0.364932803 container start 98d8757e4343a5166a83b992ad05a410f9a789e371b4490f417df2338060e6b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-21T10:02:06.347 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:06 smithi098 ceph-osd[123640]: -- 172.21.15.98:0/497336201 <== mon.2 v2:172.21.15.98:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5634569cfa40 con 0x5634569adc00 2024-09-21T10:02:06.347 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:06 smithi098 bash[123618]: 98d8757e4343a5166a83b992ad05a410f9a789e371b4490f417df2338060e6b5 2024-09-21T10:02:06.347 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:06 smithi098 systemd[1]: Started Ceph osd.3 for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:02:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:07.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:07.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:07.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:07.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:07.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:07.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:07.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:07.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:07.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:07.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:07.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:07.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:07.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:08.097 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:08 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3[123636]: 2024-09-21T10:02:08.069+0000 7f4880912640 -1 Falling back to public interface 2024-09-21T10:02:08.520 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:02:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:08 smithi098 ceph-mon[97580]: pgmap v185: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-21T10:02:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:08 smithi098 ceph-mon[97580]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T10:02:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:08 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3065786566' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:02:08.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:08 smithi098 ceph-mon[100914]: pgmap v185: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-21T10:02:08.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:08 smithi098 ceph-mon[100914]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T10:02:08.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:08 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3065786566' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:02:08.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:08 smithi196 ceph-mon[84784]: pgmap v185: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-21T10:02:08.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:08 smithi196 ceph-mon[84784]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T10:02:08.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:08 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3065786566' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:02:09.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:09 smithi196 ceph-mon[84784]: from='client.44639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:09 smithi196 ceph-mon[84784]: from='client.34773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:09.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:09 smithi098 ceph-mon[97580]: from='client.44639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:09.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:09 smithi098 ceph-mon[97580]: from='client.34773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:09.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:09 smithi098 ceph-mon[100914]: from='client.44639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:09.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:09 smithi098 ceph-mon[100914]: from='client.34773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:10.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:02:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:02:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:10.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:02:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:02:10] "GET /metrics HTTP/1.1" 200 34041 "" "Prometheus/2.51.0" 2024-09-21T10:02:10.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:10 smithi098 ceph-mon[97580]: pgmap v186: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-21T10:02:10.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:10 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:10.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:10 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:10.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:10 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:10.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:10 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:10.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:10 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:10.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:10 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:10.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:10 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 4/84 objects degraded (4.762%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:02:10.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:10 smithi098 ceph-mon[100914]: pgmap v186: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-21T10:02:10.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:10 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:10.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:10 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:10.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:10 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:10.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:10 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:10.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:10 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:10.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:10 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:10.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:10 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 4/84 objects degraded (4.762%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:02:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:10 smithi196 ceph-mon[84784]: pgmap v186: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-21T10:02:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:10 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:10 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:10 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:10 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:10 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:10 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:10 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 4/84 objects degraded (4.762%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:02:11.347 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:11 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3[123636]: 2024-09-21T10:02:11.109+0000 7f4880912640 -1 osd.3 97 log_to_monitors true 2024-09-21T10:02:11.386 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-09-21T10:02:11.738 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:11 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/771440586' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:02:11.738 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:11 smithi098 ceph-mon[97580]: from='osd.3 [v2:172.21.15.98:6826/3950952422,v1:172.21.15.98:6827/3950952422]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T10:02:11.738 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:11 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/771440586' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:02:11.738 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:11 smithi098 ceph-mon[100914]: from='osd.3 [v2:172.21.15.98:6826/3950952422,v1:172.21.15.98:6827/3950952422]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T10:02:11.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:11 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/771440586' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:02:11.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:11 smithi196 ceph-mon[84784]: from='osd.3 [v2:172.21.15.98:6826/3950952422,v1:172.21.15.98:6827/3950952422]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T10:02:12.347 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:02:12 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3[123636]: 2024-09-21T10:02:12.084+0000 7f48746b4640 -1 osd.3 97 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T10:02:12.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:12 smithi098 ceph-mon[97580]: pgmap v187: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-21T10:02:12.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:12 smithi098 ceph-mon[97580]: from='client.34788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:12.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:12 smithi098 ceph-mon[97580]: from='osd.3 [v2:172.21.15.98:6826/3950952422,v1:172.21.15.98:6827/3950952422]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T10:02:12.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:12 smithi098 ceph-mon[97580]: osdmap e100: 8 total, 7 up, 8 in 2024-09-21T10:02:12.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:12 smithi098 ceph-mon[97580]: from='osd.3 [v2:172.21.15.98:6826/3950952422,v1:172.21.15.98:6827/3950952422]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]} : dispatch 2024-09-21T10:02:12.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:12 smithi098 ceph-mon[97580]: from='client.34794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:12.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:12 smithi098 ceph-mon[97580]: from='osd.3 [v2:172.21.15.98:6826/3950952422,v1:172.21.15.98:6827/3950952422]' entity='osd.3' 2024-09-21T10:02:12.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:12 smithi098 ceph-mon[100914]: pgmap v187: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-21T10:02:12.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:12 smithi098 ceph-mon[100914]: from='client.34788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:12.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:12 smithi098 ceph-mon[100914]: from='osd.3 [v2:172.21.15.98:6826/3950952422,v1:172.21.15.98:6827/3950952422]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T10:02:12.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:12 smithi098 ceph-mon[100914]: osdmap e100: 8 total, 7 up, 8 in 2024-09-21T10:02:12.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:12 smithi098 ceph-mon[100914]: from='osd.3 [v2:172.21.15.98:6826/3950952422,v1:172.21.15.98:6827/3950952422]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]} : dispatch 2024-09-21T10:02:12.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:12 smithi098 ceph-mon[100914]: from='client.34794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:12.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:12 smithi098 ceph-mon[100914]: from='osd.3 [v2:172.21.15.98:6826/3950952422,v1:172.21.15.98:6827/3950952422]' entity='osd.3' 2024-09-21T10:02:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:12 smithi196 ceph-mon[84784]: pgmap v187: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-21T10:02:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:12 smithi196 ceph-mon[84784]: from='client.34788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:12 smithi196 ceph-mon[84784]: from='osd.3 [v2:172.21.15.98:6826/3950952422,v1:172.21.15.98:6827/3950952422]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T10:02:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:12 smithi196 ceph-mon[84784]: osdmap e100: 8 total, 7 up, 8 in 2024-09-21T10:02:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:12 smithi196 ceph-mon[84784]: from='osd.3 [v2:172.21.15.98:6826/3950952422,v1:172.21.15.98:6827/3950952422]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]} : dispatch 2024-09-21T10:02:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:12 smithi196 ceph-mon[84784]: from='client.34794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:12 smithi196 ceph-mon[84784]: from='osd.3 [v2:172.21.15.98:6826/3950952422,v1:172.21.15.98:6827/3950952422]' entity='osd.3' 2024-09-21T10:02:13.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T10:02:13.364+0000 7f6c37c82640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-09-21T10:02:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:13 smithi196 ceph-mon[84784]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:02:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:13 smithi196 ceph-mon[84784]: osd.3 [v2:172.21.15.98:6826/3950952422,v1:172.21.15.98:6827/3950952422] boot 2024-09-21T10:02:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:13 smithi196 ceph-mon[84784]: osdmap e101: 8 total, 8 up, 8 in 2024-09-21T10:02:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:13 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T10:02:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:13 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:13 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:13 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:02:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:13 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:02:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:13 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:13 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:13 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:02:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:13 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:02:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:13 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:02:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:13 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-21T10:02:13.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[97580]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:02:13.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[97580]: osd.3 [v2:172.21.15.98:6826/3950952422,v1:172.21.15.98:6827/3950952422] boot 2024-09-21T10:02:13.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[97580]: osdmap e101: 8 total, 8 up, 8 in 2024-09-21T10:02:13.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T10:02:13.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:13.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:13.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:02:13.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:02:13.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:13.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:13.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:02:13.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:02:13.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:02:13.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-21T10:02:13.969 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 4 2024-09-21T10:02:13.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[100914]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:02:13.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[100914]: osd.3 [v2:172.21.15.98:6826/3950952422,v1:172.21.15.98:6827/3950952422] boot 2024-09-21T10:02:13.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[100914]: osdmap e101: 8 total, 8 up, 8 in 2024-09-21T10:02:13.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T10:02:13.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:13.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:13.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:02:13.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:02:13.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:13.970 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:13.970 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:02:13.970 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:02:13.970 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:02:13.970 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:13 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-21T10:02:14.656 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade pause' 2024-09-21T10:02:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:14 smithi196 ceph-mon[84784]: pgmap v189: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-21T10:02:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:14 smithi196 ceph-mon[84784]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-21T10:02:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:14 smithi196 ceph-mon[84784]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-09-21T10:02:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:14 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4222048810' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:02:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:14 smithi196 ceph-mon[84784]: osdmap e102: 8 total, 8 up, 8 in 2024-09-21T10:02:14.901 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:15.022 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:14 smithi098 ceph-mon[97580]: pgmap v189: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-21T10:02:15.022 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:14 smithi098 ceph-mon[97580]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-21T10:02:15.022 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:14 smithi098 ceph-mon[97580]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-09-21T10:02:15.023 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:14 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4222048810' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:02:15.023 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:14 smithi098 ceph-mon[97580]: osdmap e102: 8 total, 8 up, 8 in 2024-09-21T10:02:15.023 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:14 smithi098 ceph-mon[100914]: pgmap v189: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-21T10:02:15.023 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:14 smithi098 ceph-mon[100914]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-21T10:02:15.023 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:14 smithi098 ceph-mon[100914]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-09-21T10:02:15.023 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:14 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4222048810' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:02:15.023 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:14 smithi098 ceph-mon[100914]: osdmap e102: 8 total, 8 up, 8 in 2024-09-21T10:02:16.732 INFO:teuthology.orchestra.run.smithi098.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:02:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:16 smithi196 ceph-mon[84784]: pgmap v192: 73 pgs: 2 peering, 26 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-21T10:02:17.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:16 smithi098 ceph-mon[97580]: pgmap v192: 73 pgs: 2 peering, 26 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-21T10:02:17.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:16 smithi098 ceph-mon[100914]: pgmap v192: 73 pgs: 2 peering, 26 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-21T10:02:17.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch ps' 2024-09-21T10:02:17.682 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:18.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:17 smithi098 ceph-mon[97580]: from='client.34806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:18.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:17 smithi098 ceph-mon[97580]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:02:18.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:17 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:18.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:17 smithi098 ceph-mon[100914]: from='client.34806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:18.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:17 smithi098 ceph-mon[100914]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:02:18.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:17 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:18.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:17 smithi196 ceph-mon[84784]: from='client.34806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:18.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:17 smithi196 ceph-mon[84784]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:02:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:17 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:19.066 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:18 smithi098 ceph-mon[97580]: pgmap v193: 73 pgs: 2 peering, 8 active+undersized, 1 active+undersized+degraded, 62 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2/84 objects degraded (2.381%) 2024-09-21T10:02:19.067 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:18 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 2/84 objects degraded (2.381%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:02:19.067 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:18 smithi098 ceph-mon[100914]: pgmap v193: 73 pgs: 2 peering, 8 active+undersized, 1 active+undersized+degraded, 62 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2/84 objects degraded (2.381%) 2024-09-21T10:02:19.067 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:18 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 2/84 objects degraded (2.381%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:02:19.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:18 smithi196 ceph-mon[84784]: pgmap v193: 73 pgs: 2 peering, 8 active+undersized, 1 active+undersized+degraded, 62 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2/84 objects degraded (2.381%) 2024-09-21T10:02:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:18 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 2/84 objects degraded (2.381%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:02:19.455 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T10:02:19.456 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (7m) 9s ago 11m 18.2M - 0.27.0 11f11916f8cd 72b57f924744 2024-09-21T10:02:19.456 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi196 *:3000 running (6m) 9s ago 10m 85.3M - 9.4.7 954c08fa6188 7724661af1a8 2024-09-21T10:02:19.456 INFO:teuthology.orchestra.run.smithi098.stdout:mds.foo.smithi098.stcwsu smithi098 running (9m) 9s ago 9m 19.6M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be d8393a5abb20 2024-09-21T10:02:19.456 INFO:teuthology.orchestra.run.smithi098.stdout:mds.foo.smithi196.ugblah smithi196 running (9m) 9s ago 9m 15.9M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 05a68593bc2b 2024-09-21T10:02:19.456 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi196 *:8443,9283,8765 running (6m) 9s ago 14m 457M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d4969d635bcd 2024-09-21T10:02:19.456 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (6m) 9s ago 16m 519M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 da66347fe5a2 2024-09-21T10:02:19.456 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (5m) 9s ago 16m 49.7M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 0a2195915f65 2024-09-21T10:02:19.456 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi196 running (3m) 9s ago 15m 36.7M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 2776b693f5c0 2024-09-21T10:02:19.456 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (4m) 9s ago 15m 39.1M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 ef4452ba8cf0 2024-09-21T10:02:19.456 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (6m) 9s ago 10m 11.4M - 1.7.0 72c9c2088986 d587eb6669b7 2024-09-21T10:02:19.456 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi196 *:9100 running (6m) 9s ago 10m 10.4M - 1.7.0 72c9c2088986 7a8564adbc71 2024-09-21T10:02:19.456 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (2m) 9s ago 14m 49.3M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 9c8da4094eee 2024-09-21T10:02:19.457 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (80s) 9s ago 14m 41.5M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 c84cfb0f2fd7 2024-09-21T10:02:19.457 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (46s) 9s ago 13m 44.1M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 47ba74506891 2024-09-21T10:02:19.457 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (13s) 9s ago 13m 14.4M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 98d8757e4343 2024-09-21T10:02:19.457 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi196 running (13m) 9s ago 13m 52.2M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 1693914f1a8b 2024-09-21T10:02:19.457 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi196 running (12m) 9s ago 12m 49.1M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be cb8e4afb2db7 2024-09-21T10:02:19.457 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi196 running (12m) 9s ago 12m 49.0M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 8ba9647073b2 2024-09-21T10:02:19.457 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi196 running (11m) 9s ago 11m 53.7M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 540c98a3c436 2024-09-21T10:02:19.457 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi196 *:9095 running (6m) 9s ago 11m 60.9M - 2.51.0 1d3b7f56885b b30b25036dc7 2024-09-21T10:02:20.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:19 smithi098 ceph-mon[97580]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/84 objects degraded (2.381%), 1 pg degraded) 2024-09-21T10:02:20.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:19 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:02:20.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:19 smithi098 ceph-mon[100914]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/84 objects degraded (2.381%), 1 pg degraded) 2024-09-21T10:02:20.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:19 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:02:20.130 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch ps' 2024-09-21T10:02:20.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:19 smithi196 ceph-mon[84784]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/84 objects degraded (2.381%), 1 pg degraded) 2024-09-21T10:02:20.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:19 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:02:20.372 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:20.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:02:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:02:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:20.792 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:02:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:02:20] "GET /metrics HTTP/1.1" 200 34041 "" "Prometheus/2.51.0" 2024-09-21T10:02:21.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:20 smithi098 ceph-mon[97580]: pgmap v194: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:21.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:20 smithi098 ceph-mon[97580]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:21.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:20 smithi098 ceph-mon[100914]: pgmap v194: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:21.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:20 smithi098 ceph-mon[100914]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:21.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:20 smithi196 ceph-mon[84784]: pgmap v194: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:20 smithi196 ceph-mon[84784]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:22.150 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T10:02:22.150 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (7m) 12s ago 11m 18.2M - 0.27.0 11f11916f8cd 72b57f924744 2024-09-21T10:02:22.150 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi196 *:3000 running (6m) 11s ago 10m 85.3M - 9.4.7 954c08fa6188 7724661af1a8 2024-09-21T10:02:22.150 INFO:teuthology.orchestra.run.smithi098.stdout:mds.foo.smithi098.stcwsu smithi098 running (9m) 12s ago 9m 19.6M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be d8393a5abb20 2024-09-21T10:02:22.150 INFO:teuthology.orchestra.run.smithi098.stdout:mds.foo.smithi196.ugblah smithi196 running (9m) 11s ago 9m 15.9M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 05a68593bc2b 2024-09-21T10:02:22.150 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi196 *:8443,9283,8765 running (6m) 11s ago 15m 457M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d4969d635bcd 2024-09-21T10:02:22.150 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (6m) 12s ago 16m 519M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 da66347fe5a2 2024-09-21T10:02:22.150 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (5m) 12s ago 16m 49.7M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 0a2195915f65 2024-09-21T10:02:22.151 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi196 running (4m) 11s ago 15m 36.7M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 2776b693f5c0 2024-09-21T10:02:22.151 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (4m) 12s ago 15m 39.1M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 ef4452ba8cf0 2024-09-21T10:02:22.151 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (7m) 12s ago 10m 11.4M - 1.7.0 72c9c2088986 d587eb6669b7 2024-09-21T10:02:22.151 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi196 *:9100 running (6m) 11s ago 10m 10.4M - 1.7.0 72c9c2088986 7a8564adbc71 2024-09-21T10:02:22.151 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (2m) 12s ago 14m 49.3M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 9c8da4094eee 2024-09-21T10:02:22.151 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (82s) 12s ago 14m 41.5M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 c84cfb0f2fd7 2024-09-21T10:02:22.151 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (49s) 12s ago 13m 44.1M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 47ba74506891 2024-09-21T10:02:22.151 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (16s) 12s ago 13m 14.4M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 98d8757e4343 2024-09-21T10:02:22.151 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi196 running (13m) 11s ago 13m 52.2M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 1693914f1a8b 2024-09-21T10:02:22.151 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi196 running (12m) 11s ago 12m 49.1M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be cb8e4afb2db7 2024-09-21T10:02:22.151 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi196 running (12m) 11s ago 12m 49.0M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 8ba9647073b2 2024-09-21T10:02:22.151 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi196 running (12m) 11s ago 12m 53.7M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 540c98a3c436 2024-09-21T10:02:22.151 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi196 *:9095 running (6m) 11s ago 11m 60.9M - 2.51.0 1d3b7f56885b b30b25036dc7 2024-09-21T10:02:22.830 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph versions' 2024-09-21T10:02:23.073 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:22 smithi098 ceph-mon[97580]: pgmap v195: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:23.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:22 smithi098 ceph-mon[100914]: pgmap v195: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:23.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:23.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:22 smithi196 ceph-mon[84784]: pgmap v195: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:24.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:23 smithi098 ceph-mon[97580]: from='client.44690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:24.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:23 smithi098 ceph-mon[100914]: from='client.44690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:24.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:23 smithi196 ceph-mon[84784]: from='client.44690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:24.886 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:02:24.886 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-09-21T10:02:24.886 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 3 2024-09-21T10:02:24.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:02:24.886 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-09-21T10:02:24.886 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T10:02:24.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:02:24.887 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-09-21T10:02:24.887 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)": 4, 2024-09-21T10:02:24.887 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 4 2024-09-21T10:02:24.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:02:24.887 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": { 2024-09-21T10:02:24.887 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)": 2 2024-09-21T10:02:24.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:02:24.887 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-09-21T10:02:24.887 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)": 6, 2024-09-21T10:02:24.887 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 9 2024-09-21T10:02:24.887 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:02:24.887 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:02:25.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:24 smithi098 ceph-mon[97580]: pgmap v196: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:25.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:24 smithi098 ceph-mon[100914]: pgmap v196: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:25.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:24 smithi196 ceph-mon[84784]: pgmap v196: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:25.646 DEBUG:teuthology.parallel:result is None 2024-09-21T10:02:25.647 INFO:teuthology.run_tasks:Running task print... 2024-09-21T10:02:25.662 INFO:teuthology.task.print:**** done upgrading first half of cluster **** 2024-09-21T10:02:25.662 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-09-21T10:02:25.674 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T10:02:25.674 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-21T10:02:25.674 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph mgr dump --format=json 2024-09-21T10:02:25.920 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:26.038 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:25 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1817994645' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:02:26.039 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:25 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1817994645' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:02:26.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:25 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1817994645' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:02:27.012 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:26 smithi098 ceph-mon[100914]: pgmap v197: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:27.012 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:26 smithi098 ceph-mon[97580]: pgmap v197: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:26 smithi196 ceph-mon[84784]: pgmap v197: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:27.731 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:02:28.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:28 smithi098 ceph-mon[97580]: pgmap v198: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:28.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:28 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:28.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:28 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3373448460' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T10:02:28.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:28 smithi098 ceph-mon[100914]: pgmap v198: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:28.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:28 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:28.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:28 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3373448460' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T10:02:28.422 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":39,"flags":0,"active_gid":26492,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":1478814869},{"type":"v1","addr":"172.21.15.98:6801","nonce":1478814869}]},"active_addr":"172.21.15.98:6801/1478814869","active_change":"2024-09-21T09:56:36.895152+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":26020,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.98:8443/","prometheus":"http://172.21.15.98:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":82,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1650823186}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":347320535}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":151849010}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":230202168}]}]} 2024-09-21T10:02:28.427 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-21T10:02:28.427 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T10:02:28.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T10:02:28.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:28 smithi196 ceph-mon[84784]: pgmap v198: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:28 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:28 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3373448460' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T10:02:28.671 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:29.445 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:29 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:29.446 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:29 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:02:29.446 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:29 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:02:29.446 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:29 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:29.446 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:29 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:29.446 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:29 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:02:29.446 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:29 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:02:29.446 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:29 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:29.446 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:29 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:29.446 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:29 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:02:29.446 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:29 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:02:29.447 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:29 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:29.447 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:29 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:29.447 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:29 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:02:29.447 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:29 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:02:29.447 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:29 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:29.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:29 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:29 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:02:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:29 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:02:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:29 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:29 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:29 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:02:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:29 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:02:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:29 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:02:30.399 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:02:30.399 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":102,"fsid":"26a15512-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:45:33.813487+0000","modified":"2024-09-21T10:02:14.082491+0000","last_up_change":"2024-09-21T10:02:13.080936+0000","last_in_change":"2024-09-21T09:50:10.725457+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:48:34.781592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:51:03.408228+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":78,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:52:19.135018+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:52:19.606631+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"579f13de-565d-4801-b65e-f27d1cde297a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":82,"up_from":86,"up_thru":101,"down_at":83,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1869651717},{"type":"v1","addr":"172.21.15.98:6803","nonce":1869651717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1869651717},{"type":"v1","addr":"172.21.15.98:6805","nonce":1869651717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1869651717},{"type":"v1","addr":"172.21.15.98:6809","nonce":1869651717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1869651717},{"type":"v1","addr":"172.21.15.98:6807","nonce":1869651717}]},"public_addr":"172.21.15.98:6803/1869651717","cluster_addr":"172.21.15.98:6805/1869651717","heartbeat_back_addr":"172.21.15.98:6809/1869651717","heartbeat_front_addr":"172.21.15.98:6807/1869651717","state":["exists","up"]},{"osd":1,"uuid":"6d31ac4e-dc1a-4051-8266-ba821106242f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":87,"up_from":91,"up_thru":101,"down_at":88,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":4249036577},{"type":"v1","addr":"172.21.15.98:6811","nonce":4249036577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":4249036577},{"type":"v1","addr":"172.21.15.98:6813","nonce":4249036577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4249036577},{"type":"v1","addr":"172.21.15.98:6817","nonce":4249036577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":4249036577},{"type":"v1","addr":"172.21.15.98:6815","nonce":4249036577}]},"public_addr":"172.21.15.98:6811/4249036577","cluster_addr":"172.21.15.98:6813/4249036577","heartbeat_back_addr":"172.21.15.98:6817/4249036577","heartbeat_front_addr":"172.21.15.98:6815/4249036577","state":["exists","up"]},{"osd":2,"uuid":"62e22789-74be-4098-a9b9-62671fb4cac1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":92,"up_from":96,"up_thru":101,"down_at":93,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1274766038},{"type":"v1","addr":"172.21.15.98:6819","nonce":1274766038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1274766038},{"type":"v1","addr":"172.21.15.98:6821","nonce":1274766038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1274766038},{"type":"v1","addr":"172.21.15.98:6825","nonce":1274766038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1274766038},{"type":"v1","addr":"172.21.15.98:6823","nonce":1274766038}]},"public_addr":"172.21.15.98:6819/1274766038","cluster_addr":"172.21.15.98:6821/1274766038","heartbeat_back_addr":"172.21.15.98:6825/1274766038","heartbeat_front_addr":"172.21.15.98:6823/1274766038","state":["exists","up"]},{"osd":3,"uuid":"77f653a6-69b0-437a-960d-c3fb036708be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":97,"up_from":101,"up_thru":101,"down_at":98,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3950952422},{"type":"v1","addr":"172.21.15.98:6827","nonce":3950952422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3950952422},{"type":"v1","addr":"172.21.15.98:6829","nonce":3950952422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3950952422},{"type":"v1","addr":"172.21.15.98:6833","nonce":3950952422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3950952422},{"type":"v1","addr":"172.21.15.98:6831","nonce":3950952422}]},"public_addr":"172.21.15.98:6827/3950952422","cluster_addr":"172.21.15.98:6829/3950952422","heartbeat_back_addr":"172.21.15.98:6833/3950952422","heartbeat_front_addr":"172.21.15.98:6831/3950952422","state":["exists","up"]},{"osd":4,"uuid":"28ec4928-7f00-46a9-b8c5-95f0b7e6be75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6801","nonce":158468112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6803","nonce":158468112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6807","nonce":158468112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6805","nonce":158468112}]},"public_addr":"172.21.15.196:6801/158468112","cluster_addr":"172.21.15.196:6803/158468112","heartbeat_back_addr":"172.21.15.196:6807/158468112","heartbeat_front_addr":"172.21.15.196:6805/158468112","state":["exists","up"]},{"osd":5,"uuid":"cdcc44eb-e27f-4d23-9ecb-03bebdc82c96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6809","nonce":2341878784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6811","nonce":2341878784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6815","nonce":2341878784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6813","nonce":2341878784}]},"public_addr":"172.21.15.196:6809/2341878784","cluster_addr":"172.21.15.196:6811/2341878784","heartbeat_back_addr":"172.21.15.196:6815/2341878784","heartbeat_front_addr":"172.21.15.196:6813/2341878784","state":["exists","up"]},{"osd":6,"uuid":"566cfb44-1b21-45fc-b053-7205b1c10d4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6817","nonce":3059702155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6819","nonce":3059702155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6823","nonce":3059702155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6821","nonce":3059702155}]},"public_addr":"172.21.15.196:6817/3059702155","cluster_addr":"172.21.15.196:6819/3059702155","heartbeat_back_addr":"172.21.15.196:6823/3059702155","heartbeat_front_addr":"172.21.15.196:6821/3059702155","state":["exists","up"]},{"osd":7,"uuid":"228404c1-3910-4933-b023-55aa8b01aee4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6825","nonce":2155020234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6827","nonce":2155020234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6831","nonce":2155020234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6829","nonce":2155020234}]},"public_addr":"172.21.15.196:6825/2155020234","cluster_addr":"172.21.15.196:6827/2155020234","heartbeat_back_addr":"172.21.15.196:6831/2155020234","heartbeat_front_addr":"172.21.15.196:6829/2155020234","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:47:43.263977+0000","dead_epoch":82},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:06.738746+0000","dead_epoch":87},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:30.883578+0000","dead_epoch":92},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:54.216866+0000","dead_epoch":97},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:49:15.599276+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:49:38.581033+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:01.504502+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:25.964220+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/227493886":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/4208010999":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/493934303":"2024-09-22T09:56:36.894927+0000","172.21.15.196:0/2330152142":"2024-09-22T09:55:58.115206+0000","172.21.15.98:0/3466051265":"2024-09-22T09:54:48.070235+0000","172.21.15.98:0/900860885":"2024-09-22T09:54:48.070235+0000","172.21.15.98:0/675672768":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/4139436413":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/3656555554":"2024-09-22T09:45:54.237657+0000","172.21.15.98:6800/1387543382":"2024-09-22T09:54:48.070235+0000","172.21.15.196:0/79302526":"2024-09-22T09:55:58.115206+0000","172.21.15.196:0/1564192582":"2024-09-22T09:55:58.115206+0000","172.21.15.196:6835/2652695583":"2024-09-22T09:55:58.115206+0000","172.21.15.98:6801/3394444334":"2024-09-22T09:56:36.894927+0000","172.21.15.98:0/262018661":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/2075804383":"2024-09-22T09:45:54.237657+0000","172.21.15.98:0/2144473621":"2024-09-22T09:45:54.237657+0000","172.21.15.98:6801/440385038":"2024-09-22T09:46:24.467967+0000","172.21.15.98:6800/440385038":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/2546189353":"2024-09-22T09:56:36.894927+0000","172.21.15.98:0/2611845896":"2024-09-22T09:56:36.894927+0000","172.21.15.98:6801/1387543382":"2024-09-22T09:54:48.070235+0000","172.21.15.98:0/3628356893":"2024-09-22T09:56:36.894927+0000","172.21.15.98:6800/1427334238":"2024-09-22T09:45:54.237657+0000","172.21.15.196:0/1546057694":"2024-09-22T09:55:58.115206+0000","172.21.15.98:6801/89887073":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/881612124":"2024-09-22T09:50:59.815356+0000","172.21.15.98:6800/3394444334":"2024-09-22T09:56:36.894927+0000","172.21.15.98:6801/1427334238":"2024-09-22T09:45:54.237657+0000","172.21.15.98:6800/89887073":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/1093274465":"2024-09-22T09:54:48.070235+0000","172.21.15.98:0/1268404616":"2024-09-22T09:56:36.894927+0000","172.21.15.98:0/271851122":"2024-09-22T09:56:36.894927+0000","172.21.15.196:0/2663325110":"2024-09-22T09:55:58.115206+0000","172.21.15.98:0/1829140501":"2024-09-22T09:54:48.070235+0000","172.21.15.196:6834/2652695583":"2024-09-22T09:55:58.115206+0000","172.21.15.98:0/3737889208":"2024-09-22T09:46:24.467967+0000","172.21.15.196:0/1036032960":"2024-09-22T09:55:58.115206+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T10:02:30.479 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:02:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:02:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:30.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:30 smithi098 ceph-mon[97580]: pgmap v199: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:30.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:30 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1956411346' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:02:30.658 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:30 smithi098 ceph-mon[100914]: pgmap v199: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:30.658 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:30 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1956411346' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:02:30.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:30 smithi196 ceph-mon[84784]: pgmap v199: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:30.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:30 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1956411346' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:02:30.979 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:02:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:02:30] "GET /metrics HTTP/1.1" 200 34030 "" "Prometheus/2.51.0" 2024-09-21T10:02:31.004 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T10:02:31.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T10:02:31.264 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:32.831 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:32 smithi098 ceph-mon[97580]: pgmap v200: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:32.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:32 smithi098 ceph-mon[100914]: pgmap v200: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:32.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:32 smithi196 ceph-mon[84784]: pgmap v200: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:02:33.239 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":102,"fsid":"26a15512-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:45:33.813487+0000","modified":"2024-09-21T10:02:14.082491+0000","last_up_change":"2024-09-21T10:02:13.080936+0000","last_in_change":"2024-09-21T09:50:10.725457+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:48:34.781592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:51:03.408228+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":78,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:52:19.135018+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:52:19.606631+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"579f13de-565d-4801-b65e-f27d1cde297a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":82,"up_from":86,"up_thru":101,"down_at":83,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1869651717},{"type":"v1","addr":"172.21.15.98:6803","nonce":1869651717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1869651717},{"type":"v1","addr":"172.21.15.98:6805","nonce":1869651717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1869651717},{"type":"v1","addr":"172.21.15.98:6809","nonce":1869651717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1869651717},{"type":"v1","addr":"172.21.15.98:6807","nonce":1869651717}]},"public_addr":"172.21.15.98:6803/1869651717","cluster_addr":"172.21.15.98:6805/1869651717","heartbeat_back_addr":"172.21.15.98:6809/1869651717","heartbeat_front_addr":"172.21.15.98:6807/1869651717","state":["exists","up"]},{"osd":1,"uuid":"6d31ac4e-dc1a-4051-8266-ba821106242f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":87,"up_from":91,"up_thru":101,"down_at":88,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":4249036577},{"type":"v1","addr":"172.21.15.98:6811","nonce":4249036577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":4249036577},{"type":"v1","addr":"172.21.15.98:6813","nonce":4249036577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4249036577},{"type":"v1","addr":"172.21.15.98:6817","nonce":4249036577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":4249036577},{"type":"v1","addr":"172.21.15.98:6815","nonce":4249036577}]},"public_addr":"172.21.15.98:6811/4249036577","cluster_addr":"172.21.15.98:6813/4249036577","heartbeat_back_addr":"172.21.15.98:6817/4249036577","heartbeat_front_addr":"172.21.15.98:6815/4249036577","state":["exists","up"]},{"osd":2,"uuid":"62e22789-74be-4098-a9b9-62671fb4cac1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":92,"up_from":96,"up_thru":101,"down_at":93,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1274766038},{"type":"v1","addr":"172.21.15.98:6819","nonce":1274766038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1274766038},{"type":"v1","addr":"172.21.15.98:6821","nonce":1274766038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1274766038},{"type":"v1","addr":"172.21.15.98:6825","nonce":1274766038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1274766038},{"type":"v1","addr":"172.21.15.98:6823","nonce":1274766038}]},"public_addr":"172.21.15.98:6819/1274766038","cluster_addr":"172.21.15.98:6821/1274766038","heartbeat_back_addr":"172.21.15.98:6825/1274766038","heartbeat_front_addr":"172.21.15.98:6823/1274766038","state":["exists","up"]},{"osd":3,"uuid":"77f653a6-69b0-437a-960d-c3fb036708be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":97,"up_from":101,"up_thru":101,"down_at":98,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3950952422},{"type":"v1","addr":"172.21.15.98:6827","nonce":3950952422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3950952422},{"type":"v1","addr":"172.21.15.98:6829","nonce":3950952422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3950952422},{"type":"v1","addr":"172.21.15.98:6833","nonce":3950952422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3950952422},{"type":"v1","addr":"172.21.15.98:6831","nonce":3950952422}]},"public_addr":"172.21.15.98:6827/3950952422","cluster_addr":"172.21.15.98:6829/3950952422","heartbeat_back_addr":"172.21.15.98:6833/3950952422","heartbeat_front_addr":"172.21.15.98:6831/3950952422","state":["exists","up"]},{"osd":4,"uuid":"28ec4928-7f00-46a9-b8c5-95f0b7e6be75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6801","nonce":158468112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6803","nonce":158468112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6807","nonce":158468112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6805","nonce":158468112}]},"public_addr":"172.21.15.196:6801/158468112","cluster_addr":"172.21.15.196:6803/158468112","heartbeat_back_addr":"172.21.15.196:6807/158468112","heartbeat_front_addr":"172.21.15.196:6805/158468112","state":["exists","up"]},{"osd":5,"uuid":"cdcc44eb-e27f-4d23-9ecb-03bebdc82c96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6809","nonce":2341878784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6811","nonce":2341878784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6815","nonce":2341878784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6813","nonce":2341878784}]},"public_addr":"172.21.15.196:6809/2341878784","cluster_addr":"172.21.15.196:6811/2341878784","heartbeat_back_addr":"172.21.15.196:6815/2341878784","heartbeat_front_addr":"172.21.15.196:6813/2341878784","state":["exists","up"]},{"osd":6,"uuid":"566cfb44-1b21-45fc-b053-7205b1c10d4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6817","nonce":3059702155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6819","nonce":3059702155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6823","nonce":3059702155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6821","nonce":3059702155}]},"public_addr":"172.21.15.196:6817/3059702155","cluster_addr":"172.21.15.196:6819/3059702155","heartbeat_back_addr":"172.21.15.196:6823/3059702155","heartbeat_front_addr":"172.21.15.196:6821/3059702155","state":["exists","up"]},{"osd":7,"uuid":"228404c1-3910-4933-b023-55aa8b01aee4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6825","nonce":2155020234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6827","nonce":2155020234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6831","nonce":2155020234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6829","nonce":2155020234}]},"public_addr":"172.21.15.196:6825/2155020234","cluster_addr":"172.21.15.196:6827/2155020234","heartbeat_back_addr":"172.21.15.196:6831/2155020234","heartbeat_front_addr":"172.21.15.196:6829/2155020234","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:47:43.263977+0000","dead_epoch":82},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:06.738746+0000","dead_epoch":87},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:30.883578+0000","dead_epoch":92},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:54.216866+0000","dead_epoch":97},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:49:15.599276+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:49:38.581033+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:01.504502+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:25.964220+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/227493886":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/4208010999":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/493934303":"2024-09-22T09:56:36.894927+0000","172.21.15.196:0/2330152142":"2024-09-22T09:55:58.115206+0000","172.21.15.98:0/3466051265":"2024-09-22T09:54:48.070235+0000","172.21.15.98:0/900860885":"2024-09-22T09:54:48.070235+0000","172.21.15.98:0/675672768":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/4139436413":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/3656555554":"2024-09-22T09:45:54.237657+0000","172.21.15.98:6800/1387543382":"2024-09-22T09:54:48.070235+0000","172.21.15.196:0/79302526":"2024-09-22T09:55:58.115206+0000","172.21.15.196:0/1564192582":"2024-09-22T09:55:58.115206+0000","172.21.15.196:6835/2652695583":"2024-09-22T09:55:58.115206+0000","172.21.15.98:6801/3394444334":"2024-09-22T09:56:36.894927+0000","172.21.15.98:0/262018661":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/2075804383":"2024-09-22T09:45:54.237657+0000","172.21.15.98:0/2144473621":"2024-09-22T09:45:54.237657+0000","172.21.15.98:6801/440385038":"2024-09-22T09:46:24.467967+0000","172.21.15.98:6800/440385038":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/2546189353":"2024-09-22T09:56:36.894927+0000","172.21.15.98:0/2611845896":"2024-09-22T09:56:36.894927+0000","172.21.15.98:6801/1387543382":"2024-09-22T09:54:48.070235+0000","172.21.15.98:0/3628356893":"2024-09-22T09:56:36.894927+0000","172.21.15.98:6800/1427334238":"2024-09-22T09:45:54.237657+0000","172.21.15.196:0/1546057694":"2024-09-22T09:55:58.115206+0000","172.21.15.98:6801/89887073":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/881612124":"2024-09-22T09:50:59.815356+0000","172.21.15.98:6800/3394444334":"2024-09-22T09:56:36.894927+0000","172.21.15.98:6801/1427334238":"2024-09-22T09:45:54.237657+0000","172.21.15.98:6800/89887073":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/1093274465":"2024-09-22T09:54:48.070235+0000","172.21.15.98:0/1268404616":"2024-09-22T09:56:36.894927+0000","172.21.15.98:0/271851122":"2024-09-22T09:56:36.894927+0000","172.21.15.196:0/2663325110":"2024-09-22T09:55:58.115206+0000","172.21.15.98:0/1829140501":"2024-09-22T09:54:48.070235+0000","172.21.15.196:6834/2652695583":"2024-09-22T09:55:58.115206+0000","172.21.15.98:0/3737889208":"2024-09-22T09:46:24.467967+0000","172.21.15.196:0/1036032960":"2024-09-22T09:55:58.115206+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T10:02:33.561 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:33 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1164184335' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:02:33.562 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:33 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1164184335' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:02:33.855 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T10:02:33.855 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T10:02:33.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T10:02:33.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T10:02:33.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T10:02:33.857 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T10:02:33.857 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T10:02:33.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T10:02:33.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:33 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1164184335' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:02:34.368 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:34.373 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:34.374 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:34.374 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:34.376 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:34.376 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:34.378 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:34.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:34 smithi098 ceph-mon[97580]: pgmap v201: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:34.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:34 smithi098 ceph-mon[100914]: pgmap v201: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:34 smithi196 ceph-mon[84784]: pgmap v201: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:36.717 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:36 smithi098 ceph-mon[97580]: pgmap v202: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:36.717 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:36 smithi098 ceph-mon[100914]: pgmap v202: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:36.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:36 smithi196 ceph-mon[84784]: pgmap v202: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:37.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:37.784 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:37.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:38.758 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:38 smithi098 ceph-mon[97580]: pgmap v203: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:38.758 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:38 smithi098 ceph-mon[100914]: pgmap v203: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:38.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:38 smithi196 ceph-mon[84784]: pgmap v203: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:40.500 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:02:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:02:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:40.830 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:02:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:02:40] "GET /metrics HTTP/1.1" 200 34032 "" "Prometheus/2.51.0" 2024-09-21T10:02:40.830 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:40 smithi098 ceph-mon[100914]: pgmap v204: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:40.830 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:40 smithi098 ceph-mon[97580]: pgmap v204: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:40.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:40 smithi196 ceph-mon[84784]: pgmap v204: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:42.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:42 smithi098 ceph-mon[100914]: pgmap v205: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:42.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:42 smithi098 ceph-mon[97580]: pgmap v205: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:42.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:42 smithi196 ceph-mon[84784]: pgmap v205: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:44.821 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:44 smithi098 ceph-mon[100914]: pgmap v206: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:44.821 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:44 smithi098 ceph-mon[97580]: pgmap v206: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:44.845 INFO:teuthology.orchestra.run.smithi098.stdout:433791696904 2024-09-21T10:02:44.846 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T10:02:44.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:44 smithi196 ceph-mon[84784]: pgmap v206: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:45.321 INFO:teuthology.orchestra.run.smithi098.stdout:158913790112 2024-09-21T10:02:45.321 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T10:02:46.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:46 smithi098 ceph-mon[97580]: pgmap v207: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:46.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:46 smithi098 ceph-mon[100914]: pgmap v207: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:46.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:46 smithi196 ceph-mon[84784]: pgmap v207: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:47.954 INFO:teuthology.orchestra.run.smithi098.stdout:369367187492 2024-09-21T10:02:47.955 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T10:02:48.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:48 smithi098 ceph-mon[97580]: pgmap v208: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:48.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:48 smithi098 ceph-mon[100914]: pgmap v208: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:48.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:48 smithi196 ceph-mon[84784]: pgmap v208: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:48.918 INFO:teuthology.orchestra.run.smithi098.stdout:184683593883 2024-09-21T10:02:48.919 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T10:02:49.505 INFO:teuthology.orchestra.run.smithi098.stdout:390842023958 2024-09-21T10:02:49.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T10:02:50.452 INFO:teuthology.orchestra.run.smithi098.stdout:133143986343 2024-09-21T10:02:50.453 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T10:02:50.514 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:02:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:02:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:50.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:50 smithi098 ceph-mon[97580]: pgmap v209: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:50.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:02:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:02:50] "GET /metrics HTTP/1.1" 200 34032 "" "Prometheus/2.51.0" 2024-09-21T10:02:50.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:50 smithi098 ceph-mon[100914]: pgmap v209: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:50 smithi196 ceph-mon[84784]: pgmap v209: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:50.998 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:51.009 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:51.040 INFO:teuthology.orchestra.run.smithi098.stdout:210453397655 2024-09-21T10:02:51.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T10:02:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:51.455 INFO:teuthology.orchestra.run.smithi098.stdout:412316860432 2024-09-21T10:02:51.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T10:02:52.763 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:52 smithi098 ceph-mon[97580]: pgmap v210: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:52.764 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:52.764 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:52 smithi098 ceph-mon[100914]: pgmap v210: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:52.764 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:52.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:52 smithi196 ceph-mon[84784]: pgmap v210: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:52.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:53.434 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:53.434 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:54.578 INFO:teuthology.orchestra.run.smithi098.stdout:433791696906 2024-09-21T10:02:54.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:54 smithi098 ceph-mon[97580]: pgmap v211: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:54.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:54 smithi098 ceph-mon[100914]: pgmap v211: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:54.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:54 smithi196 ceph-mon[84784]: pgmap v211: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:55.692 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:55 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3050680113' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T10:02:55.692 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:55 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3050680113' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T10:02:55.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:55 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3050680113' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T10:02:56.737 INFO:tasks.cephadm.ceph_manager.ceph:need seq 433791696904 got 433791696906 for osd.3 2024-09-21T10:02:56.737 DEBUG:teuthology.parallel:result is None 2024-09-21T10:02:56.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:56 smithi098 ceph-mon[97580]: pgmap v212: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:56.791 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:56 smithi098 ceph-mon[100914]: pgmap v212: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:56.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:56 smithi196 ceph-mon[84784]: pgmap v212: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:57.354 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:58.697 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:58.698 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:58.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:58 smithi098 ceph-mon[97580]: pgmap v213: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:58.793 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:58 smithi098 ceph-mon[100914]: pgmap v213: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:58 smithi196 ceph-mon[84784]: pgmap v213: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:02:59.166 INFO:teuthology.orchestra.run.smithi098.stdout:158913790115 2024-09-21T10:02:59.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:02:59 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2385130501' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T10:02:59.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:02:59 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2385130501' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T10:02:59.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:02:59 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2385130501' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T10:03:00.539 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:03:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:03:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:00.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:00 smithi098 ceph-mon[97580]: pgmap v214: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:00.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:00 smithi098 ceph-mon[100914]: pgmap v214: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:00.685 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790112 got 158913790115 for osd.5 2024-09-21T10:03:00.686 DEBUG:teuthology.parallel:result is None 2024-09-21T10:03:00.863 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:03:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:03:00] "GET /metrics HTTP/1.1" 200 34023 "" "Prometheus/2.51.0" 2024-09-21T10:03:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:00 smithi196 ceph-mon[84784]: pgmap v214: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:01.049 INFO:teuthology.orchestra.run.smithi098.stdout:369367187495 2024-09-21T10:03:01.730 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:01 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4243167934' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T10:03:01.730 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:01 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4243167934' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T10:03:01.756 INFO:tasks.cephadm.ceph_manager.ceph:need seq 369367187492 got 369367187495 for osd.0 2024-09-21T10:03:01.756 DEBUG:teuthology.parallel:result is None 2024-09-21T10:03:01.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:01 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4243167934' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T10:03:02.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:02 smithi098 ceph-mon[97580]: pgmap v215: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:02.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:02 smithi098 ceph-mon[100914]: pgmap v215: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:02.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:02 smithi196 ceph-mon[84784]: pgmap v215: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:03.251 INFO:teuthology.orchestra.run.smithi098.stdout:184683593886 2024-09-21T10:03:03.471 INFO:teuthology.orchestra.run.smithi098.stdout:133143986345 2024-09-21T10:03:03.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:03 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1520931849' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T10:03:03.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:03 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2025463399' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T10:03:03.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:03 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1520931849' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T10:03:03.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:03 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2025463399' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T10:03:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:03 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1520931849' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T10:03:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:03 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2025463399' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T10:03:04.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:04 smithi098 ceph-mon[97580]: pgmap v216: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:04.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:04 smithi098 ceph-mon[100914]: pgmap v216: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:04.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:04 smithi196 ceph-mon[84784]: pgmap v216: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:05.121 INFO:teuthology.orchestra.run.smithi098.stdout:412316860434 2024-09-21T10:03:05.492 INFO:teuthology.orchestra.run.smithi098.stdout:210453397659 2024-09-21T10:03:05.734 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:05 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1987663042' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T10:03:05.734 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:05 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/734628167' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T10:03:05.735 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:05 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1987663042' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T10:03:05.735 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:05 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/734628167' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T10:03:05.758 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986343 got 133143986345 for osd.4 2024-09-21T10:03:05.758 DEBUG:teuthology.parallel:result is None 2024-09-21T10:03:05.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:05 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1987663042' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T10:03:05.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:05 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/734628167' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T10:03:06.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:06 smithi098 ceph-mon[97580]: pgmap v217: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:06.593 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:06 smithi098 ceph-mon[100914]: pgmap v217: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:06.594 INFO:tasks.cephadm.ceph_manager.ceph:need seq 412316860432 got 412316860434 for osd.2 2024-09-21T10:03:06.594 DEBUG:teuthology.parallel:result is None 2024-09-21T10:03:06.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:06 smithi196 ceph-mon[84784]: pgmap v217: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:07.490 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593883 got 184683593886 for osd.6 2024-09-21T10:03:07.490 DEBUG:teuthology.parallel:result is None 2024-09-21T10:03:07.736 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:07.736 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:08.383 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397655 got 210453397659 for osd.7 2024-09-21T10:03:08.383 DEBUG:teuthology.parallel:result is None 2024-09-21T10:03:08.766 INFO:teuthology.orchestra.run.smithi098.stdout:390842023962 2024-09-21T10:03:08.767 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:08 smithi098 ceph-mon[97580]: pgmap v218: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:08.767 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:08 smithi098 ceph-mon[100914]: pgmap v218: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:08 smithi196 ceph-mon[84784]: pgmap v218: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:09.516 INFO:tasks.cephadm.ceph_manager.ceph:need seq 390842023958 got 390842023962 for osd.1 2024-09-21T10:03:09.516 DEBUG:teuthology.parallel:result is None 2024-09-21T10:03:09.516 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T10:03:09.516 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:03:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:09.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:09 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3836505188' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T10:03:09.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:09 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3836505188' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T10:03:09.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:09 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3836505188' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T10:03:10.584 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:03:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:03:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:10.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:10 smithi098 ceph-mon[100914]: pgmap v219: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:10.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:10 smithi098 ceph-mon[97580]: pgmap v219: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:10.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:03:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:03:10] "GET /metrics HTTP/1.1" 200 34032 "" "Prometheus/2.51.0" 2024-09-21T10:03:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:10 smithi196 ceph-mon[84784]: pgmap v219: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:11.558 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:03:11.559 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:03:12.293 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-09-21T10:03:11.017540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6289,"num_read_kb":13765,"num_write":1056,"num_write_kb":14251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1132,"ondisk_log_size":1132,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1326504,"kb_used_data":22828,"kb_used_omap":35,"kb_used_meta":1303516,"kb_avail":748634712,"statfs":{"total":767960285184,"available":766601945088,"internally_reserved":0,"allocated":23375872,"data_stored":19051694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36353,"internal_metadata":1334800895},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002878"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.098066+0000","last_change":"2024-09-21T10:02:14.097991+0000","last_active":"2024-09-21T10:02:14.098066+0000","last_peered":"2024-09-21T10:02:14.098066+0000","last_clean":"2024-09-21T10:02:14.098066+0000","last_became_active":"2024-09-21T10:02:14.097407+0000","last_became_peered":"2024-09-21T10:02:14.097407+0000","last_unstale":"2024-09-21T10:02:14.098066+0000","last_undegraded":"2024-09-21T10:02:14.098066+0000","last_fullsized":"2024-09-21T10:02:14.098066+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:18:42.098886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":69,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.085145+0000","last_change":"2024-09-21T10:01:41.360784+0000","last_active":"2024-09-21T10:02:13.085145+0000","last_peered":"2024-09-21T10:02:13.085145+0000","last_clean":"2024-09-21T10:02:13.085145+0000","last_became_active":"2024-09-21T10:01:41.360462+0000","last_became_peered":"2024-09-21T10:01:41.360462+0000","last_unstale":"2024-09-21T10:02:13.085145+0000","last_undegraded":"2024-09-21T10:02:13.085145+0000","last_fullsized":"2024-09-21T10:02:13.085145+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:11.867275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":66,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.103373+0000","last_change":"2024-09-21T10:02:14.103373+0000","last_active":"2024-09-21T10:02:14.103373+0000","last_peered":"2024-09-21T10:02:14.103373+0000","last_clean":"2024-09-21T10:02:14.103373+0000","last_became_active":"2024-09-21T10:02:14.102985+0000","last_became_peered":"2024-09-21T10:02:14.102985+0000","last_unstale":"2024-09-21T10:02:14.103373+0000","last_undegraded":"2024-09-21T10:02:14.103373+0000","last_fullsized":"2024-09-21T10:02:14.103373+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:03.418174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099028+0000","last_change":"2024-09-21T10:02:14.098949+0000","last_active":"2024-09-21T10:02:14.099028+0000","last_peered":"2024-09-21T10:02:14.099028+0000","last_clean":"2024-09-21T10:02:14.099028+0000","last_became_active":"2024-09-21T10:02:14.098646+0000","last_became_peered":"2024-09-21T10:02:14.098646+0000","last_unstale":"2024-09-21T10:02:14.099028+0000","last_undegraded":"2024-09-21T10:02:14.099028+0000","last_fullsized":"2024-09-21T10:02:14.099028+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:50:56.862529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099148+0000","last_change":"2024-09-21T10:02:14.098205+0000","last_active":"2024-09-21T10:02:14.099148+0000","last_peered":"2024-09-21T10:02:14.099148+0000","last_clean":"2024-09-21T10:02:14.099148+0000","last_became_active":"2024-09-21T10:02:14.097911+0000","last_became_peered":"2024-09-21T10:02:14.097911+0000","last_unstale":"2024-09-21T10:02:14.099148+0000","last_undegraded":"2024-09-21T10:02:14.099148+0000","last_fullsized":"2024-09-21T10:02:14.099148+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:55.971191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.085481+0000","last_change":"2024-09-21T09:52:20.371775+0000","last_active":"2024-09-21T10:02:13.085481+0000","last_peered":"2024-09-21T10:02:13.085481+0000","last_clean":"2024-09-21T10:02:13.085481+0000","last_became_active":"2024-09-21T09:52:20.371509+0000","last_became_peered":"2024-09-21T09:52:20.371509+0000","last_unstale":"2024-09-21T10:02:13.085481+0000","last_undegraded":"2024-09-21T10:02:13.085481+0000","last_fullsized":"2024-09-21T10:02:13.085481+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:43:21.822524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097891+0000","last_change":"2024-09-21T10:02:14.097817+0000","last_active":"2024-09-21T10:02:14.097891+0000","last_peered":"2024-09-21T10:02:14.097891+0000","last_clean":"2024-09-21T10:02:14.097891+0000","last_became_active":"2024-09-21T10:02:14.097466+0000","last_became_peered":"2024-09-21T10:02:14.097466+0000","last_unstale":"2024-09-21T10:02:14.097891+0000","last_undegraded":"2024-09-21T10:02:14.097891+0000","last_fullsized":"2024-09-21T10:02:14.097891+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:57.883118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":75,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.090786+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:02:13.090786+0000","last_peered":"2024-09-21T10:02:13.090786+0000","last_clean":"2024-09-21T10:02:13.090786+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:02:13.090786+0000","last_undegraded":"2024-09-21T10:02:13.090786+0000","last_fullsized":"2024-09-21T10:02:13.090786+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":68,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086446+0000","last_change":"2024-09-21T09:59:54.265455+0000","last_active":"2024-09-21T10:02:13.086446+0000","last_peered":"2024-09-21T10:02:13.086446+0000","last_clean":"2024-09-21T10:02:13.086446+0000","last_became_active":"2024-09-21T09:59:54.265110+0000","last_became_peered":"2024-09-21T09:59:54.265110+0000","last_unstale":"2024-09-21T10:02:13.086446+0000","last_undegraded":"2024-09-21T10:02:13.086446+0000","last_fullsized":"2024-09-21T10:02:13.086446+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:04:13.404874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.089734+0000","last_change":"2024-09-21T09:52:20.402753+0000","last_active":"2024-09-21T10:02:13.089734+0000","last_peered":"2024-09-21T10:02:13.089734+0000","last_clean":"2024-09-21T10:02:13.089734+0000","last_became_active":"2024-09-21T09:52:20.402462+0000","last_became_peered":"2024-09-21T09:52:20.402462+0000","last_unstale":"2024-09-21T10:02:13.089734+0000","last_undegraded":"2024-09-21T10:02:13.089734+0000","last_fullsized":"2024-09-21T10:02:13.089734+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:14.036869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":69,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086567+0000","last_change":"2024-09-21T10:01:07.420484+0000","last_active":"2024-09-21T10:02:13.086567+0000","last_peered":"2024-09-21T10:02:13.086567+0000","last_clean":"2024-09-21T10:02:13.086567+0000","last_became_active":"2024-09-21T10:01:07.420135+0000","last_became_peered":"2024-09-21T10:01:07.420135+0000","last_unstale":"2024-09-21T10:02:13.086567+0000","last_undegraded":"2024-09-21T10:02:13.086567+0000","last_fullsized":"2024-09-21T10:02:13.086567+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:33:10.241010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":83,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.098139+0000","last_change":"2024-09-21T10:02:14.098139+0000","last_active":"2024-09-21T10:02:14.098139+0000","last_peered":"2024-09-21T10:02:14.098139+0000","last_clean":"2024-09-21T10:02:14.098139+0000","last_became_active":"2024-09-21T10:02:14.097639+0000","last_became_peered":"2024-09-21T10:02:14.097639+0000","last_unstale":"2024-09-21T10:02:14.098139+0000","last_undegraded":"2024-09-21T10:02:14.098139+0000","last_fullsized":"2024-09-21T10:02:14.098139+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:11:26.925388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087557+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:02:13.087557+0000","last_peered":"2024-09-21T10:02:13.087557+0000","last_clean":"2024-09-21T10:02:13.087557+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:02:13.087557+0000","last_undegraded":"2024-09-21T10:02:13.087557+0000","last_fullsized":"2024-09-21T10:02:13.087557+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":66,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091038+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:02:13.091038+0000","last_peered":"2024-09-21T10:02:13.091038+0000","last_clean":"2024-09-21T10:02:13.091038+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:02:13.091038+0000","last_undegraded":"2024-09-21T10:02:13.091038+0000","last_fullsized":"2024-09-21T10:02:13.091038+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":67,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.104212+0000","last_change":"2024-09-21T10:02:14.104212+0000","last_active":"2024-09-21T10:02:14.104212+0000","last_peered":"2024-09-21T10:02:14.104212+0000","last_clean":"2024-09-21T10:02:14.104212+0000","last_became_active":"2024-09-21T10:02:14.103893+0000","last_became_peered":"2024-09-21T10:02:14.103893+0000","last_unstale":"2024-09-21T10:02:14.104212+0000","last_undegraded":"2024-09-21T10:02:14.104212+0000","last_fullsized":"2024-09-21T10:02:14.104212+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:59:21.191412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.091970+0000","last_change":"2024-09-21T10:01:07.419715+0000","last_active":"2024-09-21T10:02:14.091970+0000","last_peered":"2024-09-21T10:02:14.091970+0000","last_clean":"2024-09-21T10:02:14.091970+0000","last_became_active":"2024-09-21T10:01:07.419422+0000","last_became_peered":"2024-09-21T10:01:07.419422+0000","last_unstale":"2024-09-21T10:02:14.091970+0000","last_undegraded":"2024-09-21T10:02:14.091970+0000","last_fullsized":"2024-09-21T10:02:14.091970+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:36.424289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":67,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.096638+0000","last_change":"2024-09-21T10:02:14.096537+0000","last_active":"2024-09-21T10:02:14.096638+0000","last_peered":"2024-09-21T10:02:14.096638+0000","last_clean":"2024-09-21T10:02:14.096638+0000","last_became_active":"2024-09-21T10:02:14.096166+0000","last_became_peered":"2024-09-21T10:02:14.096166+0000","last_unstale":"2024-09-21T10:02:14.096638+0000","last_undegraded":"2024-09-21T10:02:14.096638+0000","last_fullsized":"2024-09-21T10:02:14.096638+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:54:18.927732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":66,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099205+0000","last_change":"2024-09-21T10:02:14.099205+0000","last_active":"2024-09-21T10:02:14.099205+0000","last_peered":"2024-09-21T10:02:14.099205+0000","last_clean":"2024-09-21T10:02:14.099205+0000","last_became_active":"2024-09-21T10:02:14.098598+0000","last_became_peered":"2024-09-21T10:02:14.098598+0000","last_unstale":"2024-09-21T10:02:14.099205+0000","last_undegraded":"2024-09-21T10:02:14.099205+0000","last_fullsized":"2024-09-21T10:02:14.099205+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:46.424612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":67,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091165+0000","last_change":"2024-09-21T10:01:07.424883+0000","last_active":"2024-09-21T10:02:13.091165+0000","last_peered":"2024-09-21T10:02:13.091165+0000","last_clean":"2024-09-21T10:02:13.091165+0000","last_became_active":"2024-09-21T10:01:07.424499+0000","last_became_peered":"2024-09-21T10:01:07.424499+0000","last_unstale":"2024-09-21T10:02:13.091165+0000","last_undegraded":"2024-09-21T10:02:13.091165+0000","last_fullsized":"2024-09-21T10:02:13.091165+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:46.101570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":83,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087300+0000","last_change":"2024-09-21T10:01:41.362291+0000","last_active":"2024-09-21T10:02:13.087300+0000","last_peered":"2024-09-21T10:02:13.087300+0000","last_clean":"2024-09-21T10:02:13.087300+0000","last_became_active":"2024-09-21T10:01:41.361954+0000","last_became_peered":"2024-09-21T10:01:41.361954+0000","last_unstale":"2024-09-21T10:02:13.087300+0000","last_undegraded":"2024-09-21T10:02:13.087300+0000","last_fullsized":"2024-09-21T10:02:13.087300+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:57.669671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":67,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087361+0000","last_change":"2024-09-21T10:01:41.361244+0000","last_active":"2024-09-21T10:02:13.087361+0000","last_peered":"2024-09-21T10:02:13.087361+0000","last_clean":"2024-09-21T10:02:13.087361+0000","last_became_active":"2024-09-21T10:01:41.360645+0000","last_became_peered":"2024-09-21T10:01:41.360645+0000","last_unstale":"2024-09-21T10:02:13.087361+0000","last_undegraded":"2024-09-21T10:02:13.087361+0000","last_fullsized":"2024-09-21T10:02:13.087361+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:07.604991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":66,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086023+0000","last_change":"2024-09-21T10:01:07.420824+0000","last_active":"2024-09-21T10:02:13.086023+0000","last_peered":"2024-09-21T10:02:13.086023+0000","last_clean":"2024-09-21T10:02:13.086023+0000","last_became_active":"2024-09-21T10:01:07.420387+0000","last_became_peered":"2024-09-21T10:01:07.420387+0000","last_unstale":"2024-09-21T10:02:13.086023+0000","last_undegraded":"2024-09-21T10:02:13.086023+0000","last_fullsized":"2024-09-21T10:02:13.086023+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:48.875842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":70,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097606+0000","last_change":"2024-09-21T10:02:14.097462+0000","last_active":"2024-09-21T10:02:14.097606+0000","last_peered":"2024-09-21T10:02:14.097606+0000","last_clean":"2024-09-21T10:02:14.097606+0000","last_became_active":"2024-09-21T10:02:14.097167+0000","last_became_peered":"2024-09-21T10:02:14.097167+0000","last_unstale":"2024-09-21T10:02:14.097606+0000","last_undegraded":"2024-09-21T10:02:14.097606+0000","last_fullsized":"2024-09-21T10:02:14.097606+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:54.922731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":54,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097753+0000","last_change":"2024-09-21T10:02:14.097676+0000","last_active":"2024-09-21T10:02:14.097753+0000","last_peered":"2024-09-21T10:02:14.097753+0000","last_clean":"2024-09-21T10:02:14.097753+0000","last_became_active":"2024-09-21T10:02:14.097391+0000","last_became_peered":"2024-09-21T10:02:14.097391+0000","last_unstale":"2024-09-21T10:02:14.097753+0000","last_undegraded":"2024-09-21T10:02:14.097753+0000","last_fullsized":"2024-09-21T10:02:14.097753+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:40:24.523178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":184,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087727+0000","last_change":"2024-09-21T09:59:54.265556+0000","last_active":"2024-09-21T10:02:13.087727+0000","last_peered":"2024-09-21T10:02:13.087727+0000","last_clean":"2024-09-21T10:02:13.087727+0000","last_became_active":"2024-09-21T09:59:54.265033+0000","last_became_peered":"2024-09-21T09:59:54.265033+0000","last_unstale":"2024-09-21T10:02:13.087727+0000","last_undegraded":"2024-09-21T10:02:13.087727+0000","last_fullsized":"2024-09-21T10:02:13.087727+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T09:48:34.850872+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:40.449756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":491,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.089982+0000","last_change":"2024-09-21T10:01:41.361089+0000","last_active":"2024-09-21T10:02:13.089982+0000","last_peered":"2024-09-21T10:02:13.089982+0000","last_clean":"2024-09-21T10:02:13.089982+0000","last_became_active":"2024-09-21T10:01:41.360658+0000","last_became_peered":"2024-09-21T10:01:41.360658+0000","last_unstale":"2024-09-21T10:02:13.089982+0000","last_undegraded":"2024-09-21T10:02:13.089982+0000","last_fullsized":"2024-09-21T10:02:13.089982+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:58:06.840024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":482,"num_read_kb":1581,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.100041+0000","last_change":"2024-09-21T10:02:14.099961+0000","last_active":"2024-09-21T10:02:14.100041+0000","last_peered":"2024-09-21T10:02:14.100041+0000","last_clean":"2024-09-21T10:02:14.100041+0000","last_became_active":"2024-09-21T10:02:14.099067+0000","last_became_peered":"2024-09-21T10:02:14.099067+0000","last_unstale":"2024-09-21T10:02:14.100041+0000","last_undegraded":"2024-09-21T10:02:14.100041+0000","last_fullsized":"2024-09-21T10:02:14.100041+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:41.037532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091668+0000","last_change":"2024-09-21T09:52:20.403534+0000","last_active":"2024-09-21T10:02:13.091668+0000","last_peered":"2024-09-21T10:02:13.091668+0000","last_clean":"2024-09-21T10:02:13.091668+0000","last_became_active":"2024-09-21T09:52:20.403258+0000","last_became_peered":"2024-09-21T09:52:20.403258+0000","last_unstale":"2024-09-21T10:02:13.091668+0000","last_undegraded":"2024-09-21T10:02:13.091668+0000","last_fullsized":"2024-09-21T10:02:13.091668+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:55.374343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":447,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087637+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:02:13.087637+0000","last_peered":"2024-09-21T10:02:13.087637+0000","last_clean":"2024-09-21T10:02:13.087637+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:02:13.087637+0000","last_undegraded":"2024-09-21T10:02:13.087637+0000","last_fullsized":"2024-09-21T10:02:13.087637+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097180+0000","last_change":"2024-09-21T10:02:14.097097+0000","last_active":"2024-09-21T10:02:14.097180+0000","last_peered":"2024-09-21T10:02:14.097180+0000","last_clean":"2024-09-21T10:02:14.097180+0000","last_became_active":"2024-09-21T10:02:14.096788+0000","last_became_peered":"2024-09-21T10:02:14.096788+0000","last_unstale":"2024-09-21T10:02:14.097180+0000","last_undegraded":"2024-09-21T10:02:14.097180+0000","last_fullsized":"2024-09-21T10:02:14.097180+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:05:29.711020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":76,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090168+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:02:14.090168+0000","last_peered":"2024-09-21T10:02:14.090168+0000","last_clean":"2024-09-21T10:02:14.090168+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:02:14.090168+0000","last_undegraded":"2024-09-21T10:02:14.090168+0000","last_fullsized":"2024-09-21T10:02:14.090168+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":815,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.100244+0000","last_change":"2024-09-21T10:02:14.100143+0000","last_active":"2024-09-21T10:02:14.100244+0000","last_peered":"2024-09-21T10:02:14.100244+0000","last_clean":"2024-09-21T10:02:14.100244+0000","last_became_active":"2024-09-21T10:02:14.099586+0000","last_became_peered":"2024-09-21T10:02:14.099586+0000","last_unstale":"2024-09-21T10:02:14.100244+0000","last_undegraded":"2024-09-21T10:02:14.100244+0000","last_fullsized":"2024-09-21T10:02:14.100244+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:34.064652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":55,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090600+0000","last_change":"2024-09-21T10:01:41.358268+0000","last_active":"2024-09-21T10:02:14.090600+0000","last_peered":"2024-09-21T10:02:14.090600+0000","last_clean":"2024-09-21T10:02:14.090600+0000","last_became_active":"2024-09-21T10:01:41.357816+0000","last_became_peered":"2024-09-21T10:01:41.357816+0000","last_unstale":"2024-09-21T10:02:14.090600+0000","last_undegraded":"2024-09-21T10:02:14.090600+0000","last_fullsized":"2024-09-21T10:02:14.090600+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:59:44.261636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":72,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.096377+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:02:14.096377+0000","last_peered":"2024-09-21T10:02:14.096377+0000","last_clean":"2024-09-21T10:02:14.096377+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:02:14.096377+0000","last_undegraded":"2024-09-21T10:02:14.096377+0000","last_fullsized":"2024-09-21T10:02:14.096377+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":675,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.090133+0000","last_change":"2024-09-21T10:01:07.424819+0000","last_active":"2024-09-21T10:02:13.090133+0000","last_peered":"2024-09-21T10:02:13.090133+0000","last_clean":"2024-09-21T10:02:13.090133+0000","last_became_active":"2024-09-21T10:01:07.424392+0000","last_became_peered":"2024-09-21T10:01:07.424392+0000","last_unstale":"2024-09-21T10:02:13.090133+0000","last_undegraded":"2024-09-21T10:02:13.090133+0000","last_fullsized":"2024-09-21T10:02:13.090133+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:27.266390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":67,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.088170+0000","last_change":"2024-09-21T09:59:54.267742+0000","last_active":"2024-09-21T10:02:13.088170+0000","last_peered":"2024-09-21T10:02:13.088170+0000","last_clean":"2024-09-21T10:02:13.088170+0000","last_became_active":"2024-09-21T09:59:54.267406+0000","last_became_peered":"2024-09-21T09:59:54.267406+0000","last_unstale":"2024-09-21T10:02:13.088170+0000","last_undegraded":"2024-09-21T10:02:13.088170+0000","last_fullsized":"2024-09-21T10:02:13.088170+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:25:35.968779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":80,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.103531+0000","last_change":"2024-09-21T10:02:14.103531+0000","last_active":"2024-09-21T10:02:14.103531+0000","last_peered":"2024-09-21T10:02:14.103531+0000","last_clean":"2024-09-21T10:02:14.103531+0000","last_became_active":"2024-09-21T10:02:14.103218+0000","last_became_peered":"2024-09-21T10:02:14.103218+0000","last_unstale":"2024-09-21T10:02:14.103531+0000","last_undegraded":"2024-09-21T10:02:14.103531+0000","last_fullsized":"2024-09-21T10:02:14.103531+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:40.014308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":84,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.103145+0000","last_change":"2024-09-21T10:02:14.103145+0000","last_active":"2024-09-21T10:02:14.103145+0000","last_peered":"2024-09-21T10:02:14.103145+0000","last_clean":"2024-09-21T10:02:14.103145+0000","last_became_active":"2024-09-21T10:02:14.102806+0000","last_became_peered":"2024-09-21T10:02:14.102806+0000","last_unstale":"2024-09-21T10:02:14.103145+0000","last_undegraded":"2024-09-21T10:02:14.103145+0000","last_fullsized":"2024-09-21T10:02:14.103145+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:24.348745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":760,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090783+0000","last_change":"2024-09-21T10:01:07.426065+0000","last_active":"2024-09-21T10:02:14.090783+0000","last_peered":"2024-09-21T10:02:14.090783+0000","last_clean":"2024-09-21T10:02:14.090783+0000","last_became_active":"2024-09-21T10:01:07.423163+0000","last_became_peered":"2024-09-21T10:01:07.423163+0000","last_unstale":"2024-09-21T10:02:14.090783+0000","last_undegraded":"2024-09-21T10:02:14.090783+0000","last_fullsized":"2024-09-21T10:02:14.090783+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:43.564978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":44,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090907+0000","last_change":"2024-09-21T10:01:07.422175+0000","last_active":"2024-09-21T10:02:14.090907+0000","last_peered":"2024-09-21T10:02:14.090907+0000","last_clean":"2024-09-21T10:02:14.090907+0000","last_became_active":"2024-09-21T10:01:07.421844+0000","last_became_peered":"2024-09-21T10:01:07.421844+0000","last_unstale":"2024-09-21T10:02:14.090907+0000","last_undegraded":"2024-09-21T10:02:14.090907+0000","last_fullsized":"2024-09-21T10:02:14.090907+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:17:15.192573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":55,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090704+0000","last_change":"2024-09-21T10:01:41.360338+0000","last_active":"2024-09-21T10:02:14.090704+0000","last_peered":"2024-09-21T10:02:14.090704+0000","last_clean":"2024-09-21T10:02:14.090704+0000","last_became_active":"2024-09-21T10:01:41.360050+0000","last_became_peered":"2024-09-21T10:01:41.360050+0000","last_unstale":"2024-09-21T10:02:14.090704+0000","last_undegraded":"2024-09-21T10:02:14.090704+0000","last_fullsized":"2024-09-21T10:02:14.090704+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:39:30.459393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":413,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087649+0000","last_change":"2024-09-21T09:59:54.265559+0000","last_active":"2024-09-21T10:02:13.087649+0000","last_peered":"2024-09-21T10:02:13.087649+0000","last_clean":"2024-09-21T10:02:13.087649+0000","last_became_active":"2024-09-21T09:59:54.264884+0000","last_became_peered":"2024-09-21T09:59:54.264884+0000","last_unstale":"2024-09-21T10:02:13.087649+0000","last_undegraded":"2024-09-21T10:02:13.087649+0000","last_fullsized":"2024-09-21T10:02:13.087649+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:24.544845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":54,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.091931+0000","last_change":"2024-09-21T09:59:54.267798+0000","last_active":"2024-09-21T10:02:14.091931+0000","last_peered":"2024-09-21T10:02:14.091931+0000","last_clean":"2024-09-21T10:02:14.091931+0000","last_became_active":"2024-09-21T09:59:54.265615+0000","last_became_peered":"2024-09-21T09:59:54.265615+0000","last_unstale":"2024-09-21T10:02:14.091931+0000","last_undegraded":"2024-09-21T10:02:14.091931+0000","last_fullsized":"2024-09-21T10:02:14.091931+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:30.363872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":67,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.102034+0000","last_change":"2024-09-21T10:02:14.101943+0000","last_active":"2024-09-21T10:02:14.102034+0000","last_peered":"2024-09-21T10:02:14.102034+0000","last_clean":"2024-09-21T10:02:14.102034+0000","last_became_active":"2024-09-21T10:02:14.101488+0000","last_became_peered":"2024-09-21T10:02:14.101488+0000","last_unstale":"2024-09-21T10:02:14.102034+0000","last_undegraded":"2024-09-21T10:02:14.102034+0000","last_fullsized":"2024-09-21T10:02:14.102034+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:52:27.866414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":423,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.089474+0000","last_change":"2024-09-21T10:01:07.425330+0000","last_active":"2024-09-21T10:02:14.089474+0000","last_peered":"2024-09-21T10:02:14.089474+0000","last_clean":"2024-09-21T10:02:14.089474+0000","last_became_active":"2024-09-21T10:01:07.424695+0000","last_became_peered":"2024-09-21T10:01:07.424695+0000","last_unstale":"2024-09-21T10:02:14.089474+0000","last_undegraded":"2024-09-21T10:02:14.089474+0000","last_fullsized":"2024-09-21T10:02:14.089474+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:13:22.297576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":64,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.096024+0000","last_change":"2024-09-21T10:02:14.095924+0000","last_active":"2024-09-21T10:02:14.096024+0000","last_peered":"2024-09-21T10:02:14.096024+0000","last_clean":"2024-09-21T10:02:14.096024+0000","last_became_active":"2024-09-21T10:02:14.095553+0000","last_became_peered":"2024-09-21T10:02:14.095553+0000","last_unstale":"2024-09-21T10:02:14.096024+0000","last_undegraded":"2024-09-21T10:02:14.096024+0000","last_fullsized":"2024-09-21T10:02:14.096024+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:08.155980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.089383+0000","last_change":"2024-09-21T10:01:07.426736+0000","last_active":"2024-09-21T10:02:14.089383+0000","last_peered":"2024-09-21T10:02:14.089383+0000","last_clean":"2024-09-21T10:02:14.089383+0000","last_became_active":"2024-09-21T10:01:07.426416+0000","last_became_peered":"2024-09-21T10:01:07.426416+0000","last_unstale":"2024-09-21T10:02:14.089383+0000","last_undegraded":"2024-09-21T10:02:14.089383+0000","last_fullsized":"2024-09-21T10:02:14.089383+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:59.784757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":329,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091262+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:02:13.091262+0000","last_peered":"2024-09-21T10:02:13.091262+0000","last_clean":"2024-09-21T10:02:13.091262+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:02:13.091262+0000","last_undegraded":"2024-09-21T10:02:13.091262+0000","last_fullsized":"2024-09-21T10:02:13.091262+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086571+0000","last_change":"2024-09-21T09:52:20.391439+0000","last_active":"2024-09-21T10:02:13.086571+0000","last_peered":"2024-09-21T10:02:13.086571+0000","last_clean":"2024-09-21T10:02:13.086571+0000","last_became_active":"2024-09-21T09:52:20.391157+0000","last_became_peered":"2024-09-21T09:52:20.391157+0000","last_unstale":"2024-09-21T10:02:13.086571+0000","last_undegraded":"2024-09-21T10:02:13.086571+0000","last_fullsized":"2024-09-21T10:02:13.086571+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:36.788330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":66,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.088062+0000","last_change":"2024-09-21T10:01:07.421150+0000","last_active":"2024-09-21T10:02:13.088062+0000","last_peered":"2024-09-21T10:02:13.088062+0000","last_clean":"2024-09-21T10:02:13.088062+0000","last_became_active":"2024-09-21T10:01:07.420830+0000","last_became_peered":"2024-09-21T10:01:07.420830+0000","last_unstale":"2024-09-21T10:02:13.088062+0000","last_undegraded":"2024-09-21T10:02:13.088062+0000","last_fullsized":"2024-09-21T10:02:13.088062+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:15:34.472626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":72,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087220+0000","last_change":"2024-09-21T09:52:20.371668+0000","last_active":"2024-09-21T10:02:13.087220+0000","last_peered":"2024-09-21T10:02:13.087220+0000","last_clean":"2024-09-21T10:02:13.087220+0000","last_became_active":"2024-09-21T09:52:20.371389+0000","last_became_peered":"2024-09-21T09:52:20.371389+0000","last_unstale":"2024-09-21T10:02:13.087220+0000","last_undegraded":"2024-09-21T10:02:13.087220+0000","last_fullsized":"2024-09-21T10:02:13.087220+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:10.013020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":54,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090703+0000","last_change":"2024-09-21T10:01:41.360337+0000","last_active":"2024-09-21T10:02:14.090703+0000","last_peered":"2024-09-21T10:02:14.090703+0000","last_clean":"2024-09-21T10:02:14.090703+0000","last_became_active":"2024-09-21T10:01:41.359996+0000","last_became_peered":"2024-09-21T10:01:41.359996+0000","last_unstale":"2024-09-21T10:02:14.090703+0000","last_undegraded":"2024-09-21T10:02:14.090703+0000","last_fullsized":"2024-09-21T10:02:14.090703+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:28:43.820241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":70,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.102896+0000","last_change":"2024-09-21T10:02:14.102896+0000","last_active":"2024-09-21T10:02:14.102896+0000","last_peered":"2024-09-21T10:02:14.102896+0000","last_clean":"2024-09-21T10:02:14.102896+0000","last_became_active":"2024-09-21T10:02:14.102493+0000","last_became_peered":"2024-09-21T10:02:14.102493+0000","last_unstale":"2024-09-21T10:02:14.102896+0000","last_undegraded":"2024-09-21T10:02:14.102896+0000","last_fullsized":"2024-09-21T10:02:14.102896+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:10:43.572005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":67,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.089930+0000","last_change":"2024-09-21T10:01:07.424062+0000","last_active":"2024-09-21T10:02:13.089930+0000","last_peered":"2024-09-21T10:02:13.089930+0000","last_clean":"2024-09-21T10:02:13.089930+0000","last_became_active":"2024-09-21T10:01:07.423629+0000","last_became_peered":"2024-09-21T10:01:07.423629+0000","last_unstale":"2024-09-21T10:02:13.089930+0000","last_undegraded":"2024-09-21T10:02:13.089930+0000","last_fullsized":"2024-09-21T10:02:13.089930+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:24.119642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.100529+0000","last_change":"2024-09-21T10:02:14.100451+0000","last_active":"2024-09-21T10:02:14.100529+0000","last_peered":"2024-09-21T10:02:14.100529+0000","last_clean":"2024-09-21T10:02:14.100529+0000","last_became_active":"2024-09-21T10:02:14.100160+0000","last_became_peered":"2024-09-21T10:02:14.100160+0000","last_unstale":"2024-09-21T10:02:14.100529+0000","last_undegraded":"2024-09-21T10:02:14.100529+0000","last_fullsized":"2024-09-21T10:02:14.100529+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:28.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.096493+0000","last_change":"2024-09-21T10:02:14.096098+0000","last_active":"2024-09-21T10:02:14.096493+0000","last_peered":"2024-09-21T10:02:14.096493+0000","last_clean":"2024-09-21T10:02:14.096493+0000","last_became_active":"2024-09-21T10:02:14.095811+0000","last_became_peered":"2024-09-21T10:02:14.095811+0000","last_unstale":"2024-09-21T10:02:14.096493+0000","last_undegraded":"2024-09-21T10:02:14.096493+0000","last_fullsized":"2024-09-21T10:02:14.096493+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:54.619333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.100791+0000","last_change":"2024-09-21T10:02:14.100712+0000","last_active":"2024-09-21T10:02:14.100791+0000","last_peered":"2024-09-21T10:02:14.100791+0000","last_clean":"2024-09-21T10:02:14.100791+0000","last_became_active":"2024-09-21T10:02:14.100427+0000","last_became_peered":"2024-09-21T10:02:14.100427+0000","last_unstale":"2024-09-21T10:02:14.100791+0000","last_undegraded":"2024-09-21T10:02:14.100791+0000","last_fullsized":"2024-09-21T10:02:14.100791+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:16:42.350642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":66,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091136+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:02:13.091136+0000","last_peered":"2024-09-21T10:02:13.091136+0000","last_clean":"2024-09-21T10:02:13.091136+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:02:13.091136+0000","last_undegraded":"2024-09-21T10:02:13.091136+0000","last_fullsized":"2024-09-21T10:02:13.091136+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":87,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097446+0000","last_change":"2024-09-21T10:02:14.097369+0000","last_active":"2024-09-21T10:02:14.097446+0000","last_peered":"2024-09-21T10:02:14.097446+0000","last_clean":"2024-09-21T10:02:14.097446+0000","last_became_active":"2024-09-21T10:02:14.097067+0000","last_became_peered":"2024-09-21T10:02:14.097067+0000","last_unstale":"2024-09-21T10:02:14.097446+0000","last_undegraded":"2024-09-21T10:02:14.097446+0000","last_fullsized":"2024-09-21T10:02:14.097446+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:50:48.123813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099046+0000","last_change":"2024-09-21T10:02:14.098970+0000","last_active":"2024-09-21T10:02:14.099046+0000","last_peered":"2024-09-21T10:02:14.099046+0000","last_clean":"2024-09-21T10:02:14.099046+0000","last_became_active":"2024-09-21T10:02:14.098641+0000","last_became_peered":"2024-09-21T10:02:14.098641+0000","last_unstale":"2024-09-21T10:02:14.099046+0000","last_undegraded":"2024-09-21T10:02:14.099046+0000","last_fullsized":"2024-09-21T10:02:14.099046+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:25.215602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":54,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097100+0000","last_change":"2024-09-21T10:02:14.097021+0000","last_active":"2024-09-21T10:02:14.097100+0000","last_peered":"2024-09-21T10:02:14.097100+0000","last_clean":"2024-09-21T10:02:14.097100+0000","last_became_active":"2024-09-21T10:02:14.096741+0000","last_became_peered":"2024-09-21T10:02:14.096741+0000","last_unstale":"2024-09-21T10:02:14.097100+0000","last_undegraded":"2024-09-21T10:02:14.097100+0000","last_fullsized":"2024-09-21T10:02:14.097100+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:18:20.631150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099544+0000","last_change":"2024-09-21T10:02:14.099460+0000","last_active":"2024-09-21T10:02:14.099544+0000","last_peered":"2024-09-21T10:02:14.099544+0000","last_clean":"2024-09-21T10:02:14.099544+0000","last_became_active":"2024-09-21T10:02:14.099159+0000","last_became_peered":"2024-09-21T10:02:14.099159+0000","last_unstale":"2024-09-21T10:02:14.099544+0000","last_undegraded":"2024-09-21T10:02:14.099544+0000","last_fullsized":"2024-09-21T10:02:14.099544+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:23:47.599107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":85,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086660+0000","last_change":"2024-09-21T10:01:41.360480+0000","last_active":"2024-09-21T10:02:13.086660+0000","last_peered":"2024-09-21T10:02:13.086660+0000","last_clean":"2024-09-21T10:02:13.086660+0000","last_became_active":"2024-09-21T10:01:41.359791+0000","last_became_peered":"2024-09-21T10:01:41.359791+0000","last_unstale":"2024-09-21T10:02:13.086660+0000","last_undegraded":"2024-09-21T10:02:13.086660+0000","last_fullsized":"2024-09-21T10:02:13.086660+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:42.812974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.102901+0000","last_change":"2024-09-21T10:02:14.102826+0000","last_active":"2024-09-21T10:02:14.102901+0000","last_peered":"2024-09-21T10:02:14.102901+0000","last_clean":"2024-09-21T10:02:14.102901+0000","last_became_active":"2024-09-21T10:02:14.102499+0000","last_became_peered":"2024-09-21T10:02:14.102499+0000","last_unstale":"2024-09-21T10:02:14.102901+0000","last_undegraded":"2024-09-21T10:02:14.102901+0000","last_fullsized":"2024-09-21T10:02:14.102901+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:30.924687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":58,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.091881+0000","last_change":"2024-09-21T09:59:54.268378+0000","last_active":"2024-09-21T10:02:14.091881+0000","last_peered":"2024-09-21T10:02:14.091881+0000","last_clean":"2024-09-21T10:02:14.091881+0000","last_became_active":"2024-09-21T09:59:54.267187+0000","last_became_peered":"2024-09-21T09:59:54.267187+0000","last_unstale":"2024-09-21T10:02:14.091881+0000","last_undegraded":"2024-09-21T10:02:14.091881+0000","last_fullsized":"2024-09-21T10:02:14.091881+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:12:06.054483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":83,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.100541+0000","last_change":"2024-09-21T10:02:14.100541+0000","last_active":"2024-09-21T10:02:14.100541+0000","last_peered":"2024-09-21T10:02:14.100541+0000","last_clean":"2024-09-21T10:02:14.100541+0000","last_became_active":"2024-09-21T10:02:14.100327+0000","last_became_peered":"2024-09-21T10:02:14.100327+0000","last_unstale":"2024-09-21T10:02:14.100541+0000","last_undegraded":"2024-09-21T10:02:14.100541+0000","last_fullsized":"2024-09-21T10:02:14.100541+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:34:34.304188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":69,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.090359+0000","last_change":"2024-09-21T10:01:07.424248+0000","last_active":"2024-09-21T10:02:13.090359+0000","last_peered":"2024-09-21T10:02:13.090359+0000","last_clean":"2024-09-21T10:02:13.090359+0000","last_became_active":"2024-09-21T10:01:07.423738+0000","last_became_peered":"2024-09-21T10:01:07.423738+0000","last_unstale":"2024-09-21T10:02:13.090359+0000","last_undegraded":"2024-09-21T10:02:13.090359+0000","last_fullsized":"2024-09-21T10:02:13.090359+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:44:35.663555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":83,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.096583+0000","last_change":"2024-09-21T10:02:14.096583+0000","last_active":"2024-09-21T10:02:14.096583+0000","last_peered":"2024-09-21T10:02:14.096583+0000","last_clean":"2024-09-21T10:02:14.096583+0000","last_became_active":"2024-09-21T10:02:14.096196+0000","last_became_peered":"2024-09-21T10:02:14.096196+0000","last_unstale":"2024-09-21T10:02:14.096583+0000","last_undegraded":"2024-09-21T10:02:14.096583+0000","last_fullsized":"2024-09-21T10:02:14.096583+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:07:01.888435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091542+0000","last_change":"2024-09-21T09:52:20.371638+0000","last_active":"2024-09-21T10:02:13.091542+0000","last_peered":"2024-09-21T10:02:13.091542+0000","last_clean":"2024-09-21T10:02:13.091542+0000","last_became_active":"2024-09-21T09:52:20.371383+0000","last_became_peered":"2024-09-21T09:52:20.371383+0000","last_unstale":"2024-09-21T10:02:13.091542+0000","last_undegraded":"2024-09-21T10:02:13.091542+0000","last_fullsized":"2024-09-21T10:02:13.091542+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:33:45.384397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099880+0000","last_change":"2024-09-21T10:02:14.099799+0000","last_active":"2024-09-21T10:02:14.099880+0000","last_peered":"2024-09-21T10:02:14.099880+0000","last_clean":"2024-09-21T10:02:14.099880+0000","last_became_active":"2024-09-21T10:02:14.099375+0000","last_became_peered":"2024-09-21T10:02:14.099375+0000","last_unstale":"2024-09-21T10:02:14.099880+0000","last_undegraded":"2024-09-21T10:02:14.099880+0000","last_fullsized":"2024-09-21T10:02:14.099880+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:13.407638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":75,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099528+0000","last_change":"2024-09-21T10:02:14.099439+0000","last_active":"2024-09-21T10:02:14.099528+0000","last_peered":"2024-09-21T10:02:14.099528+0000","last_clean":"2024-09-21T10:02:14.099528+0000","last_became_active":"2024-09-21T10:02:14.099117+0000","last_became_peered":"2024-09-21T10:02:14.099117+0000","last_unstale":"2024-09-21T10:02:14.099528+0000","last_undegraded":"2024-09-21T10:02:14.099528+0000","last_fullsized":"2024-09-21T10:02:14.099528+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:39.681817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":76,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086376+0000","last_change":"2024-09-21T10:01:41.360674+0000","last_active":"2024-09-21T10:02:13.086376+0000","last_peered":"2024-09-21T10:02:13.086376+0000","last_clean":"2024-09-21T10:02:13.086376+0000","last_became_active":"2024-09-21T10:01:41.360258+0000","last_became_peered":"2024-09-21T10:01:41.360258+0000","last_unstale":"2024-09-21T10:02:13.086376+0000","last_undegraded":"2024-09-21T10:02:13.086376+0000","last_fullsized":"2024-09-21T10:02:13.086376+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:06.200022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.088916+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:02:14.088916+0000","last_peered":"2024-09-21T10:02:14.088916+0000","last_clean":"2024-09-21T10:02:14.088916+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:02:14.088916+0000","last_undegraded":"2024-09-21T10:02:14.088916+0000","last_fullsized":"2024-09-21T10:02:14.088916+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5983,"num_read_kb":13006,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397660,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307316,"kb_used_data":5792,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93437836,"statfs":{"total":95995035648,"available":95680344064,"internally_reserved":0,"allocated":5931008,"data_stored":5374871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593888,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299044,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446108,"statfs":{"total":95995035648,"available":95688814592,"internally_reserved":0,"allocated":1728512,"data_stored":1180440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790118,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302552,"kb_used_data":5192,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93442600,"statfs":{"total":95995035648,"available":95685222400,"internally_reserved":0,"allocated":5316608,"data_stored":4779887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986347,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298476,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446676,"statfs":{"total":95995035648,"available":95689396224,"internally_reserved":0,"allocated":1138688,"data_stored":590121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":101,"seq":433791696909,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28556,"kb_used_data":1092,"kb_used_omap":5,"kb_used_meta":27450,"kb_avail":93716596,"statfs":{"total":95995035648,"available":95965794304,"internally_reserved":0,"allocated":1118208,"data_stored":585341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":28109584},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860436,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32732,"kb_used_data":5192,"kb_used_omap":8,"kb_used_meta":27511,"kb_avail":93712420,"statfs":{"total":95995035648,"available":95961518080,"internally_reserved":0,"allocated":5316608,"data_stored":4780167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":28172140},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":91,"seq":390842023963,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28616,"kb_used_data":1084,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93716536,"statfs":{"total":95995035648,"available":95965732864,"internally_reserved":0,"allocated":1110016,"data_stored":584648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":28172850},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187497,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29212,"kb_used_data":1676,"kb_used_omap":8,"kb_used_meta":27511,"kb_avail":93715940,"statfs":{"total":95995035648,"available":95965122560,"internally_reserved":0,"allocated":1716224,"data_stored":1176219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":28172273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:03:12.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:03:12.545 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:12.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:12 smithi098 ceph-mon[97580]: pgmap v220: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:12.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:12 smithi098 ceph-mon[97580]: from='client.34944 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:03:12.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:12 smithi098 ceph-mon[100914]: pgmap v220: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:12.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:12 smithi098 ceph-mon[100914]: from='client.34944 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:03:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:12 smithi196 ceph-mon[84784]: pgmap v220: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:12 smithi196 ceph-mon[84784]: from='client.34944 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:03:14.316 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:03:14.317 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:03:14.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:14 smithi098 ceph-mon[97580]: pgmap v221: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:14.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:14 smithi098 ceph-mon[100914]: pgmap v221: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:14 smithi196 ceph-mon[84784]: pgmap v221: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:15.027 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-09-21T10:03:13.017969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6289,"num_read_kb":13765,"num_write":1056,"num_write_kb":14251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1132,"ondisk_log_size":1132,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1326504,"kb_used_data":22828,"kb_used_omap":35,"kb_used_meta":1303516,"kb_avail":748634712,"statfs":{"total":767960285184,"available":766601945088,"internally_reserved":0,"allocated":23375872,"data_stored":19051694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36353,"internal_metadata":1334800895},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002877"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.098066+0000","last_change":"2024-09-21T10:02:14.097991+0000","last_active":"2024-09-21T10:02:14.098066+0000","last_peered":"2024-09-21T10:02:14.098066+0000","last_clean":"2024-09-21T10:02:14.098066+0000","last_became_active":"2024-09-21T10:02:14.097407+0000","last_became_peered":"2024-09-21T10:02:14.097407+0000","last_unstale":"2024-09-21T10:02:14.098066+0000","last_undegraded":"2024-09-21T10:02:14.098066+0000","last_fullsized":"2024-09-21T10:02:14.098066+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:18:42.098886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":69,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.085145+0000","last_change":"2024-09-21T10:01:41.360784+0000","last_active":"2024-09-21T10:02:13.085145+0000","last_peered":"2024-09-21T10:02:13.085145+0000","last_clean":"2024-09-21T10:02:13.085145+0000","last_became_active":"2024-09-21T10:01:41.360462+0000","last_became_peered":"2024-09-21T10:01:41.360462+0000","last_unstale":"2024-09-21T10:02:13.085145+0000","last_undegraded":"2024-09-21T10:02:13.085145+0000","last_fullsized":"2024-09-21T10:02:13.085145+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:11.867275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":66,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.103373+0000","last_change":"2024-09-21T10:02:14.103373+0000","last_active":"2024-09-21T10:02:14.103373+0000","last_peered":"2024-09-21T10:02:14.103373+0000","last_clean":"2024-09-21T10:02:14.103373+0000","last_became_active":"2024-09-21T10:02:14.102985+0000","last_became_peered":"2024-09-21T10:02:14.102985+0000","last_unstale":"2024-09-21T10:02:14.103373+0000","last_undegraded":"2024-09-21T10:02:14.103373+0000","last_fullsized":"2024-09-21T10:02:14.103373+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:03.418174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099028+0000","last_change":"2024-09-21T10:02:14.098949+0000","last_active":"2024-09-21T10:02:14.099028+0000","last_peered":"2024-09-21T10:02:14.099028+0000","last_clean":"2024-09-21T10:02:14.099028+0000","last_became_active":"2024-09-21T10:02:14.098646+0000","last_became_peered":"2024-09-21T10:02:14.098646+0000","last_unstale":"2024-09-21T10:02:14.099028+0000","last_undegraded":"2024-09-21T10:02:14.099028+0000","last_fullsized":"2024-09-21T10:02:14.099028+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:50:56.862529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099148+0000","last_change":"2024-09-21T10:02:14.098205+0000","last_active":"2024-09-21T10:02:14.099148+0000","last_peered":"2024-09-21T10:02:14.099148+0000","last_clean":"2024-09-21T10:02:14.099148+0000","last_became_active":"2024-09-21T10:02:14.097911+0000","last_became_peered":"2024-09-21T10:02:14.097911+0000","last_unstale":"2024-09-21T10:02:14.099148+0000","last_undegraded":"2024-09-21T10:02:14.099148+0000","last_fullsized":"2024-09-21T10:02:14.099148+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:55.971191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.085481+0000","last_change":"2024-09-21T09:52:20.371775+0000","last_active":"2024-09-21T10:02:13.085481+0000","last_peered":"2024-09-21T10:02:13.085481+0000","last_clean":"2024-09-21T10:02:13.085481+0000","last_became_active":"2024-09-21T09:52:20.371509+0000","last_became_peered":"2024-09-21T09:52:20.371509+0000","last_unstale":"2024-09-21T10:02:13.085481+0000","last_undegraded":"2024-09-21T10:02:13.085481+0000","last_fullsized":"2024-09-21T10:02:13.085481+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:43:21.822524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097891+0000","last_change":"2024-09-21T10:02:14.097817+0000","last_active":"2024-09-21T10:02:14.097891+0000","last_peered":"2024-09-21T10:02:14.097891+0000","last_clean":"2024-09-21T10:02:14.097891+0000","last_became_active":"2024-09-21T10:02:14.097466+0000","last_became_peered":"2024-09-21T10:02:14.097466+0000","last_unstale":"2024-09-21T10:02:14.097891+0000","last_undegraded":"2024-09-21T10:02:14.097891+0000","last_fullsized":"2024-09-21T10:02:14.097891+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:57.883118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":75,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.090786+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:02:13.090786+0000","last_peered":"2024-09-21T10:02:13.090786+0000","last_clean":"2024-09-21T10:02:13.090786+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:02:13.090786+0000","last_undegraded":"2024-09-21T10:02:13.090786+0000","last_fullsized":"2024-09-21T10:02:13.090786+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":68,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086446+0000","last_change":"2024-09-21T09:59:54.265455+0000","last_active":"2024-09-21T10:02:13.086446+0000","last_peered":"2024-09-21T10:02:13.086446+0000","last_clean":"2024-09-21T10:02:13.086446+0000","last_became_active":"2024-09-21T09:59:54.265110+0000","last_became_peered":"2024-09-21T09:59:54.265110+0000","last_unstale":"2024-09-21T10:02:13.086446+0000","last_undegraded":"2024-09-21T10:02:13.086446+0000","last_fullsized":"2024-09-21T10:02:13.086446+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:04:13.404874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.089734+0000","last_change":"2024-09-21T09:52:20.402753+0000","last_active":"2024-09-21T10:02:13.089734+0000","last_peered":"2024-09-21T10:02:13.089734+0000","last_clean":"2024-09-21T10:02:13.089734+0000","last_became_active":"2024-09-21T09:52:20.402462+0000","last_became_peered":"2024-09-21T09:52:20.402462+0000","last_unstale":"2024-09-21T10:02:13.089734+0000","last_undegraded":"2024-09-21T10:02:13.089734+0000","last_fullsized":"2024-09-21T10:02:13.089734+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:14.036869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":69,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086567+0000","last_change":"2024-09-21T10:01:07.420484+0000","last_active":"2024-09-21T10:02:13.086567+0000","last_peered":"2024-09-21T10:02:13.086567+0000","last_clean":"2024-09-21T10:02:13.086567+0000","last_became_active":"2024-09-21T10:01:07.420135+0000","last_became_peered":"2024-09-21T10:01:07.420135+0000","last_unstale":"2024-09-21T10:02:13.086567+0000","last_undegraded":"2024-09-21T10:02:13.086567+0000","last_fullsized":"2024-09-21T10:02:13.086567+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:33:10.241010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":83,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.098139+0000","last_change":"2024-09-21T10:02:14.098139+0000","last_active":"2024-09-21T10:02:14.098139+0000","last_peered":"2024-09-21T10:02:14.098139+0000","last_clean":"2024-09-21T10:02:14.098139+0000","last_became_active":"2024-09-21T10:02:14.097639+0000","last_became_peered":"2024-09-21T10:02:14.097639+0000","last_unstale":"2024-09-21T10:02:14.098139+0000","last_undegraded":"2024-09-21T10:02:14.098139+0000","last_fullsized":"2024-09-21T10:02:14.098139+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:11:26.925388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087557+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:02:13.087557+0000","last_peered":"2024-09-21T10:02:13.087557+0000","last_clean":"2024-09-21T10:02:13.087557+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:02:13.087557+0000","last_undegraded":"2024-09-21T10:02:13.087557+0000","last_fullsized":"2024-09-21T10:02:13.087557+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":66,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091038+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:02:13.091038+0000","last_peered":"2024-09-21T10:02:13.091038+0000","last_clean":"2024-09-21T10:02:13.091038+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:02:13.091038+0000","last_undegraded":"2024-09-21T10:02:13.091038+0000","last_fullsized":"2024-09-21T10:02:13.091038+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":67,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.104212+0000","last_change":"2024-09-21T10:02:14.104212+0000","last_active":"2024-09-21T10:02:14.104212+0000","last_peered":"2024-09-21T10:02:14.104212+0000","last_clean":"2024-09-21T10:02:14.104212+0000","last_became_active":"2024-09-21T10:02:14.103893+0000","last_became_peered":"2024-09-21T10:02:14.103893+0000","last_unstale":"2024-09-21T10:02:14.104212+0000","last_undegraded":"2024-09-21T10:02:14.104212+0000","last_fullsized":"2024-09-21T10:02:14.104212+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:59:21.191412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.091970+0000","last_change":"2024-09-21T10:01:07.419715+0000","last_active":"2024-09-21T10:02:14.091970+0000","last_peered":"2024-09-21T10:02:14.091970+0000","last_clean":"2024-09-21T10:02:14.091970+0000","last_became_active":"2024-09-21T10:01:07.419422+0000","last_became_peered":"2024-09-21T10:01:07.419422+0000","last_unstale":"2024-09-21T10:02:14.091970+0000","last_undegraded":"2024-09-21T10:02:14.091970+0000","last_fullsized":"2024-09-21T10:02:14.091970+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:36.424289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":67,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.096638+0000","last_change":"2024-09-21T10:02:14.096537+0000","last_active":"2024-09-21T10:02:14.096638+0000","last_peered":"2024-09-21T10:02:14.096638+0000","last_clean":"2024-09-21T10:02:14.096638+0000","last_became_active":"2024-09-21T10:02:14.096166+0000","last_became_peered":"2024-09-21T10:02:14.096166+0000","last_unstale":"2024-09-21T10:02:14.096638+0000","last_undegraded":"2024-09-21T10:02:14.096638+0000","last_fullsized":"2024-09-21T10:02:14.096638+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:54:18.927732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":66,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099205+0000","last_change":"2024-09-21T10:02:14.099205+0000","last_active":"2024-09-21T10:02:14.099205+0000","last_peered":"2024-09-21T10:02:14.099205+0000","last_clean":"2024-09-21T10:02:14.099205+0000","last_became_active":"2024-09-21T10:02:14.098598+0000","last_became_peered":"2024-09-21T10:02:14.098598+0000","last_unstale":"2024-09-21T10:02:14.099205+0000","last_undegraded":"2024-09-21T10:02:14.099205+0000","last_fullsized":"2024-09-21T10:02:14.099205+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:46.424612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":67,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091165+0000","last_change":"2024-09-21T10:01:07.424883+0000","last_active":"2024-09-21T10:02:13.091165+0000","last_peered":"2024-09-21T10:02:13.091165+0000","last_clean":"2024-09-21T10:02:13.091165+0000","last_became_active":"2024-09-21T10:01:07.424499+0000","last_became_peered":"2024-09-21T10:01:07.424499+0000","last_unstale":"2024-09-21T10:02:13.091165+0000","last_undegraded":"2024-09-21T10:02:13.091165+0000","last_fullsized":"2024-09-21T10:02:13.091165+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:46.101570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":83,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087300+0000","last_change":"2024-09-21T10:01:41.362291+0000","last_active":"2024-09-21T10:02:13.087300+0000","last_peered":"2024-09-21T10:02:13.087300+0000","last_clean":"2024-09-21T10:02:13.087300+0000","last_became_active":"2024-09-21T10:01:41.361954+0000","last_became_peered":"2024-09-21T10:01:41.361954+0000","last_unstale":"2024-09-21T10:02:13.087300+0000","last_undegraded":"2024-09-21T10:02:13.087300+0000","last_fullsized":"2024-09-21T10:02:13.087300+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:57.669671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":67,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087361+0000","last_change":"2024-09-21T10:01:41.361244+0000","last_active":"2024-09-21T10:02:13.087361+0000","last_peered":"2024-09-21T10:02:13.087361+0000","last_clean":"2024-09-21T10:02:13.087361+0000","last_became_active":"2024-09-21T10:01:41.360645+0000","last_became_peered":"2024-09-21T10:01:41.360645+0000","last_unstale":"2024-09-21T10:02:13.087361+0000","last_undegraded":"2024-09-21T10:02:13.087361+0000","last_fullsized":"2024-09-21T10:02:13.087361+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:07.604991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":66,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086023+0000","last_change":"2024-09-21T10:01:07.420824+0000","last_active":"2024-09-21T10:02:13.086023+0000","last_peered":"2024-09-21T10:02:13.086023+0000","last_clean":"2024-09-21T10:02:13.086023+0000","last_became_active":"2024-09-21T10:01:07.420387+0000","last_became_peered":"2024-09-21T10:01:07.420387+0000","last_unstale":"2024-09-21T10:02:13.086023+0000","last_undegraded":"2024-09-21T10:02:13.086023+0000","last_fullsized":"2024-09-21T10:02:13.086023+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:48.875842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":70,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097606+0000","last_change":"2024-09-21T10:02:14.097462+0000","last_active":"2024-09-21T10:02:14.097606+0000","last_peered":"2024-09-21T10:02:14.097606+0000","last_clean":"2024-09-21T10:02:14.097606+0000","last_became_active":"2024-09-21T10:02:14.097167+0000","last_became_peered":"2024-09-21T10:02:14.097167+0000","last_unstale":"2024-09-21T10:02:14.097606+0000","last_undegraded":"2024-09-21T10:02:14.097606+0000","last_fullsized":"2024-09-21T10:02:14.097606+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:54.922731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":54,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097753+0000","last_change":"2024-09-21T10:02:14.097676+0000","last_active":"2024-09-21T10:02:14.097753+0000","last_peered":"2024-09-21T10:02:14.097753+0000","last_clean":"2024-09-21T10:02:14.097753+0000","last_became_active":"2024-09-21T10:02:14.097391+0000","last_became_peered":"2024-09-21T10:02:14.097391+0000","last_unstale":"2024-09-21T10:02:14.097753+0000","last_undegraded":"2024-09-21T10:02:14.097753+0000","last_fullsized":"2024-09-21T10:02:14.097753+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:40:24.523178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":184,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087727+0000","last_change":"2024-09-21T09:59:54.265556+0000","last_active":"2024-09-21T10:02:13.087727+0000","last_peered":"2024-09-21T10:02:13.087727+0000","last_clean":"2024-09-21T10:02:13.087727+0000","last_became_active":"2024-09-21T09:59:54.265033+0000","last_became_peered":"2024-09-21T09:59:54.265033+0000","last_unstale":"2024-09-21T10:02:13.087727+0000","last_undegraded":"2024-09-21T10:02:13.087727+0000","last_fullsized":"2024-09-21T10:02:13.087727+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T09:48:34.850872+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:40.449756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":491,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.089982+0000","last_change":"2024-09-21T10:01:41.361089+0000","last_active":"2024-09-21T10:02:13.089982+0000","last_peered":"2024-09-21T10:02:13.089982+0000","last_clean":"2024-09-21T10:02:13.089982+0000","last_became_active":"2024-09-21T10:01:41.360658+0000","last_became_peered":"2024-09-21T10:01:41.360658+0000","last_unstale":"2024-09-21T10:02:13.089982+0000","last_undegraded":"2024-09-21T10:02:13.089982+0000","last_fullsized":"2024-09-21T10:02:13.089982+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:58:06.840024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":482,"num_read_kb":1581,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.100041+0000","last_change":"2024-09-21T10:02:14.099961+0000","last_active":"2024-09-21T10:02:14.100041+0000","last_peered":"2024-09-21T10:02:14.100041+0000","last_clean":"2024-09-21T10:02:14.100041+0000","last_became_active":"2024-09-21T10:02:14.099067+0000","last_became_peered":"2024-09-21T10:02:14.099067+0000","last_unstale":"2024-09-21T10:02:14.100041+0000","last_undegraded":"2024-09-21T10:02:14.100041+0000","last_fullsized":"2024-09-21T10:02:14.100041+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:41.037532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091668+0000","last_change":"2024-09-21T09:52:20.403534+0000","last_active":"2024-09-21T10:02:13.091668+0000","last_peered":"2024-09-21T10:02:13.091668+0000","last_clean":"2024-09-21T10:02:13.091668+0000","last_became_active":"2024-09-21T09:52:20.403258+0000","last_became_peered":"2024-09-21T09:52:20.403258+0000","last_unstale":"2024-09-21T10:02:13.091668+0000","last_undegraded":"2024-09-21T10:02:13.091668+0000","last_fullsized":"2024-09-21T10:02:13.091668+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:55.374343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":447,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087637+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:02:13.087637+0000","last_peered":"2024-09-21T10:02:13.087637+0000","last_clean":"2024-09-21T10:02:13.087637+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:02:13.087637+0000","last_undegraded":"2024-09-21T10:02:13.087637+0000","last_fullsized":"2024-09-21T10:02:13.087637+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097180+0000","last_change":"2024-09-21T10:02:14.097097+0000","last_active":"2024-09-21T10:02:14.097180+0000","last_peered":"2024-09-21T10:02:14.097180+0000","last_clean":"2024-09-21T10:02:14.097180+0000","last_became_active":"2024-09-21T10:02:14.096788+0000","last_became_peered":"2024-09-21T10:02:14.096788+0000","last_unstale":"2024-09-21T10:02:14.097180+0000","last_undegraded":"2024-09-21T10:02:14.097180+0000","last_fullsized":"2024-09-21T10:02:14.097180+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:05:29.711020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":76,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090168+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:02:14.090168+0000","last_peered":"2024-09-21T10:02:14.090168+0000","last_clean":"2024-09-21T10:02:14.090168+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:02:14.090168+0000","last_undegraded":"2024-09-21T10:02:14.090168+0000","last_fullsized":"2024-09-21T10:02:14.090168+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":815,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.100244+0000","last_change":"2024-09-21T10:02:14.100143+0000","last_active":"2024-09-21T10:02:14.100244+0000","last_peered":"2024-09-21T10:02:14.100244+0000","last_clean":"2024-09-21T10:02:14.100244+0000","last_became_active":"2024-09-21T10:02:14.099586+0000","last_became_peered":"2024-09-21T10:02:14.099586+0000","last_unstale":"2024-09-21T10:02:14.100244+0000","last_undegraded":"2024-09-21T10:02:14.100244+0000","last_fullsized":"2024-09-21T10:02:14.100244+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:34.064652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":55,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090600+0000","last_change":"2024-09-21T10:01:41.358268+0000","last_active":"2024-09-21T10:02:14.090600+0000","last_peered":"2024-09-21T10:02:14.090600+0000","last_clean":"2024-09-21T10:02:14.090600+0000","last_became_active":"2024-09-21T10:01:41.357816+0000","last_became_peered":"2024-09-21T10:01:41.357816+0000","last_unstale":"2024-09-21T10:02:14.090600+0000","last_undegraded":"2024-09-21T10:02:14.090600+0000","last_fullsized":"2024-09-21T10:02:14.090600+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:59:44.261636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":72,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.096377+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:02:14.096377+0000","last_peered":"2024-09-21T10:02:14.096377+0000","last_clean":"2024-09-21T10:02:14.096377+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:02:14.096377+0000","last_undegraded":"2024-09-21T10:02:14.096377+0000","last_fullsized":"2024-09-21T10:02:14.096377+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":675,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.090133+0000","last_change":"2024-09-21T10:01:07.424819+0000","last_active":"2024-09-21T10:02:13.090133+0000","last_peered":"2024-09-21T10:02:13.090133+0000","last_clean":"2024-09-21T10:02:13.090133+0000","last_became_active":"2024-09-21T10:01:07.424392+0000","last_became_peered":"2024-09-21T10:01:07.424392+0000","last_unstale":"2024-09-21T10:02:13.090133+0000","last_undegraded":"2024-09-21T10:02:13.090133+0000","last_fullsized":"2024-09-21T10:02:13.090133+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:27.266390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":67,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.088170+0000","last_change":"2024-09-21T09:59:54.267742+0000","last_active":"2024-09-21T10:02:13.088170+0000","last_peered":"2024-09-21T10:02:13.088170+0000","last_clean":"2024-09-21T10:02:13.088170+0000","last_became_active":"2024-09-21T09:59:54.267406+0000","last_became_peered":"2024-09-21T09:59:54.267406+0000","last_unstale":"2024-09-21T10:02:13.088170+0000","last_undegraded":"2024-09-21T10:02:13.088170+0000","last_fullsized":"2024-09-21T10:02:13.088170+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:25:35.968779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":80,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.103531+0000","last_change":"2024-09-21T10:02:14.103531+0000","last_active":"2024-09-21T10:02:14.103531+0000","last_peered":"2024-09-21T10:02:14.103531+0000","last_clean":"2024-09-21T10:02:14.103531+0000","last_became_active":"2024-09-21T10:02:14.103218+0000","last_became_peered":"2024-09-21T10:02:14.103218+0000","last_unstale":"2024-09-21T10:02:14.103531+0000","last_undegraded":"2024-09-21T10:02:14.103531+0000","last_fullsized":"2024-09-21T10:02:14.103531+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:40.014308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":84,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.103145+0000","last_change":"2024-09-21T10:02:14.103145+0000","last_active":"2024-09-21T10:02:14.103145+0000","last_peered":"2024-09-21T10:02:14.103145+0000","last_clean":"2024-09-21T10:02:14.103145+0000","last_became_active":"2024-09-21T10:02:14.102806+0000","last_became_peered":"2024-09-21T10:02:14.102806+0000","last_unstale":"2024-09-21T10:02:14.103145+0000","last_undegraded":"2024-09-21T10:02:14.103145+0000","last_fullsized":"2024-09-21T10:02:14.103145+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:24.348745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":760,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090783+0000","last_change":"2024-09-21T10:01:07.426065+0000","last_active":"2024-09-21T10:02:14.090783+0000","last_peered":"2024-09-21T10:02:14.090783+0000","last_clean":"2024-09-21T10:02:14.090783+0000","last_became_active":"2024-09-21T10:01:07.423163+0000","last_became_peered":"2024-09-21T10:01:07.423163+0000","last_unstale":"2024-09-21T10:02:14.090783+0000","last_undegraded":"2024-09-21T10:02:14.090783+0000","last_fullsized":"2024-09-21T10:02:14.090783+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:43.564978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":44,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090907+0000","last_change":"2024-09-21T10:01:07.422175+0000","last_active":"2024-09-21T10:02:14.090907+0000","last_peered":"2024-09-21T10:02:14.090907+0000","last_clean":"2024-09-21T10:02:14.090907+0000","last_became_active":"2024-09-21T10:01:07.421844+0000","last_became_peered":"2024-09-21T10:01:07.421844+0000","last_unstale":"2024-09-21T10:02:14.090907+0000","last_undegraded":"2024-09-21T10:02:14.090907+0000","last_fullsized":"2024-09-21T10:02:14.090907+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:17:15.192573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":55,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090704+0000","last_change":"2024-09-21T10:01:41.360338+0000","last_active":"2024-09-21T10:02:14.090704+0000","last_peered":"2024-09-21T10:02:14.090704+0000","last_clean":"2024-09-21T10:02:14.090704+0000","last_became_active":"2024-09-21T10:01:41.360050+0000","last_became_peered":"2024-09-21T10:01:41.360050+0000","last_unstale":"2024-09-21T10:02:14.090704+0000","last_undegraded":"2024-09-21T10:02:14.090704+0000","last_fullsized":"2024-09-21T10:02:14.090704+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:39:30.459393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":413,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087649+0000","last_change":"2024-09-21T09:59:54.265559+0000","last_active":"2024-09-21T10:02:13.087649+0000","last_peered":"2024-09-21T10:02:13.087649+0000","last_clean":"2024-09-21T10:02:13.087649+0000","last_became_active":"2024-09-21T09:59:54.264884+0000","last_became_peered":"2024-09-21T09:59:54.264884+0000","last_unstale":"2024-09-21T10:02:13.087649+0000","last_undegraded":"2024-09-21T10:02:13.087649+0000","last_fullsized":"2024-09-21T10:02:13.087649+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:24.544845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":54,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.091931+0000","last_change":"2024-09-21T09:59:54.267798+0000","last_active":"2024-09-21T10:02:14.091931+0000","last_peered":"2024-09-21T10:02:14.091931+0000","last_clean":"2024-09-21T10:02:14.091931+0000","last_became_active":"2024-09-21T09:59:54.265615+0000","last_became_peered":"2024-09-21T09:59:54.265615+0000","last_unstale":"2024-09-21T10:02:14.091931+0000","last_undegraded":"2024-09-21T10:02:14.091931+0000","last_fullsized":"2024-09-21T10:02:14.091931+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:30.363872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":67,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.102034+0000","last_change":"2024-09-21T10:02:14.101943+0000","last_active":"2024-09-21T10:02:14.102034+0000","last_peered":"2024-09-21T10:02:14.102034+0000","last_clean":"2024-09-21T10:02:14.102034+0000","last_became_active":"2024-09-21T10:02:14.101488+0000","last_became_peered":"2024-09-21T10:02:14.101488+0000","last_unstale":"2024-09-21T10:02:14.102034+0000","last_undegraded":"2024-09-21T10:02:14.102034+0000","last_fullsized":"2024-09-21T10:02:14.102034+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:52:27.866414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":423,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.089474+0000","last_change":"2024-09-21T10:01:07.425330+0000","last_active":"2024-09-21T10:02:14.089474+0000","last_peered":"2024-09-21T10:02:14.089474+0000","last_clean":"2024-09-21T10:02:14.089474+0000","last_became_active":"2024-09-21T10:01:07.424695+0000","last_became_peered":"2024-09-21T10:01:07.424695+0000","last_unstale":"2024-09-21T10:02:14.089474+0000","last_undegraded":"2024-09-21T10:02:14.089474+0000","last_fullsized":"2024-09-21T10:02:14.089474+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:13:22.297576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":64,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.096024+0000","last_change":"2024-09-21T10:02:14.095924+0000","last_active":"2024-09-21T10:02:14.096024+0000","last_peered":"2024-09-21T10:02:14.096024+0000","last_clean":"2024-09-21T10:02:14.096024+0000","last_became_active":"2024-09-21T10:02:14.095553+0000","last_became_peered":"2024-09-21T10:02:14.095553+0000","last_unstale":"2024-09-21T10:02:14.096024+0000","last_undegraded":"2024-09-21T10:02:14.096024+0000","last_fullsized":"2024-09-21T10:02:14.096024+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:08.155980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.089383+0000","last_change":"2024-09-21T10:01:07.426736+0000","last_active":"2024-09-21T10:02:14.089383+0000","last_peered":"2024-09-21T10:02:14.089383+0000","last_clean":"2024-09-21T10:02:14.089383+0000","last_became_active":"2024-09-21T10:01:07.426416+0000","last_became_peered":"2024-09-21T10:01:07.426416+0000","last_unstale":"2024-09-21T10:02:14.089383+0000","last_undegraded":"2024-09-21T10:02:14.089383+0000","last_fullsized":"2024-09-21T10:02:14.089383+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:59.784757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":329,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091262+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:02:13.091262+0000","last_peered":"2024-09-21T10:02:13.091262+0000","last_clean":"2024-09-21T10:02:13.091262+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:02:13.091262+0000","last_undegraded":"2024-09-21T10:02:13.091262+0000","last_fullsized":"2024-09-21T10:02:13.091262+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086571+0000","last_change":"2024-09-21T09:52:20.391439+0000","last_active":"2024-09-21T10:02:13.086571+0000","last_peered":"2024-09-21T10:02:13.086571+0000","last_clean":"2024-09-21T10:02:13.086571+0000","last_became_active":"2024-09-21T09:52:20.391157+0000","last_became_peered":"2024-09-21T09:52:20.391157+0000","last_unstale":"2024-09-21T10:02:13.086571+0000","last_undegraded":"2024-09-21T10:02:13.086571+0000","last_fullsized":"2024-09-21T10:02:13.086571+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:36.788330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":66,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.088062+0000","last_change":"2024-09-21T10:01:07.421150+0000","last_active":"2024-09-21T10:02:13.088062+0000","last_peered":"2024-09-21T10:02:13.088062+0000","last_clean":"2024-09-21T10:02:13.088062+0000","last_became_active":"2024-09-21T10:01:07.420830+0000","last_became_peered":"2024-09-21T10:01:07.420830+0000","last_unstale":"2024-09-21T10:02:13.088062+0000","last_undegraded":"2024-09-21T10:02:13.088062+0000","last_fullsized":"2024-09-21T10:02:13.088062+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:15:34.472626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":72,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087220+0000","last_change":"2024-09-21T09:52:20.371668+0000","last_active":"2024-09-21T10:02:13.087220+0000","last_peered":"2024-09-21T10:02:13.087220+0000","last_clean":"2024-09-21T10:02:13.087220+0000","last_became_active":"2024-09-21T09:52:20.371389+0000","last_became_peered":"2024-09-21T09:52:20.371389+0000","last_unstale":"2024-09-21T10:02:13.087220+0000","last_undegraded":"2024-09-21T10:02:13.087220+0000","last_fullsized":"2024-09-21T10:02:13.087220+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:10.013020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":54,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090703+0000","last_change":"2024-09-21T10:01:41.360337+0000","last_active":"2024-09-21T10:02:14.090703+0000","last_peered":"2024-09-21T10:02:14.090703+0000","last_clean":"2024-09-21T10:02:14.090703+0000","last_became_active":"2024-09-21T10:01:41.359996+0000","last_became_peered":"2024-09-21T10:01:41.359996+0000","last_unstale":"2024-09-21T10:02:14.090703+0000","last_undegraded":"2024-09-21T10:02:14.090703+0000","last_fullsized":"2024-09-21T10:02:14.090703+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:28:43.820241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":70,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.102896+0000","last_change":"2024-09-21T10:02:14.102896+0000","last_active":"2024-09-21T10:02:14.102896+0000","last_peered":"2024-09-21T10:02:14.102896+0000","last_clean":"2024-09-21T10:02:14.102896+0000","last_became_active":"2024-09-21T10:02:14.102493+0000","last_became_peered":"2024-09-21T10:02:14.102493+0000","last_unstale":"2024-09-21T10:02:14.102896+0000","last_undegraded":"2024-09-21T10:02:14.102896+0000","last_fullsized":"2024-09-21T10:02:14.102896+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:10:43.572005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":67,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.089930+0000","last_change":"2024-09-21T10:01:07.424062+0000","last_active":"2024-09-21T10:02:13.089930+0000","last_peered":"2024-09-21T10:02:13.089930+0000","last_clean":"2024-09-21T10:02:13.089930+0000","last_became_active":"2024-09-21T10:01:07.423629+0000","last_became_peered":"2024-09-21T10:01:07.423629+0000","last_unstale":"2024-09-21T10:02:13.089930+0000","last_undegraded":"2024-09-21T10:02:13.089930+0000","last_fullsized":"2024-09-21T10:02:13.089930+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:24.119642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.100529+0000","last_change":"2024-09-21T10:02:14.100451+0000","last_active":"2024-09-21T10:02:14.100529+0000","last_peered":"2024-09-21T10:02:14.100529+0000","last_clean":"2024-09-21T10:02:14.100529+0000","last_became_active":"2024-09-21T10:02:14.100160+0000","last_became_peered":"2024-09-21T10:02:14.100160+0000","last_unstale":"2024-09-21T10:02:14.100529+0000","last_undegraded":"2024-09-21T10:02:14.100529+0000","last_fullsized":"2024-09-21T10:02:14.100529+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:28.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.096493+0000","last_change":"2024-09-21T10:02:14.096098+0000","last_active":"2024-09-21T10:02:14.096493+0000","last_peered":"2024-09-21T10:02:14.096493+0000","last_clean":"2024-09-21T10:02:14.096493+0000","last_became_active":"2024-09-21T10:02:14.095811+0000","last_became_peered":"2024-09-21T10:02:14.095811+0000","last_unstale":"2024-09-21T10:02:14.096493+0000","last_undegraded":"2024-09-21T10:02:14.096493+0000","last_fullsized":"2024-09-21T10:02:14.096493+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:54.619333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.100791+0000","last_change":"2024-09-21T10:02:14.100712+0000","last_active":"2024-09-21T10:02:14.100791+0000","last_peered":"2024-09-21T10:02:14.100791+0000","last_clean":"2024-09-21T10:02:14.100791+0000","last_became_active":"2024-09-21T10:02:14.100427+0000","last_became_peered":"2024-09-21T10:02:14.100427+0000","last_unstale":"2024-09-21T10:02:14.100791+0000","last_undegraded":"2024-09-21T10:02:14.100791+0000","last_fullsized":"2024-09-21T10:02:14.100791+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:16:42.350642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":66,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091136+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:02:13.091136+0000","last_peered":"2024-09-21T10:02:13.091136+0000","last_clean":"2024-09-21T10:02:13.091136+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:02:13.091136+0000","last_undegraded":"2024-09-21T10:02:13.091136+0000","last_fullsized":"2024-09-21T10:02:13.091136+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":87,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097446+0000","last_change":"2024-09-21T10:02:14.097369+0000","last_active":"2024-09-21T10:02:14.097446+0000","last_peered":"2024-09-21T10:02:14.097446+0000","last_clean":"2024-09-21T10:02:14.097446+0000","last_became_active":"2024-09-21T10:02:14.097067+0000","last_became_peered":"2024-09-21T10:02:14.097067+0000","last_unstale":"2024-09-21T10:02:14.097446+0000","last_undegraded":"2024-09-21T10:02:14.097446+0000","last_fullsized":"2024-09-21T10:02:14.097446+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:50:48.123813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099046+0000","last_change":"2024-09-21T10:02:14.098970+0000","last_active":"2024-09-21T10:02:14.099046+0000","last_peered":"2024-09-21T10:02:14.099046+0000","last_clean":"2024-09-21T10:02:14.099046+0000","last_became_active":"2024-09-21T10:02:14.098641+0000","last_became_peered":"2024-09-21T10:02:14.098641+0000","last_unstale":"2024-09-21T10:02:14.099046+0000","last_undegraded":"2024-09-21T10:02:14.099046+0000","last_fullsized":"2024-09-21T10:02:14.099046+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:25.215602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":54,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097100+0000","last_change":"2024-09-21T10:02:14.097021+0000","last_active":"2024-09-21T10:02:14.097100+0000","last_peered":"2024-09-21T10:02:14.097100+0000","last_clean":"2024-09-21T10:02:14.097100+0000","last_became_active":"2024-09-21T10:02:14.096741+0000","last_became_peered":"2024-09-21T10:02:14.096741+0000","last_unstale":"2024-09-21T10:02:14.097100+0000","last_undegraded":"2024-09-21T10:02:14.097100+0000","last_fullsized":"2024-09-21T10:02:14.097100+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:18:20.631150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099544+0000","last_change":"2024-09-21T10:02:14.099460+0000","last_active":"2024-09-21T10:02:14.099544+0000","last_peered":"2024-09-21T10:02:14.099544+0000","last_clean":"2024-09-21T10:02:14.099544+0000","last_became_active":"2024-09-21T10:02:14.099159+0000","last_became_peered":"2024-09-21T10:02:14.099159+0000","last_unstale":"2024-09-21T10:02:14.099544+0000","last_undegraded":"2024-09-21T10:02:14.099544+0000","last_fullsized":"2024-09-21T10:02:14.099544+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:23:47.599107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":85,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086660+0000","last_change":"2024-09-21T10:01:41.360480+0000","last_active":"2024-09-21T10:02:13.086660+0000","last_peered":"2024-09-21T10:02:13.086660+0000","last_clean":"2024-09-21T10:02:13.086660+0000","last_became_active":"2024-09-21T10:01:41.359791+0000","last_became_peered":"2024-09-21T10:01:41.359791+0000","last_unstale":"2024-09-21T10:02:13.086660+0000","last_undegraded":"2024-09-21T10:02:13.086660+0000","last_fullsized":"2024-09-21T10:02:13.086660+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:42.812974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.102901+0000","last_change":"2024-09-21T10:02:14.102826+0000","last_active":"2024-09-21T10:02:14.102901+0000","last_peered":"2024-09-21T10:02:14.102901+0000","last_clean":"2024-09-21T10:02:14.102901+0000","last_became_active":"2024-09-21T10:02:14.102499+0000","last_became_peered":"2024-09-21T10:02:14.102499+0000","last_unstale":"2024-09-21T10:02:14.102901+0000","last_undegraded":"2024-09-21T10:02:14.102901+0000","last_fullsized":"2024-09-21T10:02:14.102901+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:30.924687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":58,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.091881+0000","last_change":"2024-09-21T09:59:54.268378+0000","last_active":"2024-09-21T10:02:14.091881+0000","last_peered":"2024-09-21T10:02:14.091881+0000","last_clean":"2024-09-21T10:02:14.091881+0000","last_became_active":"2024-09-21T09:59:54.267187+0000","last_became_peered":"2024-09-21T09:59:54.267187+0000","last_unstale":"2024-09-21T10:02:14.091881+0000","last_undegraded":"2024-09-21T10:02:14.091881+0000","last_fullsized":"2024-09-21T10:02:14.091881+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:12:06.054483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":83,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.100541+0000","last_change":"2024-09-21T10:02:14.100541+0000","last_active":"2024-09-21T10:02:14.100541+0000","last_peered":"2024-09-21T10:02:14.100541+0000","last_clean":"2024-09-21T10:02:14.100541+0000","last_became_active":"2024-09-21T10:02:14.100327+0000","last_became_peered":"2024-09-21T10:02:14.100327+0000","last_unstale":"2024-09-21T10:02:14.100541+0000","last_undegraded":"2024-09-21T10:02:14.100541+0000","last_fullsized":"2024-09-21T10:02:14.100541+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:34:34.304188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":69,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.090359+0000","last_change":"2024-09-21T10:01:07.424248+0000","last_active":"2024-09-21T10:02:13.090359+0000","last_peered":"2024-09-21T10:02:13.090359+0000","last_clean":"2024-09-21T10:02:13.090359+0000","last_became_active":"2024-09-21T10:01:07.423738+0000","last_became_peered":"2024-09-21T10:01:07.423738+0000","last_unstale":"2024-09-21T10:02:13.090359+0000","last_undegraded":"2024-09-21T10:02:13.090359+0000","last_fullsized":"2024-09-21T10:02:13.090359+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:44:35.663555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":83,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.096583+0000","last_change":"2024-09-21T10:02:14.096583+0000","last_active":"2024-09-21T10:02:14.096583+0000","last_peered":"2024-09-21T10:02:14.096583+0000","last_clean":"2024-09-21T10:02:14.096583+0000","last_became_active":"2024-09-21T10:02:14.096196+0000","last_became_peered":"2024-09-21T10:02:14.096196+0000","last_unstale":"2024-09-21T10:02:14.096583+0000","last_undegraded":"2024-09-21T10:02:14.096583+0000","last_fullsized":"2024-09-21T10:02:14.096583+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:07:01.888435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091542+0000","last_change":"2024-09-21T09:52:20.371638+0000","last_active":"2024-09-21T10:02:13.091542+0000","last_peered":"2024-09-21T10:02:13.091542+0000","last_clean":"2024-09-21T10:02:13.091542+0000","last_became_active":"2024-09-21T09:52:20.371383+0000","last_became_peered":"2024-09-21T09:52:20.371383+0000","last_unstale":"2024-09-21T10:02:13.091542+0000","last_undegraded":"2024-09-21T10:02:13.091542+0000","last_fullsized":"2024-09-21T10:02:13.091542+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:33:45.384397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099880+0000","last_change":"2024-09-21T10:02:14.099799+0000","last_active":"2024-09-21T10:02:14.099880+0000","last_peered":"2024-09-21T10:02:14.099880+0000","last_clean":"2024-09-21T10:02:14.099880+0000","last_became_active":"2024-09-21T10:02:14.099375+0000","last_became_peered":"2024-09-21T10:02:14.099375+0000","last_unstale":"2024-09-21T10:02:14.099880+0000","last_undegraded":"2024-09-21T10:02:14.099880+0000","last_fullsized":"2024-09-21T10:02:14.099880+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:13.407638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":75,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099528+0000","last_change":"2024-09-21T10:02:14.099439+0000","last_active":"2024-09-21T10:02:14.099528+0000","last_peered":"2024-09-21T10:02:14.099528+0000","last_clean":"2024-09-21T10:02:14.099528+0000","last_became_active":"2024-09-21T10:02:14.099117+0000","last_became_peered":"2024-09-21T10:02:14.099117+0000","last_unstale":"2024-09-21T10:02:14.099528+0000","last_undegraded":"2024-09-21T10:02:14.099528+0000","last_fullsized":"2024-09-21T10:02:14.099528+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:39.681817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":76,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086376+0000","last_change":"2024-09-21T10:01:41.360674+0000","last_active":"2024-09-21T10:02:13.086376+0000","last_peered":"2024-09-21T10:02:13.086376+0000","last_clean":"2024-09-21T10:02:13.086376+0000","last_became_active":"2024-09-21T10:01:41.360258+0000","last_became_peered":"2024-09-21T10:01:41.360258+0000","last_unstale":"2024-09-21T10:02:13.086376+0000","last_undegraded":"2024-09-21T10:02:13.086376+0000","last_fullsized":"2024-09-21T10:02:13.086376+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:06.200022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.088916+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:02:14.088916+0000","last_peered":"2024-09-21T10:02:14.088916+0000","last_clean":"2024-09-21T10:02:14.088916+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:02:14.088916+0000","last_undegraded":"2024-09-21T10:02:14.088916+0000","last_fullsized":"2024-09-21T10:02:14.088916+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5983,"num_read_kb":13006,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397661,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307316,"kb_used_data":5792,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93437836,"statfs":{"total":95995035648,"available":95680344064,"internally_reserved":0,"allocated":5931008,"data_stored":5374871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593889,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299044,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446108,"statfs":{"total":95995035648,"available":95688814592,"internally_reserved":0,"allocated":1728512,"data_stored":1180440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790119,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302552,"kb_used_data":5192,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93442600,"statfs":{"total":95995035648,"available":95685222400,"internally_reserved":0,"allocated":5316608,"data_stored":4779887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986348,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298476,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446676,"statfs":{"total":95995035648,"available":95689396224,"internally_reserved":0,"allocated":1138688,"data_stored":590121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":101,"seq":433791696910,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28556,"kb_used_data":1092,"kb_used_omap":5,"kb_used_meta":27450,"kb_avail":93716596,"statfs":{"total":95995035648,"available":95965794304,"internally_reserved":0,"allocated":1118208,"data_stored":585341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":28109584},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860436,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32732,"kb_used_data":5192,"kb_used_omap":8,"kb_used_meta":27511,"kb_avail":93712420,"statfs":{"total":95995035648,"available":95961518080,"internally_reserved":0,"allocated":5316608,"data_stored":4780167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":28172140},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":91,"seq":390842023963,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28616,"kb_used_data":1084,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93716536,"statfs":{"total":95995035648,"available":95965732864,"internally_reserved":0,"allocated":1110016,"data_stored":584648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":28172850},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187498,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29212,"kb_used_data":1676,"kb_used_omap":8,"kb_used_meta":27511,"kb_avail":93715940,"statfs":{"total":95995035648,"available":95965122560,"internally_reserved":0,"allocated":1716224,"data_stored":1176219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":28172273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:03:15.031 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T10:03:15.031 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T10:03:15.031 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-21T10:03:15.031 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph health --format=json 2024-09-21T10:03:15.274 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:15.680 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:15 smithi098 ceph-mon[97580]: from='client.44819 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:03:15.680 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:15 smithi098 ceph-mon[100914]: from='client.44819 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:03:15.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:15 smithi196 ceph-mon[84784]: from='client.44819 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:03:16.710 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:16 smithi098 ceph-mon[97580]: pgmap v222: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:16.711 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:16 smithi098 ceph-mon[100914]: pgmap v222: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:16 smithi196 ceph-mon[84784]: pgmap v222: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:17.298 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:03:17.298 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T10:03:17.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:17 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3333194290' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T10:03:17.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:17 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3333194290' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T10:03:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:17 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3333194290' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T10:03:17.992 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-21T10:03:17.992 INFO:teuthology.run_tasks:Running task print... 2024-09-21T10:03:18.004 INFO:teuthology.task.print:**** applying stress + thrashing to mixed-version cluster **** 2024-09-21T10:03:18.004 INFO:teuthology.run_tasks:Running task parallel... 2024-09-21T10:03:18.023 INFO:teuthology.task.parallel:starting parallel... 2024-09-21T10:03:18.024 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-21T10:03:18.024 INFO:teuthology.task.sequential:In sequential, running task thrashosds... 2024-09-21T10:03:18.026 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-21T10:03:18.026 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'OSD_DOWN', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSDMAP_FLAGS', 'PG_AVAILABILITY'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'main', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}} 2024-09-21T10:03:18.026 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-21T10:03:18.026 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-21T10:03:18.026 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T10:03:18.027 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:03:18.274 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:18.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:18 smithi098 ceph-mon[97580]: pgmap v223: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:18.654 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:18 smithi098 ceph-mon[100914]: pgmap v223: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:18.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:18 smithi196 ceph-mon[84784]: pgmap v223: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:20.161 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:03:20.162 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:03:20.607 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:03:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:03:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:20.851 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-09-21T10:03:19.019290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6289,"num_read_kb":13765,"num_write":1056,"num_write_kb":14251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1132,"ondisk_log_size":1132,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1326504,"kb_used_data":22828,"kb_used_omap":35,"kb_used_meta":1303516,"kb_avail":748634712,"statfs":{"total":767960285184,"available":766601945088,"internally_reserved":0,"allocated":23375872,"data_stored":19051694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36353,"internal_metadata":1334800895},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002906"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.098066+0000","last_change":"2024-09-21T10:02:14.097991+0000","last_active":"2024-09-21T10:02:14.098066+0000","last_peered":"2024-09-21T10:02:14.098066+0000","last_clean":"2024-09-21T10:02:14.098066+0000","last_became_active":"2024-09-21T10:02:14.097407+0000","last_became_peered":"2024-09-21T10:02:14.097407+0000","last_unstale":"2024-09-21T10:02:14.098066+0000","last_undegraded":"2024-09-21T10:02:14.098066+0000","last_fullsized":"2024-09-21T10:02:14.098066+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:18:42.098886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":69,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.085145+0000","last_change":"2024-09-21T10:01:41.360784+0000","last_active":"2024-09-21T10:02:13.085145+0000","last_peered":"2024-09-21T10:02:13.085145+0000","last_clean":"2024-09-21T10:02:13.085145+0000","last_became_active":"2024-09-21T10:01:41.360462+0000","last_became_peered":"2024-09-21T10:01:41.360462+0000","last_unstale":"2024-09-21T10:02:13.085145+0000","last_undegraded":"2024-09-21T10:02:13.085145+0000","last_fullsized":"2024-09-21T10:02:13.085145+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:11.867275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":66,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.103373+0000","last_change":"2024-09-21T10:02:14.103373+0000","last_active":"2024-09-21T10:02:14.103373+0000","last_peered":"2024-09-21T10:02:14.103373+0000","last_clean":"2024-09-21T10:02:14.103373+0000","last_became_active":"2024-09-21T10:02:14.102985+0000","last_became_peered":"2024-09-21T10:02:14.102985+0000","last_unstale":"2024-09-21T10:02:14.103373+0000","last_undegraded":"2024-09-21T10:02:14.103373+0000","last_fullsized":"2024-09-21T10:02:14.103373+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:03.418174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099028+0000","last_change":"2024-09-21T10:02:14.098949+0000","last_active":"2024-09-21T10:02:14.099028+0000","last_peered":"2024-09-21T10:02:14.099028+0000","last_clean":"2024-09-21T10:02:14.099028+0000","last_became_active":"2024-09-21T10:02:14.098646+0000","last_became_peered":"2024-09-21T10:02:14.098646+0000","last_unstale":"2024-09-21T10:02:14.099028+0000","last_undegraded":"2024-09-21T10:02:14.099028+0000","last_fullsized":"2024-09-21T10:02:14.099028+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:50:56.862529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099148+0000","last_change":"2024-09-21T10:02:14.098205+0000","last_active":"2024-09-21T10:02:14.099148+0000","last_peered":"2024-09-21T10:02:14.099148+0000","last_clean":"2024-09-21T10:02:14.099148+0000","last_became_active":"2024-09-21T10:02:14.097911+0000","last_became_peered":"2024-09-21T10:02:14.097911+0000","last_unstale":"2024-09-21T10:02:14.099148+0000","last_undegraded":"2024-09-21T10:02:14.099148+0000","last_fullsized":"2024-09-21T10:02:14.099148+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:55.971191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.085481+0000","last_change":"2024-09-21T09:52:20.371775+0000","last_active":"2024-09-21T10:02:13.085481+0000","last_peered":"2024-09-21T10:02:13.085481+0000","last_clean":"2024-09-21T10:02:13.085481+0000","last_became_active":"2024-09-21T09:52:20.371509+0000","last_became_peered":"2024-09-21T09:52:20.371509+0000","last_unstale":"2024-09-21T10:02:13.085481+0000","last_undegraded":"2024-09-21T10:02:13.085481+0000","last_fullsized":"2024-09-21T10:02:13.085481+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:43:21.822524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097891+0000","last_change":"2024-09-21T10:02:14.097817+0000","last_active":"2024-09-21T10:02:14.097891+0000","last_peered":"2024-09-21T10:02:14.097891+0000","last_clean":"2024-09-21T10:02:14.097891+0000","last_became_active":"2024-09-21T10:02:14.097466+0000","last_became_peered":"2024-09-21T10:02:14.097466+0000","last_unstale":"2024-09-21T10:02:14.097891+0000","last_undegraded":"2024-09-21T10:02:14.097891+0000","last_fullsized":"2024-09-21T10:02:14.097891+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:57.883118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":75,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.090786+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:02:13.090786+0000","last_peered":"2024-09-21T10:02:13.090786+0000","last_clean":"2024-09-21T10:02:13.090786+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:02:13.090786+0000","last_undegraded":"2024-09-21T10:02:13.090786+0000","last_fullsized":"2024-09-21T10:02:13.090786+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":68,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086446+0000","last_change":"2024-09-21T09:59:54.265455+0000","last_active":"2024-09-21T10:02:13.086446+0000","last_peered":"2024-09-21T10:02:13.086446+0000","last_clean":"2024-09-21T10:02:13.086446+0000","last_became_active":"2024-09-21T09:59:54.265110+0000","last_became_peered":"2024-09-21T09:59:54.265110+0000","last_unstale":"2024-09-21T10:02:13.086446+0000","last_undegraded":"2024-09-21T10:02:13.086446+0000","last_fullsized":"2024-09-21T10:02:13.086446+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:04:13.404874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.089734+0000","last_change":"2024-09-21T09:52:20.402753+0000","last_active":"2024-09-21T10:02:13.089734+0000","last_peered":"2024-09-21T10:02:13.089734+0000","last_clean":"2024-09-21T10:02:13.089734+0000","last_became_active":"2024-09-21T09:52:20.402462+0000","last_became_peered":"2024-09-21T09:52:20.402462+0000","last_unstale":"2024-09-21T10:02:13.089734+0000","last_undegraded":"2024-09-21T10:02:13.089734+0000","last_fullsized":"2024-09-21T10:02:13.089734+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:14.036869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":69,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086567+0000","last_change":"2024-09-21T10:01:07.420484+0000","last_active":"2024-09-21T10:02:13.086567+0000","last_peered":"2024-09-21T10:02:13.086567+0000","last_clean":"2024-09-21T10:02:13.086567+0000","last_became_active":"2024-09-21T10:01:07.420135+0000","last_became_peered":"2024-09-21T10:01:07.420135+0000","last_unstale":"2024-09-21T10:02:13.086567+0000","last_undegraded":"2024-09-21T10:02:13.086567+0000","last_fullsized":"2024-09-21T10:02:13.086567+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:33:10.241010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":83,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.098139+0000","last_change":"2024-09-21T10:02:14.098139+0000","last_active":"2024-09-21T10:02:14.098139+0000","last_peered":"2024-09-21T10:02:14.098139+0000","last_clean":"2024-09-21T10:02:14.098139+0000","last_became_active":"2024-09-21T10:02:14.097639+0000","last_became_peered":"2024-09-21T10:02:14.097639+0000","last_unstale":"2024-09-21T10:02:14.098139+0000","last_undegraded":"2024-09-21T10:02:14.098139+0000","last_fullsized":"2024-09-21T10:02:14.098139+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:11:26.925388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087557+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:02:13.087557+0000","last_peered":"2024-09-21T10:02:13.087557+0000","last_clean":"2024-09-21T10:02:13.087557+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:02:13.087557+0000","last_undegraded":"2024-09-21T10:02:13.087557+0000","last_fullsized":"2024-09-21T10:02:13.087557+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":66,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091038+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:02:13.091038+0000","last_peered":"2024-09-21T10:02:13.091038+0000","last_clean":"2024-09-21T10:02:13.091038+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:02:13.091038+0000","last_undegraded":"2024-09-21T10:02:13.091038+0000","last_fullsized":"2024-09-21T10:02:13.091038+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":67,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.104212+0000","last_change":"2024-09-21T10:02:14.104212+0000","last_active":"2024-09-21T10:02:14.104212+0000","last_peered":"2024-09-21T10:02:14.104212+0000","last_clean":"2024-09-21T10:02:14.104212+0000","last_became_active":"2024-09-21T10:02:14.103893+0000","last_became_peered":"2024-09-21T10:02:14.103893+0000","last_unstale":"2024-09-21T10:02:14.104212+0000","last_undegraded":"2024-09-21T10:02:14.104212+0000","last_fullsized":"2024-09-21T10:02:14.104212+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:59:21.191412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.091970+0000","last_change":"2024-09-21T10:01:07.419715+0000","last_active":"2024-09-21T10:02:14.091970+0000","last_peered":"2024-09-21T10:02:14.091970+0000","last_clean":"2024-09-21T10:02:14.091970+0000","last_became_active":"2024-09-21T10:01:07.419422+0000","last_became_peered":"2024-09-21T10:01:07.419422+0000","last_unstale":"2024-09-21T10:02:14.091970+0000","last_undegraded":"2024-09-21T10:02:14.091970+0000","last_fullsized":"2024-09-21T10:02:14.091970+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:36.424289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":67,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.096638+0000","last_change":"2024-09-21T10:02:14.096537+0000","last_active":"2024-09-21T10:02:14.096638+0000","last_peered":"2024-09-21T10:02:14.096638+0000","last_clean":"2024-09-21T10:02:14.096638+0000","last_became_active":"2024-09-21T10:02:14.096166+0000","last_became_peered":"2024-09-21T10:02:14.096166+0000","last_unstale":"2024-09-21T10:02:14.096638+0000","last_undegraded":"2024-09-21T10:02:14.096638+0000","last_fullsized":"2024-09-21T10:02:14.096638+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:54:18.927732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":66,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099205+0000","last_change":"2024-09-21T10:02:14.099205+0000","last_active":"2024-09-21T10:02:14.099205+0000","last_peered":"2024-09-21T10:02:14.099205+0000","last_clean":"2024-09-21T10:02:14.099205+0000","last_became_active":"2024-09-21T10:02:14.098598+0000","last_became_peered":"2024-09-21T10:02:14.098598+0000","last_unstale":"2024-09-21T10:02:14.099205+0000","last_undegraded":"2024-09-21T10:02:14.099205+0000","last_fullsized":"2024-09-21T10:02:14.099205+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:46.424612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":67,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091165+0000","last_change":"2024-09-21T10:01:07.424883+0000","last_active":"2024-09-21T10:02:13.091165+0000","last_peered":"2024-09-21T10:02:13.091165+0000","last_clean":"2024-09-21T10:02:13.091165+0000","last_became_active":"2024-09-21T10:01:07.424499+0000","last_became_peered":"2024-09-21T10:01:07.424499+0000","last_unstale":"2024-09-21T10:02:13.091165+0000","last_undegraded":"2024-09-21T10:02:13.091165+0000","last_fullsized":"2024-09-21T10:02:13.091165+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:46.101570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":83,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087300+0000","last_change":"2024-09-21T10:01:41.362291+0000","last_active":"2024-09-21T10:02:13.087300+0000","last_peered":"2024-09-21T10:02:13.087300+0000","last_clean":"2024-09-21T10:02:13.087300+0000","last_became_active":"2024-09-21T10:01:41.361954+0000","last_became_peered":"2024-09-21T10:01:41.361954+0000","last_unstale":"2024-09-21T10:02:13.087300+0000","last_undegraded":"2024-09-21T10:02:13.087300+0000","last_fullsized":"2024-09-21T10:02:13.087300+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:57.669671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":67,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087361+0000","last_change":"2024-09-21T10:01:41.361244+0000","last_active":"2024-09-21T10:02:13.087361+0000","last_peered":"2024-09-21T10:02:13.087361+0000","last_clean":"2024-09-21T10:02:13.087361+0000","last_became_active":"2024-09-21T10:01:41.360645+0000","last_became_peered":"2024-09-21T10:01:41.360645+0000","last_unstale":"2024-09-21T10:02:13.087361+0000","last_undegraded":"2024-09-21T10:02:13.087361+0000","last_fullsized":"2024-09-21T10:02:13.087361+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:07.604991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":66,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086023+0000","last_change":"2024-09-21T10:01:07.420824+0000","last_active":"2024-09-21T10:02:13.086023+0000","last_peered":"2024-09-21T10:02:13.086023+0000","last_clean":"2024-09-21T10:02:13.086023+0000","last_became_active":"2024-09-21T10:01:07.420387+0000","last_became_peered":"2024-09-21T10:01:07.420387+0000","last_unstale":"2024-09-21T10:02:13.086023+0000","last_undegraded":"2024-09-21T10:02:13.086023+0000","last_fullsized":"2024-09-21T10:02:13.086023+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:48.875842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":70,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097606+0000","last_change":"2024-09-21T10:02:14.097462+0000","last_active":"2024-09-21T10:02:14.097606+0000","last_peered":"2024-09-21T10:02:14.097606+0000","last_clean":"2024-09-21T10:02:14.097606+0000","last_became_active":"2024-09-21T10:02:14.097167+0000","last_became_peered":"2024-09-21T10:02:14.097167+0000","last_unstale":"2024-09-21T10:02:14.097606+0000","last_undegraded":"2024-09-21T10:02:14.097606+0000","last_fullsized":"2024-09-21T10:02:14.097606+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:54.922731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":54,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097753+0000","last_change":"2024-09-21T10:02:14.097676+0000","last_active":"2024-09-21T10:02:14.097753+0000","last_peered":"2024-09-21T10:02:14.097753+0000","last_clean":"2024-09-21T10:02:14.097753+0000","last_became_active":"2024-09-21T10:02:14.097391+0000","last_became_peered":"2024-09-21T10:02:14.097391+0000","last_unstale":"2024-09-21T10:02:14.097753+0000","last_undegraded":"2024-09-21T10:02:14.097753+0000","last_fullsized":"2024-09-21T10:02:14.097753+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:40:24.523178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":184,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087727+0000","last_change":"2024-09-21T09:59:54.265556+0000","last_active":"2024-09-21T10:02:13.087727+0000","last_peered":"2024-09-21T10:02:13.087727+0000","last_clean":"2024-09-21T10:02:13.087727+0000","last_became_active":"2024-09-21T09:59:54.265033+0000","last_became_peered":"2024-09-21T09:59:54.265033+0000","last_unstale":"2024-09-21T10:02:13.087727+0000","last_undegraded":"2024-09-21T10:02:13.087727+0000","last_fullsized":"2024-09-21T10:02:13.087727+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T09:48:34.850872+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:40.449756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":491,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.089982+0000","last_change":"2024-09-21T10:01:41.361089+0000","last_active":"2024-09-21T10:02:13.089982+0000","last_peered":"2024-09-21T10:02:13.089982+0000","last_clean":"2024-09-21T10:02:13.089982+0000","last_became_active":"2024-09-21T10:01:41.360658+0000","last_became_peered":"2024-09-21T10:01:41.360658+0000","last_unstale":"2024-09-21T10:02:13.089982+0000","last_undegraded":"2024-09-21T10:02:13.089982+0000","last_fullsized":"2024-09-21T10:02:13.089982+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:58:06.840024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":482,"num_read_kb":1581,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.100041+0000","last_change":"2024-09-21T10:02:14.099961+0000","last_active":"2024-09-21T10:02:14.100041+0000","last_peered":"2024-09-21T10:02:14.100041+0000","last_clean":"2024-09-21T10:02:14.100041+0000","last_became_active":"2024-09-21T10:02:14.099067+0000","last_became_peered":"2024-09-21T10:02:14.099067+0000","last_unstale":"2024-09-21T10:02:14.100041+0000","last_undegraded":"2024-09-21T10:02:14.100041+0000","last_fullsized":"2024-09-21T10:02:14.100041+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:41.037532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091668+0000","last_change":"2024-09-21T09:52:20.403534+0000","last_active":"2024-09-21T10:02:13.091668+0000","last_peered":"2024-09-21T10:02:13.091668+0000","last_clean":"2024-09-21T10:02:13.091668+0000","last_became_active":"2024-09-21T09:52:20.403258+0000","last_became_peered":"2024-09-21T09:52:20.403258+0000","last_unstale":"2024-09-21T10:02:13.091668+0000","last_undegraded":"2024-09-21T10:02:13.091668+0000","last_fullsized":"2024-09-21T10:02:13.091668+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:55.374343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":447,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087637+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:02:13.087637+0000","last_peered":"2024-09-21T10:02:13.087637+0000","last_clean":"2024-09-21T10:02:13.087637+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:02:13.087637+0000","last_undegraded":"2024-09-21T10:02:13.087637+0000","last_fullsized":"2024-09-21T10:02:13.087637+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097180+0000","last_change":"2024-09-21T10:02:14.097097+0000","last_active":"2024-09-21T10:02:14.097180+0000","last_peered":"2024-09-21T10:02:14.097180+0000","last_clean":"2024-09-21T10:02:14.097180+0000","last_became_active":"2024-09-21T10:02:14.096788+0000","last_became_peered":"2024-09-21T10:02:14.096788+0000","last_unstale":"2024-09-21T10:02:14.097180+0000","last_undegraded":"2024-09-21T10:02:14.097180+0000","last_fullsized":"2024-09-21T10:02:14.097180+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:05:29.711020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":76,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090168+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:02:14.090168+0000","last_peered":"2024-09-21T10:02:14.090168+0000","last_clean":"2024-09-21T10:02:14.090168+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:02:14.090168+0000","last_undegraded":"2024-09-21T10:02:14.090168+0000","last_fullsized":"2024-09-21T10:02:14.090168+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":815,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.100244+0000","last_change":"2024-09-21T10:02:14.100143+0000","last_active":"2024-09-21T10:02:14.100244+0000","last_peered":"2024-09-21T10:02:14.100244+0000","last_clean":"2024-09-21T10:02:14.100244+0000","last_became_active":"2024-09-21T10:02:14.099586+0000","last_became_peered":"2024-09-21T10:02:14.099586+0000","last_unstale":"2024-09-21T10:02:14.100244+0000","last_undegraded":"2024-09-21T10:02:14.100244+0000","last_fullsized":"2024-09-21T10:02:14.100244+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:34.064652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":55,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090600+0000","last_change":"2024-09-21T10:01:41.358268+0000","last_active":"2024-09-21T10:02:14.090600+0000","last_peered":"2024-09-21T10:02:14.090600+0000","last_clean":"2024-09-21T10:02:14.090600+0000","last_became_active":"2024-09-21T10:01:41.357816+0000","last_became_peered":"2024-09-21T10:01:41.357816+0000","last_unstale":"2024-09-21T10:02:14.090600+0000","last_undegraded":"2024-09-21T10:02:14.090600+0000","last_fullsized":"2024-09-21T10:02:14.090600+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:59:44.261636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":72,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.096377+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:02:14.096377+0000","last_peered":"2024-09-21T10:02:14.096377+0000","last_clean":"2024-09-21T10:02:14.096377+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:02:14.096377+0000","last_undegraded":"2024-09-21T10:02:14.096377+0000","last_fullsized":"2024-09-21T10:02:14.096377+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":675,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.090133+0000","last_change":"2024-09-21T10:01:07.424819+0000","last_active":"2024-09-21T10:02:13.090133+0000","last_peered":"2024-09-21T10:02:13.090133+0000","last_clean":"2024-09-21T10:02:13.090133+0000","last_became_active":"2024-09-21T10:01:07.424392+0000","last_became_peered":"2024-09-21T10:01:07.424392+0000","last_unstale":"2024-09-21T10:02:13.090133+0000","last_undegraded":"2024-09-21T10:02:13.090133+0000","last_fullsized":"2024-09-21T10:02:13.090133+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:27.266390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":67,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.088170+0000","last_change":"2024-09-21T09:59:54.267742+0000","last_active":"2024-09-21T10:02:13.088170+0000","last_peered":"2024-09-21T10:02:13.088170+0000","last_clean":"2024-09-21T10:02:13.088170+0000","last_became_active":"2024-09-21T09:59:54.267406+0000","last_became_peered":"2024-09-21T09:59:54.267406+0000","last_unstale":"2024-09-21T10:02:13.088170+0000","last_undegraded":"2024-09-21T10:02:13.088170+0000","last_fullsized":"2024-09-21T10:02:13.088170+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:25:35.968779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":80,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.103531+0000","last_change":"2024-09-21T10:02:14.103531+0000","last_active":"2024-09-21T10:02:14.103531+0000","last_peered":"2024-09-21T10:02:14.103531+0000","last_clean":"2024-09-21T10:02:14.103531+0000","last_became_active":"2024-09-21T10:02:14.103218+0000","last_became_peered":"2024-09-21T10:02:14.103218+0000","last_unstale":"2024-09-21T10:02:14.103531+0000","last_undegraded":"2024-09-21T10:02:14.103531+0000","last_fullsized":"2024-09-21T10:02:14.103531+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:40.014308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":84,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.103145+0000","last_change":"2024-09-21T10:02:14.103145+0000","last_active":"2024-09-21T10:02:14.103145+0000","last_peered":"2024-09-21T10:02:14.103145+0000","last_clean":"2024-09-21T10:02:14.103145+0000","last_became_active":"2024-09-21T10:02:14.102806+0000","last_became_peered":"2024-09-21T10:02:14.102806+0000","last_unstale":"2024-09-21T10:02:14.103145+0000","last_undegraded":"2024-09-21T10:02:14.103145+0000","last_fullsized":"2024-09-21T10:02:14.103145+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:24.348745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":760,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090783+0000","last_change":"2024-09-21T10:01:07.426065+0000","last_active":"2024-09-21T10:02:14.090783+0000","last_peered":"2024-09-21T10:02:14.090783+0000","last_clean":"2024-09-21T10:02:14.090783+0000","last_became_active":"2024-09-21T10:01:07.423163+0000","last_became_peered":"2024-09-21T10:01:07.423163+0000","last_unstale":"2024-09-21T10:02:14.090783+0000","last_undegraded":"2024-09-21T10:02:14.090783+0000","last_fullsized":"2024-09-21T10:02:14.090783+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:43.564978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":44,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090907+0000","last_change":"2024-09-21T10:01:07.422175+0000","last_active":"2024-09-21T10:02:14.090907+0000","last_peered":"2024-09-21T10:02:14.090907+0000","last_clean":"2024-09-21T10:02:14.090907+0000","last_became_active":"2024-09-21T10:01:07.421844+0000","last_became_peered":"2024-09-21T10:01:07.421844+0000","last_unstale":"2024-09-21T10:02:14.090907+0000","last_undegraded":"2024-09-21T10:02:14.090907+0000","last_fullsized":"2024-09-21T10:02:14.090907+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:17:15.192573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":55,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090704+0000","last_change":"2024-09-21T10:01:41.360338+0000","last_active":"2024-09-21T10:02:14.090704+0000","last_peered":"2024-09-21T10:02:14.090704+0000","last_clean":"2024-09-21T10:02:14.090704+0000","last_became_active":"2024-09-21T10:01:41.360050+0000","last_became_peered":"2024-09-21T10:01:41.360050+0000","last_unstale":"2024-09-21T10:02:14.090704+0000","last_undegraded":"2024-09-21T10:02:14.090704+0000","last_fullsized":"2024-09-21T10:02:14.090704+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:39:30.459393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":413,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087649+0000","last_change":"2024-09-21T09:59:54.265559+0000","last_active":"2024-09-21T10:02:13.087649+0000","last_peered":"2024-09-21T10:02:13.087649+0000","last_clean":"2024-09-21T10:02:13.087649+0000","last_became_active":"2024-09-21T09:59:54.264884+0000","last_became_peered":"2024-09-21T09:59:54.264884+0000","last_unstale":"2024-09-21T10:02:13.087649+0000","last_undegraded":"2024-09-21T10:02:13.087649+0000","last_fullsized":"2024-09-21T10:02:13.087649+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:24.544845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":54,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.091931+0000","last_change":"2024-09-21T09:59:54.267798+0000","last_active":"2024-09-21T10:02:14.091931+0000","last_peered":"2024-09-21T10:02:14.091931+0000","last_clean":"2024-09-21T10:02:14.091931+0000","last_became_active":"2024-09-21T09:59:54.265615+0000","last_became_peered":"2024-09-21T09:59:54.265615+0000","last_unstale":"2024-09-21T10:02:14.091931+0000","last_undegraded":"2024-09-21T10:02:14.091931+0000","last_fullsized":"2024-09-21T10:02:14.091931+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:30.363872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":67,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.102034+0000","last_change":"2024-09-21T10:02:14.101943+0000","last_active":"2024-09-21T10:02:14.102034+0000","last_peered":"2024-09-21T10:02:14.102034+0000","last_clean":"2024-09-21T10:02:14.102034+0000","last_became_active":"2024-09-21T10:02:14.101488+0000","last_became_peered":"2024-09-21T10:02:14.101488+0000","last_unstale":"2024-09-21T10:02:14.102034+0000","last_undegraded":"2024-09-21T10:02:14.102034+0000","last_fullsized":"2024-09-21T10:02:14.102034+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:52:27.866414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":424,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.089522+0000","last_change":"2024-09-21T10:01:07.425330+0000","last_active":"2024-09-21T10:02:14.089522+0000","last_peered":"2024-09-21T10:02:14.089522+0000","last_clean":"2024-09-21T10:02:14.089522+0000","last_became_active":"2024-09-21T10:01:07.424695+0000","last_became_peered":"2024-09-21T10:01:07.424695+0000","last_unstale":"2024-09-21T10:02:14.089522+0000","last_undegraded":"2024-09-21T10:02:14.089522+0000","last_fullsized":"2024-09-21T10:02:14.089522+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:13:22.297576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":64,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.096024+0000","last_change":"2024-09-21T10:02:14.095924+0000","last_active":"2024-09-21T10:02:14.096024+0000","last_peered":"2024-09-21T10:02:14.096024+0000","last_clean":"2024-09-21T10:02:14.096024+0000","last_became_active":"2024-09-21T10:02:14.095553+0000","last_became_peered":"2024-09-21T10:02:14.095553+0000","last_unstale":"2024-09-21T10:02:14.096024+0000","last_undegraded":"2024-09-21T10:02:14.096024+0000","last_fullsized":"2024-09-21T10:02:14.096024+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:08.155980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.089383+0000","last_change":"2024-09-21T10:01:07.426736+0000","last_active":"2024-09-21T10:02:14.089383+0000","last_peered":"2024-09-21T10:02:14.089383+0000","last_clean":"2024-09-21T10:02:14.089383+0000","last_became_active":"2024-09-21T10:01:07.426416+0000","last_became_peered":"2024-09-21T10:01:07.426416+0000","last_unstale":"2024-09-21T10:02:14.089383+0000","last_undegraded":"2024-09-21T10:02:14.089383+0000","last_fullsized":"2024-09-21T10:02:14.089383+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:59.784757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":329,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091262+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:02:13.091262+0000","last_peered":"2024-09-21T10:02:13.091262+0000","last_clean":"2024-09-21T10:02:13.091262+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:02:13.091262+0000","last_undegraded":"2024-09-21T10:02:13.091262+0000","last_fullsized":"2024-09-21T10:02:13.091262+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086571+0000","last_change":"2024-09-21T09:52:20.391439+0000","last_active":"2024-09-21T10:02:13.086571+0000","last_peered":"2024-09-21T10:02:13.086571+0000","last_clean":"2024-09-21T10:02:13.086571+0000","last_became_active":"2024-09-21T09:52:20.391157+0000","last_became_peered":"2024-09-21T09:52:20.391157+0000","last_unstale":"2024-09-21T10:02:13.086571+0000","last_undegraded":"2024-09-21T10:02:13.086571+0000","last_fullsized":"2024-09-21T10:02:13.086571+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:36.788330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":66,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.088062+0000","last_change":"2024-09-21T10:01:07.421150+0000","last_active":"2024-09-21T10:02:13.088062+0000","last_peered":"2024-09-21T10:02:13.088062+0000","last_clean":"2024-09-21T10:02:13.088062+0000","last_became_active":"2024-09-21T10:01:07.420830+0000","last_became_peered":"2024-09-21T10:01:07.420830+0000","last_unstale":"2024-09-21T10:02:13.088062+0000","last_undegraded":"2024-09-21T10:02:13.088062+0000","last_fullsized":"2024-09-21T10:02:13.088062+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:15:34.472626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":72,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087220+0000","last_change":"2024-09-21T09:52:20.371668+0000","last_active":"2024-09-21T10:02:13.087220+0000","last_peered":"2024-09-21T10:02:13.087220+0000","last_clean":"2024-09-21T10:02:13.087220+0000","last_became_active":"2024-09-21T09:52:20.371389+0000","last_became_peered":"2024-09-21T09:52:20.371389+0000","last_unstale":"2024-09-21T10:02:13.087220+0000","last_undegraded":"2024-09-21T10:02:13.087220+0000","last_fullsized":"2024-09-21T10:02:13.087220+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:10.013020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":54,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090703+0000","last_change":"2024-09-21T10:01:41.360337+0000","last_active":"2024-09-21T10:02:14.090703+0000","last_peered":"2024-09-21T10:02:14.090703+0000","last_clean":"2024-09-21T10:02:14.090703+0000","last_became_active":"2024-09-21T10:01:41.359996+0000","last_became_peered":"2024-09-21T10:01:41.359996+0000","last_unstale":"2024-09-21T10:02:14.090703+0000","last_undegraded":"2024-09-21T10:02:14.090703+0000","last_fullsized":"2024-09-21T10:02:14.090703+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:28:43.820241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":70,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.102896+0000","last_change":"2024-09-21T10:02:14.102896+0000","last_active":"2024-09-21T10:02:14.102896+0000","last_peered":"2024-09-21T10:02:14.102896+0000","last_clean":"2024-09-21T10:02:14.102896+0000","last_became_active":"2024-09-21T10:02:14.102493+0000","last_became_peered":"2024-09-21T10:02:14.102493+0000","last_unstale":"2024-09-21T10:02:14.102896+0000","last_undegraded":"2024-09-21T10:02:14.102896+0000","last_fullsized":"2024-09-21T10:02:14.102896+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:10:43.572005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":67,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.089930+0000","last_change":"2024-09-21T10:01:07.424062+0000","last_active":"2024-09-21T10:02:13.089930+0000","last_peered":"2024-09-21T10:02:13.089930+0000","last_clean":"2024-09-21T10:02:13.089930+0000","last_became_active":"2024-09-21T10:01:07.423629+0000","last_became_peered":"2024-09-21T10:01:07.423629+0000","last_unstale":"2024-09-21T10:02:13.089930+0000","last_undegraded":"2024-09-21T10:02:13.089930+0000","last_fullsized":"2024-09-21T10:02:13.089930+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:24.119642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.100529+0000","last_change":"2024-09-21T10:02:14.100451+0000","last_active":"2024-09-21T10:02:14.100529+0000","last_peered":"2024-09-21T10:02:14.100529+0000","last_clean":"2024-09-21T10:02:14.100529+0000","last_became_active":"2024-09-21T10:02:14.100160+0000","last_became_peered":"2024-09-21T10:02:14.100160+0000","last_unstale":"2024-09-21T10:02:14.100529+0000","last_undegraded":"2024-09-21T10:02:14.100529+0000","last_fullsized":"2024-09-21T10:02:14.100529+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:28.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.096493+0000","last_change":"2024-09-21T10:02:14.096098+0000","last_active":"2024-09-21T10:02:14.096493+0000","last_peered":"2024-09-21T10:02:14.096493+0000","last_clean":"2024-09-21T10:02:14.096493+0000","last_became_active":"2024-09-21T10:02:14.095811+0000","last_became_peered":"2024-09-21T10:02:14.095811+0000","last_unstale":"2024-09-21T10:02:14.096493+0000","last_undegraded":"2024-09-21T10:02:14.096493+0000","last_fullsized":"2024-09-21T10:02:14.096493+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:54.619333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.100791+0000","last_change":"2024-09-21T10:02:14.100712+0000","last_active":"2024-09-21T10:02:14.100791+0000","last_peered":"2024-09-21T10:02:14.100791+0000","last_clean":"2024-09-21T10:02:14.100791+0000","last_became_active":"2024-09-21T10:02:14.100427+0000","last_became_peered":"2024-09-21T10:02:14.100427+0000","last_unstale":"2024-09-21T10:02:14.100791+0000","last_undegraded":"2024-09-21T10:02:14.100791+0000","last_fullsized":"2024-09-21T10:02:14.100791+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:16:42.350642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":66,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091136+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:02:13.091136+0000","last_peered":"2024-09-21T10:02:13.091136+0000","last_clean":"2024-09-21T10:02:13.091136+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:02:13.091136+0000","last_undegraded":"2024-09-21T10:02:13.091136+0000","last_fullsized":"2024-09-21T10:02:13.091136+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":87,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097446+0000","last_change":"2024-09-21T10:02:14.097369+0000","last_active":"2024-09-21T10:02:14.097446+0000","last_peered":"2024-09-21T10:02:14.097446+0000","last_clean":"2024-09-21T10:02:14.097446+0000","last_became_active":"2024-09-21T10:02:14.097067+0000","last_became_peered":"2024-09-21T10:02:14.097067+0000","last_unstale":"2024-09-21T10:02:14.097446+0000","last_undegraded":"2024-09-21T10:02:14.097446+0000","last_fullsized":"2024-09-21T10:02:14.097446+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:50:48.123813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099046+0000","last_change":"2024-09-21T10:02:14.098970+0000","last_active":"2024-09-21T10:02:14.099046+0000","last_peered":"2024-09-21T10:02:14.099046+0000","last_clean":"2024-09-21T10:02:14.099046+0000","last_became_active":"2024-09-21T10:02:14.098641+0000","last_became_peered":"2024-09-21T10:02:14.098641+0000","last_unstale":"2024-09-21T10:02:14.099046+0000","last_undegraded":"2024-09-21T10:02:14.099046+0000","last_fullsized":"2024-09-21T10:02:14.099046+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:25.215602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":54,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097100+0000","last_change":"2024-09-21T10:02:14.097021+0000","last_active":"2024-09-21T10:02:14.097100+0000","last_peered":"2024-09-21T10:02:14.097100+0000","last_clean":"2024-09-21T10:02:14.097100+0000","last_became_active":"2024-09-21T10:02:14.096741+0000","last_became_peered":"2024-09-21T10:02:14.096741+0000","last_unstale":"2024-09-21T10:02:14.097100+0000","last_undegraded":"2024-09-21T10:02:14.097100+0000","last_fullsized":"2024-09-21T10:02:14.097100+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:18:20.631150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099544+0000","last_change":"2024-09-21T10:02:14.099460+0000","last_active":"2024-09-21T10:02:14.099544+0000","last_peered":"2024-09-21T10:02:14.099544+0000","last_clean":"2024-09-21T10:02:14.099544+0000","last_became_active":"2024-09-21T10:02:14.099159+0000","last_became_peered":"2024-09-21T10:02:14.099159+0000","last_unstale":"2024-09-21T10:02:14.099544+0000","last_undegraded":"2024-09-21T10:02:14.099544+0000","last_fullsized":"2024-09-21T10:02:14.099544+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:23:47.599107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":85,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086660+0000","last_change":"2024-09-21T10:01:41.360480+0000","last_active":"2024-09-21T10:02:13.086660+0000","last_peered":"2024-09-21T10:02:13.086660+0000","last_clean":"2024-09-21T10:02:13.086660+0000","last_became_active":"2024-09-21T10:01:41.359791+0000","last_became_peered":"2024-09-21T10:01:41.359791+0000","last_unstale":"2024-09-21T10:02:13.086660+0000","last_undegraded":"2024-09-21T10:02:13.086660+0000","last_fullsized":"2024-09-21T10:02:13.086660+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:42.812974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.102901+0000","last_change":"2024-09-21T10:02:14.102826+0000","last_active":"2024-09-21T10:02:14.102901+0000","last_peered":"2024-09-21T10:02:14.102901+0000","last_clean":"2024-09-21T10:02:14.102901+0000","last_became_active":"2024-09-21T10:02:14.102499+0000","last_became_peered":"2024-09-21T10:02:14.102499+0000","last_unstale":"2024-09-21T10:02:14.102901+0000","last_undegraded":"2024-09-21T10:02:14.102901+0000","last_fullsized":"2024-09-21T10:02:14.102901+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:30.924687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":58,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.091881+0000","last_change":"2024-09-21T09:59:54.268378+0000","last_active":"2024-09-21T10:02:14.091881+0000","last_peered":"2024-09-21T10:02:14.091881+0000","last_clean":"2024-09-21T10:02:14.091881+0000","last_became_active":"2024-09-21T09:59:54.267187+0000","last_became_peered":"2024-09-21T09:59:54.267187+0000","last_unstale":"2024-09-21T10:02:14.091881+0000","last_undegraded":"2024-09-21T10:02:14.091881+0000","last_fullsized":"2024-09-21T10:02:14.091881+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:12:06.054483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":83,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.100541+0000","last_change":"2024-09-21T10:02:14.100541+0000","last_active":"2024-09-21T10:02:14.100541+0000","last_peered":"2024-09-21T10:02:14.100541+0000","last_clean":"2024-09-21T10:02:14.100541+0000","last_became_active":"2024-09-21T10:02:14.100327+0000","last_became_peered":"2024-09-21T10:02:14.100327+0000","last_unstale":"2024-09-21T10:02:14.100541+0000","last_undegraded":"2024-09-21T10:02:14.100541+0000","last_fullsized":"2024-09-21T10:02:14.100541+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:34:34.304188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":69,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.090359+0000","last_change":"2024-09-21T10:01:07.424248+0000","last_active":"2024-09-21T10:02:13.090359+0000","last_peered":"2024-09-21T10:02:13.090359+0000","last_clean":"2024-09-21T10:02:13.090359+0000","last_became_active":"2024-09-21T10:01:07.423738+0000","last_became_peered":"2024-09-21T10:01:07.423738+0000","last_unstale":"2024-09-21T10:02:13.090359+0000","last_undegraded":"2024-09-21T10:02:13.090359+0000","last_fullsized":"2024-09-21T10:02:13.090359+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:44:35.663555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":83,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.096583+0000","last_change":"2024-09-21T10:02:14.096583+0000","last_active":"2024-09-21T10:02:14.096583+0000","last_peered":"2024-09-21T10:02:14.096583+0000","last_clean":"2024-09-21T10:02:14.096583+0000","last_became_active":"2024-09-21T10:02:14.096196+0000","last_became_peered":"2024-09-21T10:02:14.096196+0000","last_unstale":"2024-09-21T10:02:14.096583+0000","last_undegraded":"2024-09-21T10:02:14.096583+0000","last_fullsized":"2024-09-21T10:02:14.096583+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:07:01.888435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091542+0000","last_change":"2024-09-21T09:52:20.371638+0000","last_active":"2024-09-21T10:02:13.091542+0000","last_peered":"2024-09-21T10:02:13.091542+0000","last_clean":"2024-09-21T10:02:13.091542+0000","last_became_active":"2024-09-21T09:52:20.371383+0000","last_became_peered":"2024-09-21T09:52:20.371383+0000","last_unstale":"2024-09-21T10:02:13.091542+0000","last_undegraded":"2024-09-21T10:02:13.091542+0000","last_fullsized":"2024-09-21T10:02:13.091542+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:33:45.384397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099880+0000","last_change":"2024-09-21T10:02:14.099799+0000","last_active":"2024-09-21T10:02:14.099880+0000","last_peered":"2024-09-21T10:02:14.099880+0000","last_clean":"2024-09-21T10:02:14.099880+0000","last_became_active":"2024-09-21T10:02:14.099375+0000","last_became_peered":"2024-09-21T10:02:14.099375+0000","last_unstale":"2024-09-21T10:02:14.099880+0000","last_undegraded":"2024-09-21T10:02:14.099880+0000","last_fullsized":"2024-09-21T10:02:14.099880+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:13.407638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":75,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099528+0000","last_change":"2024-09-21T10:02:14.099439+0000","last_active":"2024-09-21T10:02:14.099528+0000","last_peered":"2024-09-21T10:02:14.099528+0000","last_clean":"2024-09-21T10:02:14.099528+0000","last_became_active":"2024-09-21T10:02:14.099117+0000","last_became_peered":"2024-09-21T10:02:14.099117+0000","last_unstale":"2024-09-21T10:02:14.099528+0000","last_undegraded":"2024-09-21T10:02:14.099528+0000","last_fullsized":"2024-09-21T10:02:14.099528+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:39.681817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":76,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086376+0000","last_change":"2024-09-21T10:01:41.360674+0000","last_active":"2024-09-21T10:02:13.086376+0000","last_peered":"2024-09-21T10:02:13.086376+0000","last_clean":"2024-09-21T10:02:13.086376+0000","last_became_active":"2024-09-21T10:01:41.360258+0000","last_became_peered":"2024-09-21T10:01:41.360258+0000","last_unstale":"2024-09-21T10:02:13.086376+0000","last_undegraded":"2024-09-21T10:02:13.086376+0000","last_fullsized":"2024-09-21T10:02:13.086376+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:06.200022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.088916+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:02:14.088916+0000","last_peered":"2024-09-21T10:02:14.088916+0000","last_clean":"2024-09-21T10:02:14.088916+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:02:14.088916+0000","last_undegraded":"2024-09-21T10:02:14.088916+0000","last_fullsized":"2024-09-21T10:02:14.088916+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5983,"num_read_kb":13006,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397662,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307316,"kb_used_data":5792,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93437836,"statfs":{"total":95995035648,"available":95680344064,"internally_reserved":0,"allocated":5931008,"data_stored":5374871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593890,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299044,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446108,"statfs":{"total":95995035648,"available":95688814592,"internally_reserved":0,"allocated":1728512,"data_stored":1180440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790120,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302552,"kb_used_data":5192,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93442600,"statfs":{"total":95995035648,"available":95685222400,"internally_reserved":0,"allocated":5316608,"data_stored":4779887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986349,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298476,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446676,"statfs":{"total":95995035648,"available":95689396224,"internally_reserved":0,"allocated":1138688,"data_stored":590121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":101,"seq":433791696911,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28556,"kb_used_data":1092,"kb_used_omap":5,"kb_used_meta":27450,"kb_avail":93716596,"statfs":{"total":95995035648,"available":95965794304,"internally_reserved":0,"allocated":1118208,"data_stored":585341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":28109584},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860438,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32732,"kb_used_data":5192,"kb_used_omap":8,"kb_used_meta":27511,"kb_avail":93712420,"statfs":{"total":95995035648,"available":95961518080,"internally_reserved":0,"allocated":5316608,"data_stored":4780167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":28172140},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":91,"seq":390842023964,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28616,"kb_used_data":1084,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93716536,"statfs":{"total":95995035648,"available":95965732864,"internally_reserved":0,"allocated":1110016,"data_stored":584648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":28172850},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187499,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29212,"kb_used_data":1676,"kb_used_omap":8,"kb_used_meta":27511,"kb_avail":93715940,"statfs":{"total":95995035648,"available":95965122560,"internally_reserved":0,"allocated":1716224,"data_stored":1176219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":28172273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:03:20.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:03:20.878 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:03:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:03:20] "GET /metrics HTTP/1.1" 200 34032 "" "Prometheus/2.51.0" 2024-09-21T10:03:20.879 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:20 smithi098 ceph-mon[100914]: pgmap v224: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:20.879 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:20 smithi098 ceph-mon[97580]: pgmap v224: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:20.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:20 smithi196 ceph-mon[84784]: pgmap v224: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:21.099 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:21.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:21 smithi098 ceph-mon[97580]: from='client.44831 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:03:21.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:21 smithi098 ceph-mon[100914]: from='client.44831 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:03:21.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:21 smithi196 ceph-mon[84784]: from='client.44831 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:03:22.729 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:22 smithi098 ceph-mon[97580]: pgmap v225: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:22.729 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:22.729 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:22 smithi098 ceph-mon[100914]: pgmap v225: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:22.729 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:22.883 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:03:22.884 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:03:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:22 smithi196 ceph-mon[84784]: pgmap v225: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:23.516 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-09-21T10:03:21.019734+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6289,"num_read_kb":13765,"num_write":1056,"num_write_kb":14251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1132,"ondisk_log_size":1132,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1326504,"kb_used_data":22828,"kb_used_omap":35,"kb_used_meta":1303516,"kb_avail":748634712,"statfs":{"total":767960285184,"available":766601945088,"internally_reserved":0,"allocated":23375872,"data_stored":19051694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36353,"internal_metadata":1334800895},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002815"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.098066+0000","last_change":"2024-09-21T10:02:14.097991+0000","last_active":"2024-09-21T10:02:14.098066+0000","last_peered":"2024-09-21T10:02:14.098066+0000","last_clean":"2024-09-21T10:02:14.098066+0000","last_became_active":"2024-09-21T10:02:14.097407+0000","last_became_peered":"2024-09-21T10:02:14.097407+0000","last_unstale":"2024-09-21T10:02:14.098066+0000","last_undegraded":"2024-09-21T10:02:14.098066+0000","last_fullsized":"2024-09-21T10:02:14.098066+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:18:42.098886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":69,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.085145+0000","last_change":"2024-09-21T10:01:41.360784+0000","last_active":"2024-09-21T10:02:13.085145+0000","last_peered":"2024-09-21T10:02:13.085145+0000","last_clean":"2024-09-21T10:02:13.085145+0000","last_became_active":"2024-09-21T10:01:41.360462+0000","last_became_peered":"2024-09-21T10:01:41.360462+0000","last_unstale":"2024-09-21T10:02:13.085145+0000","last_undegraded":"2024-09-21T10:02:13.085145+0000","last_fullsized":"2024-09-21T10:02:13.085145+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:11.867275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":66,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.103373+0000","last_change":"2024-09-21T10:02:14.103373+0000","last_active":"2024-09-21T10:02:14.103373+0000","last_peered":"2024-09-21T10:02:14.103373+0000","last_clean":"2024-09-21T10:02:14.103373+0000","last_became_active":"2024-09-21T10:02:14.102985+0000","last_became_peered":"2024-09-21T10:02:14.102985+0000","last_unstale":"2024-09-21T10:02:14.103373+0000","last_undegraded":"2024-09-21T10:02:14.103373+0000","last_fullsized":"2024-09-21T10:02:14.103373+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:03.418174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099028+0000","last_change":"2024-09-21T10:02:14.098949+0000","last_active":"2024-09-21T10:02:14.099028+0000","last_peered":"2024-09-21T10:02:14.099028+0000","last_clean":"2024-09-21T10:02:14.099028+0000","last_became_active":"2024-09-21T10:02:14.098646+0000","last_became_peered":"2024-09-21T10:02:14.098646+0000","last_unstale":"2024-09-21T10:02:14.099028+0000","last_undegraded":"2024-09-21T10:02:14.099028+0000","last_fullsized":"2024-09-21T10:02:14.099028+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:50:56.862529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099148+0000","last_change":"2024-09-21T10:02:14.098205+0000","last_active":"2024-09-21T10:02:14.099148+0000","last_peered":"2024-09-21T10:02:14.099148+0000","last_clean":"2024-09-21T10:02:14.099148+0000","last_became_active":"2024-09-21T10:02:14.097911+0000","last_became_peered":"2024-09-21T10:02:14.097911+0000","last_unstale":"2024-09-21T10:02:14.099148+0000","last_undegraded":"2024-09-21T10:02:14.099148+0000","last_fullsized":"2024-09-21T10:02:14.099148+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:55.971191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.085481+0000","last_change":"2024-09-21T09:52:20.371775+0000","last_active":"2024-09-21T10:02:13.085481+0000","last_peered":"2024-09-21T10:02:13.085481+0000","last_clean":"2024-09-21T10:02:13.085481+0000","last_became_active":"2024-09-21T09:52:20.371509+0000","last_became_peered":"2024-09-21T09:52:20.371509+0000","last_unstale":"2024-09-21T10:02:13.085481+0000","last_undegraded":"2024-09-21T10:02:13.085481+0000","last_fullsized":"2024-09-21T10:02:13.085481+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:43:21.822524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097891+0000","last_change":"2024-09-21T10:02:14.097817+0000","last_active":"2024-09-21T10:02:14.097891+0000","last_peered":"2024-09-21T10:02:14.097891+0000","last_clean":"2024-09-21T10:02:14.097891+0000","last_became_active":"2024-09-21T10:02:14.097466+0000","last_became_peered":"2024-09-21T10:02:14.097466+0000","last_unstale":"2024-09-21T10:02:14.097891+0000","last_undegraded":"2024-09-21T10:02:14.097891+0000","last_fullsized":"2024-09-21T10:02:14.097891+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:57.883118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":75,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.090786+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:02:13.090786+0000","last_peered":"2024-09-21T10:02:13.090786+0000","last_clean":"2024-09-21T10:02:13.090786+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:02:13.090786+0000","last_undegraded":"2024-09-21T10:02:13.090786+0000","last_fullsized":"2024-09-21T10:02:13.090786+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":68,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086446+0000","last_change":"2024-09-21T09:59:54.265455+0000","last_active":"2024-09-21T10:02:13.086446+0000","last_peered":"2024-09-21T10:02:13.086446+0000","last_clean":"2024-09-21T10:02:13.086446+0000","last_became_active":"2024-09-21T09:59:54.265110+0000","last_became_peered":"2024-09-21T09:59:54.265110+0000","last_unstale":"2024-09-21T10:02:13.086446+0000","last_undegraded":"2024-09-21T10:02:13.086446+0000","last_fullsized":"2024-09-21T10:02:13.086446+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:04:13.404874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.089734+0000","last_change":"2024-09-21T09:52:20.402753+0000","last_active":"2024-09-21T10:02:13.089734+0000","last_peered":"2024-09-21T10:02:13.089734+0000","last_clean":"2024-09-21T10:02:13.089734+0000","last_became_active":"2024-09-21T09:52:20.402462+0000","last_became_peered":"2024-09-21T09:52:20.402462+0000","last_unstale":"2024-09-21T10:02:13.089734+0000","last_undegraded":"2024-09-21T10:02:13.089734+0000","last_fullsized":"2024-09-21T10:02:13.089734+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:14.036869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":69,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086567+0000","last_change":"2024-09-21T10:01:07.420484+0000","last_active":"2024-09-21T10:02:13.086567+0000","last_peered":"2024-09-21T10:02:13.086567+0000","last_clean":"2024-09-21T10:02:13.086567+0000","last_became_active":"2024-09-21T10:01:07.420135+0000","last_became_peered":"2024-09-21T10:01:07.420135+0000","last_unstale":"2024-09-21T10:02:13.086567+0000","last_undegraded":"2024-09-21T10:02:13.086567+0000","last_fullsized":"2024-09-21T10:02:13.086567+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:33:10.241010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":83,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.098139+0000","last_change":"2024-09-21T10:02:14.098139+0000","last_active":"2024-09-21T10:02:14.098139+0000","last_peered":"2024-09-21T10:02:14.098139+0000","last_clean":"2024-09-21T10:02:14.098139+0000","last_became_active":"2024-09-21T10:02:14.097639+0000","last_became_peered":"2024-09-21T10:02:14.097639+0000","last_unstale":"2024-09-21T10:02:14.098139+0000","last_undegraded":"2024-09-21T10:02:14.098139+0000","last_fullsized":"2024-09-21T10:02:14.098139+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:11:26.925388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087557+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:02:13.087557+0000","last_peered":"2024-09-21T10:02:13.087557+0000","last_clean":"2024-09-21T10:02:13.087557+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:02:13.087557+0000","last_undegraded":"2024-09-21T10:02:13.087557+0000","last_fullsized":"2024-09-21T10:02:13.087557+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":66,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091038+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:02:13.091038+0000","last_peered":"2024-09-21T10:02:13.091038+0000","last_clean":"2024-09-21T10:02:13.091038+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:02:13.091038+0000","last_undegraded":"2024-09-21T10:02:13.091038+0000","last_fullsized":"2024-09-21T10:02:13.091038+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":67,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.104212+0000","last_change":"2024-09-21T10:02:14.104212+0000","last_active":"2024-09-21T10:02:14.104212+0000","last_peered":"2024-09-21T10:02:14.104212+0000","last_clean":"2024-09-21T10:02:14.104212+0000","last_became_active":"2024-09-21T10:02:14.103893+0000","last_became_peered":"2024-09-21T10:02:14.103893+0000","last_unstale":"2024-09-21T10:02:14.104212+0000","last_undegraded":"2024-09-21T10:02:14.104212+0000","last_fullsized":"2024-09-21T10:02:14.104212+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:59:21.191412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.091970+0000","last_change":"2024-09-21T10:01:07.419715+0000","last_active":"2024-09-21T10:02:14.091970+0000","last_peered":"2024-09-21T10:02:14.091970+0000","last_clean":"2024-09-21T10:02:14.091970+0000","last_became_active":"2024-09-21T10:01:07.419422+0000","last_became_peered":"2024-09-21T10:01:07.419422+0000","last_unstale":"2024-09-21T10:02:14.091970+0000","last_undegraded":"2024-09-21T10:02:14.091970+0000","last_fullsized":"2024-09-21T10:02:14.091970+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:36.424289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":67,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.096638+0000","last_change":"2024-09-21T10:02:14.096537+0000","last_active":"2024-09-21T10:02:14.096638+0000","last_peered":"2024-09-21T10:02:14.096638+0000","last_clean":"2024-09-21T10:02:14.096638+0000","last_became_active":"2024-09-21T10:02:14.096166+0000","last_became_peered":"2024-09-21T10:02:14.096166+0000","last_unstale":"2024-09-21T10:02:14.096638+0000","last_undegraded":"2024-09-21T10:02:14.096638+0000","last_fullsized":"2024-09-21T10:02:14.096638+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:54:18.927732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":66,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099205+0000","last_change":"2024-09-21T10:02:14.099205+0000","last_active":"2024-09-21T10:02:14.099205+0000","last_peered":"2024-09-21T10:02:14.099205+0000","last_clean":"2024-09-21T10:02:14.099205+0000","last_became_active":"2024-09-21T10:02:14.098598+0000","last_became_peered":"2024-09-21T10:02:14.098598+0000","last_unstale":"2024-09-21T10:02:14.099205+0000","last_undegraded":"2024-09-21T10:02:14.099205+0000","last_fullsized":"2024-09-21T10:02:14.099205+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:46.424612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":67,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091165+0000","last_change":"2024-09-21T10:01:07.424883+0000","last_active":"2024-09-21T10:02:13.091165+0000","last_peered":"2024-09-21T10:02:13.091165+0000","last_clean":"2024-09-21T10:02:13.091165+0000","last_became_active":"2024-09-21T10:01:07.424499+0000","last_became_peered":"2024-09-21T10:01:07.424499+0000","last_unstale":"2024-09-21T10:02:13.091165+0000","last_undegraded":"2024-09-21T10:02:13.091165+0000","last_fullsized":"2024-09-21T10:02:13.091165+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:46.101570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":83,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087300+0000","last_change":"2024-09-21T10:01:41.362291+0000","last_active":"2024-09-21T10:02:13.087300+0000","last_peered":"2024-09-21T10:02:13.087300+0000","last_clean":"2024-09-21T10:02:13.087300+0000","last_became_active":"2024-09-21T10:01:41.361954+0000","last_became_peered":"2024-09-21T10:01:41.361954+0000","last_unstale":"2024-09-21T10:02:13.087300+0000","last_undegraded":"2024-09-21T10:02:13.087300+0000","last_fullsized":"2024-09-21T10:02:13.087300+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:57.669671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":67,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087361+0000","last_change":"2024-09-21T10:01:41.361244+0000","last_active":"2024-09-21T10:02:13.087361+0000","last_peered":"2024-09-21T10:02:13.087361+0000","last_clean":"2024-09-21T10:02:13.087361+0000","last_became_active":"2024-09-21T10:01:41.360645+0000","last_became_peered":"2024-09-21T10:01:41.360645+0000","last_unstale":"2024-09-21T10:02:13.087361+0000","last_undegraded":"2024-09-21T10:02:13.087361+0000","last_fullsized":"2024-09-21T10:02:13.087361+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:07.604991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":66,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086023+0000","last_change":"2024-09-21T10:01:07.420824+0000","last_active":"2024-09-21T10:02:13.086023+0000","last_peered":"2024-09-21T10:02:13.086023+0000","last_clean":"2024-09-21T10:02:13.086023+0000","last_became_active":"2024-09-21T10:01:07.420387+0000","last_became_peered":"2024-09-21T10:01:07.420387+0000","last_unstale":"2024-09-21T10:02:13.086023+0000","last_undegraded":"2024-09-21T10:02:13.086023+0000","last_fullsized":"2024-09-21T10:02:13.086023+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:48.875842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":70,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097606+0000","last_change":"2024-09-21T10:02:14.097462+0000","last_active":"2024-09-21T10:02:14.097606+0000","last_peered":"2024-09-21T10:02:14.097606+0000","last_clean":"2024-09-21T10:02:14.097606+0000","last_became_active":"2024-09-21T10:02:14.097167+0000","last_became_peered":"2024-09-21T10:02:14.097167+0000","last_unstale":"2024-09-21T10:02:14.097606+0000","last_undegraded":"2024-09-21T10:02:14.097606+0000","last_fullsized":"2024-09-21T10:02:14.097606+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:54.922731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":55,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.098313+0000","last_change":"2024-09-21T10:02:14.097676+0000","last_active":"2024-09-21T10:02:14.098313+0000","last_peered":"2024-09-21T10:02:14.098313+0000","last_clean":"2024-09-21T10:02:14.098313+0000","last_became_active":"2024-09-21T10:02:14.097391+0000","last_became_peered":"2024-09-21T10:02:14.097391+0000","last_unstale":"2024-09-21T10:02:14.098313+0000","last_undegraded":"2024-09-21T10:02:14.098313+0000","last_fullsized":"2024-09-21T10:02:14.098313+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:40:24.523178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":184,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087727+0000","last_change":"2024-09-21T09:59:54.265556+0000","last_active":"2024-09-21T10:02:13.087727+0000","last_peered":"2024-09-21T10:02:13.087727+0000","last_clean":"2024-09-21T10:02:13.087727+0000","last_became_active":"2024-09-21T09:59:54.265033+0000","last_became_peered":"2024-09-21T09:59:54.265033+0000","last_unstale":"2024-09-21T10:02:13.087727+0000","last_undegraded":"2024-09-21T10:02:13.087727+0000","last_fullsized":"2024-09-21T10:02:13.087727+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T09:48:34.850872+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:40.449756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":491,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.089982+0000","last_change":"2024-09-21T10:01:41.361089+0000","last_active":"2024-09-21T10:02:13.089982+0000","last_peered":"2024-09-21T10:02:13.089982+0000","last_clean":"2024-09-21T10:02:13.089982+0000","last_became_active":"2024-09-21T10:01:41.360658+0000","last_became_peered":"2024-09-21T10:01:41.360658+0000","last_unstale":"2024-09-21T10:02:13.089982+0000","last_undegraded":"2024-09-21T10:02:13.089982+0000","last_fullsized":"2024-09-21T10:02:13.089982+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:58:06.840024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":482,"num_read_kb":1581,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.100041+0000","last_change":"2024-09-21T10:02:14.099961+0000","last_active":"2024-09-21T10:02:14.100041+0000","last_peered":"2024-09-21T10:02:14.100041+0000","last_clean":"2024-09-21T10:02:14.100041+0000","last_became_active":"2024-09-21T10:02:14.099067+0000","last_became_peered":"2024-09-21T10:02:14.099067+0000","last_unstale":"2024-09-21T10:02:14.100041+0000","last_undegraded":"2024-09-21T10:02:14.100041+0000","last_fullsized":"2024-09-21T10:02:14.100041+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:41.037532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091668+0000","last_change":"2024-09-21T09:52:20.403534+0000","last_active":"2024-09-21T10:02:13.091668+0000","last_peered":"2024-09-21T10:02:13.091668+0000","last_clean":"2024-09-21T10:02:13.091668+0000","last_became_active":"2024-09-21T09:52:20.403258+0000","last_became_peered":"2024-09-21T09:52:20.403258+0000","last_unstale":"2024-09-21T10:02:13.091668+0000","last_undegraded":"2024-09-21T10:02:13.091668+0000","last_fullsized":"2024-09-21T10:02:13.091668+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:55.374343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":447,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087637+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:02:13.087637+0000","last_peered":"2024-09-21T10:02:13.087637+0000","last_clean":"2024-09-21T10:02:13.087637+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:02:13.087637+0000","last_undegraded":"2024-09-21T10:02:13.087637+0000","last_fullsized":"2024-09-21T10:02:13.087637+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097180+0000","last_change":"2024-09-21T10:02:14.097097+0000","last_active":"2024-09-21T10:02:14.097180+0000","last_peered":"2024-09-21T10:02:14.097180+0000","last_clean":"2024-09-21T10:02:14.097180+0000","last_became_active":"2024-09-21T10:02:14.096788+0000","last_became_peered":"2024-09-21T10:02:14.096788+0000","last_unstale":"2024-09-21T10:02:14.097180+0000","last_undegraded":"2024-09-21T10:02:14.097180+0000","last_fullsized":"2024-09-21T10:02:14.097180+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:05:29.711020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":76,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090168+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:02:14.090168+0000","last_peered":"2024-09-21T10:02:14.090168+0000","last_clean":"2024-09-21T10:02:14.090168+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:02:14.090168+0000","last_undegraded":"2024-09-21T10:02:14.090168+0000","last_fullsized":"2024-09-21T10:02:14.090168+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":815,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.100244+0000","last_change":"2024-09-21T10:02:14.100143+0000","last_active":"2024-09-21T10:02:14.100244+0000","last_peered":"2024-09-21T10:02:14.100244+0000","last_clean":"2024-09-21T10:02:14.100244+0000","last_became_active":"2024-09-21T10:02:14.099586+0000","last_became_peered":"2024-09-21T10:02:14.099586+0000","last_unstale":"2024-09-21T10:02:14.100244+0000","last_undegraded":"2024-09-21T10:02:14.100244+0000","last_fullsized":"2024-09-21T10:02:14.100244+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:34.064652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":55,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090600+0000","last_change":"2024-09-21T10:01:41.358268+0000","last_active":"2024-09-21T10:02:14.090600+0000","last_peered":"2024-09-21T10:02:14.090600+0000","last_clean":"2024-09-21T10:02:14.090600+0000","last_became_active":"2024-09-21T10:01:41.357816+0000","last_became_peered":"2024-09-21T10:01:41.357816+0000","last_unstale":"2024-09-21T10:02:14.090600+0000","last_undegraded":"2024-09-21T10:02:14.090600+0000","last_fullsized":"2024-09-21T10:02:14.090600+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:59:44.261636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":72,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.096377+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:02:14.096377+0000","last_peered":"2024-09-21T10:02:14.096377+0000","last_clean":"2024-09-21T10:02:14.096377+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:02:14.096377+0000","last_undegraded":"2024-09-21T10:02:14.096377+0000","last_fullsized":"2024-09-21T10:02:14.096377+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":675,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.090133+0000","last_change":"2024-09-21T10:01:07.424819+0000","last_active":"2024-09-21T10:02:13.090133+0000","last_peered":"2024-09-21T10:02:13.090133+0000","last_clean":"2024-09-21T10:02:13.090133+0000","last_became_active":"2024-09-21T10:01:07.424392+0000","last_became_peered":"2024-09-21T10:01:07.424392+0000","last_unstale":"2024-09-21T10:02:13.090133+0000","last_undegraded":"2024-09-21T10:02:13.090133+0000","last_fullsized":"2024-09-21T10:02:13.090133+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:27.266390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":67,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.088170+0000","last_change":"2024-09-21T09:59:54.267742+0000","last_active":"2024-09-21T10:02:13.088170+0000","last_peered":"2024-09-21T10:02:13.088170+0000","last_clean":"2024-09-21T10:02:13.088170+0000","last_became_active":"2024-09-21T09:59:54.267406+0000","last_became_peered":"2024-09-21T09:59:54.267406+0000","last_unstale":"2024-09-21T10:02:13.088170+0000","last_undegraded":"2024-09-21T10:02:13.088170+0000","last_fullsized":"2024-09-21T10:02:13.088170+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:25:35.968779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":80,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.103531+0000","last_change":"2024-09-21T10:02:14.103531+0000","last_active":"2024-09-21T10:02:14.103531+0000","last_peered":"2024-09-21T10:02:14.103531+0000","last_clean":"2024-09-21T10:02:14.103531+0000","last_became_active":"2024-09-21T10:02:14.103218+0000","last_became_peered":"2024-09-21T10:02:14.103218+0000","last_unstale":"2024-09-21T10:02:14.103531+0000","last_undegraded":"2024-09-21T10:02:14.103531+0000","last_fullsized":"2024-09-21T10:02:14.103531+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:40.014308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":84,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.103145+0000","last_change":"2024-09-21T10:02:14.103145+0000","last_active":"2024-09-21T10:02:14.103145+0000","last_peered":"2024-09-21T10:02:14.103145+0000","last_clean":"2024-09-21T10:02:14.103145+0000","last_became_active":"2024-09-21T10:02:14.102806+0000","last_became_peered":"2024-09-21T10:02:14.102806+0000","last_unstale":"2024-09-21T10:02:14.103145+0000","last_undegraded":"2024-09-21T10:02:14.103145+0000","last_fullsized":"2024-09-21T10:02:14.103145+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:24.348745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":760,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090783+0000","last_change":"2024-09-21T10:01:07.426065+0000","last_active":"2024-09-21T10:02:14.090783+0000","last_peered":"2024-09-21T10:02:14.090783+0000","last_clean":"2024-09-21T10:02:14.090783+0000","last_became_active":"2024-09-21T10:01:07.423163+0000","last_became_peered":"2024-09-21T10:01:07.423163+0000","last_unstale":"2024-09-21T10:02:14.090783+0000","last_undegraded":"2024-09-21T10:02:14.090783+0000","last_fullsized":"2024-09-21T10:02:14.090783+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:43.564978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":44,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090907+0000","last_change":"2024-09-21T10:01:07.422175+0000","last_active":"2024-09-21T10:02:14.090907+0000","last_peered":"2024-09-21T10:02:14.090907+0000","last_clean":"2024-09-21T10:02:14.090907+0000","last_became_active":"2024-09-21T10:01:07.421844+0000","last_became_peered":"2024-09-21T10:01:07.421844+0000","last_unstale":"2024-09-21T10:02:14.090907+0000","last_undegraded":"2024-09-21T10:02:14.090907+0000","last_fullsized":"2024-09-21T10:02:14.090907+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:17:15.192573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":55,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090704+0000","last_change":"2024-09-21T10:01:41.360338+0000","last_active":"2024-09-21T10:02:14.090704+0000","last_peered":"2024-09-21T10:02:14.090704+0000","last_clean":"2024-09-21T10:02:14.090704+0000","last_became_active":"2024-09-21T10:01:41.360050+0000","last_became_peered":"2024-09-21T10:01:41.360050+0000","last_unstale":"2024-09-21T10:02:14.090704+0000","last_undegraded":"2024-09-21T10:02:14.090704+0000","last_fullsized":"2024-09-21T10:02:14.090704+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:39:30.459393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":413,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087649+0000","last_change":"2024-09-21T09:59:54.265559+0000","last_active":"2024-09-21T10:02:13.087649+0000","last_peered":"2024-09-21T10:02:13.087649+0000","last_clean":"2024-09-21T10:02:13.087649+0000","last_became_active":"2024-09-21T09:59:54.264884+0000","last_became_peered":"2024-09-21T09:59:54.264884+0000","last_unstale":"2024-09-21T10:02:13.087649+0000","last_undegraded":"2024-09-21T10:02:13.087649+0000","last_fullsized":"2024-09-21T10:02:13.087649+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:24.544845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":54,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.091931+0000","last_change":"2024-09-21T09:59:54.267798+0000","last_active":"2024-09-21T10:02:14.091931+0000","last_peered":"2024-09-21T10:02:14.091931+0000","last_clean":"2024-09-21T10:02:14.091931+0000","last_became_active":"2024-09-21T09:59:54.265615+0000","last_became_peered":"2024-09-21T09:59:54.265615+0000","last_unstale":"2024-09-21T10:02:14.091931+0000","last_undegraded":"2024-09-21T10:02:14.091931+0000","last_fullsized":"2024-09-21T10:02:14.091931+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:30.363872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":67,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.102034+0000","last_change":"2024-09-21T10:02:14.101943+0000","last_active":"2024-09-21T10:02:14.102034+0000","last_peered":"2024-09-21T10:02:14.102034+0000","last_clean":"2024-09-21T10:02:14.102034+0000","last_became_active":"2024-09-21T10:02:14.101488+0000","last_became_peered":"2024-09-21T10:02:14.101488+0000","last_unstale":"2024-09-21T10:02:14.102034+0000","last_undegraded":"2024-09-21T10:02:14.102034+0000","last_fullsized":"2024-09-21T10:02:14.102034+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:52:27.866414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":424,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.089522+0000","last_change":"2024-09-21T10:01:07.425330+0000","last_active":"2024-09-21T10:02:14.089522+0000","last_peered":"2024-09-21T10:02:14.089522+0000","last_clean":"2024-09-21T10:02:14.089522+0000","last_became_active":"2024-09-21T10:01:07.424695+0000","last_became_peered":"2024-09-21T10:01:07.424695+0000","last_unstale":"2024-09-21T10:02:14.089522+0000","last_undegraded":"2024-09-21T10:02:14.089522+0000","last_fullsized":"2024-09-21T10:02:14.089522+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:13:22.297576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":64,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.096024+0000","last_change":"2024-09-21T10:02:14.095924+0000","last_active":"2024-09-21T10:02:14.096024+0000","last_peered":"2024-09-21T10:02:14.096024+0000","last_clean":"2024-09-21T10:02:14.096024+0000","last_became_active":"2024-09-21T10:02:14.095553+0000","last_became_peered":"2024-09-21T10:02:14.095553+0000","last_unstale":"2024-09-21T10:02:14.096024+0000","last_undegraded":"2024-09-21T10:02:14.096024+0000","last_fullsized":"2024-09-21T10:02:14.096024+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:08.155980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.089383+0000","last_change":"2024-09-21T10:01:07.426736+0000","last_active":"2024-09-21T10:02:14.089383+0000","last_peered":"2024-09-21T10:02:14.089383+0000","last_clean":"2024-09-21T10:02:14.089383+0000","last_became_active":"2024-09-21T10:01:07.426416+0000","last_became_peered":"2024-09-21T10:01:07.426416+0000","last_unstale":"2024-09-21T10:02:14.089383+0000","last_undegraded":"2024-09-21T10:02:14.089383+0000","last_fullsized":"2024-09-21T10:02:14.089383+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:59.784757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":329,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091262+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:02:13.091262+0000","last_peered":"2024-09-21T10:02:13.091262+0000","last_clean":"2024-09-21T10:02:13.091262+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:02:13.091262+0000","last_undegraded":"2024-09-21T10:02:13.091262+0000","last_fullsized":"2024-09-21T10:02:13.091262+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086571+0000","last_change":"2024-09-21T09:52:20.391439+0000","last_active":"2024-09-21T10:02:13.086571+0000","last_peered":"2024-09-21T10:02:13.086571+0000","last_clean":"2024-09-21T10:02:13.086571+0000","last_became_active":"2024-09-21T09:52:20.391157+0000","last_became_peered":"2024-09-21T09:52:20.391157+0000","last_unstale":"2024-09-21T10:02:13.086571+0000","last_undegraded":"2024-09-21T10:02:13.086571+0000","last_fullsized":"2024-09-21T10:02:13.086571+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:36.788330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":66,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.088062+0000","last_change":"2024-09-21T10:01:07.421150+0000","last_active":"2024-09-21T10:02:13.088062+0000","last_peered":"2024-09-21T10:02:13.088062+0000","last_clean":"2024-09-21T10:02:13.088062+0000","last_became_active":"2024-09-21T10:01:07.420830+0000","last_became_peered":"2024-09-21T10:01:07.420830+0000","last_unstale":"2024-09-21T10:02:13.088062+0000","last_undegraded":"2024-09-21T10:02:13.088062+0000","last_fullsized":"2024-09-21T10:02:13.088062+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:15:34.472626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":72,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.087220+0000","last_change":"2024-09-21T09:52:20.371668+0000","last_active":"2024-09-21T10:02:13.087220+0000","last_peered":"2024-09-21T10:02:13.087220+0000","last_clean":"2024-09-21T10:02:13.087220+0000","last_became_active":"2024-09-21T09:52:20.371389+0000","last_became_peered":"2024-09-21T09:52:20.371389+0000","last_unstale":"2024-09-21T10:02:13.087220+0000","last_undegraded":"2024-09-21T10:02:13.087220+0000","last_fullsized":"2024-09-21T10:02:13.087220+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:10.013020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":54,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.090703+0000","last_change":"2024-09-21T10:01:41.360337+0000","last_active":"2024-09-21T10:02:14.090703+0000","last_peered":"2024-09-21T10:02:14.090703+0000","last_clean":"2024-09-21T10:02:14.090703+0000","last_became_active":"2024-09-21T10:01:41.359996+0000","last_became_peered":"2024-09-21T10:01:41.359996+0000","last_unstale":"2024-09-21T10:02:14.090703+0000","last_undegraded":"2024-09-21T10:02:14.090703+0000","last_fullsized":"2024-09-21T10:02:14.090703+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:28:43.820241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":70,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.102896+0000","last_change":"2024-09-21T10:02:14.102896+0000","last_active":"2024-09-21T10:02:14.102896+0000","last_peered":"2024-09-21T10:02:14.102896+0000","last_clean":"2024-09-21T10:02:14.102896+0000","last_became_active":"2024-09-21T10:02:14.102493+0000","last_became_peered":"2024-09-21T10:02:14.102493+0000","last_unstale":"2024-09-21T10:02:14.102896+0000","last_undegraded":"2024-09-21T10:02:14.102896+0000","last_fullsized":"2024-09-21T10:02:14.102896+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:10:43.572005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":67,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.089930+0000","last_change":"2024-09-21T10:01:07.424062+0000","last_active":"2024-09-21T10:02:13.089930+0000","last_peered":"2024-09-21T10:02:13.089930+0000","last_clean":"2024-09-21T10:02:13.089930+0000","last_became_active":"2024-09-21T10:01:07.423629+0000","last_became_peered":"2024-09-21T10:01:07.423629+0000","last_unstale":"2024-09-21T10:02:13.089930+0000","last_undegraded":"2024-09-21T10:02:13.089930+0000","last_fullsized":"2024-09-21T10:02:13.089930+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:24.119642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.100529+0000","last_change":"2024-09-21T10:02:14.100451+0000","last_active":"2024-09-21T10:02:14.100529+0000","last_peered":"2024-09-21T10:02:14.100529+0000","last_clean":"2024-09-21T10:02:14.100529+0000","last_became_active":"2024-09-21T10:02:14.100160+0000","last_became_peered":"2024-09-21T10:02:14.100160+0000","last_unstale":"2024-09-21T10:02:14.100529+0000","last_undegraded":"2024-09-21T10:02:14.100529+0000","last_fullsized":"2024-09-21T10:02:14.100529+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:28.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.096493+0000","last_change":"2024-09-21T10:02:14.096098+0000","last_active":"2024-09-21T10:02:14.096493+0000","last_peered":"2024-09-21T10:02:14.096493+0000","last_clean":"2024-09-21T10:02:14.096493+0000","last_became_active":"2024-09-21T10:02:14.095811+0000","last_became_peered":"2024-09-21T10:02:14.095811+0000","last_unstale":"2024-09-21T10:02:14.096493+0000","last_undegraded":"2024-09-21T10:02:14.096493+0000","last_fullsized":"2024-09-21T10:02:14.096493+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:54.619333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.100791+0000","last_change":"2024-09-21T10:02:14.100712+0000","last_active":"2024-09-21T10:02:14.100791+0000","last_peered":"2024-09-21T10:02:14.100791+0000","last_clean":"2024-09-21T10:02:14.100791+0000","last_became_active":"2024-09-21T10:02:14.100427+0000","last_became_peered":"2024-09-21T10:02:14.100427+0000","last_unstale":"2024-09-21T10:02:14.100791+0000","last_undegraded":"2024-09-21T10:02:14.100791+0000","last_fullsized":"2024-09-21T10:02:14.100791+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:16:42.350642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":66,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091136+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:02:13.091136+0000","last_peered":"2024-09-21T10:02:13.091136+0000","last_clean":"2024-09-21T10:02:13.091136+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:02:13.091136+0000","last_undegraded":"2024-09-21T10:02:13.091136+0000","last_fullsized":"2024-09-21T10:02:13.091136+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":87,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097446+0000","last_change":"2024-09-21T10:02:14.097369+0000","last_active":"2024-09-21T10:02:14.097446+0000","last_peered":"2024-09-21T10:02:14.097446+0000","last_clean":"2024-09-21T10:02:14.097446+0000","last_became_active":"2024-09-21T10:02:14.097067+0000","last_became_peered":"2024-09-21T10:02:14.097067+0000","last_unstale":"2024-09-21T10:02:14.097446+0000","last_undegraded":"2024-09-21T10:02:14.097446+0000","last_fullsized":"2024-09-21T10:02:14.097446+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:50:48.123813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":65,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099046+0000","last_change":"2024-09-21T10:02:14.098970+0000","last_active":"2024-09-21T10:02:14.099046+0000","last_peered":"2024-09-21T10:02:14.099046+0000","last_clean":"2024-09-21T10:02:14.099046+0000","last_became_active":"2024-09-21T10:02:14.098641+0000","last_became_peered":"2024-09-21T10:02:14.098641+0000","last_unstale":"2024-09-21T10:02:14.099046+0000","last_undegraded":"2024-09-21T10:02:14.099046+0000","last_fullsized":"2024-09-21T10:02:14.099046+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:25.215602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":54,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.097100+0000","last_change":"2024-09-21T10:02:14.097021+0000","last_active":"2024-09-21T10:02:14.097100+0000","last_peered":"2024-09-21T10:02:14.097100+0000","last_clean":"2024-09-21T10:02:14.097100+0000","last_became_active":"2024-09-21T10:02:14.096741+0000","last_became_peered":"2024-09-21T10:02:14.096741+0000","last_unstale":"2024-09-21T10:02:14.097100+0000","last_undegraded":"2024-09-21T10:02:14.097100+0000","last_fullsized":"2024-09-21T10:02:14.097100+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:18:20.631150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099544+0000","last_change":"2024-09-21T10:02:14.099460+0000","last_active":"2024-09-21T10:02:14.099544+0000","last_peered":"2024-09-21T10:02:14.099544+0000","last_clean":"2024-09-21T10:02:14.099544+0000","last_became_active":"2024-09-21T10:02:14.099159+0000","last_became_peered":"2024-09-21T10:02:14.099159+0000","last_unstale":"2024-09-21T10:02:14.099544+0000","last_undegraded":"2024-09-21T10:02:14.099544+0000","last_fullsized":"2024-09-21T10:02:14.099544+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:23:47.599107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":85,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086660+0000","last_change":"2024-09-21T10:01:41.360480+0000","last_active":"2024-09-21T10:02:13.086660+0000","last_peered":"2024-09-21T10:02:13.086660+0000","last_clean":"2024-09-21T10:02:13.086660+0000","last_became_active":"2024-09-21T10:01:41.359791+0000","last_became_peered":"2024-09-21T10:01:41.359791+0000","last_unstale":"2024-09-21T10:02:13.086660+0000","last_undegraded":"2024-09-21T10:02:13.086660+0000","last_fullsized":"2024-09-21T10:02:13.086660+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:42.812974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.102901+0000","last_change":"2024-09-21T10:02:14.102826+0000","last_active":"2024-09-21T10:02:14.102901+0000","last_peered":"2024-09-21T10:02:14.102901+0000","last_clean":"2024-09-21T10:02:14.102901+0000","last_became_active":"2024-09-21T10:02:14.102499+0000","last_became_peered":"2024-09-21T10:02:14.102499+0000","last_unstale":"2024-09-21T10:02:14.102901+0000","last_undegraded":"2024-09-21T10:02:14.102901+0000","last_fullsized":"2024-09-21T10:02:14.102901+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:30.924687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":58,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.091881+0000","last_change":"2024-09-21T09:59:54.268378+0000","last_active":"2024-09-21T10:02:14.091881+0000","last_peered":"2024-09-21T10:02:14.091881+0000","last_clean":"2024-09-21T10:02:14.091881+0000","last_became_active":"2024-09-21T09:59:54.267187+0000","last_became_peered":"2024-09-21T09:59:54.267187+0000","last_unstale":"2024-09-21T10:02:14.091881+0000","last_undegraded":"2024-09-21T10:02:14.091881+0000","last_fullsized":"2024-09-21T10:02:14.091881+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:12:06.054483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":83,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.100541+0000","last_change":"2024-09-21T10:02:14.100541+0000","last_active":"2024-09-21T10:02:14.100541+0000","last_peered":"2024-09-21T10:02:14.100541+0000","last_clean":"2024-09-21T10:02:14.100541+0000","last_became_active":"2024-09-21T10:02:14.100327+0000","last_became_peered":"2024-09-21T10:02:14.100327+0000","last_unstale":"2024-09-21T10:02:14.100541+0000","last_undegraded":"2024-09-21T10:02:14.100541+0000","last_fullsized":"2024-09-21T10:02:14.100541+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:34:34.304188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":69,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.090359+0000","last_change":"2024-09-21T10:01:07.424248+0000","last_active":"2024-09-21T10:02:13.090359+0000","last_peered":"2024-09-21T10:02:13.090359+0000","last_clean":"2024-09-21T10:02:13.090359+0000","last_became_active":"2024-09-21T10:01:07.423738+0000","last_became_peered":"2024-09-21T10:01:07.423738+0000","last_unstale":"2024-09-21T10:02:13.090359+0000","last_undegraded":"2024-09-21T10:02:13.090359+0000","last_fullsized":"2024-09-21T10:02:13.090359+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:44:35.663555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":83,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.096583+0000","last_change":"2024-09-21T10:02:14.096583+0000","last_active":"2024-09-21T10:02:14.096583+0000","last_peered":"2024-09-21T10:02:14.096583+0000","last_clean":"2024-09-21T10:02:14.096583+0000","last_became_active":"2024-09-21T10:02:14.096196+0000","last_became_peered":"2024-09-21T10:02:14.096196+0000","last_unstale":"2024-09-21T10:02:14.096583+0000","last_undegraded":"2024-09-21T10:02:14.096583+0000","last_fullsized":"2024-09-21T10:02:14.096583+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:07:01.888435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.091542+0000","last_change":"2024-09-21T09:52:20.371638+0000","last_active":"2024-09-21T10:02:13.091542+0000","last_peered":"2024-09-21T10:02:13.091542+0000","last_clean":"2024-09-21T10:02:13.091542+0000","last_became_active":"2024-09-21T09:52:20.371383+0000","last_became_peered":"2024-09-21T09:52:20.371383+0000","last_unstale":"2024-09-21T10:02:13.091542+0000","last_undegraded":"2024-09-21T10:02:13.091542+0000","last_fullsized":"2024-09-21T10:02:13.091542+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:33:45.384397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099880+0000","last_change":"2024-09-21T10:02:14.099799+0000","last_active":"2024-09-21T10:02:14.099880+0000","last_peered":"2024-09-21T10:02:14.099880+0000","last_clean":"2024-09-21T10:02:14.099880+0000","last_became_active":"2024-09-21T10:02:14.099375+0000","last_became_peered":"2024-09-21T10:02:14.099375+0000","last_unstale":"2024-09-21T10:02:14.099880+0000","last_undegraded":"2024-09-21T10:02:14.099880+0000","last_fullsized":"2024-09-21T10:02:14.099880+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:13.407638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":75,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.099528+0000","last_change":"2024-09-21T10:02:14.099439+0000","last_active":"2024-09-21T10:02:14.099528+0000","last_peered":"2024-09-21T10:02:14.099528+0000","last_clean":"2024-09-21T10:02:14.099528+0000","last_became_active":"2024-09-21T10:02:14.099117+0000","last_became_peered":"2024-09-21T10:02:14.099117+0000","last_unstale":"2024-09-21T10:02:14.099528+0000","last_undegraded":"2024-09-21T10:02:14.099528+0000","last_fullsized":"2024-09-21T10:02:14.099528+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:39.681817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":76,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.086376+0000","last_change":"2024-09-21T10:01:41.360674+0000","last_active":"2024-09-21T10:02:13.086376+0000","last_peered":"2024-09-21T10:02:13.086376+0000","last_clean":"2024-09-21T10:02:13.086376+0000","last_became_active":"2024-09-21T10:01:41.360258+0000","last_became_peered":"2024-09-21T10:01:41.360258+0000","last_unstale":"2024-09-21T10:02:13.086376+0000","last_undegraded":"2024-09-21T10:02:13.086376+0000","last_fullsized":"2024-09-21T10:02:13.086376+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:06.200022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":69,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T10:02:14.088916+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:02:14.088916+0000","last_peered":"2024-09-21T10:02:14.088916+0000","last_clean":"2024-09-21T10:02:14.088916+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:02:14.088916+0000","last_undegraded":"2024-09-21T10:02:14.088916+0000","last_fullsized":"2024-09-21T10:02:14.088916+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5983,"num_read_kb":13006,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397662,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307316,"kb_used_data":5792,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93437836,"statfs":{"total":95995035648,"available":95680344064,"internally_reserved":0,"allocated":5931008,"data_stored":5374871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593890,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299044,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446108,"statfs":{"total":95995035648,"available":95688814592,"internally_reserved":0,"allocated":1728512,"data_stored":1180440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790120,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302552,"kb_used_data":5192,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93442600,"statfs":{"total":95995035648,"available":95685222400,"internally_reserved":0,"allocated":5316608,"data_stored":4779887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986349,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298476,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446676,"statfs":{"total":95995035648,"available":95689396224,"internally_reserved":0,"allocated":1138688,"data_stored":590121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":101,"seq":433791696911,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28556,"kb_used_data":1092,"kb_used_omap":5,"kb_used_meta":27450,"kb_avail":93716596,"statfs":{"total":95995035648,"available":95965794304,"internally_reserved":0,"allocated":1118208,"data_stored":585341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":28109584},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860438,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32732,"kb_used_data":5192,"kb_used_omap":8,"kb_used_meta":27511,"kb_avail":93712420,"statfs":{"total":95995035648,"available":95961518080,"internally_reserved":0,"allocated":5316608,"data_stored":4780167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":28172140},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":91,"seq":390842023965,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28616,"kb_used_data":1084,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93716536,"statfs":{"total":95995035648,"available":95965732864,"internally_reserved":0,"allocated":1110016,"data_stored":584648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":28172850},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187499,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29212,"kb_used_data":1676,"kb_used_omap":8,"kb_used_meta":27511,"kb_avail":93715940,"statfs":{"total":95995035648,"available":95965122560,"internally_reserved":0,"allocated":1716224,"data_stored":1176219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":28172273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:03:23.519 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T10:03:23.519 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd dump 2024-09-21T10:03:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:23.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:23 smithi098 ceph-mon[97580]: from='client.34968 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:03:23.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:23 smithi098 ceph-mon[100914]: from='client.34968 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:03:23.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:23 smithi196 ceph-mon[84784]: from='client.34968 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:03:24.874 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:24 smithi098 ceph-mon[97580]: pgmap v226: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:24.874 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:24 smithi098 ceph-mon[100914]: pgmap v226: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:24 smithi196 ceph-mon[84784]: pgmap v226: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:25.574 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 102 2024-09-21T10:03:25.575 INFO:teuthology.orchestra.run.smithi098.stdout:fsid 26a15512-77fe-11ef-baf6-efdc52797490 2024-09-21T10:03:25.575 INFO:teuthology.orchestra.run.smithi098.stdout:created 2024-09-21T09:45:33.813487+0000 2024-09-21T10:03:25.575 INFO:teuthology.orchestra.run.smithi098.stdout:modified 2024-09-21T10:02:14.082491+0000 2024-09-21T10:03:25.575 INFO:teuthology.orchestra.run.smithi098.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-21T10:03:25.575 INFO:teuthology.orchestra.run.smithi098.stdout:crush_version 22 2024-09-21T10:03:25.575 INFO:teuthology.orchestra.run.smithi098.stdout:full_ratio 0.95 2024-09-21T10:03:25.575 INFO:teuthology.orchestra.run.smithi098.stdout:backfillfull_ratio 0.9 2024-09-21T10:03:25.575 INFO:teuthology.orchestra.run.smithi098.stdout:nearfull_ratio 0.85 2024-09-21T10:03:25.575 INFO:teuthology.orchestra.run.smithi098.stdout:require_min_compat_client luminous 2024-09-21T10:03:25.575 INFO:teuthology.orchestra.run.smithi098.stdout:min_compat_client luminous 2024-09-21T10:03:25.575 INFO:teuthology.orchestra.run.smithi098.stdout:require_osd_release quincy 2024-09-21T10:03:25.575 INFO:teuthology.orchestra.run.smithi098.stdout:stretch_mode_enabled false 2024-09-21T10:03:25.576 INFO:teuthology.orchestra.run.smithi098.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-09-21T10:03:25.576 INFO:teuthology.orchestra.run.smithi098.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 78 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.00 2024-09-21T10:03:25.576 INFO:teuthology.orchestra.run.smithi098.stdout:pool 3 'cephfs.foo.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 63 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.50 2024-09-21T10:03:25.576 INFO:teuthology.orchestra.run.smithi098.stdout:pool 4 'cephfs.foo.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 64 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 2.25 2024-09-21T10:03:25.576 INFO:teuthology.orchestra.run.smithi098.stdout:max_osd 8 2024-09-21T10:03:25.576 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 up in weight 1 up_from 86 up_thru 101 down_at 83 last_clean_interval [8,82) [v2:172.21.15.98:6802/1869651717,v1:172.21.15.98:6803/1869651717] [v2:172.21.15.98:6804/1869651717,v1:172.21.15.98:6805/1869651717] exists,up 579f13de-565d-4801-b65e-f27d1cde297a 2024-09-21T10:03:25.576 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 up in weight 1 up_from 91 up_thru 101 down_at 88 last_clean_interval [13,87) [v2:172.21.15.98:6810/4249036577,v1:172.21.15.98:6811/4249036577] [v2:172.21.15.98:6812/4249036577,v1:172.21.15.98:6813/4249036577] exists,up 6d31ac4e-dc1a-4051-8266-ba821106242f 2024-09-21T10:03:25.576 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 up in weight 1 up_from 96 up_thru 101 down_at 93 last_clean_interval [18,92) [v2:172.21.15.98:6818/1274766038,v1:172.21.15.98:6819/1274766038] [v2:172.21.15.98:6820/1274766038,v1:172.21.15.98:6821/1274766038] exists,up 62e22789-74be-4098-a9b9-62671fb4cac1 2024-09-21T10:03:25.576 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 up in weight 1 up_from 101 up_thru 101 down_at 98 last_clean_interval [26,97) [v2:172.21.15.98:6826/3950952422,v1:172.21.15.98:6827/3950952422] [v2:172.21.15.98:6828/3950952422,v1:172.21.15.98:6829/3950952422] exists,up 77f653a6-69b0-437a-960d-c3fb036708be 2024-09-21T10:03:25.577 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 up in weight 1 up_from 31 up_thru 101 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6800/158468112,v1:172.21.15.196:6801/158468112] [v2:172.21.15.196:6802/158468112,v1:172.21.15.196:6803/158468112] exists,up 28ec4928-7f00-46a9-b8c5-95f0b7e6be75 2024-09-21T10:03:25.577 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 up in weight 1 up_from 37 up_thru 101 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6808/2341878784,v1:172.21.15.196:6809/2341878784] [v2:172.21.15.196:6810/2341878784,v1:172.21.15.196:6811/2341878784] exists,up cdcc44eb-e27f-4d23-9ecb-03bebdc82c96 2024-09-21T10:03:25.577 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 up in weight 1 up_from 43 up_thru 98 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6816/3059702155,v1:172.21.15.196:6817/3059702155] [v2:172.21.15.196:6818/3059702155,v1:172.21.15.196:6819/3059702155] exists,up 566cfb44-1b21-45fc-b053-7205b1c10d4a 2024-09-21T10:03:25.577 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 up in weight 1 up_from 49 up_thru 101 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6824/2155020234,v1:172.21.15.196:6825/2155020234] [v2:172.21.15.196:6826/2155020234,v1:172.21.15.196:6827/2155020234] exists,up 228404c1-3910-4933-b023-55aa8b01aee4 2024-09-21T10:03:25.577 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 4.4 [1,5] 2024-09-21T10:03:25.577 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 4.7 [1,2] 2024-09-21T10:03:25.577 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 4.18 [1,2] 2024-09-21T10:03:25.577 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/227493886 expires 2024-09-22T09:46:24.467967+0000 2024-09-21T10:03:25.577 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/4208010999 expires 2024-09-22T09:50:59.815356+0000 2024-09-21T10:03:25.577 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/493934303 expires 2024-09-22T09:56:36.894927+0000 2024-09-21T10:03:25.578 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.196:0/2330152142 expires 2024-09-22T09:55:58.115206+0000 2024-09-21T10:03:25.578 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/3466051265 expires 2024-09-22T09:54:48.070235+0000 2024-09-21T10:03:25.578 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/900860885 expires 2024-09-22T09:54:48.070235+0000 2024-09-21T10:03:25.578 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/675672768 expires 2024-09-22T09:50:59.815356+0000 2024-09-21T10:03:25.578 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/4139436413 expires 2024-09-22T09:50:59.815356+0000 2024-09-21T10:03:25.578 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/3656555554 expires 2024-09-22T09:45:54.237657+0000 2024-09-21T10:03:25.578 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6800/1387543382 expires 2024-09-22T09:54:48.070235+0000 2024-09-21T10:03:25.578 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.196:0/79302526 expires 2024-09-22T09:55:58.115206+0000 2024-09-21T10:03:25.578 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.196:0/1564192582 expires 2024-09-22T09:55:58.115206+0000 2024-09-21T10:03:25.578 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.196:6835/2652695583 expires 2024-09-22T09:55:58.115206+0000 2024-09-21T10:03:25.578 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6801/3394444334 expires 2024-09-22T09:56:36.894927+0000 2024-09-21T10:03:25.579 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/262018661 expires 2024-09-22T09:46:24.467967+0000 2024-09-21T10:03:25.579 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/2075804383 expires 2024-09-22T09:45:54.237657+0000 2024-09-21T10:03:25.579 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/2144473621 expires 2024-09-22T09:45:54.237657+0000 2024-09-21T10:03:25.579 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6801/440385038 expires 2024-09-22T09:46:24.467967+0000 2024-09-21T10:03:25.579 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6800/440385038 expires 2024-09-22T09:46:24.467967+0000 2024-09-21T10:03:25.579 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/2546189353 expires 2024-09-22T09:56:36.894927+0000 2024-09-21T10:03:25.579 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/2611845896 expires 2024-09-22T09:56:36.894927+0000 2024-09-21T10:03:25.579 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6801/1387543382 expires 2024-09-22T09:54:48.070235+0000 2024-09-21T10:03:25.579 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/3628356893 expires 2024-09-22T09:56:36.894927+0000 2024-09-21T10:03:25.579 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6800/1427334238 expires 2024-09-22T09:45:54.237657+0000 2024-09-21T10:03:25.579 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.196:0/1546057694 expires 2024-09-22T09:55:58.115206+0000 2024-09-21T10:03:25.579 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6801/89887073 expires 2024-09-22T09:50:59.815356+0000 2024-09-21T10:03:25.579 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/881612124 expires 2024-09-22T09:50:59.815356+0000 2024-09-21T10:03:25.580 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6800/3394444334 expires 2024-09-22T09:56:36.894927+0000 2024-09-21T10:03:25.580 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6801/1427334238 expires 2024-09-22T09:45:54.237657+0000 2024-09-21T10:03:25.580 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6800/89887073 expires 2024-09-22T09:50:59.815356+0000 2024-09-21T10:03:25.580 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/1093274465 expires 2024-09-22T09:54:48.070235+0000 2024-09-21T10:03:25.580 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/1268404616 expires 2024-09-22T09:56:36.894927+0000 2024-09-21T10:03:25.580 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/271851122 expires 2024-09-22T09:56:36.894927+0000 2024-09-21T10:03:25.580 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.196:0/2663325110 expires 2024-09-22T09:55:58.115206+0000 2024-09-21T10:03:25.580 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/1829140501 expires 2024-09-22T09:54:48.070235+0000 2024-09-21T10:03:25.580 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.196:6834/2652695583 expires 2024-09-22T09:55:58.115206+0000 2024-09-21T10:03:25.580 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/3737889208 expires 2024-09-22T09:46:24.467967+0000 2024-09-21T10:03:25.580 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.196:0/1036032960 expires 2024-09-22T09:55:58.115206+0000 2024-09-21T10:03:25.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:25 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1917677406' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T10:03:25.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:25 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1917677406' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T10:03:25.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:25 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1917677406' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T10:03:26.223 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 86 up_thru 101 down_at 83 last_clean_interval [8,82) [v2:172.21.15.98:6802/1869651717,v1:172.21.15.98:6803/1869651717] [v2:172.21.15.98:6804/1869651717,v1:172.21.15.98:6805/1869651717] exists,up 579f13de-565d-4801-b65e-f27d1cde297a', 'osd.1 up in weight 1 up_from 91 up_thru 101 down_at 88 last_clean_interval [13,87) [v2:172.21.15.98:6810/4249036577,v1:172.21.15.98:6811/4249036577] [v2:172.21.15.98:6812/4249036577,v1:172.21.15.98:6813/4249036577] exists,up 6d31ac4e-dc1a-4051-8266-ba821106242f', 'osd.2 up in weight 1 up_from 96 up_thru 101 down_at 93 last_clean_interval [18,92) [v2:172.21.15.98:6818/1274766038,v1:172.21.15.98:6819/1274766038] [v2:172.21.15.98:6820/1274766038,v1:172.21.15.98:6821/1274766038] exists,up 62e22789-74be-4098-a9b9-62671fb4cac1', 'osd.3 up in weight 1 up_from 101 up_thru 101 down_at 98 last_clean_interval [26,97) [v2:172.21.15.98:6826/3950952422,v1:172.21.15.98:6827/3950952422] [v2:172.21.15.98:6828/3950952422,v1:172.21.15.98:6829/3950952422] exists,up 77f653a6-69b0-437a-960d-c3fb036708be', 'osd.4 up in weight 1 up_from 31 up_thru 101 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6800/158468112,v1:172.21.15.196:6801/158468112] [v2:172.21.15.196:6802/158468112,v1:172.21.15.196:6803/158468112] exists,up 28ec4928-7f00-46a9-b8c5-95f0b7e6be75', 'osd.5 up in weight 1 up_from 37 up_thru 101 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6808/2341878784,v1:172.21.15.196:6809/2341878784] [v2:172.21.15.196:6810/2341878784,v1:172.21.15.196:6811/2341878784] exists,up cdcc44eb-e27f-4d23-9ecb-03bebdc82c96', 'osd.6 up in weight 1 up_from 43 up_thru 98 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6816/3059702155,v1:172.21.15.196:6817/3059702155] [v2:172.21.15.196:6818/3059702155,v1:172.21.15.196:6819/3059702155] exists,up 566cfb44-1b21-45fc-b053-7205b1c10d4a', 'osd.7 up in weight 1 up_from 49 up_thru 101 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6824/2155020234,v1:172.21.15.196:6825/2155020234] [v2:172.21.15.196:6826/2155020234,v1:172.21.15.196:6827/2155020234] exists,up 228404c1-3910-4933-b023-55aa8b01aee4'] 2024-09-21T10:03:26.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon mon.a config show 2024-09-21T10:03:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:26.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:26 smithi098 ceph-mon[97580]: pgmap v227: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:26.844 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:26 smithi098 ceph-mon[100914]: pgmap v227: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:26 smithi196 ceph-mon[84784]: pgmap v227: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:28.102 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:03:28.102 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mon.a", 2024-09-21T10:03:28.102 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster": "ceph", 2024-09-21T10:03:28.102 INFO:teuthology.orchestra.run.smithi098.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-21T10:03:28.102 INFO:teuthology.orchestra.run.smithi098.stdout: "admin_socket_mode": "", 2024-09-21T10:03:28.102 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-21T10:03:28.102 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_client_required": "cephx, none", 2024-09-21T10:03:28.102 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_cluster_required": "cephx", 2024-09-21T10:03:28.103 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_debug": "true", 2024-09-21T10:03:28.103 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-21T10:03:28.103 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-21T10:03:28.103 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_service_required": "cephx", 2024-09-21T10:03:28.103 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-21T10:03:28.103 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_supported": "", 2024-09-21T10:03:28.103 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_aio": "true", 2024-09-21T10:03:28.103 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-21T10:03:28.103 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_aio_poll_ms": "250", 2024-09-21T10:03:28.103 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_aio_reap_max": "16", 2024-09-21T10:03:28.103 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_async_discard_threads": "0", 2024-09-21T10:03:28.103 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_block_size": "4096", 2024-09-21T10:03:28.103 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_debug_aio": "false", 2024-09-21T10:03:28.103 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-21T10:03:28.104 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-21T10:03:28.104 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_debug_inflight_ios": "false", 2024-09-21T10:03:28.104 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_enable_discard": "false", 2024-09-21T10:03:28.104 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_flock_retry": "3", 2024-09-21T10:03:28.104 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-21T10:03:28.104 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_inject_crash": "0", 2024-09-21T10:03:28.104 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-21T10:03:28.104 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_ioring": "false", 2024-09-21T10:03:28.104 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_ioring_hipri": "false", 2024-09-21T10:03:28.104 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-21T10:03:28.104 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-21T10:03:28.104 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-21T10:03:28.104 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-21T10:03:28.105 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-21T10:03:28.105 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-21T10:03:28.105 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_type": "", 2024-09-21T10:03:28.105 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_alloc_size": "1048576", 2024-09-21T10:03:28.105 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_allocator": "hybrid", 2024-09-21T10:03:28.105 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_buffered_io": "true", 2024-09-21T10:03:28.105 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_check_for_zeros": "false", 2024-09-21T10:03:28.105 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-21T10:03:28.105 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-21T10:03:28.105 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_compact_log_sync": "false", 2024-09-21T10:03:28.105 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-21T10:03:28.105 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-21T10:03:28.105 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-21T10:03:28.105 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-21T10:03:28.106 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_max_log_runway": "4194304", 2024-09-21T10:03:28.106 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_max_prefetch": "1048576", 2024-09-21T10:03:28.106 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_min_flush_size": "524288", 2024-09-21T10:03:28.106 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_min_log_runway": "1048576", 2024-09-21T10:03:28.106 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_replay_recovery": "false", 2024-09-21T10:03:28.106 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-21T10:03:28.106 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-21T10:03:28.106 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_sync_write": "false", 2024-09-21T10:03:28.106 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-21T10:03:28.106 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-21T10:03:28.106 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-21T10:03:28.106 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_allocation_from_file": "true", 2024-09-21T10:03:28.106 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_allocator": "hybrid", 2024-09-21T10:03:28.107 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_async_db_compaction": "true", 2024-09-21T10:03:28.107 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-21T10:03:28.107 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-21T10:03:28.107 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-21T10:03:28.107 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-21T10:03:28.107 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_bdev_label_multi": "true", 2024-09-21T10:03:28.107 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-21T10:03:28.107 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-21T10:03:28.107 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-21T10:03:28.107 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-21T10:03:28.107 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-21T10:03:28.107 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_block_create": "true", 2024-09-21T10:03:28.107 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_block_db_create": "false", 2024-09-21T10:03:28.107 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_block_db_path": "", 2024-09-21T10:03:28.108 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_block_db_size": "0", 2024-09-21T10:03:28.108 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_block_path": "", 2024-09-21T10:03:28.108 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_block_preallocate_file": "false", 2024-09-21T10:03:28.108 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_block_size": "107374182400", 2024-09-21T10:03:28.108 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_block_wal_create": "false", 2024-09-21T10:03:28.108 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_block_wal_path": "", 2024-09-21T10:03:28.108 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_block_wal_size": "100663296", 2024-09-21T10:03:28.108 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_bluefs": "true", 2024-09-21T10:03:28.108 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-21T10:03:28.108 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-21T10:03:28.108 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-21T10:03:28.109 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-21T10:03:28.109 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-21T10:03:28.109 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T10:03:28.109 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T10:03:28.109 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-21T10:03:28.109 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T10:03:28.109 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_autotune": "true", 2024-09-21T10:03:28.109 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-21T10:03:28.109 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-21T10:03:28.109 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-21T10:03:28.109 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-21T10:03:28.109 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_size": "0", 2024-09-21T10:03:28.109 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-21T10:03:28.109 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-21T10:03:28.110 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-21T10:03:28.110 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-21T10:03:28.110 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_type": "2q", 2024-09-21T10:03:28.110 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-21T10:03:28.110 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_clone_cow": "true", 2024-09-21T10:03:28.110 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-21T10:03:28.110 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-21T10:03:28.110 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-21T10:03:28.111 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-21T10:03:28.111 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-21T10:03:28.111 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-21T10:03:28.111 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-21T10:03:28.111 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_compression_mode": "none", 2024-09-21T10:03:28.111 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-21T10:03:28.111 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_csum_type": "crc32c", 2024-09-21T10:03:28.111 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-21T10:03:28.111 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-21T10:03:28.111 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_freelist": "false", 2024-09-21T10:03:28.111 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-21T10:03:28.111 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-21T10:03:28.112 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-21T10:03:28.112 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-21T10:03:28.112 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-21T10:03:28.112 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-21T10:03:28.112 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-21T10:03:28.112 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-21T10:03:28.112 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-21T10:03:28.112 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-21T10:03:28.112 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-21T10:03:28.112 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-21T10:03:28.112 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_small_allocations": "0", 2024-09-21T10:03:28.112 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-21T10:03:28.113 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_default_buffered_read": "true", 2024-09-21T10:03:28.113 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_default_buffered_write": "false", 2024-09-21T10:03:28.113 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-21T10:03:28.113 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-21T10:03:28.113 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-21T10:03:28.113 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-21T10:03:28.113 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-21T10:03:28.113 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-21T10:03:28.113 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-21T10:03:28.113 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-21T10:03:28.114 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-21T10:03:28.114 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fail_eio": "false", 2024-09-21T10:03:28.114 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-21T10:03:28.114 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-21T10:03:28.114 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-21T10:03:28.114 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-21T10:03:28.114 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-21T10:03:28.114 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-21T10:03:28.114 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_on_mount": "false", 2024-09-21T10:03:28.114 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-21T10:03:28.114 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_on_umount": "false", 2024-09-21T10:03:28.114 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-21T10:03:28.114 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-21T10:03:28.115 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-21T10:03:28.115 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-21T10:03:28.115 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-21T10:03:28.115 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-21T10:03:28.115 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-21T10:03:28.115 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-21T10:03:28.115 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_ignore_data_csum": "false", 2024-09-21T10:03:28.115 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-21T10:03:28.115 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-21T10:03:28.115 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-21T10:03:28.115 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-21T10:03:28.115 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_log_op_age": "5.000000", 2024-09-21T10:03:28.115 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_max_alloc_size": "0", 2024-09-21T10:03:28.116 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_max_blob_size": "0", 2024-09-21T10:03:28.116 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-21T10:03:28.116 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-21T10:03:28.116 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-21T10:03:28.116 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_max_deferred_txc": "32", 2024-09-21T10:03:28.116 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_min_alloc_size": "0", 2024-09-21T10:03:28.116 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-21T10:03:28.116 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-21T10:03:28.116 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_nid_prealloc": "1024", 2024-09-21T10:03:28.116 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-21T10:03:28.116 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-21T10:03:28.116 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-21T10:03:28.116 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-21T10:03:28.116 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_retry_disk_reads": "3", 2024-09-21T10:03:28.117 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_rocksdb_cf": "true", 2024-09-21T10:03:28.117 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-21T10:03:28.117 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-21T10:03:28.117 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-21T10:03:28.117 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-21T10:03:28.117 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-21T10:03:28.117 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-21T10:03:28.117 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-21T10:03:28.117 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-21T10:03:28.117 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_spdk_mem": "512", 2024-09-21T10:03:28.118 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-21T10:03:28.118 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-21T10:03:28.118 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-21T10:03:28.118 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-21T10:03:28.118 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-21T10:03:28.118 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-21T10:03:28.118 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-21T10:03:28.118 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_tracing": "false", 2024-09-21T10:03:28.118 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-21T10:03:28.118 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-21T10:03:28.118 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-21T10:03:28.118 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-21T10:03:28.118 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-21T10:03:28.119 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-21T10:03:28.119 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-21T10:03:28.119 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-21T10:03:28.119 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-21T10:03:28.119 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_write_v2": "false", 2024-09-21T10:03:28.119 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_write_v2_random": "false", 2024-09-21T10:03:28.119 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_zero_block_detection": "false", 2024-09-21T10:03:28.119 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-21T10:03:28.119 INFO:teuthology.orchestra.run.smithi098.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-21T10:03:28.119 INFO:teuthology.orchestra.run.smithi098.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-21T10:03:28.119 INFO:teuthology.orchestra.run.smithi098.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-21T10:03:28.120 INFO:teuthology.orchestra.run.smithi098.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-21T10:03:28.120 INFO:teuthology.orchestra.run.smithi098.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-21T10:03:28.120 INFO:teuthology.orchestra.run.smithi098.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-21T10:03:28.120 INFO:teuthology.orchestra.run.smithi098.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-21T10:03:28.120 INFO:teuthology.orchestra.run.smithi098.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-21T10:03:28.120 INFO:teuthology.orchestra.run.smithi098.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-21T10:03:28.120 INFO:teuthology.orchestra.run.smithi098.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-21T10:03:28.120 INFO:teuthology.orchestra.run.smithi098.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-21T10:03:28.120 INFO:teuthology.orchestra.run.smithi098.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-21T10:03:28.120 INFO:teuthology.orchestra.run.smithi098.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-21T10:03:28.120 INFO:teuthology.orchestra.run.smithi098.stdout: "cephx_cluster_require_signatures": "false", 2024-09-21T10:03:28.120 INFO:teuthology.orchestra.run.smithi098.stdout: "cephx_cluster_require_version": "2", 2024-09-21T10:03:28.120 INFO:teuthology.orchestra.run.smithi098.stdout: "cephx_require_signatures": "false", 2024-09-21T10:03:28.121 INFO:teuthology.orchestra.run.smithi098.stdout: "cephx_require_version": "2", 2024-09-21T10:03:28.121 INFO:teuthology.orchestra.run.smithi098.stdout: "cephx_service_require_signatures": "false", 2024-09-21T10:03:28.121 INFO:teuthology.orchestra.run.smithi098.stdout: "cephx_service_require_version": "2", 2024-09-21T10:03:28.121 INFO:teuthology.orchestra.run.smithi098.stdout: "cephx_sign_messages": "true", 2024-09-21T10:03:28.121 INFO:teuthology.orchestra.run.smithi098.stdout: "chdir": "", 2024-09-21T10:03:28.121 INFO:teuthology.orchestra.run.smithi098.stdout: "client_acl_type": "", 2024-09-21T10:03:28.121 INFO:teuthology.orchestra.run.smithi098.stdout: "client_asio_thread_count": "2", 2024-09-21T10:03:28.121 INFO:teuthology.orchestra.run.smithi098.stdout: "client_cache_mid": "0.750000", 2024-09-21T10:03:28.121 INFO:teuthology.orchestra.run.smithi098.stdout: "client_cache_size": "16384", 2024-09-21T10:03:28.121 INFO:teuthology.orchestra.run.smithi098.stdout: "client_caps_release_delay": "5", 2024-09-21T10:03:28.121 INFO:teuthology.orchestra.run.smithi098.stdout: "client_check_pool_perm": "true", 2024-09-21T10:03:28.121 INFO:teuthology.orchestra.run.smithi098.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-21T10:03:28.122 INFO:teuthology.orchestra.run.smithi098.stdout: "client_debug_force_sync_read": "false", 2024-09-21T10:03:28.122 INFO:teuthology.orchestra.run.smithi098.stdout: "client_debug_getattr_caps": "false", 2024-09-21T10:03:28.122 INFO:teuthology.orchestra.run.smithi098.stdout: "client_debug_inject_features": "", 2024-09-21T10:03:28.122 INFO:teuthology.orchestra.run.smithi098.stdout: "client_debug_inject_tick_delay": "0", 2024-09-21T10:03:28.122 INFO:teuthology.orchestra.run.smithi098.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-21T10:03:28.122 INFO:teuthology.orchestra.run.smithi098.stdout: "client_die_on_failed_remount": "false", 2024-09-21T10:03:28.122 INFO:teuthology.orchestra.run.smithi098.stdout: "client_dirsize_rbytes": "true", 2024-09-21T10:03:28.122 INFO:teuthology.orchestra.run.smithi098.stdout: "client_force_lazyio": "false", 2024-09-21T10:03:28.122 INFO:teuthology.orchestra.run.smithi098.stdout: "client_fs": "", 2024-09-21T10:03:28.122 INFO:teuthology.orchestra.run.smithi098.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-21T10:03:28.122 INFO:teuthology.orchestra.run.smithi098.stdout: "client_inject_release_failure": "false", 2024-09-21T10:03:28.122 INFO:teuthology.orchestra.run.smithi098.stdout: "client_max_inline_size": "4096", 2024-09-21T10:03:28.123 INFO:teuthology.orchestra.run.smithi098.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-21T10:03:28.123 INFO:teuthology.orchestra.run.smithi098.stdout: "client_mds_namespace": "", 2024-09-21T10:03:28.123 INFO:teuthology.orchestra.run.smithi098.stdout: "client_metadata": "", 2024-09-21T10:03:28.123 INFO:teuthology.orchestra.run.smithi098.stdout: "client_mount_gid": "-1", 2024-09-21T10:03:28.123 INFO:teuthology.orchestra.run.smithi098.stdout: "client_mount_timeout": "300", 2024-09-21T10:03:28.123 INFO:teuthology.orchestra.run.smithi098.stdout: "client_mount_uid": "-1", 2024-09-21T10:03:28.123 INFO:teuthology.orchestra.run.smithi098.stdout: "client_mountpoint": "/", 2024-09-21T10:03:28.123 INFO:teuthology.orchestra.run.smithi098.stdout: "client_notify_timeout": "10", 2024-09-21T10:03:28.123 INFO:teuthology.orchestra.run.smithi098.stdout: "client_oc": "true", 2024-09-21T10:03:28.123 INFO:teuthology.orchestra.run.smithi098.stdout: "client_oc_max_dirty": "104857600", 2024-09-21T10:03:28.123 INFO:teuthology.orchestra.run.smithi098.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-21T10:03:28.123 INFO:teuthology.orchestra.run.smithi098.stdout: "client_oc_max_objects": "1000", 2024-09-21T10:03:28.123 INFO:teuthology.orchestra.run.smithi098.stdout: "client_oc_size": "209715200", 2024-09-21T10:03:28.124 INFO:teuthology.orchestra.run.smithi098.stdout: "client_oc_target_dirty": "8388608", 2024-09-21T10:03:28.124 INFO:teuthology.orchestra.run.smithi098.stdout: "client_permissions": "true", 2024-09-21T10:03:28.124 INFO:teuthology.orchestra.run.smithi098.stdout: "client_quota": "true", 2024-09-21T10:03:28.124 INFO:teuthology.orchestra.run.smithi098.stdout: "client_quota_df": "true", 2024-09-21T10:03:28.124 INFO:teuthology.orchestra.run.smithi098.stdout: "client_readahead_max_bytes": "0", 2024-09-21T10:03:28.124 INFO:teuthology.orchestra.run.smithi098.stdout: "client_readahead_max_periods": "4", 2024-09-21T10:03:28.124 INFO:teuthology.orchestra.run.smithi098.stdout: "client_readahead_min": "131072", 2024-09-21T10:03:28.124 INFO:teuthology.orchestra.run.smithi098.stdout: "client_reconnect_stale": "false", 2024-09-21T10:03:28.124 INFO:teuthology.orchestra.run.smithi098.stdout: "client_shutdown_timeout": "30", 2024-09-21T10:03:28.124 INFO:teuthology.orchestra.run.smithi098.stdout: "client_snapdir": ".snap", 2024-09-21T10:03:28.124 INFO:teuthology.orchestra.run.smithi098.stdout: "client_tick_interval": "1", 2024-09-21T10:03:28.124 INFO:teuthology.orchestra.run.smithi098.stdout: "client_trace": "", 2024-09-21T10:03:28.125 INFO:teuthology.orchestra.run.smithi098.stdout: "client_try_dentry_invalidate": "false", 2024-09-21T10:03:28.125 INFO:teuthology.orchestra.run.smithi098.stdout: "client_use_faked_inos": "false", 2024-09-21T10:03:28.125 INFO:teuthology.orchestra.run.smithi098.stdout: "client_use_random_mds": "false", 2024-09-21T10:03:28.125 INFO:teuthology.orchestra.run.smithi098.stdout: "clog_to_graylog": "false", 2024-09-21T10:03:28.125 INFO:teuthology.orchestra.run.smithi098.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-21T10:03:28.125 INFO:teuthology.orchestra.run.smithi098.stdout: "clog_to_graylog_port": "12201", 2024-09-21T10:03:28.125 INFO:teuthology.orchestra.run.smithi098.stdout: "clog_to_monitors": "default=true", 2024-09-21T10:03:28.125 INFO:teuthology.orchestra.run.smithi098.stdout: "clog_to_syslog": "false", 2024-09-21T10:03:28.125 INFO:teuthology.orchestra.run.smithi098.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-21T10:03:28.125 INFO:teuthology.orchestra.run.smithi098.stdout: "clog_to_syslog_level": "info", 2024-09-21T10:03:28.125 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "-", 2024-09-21T10:03:28.125 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_network": "", 2024-09-21T10:03:28.125 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_network_interface": "", 2024-09-21T10:03:28.125 INFO:teuthology.orchestra.run.smithi098.stdout: "compressor_zlib_isal": "false", 2024-09-21T10:03:28.126 INFO:teuthology.orchestra.run.smithi098.stdout: "compressor_zlib_level": "5", 2024-09-21T10:03:28.126 INFO:teuthology.orchestra.run.smithi098.stdout: "compressor_zlib_winsize": "-15", 2024-09-21T10:03:28.126 INFO:teuthology.orchestra.run.smithi098.stdout: "compressor_zstd_level": "1", 2024-09-21T10:03:28.126 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image": "quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47", 2024-09-21T10:03:28.126 INFO:teuthology.orchestra.run.smithi098.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-21T10:03:28.126 INFO:teuthology.orchestra.run.smithi098.stdout: "crimson_alien_op_num_threads": "6", 2024-09-21T10:03:28.126 INFO:teuthology.orchestra.run.smithi098.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-21T10:03:28.126 INFO:teuthology.orchestra.run.smithi098.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-21T10:03:28.126 INFO:teuthology.orchestra.run.smithi098.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-21T10:03:28.126 INFO:teuthology.orchestra.run.smithi098.stdout: "crimson_osd_stat_interval": "0", 2024-09-21T10:03:28.126 INFO:teuthology.orchestra.run.smithi098.stdout: "crimson_seastar_cpu_cores": "", 2024-09-21T10:03:28.126 INFO:teuthology.orchestra.run.smithi098.stdout: "crimson_seastar_num_threads": "0", 2024-09-21T10:03:28.126 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "", 2024-09-21T10:03:28.126 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location_hook": "", 2024-09-21T10:03:28.127 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location_hook_timeout": "10", 2024-09-21T10:03:28.127 INFO:teuthology.orchestra.run.smithi098.stdout: "daemonize": "false", 2024-09-21T10:03:28.127 INFO:teuthology.orchestra.run.smithi098.stdout: "daos_pool": "tank", 2024-09-21T10:03:28.127 INFO:teuthology.orchestra.run.smithi098.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-21T10:03:28.127 INFO:teuthology.orchestra.run.smithi098.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-21T10:03:28.127 INFO:teuthology.orchestra.run.smithi098.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-21T10:03:28.127 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_alienstore": "0/5", 2024-09-21T10:03:28.127 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_allow_any_pool_priority": "false", 2024-09-21T10:03:28.127 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_asok": "1/5", 2024-09-21T10:03:28.127 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_asok_assert_abort": "false", 2024-09-21T10:03:28.127 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_asserts_on_shutdown": "true", 2024-09-21T10:03:28.127 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_auth": "1/5", 2024-09-21T10:03:28.127 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_bdev": "1/3", 2024-09-21T10:03:28.127 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_bluefs": "1/5", 2024-09-21T10:03:28.128 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_bluestore": "1/5", 2024-09-21T10:03:28.128 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_buffer": "0/1", 2024-09-21T10:03:28.128 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_ceph_dedup": "0/5", 2024-09-21T10:03:28.128 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_ceph_exporter": "1/5", 2024-09-21T10:03:28.128 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_cephfs_mirror": "0/5", 2024-09-21T10:03:28.128 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_cephsqlite": "0/5", 2024-09-21T10:03:28.128 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_client": "0/5", 2024-09-21T10:03:28.128 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_compressor": "1/5", 2024-09-21T10:03:28.128 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_context": "0/1", 2024-09-21T10:03:28.128 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_crimson_interrupt": "0/5", 2024-09-21T10:03:28.128 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_crush": "1/1", 2024-09-21T10:03:28.128 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_crypto": "1/5", 2024-09-21T10:03:28.128 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_cyanstore": "0/5", 2024-09-21T10:03:28.128 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_deliberately_leak_memory": "false", 2024-09-21T10:03:28.129 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_disable_randomized_ping": "false", 2024-09-21T10:03:28.129 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_dpdk": "1/5", 2024-09-21T10:03:28.129 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_eventtrace": "1/5", 2024-09-21T10:03:28.129 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_filer": "0/1", 2024-09-21T10:03:28.129 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_filestore": "1/3", 2024-09-21T10:03:28.129 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_finisher": "1/1", 2024-09-21T10:03:28.129 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_fuse": "1/5", 2024-09-21T10:03:28.129 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_heartbeat_testing_span": "0", 2024-09-21T10:03:28.129 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_heartbeatmap": "1/5", 2024-09-21T10:03:28.129 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-21T10:03:28.130 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_javaclient": "1/5", 2024-09-21T10:03:28.130 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_journal": "1/3", 2024-09-21T10:03:28.130 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_journaler": "0/5", 2024-09-21T10:03:28.130 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_kstore": "1/5", 2024-09-21T10:03:28.130 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_lockdep": "0/1", 2024-09-21T10:03:28.130 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_mclock": "1/5", 2024-09-21T10:03:28.130 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_mds": "1/5", 2024-09-21T10:03:28.130 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_mds_balancer": "1/5", 2024-09-21T10:03:28.130 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_mds_locker": "1/5", 2024-09-21T10:03:28.130 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_mds_log": "1/5", 2024-09-21T10:03:28.130 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_mds_log_expire": "1/5", 2024-09-21T10:03:28.130 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_mds_migrator": "1/5", 2024-09-21T10:03:28.130 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_mds_quiesce": "3/5", 2024-09-21T10:03:28.130 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_memstore": "1/5", 2024-09-21T10:03:28.131 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_mgr": "2/5", 2024-09-21T10:03:28.131 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_mgrc": "1/5", 2024-09-21T10:03:28.131 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_mon": "20/20", 2024-09-21T10:03:28.131 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_monc": "0/10", 2024-09-21T10:03:28.131 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_ms": "1/1", 2024-09-21T10:03:28.131 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_none": "0/5", 2024-09-21T10:03:28.131 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_objclass": "0/5", 2024-09-21T10:03:28.131 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_objectcacher": "0/5", 2024-09-21T10:03:28.131 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_objecter": "0/1", 2024-09-21T10:03:28.131 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_optracker": "0/5", 2024-09-21T10:03:28.132 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_osd": "1/5", 2024-09-21T10:03:28.132 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_paxos": "20/20", 2024-09-21T10:03:28.132 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_perfcounter": "1/5", 2024-09-21T10:03:28.132 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_prioritycache": "1/5", 2024-09-21T10:03:28.132 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rados": "0/5", 2024-09-21T10:03:28.132 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rbd": "0/5", 2024-09-21T10:03:28.132 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rbd_mirror": "0/5", 2024-09-21T10:03:28.132 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rbd_pwl": "0/5", 2024-09-21T10:03:28.132 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rbd_replay": "0/5", 2024-09-21T10:03:28.132 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_refs": "0/0", 2024-09-21T10:03:28.132 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_reserver": "1/1", 2024-09-21T10:03:28.132 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rgw": "1/5", 2024-09-21T10:03:28.132 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rgw_access": "1/5", 2024-09-21T10:03:28.132 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rgw_datacache": "1/5", 2024-09-21T10:03:28.133 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rgw_dbstore": "1/5", 2024-09-21T10:03:28.133 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rgw_flight": "1/5", 2024-09-21T10:03:28.133 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-21T10:03:28.133 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rgw_notification": "1/5", 2024-09-21T10:03:28.133 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rgw_sync": "1/5", 2024-09-21T10:03:28.133 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rocksdb": "4/5", 2024-09-21T10:03:28.133 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore": "0/5", 2024-09-21T10:03:28.133 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore_backref": "0/5", 2024-09-21T10:03:28.133 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore_cache": "0/5", 2024-09-21T10:03:28.133 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore_cleaner": "0/5", 2024-09-21T10:03:28.133 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore_device": "0/5", 2024-09-21T10:03:28.133 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore_epm": "0/5", 2024-09-21T10:03:28.133 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-21T10:03:28.133 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore_journal": "0/5", 2024-09-21T10:03:28.134 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore_lba": "0/5", 2024-09-21T10:03:28.134 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore_odata": "0/5", 2024-09-21T10:03:28.134 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore_omap": "0/5", 2024-09-21T10:03:28.134 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore_onode": "0/5", 2024-09-21T10:03:28.134 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore_t": "0/5", 2024-09-21T10:03:28.134 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore_tm": "0/5", 2024-09-21T10:03:28.134 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_striper": "0/1", 2024-09-21T10:03:28.134 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_test": "0/5", 2024-09-21T10:03:28.134 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_throttle": "1/1", 2024-09-21T10:03:28.134 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_timer": "0/1", 2024-09-21T10:03:28.134 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_tp": "0/5", 2024-09-21T10:03:28.134 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_trace": "1/5", 2024-09-21T10:03:28.134 INFO:teuthology.orchestra.run.smithi098.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-21T10:03:28.135 INFO:teuthology.orchestra.run.smithi098.stdout: "device_failure_prediction_mode": "none", 2024-09-21T10:03:28.135 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-21T10:03:28.135 INFO:teuthology.orchestra.run.smithi098.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-21T10:03:28.135 INFO:teuthology.orchestra.run.smithi098.stdout: "err_to_graylog": "false", 2024-09-21T10:03:28.135 INFO:teuthology.orchestra.run.smithi098.stdout: "err_to_journald": "false", 2024-09-21T10:03:28.135 INFO:teuthology.orchestra.run.smithi098.stdout: "err_to_stderr": "true", 2024-09-21T10:03:28.135 INFO:teuthology.orchestra.run.smithi098.stdout: "err_to_syslog": "false", 2024-09-21T10:03:28.135 INFO:teuthology.orchestra.run.smithi098.stdout: "event_tracing": "false", 2024-09-21T10:03:28.135 INFO:teuthology.orchestra.run.smithi098.stdout: "exporter_addr": "0.0.0.0", 2024-09-21T10:03:28.135 INFO:teuthology.orchestra.run.smithi098.stdout: "exporter_cert_file": "", 2024-09-21T10:03:28.135 INFO:teuthology.orchestra.run.smithi098.stdout: "exporter_http_port": "9926", 2024-09-21T10:03:28.135 INFO:teuthology.orchestra.run.smithi098.stdout: "exporter_key_file": "", 2024-09-21T10:03:28.135 INFO:teuthology.orchestra.run.smithi098.stdout: "exporter_prio_limit": "5", 2024-09-21T10:03:28.135 INFO:teuthology.orchestra.run.smithi098.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-21T10:03:28.136 INFO:teuthology.orchestra.run.smithi098.stdout: "exporter_sort_metrics": "true", 2024-09-21T10:03:28.136 INFO:teuthology.orchestra.run.smithi098.stdout: "exporter_stats_period": "5", 2024-09-21T10:03:28.136 INFO:teuthology.orchestra.run.smithi098.stdout: "fake_statfs_for_testing": "0", 2024-09-21T10:03:28.136 INFO:teuthology.orchestra.run.smithi098.stdout: "fatal_signal_handlers": "true", 2024-09-21T10:03:28.136 INFO:teuthology.orchestra.run.smithi098.stdout: "filer_max_purge_ops": "10", 2024-09-21T10:03:28.136 INFO:teuthology.orchestra.run.smithi098.stdout: "filer_max_truncate_ops": "128", 2024-09-21T10:03:28.136 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_apply_finisher_threads": "1", 2024-09-21T10:03:28.136 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_blackhole": "false", 2024-09-21T10:03:28.136 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_btrfs_clone_range": "true", 2024-09-21T10:03:28.136 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_btrfs_snap": "true", 2024-09-21T10:03:28.136 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_caller_concurrency": "10", 2024-09-21T10:03:28.136 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_collect_device_partition_information": "true", 2024-09-21T10:03:28.136 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_commit_timeout": "600.000000", 2024-09-21T10:03:28.136 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_debug_inject_read_err": "false", 2024-09-21T10:03:28.137 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_debug_omap_check": "false", 2024-09-21T10:03:28.137 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_debug_verify_split": "false", 2024-09-21T10:03:28.137 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_dump_file": "", 2024-09-21T10:03:28.137 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-21T10:03:28.137 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-21T10:03:28.137 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_fadvise": "true", 2024-09-21T10:03:28.137 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_fail_eio": "true", 2024-09-21T10:03:28.137 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_fd_cache_shards": "16", 2024-09-21T10:03:28.137 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_fd_cache_size": "128", 2024-09-21T10:03:28.137 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_fiemap": "false", 2024-09-21T10:03:28.137 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_fiemap_threshold": "4096", 2024-09-21T10:03:28.137 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-21T10:03:28.137 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-21T10:03:28.137 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_inject_stall": "0", 2024-09-21T10:03:28.138 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_journal_parallel": "false", 2024-09-21T10:03:28.138 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_journal_trailing": "false", 2024-09-21T10:03:28.138 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_journal_writeahead": "false", 2024-09-21T10:03:28.138 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_kill_at": "0", 2024-09-21T10:03:28.138 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-21T10:03:28.138 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-21T10:03:28.138 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-21T10:03:28.138 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-21T10:03:28.138 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-21T10:03:28.138 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_inline_xattrs": "0", 2024-09-21T10:03:28.138 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-21T10:03:28.139 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-21T10:03:28.139 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-21T10:03:28.139 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-21T10:03:28.139 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_xattr_value_size": "0", 2024-09-21T10:03:28.139 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-21T10:03:28.139 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-21T10:03:28.139 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-21T10:03:28.139 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_merge_threshold": "-10", 2024-09-21T10:03:28.139 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-21T10:03:28.139 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_odsync_write": "false", 2024-09-21T10:03:28.139 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_omap_backend": "rocksdb", 2024-09-21T10:03:28.139 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_omap_backend_path": "", 2024-09-21T10:03:28.139 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-21T10:03:28.139 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-21T10:03:28.140 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-21T10:03:28.140 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_op_thread_timeout": "60", 2024-09-21T10:03:28.140 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_op_threads": "2", 2024-09-21T10:03:28.140 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_punch_hole": "false", 2024-09-21T10:03:28.140 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-21T10:03:28.140 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-21T10:03:28.140 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-21T10:03:28.140 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-21T10:03:28.140 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-21T10:03:28.140 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-21T10:03:28.140 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-21T10:03:28.140 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-21T10:03:28.140 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-21T10:03:28.140 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_queue_max_ops": "50", 2024-09-21T10:03:28.141 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-21T10:03:28.141 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_seek_data_hole": "false", 2024-09-21T10:03:28.141 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_sloppy_crc": "false", 2024-09-21T10:03:28.141 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-21T10:03:28.141 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_splice": "false", 2024-09-21T10:03:28.141 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_split_multiple": "2", 2024-09-21T10:03:28.141 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_split_rand_factor": "20", 2024-09-21T10:03:28.141 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_update_to": "1000", 2024-09-21T10:03:28.141 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-21T10:03:28.141 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-21T10:03:28.141 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-21T10:03:28.141 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-21T10:03:28.141 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-21T10:03:28.142 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-21T10:03:28.142 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_enable": "true", 2024-09-21T10:03:28.142 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-21T10:03:28.142 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-21T10:03:28.142 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-21T10:03:28.142 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-21T10:03:28.142 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-21T10:03:28.142 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-21T10:03:28.142 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_xfs_extsize": "false", 2024-09-21T10:03:28.142 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_zfs_snap": "false", 2024-09-21T10:03:28.142 INFO:teuthology.orchestra.run.smithi098.stdout: "fio_dir": "/tmp/fio", 2024-09-21T10:03:28.142 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "26a15512-77fe-11ef-baf6-efdc52797490", 2024-09-21T10:03:28.142 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_allow_other": "true", 2024-09-21T10:03:28.142 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_atomic_o_trunc": "true", 2024-09-21T10:03:28.143 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_big_writes": "true", 2024-09-21T10:03:28.143 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_debug": "false", 2024-09-21T10:03:28.143 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_default_permissions": "false", 2024-09-21T10:03:28.143 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_disable_pagecache": "false", 2024-09-21T10:03:28.143 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_max_write": "0", 2024-09-21T10:03:28.143 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_multithreaded": "true", 2024-09-21T10:03:28.143 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_require_active_mds": "true", 2024-09-21T10:03:28.143 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_set_user_groups": "true", 2024-09-21T10:03:28.143 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_splice_move": "true", 2024-09-21T10:03:28.143 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_splice_read": "true", 2024-09-21T10:03:28.143 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_splice_write": "true", 2024-09-21T10:03:28.144 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-21T10:03:28.144 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_use_invalidate_cb": "true", 2024-09-21T10:03:28.144 INFO:teuthology.orchestra.run.smithi098.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-21T10:03:28.144 INFO:teuthology.orchestra.run.smithi098.stdout: "gss_target_name": "ceph", 2024-09-21T10:03:28.144 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_file": "", 2024-09-21T10:03:28.144 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_inject_failure": "0", 2024-09-21T10:03:28.144 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_interval": "5", 2024-09-21T10:03:28.144 INFO:teuthology.orchestra.run.smithi098.stdout: "host": "smithi098", 2024-09-21T10:03:28.144 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-21T10:03:28.144 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-21T10:03:28.144 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-21T10:03:28.144 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-21T10:03:28.144 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-21T10:03:28.145 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-21T10:03:28.145 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-21T10:03:28.145 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-21T10:03:28.145 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-21T10:03:28.145 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-21T10:03:28.145 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-21T10:03:28.145 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-21T10:03:28.145 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-21T10:03:28.145 INFO:teuthology.orchestra.run.smithi098.stdout: "inject_early_sigterm": "false", 2024-09-21T10:03:28.145 INFO:teuthology.orchestra.run.smithi098.stdout: "jaeger_agent_port": "6799", 2024-09-21T10:03:28.146 INFO:teuthology.orchestra.run.smithi098.stdout: "jaeger_tracing_enable": "false", 2024-09-21T10:03:28.146 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_aio": "true", 2024-09-21T10:03:28.146 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_align_min_size": "65536", 2024-09-21T10:03:28.146 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_block_align": "true", 2024-09-21T10:03:28.146 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_block_size": "4096", 2024-09-21T10:03:28.146 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_dio": "true", 2024-09-21T10:03:28.146 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_discard": "false", 2024-09-21T10:03:28.146 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_force_aio": "false", 2024-09-21T10:03:28.146 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_ignore_corruption": "false", 2024-09-21T10:03:28.146 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_max_write_bytes": "10485760", 2024-09-21T10:03:28.146 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_max_write_entries": "100", 2024-09-21T10:03:28.146 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_replay_from": "0", 2024-09-21T10:03:28.146 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-21T10:03:28.147 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-21T10:03:28.147 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-21T10:03:28.147 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-21T10:03:28.147 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_write_header_frequency": "0", 2024-09-21T10:03:28.147 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_zero_on_create": "false", 2024-09-21T10:03:28.147 INFO:teuthology.orchestra.run.smithi098.stdout: "journaler_prefetch_periods": "10", 2024-09-21T10:03:28.147 INFO:teuthology.orchestra.run.smithi098.stdout: "journaler_prezero_periods": "5", 2024-09-21T10:03:28.147 INFO:teuthology.orchestra.run.smithi098.stdout: "journaler_write_head_interval": "15", 2024-09-21T10:03:28.147 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "", 2024-09-21T10:03:28.147 INFO:teuthology.orchestra.run.smithi098.stdout: "keyfile": "", 2024-09-21T10:03:28.147 INFO:teuthology.orchestra.run.smithi098.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-21T10:03:28.147 INFO:teuthology.orchestra.run.smithi098.stdout: "kstore_backend": "rocksdb", 2024-09-21T10:03:28.148 INFO:teuthology.orchestra.run.smithi098.stdout: "kstore_default_stripe_size": "65536", 2024-09-21T10:03:28.148 INFO:teuthology.orchestra.run.smithi098.stdout: "kstore_fsck_on_mount": "false", 2024-09-21T10:03:28.148 INFO:teuthology.orchestra.run.smithi098.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-21T10:03:28.148 INFO:teuthology.orchestra.run.smithi098.stdout: "kstore_max_bytes": "67108864", 2024-09-21T10:03:28.148 INFO:teuthology.orchestra.run.smithi098.stdout: "kstore_max_ops": "512", 2024-09-21T10:03:28.148 INFO:teuthology.orchestra.run.smithi098.stdout: "kstore_nid_prealloc": "1024", 2024-09-21T10:03:28.148 INFO:teuthology.orchestra.run.smithi098.stdout: "kstore_onode_map_size": "1024", 2024-09-21T10:03:28.148 INFO:teuthology.orchestra.run.smithi098.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-21T10:03:28.148 INFO:teuthology.orchestra.run.smithi098.stdout: "kstore_sync_submit_transaction": "false", 2024-09-21T10:03:28.148 INFO:teuthology.orchestra.run.smithi098.stdout: "kstore_sync_transaction": "false", 2024-09-21T10:03:28.148 INFO:teuthology.orchestra.run.smithi098.stdout: "librados_thread_count": "2", 2024-09-21T10:03:28.148 INFO:teuthology.orchestra.run.smithi098.stdout: "lockdep": "false", 2024-09-21T10:03:28.148 INFO:teuthology.orchestra.run.smithi098.stdout: "lockdep_force_backtrace": "false", 2024-09-21T10:03:28.149 INFO:teuthology.orchestra.run.smithi098.stdout: "log_coarse_timestamps": "true", 2024-09-21T10:03:28.149 INFO:teuthology.orchestra.run.smithi098.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-21T10:03:28.149 INFO:teuthology.orchestra.run.smithi098.stdout: "log_flush_on_exit": "false", 2024-09-21T10:03:28.149 INFO:teuthology.orchestra.run.smithi098.stdout: "log_graylog_host": "127.0.0.1", 2024-09-21T10:03:28.149 INFO:teuthology.orchestra.run.smithi098.stdout: "log_graylog_port": "12201", 2024-09-21T10:03:28.149 INFO:teuthology.orchestra.run.smithi098.stdout: "log_max_new": "1000", 2024-09-21T10:03:28.149 INFO:teuthology.orchestra.run.smithi098.stdout: "log_max_recent": "10000", 2024-09-21T10:03:28.149 INFO:teuthology.orchestra.run.smithi098.stdout: "log_stderr_prefix": "", 2024-09-21T10:03:28.149 INFO:teuthology.orchestra.run.smithi098.stdout: "log_stop_at_utilization": "0.970000", 2024-09-21T10:03:28.149 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": "true", 2024-09-21T10:03:28.149 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_graylog": "false", 2024-09-21T10:03:28.149 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_journald": "false", 2024-09-21T10:03:28.149 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_stderr": "false", 2024-09-21T10:03:28.150 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_syslog": "false", 2024-09-21T10:03:28.150 INFO:teuthology.orchestra.run.smithi098.stdout: "max_rotating_auth_attempts": "10", 2024-09-21T10:03:28.150 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-21T10:03:28.150 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_action_on_write_error": "1", 2024-09-21T10:03:28.150 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_alternate_name_max": "8192", 2024-09-21T10:03:28.150 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_asio_thread_count": "2", 2024-09-21T10:03:28.150 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_export_pin": "true", 2024-09-21T10:03:28.150 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_fragment_dirs": "true", 2024-09-21T10:03:28.150 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-21T10:03:28.150 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_fragment_interval": "5", 2024-09-21T10:03:28.150 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-21T10:03:28.150 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-21T10:03:28.151 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_interval": "10", 2024-09-21T10:03:28.151 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_max": "-1", 2024-09-21T10:03:28.151 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_max_until": "-1", 2024-09-21T10:03:28.151 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_merge_size": "50", 2024-09-21T10:03:28.151 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_midchunk": "0.300000", 2024-09-21T10:03:28.151 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-21T10:03:28.151 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_min_start": "0.200000", 2024-09-21T10:03:28.151 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_minchunk": "0.001000", 2024-09-21T10:03:28.151 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_mode": "0", 2024-09-21T10:03:28.151 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_need_max": "1.200000", 2024-09-21T10:03:28.151 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_need_min": "0.800000", 2024-09-21T10:03:28.151 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_overload_epochs": "2", 2024-09-21T10:03:28.152 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-21T10:03:28.152 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-21T10:03:28.152 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_split_bits": "3", 2024-09-21T10:03:28.152 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-21T10:03:28.152 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_split_size": "10000", 2024-09-21T10:03:28.152 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-21T10:03:28.152 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_target_decay": "10.000000", 2024-09-21T10:03:28.152 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-21T10:03:28.152 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_beacon_grace": "15.000000", 2024-09-21T10:03:28.152 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_beacon_interval": "4.000000", 2024-09-21T10:03:28.152 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-21T10:03:28.152 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-21T10:03:28.152 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache_mid": "0.700000", 2024-09-21T10:03:28.153 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-21T10:03:28.153 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache_quiesce_delay": "0", 2024-09-21T10:03:28.153 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-21T10:03:28.153 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-21T10:03:28.153 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-21T10:03:28.153 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache_release_free_interval": "10", 2024-09-21T10:03:28.153 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache_reservation": "0.050000", 2024-09-21T10:03:28.153 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-21T10:03:28.153 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache_trim_interval": "1", 2024-09-21T10:03:28.153 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache_trim_threshold": "262144", 2024-09-21T10:03:28.153 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-21T10:03:28.154 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-21T10:03:28.154 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-21T10:03:28.154 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_client_prealloc_inos": "1000", 2024-09-21T10:03:28.154 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-21T10:03:28.154 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_connect_bootstrapping": "false", 2024-09-21T10:03:28.154 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_damage_table_max_entries": "10000", 2024-09-21T10:03:28.154 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-21T10:03:28.154 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_debug_auth_pins": "false", 2024-09-21T10:03:28.154 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_debug_frag": "false", 2024-09-21T10:03:28.154 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_debug_scatterstat": "false", 2024-09-21T10:03:28.154 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_debug_subtrees": "false", 2024-09-21T10:03:28.155 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_decay_halflife": "5.000000", 2024-09-21T10:03:28.155 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_default_dir_hash": "2", 2024-09-21T10:03:28.155 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_defer_session_stale": "true", 2024-09-21T10:03:28.155 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_deny_all_reconnect": "false", 2024-09-21T10:03:28.155 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_dir_keys_per_op": "16384", 2024-09-21T10:03:28.155 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_dir_max_commit_size": "10", 2024-09-21T10:03:28.155 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_dir_max_entries": "0", 2024-09-21T10:03:28.155 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_dir_prefetch": "true", 2024-09-21T10:03:28.155 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-21T10:03:28.155 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-21T10:03:28.155 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_dump_cache_on_map": "false", 2024-09-21T10:03:28.155 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-21T10:03:28.155 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-21T10:03:28.156 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_early_reply": "true", 2024-09-21T10:03:28.156 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_enable_op_tracker": "true", 2024-09-21T10:03:28.156 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_enforce_unique_name": "true", 2024-09-21T10:03:28.156 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-21T10:03:28.156 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-21T10:03:28.156 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_export_ephemeral_random": "true", 2024-09-21T10:03:28.156 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-21T10:03:28.156 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-21T10:03:28.156 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-21T10:03:28.156 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-21T10:03:28.156 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-21T10:03:28.156 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-21T10:03:28.156 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-21T10:03:28.157 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-21T10:03:28.157 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_health_summarize_threshold": "10", 2024-09-21T10:03:28.157 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-21T10:03:28.161 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-21T10:03:28.161 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_inject_health_dummy": "false", 2024-09-21T10:03:28.161 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-21T10:03:28.161 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_inject_migrator_session_race": "false", 2024-09-21T10:03:28.161 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-21T10:03:28.161 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-21T10:03:28.161 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-21T10:03:28.161 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_join_fs": "", 2024-09-21T10:03:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_journal_format": "1", 2024-09-21T10:03:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-21T10:03:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_create_at": "0", 2024-09-21T10:03:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_dirfrag_at": "0", 2024-09-21T10:03:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_export_at": "0", 2024-09-21T10:03:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_import_at": "0", 2024-09-21T10:03:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_journal_at": "0", 2024-09-21T10:03:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_journal_expire_at": "0", 2024-09-21T10:03:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_journal_replay_at": "0", 2024-09-21T10:03:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_link_at": "0", 2024-09-21T10:03:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_mdstable_at": "0", 2024-09-21T10:03:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_openc_at": "0", 2024-09-21T10:03:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_rename_at": "0", 2024-09-21T10:03:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_shutdown_at": "0", 2024-09-21T10:03:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log_event_large_threshold": "524288", 2024-09-21T10:03:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log_events_per_segment": "1024", 2024-09-21T10:03:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-21T10:03:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log_max_events": "-1", 2024-09-21T10:03:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log_max_segments": "128", 2024-09-21T10:03:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log_pause": "false", 2024-09-21T10:03:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log_segment_size": "0", 2024-09-21T10:03:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-21T10:03:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-21T10:03:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-21T10:03:28.163 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log_trim_threshold": "128", 2024-09-21T10:03:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-21T10:03:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log_warn_factor": "2.000000", 2024-09-21T10:03:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_max_caps_per_client": "1048576", 2024-09-21T10:03:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_max_completed_flushes": "100000", 2024-09-21T10:03:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_max_completed_requests": "100000", 2024-09-21T10:03:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_max_export_size": "20971520", 2024-09-21T10:03:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_max_file_recover": "32", 2024-09-21T10:03:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_max_purge_files": "64", 2024-09-21T10:03:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_max_purge_ops": "8192", 2024-09-21T10:03:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-21T10:03:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-21T10:03:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_max_snaps_per_dir": "100", 2024-09-21T10:03:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_metrics_update_interval": "2", 2024-09-21T10:03:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_min_caps_per_client": "100", 2024-09-21T10:03:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_min_caps_working_set": "10000", 2024-09-21T10:03:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-21T10:03:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_numa_node": "-1", 2024-09-21T10:03:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-21T10:03:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_op_complaint_time": "30.000000", 2024-09-21T10:03:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_op_history_duration": "600", 2024-09-21T10:03:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_op_history_size": "20", 2024-09-21T10:03:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_op_history_slow_op_size": "20", 2024-09-21T10:03:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-21T10:03:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_op_log_threshold": "5", 2024-09-21T10:03:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_ping_grace": "15", 2024-09-21T10:03:28.165 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_ping_interval": "5", 2024-09-21T10:03:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-21T10:03:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-21T10:03:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_recall_max_caps": "30000", 2024-09-21T10:03:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-21T10:03:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-21T10:03:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-21T10:03:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_recall_warning_threshold": "262144", 2024-09-21T10:03:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-21T10:03:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_replay_interval": "1.000000", 2024-09-21T10:03:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-21T10:03:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-21T10:03:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_root_ino_gid": "0", 2024-09-21T10:03:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_root_ino_uid": "0", 2024-09-21T10:03:28.166 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-21T10:03:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_scrub_stats_review_period": "1", 2024-09-21T10:03:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-21T10:03:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-21T10:03:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-21T10:03:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-21T10:03:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-21T10:03:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-21T10:03:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-21T10:03:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-21T10:03:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-21T10:03:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-21T10:03:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-21T10:03:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_shutdown_check": "0", 2024-09-21T10:03:28.167 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_skip_ino": "0", 2024-09-21T10:03:28.168 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-21T10:03:28.168 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_snap_max_uid": "4294967294", 2024-09-21T10:03:28.168 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_snap_min_uid": "0", 2024-09-21T10:03:28.168 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_snap_rstat": "false", 2024-09-21T10:03:28.169 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_standby_replay_damaged": "false", 2024-09-21T10:03:28.169 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_symlink_recovery": "true", 2024-09-21T10:03:28.169 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-21T10:03:28.169 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_thrash_exports": "0", 2024-09-21T10:03:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_thrash_fragments": "0", 2024-09-21T10:03:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_tick_interval": "5.000000", 2024-09-21T10:03:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_valgrind_exit": "false", 2024-09-21T10:03:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_verify_backtrace": "1", 2024-09-21T10:03:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_verify_scatter": "false", 2024-09-21T10:03:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-21T10:03:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_wipe_sessions": "false", 2024-09-21T10:03:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool_debug": "false", 2024-09-21T10:03:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-21T10:03:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: "memstore_device_bytes": "1073741824", 2024-09-21T10:03:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: "memstore_page_set": "false", 2024-09-21T10:03:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: "memstore_page_size": "65536", 2024-09-21T10:03:28.170 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_client_bytes": "134217728", 2024-09-21T10:03:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_client_messages": "512", 2024-09-21T10:03:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-21T10:03:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-21T10:03:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-21T10:03:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-21T10:03:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_disabled_modules": "", 2024-09-21T10:03:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_enable_op_tracker": "true", 2024-09-21T10:03:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-21T10:03:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_max_pg_creating": "1024", 2024-09-21T10:03:28.171 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_max_pg_num_change": "128", 2024-09-21T10:03:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_mds_bytes": "134217728", 2024-09-21T10:03:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_mds_messages": "128", 2024-09-21T10:03:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-21T10:03:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_mon_bytes": "134217728", 2024-09-21T10:03:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_mon_messages": "128", 2024-09-21T10:03:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-21T10:03:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-21T10:03:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_op_history_duration": "600", 2024-09-21T10:03:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_op_history_size": "20", 2024-09-21T10:03:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-21T10:03:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-21T10:03:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_op_log_threshold": "5", 2024-09-21T10:03:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_osd_bytes": "536870912", 2024-09-21T10:03:28.172 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_osd_messages": "8192", 2024-09-21T10:03:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_pool": "true", 2024-09-21T10:03:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-21T10:03:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_standby_modules": "true", 2024-09-21T10:03:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_stats_period": "5", 2024-09-21T10:03:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_stats_threshold": "5", 2024-09-21T10:03:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_tick_period": "2", 2024-09-21T10:03:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-21T10:03:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-21T10:03:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_allow_pool_delete": "true", 2024-09-21T10:03:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_allow_pool_size_one": "false", 2024-09-21T10:03:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_auth_validate_all_caps": "true", 2024-09-21T10:03:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-21T10:03:28.173 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-21T10:03:28.174 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_bytes": "104857600", 2024-09-21T10:03:28.174 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_directed_command_retry": "2", 2024-09-21T10:03:28.174 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-21T10:03:28.174 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-21T10:03:28.174 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-21T10:03:28.174 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-21T10:03:28.174 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_hunt_parallel": "3", 2024-09-21T10:03:28.174 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_log_interval": "1.000000", 2024-09-21T10:03:28.174 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-21T10:03:28.174 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_ping_interval": "10.000000", 2024-09-21T10:03:28.174 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-21T10:03:28.174 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_target_rank": "-1", 2024-09-21T10:03:28.174 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-21T10:03:28.174 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-21T10:03:28.175 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-21T10:03:28.175 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cluster_log_level": "debug", 2024-09-21T10:03:28.175 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cluster_log_to_file": "true", 2024-09-21T10:03:28.175 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-21T10:03:28.175 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-21T10:03:28.175 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-21T10:03:28.175 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cluster_log_to_journald": "true", 2024-09-21T10:03:28.175 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-21T10:03:28.175 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-21T10:03:28.175 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-21T10:03:28.175 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_compact_on_bootstrap": "false", 2024-09-21T10:03:28.175 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_compact_on_start": "false", 2024-09-21T10:03:28.175 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_compact_on_trim": "true", 2024-09-21T10:03:28.176 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-21T10:03:28.176 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-21T10:03:28.176 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-21T10:03:28.176 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cpu_threads": "4", 2024-09-21T10:03:28.176 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_crush_min_required_version": "hammer", 2024-09-21T10:03:28.176 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_daemon_bytes": "419430400", 2024-09-21T10:03:28.176 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-21T10:03:28.176 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_data_avail_crit": "5", 2024-09-21T10:03:28.176 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_data_avail_warn": "5", 2024-09-21T10:03:28.177 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_data_size_warn": "16106127360", 2024-09-21T10:03:28.177 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-21T10:03:28.177 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-21T10:03:28.177 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_debug_dump_json": "false", 2024-09-21T10:03:28.177 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-21T10:03:28.177 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_debug_dump_transactions": "false", 2024-09-21T10:03:28.177 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_debug_extra_checks": "false", 2024-09-21T10:03:28.177 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-21T10:03:28.177 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-21T10:03:28.177 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_debug_no_require_reef": "false", 2024-09-21T10:03:28.177 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_debug_no_require_squid": "false", 2024-09-21T10:03:28.177 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-21T10:03:28.177 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-21T10:03:28.178 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-21T10:03:28.178 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_down_mkfs_grace": "60", 2024-09-21T10:03:28.178 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_down_uptime_grace": "60", 2024-09-21T10:03:28.178 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_election_default_strategy": "3", 2024-09-21T10:03:28.178 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_election_timeout": "5.000000", 2024-09-21T10:03:28.178 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-21T10:03:28.178 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_elector_ping_divisor": "2", 2024-09-21T10:03:28.178 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-21T10:03:28.178 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_enable_op_tracker": "true", 2024-09-21T10:03:28.178 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_fake_pool_delete": "false", 2024-09-21T10:03:28.179 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_force_quorum_join": "false", 2024-09-21T10:03:28.179 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-21T10:03:28.179 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_globalid_prealloc": "10000", 2024-09-21T10:03:28.179 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_health_detail_to_clog": "true", 2024-09-21T10:03:28.179 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_health_log_update_period": "5", 2024-09-21T10:03:28.179 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_health_max_detail": "50", 2024-09-21T10:03:28.179 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_health_to_clog": "true", 2024-09-21T10:03:28.179 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_health_to_clog_interval": "600", 2024-09-21T10:03:28.179 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-21T10:03:28.179 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_host": "[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0]", 2024-09-21T10:03:28.179 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_host_override": "", 2024-09-21T10:03:28.180 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_initial_members": "", 2024-09-21T10:03:28.180 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-21T10:03:28.180 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-21T10:03:28.180 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-21T10:03:28.180 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-21T10:03:28.180 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-21T10:03:28.180 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_lease": "5.000000", 2024-09-21T10:03:28.180 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-21T10:03:28.180 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-21T10:03:28.180 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_log_full_interval": "50", 2024-09-21T10:03:28.180 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_log_max": "10000", 2024-09-21T10:03:28.180 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_log_max_summary": "50", 2024-09-21T10:03:28.180 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-21T10:03:28.180 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_max_log_epochs": "500", 2024-09-21T10:03:28.181 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-21T10:03:28.181 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-21T10:03:28.181 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-21T10:03:28.181 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_max_osd": "10000", 2024-09-21T10:03:28.181 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_max_pg_per_osd": "10000", 2024-09-21T10:03:28.181 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_max_pool_pg_num": "65536", 2024-09-21T10:03:28.181 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-21T10:03:28.181 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-21T10:03:28.181 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_mds_force_trim_to": "0", 2024-09-21T10:03:28.181 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_mds_skip_sanity": "true", 2024-09-21T10:03:28.181 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_memory_autotune": "true", 2024-09-21T10:03:28.181 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_memory_target": "2147483648", 2024-09-21T10:03:28.181 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_mgr_beacon_grace": "30", 2024-09-21T10:03:28.182 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-21T10:03:28.182 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_mgr_digest_period": "5", 2024-09-21T10:03:28.182 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_mgr_inactive_grace": "60", 2024-09-21T10:03:28.182 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-21T10:03:28.182 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-21T10:03:28.182 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_min_osdmap_epochs": "500", 2024-09-21T10:03:28.182 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-21T10:03:28.182 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-21T10:03:28.182 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_op_complaint_time": "30", 2024-09-21T10:03:28.182 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_op_history_duration": "600", 2024-09-21T10:03:28.182 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_op_history_size": "20", 2024-09-21T10:03:28.183 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_op_history_slow_op_size": "20", 2024-09-21T10:03:28.183 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-21T10:03:28.183 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_op_log_threshold": "5", 2024-09-21T10:03:28.183 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-21T10:03:28.183 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-21T10:03:28.183 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-21T10:03:28.183 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_auto_mark_in": "false", 2024-09-21T10:03:28.183 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-21T10:03:28.183 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-21T10:03:28.183 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-21T10:03:28.183 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_cache_size": "500", 2024-09-21T10:03:28.184 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-21T10:03:28.184 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-21T10:03:28.184 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-21T10:03:28.184 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_down_out_interval": "600", 2024-09-21T10:03:28.184 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-21T10:03:28.184 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-21T10:03:28.184 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_force_trim_to": "0", 2024-09-21T10:03:28.184 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-21T10:03:28.184 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-21T10:03:28.184 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-21T10:03:28.184 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-21T10:03:28.184 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-21T10:03:28.184 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-21T10:03:28.184 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-21T10:03:28.185 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_min_down_reporters": "2", 2024-09-21T10:03:28.185 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-21T10:03:28.185 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-21T10:03:28.185 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-21T10:03:28.185 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-21T10:03:28.185 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-21T10:03:28.185 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-21T10:03:28.185 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_report_timeout": "900", 2024-09-21T10:03:28.185 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-21T10:03:28.185 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-21T10:03:28.185 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-21T10:03:28.185 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-21T10:03:28.185 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-21T10:03:28.186 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-21T10:03:28.186 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-21T10:03:28.186 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-21T10:03:28.186 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-21T10:03:28.186 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_pg_stuck_threshold": "60", 2024-09-21T10:03:28.186 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-21T10:03:28.186 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-21T10:03:28.186 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-21T10:03:28.186 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-21T10:03:28.186 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-21T10:03:28.186 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-21T10:03:28.186 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_probe_timeout": "2.000000", 2024-09-21T10:03:28.186 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_reweight_max_change": "0.050000", 2024-09-21T10:03:28.186 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_reweight_max_osds": "4", 2024-09-21T10:03:28.187 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-21T10:03:28.187 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-21T10:03:28.187 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-21T10:03:28.187 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-21T10:03:28.187 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-21T10:03:28.187 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_scrub_interval": "86400", 2024-09-21T10:03:28.187 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_scrub_max_keys": "100", 2024-09-21T10:03:28.187 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_scrub_timeout": "300", 2024-09-21T10:03:28.187 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_session_timeout": "300", 2024-09-21T10:03:28.187 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_smart_report_timeout": "5", 2024-09-21T10:03:28.187 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_stat_smooth_intervals": "6", 2024-09-21T10:03:28.187 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-21T10:03:28.187 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_stretch_pool_min_size": "2", 2024-09-21T10:03:28.188 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_stretch_pool_size": "4", 2024-09-21T10:03:28.188 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-21T10:03:28.188 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-21T10:03:28.188 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_sync_debug": "false", 2024-09-21T10:03:28.188 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-21T10:03:28.188 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-21T10:03:28.188 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_sync_provider_kill_at": "0", 2024-09-21T10:03:28.188 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_sync_requester_kill_at": "0", 2024-09-21T10:03:28.188 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_sync_timeout": "60.000000", 2024-09-21T10:03:28.188 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_target_pg_per_osd": "100", 2024-09-21T10:03:28.188 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_tick_interval": "5", 2024-09-21T10:03:28.188 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_timecheck_interval": "300.000000", 2024-09-21T10:03:28.188 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-21T10:03:28.188 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_use_min_delay_socket": "false", 2024-09-21T10:03:28.189 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_older_version_delay": "604800", 2024-09-21T10:03:28.189 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-21T10:03:28.189 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-21T10:03:28.189 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-21T10:03:28.189 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-21T10:03:28.189 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-21T10:03:28.189 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-21T10:03:28.189 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-21T10:03:28.189 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_misplaced": "false", 2024-09-21T10:03:28.189 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-21T10:03:28.189 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_older_version": "true", 2024-09-21T10:03:28.189 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-21T10:03:28.189 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-21T10:03:28.190 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-21T10:03:28.190 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-21T10:03:28.190 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-21T10:03:28.190 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-21T10:03:28.190 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-21T10:03:28.190 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-21T10:03:28.190 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-21T10:03:28.190 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": "", 2024-09-21T10:03:28.190 INFO:teuthology.orchestra.run.smithi098.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T10:03:28.190 INFO:teuthology.orchestra.run.smithi098.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-21T10:03:28.190 INFO:teuthology.orchestra.run.smithi098.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-21T10:03:28.190 INFO:teuthology.orchestra.run.smithi098.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T10:03:28.190 INFO:teuthology.orchestra.run.smithi098.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-21T10:03:28.191 INFO:teuthology.orchestra.run.smithi098.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-21T10:03:28.191 INFO:teuthology.orchestra.run.smithi098.stdout: "motr_tracing_enabled": "false", 2024-09-21T10:03:28.191 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_op_threads": "3", 2024-09-21T10:03:28.191 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-21T10:03:28.191 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_cm": "false", 2024-09-21T10:03:28.191 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_device_name": "", 2024-09-21T10:03:28.191 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_dscp": "96", 2024-09-21T10:03:28.191 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-21T10:03:28.191 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-21T10:03:28.191 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_local_gid": "", 2024-09-21T10:03:28.191 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-21T10:03:28.191 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_port_num": "1", 2024-09-21T10:03:28.191 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-21T10:03:28.191 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-21T10:03:28.192 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-21T10:03:28.192 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-21T10:03:28.192 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_sl": "3", 2024-09-21T10:03:28.192 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_support_srq": "true", 2024-09-21T10:03:28.192 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_type": "ib", 2024-09-21T10:03:28.192 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_reap_threshold": "5", 2024-09-21T10:03:28.192 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_bind_before_connect": "false", 2024-09-21T10:03:28.192 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_bind_ipv4": "true", 2024-09-21T10:03:28.192 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_bind_ipv6": "false", 2024-09-21T10:03:28.192 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_bind_msgr1": "true", 2024-09-21T10:03:28.192 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_bind_msgr2": "true", 2024-09-21T10:03:28.192 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_bind_port_max": "7568", 2024-09-21T10:03:28.192 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_bind_port_min": "6800", 2024-09-21T10:03:28.192 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-21T10:03:28.193 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_bind_retry_count": "3", 2024-09-21T10:03:28.193 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_bind_retry_delay": "5", 2024-09-21T10:03:28.193 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_blackhole_client": "false", 2024-09-21T10:03:28.193 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_blackhole_mds": "false", 2024-09-21T10:03:28.193 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_blackhole_mgr": "false", 2024-09-21T10:03:28.193 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_blackhole_mon": "false", 2024-09-21T10:03:28.193 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_blackhole_osd": "false", 2024-09-21T10:03:28.193 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_client_mode": "crc secure", 2024-09-21T10:03:28.193 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-21T10:03:28.193 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_cluster_mode": "crc secure", 2024-09-21T10:03:28.193 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_cluster_type": "", 2024-09-21T10:03:28.193 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_compress_secure": "false", 2024-09-21T10:03:28.193 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_connection_idle_timeout": "900", 2024-09-21T10:03:28.193 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_connection_ready_timeout": "10", 2024-09-21T10:03:28.194 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_crc_data": "true", 2024-09-21T10:03:28.194 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_crc_header": "true", 2024-09-21T10:03:28.194 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_die_on_bad_msg": "false", 2024-09-21T10:03:28.194 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_die_on_bug": "true", 2024-09-21T10:03:28.194 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_die_on_old_message": "true", 2024-09-21T10:03:28.194 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_die_on_skipped_message": "false", 2024-09-21T10:03:28.194 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-21T10:03:28.194 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-21T10:03:28.194 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_coremask": "0xF", 2024-09-21T10:03:28.194 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-21T10:03:28.195 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-21T10:03:28.195 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_enable_tso": "true", 2024-09-21T10:03:28.195 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-21T10:03:28.195 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-21T10:03:28.195 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_hugepages": "", 2024-09-21T10:03:28.195 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-21T10:03:28.195 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-21T10:03:28.195 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_lro": "true", 2024-09-21T10:03:28.195 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_memory_channel": "4", 2024-09-21T10:03:28.196 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-21T10:03:28.196 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_pmd": "", 2024-09-21T10:03:28.196 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_port_id": "0", 2024-09-21T10:03:28.196 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-21T10:03:28.196 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-21T10:03:28.196 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dump_on_send": "false", 2024-09-21T10:03:28.196 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_initial_backoff": "0.200000", 2024-09-21T10:03:28.197 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_inject_delay_max": "1.000000", 2024-09-21T10:03:28.197 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-21T10:03:28.197 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_inject_delay_type": "", 2024-09-21T10:03:28.197 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-21T10:03:28.197 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_inject_network_congestion": "0", 2024-09-21T10:03:28.197 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_inject_socket_failures": "0", 2024-09-21T10:03:28.197 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_learn_addr_from_peer": "true", 2024-09-21T10:03:28.197 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_max_accept_failures": "4", 2024-09-21T10:03:28.197 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_max_backoff": "15.000000", 2024-09-21T10:03:28.197 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_mon_client_mode": "secure crc", 2024-09-21T10:03:28.197 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-21T10:03:28.197 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_mon_service_mode": "secure crc", 2024-09-21T10:03:28.198 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_osd_compress_min_size": "1024", 2024-09-21T10:03:28.198 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_osd_compress_mode": "none", 2024-09-21T10:03:28.198 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-21T10:03:28.198 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-21T10:03:28.198 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_pq_min_cost": "65536", 2024-09-21T10:03:28.198 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_public_type": "", 2024-09-21T10:03:28.198 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_service_mode": "crc secure", 2024-09-21T10:03:28.198 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_tcp_listen_backlog": "512", 2024-09-21T10:03:28.198 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_tcp_nodelay": "true", 2024-09-21T10:03:28.198 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-21T10:03:28.199 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_tcp_rcvbuf": "0", 2024-09-21T10:03:28.199 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-21T10:03:28.199 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_type": "async+posix", 2024-09-21T10:03:28.199 INFO:teuthology.orchestra.run.smithi098.stdout: "no_config_file": "false", 2024-09-21T10:03:28.199 INFO:teuthology.orchestra.run.smithi098.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-21T10:03:28.199 INFO:teuthology.orchestra.run.smithi098.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-21T10:03:28.199 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter_completion_locks_per_session": "32", 2024-09-21T10:03:28.199 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-21T10:03:28.199 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-21T10:03:28.199 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter_inflight_ops": "1024", 2024-09-21T10:03:28.199 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-21T10:03:28.199 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-21T10:03:28.199 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter_tick_interval": "5.000000", 2024-09-21T10:03:28.200 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter_timeout": "10.000000", 2024-09-21T10:03:28.200 INFO:teuthology.orchestra.run.smithi098.stdout: "objectstore_blackhole": "false", 2024-09-21T10:03:28.200 INFO:teuthology.orchestra.run.smithi098.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-21T10:03:28.200 INFO:teuthology.orchestra.run.smithi098.stdout: "openssl_engine_opts": "", 2024-09-21T10:03:28.200 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_agent_delay_time": "5.000000", 2024-09-21T10:03:28.200 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_agent_hist_halflife": "1000", 2024-09-21T10:03:28.200 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_agent_max_low_ops": "2", 2024-09-21T10:03:28.200 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_agent_max_ops": "4", 2024-09-21T10:03:28.200 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-21T10:03:28.200 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-21T10:03:28.200 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_agent_slop": "0.020000", 2024-09-21T10:03:28.200 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-21T10:03:28.200 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-21T10:03:28.200 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_asio_thread_count": "2", 2024-09-21T10:03:28.201 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_async_recovery_min_cost": "100", 2024-09-21T10:03:28.201 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-21T10:03:28.201 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-21T10:03:28.201 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_backfill_scan_max": "512", 2024-09-21T10:03:28.201 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_backfill_scan_min": "64", 2024-09-21T10:03:28.201 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_backoff_on_degraded": "false", 2024-09-21T10:03:28.201 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_backoff_on_peering": "false", 2024-09-21T10:03:28.201 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_backoff_on_unfound": "true", 2024-09-21T10:03:28.201 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_beacon_report_interval": "300", 2024-09-21T10:03:28.201 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bench_duration": "30", 2024-09-21T10:03:28.201 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-21T10:03:28.201 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bench_max_block_size": "67108864", 2024-09-21T10:03:28.201 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-21T10:03:28.201 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_blkin_trace_all": "false", 2024-09-21T10:03:28.202 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-21T10:03:28.202 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-21T10:03:28.202 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-21T10:03:28.202 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-21T10:03:28.202 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_check_for_log_corruption": "false", 2024-09-21T10:03:28.202 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-21T10:03:28.202 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T10:03:28.202 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-21T10:03:28.202 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T10:03:28.202 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_class_update_on_start": "true", 2024-09-21T10:03:28.202 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_client_message_cap": "256", 2024-09-21T10:03:28.202 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_client_message_size_cap": "524288000", 2024-09-21T10:03:28.202 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_client_op_priority": "63", 2024-09-21T10:03:28.203 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_client_watch_timeout": "30", 2024-09-21T10:03:28.203 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_command_max_records": "256", 2024-09-21T10:03:28.203 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_compact_on_start": "false", 2024-09-21T10:03:28.203 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-21T10:03:28.203 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-21T10:03:28.203 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-21T10:03:28.203 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_crush_update_on_start": "true", 2024-09-21T10:03:28.203 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_crush_update_weight_set": "true", 2024-09-21T10:03:28.203 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-21T10:03:28.203 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-21T10:03:28.203 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-21T10:03:28.203 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-21T10:03:28.203 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-21T10:03:28.203 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_feed_pullee": "-1", 2024-09-21T10:03:28.204 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-21T10:03:28.204 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-21T10:03:28.204 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-21T10:03:28.204 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_misdirected_ops": "false", 2024-09-21T10:03:28.204 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_no_acting_change": "false", 2024-09-21T10:03:28.204 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_no_purge_strays": "false", 2024-09-21T10:03:28.204 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_op_order": "false", 2024-09-21T10:03:28.204 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-21T10:03:28.204 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-21T10:03:28.204 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-21T10:03:28.204 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-21T10:03:28.204 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_shutdown": "false", 2024-09-21T10:03:28.204 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-21T10:03:28.204 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-21T10:03:28.205 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_trim_objects": "false", 2024-09-21T10:03:28.205 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-21T10:03:28.205 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-21T10:03:28.205 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_verify_snaps": "false", 2024-09-21T10:03:28.205 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-21T10:03:28.205 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-21T10:03:28.205 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_deep_scrub_keys": "1024", 2024-09-21T10:03:28.205 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-21T10:03:28.205 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-21T10:03:28.205 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-21T10:03:28.205 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_deep_scrub_stride": "524288", 2024-09-21T10:03:28.205 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-21T10:03:28.205 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-21T10:03:28.206 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_default_notify_timeout": "30", 2024-09-21T10:03:28.206 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_delete_sleep": "0.000000", 2024-09-21T10:03:28.206 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-21T10:03:28.206 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-21T10:03:28.206 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-21T10:03:28.206 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_discard_disconnected_ops": "true", 2024-09-21T10:03:28.206 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_ec_partial_reads": "true", 2024-09-21T10:03:28.206 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_enable_op_tracker": "true", 2024-09-21T10:03:28.206 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-21T10:03:28.206 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-21T10:03:28.206 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-21T10:03:28.206 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-21T10:03:28.206 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_fast_info": "true", 2024-09-21T10:03:28.207 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_fast_shutdown": "true", 2024-09-21T10:03:28.207 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-21T10:03:28.207 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-21T10:03:28.207 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-21T10:03:28.207 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-21T10:03:28.207 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-21T10:03:28.207 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_function_tracing": "false", 2024-09-21T10:03:28.207 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_heartbeat_grace": "20", 2024-09-21T10:03:28.207 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_heartbeat_interval": "6", 2024-09-21T10:03:28.207 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-21T10:03:28.207 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_heartbeat_min_peers": "10", 2024-09-21T10:03:28.207 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_heartbeat_min_size": "2000", 2024-09-21T10:03:28.207 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_heartbeat_stale": "600", 2024-09-21T10:03:28.207 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-21T10:03:28.208 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_hit_set_max_size": "100000", 2024-09-21T10:03:28.208 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_hit_set_min_size": "1000", 2024-09-21T10:03:28.208 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-21T10:03:28.208 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-21T10:03:28.208 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-21T10:03:28.208 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-21T10:03:28.208 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-21T10:03:28.208 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-21T10:03:28.208 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_journal_size": "5120", 2024-09-21T10:03:28.208 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_kill_backfill_at": "0", 2024-09-21T10:03:28.208 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-21T10:03:28.208 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_map_cache_size": "50", 2024-09-21T10:03:28.208 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_map_dedup": "true", 2024-09-21T10:03:28.209 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_map_message_max": "40", 2024-09-21T10:03:28.209 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-21T10:03:28.209 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_map_share_max_epochs": "40", 2024-09-21T10:03:28.209 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_attr_name_len": "100", 2024-09-21T10:03:28.209 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_attr_size": "0", 2024-09-21T10:03:28.209 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_backfills": "1", 2024-09-21T10:03:28.209 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_markdown_count": "5", 2024-09-21T10:03:28.209 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_markdown_period": "600", 2024-09-21T10:03:28.209 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_object_name_len": "2048", 2024-09-21T10:03:28.209 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_object_namespace_len": "256", 2024-09-21T10:03:28.209 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_object_size": "134217728", 2024-09-21T10:03:28.209 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-21T10:03:28.209 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-21T10:03:28.209 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_pg_blocked_by": "16", 2024-09-21T10:03:28.210 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_pg_log_entries": "10000", 2024-09-21T10:03:28.210 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-21T10:03:28.210 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_pgls": "1024", 2024-09-21T10:03:28.210 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_push_cost": "8388608", 2024-09-21T10:03:28.210 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_push_objects": "10", 2024-09-21T10:03:28.210 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_scrubs": "3", 2024-09-21T10:03:28.210 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-21T10:03:28.210 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_trimming_pgs": "2", 2024-09-21T10:03:28.210 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_write_op_reply_len": "64", 2024-09-21T10:03:28.210 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_write_size": "90", 2024-09-21T10:03:28.210 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-21T10:03:28.210 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-21T10:03:28.210 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-21T10:03:28.211 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-21T10:03:28.211 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-21T10:03:28.211 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-21T10:03:28.211 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-21T10:03:28.211 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-21T10:03:28.211 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_profile": "balanced", 2024-09-21T10:03:28.211 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-21T10:03:28.211 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-21T10:03:28.211 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-21T10:03:28.211 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-21T10:03:28.211 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-21T10:03:28.211 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-21T10:03:28.211 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-21T10:03:28.211 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-21T10:03:28.211 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-21T10:03:28.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-21T10:03:28.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-21T10:03:28.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_memory_base": "805306368", 2024-09-21T10:03:28.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_memory_cache_min": "134217728", 2024-09-21T10:03:28.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-21T10:03:28.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-21T10:03:28.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_memory_target": "4294967296", 2024-09-21T10:03:28.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_memory_target_autotune": "false", 2024-09-21T10:03:28.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-21T10:03:28.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_min_pg_log_entries": "250", 2024-09-21T10:03:28.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_min_recovery_priority": "0", 2024-09-21T10:03:28.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-21T10:03:28.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-21T10:03:28.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mon_report_interval": "5", 2024-09-21T10:03:28.213 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-21T10:03:28.213 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-21T10:03:28.213 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_num_cache_shards": "32", 2024-09-21T10:03:28.213 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_num_op_tracker_shard": "32", 2024-09-21T10:03:28.213 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_numa_auto_affinity": "true", 2024-09-21T10:03:28.213 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_numa_node": "-1", 2024-09-21T10:03:28.213 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_numa_prefer_iface": "true", 2024-09-21T10:03:28.213 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-21T10:03:28.213 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_objecter_finishers": "1", 2024-09-21T10:03:28.213 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_objectstore": "bluestore", 2024-09-21T10:03:28.213 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_objectstore_fuse": "false", 2024-09-21T10:03:28.213 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-21T10:03:28.213 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_objectstore_tracing": "false", 2024-09-21T10:03:28.213 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_complaint_time": "30.000000", 2024-09-21T10:03:28.214 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_history_duration": "600", 2024-09-21T10:03:28.214 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_history_size": "20", 2024-09-21T10:03:28.214 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_history_slow_op_size": "20", 2024-09-21T10:03:28.214 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-21T10:03:28.214 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_log_threshold": "5", 2024-09-21T10:03:28.214 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_num_shards": "0", 2024-09-21T10:03:28.214 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_num_shards_hdd": "5", 2024-09-21T10:03:28.214 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_num_shards_ssd": "8", 2024-09-21T10:03:28.214 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-21T10:03:28.214 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-21T10:03:28.214 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-21T10:03:28.214 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-21T10:03:28.214 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_pq_min_cost": "65536", 2024-09-21T10:03:28.215 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-21T10:03:28.215 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_queue_cut_off": "high", 2024-09-21T10:03:28.215 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-21T10:03:28.215 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_thread_timeout": "15", 2024-09-21T10:03:28.215 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_open_classes_on_start": "true", 2024-09-21T10:03:28.215 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_os_flags": "0", 2024-09-21T10:03:28.215 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_peering_op_priority": "255", 2024-09-21T10:03:28.215 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pg_delete_cost": "1048576", 2024-09-21T10:03:28.215 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pg_delete_priority": "5", 2024-09-21T10:03:28.215 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-21T10:03:28.215 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-21T10:03:28.215 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-21T10:03:28.215 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pg_log_trim_max": "10000", 2024-09-21T10:03:28.215 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pg_log_trim_min": "100", 2024-09-21T10:03:28.216 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-21T10:03:28.216 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-21T10:03:28.216 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-21T10:03:28.216 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-21T10:03:28.216 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-21T10:03:28.216 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-21T10:03:28.216 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-21T10:03:28.216 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-21T10:03:28.216 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-21T10:03:28.216 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-21T10:03:28.216 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_crimson": "false", 2024-09-21T10:03:28.216 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-21T10:03:28.216 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-21T10:03:28.217 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-21T10:03:28.217 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-21T10:03:28.217 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-21T10:03:28.217 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-21T10:03:28.217 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-21T10:03:28.217 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-21T10:03:28.217 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_flags": "0", 2024-09-21T10:03:28.217 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-21T10:03:28.217 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_min_size": "0", 2024-09-21T10:03:28.217 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-21T10:03:28.217 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_pg_num": "32", 2024-09-21T10:03:28.217 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_pgp_num": "0", 2024-09-21T10:03:28.217 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-21T10:03:28.217 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_read_ratio": "70", 2024-09-21T10:03:28.218 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_size": "3", 2024-09-21T10:03:28.218 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_type": "replicated", 2024-09-21T10:03:28.218 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-21T10:03:28.218 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-21T10:03:28.218 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_push_per_object_cost": "1000", 2024-09-21T10:03:28.218 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-21T10:03:28.218 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recover_clone_overlap": "true", 2024-09-21T10:03:28.218 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-21T10:03:28.218 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_cost": "20971520", 2024-09-21T10:03:28.218 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-21T10:03:28.218 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_max_active": "0", 2024-09-21T10:03:28.218 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-21T10:03:28.218 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-21T10:03:28.219 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-21T10:03:28.219 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-21T10:03:28.219 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_max_single_start": "1", 2024-09-21T10:03:28.219 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_op_priority": "3", 2024-09-21T10:03:28.219 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-21T10:03:28.219 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_priority": "5", 2024-09-21T10:03:28.219 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-21T10:03:28.219 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_sleep": "0.000000", 2024-09-21T10:03:28.219 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-21T10:03:28.219 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-21T10:03:28.219 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-21T10:03:28.219 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_repair_during_recovery": "false", 2024-09-21T10:03:28.219 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_requested_scrub_priority": "5", 2024-09-21T10:03:28.220 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-21T10:03:28.220 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-21T10:03:28.220 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_auto_repair": "false", 2024-09-21T10:03:28.220 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-21T10:03:28.220 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-21T10:03:28.220 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_begin_hour": "0", 2024-09-21T10:03:28.220 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_begin_week_day": "0", 2024-09-21T10:03:28.220 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_chunk_max": "15", 2024-09-21T10:03:28.220 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_chunk_min": "5", 2024-09-21T10:03:28.220 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_cost": "52428800", 2024-09-21T10:03:28.220 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-21T10:03:28.220 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_during_recovery": "false", 2024-09-21T10:03:28.220 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_end_hour": "0", 2024-09-21T10:03:28.220 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_end_week_day": "0", 2024-09-21T10:03:28.221 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_event_cost": "4096", 2024-09-21T10:03:28.221 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-21T10:03:28.221 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-21T10:03:28.221 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_invalid_stats": "true", 2024-09-21T10:03:28.221 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-21T10:03:28.221 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-21T10:03:28.221 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_max_preemptions": "5", 2024-09-21T10:03:28.221 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-21T10:03:28.221 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_priority": "5", 2024-09-21T10:03:28.221 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_retry_after_noscrub": "60", 2024-09-21T10:03:28.221 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_retry_delay": "30", 2024-09-21T10:03:28.221 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_retry_new_interval": "10", 2024-09-21T10:03:28.221 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_retry_pg_state": "60", 2024-09-21T10:03:28.221 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_retry_trimming": "10", 2024-09-21T10:03:28.222 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_sleep": "0.000000", 2024-09-21T10:03:28.222 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-21T10:03:28.222 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-21T10:03:28.222 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-21T10:03:28.222 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-21T10:03:28.222 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_skip_data_digest": "false", 2024-09-21T10:03:28.222 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_smart_report_timeout": "5", 2024-09-21T10:03:28.222 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_snap_trim_cost": "1048576", 2024-09-21T10:03:28.222 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_snap_trim_priority": "5", 2024-09-21T10:03:28.222 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-21T10:03:28.222 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-21T10:03:28.222 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-21T10:03:28.222 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-21T10:03:28.223 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-21T10:03:28.223 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-21T10:03:28.223 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-21T10:03:28.223 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_target_transaction_size": "30", 2024-09-21T10:03:28.223 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-21T10:03:28.223 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-21T10:03:28.223 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-21T10:03:28.223 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-21T10:03:28.223 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-21T10:03:28.223 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-21T10:03:28.223 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-21T10:03:28.223 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-21T10:03:28.223 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-21T10:03:28.223 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-21T10:03:28.224 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_tracing": "false", 2024-09-21T10:03:28.224 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_use_stale_snap": "false", 2024-09-21T10:03:28.224 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-21T10:03:28.224 INFO:teuthology.orchestra.run.smithi098.stdout: "osdc_blkin_trace_all": "false", 2024-09-21T10:03:28.224 INFO:teuthology.orchestra.run.smithi098.stdout: "paxos_kill_at": "0", 2024-09-21T10:03:28.224 INFO:teuthology.orchestra.run.smithi098.stdout: "paxos_max_join_drift": "10", 2024-09-21T10:03:28.224 INFO:teuthology.orchestra.run.smithi098.stdout: "paxos_min": "500", 2024-09-21T10:03:28.224 INFO:teuthology.orchestra.run.smithi098.stdout: "paxos_min_wait": "0.050000", 2024-09-21T10:03:28.224 INFO:teuthology.orchestra.run.smithi098.stdout: "paxos_propose_interval": "1.000000", 2024-09-21T10:03:28.224 INFO:teuthology.orchestra.run.smithi098.stdout: "paxos_service_trim_max": "500", 2024-09-21T10:03:28.224 INFO:teuthology.orchestra.run.smithi098.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-21T10:03:28.224 INFO:teuthology.orchestra.run.smithi098.stdout: "paxos_service_trim_min": "250", 2024-09-21T10:03:28.224 INFO:teuthology.orchestra.run.smithi098.stdout: "paxos_stash_full_interval": "25", 2024-09-21T10:03:28.224 INFO:teuthology.orchestra.run.smithi098.stdout: "paxos_trim_max": "500", 2024-09-21T10:03:28.225 INFO:teuthology.orchestra.run.smithi098.stdout: "paxos_trim_min": "250", 2024-09-21T10:03:28.225 INFO:teuthology.orchestra.run.smithi098.stdout: "perf": "true", 2024-09-21T10:03:28.225 INFO:teuthology.orchestra.run.smithi098.stdout: "pid_file": "", 2024-09-21T10:03:28.225 INFO:teuthology.orchestra.run.smithi098.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-21T10:03:28.225 INFO:teuthology.orchestra.run.smithi098.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-21T10:03:28.225 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "-", 2024-09-21T10:03:28.225 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrv": "", 2024-09-21T10:03:28.225 INFO:teuthology.orchestra.run.smithi098.stdout: "public_bind_addr": "-", 2024-09-21T10:03:28.225 INFO:teuthology.orchestra.run.smithi098.stdout: "public_network": "172.21.0.0/20", 2024-09-21T10:03:28.225 INFO:teuthology.orchestra.run.smithi098.stdout: "public_network_interface": "", 2024-09-21T10:03:28.225 INFO:teuthology.orchestra.run.smithi098.stdout: "qat_compressor_busy_polling": "false", 2024-09-21T10:03:28.225 INFO:teuthology.orchestra.run.smithi098.stdout: "qat_compressor_enabled": "false", 2024-09-21T10:03:28.225 INFO:teuthology.orchestra.run.smithi098.stdout: "qat_compressor_session_max_number": "256", 2024-09-21T10:03:28.225 INFO:teuthology.orchestra.run.smithi098.stdout: "rados_mon_op_timeout": "0", 2024-09-21T10:03:28.226 INFO:teuthology.orchestra.run.smithi098.stdout: "rados_osd_op_timeout": "0", 2024-09-21T10:03:28.226 INFO:teuthology.orchestra.run.smithi098.stdout: "rados_tracing": "false", 2024-09-21T10:03:28.226 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_atime_update_interval": "60", 2024-09-21T10:03:28.226 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-21T10:03:28.226 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_balance_parent_reads": "false", 2024-09-21T10:03:28.226 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_balance_snap_reads": "false", 2024-09-21T10:03:28.226 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_blkin_trace_all": "false", 2024-09-21T10:03:28.226 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-21T10:03:28.226 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-21T10:03:28.226 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_cache": "true", 2024-09-21T10:03:28.226 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-21T10:03:28.226 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-21T10:03:28.226 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-21T10:03:28.227 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-21T10:03:28.227 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_cache_policy": "writearound", 2024-09-21T10:03:28.227 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_cache_size": "33554432", 2024-09-21T10:03:28.227 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-21T10:03:28.227 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-21T10:03:28.227 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_clone_copy_on_read": "false", 2024-09-21T10:03:28.227 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_compression_hint": "none", 2024-09-21T10:03:28.227 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_concurrent_management_ops": "10", 2024-09-21T10:03:28.227 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-21T10:03:28.227 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_default_clone_format": "auto", 2024-09-21T10:03:28.227 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_default_data_pool": "", 2024-09-21T10:03:28.227 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_default_features": "61", 2024-09-21T10:03:28.227 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_default_format": "2", 2024-09-21T10:03:28.227 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_default_map_options": "", 2024-09-21T10:03:28.228 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_default_order": "22", 2024-09-21T10:03:28.228 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_default_pool": "rbd", 2024-09-21T10:03:28.228 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-21T10:03:28.228 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_default_stripe_count": "0", 2024-09-21T10:03:28.228 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_default_stripe_unit": "0", 2024-09-21T10:03:28.228 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-21T10:03:28.228 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-21T10:03:28.228 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-21T10:03:28.228 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_enable_alloc_hint": "true", 2024-09-21T10:03:28.228 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-21T10:03:28.228 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_io_scheduler": "simple", 2024-09-21T10:03:28.228 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-21T10:03:28.228 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-21T10:03:28.229 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-21T10:03:28.229 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-21T10:03:28.229 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-21T10:03:28.229 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-21T10:03:28.229 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-21T10:03:28.229 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-21T10:03:28.229 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-21T10:03:28.229 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_journal_order": "24", 2024-09-21T10:03:28.229 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_journal_pool": "", 2024-09-21T10:03:28.229 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_journal_splay_width": "4", 2024-09-21T10:03:28.229 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_localize_parent_reads": "false", 2024-09-21T10:03:28.229 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_localize_snap_reads": "false", 2024-09-21T10:03:28.230 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-21T10:03:28.230 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-21T10:03:28.230 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-21T10:03:28.230 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-21T10:03:28.230 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-21T10:03:28.230 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-21T10:03:28.230 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-21T10:03:28.230 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-21T10:03:28.230 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-21T10:03:28.230 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-21T10:03:28.230 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-21T10:03:28.230 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-21T10:03:28.230 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-21T10:03:28.231 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-21T10:03:28.231 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-21T10:03:28.231 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-21T10:03:28.231 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-21T10:03:28.231 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-21T10:03:28.231 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-21T10:03:28.231 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-21T10:03:28.231 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-21T10:03:28.231 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-21T10:03:28.231 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-21T10:03:28.231 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-21T10:03:28.231 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-21T10:03:28.231 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-21T10:03:28.231 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-21T10:03:28.232 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-21T10:03:28.232 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-21T10:03:28.232 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-21T10:03:28.232 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-21T10:03:28.232 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mtime_update_interval": "60", 2024-09-21T10:03:28.232 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_non_blocking_aio": "true", 2024-09-21T10:03:28.232 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_op_thread_timeout": "60", 2024-09-21T10:03:28.232 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_op_threads": "1", 2024-09-21T10:03:28.232 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_parent_cache_enabled": "false", 2024-09-21T10:03:28.232 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-21T10:03:28.232 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-21T10:03:28.232 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-21T10:03:28.232 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_plugins": "", 2024-09-21T10:03:28.232 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_bps_burst": "0", 2024-09-21T10:03:28.233 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-21T10:03:28.233 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_bps_limit": "0", 2024-09-21T10:03:28.233 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_exclude_ops": "0", 2024-09-21T10:03:28.233 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_iops_burst": "0", 2024-09-21T10:03:28.233 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-21T10:03:28.233 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_iops_limit": "0", 2024-09-21T10:03:28.233 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-21T10:03:28.233 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-21T10:03:28.233 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-21T10:03:28.233 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-21T10:03:28.233 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-21T10:03:28.233 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-21T10:03:28.233 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-21T10:03:28.234 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-21T10:03:28.234 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-21T10:03:28.234 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-21T10:03:28.234 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-21T10:03:28.234 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-21T10:03:28.234 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-21T10:03:28.234 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-21T10:03:28.234 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_read_from_replica_policy": "default", 2024-09-21T10:03:28.234 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-21T10:03:28.234 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-21T10:03:28.234 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-21T10:03:28.234 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-21T10:03:28.234 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_skip_partial_discard": "true", 2024-09-21T10:03:28.235 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-21T10:03:28.235 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_tracing": "false", 2024-09-21T10:03:28.235 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_validate_names": "true", 2024-09-21T10:03:28.235 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_validate_pool": "true", 2024-09-21T10:03:28.235 INFO:teuthology.orchestra.run.smithi098.stdout: "restapi_base_url": "", 2024-09-21T10:03:28.235 INFO:teuthology.orchestra.run.smithi098.stdout: "restapi_log_level": "", 2024-09-21T10:03:28.235 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-21T10:03:28.235 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-21T10:03:28.235 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_acl_grants_max_num": "100", 2024-09-21T10:03:28.235 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_admin_entry": "admin", 2024-09-21T10:03:28.235 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-21T10:03:28.235 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_asio_assert_yielding": "false", 2024-09-21T10:03:28.235 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_backend_store": "rados", 2024-09-21T10:03:28.235 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_barbican_url": "", 2024-09-21T10:03:28.236 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_beast_enable_async": "true", 2024-09-21T10:03:28.236 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_bucket_counters_cache": "false", 2024-09-21T10:03:28.236 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-21T10:03:28.236 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-21T10:03:28.236 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-21T10:03:28.236 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-21T10:03:28.236 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-21T10:03:28.236 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-21T10:03:28.236 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-21T10:03:28.236 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-21T10:03:28.236 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_cache_enabled": "true", 2024-09-21T10:03:28.236 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_cache_expiry_interval": "900", 2024-09-21T10:03:28.236 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_cache_lru_size": "25000", 2024-09-21T10:03:28.236 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_config_store": "rados", 2024-09-21T10:03:28.236 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_content_length_compat": "false", 2024-09-21T10:03:28.237 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_copy_obj_progress": "true", 2024-09-21T10:03:28.237 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-21T10:03:28.237 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_cors_rules_max_num": "100", 2024-09-21T10:03:28.237 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_cross_domain_policy": "", 2024-09-21T10:03:28.237 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-21T10:03:28.237 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_kmip_addr": "", 2024-09-21T10:03:28.237 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-21T10:03:28.237 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-21T10:03:28.237 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-21T10:03:28.237 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-21T10:03:28.237 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_kmip_password": "", 2024-09-21T10:03:28.237 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-21T10:03:28.237 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_kmip_username": "", 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_require_ssl": "true", 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-21T10:03:28.238 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-21T10:03:28.239 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-21T10:03:28.239 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-21T10:03:28.239 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-21T10:03:28.239 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_vault_addr": "", 2024-09-21T10:03:28.239 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_vault_auth": "token", 2024-09-21T10:03:28.239 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_vault_namespace": "", 2024-09-21T10:03:28.239 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_vault_prefix": "", 2024-09-21T10:03:28.239 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-21T10:03:28.239 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-21T10:03:28.239 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-21T10:03:28.239 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-21T10:03:28.239 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_vault_token_file": "", 2024-09-21T10:03:28.239 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-21T10:03:28.239 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_curl_buffersize": "524288", 2024-09-21T10:03:28.240 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-21T10:03:28.240 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_curl_low_speed_time": "300", 2024-09-21T10:03:28.240 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-21T10:03:28.240 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-21T10:03:28.240 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-21T10:03:28.240 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-21T10:03:28.240 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-21T10:03:28.240 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-21T10:03:28.240 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-21T10:03:28.240 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-21T10:03:28.240 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-21T10:03:28.240 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-21T10:03:28.240 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-21T10:03:28.240 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-21T10:03:28.241 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-21T10:03:28.241 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-21T10:03:28.241 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-21T10:03:28.241 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-21T10:03:28.241 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-21T10:03:28.241 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-21T10:03:28.241 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-21T10:03:28.241 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_data_log_changes_size": "1000", 2024-09-21T10:03:28.241 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_data_log_num_shards": "128", 2024-09-21T10:03:28.241 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-21T10:03:28.241 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_data_log_window": "30", 2024-09-21T10:03:28.241 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-21T10:03:28.241 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-21T10:03:28.241 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-21T10:03:28.242 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-21T10:03:28.242 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-21T10:03:28.242 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-21T10:03:28.242 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-21T10:03:28.242 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-21T10:03:28.242 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-21T10:03:28.242 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-21T10:03:28.242 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-21T10:03:28.242 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-21T10:03:28.242 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-21T10:03:28.242 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_disable_s3select": "false", 2024-09-21T10:03:28.242 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-21T10:03:28.242 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-21T10:03:28.243 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-21T10:03:28.243 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-21T10:03:28.243 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-21T10:03:28.243 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-21T10:03:28.243 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-21T10:03:28.243 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-21T10:03:28.243 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-21T10:03:28.243 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-21T10:03:28.243 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-21T10:03:28.243 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-21T10:03:28.243 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dns_name": "", 2024-09-21T10:03:28.243 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dns_s3website_name": "", 2024-09-21T10:03:28.243 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dynamic_resharding": "true", 2024-09-21T10:03:28.243 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-21T10:03:28.244 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-21T10:03:28.244 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-21T10:03:28.244 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_enable_gc_threads": "true", 2024-09-21T10:03:28.244 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_enable_lc_threads": "true", 2024-09-21T10:03:28.244 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_enable_mdsearch": "true", 2024-09-21T10:03:28.244 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_enable_ops_log": "false", 2024-09-21T10:03:28.244 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_enable_quota_threads": "true", 2024-09-21T10:03:28.244 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_enable_static_website": "false", 2024-09-21T10:03:28.244 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_enable_usage_log": "false", 2024-09-21T10:03:28.244 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_enforce_swift_acls": "true", 2024-09-21T10:03:28.244 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_exit_timeout_secs": "120", 2024-09-21T10:03:28.244 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_expose_bucket": "false", 2024-09-21T10:03:28.244 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_extended_http_attrs": "", 2024-09-21T10:03:28.244 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_filter": "none", 2024-09-21T10:03:28.245 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-21T10:03:28.245 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_frontends": "beast port=7480", 2024-09-21T10:03:28.245 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-21T10:03:28.245 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_gc_max_deferred": "50", 2024-09-21T10:03:28.245 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-21T10:03:28.245 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_gc_max_objs": "32", 2024-09-21T10:03:28.245 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-21T10:03:28.245 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-21T10:03:28.245 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-21T10:03:28.245 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-21T10:03:28.245 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_gc_processor_period": "3600", 2024-09-21T10:03:28.245 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-21T10:03:28.245 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-21T10:03:28.246 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-21T10:03:28.246 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-21T10:03:28.246 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_init_timeout": "300", 2024-09-21T10:03:28.246 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-21T10:03:28.246 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-21T10:03:28.246 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_kafka_connection_idle": "300", 2024-09-21T10:03:28.246 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-21T10:03:28.246 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-21T10:03:28.246 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-21T10:03:28.246 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-21T10:03:28.246 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-21T10:03:28.246 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_admin_domain": "", 2024-09-21T10:03:28.246 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_admin_password": "", 2024-09-21T10:03:28.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_admin_password_path": "", 2024-09-21T10:03:28.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_admin_project": "", 2024-09-21T10:03:28.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_admin_tenant": "", 2024-09-21T10:03:28.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_admin_token": "", 2024-09-21T10:03:28.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_admin_token_path": "", 2024-09-21T10:03:28.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_admin_user": "", 2024-09-21T10:03:28.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_barbican_domain": "", 2024-09-21T10:03:28.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_barbican_password": "", 2024-09-21T10:03:28.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_barbican_project": "", 2024-09-21T10:03:28.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-21T10:03:28.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_barbican_user": "", 2024-09-21T10:03:28.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-21T10:03:28.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-21T10:03:28.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-21T10:03:28.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-21T10:03:28.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-21T10:03:28.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_url": "", 2024-09-21T10:03:28.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-21T10:03:28.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_lc_debug_interval": "-1", 2024-09-21T10:03:28.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_lc_lock_max_time": "90", 2024-09-21T10:03:28.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_lc_max_objs": "32", 2024-09-21T10:03:28.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_lc_max_rules": "1000", 2024-09-21T10:03:28.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_lc_max_worker": "3", 2024-09-21T10:03:28.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-21T10:03:28.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_lc_thread_delay": "0", 2024-09-21T10:03:28.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-21T10:03:28.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_ldap_dnattr": "uid", 2024-09-21T10:03:28.249 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-21T10:03:28.249 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_ldap_searchfilter": "", 2024-09-21T10:03:28.249 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-21T10:03:28.249 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_ldap_uri": "", 2024-09-21T10:03:28.249 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-21T10:03:28.249 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-21T10:03:28.249 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-21T10:03:28.249 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-21T10:03:28.249 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_log_http_headers": "", 2024-09-21T10:03:28.249 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-21T10:03:28.249 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-21T10:03:28.249 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_log_object_name_utc": "false", 2024-09-21T10:03:28.249 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-21T10:03:28.250 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-21T10:03:28.250 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_attr_name_len": "0", 2024-09-21T10:03:28.250 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_attr_size": "0", 2024-09-21T10:03:28.250 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-21T10:03:28.250 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_chunk_size": "4194304", 2024-09-21T10:03:28.250 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-21T10:03:28.250 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-21T10:03:28.250 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-21T10:03:28.250 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_listing_results": "5000", 2024-09-21T10:03:28.250 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_notify_retries": "10", 2024-09-21T10:03:28.250 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-21T10:03:28.250 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_put_param_size": "1048576", 2024-09-21T10:03:28.250 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_put_size": "5368709120", 2024-09-21T10:03:28.250 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_slo_entries": "1000", 2024-09-21T10:03:28.251 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_md_log_max_shards": "64", 2024-09-21T10:03:28.251 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-21T10:03:28.251 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-21T10:03:28.251 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-21T10:03:28.251 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-21T10:03:28.251 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_mp_lock_max_time": "600", 2024-09-21T10:03:28.251 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-21T10:03:28.251 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-21T10:03:28.251 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-21T10:03:28.251 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-21T10:03:28.251 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-21T10:03:28.251 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-21T10:03:28.251 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-21T10:03:28.251 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-21T10:03:28.252 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_max_gc": "300", 2024-09-21T10:03:28.252 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-21T10:03:28.252 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-21T10:03:28.252 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-21T10:03:28.252 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-21T10:03:28.252 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-21T10:03:28.252 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-21T10:03:28.252 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-21T10:03:28.252 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_num_async_rados_threads": "32", 2024-09-21T10:03:28.252 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_num_control_oids": "8", 2024-09-21T10:03:28.252 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_numa_node": "-1", 2024-09-21T10:03:28.252 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-21T10:03:28.252 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-21T10:03:28.253 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_objexp_chunk_size": "100", 2024-09-21T10:03:28.253 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_objexp_gc_interval": "600", 2024-09-21T10:03:28.253 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-21T10:03:28.253 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-21T10:03:28.253 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-21T10:03:28.253 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_op_thread_timeout": "600", 2024-09-21T10:03:28.253 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_op_tracing": "false", 2024-09-21T10:03:28.253 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_opa_token": "", 2024-09-21T10:03:28.253 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_opa_url": "", 2024-09-21T10:03:28.253 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_opa_verify_ssl": "true", 2024-09-21T10:03:28.253 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-21T10:03:28.253 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-21T10:03:28.253 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_ops_log_rados": "false", 2024-09-21T10:03:28.253 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_ops_log_socket_path": "", 2024-09-21T10:03:28.254 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-21T10:03:28.254 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-21T10:03:28.254 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-21T10:03:28.254 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_period_push_interval": "2.000000", 2024-09-21T10:03:28.254 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-21T10:03:28.254 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-21T10:03:28.254 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-21T10:03:28.254 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-21T10:03:28.254 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_posix_cache_lanes": "3", 2024-09-21T10:03:28.254 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-21T10:03:28.254 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-21T10:03:28.254 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_posix_cache_partitions": "3", 2024-09-21T10:03:28.254 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-21T10:03:28.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_print_continue": "true", 2024-09-21T10:03:28.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-21T10:03:28.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-21T10:03:28.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-21T10:03:28.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-21T10:03:28.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-21T10:03:28.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_rados_tracing": "false", 2024-09-21T10:03:28.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_realm": "", 2024-09-21T10:03:28.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_realm_id": "", 2024-09-21T10:03:28.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-21T10:03:28.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_region": "", 2024-09-21T10:03:28.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-21T10:03:28.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-21T10:03:28.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-21T10:03:28.256 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-21T10:03:28.256 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_request_uri": "", 2024-09-21T10:03:28.256 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_reshard_batch_size": "64", 2024-09-21T10:03:28.256 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-21T10:03:28.256 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-21T10:03:28.256 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_reshard_max_aio": "128", 2024-09-21T10:03:28.256 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_reshard_num_logs": "16", 2024-09-21T10:03:28.256 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_reshard_progress_judge_interval": "120", 2024-09-21T10:03:28.256 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2024-09-21T10:03:28.256 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_reshard_thread_interval": "600", 2024-09-21T10:03:28.256 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_reshardlog_threshold": "30000", 2024-09-21T10:03:28.256 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_resolve_cname": "false", 2024-09-21T10:03:28.256 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-21T10:03:28.256 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_run_sync_thread": "true", 2024-09-21T10:03:28.257 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-21T10:03:28.257 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-21T10:03:28.257 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-21T10:03:28.257 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-21T10:03:28.257 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-21T10:03:28.257 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-21T10:03:28.257 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-21T10:03:28.257 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-21T10:03:28.257 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-21T10:03:28.257 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_scheduler_type": "throttler", 2024-09-21T10:03:28.257 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_script_uri": "", 2024-09-21T10:03:28.257 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_service_provider_name": "", 2024-09-21T10:03:28.257 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-21T10:03:28.257 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sts_client_id": "", 2024-09-21T10:03:28.258 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sts_client_secret": "", 2024-09-21T10:03:28.258 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sts_entry": "sts", 2024-09-21T10:03:28.258 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sts_key": "sts", 2024-09-21T10:03:28.258 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-21T10:03:28.258 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sts_min_session_duration": "900", 2024-09-21T10:03:28.258 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sts_token_introspection_url": "", 2024-09-21T10:03:28.258 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_swift_account_in_url": "false", 2024-09-21T10:03:28.258 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_swift_auth_entry": "auth", 2024-09-21T10:03:28.258 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_swift_auth_url": "", 2024-09-21T10:03:28.258 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_swift_custom_header": "", 2024-09-21T10:03:28.258 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-21T10:03:28.258 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_swift_need_stats": "true", 2024-09-21T10:03:28.258 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_swift_tenant_name": "", 2024-09-21T10:03:28.259 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_swift_token_expiration": "86400", 2024-09-21T10:03:28.259 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_swift_url": "", 2024-09-21T10:03:28.259 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_swift_url_prefix": "swift", 2024-09-21T10:03:28.259 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-21T10:03:28.259 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-21T10:03:28.259 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-21T10:03:28.259 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_lease_period": "120", 2024-09-21T10:03:28.259 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-21T10:03:28.259 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-21T10:03:28.259 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-21T10:03:28.259 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-21T10:03:28.259 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-21T10:03:28.259 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-21T10:03:28.259 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-21T10:03:28.260 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-21T10:03:28.260 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-21T10:03:28.260 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_thread_pool_size": "512", 2024-09-21T10:03:28.260 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-21T10:03:28.260 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-21T10:03:28.260 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-21T10:03:28.260 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-21T10:03:28.260 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_torrent_comment": "", 2024-09-21T10:03:28.260 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_torrent_createby": "", 2024-09-21T10:03:28.260 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_torrent_encoding": "", 2024-09-21T10:03:28.260 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_torrent_flag": "false", 2024-09-21T10:03:28.260 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-21T10:03:28.260 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_torrent_origin": "", 2024-09-21T10:03:28.261 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-21T10:03:28.261 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_torrent_tracker": "", 2024-09-21T10:03:28.261 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_trust_forwarded_https": "false", 2024-09-21T10:03:28.261 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-21T10:03:28.261 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-21T10:03:28.261 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_usage_max_shards": "32", 2024-09-21T10:03:28.261 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_usage_max_user_shards": "1", 2024-09-21T10:03:28.261 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_use_opa_authz": "false", 2024-09-21T10:03:28.261 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_user_counters_cache": "false", 2024-09-21T10:03:28.261 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-21T10:03:28.261 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-21T10:03:28.261 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-21T10:03:28.261 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_user_max_buckets": "1000", 2024-09-21T10:03:28.261 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_user_policies_max_num": "100", 2024-09-21T10:03:28.262 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-21T10:03:28.262 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-21T10:03:28.262 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-21T10:03:28.262 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-21T10:03:28.262 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_user_unique_email": "true", 2024-09-21T10:03:28.262 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_verify_ssl": "true", 2024-09-21T10:03:28.262 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-21T10:03:28.262 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_zone": "", 2024-09-21T10:03:28.262 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_zone_id": "", 2024-09-21T10:03:28.262 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-21T10:03:28.262 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_zonegroup": "", 2024-09-21T10:03:28.262 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_zonegroup_id": "", 2024-09-21T10:03:28.262 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-21T10:03:28.262 INFO:teuthology.orchestra.run.smithi098.stdout: "rgwlc_auto_session_clear": "true", 2024-09-21T10:03:28.263 INFO:teuthology.orchestra.run.smithi098.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-21T10:03:28.263 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_block_size": "4096", 2024-09-21T10:03:28.263 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-21T10:03:28.263 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-21T10:03:28.263 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-21T10:03:28.263 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-21T10:03:28.263 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-21T10:03:28.263 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_cache_size": "536870912", 2024-09-21T10:03:28.263 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-21T10:03:28.263 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-21T10:03:28.263 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-21T10:03:28.263 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-21T10:03:28.263 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-21T10:03:28.263 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-21T10:03:28.264 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-21T10:03:28.264 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-21T10:03:28.264 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_index_type": "binary_search", 2024-09-21T10:03:28.264 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-21T10:03:28.264 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-21T10:03:28.264 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_partition_filters": "false", 2024-09-21T10:03:28.264 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_perf": "false", 2024-09-21T10:03:28.264 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-21T10:03:28.264 INFO:teuthology.orchestra.run.smithi098.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-21T10:03:28.264 INFO:teuthology.orchestra.run.smithi098.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-21T10:03:28.264 INFO:teuthology.orchestra.run.smithi098.stdout: "run_dir": "/var/run/ceph", 2024-09-21T10:03:28.264 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_block_create": "true", 2024-09-21T10:03:28.264 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_cache_lru_size": "67108864", 2024-09-21T10:03:28.264 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-21T10:03:28.265 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-21T10:03:28.265 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_default_max_object_size": "16777216", 2024-09-21T10:03:28.265 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-21T10:03:28.265 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_device_size": "53687091200", 2024-09-21T10:03:28.265 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-21T10:03:28.265 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_full_integrity_check": "false", 2024-09-21T10:03:28.265 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_journal_batch_capacity": "16", 2024-09-21T10:03:28.265 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-21T10:03:28.265 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-21T10:03:28.265 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-21T10:03:28.265 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_main_device_type": "SSD", 2024-09-21T10:03:28.265 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-21T10:03:28.265 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-21T10:03:28.266 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-21T10:03:28.266 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-21T10:03:28.266 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-21T10:03:28.266 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-21T10:03:28.266 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_segment_size": "67108864", 2024-09-21T10:03:28.266 INFO:teuthology.orchestra.run.smithi098.stdout: "set_keepcaps": "false", 2024-09-21T10:03:28.266 INFO:teuthology.orchestra.run.smithi098.stdout: "setgroup": "ceph", 2024-09-21T10:03:28.266 INFO:teuthology.orchestra.run.smithi098.stdout: "setuser": "ceph", 2024-09-21T10:03:28.266 INFO:teuthology.orchestra.run.smithi098.stdout: "setuser_match_path": "", 2024-09-21T10:03:28.266 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-21T10:03:28.266 INFO:teuthology.orchestra.run.smithi098.stdout: "thp": "false", 2024-09-21T10:03:28.266 INFO:teuthology.orchestra.run.smithi098.stdout: "threadpool_default_timeout": "60", 2024-09-21T10:03:28.266 INFO:teuthology.orchestra.run.smithi098.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-21T10:03:28.266 INFO:teuthology.orchestra.run.smithi098.stdout: "throttler_perf_counter": "true", 2024-09-21T10:03:28.267 INFO:teuthology.orchestra.run.smithi098.stdout: "tmp_dir": "/tmp", 2024-09-21T10:03:28.267 INFO:teuthology.orchestra.run.smithi098.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-21T10:03:28.267 INFO:teuthology.orchestra.run.smithi098.stdout: "uadk_compressor_enabled": "false", 2024-09-21T10:03:28.267 INFO:teuthology.orchestra.run.smithi098.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-21T10:03:28.267 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:03:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:28 smithi196 ceph-mon[84784]: pgmap v228: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:28 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:03:28.950 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:28 smithi098 ceph-mon[97580]: pgmap v228: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:28.950 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:28 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:03:28.951 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:28 smithi098 ceph-mon[100914]: pgmap v228: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:28.951 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:28 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:03:28.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-21T10:03:29.779 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:30.598 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:03:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:03:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:30.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:30 smithi098 ceph-mon[97580]: pgmap v229: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:30.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:03:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:03:30] "GET /metrics HTTP/1.1" 200 34032 "" "Prometheus/2.51.0" 2024-09-21T10:03:30.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:30 smithi098 ceph-mon[100914]: pgmap v229: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:30.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:30 smithi196 ceph-mon[84784]: pgmap v229: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:31.514 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a: {} 2024-09-21T10:03:31.515 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b: {} 2024-09-21T10:03:31.515 INFO:teuthology.orchestra.run.smithi098.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T10:03:31.515 INFO:teuthology.orchestra.run.smithi098.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T10:03:31.522 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c: {} 2024-09-21T10:03:31.522 INFO:teuthology.orchestra.run.smithi098.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T10:03:31.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:31 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2629790213' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T10:03:31.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:31 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2629790213' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T10:03:32.013 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:31 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2629790213' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T10:03:32.295 INFO:tasks.thrashosds.thrasher:Unable to test ceph-objectstore-tool, not available on all OSD nodes 2024-09-21T10:03:32.296 INFO:teuthology.task.sequential:In sequential, running task full_sequential... 2024-09-21T10:03:32.296 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:03:32.298 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:03:32.298 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-21T10:03:32.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_0 16 2024-09-21T10:03:32.299 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-21T10:03:32.299 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:03:32.299 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:03:32.299 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T10:03:32.299 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.176620 2024-09-21T10:03:32.299 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 1 0.176620018855292 2024-09-21T10:03:32.299 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-21T10:03:32.300 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:32.300 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-21T10:03:32.300 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:03:32.300 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-21T10:03:32.301 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:03:32.301 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-21T10:03:32.301 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:03:32.332 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:32.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:32.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:32.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:32.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:32.474 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:32.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:32.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:32.612 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:32.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:32.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:32.714 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:32.735 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:32.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:32.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:32.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:32.859 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:32.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:32.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:32.912 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:32 smithi098 ceph-mon[97580]: pgmap v230: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:32.912 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:32 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:03:32.912 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:32 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:03:32.912 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:32 smithi098 ceph-mon[100914]: pgmap v230: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:32.912 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:32 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:03:32.912 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:32 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:03:32.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:32.982 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:32.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:32.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:33.083 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:33.104 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:33.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:33.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:33.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:32 smithi196 ceph-mon[84784]: pgmap v230: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:33.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:32 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:03:33.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:32 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:03:33.141 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:33.206 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:33.227 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:33.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:33.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:33.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:33.351 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:33.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:33.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:33.429 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:33.453 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:33.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:33.475 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:33.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:33.576 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:33.598 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:33.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:33.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:33.700 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:33.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:33.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:33.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:33.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:33.844 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:33.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:33.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:33.946 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:33.967 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:33.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:33.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:34.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:34.090 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:34.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:34.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:34.192 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:34.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:34.215 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:34.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:34.315 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:34.337 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:34.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:34.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:34.439 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:34.460 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:34.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:34.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:34.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:34.583 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:34.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:34.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:34.684 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:34.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:34.708 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:34.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:34.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:34.808 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:34 smithi098 ceph-mon[97580]: pgmap v231: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:34.809 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:34 smithi098 ceph-mon[100914]: pgmap v231: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:34.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:34.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:34.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:34.932 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:34.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:34.955 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:34.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:35.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:35.078 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:35.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:35.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:35.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:34 smithi196 ceph-mon[84784]: pgmap v231: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:35.180 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:35.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:35.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:35.302 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:35.323 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:35.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:35.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:35.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:35.446 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:35.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:35.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:35.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:35.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:35.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:35.670 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:35.696 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:35.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:35.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:35.798 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:35.823 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:35.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:35.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:35.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:35.950 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:35.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:35.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:36.052 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:36.077 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:36.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:36.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:36.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:36.200 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:36.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:36.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:36.301 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:36.323 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:36.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:36.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:36.391 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:36.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:36.446 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:36.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:36.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:36.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:36.569 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:36.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:36.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:36.671 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:36.696 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:36.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:36.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:36.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:36 smithi098 ceph-mon[97580]: pgmap v232: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:36.750 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:36 smithi098 ceph-mon[100914]: pgmap v232: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:36.798 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:36.823 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:36.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:36.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:36.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:36.940 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:36 smithi196 ceph-mon[84784]: pgmap v232: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:36.951 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:36.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:36.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:37.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:37.075 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:37.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:37.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:37.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:37.198 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:37.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:37.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:37.299 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:37.321 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:37.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:37.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:37.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:37.445 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:37.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:37.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:37.546 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:37.568 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:37.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:37.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:37.669 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:37.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:37.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:37.791 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:37.813 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:37.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:37.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:37.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:37.937 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:37.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:37.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:38.038 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:38.059 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:38.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:38.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:38.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:38.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:38.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:38.161 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:38.182 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:38.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:38.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:38.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:38.305 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:38.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:38.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:38.406 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:38.456 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:38.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:38.556 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:38.583 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:38.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:38.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:38.659 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.1 primary-affinity to 0.17662 (8115742) 2024-09-21T10:03:38.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:38.708 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:38.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:38.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:38.809 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:38.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:38.832 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:38.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:38.933 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:38.954 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:38.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:38.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:38.978 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:39.008 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:38 smithi098 ceph-mon[97580]: pgmap v233: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:39.009 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:38 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3533101382' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.176620018855292} : dispatch 2024-09-21T10:03:39.009 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:38 smithi098 ceph-mon[100914]: pgmap v233: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:39.009 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:38 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3533101382' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.176620018855292} : dispatch 2024-09-21T10:03:39.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:39.077 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:39.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:39.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:39.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:38 smithi196 ceph-mon[84784]: pgmap v233: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:39.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:38 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3533101382' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.176620018855292} : dispatch 2024-09-21T10:03:39.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:39.200 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:39.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:39.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:39.301 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:39.322 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:39.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:39.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:39.424 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:39.446 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:39.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:39.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:39.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:39.570 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:39.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:39.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:39.671 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:39.693 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:39.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:39.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:39.795 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:39.817 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:39.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:39.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:39.919 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:39.942 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:39.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:39.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:40.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:39 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3533101382' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.176620018855292}]': finished 2024-09-21T10:03:40.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:39 smithi098 ceph-mon[97580]: osdmap e103: 8 total, 8 up, 8 in 2024-09-21T10:03:40.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:39 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:03:40.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:39 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:03:40.033 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:39 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3533101382' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.176620018855292}]': finished 2024-09-21T10:03:40.033 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:39 smithi098 ceph-mon[100914]: osdmap e103: 8 total, 8 up, 8 in 2024-09-21T10:03:40.033 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:39 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:03:40.034 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:39 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:03:40.044 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:40.067 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:40.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:40.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:40.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:39 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3533101382' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.176620018855292}]': finished 2024-09-21T10:03:40.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:39 smithi196 ceph-mon[84784]: osdmap e103: 8 total, 8 up, 8 in 2024-09-21T10:03:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:39 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:03:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:39 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:03:40.168 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:40.191 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:40.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:40.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:40.293 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:40.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:40.318 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:40.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:40.418 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:40.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:40.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:40.543 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:40.568 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:40.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:40.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:40.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:03:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:03:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:40.670 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:40.695 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:40.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:40.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:40.797 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:40.822 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:40.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:40.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:40.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:03:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:03:40] "GET /metrics HTTP/1.1" 200 34032 "" "Prometheus/2.51.0" 2024-09-21T10:03:40.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:40 smithi098 ceph-mon[97580]: pgmap v235: 73 pgs: 2 peering, 71 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:40.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:40 smithi098 ceph-mon[97580]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:03:40.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:40 smithi098 ceph-mon[97580]: osdmap e104: 8 total, 8 up, 8 in 2024-09-21T10:03:40.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:40 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:40.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:40 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:03:40.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:40 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:03:40.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:40 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/719093181' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-21T10:03:40.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:40 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-21T10:03:40.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:40 smithi098 ceph-mon[100914]: pgmap v235: 73 pgs: 2 peering, 71 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:40.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:40 smithi098 ceph-mon[100914]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:03:40.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:40 smithi098 ceph-mon[100914]: osdmap e104: 8 total, 8 up, 8 in 2024-09-21T10:03:40.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:40 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:40.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:40 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:03:40.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:40 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:03:40.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:40 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/719093181' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-21T10:03:40.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:40 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-21T10:03:40.924 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'unique_pool_0' created 2024-09-21T10:03:40.951 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:40.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:40.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:41.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:41.074 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:41.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:41.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:41.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:40 smithi196 ceph-mon[84784]: pgmap v235: 73 pgs: 2 peering, 71 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:40 smithi196 ceph-mon[84784]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:03:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:40 smithi196 ceph-mon[84784]: osdmap e104: 8 total, 8 up, 8 in 2024-09-21T10:03:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:40 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:40 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:03:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:40 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:03:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:40 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/719093181' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-21T10:03:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:40 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-21T10:03:41.175 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:41.201 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:41.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:41.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:41.303 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:41.325 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:41.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:41.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:41.426 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:41.450 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:41.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:41.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:41.552 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:41.574 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:41.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:41.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:41.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-21T10:03:41.677 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:41.703 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:41.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:41.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:41.805 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:41.830 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:41.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:41.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:41.932 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:41.956 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:41.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:41.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:42.058 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:42.079 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:42.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:42.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:42.180 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:42.201 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:42.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:42.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:42.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:41 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T10:03:42.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:41 smithi098 ceph-mon[97580]: osdmap e105: 8 total, 8 up, 8 in 2024-09-21T10:03:42.282 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:41 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T10:03:42.282 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:41 smithi098 ceph-mon[100914]: osdmap e105: 8 total, 8 up, 8 in 2024-09-21T10:03:42.303 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:42.324 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:42.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:42.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:42.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:41 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T10:03:42.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:41 smithi196 ceph-mon[84784]: osdmap e105: 8 total, 8 up, 8 in 2024-09-21T10:03:42.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:42.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:42.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:42.548 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:42.574 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:42.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:42.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:42.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:42.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:42.699 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:42.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:42.800 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:42.826 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:42.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:42.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:42.928 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:42.954 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:42.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:42.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:43.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:43.077 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:43.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:43.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:43.163 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:42 smithi098 ceph-mon[97580]: pgmap v238: 89 pgs: 16 unknown, 2 peering, 71 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:43.163 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:42 smithi098 ceph-mon[97580]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:03:43.163 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:42 smithi098 ceph-mon[97580]: osdmap e106: 8 total, 8 up, 8 in 2024-09-21T10:03:43.164 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:42 smithi098 ceph-mon[100914]: pgmap v238: 89 pgs: 16 unknown, 2 peering, 71 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:43.164 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:42 smithi098 ceph-mon[100914]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:03:43.164 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:42 smithi098 ceph-mon[100914]: osdmap e106: 8 total, 8 up, 8 in 2024-09-21T10:03:43.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:43.205 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:43.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:43.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:43.306 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:43.332 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:43.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:43.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:43.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:42 smithi196 ceph-mon[84784]: pgmap v238: 89 pgs: 16 unknown, 2 peering, 71 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:43.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:42 smithi196 ceph-mon[84784]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:03:43.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:42 smithi196 ceph-mon[84784]: osdmap e106: 8 total, 8 up, 8 in 2024-09-21T10:03:43.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:43.451 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:03:43.451 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:03:43.456 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:43.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:43.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:43.468 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:03:43.468 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:03:43.484 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:03:43.484 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:03:43.500 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:03:43.500 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:03:43.516 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:03:43.516 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:03:43.532 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:03:43.532 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:03:43.548 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:03:43.548 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:03:43.558 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:43.563 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:03:43.564 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:03:43.579 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:43.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:43.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:43.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:43.702 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:43.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:43.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:43.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:43.825 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:43.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:43.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:43.926 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:43.948 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:43.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:43.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:43.989 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:03:44.049 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:44.070 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:44.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:44.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:44.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:43 smithi098 ceph-mon[97580]: osdmap e107: 8 total, 8 up, 8 in 2024-09-21T10:03:44.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:43 smithi098 ceph-mon[97580]: pgmap v241: 89 pgs: 3 creating+peering, 5 unknown, 2 peering, 79 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:44.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:43 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4140779256' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:03:44.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:43 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1486350927' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:03:44.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:43 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:03:44.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:43 smithi098 ceph-mon[100914]: osdmap e107: 8 total, 8 up, 8 in 2024-09-21T10:03:44.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:43 smithi098 ceph-mon[100914]: pgmap v241: 89 pgs: 3 creating+peering, 5 unknown, 2 peering, 79 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:44.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:43 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4140779256' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:03:44.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:43 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1486350927' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:03:44.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:43 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:03:44.172 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:44.198 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:44.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:44.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:44.300 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:44.325 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:44.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:44.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:44.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:43 smithi196 ceph-mon[84784]: osdmap e107: 8 total, 8 up, 8 in 2024-09-21T10:03:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:43 smithi196 ceph-mon[84784]: pgmap v241: 89 pgs: 3 creating+peering, 5 unknown, 2 peering, 79 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:43 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4140779256' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:03:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:43 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1486350927' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:03:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:43 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:03:44.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:44.449 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:44.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:44.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:44.550 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:44.577 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:44.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:44.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:44.644 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:03:44.644 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:03:44.678 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:44.699 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:44.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:44.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:44.800 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:44.826 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:44.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:44.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:44.928 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:44.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:44.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:44.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:45.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:45.076 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:45.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:45.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:45.177 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:45.203 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:45.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:45.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:45.271 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:44 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:03:45.271 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:44 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:03:45.271 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:44 smithi098 ceph-mon[97580]: osdmap e108: 8 total, 8 up, 8 in 2024-09-21T10:03:45.272 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:44 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:03:45.272 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:44 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:03:45.272 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:44 smithi098 ceph-mon[100914]: osdmap e108: 8 total, 8 up, 8 in 2024-09-21T10:03:45.305 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:45.326 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:45.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:45.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:45.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:44 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:03:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:44 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:03:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:44 smithi196 ceph-mon[84784]: osdmap e108: 8 total, 8 up, 8 in 2024-09-21T10:03:45.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:45.453 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:45.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:45.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:45.554 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:45.580 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:45.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:45.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:45.682 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:45.707 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:45.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:45.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:45.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:03:45.731 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:45.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:45.829 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:45.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:45.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:45.931 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:45.956 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:45.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:45.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:46.058 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:46.082 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:46.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:46.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:46.128 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:45 smithi098 ceph-mon[97580]: pgmap v243: 89 pgs: 3 creating+peering, 86 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:46.128 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:45 smithi098 ceph-mon[100914]: pgmap v243: 89 pgs: 3 creating+peering, 86 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:46.184 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:46.209 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:46.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:46.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:46.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:03:46.311 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:46.332 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:46.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:46.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:46.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:45 smithi196 ceph-mon[84784]: pgmap v243: 89 pgs: 3 creating+peering, 86 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:46.434 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:46.459 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:46.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:46.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:46.561 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:46.586 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:46.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:46.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:46.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:46.710 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:46.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:46.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:46.811 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:46.837 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:46.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:46.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:46.939 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:46.961 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:46.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:46.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:47.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:47.067 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:47.087 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:47.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:47.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:47.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:47.189 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:46 smithi098 ceph-mon[97580]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T10:03:47.189 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:46 smithi098 ceph-mon[100914]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T10:03:47.209 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:47.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:47.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:47.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:03:47.311 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:47.361 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:47.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:47.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:47.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:46 smithi196 ceph-mon[84784]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T10:03:47.462 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:47.488 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:47.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:47.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:47.590 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:47.611 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:47.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:47.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:47.666 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:03:47.667 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:03:47.667 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-21T10:03:47.667 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:03:47.667 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:03:47.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-21T10:03:47.715 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:47.741 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:47.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:47.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:47.842 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:47.864 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:47.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:47.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:47.966 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:47.987 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:47.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:47.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:47.996 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-21T10:03:48.089 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:48.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:48 smithi098 ceph-mon[97580]: pgmap v244: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:48.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:48 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/277023948' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:03:48.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:48 smithi098 ceph-mon[100914]: pgmap v244: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:48.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:48 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/277023948' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:03:48.111 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:48.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:48.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:48.212 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:48.237 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:48.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:48.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:48.339 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:48.360 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:48.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:48.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:48.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:47 smithi196 ceph-mon[84784]: pgmap v244: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:48.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:47 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/277023948' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:03:48.462 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:48.483 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:48.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:48.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:48.584 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:48.609 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:48.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:48.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:48.710 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:48.735 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:48.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:48.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:48.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:48.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:48.860 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:48.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:48.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:48.982 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:48.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:48.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:49.084 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:49.093 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:49.110 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:49.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:49.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:49.212 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:49.237 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:49.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:49.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:49.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:48 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/277023948' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:03:49.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:48 smithi098 ceph-mon[97580]: osdmap e109: 8 total, 8 up, 8 in 2024-09-21T10:03:49.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:49 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/277023948' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:03:49.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:49 smithi098 ceph-mon[100914]: osdmap e109: 8 total, 8 up, 8 in 2024-09-21T10:03:49.327 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:49.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:49.364 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:49.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:49.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:49.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:48 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/277023948' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:03:49.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:48 smithi196 ceph-mon[84784]: osdmap e109: 8 total, 8 up, 8 in 2024-09-21T10:03:49.466 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:49.491 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:49.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:49.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:49.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:49.606 DEBUG:teuthology.orchestra.run.smithi098:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-21T10:03:49.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:49.615 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:49.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:49.615 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:03:49.615 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:03:49.660 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:hints = 1 2024-09-21T10:03:49.660 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:03:49.660 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Object prefix: benchmark_data_smithi098_135187 2024-09-21T10:03:49.660 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:03:49.660 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:03:49.716 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:49.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:49.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:49.855 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:49.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:49.892 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:49.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:49.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:50.030 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:50.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:50.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:50.116 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:50 smithi098 ceph-mon[97580]: pgmap v246: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:50.116 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:50 smithi098 ceph-mon[100914]: pgmap v246: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:50.132 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:50.168 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:50.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:50.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:50.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:50.306 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:50.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:50.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:50.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:50 smithi196 ceph-mon[84784]: pgmap v246: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:50.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:03:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:03:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:50.408 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:50.442 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:50.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:50.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:50.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:50.567 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:03:50.567 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:03:50.577 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:50.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:50.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:50.599 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:03:50.599 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:03:50.636 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:03:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:03:50.660 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 1 16 3956 3940 246.236 246.25 0.00431872 0.00404427 2024-09-21T10:03:50.670 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:03:50.670 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:03:50.679 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:50.700 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:03:50.701 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:03:50.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:50.716 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:50.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:50.733 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:03:50.734 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:03:50.764 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:03:50.764 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:03:50.793 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:03:50.794 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:03:50.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:50.851 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:50.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:50.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:50.924 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:03:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:03:50] "GET /metrics HTTP/1.1" 200 34032 "" "Prometheus/2.51.0" 2024-09-21T10:03:50.953 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:50.989 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:50.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:50.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:51.091 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:51.125 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:51.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:51.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:51.227 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:51.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:51.265 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:51.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:51.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:51 smithi098 ceph-mon[97580]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:03:51.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:51 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3379023670' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:03:51.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:51 smithi098 ceph-mon[100914]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:03:51.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:51 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3379023670' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:03:51.366 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:51.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:51 smithi196 ceph-mon[84784]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:03:51.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:51 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3379023670' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:03:51.398 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:51.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:51.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:51.500 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:51.536 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:51.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:51.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:51.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:51.660 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 2 16 7929 7913 247.263 248.312 0.00625234 0.00403546 2024-09-21T10:03:51.671 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:51.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:51.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:51.773 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:51.806 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:51.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:51.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:51.908 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:51.944 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:51.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:51.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:52.046 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:52.077 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:52.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:52.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:52.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:52 smithi098 ceph-mon[97580]: pgmap v247: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:52.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:52 smithi098 ceph-mon[100914]: pgmap v247: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:52.179 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:52.210 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:52.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:52.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:52.312 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:52.348 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:52.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:52.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:52.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:52 smithi196 ceph-mon[84784]: pgmap v247: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:03:52.450 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:52.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:52.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:52.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:52.620 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:52.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:52.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:52.660 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 3 16 11641 11625 242.169 232 0.00664711 0.00412099 2024-09-21T10:03:52.722 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:52.751 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:52.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:52.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:52.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:52.887 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:52.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:52.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:52.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:53.025 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:53.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:53.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:53.127 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:53.161 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:53.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:53.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:53.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:53.297 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:53.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:53.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:53.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:53.432 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:53.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:53.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:53.534 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:53.567 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:53.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:53.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:53.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:03:53.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:53.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:03:53.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:03:53.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:53.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:03:53.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:03:53.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:03:53.660 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 4 16 14841 14825 231.622 200 0.00441936 0.00431119 2024-09-21T10:03:53.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:53.701 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:53.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:53.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:53.803 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:53.835 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:53.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:53.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:53.937 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:53.971 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:53.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:53.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:54.073 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:54.109 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:54.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:54.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:54.211 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:54.250 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:54.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:54.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:54.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:54 smithi098 ceph-mon[97580]: pgmap v248: 89 pgs: 89 active+clean; 491 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 54 MiB/s wr, 861 op/s 2024-09-21T10:03:54.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:54 smithi098 ceph-mon[100914]: pgmap v248: 89 pgs: 89 active+clean; 491 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 54 MiB/s wr, 861 op/s 2024-09-21T10:03:54.352 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:54.389 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:54.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:54.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:54.491 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:54.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:54.514 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:54.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:54.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:03:54.615 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:54 smithi196 ceph-mon[84784]: pgmap v248: 89 pgs: 89 active+clean; 491 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 54 MiB/s wr, 861 op/s 2024-09-21T10:03:54.649 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:54.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:54.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:54.660 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 5 16 17877 17861 223.245 189.75 0.00300661 0.0044737 2024-09-21T10:03:54.752 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:54.784 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:54.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:54.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:54.886 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:54.918 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:54.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:54.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:55.019 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:55.054 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:55.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:55.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:55.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:55.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:55.193 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:55.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:55.294 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:55.332 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:55.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:55.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:55.434 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:55.469 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:55.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:55.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:55.571 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:55.605 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:55.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:55.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:55.661 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 6 16 20841 20825 216.909 185.25 0.00361951 0.00460421 2024-09-21T10:03:55.707 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:55.742 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:55.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:55.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:55.844 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:55.877 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:55.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:55.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:55.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:56.013 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:56.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:56.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:56.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:56.149 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:56.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:56.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:56.251 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:56.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:56.284 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:56.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:56.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:56.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:56.421 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:56.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:56.522 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:56.557 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:56.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:56.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:56.574 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:03:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:03:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-21T10:03:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-21T10:03:56.574 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:03:56.659 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:56.662 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 7 16 23539 23523 210.009 168.625 0.00502 0.00475697 2024-09-21T10:03:56.662 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:03:56.662 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:03:56.663 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:03:56.694 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:56.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:56.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:56.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:56.831 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:56.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:56.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:56.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:56 smithi098 ceph-mon[97580]: pgmap v249: 89 pgs: 89 active+clean; 799 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-21T10:03:56.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:56 smithi098 ceph-mon[100914]: pgmap v249: 89 pgs: 89 active+clean; 799 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-21T10:03:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:56 smithi196 ceph-mon[84784]: pgmap v249: 89 pgs: 89 active+clean; 799 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-21T10:03:56.933 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:56.969 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:56.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:56.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:57.070 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:57.103 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:57.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:57.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:57.205 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:57.237 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:57.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:57.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:57.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:57.374 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:57.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:57.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:57.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:57.512 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:57.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:57.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:57.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:57.653 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:57.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:57.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:57.661 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 8 16 26332 26316 205.576 174.562 0.0042705 0.00485577 2024-09-21T10:03:57.754 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:57.790 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:57.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:57.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:57.892 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:57.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:57.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:58.030 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:58.064 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:58.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:58.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:58.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:58.197 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:58.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:58.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:58.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:58.339 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:58.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:58.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:58.441 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:58.472 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:58.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:58.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:58.488 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:03:58.574 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:58.590 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:03:58.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:58.612 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:58.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:58.620 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:58 smithi098 ceph-mon[97580]: pgmap v250: 89 pgs: 89 active+clean; 984 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:03:58.620 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:58 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/846680098' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:03:58.620 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:58 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:03:58.621 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:58 smithi098 ceph-mon[100914]: pgmap v250: 89 pgs: 89 active+clean; 984 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:03:58.621 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:58 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/846680098' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:03:58.621 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:58 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:03:58.661 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 9 16 28846 28830 200.19 157.125 0.00738319 0.00499024 2024-09-21T10:03:58.713 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:58.747 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:58.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:58.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:58.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:58 smithi196 ceph-mon[84784]: pgmap v250: 89 pgs: 89 active+clean; 984 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:03:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:58 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/846680098' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:03:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:58 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:03:58.890 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:58.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:58.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:58.995 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:59.033 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:59.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:59.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:59.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:59.171 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:59.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:59.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:59.222 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:59.274 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:59.311 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:59.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:59.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:59.413 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:59.447 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:59.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:59.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:59.549 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:59.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:59.580 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:59.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:59.661 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 10 16 31183 31167 194.776 146.062 0.00253685 0.00511464 2024-09-21T10:03:59.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:59.701 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:59.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:59.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:59.777 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:59 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:03:59.777 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:59 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:03:59.777 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:03:59 smithi098 ceph-mon[97580]: osdmap e110: 8 total, 8 up, 8 in 2024-09-21T10:03:59.778 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:59 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:03:59.778 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:59 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:03:59.778 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:03:59 smithi098 ceph-mon[100914]: osdmap e110: 8 total, 8 up, 8 in 2024-09-21T10:03:59.803 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:59.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:59.839 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:03:59.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:59.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:59 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:03:59.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:59 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:03:59.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:03:59 smithi196 ceph-mon[84784]: osdmap e110: 8 total, 8 up, 8 in 2024-09-21T10:03:59.941 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:03:59.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:59.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:00.082 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:00.116 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:00.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:00.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:00.218 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:00.246 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:00.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:00.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:00.348 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:00.383 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:00.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:00.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:00.485 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:00.500 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:04:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:04:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:00.520 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:00.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:00.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:00.621 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:00.655 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:00.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:00.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:00.661 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 11 16 33288 33272 189.028 131.562 0.00389391 0.00528495 2024-09-21T10:04:00.758 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:00.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:00.790 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:00.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:00.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:00 smithi098 ceph-mon[97580]: pgmap v252: 89 pgs: 89 active+clean; 1.7 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 170 MiB/s wr, 2.72k op/s 2024-09-21T10:04:00.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:04:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:04:00] "GET /metrics HTTP/1.1" 200 36424 "" "Prometheus/2.51.0" 2024-09-21T10:04:00.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:00 smithi098 ceph-mon[100914]: pgmap v252: 89 pgs: 89 active+clean; 1.7 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 170 MiB/s wr, 2.72k op/s 2024-09-21T10:04:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:00 smithi196 ceph-mon[84784]: pgmap v252: 89 pgs: 89 active+clean; 1.7 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 170 MiB/s wr, 2.72k op/s 2024-09-21T10:04:00.891 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:00.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:00.928 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:00.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:01.029 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:01.064 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:01.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:01.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:01.165 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:01.203 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:01.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:01.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:01.211 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:04:01.305 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:01.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:01.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:01.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:01.490 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:01.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:01.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:01.591 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:01.627 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:01.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:01.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:01.661 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 12 16 35236 35220 183.42 121.75 0.0065291 0.005438 2024-09-21T10:04:01.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:01.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:01.754 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:01.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:01.855 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:01.888 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:01.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:01.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:01.990 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:02.012 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:02.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:02.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:02.113 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:02.147 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:02.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:02.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:02.249 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:02.284 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:02.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:02.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:02.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:02.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:02.424 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:02.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:02.432 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:04:02.433 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T10:04:02.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:02.563 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:02.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:02.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:02.661 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 13 16 37038 37022 177.974 112.625 0.00951974 0.00561365 2024-09-21T10:04:02.665 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:02.699 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:02.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:02.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:02.801 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:02.836 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:02.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:02.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:02.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:02 smithi098 ceph-mon[97580]: pgmap v253: 89 pgs: 89 active+clean; 1.7 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 170 MiB/s wr, 2.72k op/s 2024-09-21T10:04:02.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:02 smithi098 ceph-mon[100914]: pgmap v253: 89 pgs: 89 active+clean; 1.7 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 170 MiB/s wr, 2.72k op/s 2024-09-21T10:04:02.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:02 smithi196 ceph-mon[84784]: pgmap v253: 89 pgs: 89 active+clean; 1.7 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 170 MiB/s wr, 2.72k op/s 2024-09-21T10:04:02.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:02.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:02.976 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:02.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:03.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:03.113 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:03.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:03.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:03.215 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:03.253 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:03.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:03.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:03.355 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:03.392 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:03.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:03.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:03.494 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:03.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:03.528 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:03.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:03.629 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:03.661 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:03.662 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 14 16 39234 39218 175.065 137.25 0.00913924 0.00570722 2024-09-21T10:04:03.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:03.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:03.712 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:03.763 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:03.800 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:03.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:03.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:03.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:03.942 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:03.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:03.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:04.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:04.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:04.081 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:04.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:04.182 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:04.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:04.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:04.319 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:04.354 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:04.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:04.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:04.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:04.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:04.502 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:04.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:04.576 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:04.603 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:04.634 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:04:04.635 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:04:04.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:04.658 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:04.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:04.663 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 15 16 41066 41050 171.025 114.5 0.00881387 0.00584169 2024-09-21T10:04:04.668 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:04:04.669 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:04:04.701 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:04:04.702 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:04:04.717 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:04 smithi098 ceph-mon[97580]: pgmap v254: 89 pgs: 89 active+clean; 2.1 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 168 MiB/s wr, 2.69k op/s 2024-09-21T10:04:04.718 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:04 smithi098 ceph-mon[100914]: pgmap v254: 89 pgs: 89 active+clean; 2.1 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 168 MiB/s wr, 2.69k op/s 2024-09-21T10:04:04.732 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:04:04.732 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:04:04.760 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:04.763 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:04:04.763 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:04:04.794 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:04:04.794 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:04:04.795 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:04.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:04.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:04.823 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:04:04.823 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:04:04.851 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:04:04.851 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:04:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:04 smithi196 ceph-mon[84784]: pgmap v254: 89 pgs: 89 active+clean; 2.1 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 168 MiB/s wr, 2.69k op/s 2024-09-21T10:04:04.898 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:04.932 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:04.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:04.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:05.033 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:05.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:05.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:05.171 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:05.208 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:05.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:05.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:05.310 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:05.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:05.346 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:05.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:05.447 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:05.480 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:05.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:05.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:05.582 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:05.615 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:05.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:05.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:05.662 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 16 16 43228 43212 168.78 135.125 0.00489913 0.00591987 2024-09-21T10:04:05.717 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:05.750 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:05.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:05.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:05.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:05 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4241096377' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:04:05.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:05 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4241096377' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:04:05.852 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:05.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:05 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4241096377' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:04:05.888 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:05.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:05.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:05.990 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:06.023 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:06.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:06.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:06.125 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:06.160 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:06.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:06.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:06.262 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:06.296 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:06.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:06.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:06.398 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:06.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:06.435 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:06.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:06.491 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:06.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:06.575 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:06.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:06.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:06.662 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 17 16 45380 45364 166.763 134.5 0.00686355 0.00599216 2024-09-21T10:04:06.677 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:06.719 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:06.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:06.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:06.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:06 smithi196 ceph-mon[84784]: pgmap v255: 89 pgs: 89 active+clean; 2.3 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 157 MiB/s wr, 2.52k op/s 2024-09-21T10:04:06.821 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:06.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:06 smithi098 ceph-mon[97580]: pgmap v255: 89 pgs: 89 active+clean; 2.3 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 157 MiB/s wr, 2.52k op/s 2024-09-21T10:04:06.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:06 smithi098 ceph-mon[100914]: pgmap v255: 89 pgs: 89 active+clean; 2.3 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 157 MiB/s wr, 2.52k op/s 2024-09-21T10:04:06.855 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:06.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:06.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:06.957 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:06.994 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:06.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:06.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:07.096 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:07.133 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:07.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:07.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:07.235 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:07.275 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:07.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:07.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:07.376 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:07.413 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:07.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:07.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:07.515 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:07.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:07.549 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:07.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:07.650 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:07.662 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 18 16 47200 47184 163.818 113.75 0.0106834 0.0061002 2024-09-21T10:04:07.688 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:07.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:07.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:07.789 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:07.823 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:07.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:07.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:07.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:07.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:07.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:07.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:07.960 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:07.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:07.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:08.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:08.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:08.097 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:08.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:08.198 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:08.235 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:08.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:08.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:08.337 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:08.376 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:08.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:08.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:08.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:08.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:08.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:08.616 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:08.652 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:08.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:08.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:08.662 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 19 16 49192 49176 161.747 124.5 0.00977237 0.00617756 2024-09-21T10:04:08.667 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:08 smithi098 ceph-mon[97580]: pgmap v256: 89 pgs: 89 active+clean; 2.4 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 151 MiB/s wr, 2.41k op/s 2024-09-21T10:04:08.668 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:08 smithi098 ceph-mon[100914]: pgmap v256: 89 pgs: 89 active+clean; 2.4 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 151 MiB/s wr, 2.41k op/s 2024-09-21T10:04:08.754 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:08.789 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:08.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:08.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:08 smithi196 ceph-mon[84784]: pgmap v256: 89 pgs: 89 active+clean; 2.4 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 151 MiB/s wr, 2.41k op/s 2024-09-21T10:04:08.891 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:08.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:08.930 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:08.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:08.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:04:09.031 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:09.064 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:09.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:09.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:09.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:09.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:09.213 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:09.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:09.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:09.363 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:09.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:09.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:09.465 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:09.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:09.503 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:09.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:09.604 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:09.662 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:04:09.661383+0000 min lat: 0.00106087 max lat: 0.125969 avg lat: 0.00627008 2024-09-21T10:04:09.662 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:04:09.662 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 20 16 51014 50998 159.353 113.875 0.00874185 0.00627008 2024-09-21T10:04:09.669 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:09.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:09.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:09.771 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:09.803 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:09.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:09.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:09.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:09.940 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:09.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:09.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:10.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:10.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:10.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:10.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:10.215 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:10.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:10.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:10.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:10.353 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:10.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:10.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:10.455 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:10.487 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:10.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:10.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:10.537 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:04:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:04:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:10.552 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:04:10.590 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:10.623 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:10.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:10.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:10.662 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 21 16 52863 52847 157.267 115.562 0.00906315 0.00635352 2024-09-21T10:04:10.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:10.761 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:10.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:10.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:10.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:04:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:04:10] "GET /metrics HTTP/1.1" 200 36447 "" "Prometheus/2.51.0" 2024-09-21T10:04:10.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:10 smithi098 ceph-mon[100914]: pgmap v257: 89 pgs: 89 active+clean; 2.9 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 149 MiB/s wr, 2.38k op/s 2024-09-21T10:04:10.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:10 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/672951432' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:04:10.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:10 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:04:10.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:10 smithi098 ceph-mon[97580]: pgmap v257: 89 pgs: 89 active+clean; 2.9 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 149 MiB/s wr, 2.38k op/s 2024-09-21T10:04:10.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:10 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/672951432' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:04:10.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:10 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:04:10.863 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:10 smithi196 ceph-mon[84784]: pgmap v257: 89 pgs: 89 active+clean; 2.9 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 149 MiB/s wr, 2.38k op/s 2024-09-21T10:04:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:10 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/672951432' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:04:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:10 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:04:10.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:10.900 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:10.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:11.001 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:11.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:11.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:11.137 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:11.158 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:11.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:11.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:11.260 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:11.299 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:11.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:11.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:11.401 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:11.429 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:11.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:11.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:11.531 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:11.568 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:11.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:11.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:11.662 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 22 16 54554 54538 154.922 105.688 0.00522124 0.00645013 2024-09-21T10:04:11.670 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:11.706 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:11.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:11.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:11.808 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:11.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:11.843 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:11.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:11.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:11 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:04:11.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:11 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:04:11.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:11 smithi098 ceph-mon[97580]: osdmap e111: 8 total, 8 up, 8 in 2024-09-21T10:04:11.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:11 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:04:11.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:11 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:04:11.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:11 smithi098 ceph-mon[100914]: osdmap e111: 8 total, 8 up, 8 in 2024-09-21T10:04:11.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:11 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:04:11.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:11 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:04:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:11 smithi196 ceph-mon[84784]: osdmap e111: 8 total, 8 up, 8 in 2024-09-21T10:04:11.945 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:11.982 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:11.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:11.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:12.084 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:12.120 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:12.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:12.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:12.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:12.256 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:12.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:12.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:12.358 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:12.396 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:12.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:12.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:12.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:12.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:12.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:12.633 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:12.648 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:12 smithi098 ceph-mon[97580]: pgmap v259: 89 pgs: 89 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-09-21T10:04:12.648 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:12 smithi098 ceph-mon[100914]: pgmap v259: 89 pgs: 89 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-09-21T10:04:12.663 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 23 16 56378 56362 153.139 114 0.00844375 0.00652483 2024-09-21T10:04:12.671 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:12.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:12.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:12.774 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:12.809 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:12.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:12.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:12 smithi196 ceph-mon[84784]: pgmap v259: 89 pgs: 89 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-09-21T10:04:12.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:12.947 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:12.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:12.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:12.976 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:04:12.976 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:04:13.049 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:13.083 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:13.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:13.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:13.184 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:13.215 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:13.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:13.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:13.317 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:13.353 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:13.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:13.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:13.454 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:13.491 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:13.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:13.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:13.593 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:13.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:13.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:13.663 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 24 16 58109 58093 151.266 108.188 0.00861164 0.00660584 2024-09-21T10:04:13.670 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:04:13.732 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:13.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:13.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:13.892 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:13.930 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:13.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:13.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:14.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:14.063 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:14.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:14.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:14.165 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:14.199 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:14.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:14.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:14.301 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:14.335 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:14.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:14.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:14.380 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:14.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:04:14.434 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:14.437 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:14.465 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:14.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:14.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:14.567 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:14.598 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:14.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:14.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:14.663 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 25 16 59651 59635 149.068 96.375 0.0079082 0.0067029 2024-09-21T10:04:14.700 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:14.728 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:14.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:14.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:14.819 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:14 smithi098 ceph-mon[97580]: pgmap v260: 89 pgs: 89 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 122 MiB/s wr, 1.96k op/s 2024-09-21T10:04:14.819 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:14 smithi098 ceph-mon[100914]: pgmap v260: 89 pgs: 89 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 122 MiB/s wr, 1.96k op/s 2024-09-21T10:04:14.830 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:14.869 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:14.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:14.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:14 smithi196 ceph-mon[84784]: pgmap v260: 89 pgs: 89 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 122 MiB/s wr, 1.96k op/s 2024-09-21T10:04:14.971 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:15.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:15.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:15.112 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:15.153 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:15.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:15.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:15.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:15.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:15.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:15.398 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:15.435 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:15.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:15.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:15.537 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:15.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:15.575 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:15.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:15.579 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T10:04:15.663 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 26 16 61331 61315 147.373 105 0.0125993 0.00678078 2024-09-21T10:04:15.676 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:15.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:15 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4095785063' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:04:15.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:15 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:04:15.704 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:15 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4095785063' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:04:15.704 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:15 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:04:15.713 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:15.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:15.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:15.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:15.853 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:15.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:15 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4095785063' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:04:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:15 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:04:15.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:15.990 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:15.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:15.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:16.094 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:16.124 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:16.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:16.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:16.226 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:16.267 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:16.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:16.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:16.366 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T10:04:16.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:16.403 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:16.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:16.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:16.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:16.541 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:16.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:16.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:16.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:16 smithi098 ceph-mon[97580]: pgmap v261: 89 pgs: 89 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 120 MiB/s wr, 1.91k op/s 2024-09-21T10:04:16.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:16 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:04:16.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:16 smithi098 ceph-mon[97580]: osdmap e112: 8 total, 8 up, 8 in 2024-09-21T10:04:16.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:16 smithi098 ceph-mon[100914]: pgmap v261: 89 pgs: 89 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 120 MiB/s wr, 1.91k op/s 2024-09-21T10:04:16.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:16 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:04:16.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:16 smithi098 ceph-mon[100914]: osdmap e112: 8 total, 8 up, 8 in 2024-09-21T10:04:16.643 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:16.666 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 27 16 62999 62983 145.766 104.25 0.00859483 0.0068549 2024-09-21T10:04:16.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:16.690 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:16.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:16.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:16.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:16.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:16 smithi196 ceph-mon[84784]: pgmap v261: 89 pgs: 89 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 120 MiB/s wr, 1.91k op/s 2024-09-21T10:04:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:16 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:04:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:16 smithi196 ceph-mon[84784]: osdmap e112: 8 total, 8 up, 8 in 2024-09-21T10:04:16.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:16.966 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:16.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:16.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:17.068 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:17.111 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:17.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:17.213 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:17.249 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:17.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:17.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:17.278 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:04:17.278 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:04:17.310 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:04:17.310 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:04:17.341 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:04:17.341 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:04:17.350 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:17.369 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:04:17.369 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:04:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:17.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:17.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:17.399 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:04:17.399 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:04:17.429 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:04:17.430 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:04:17.459 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:04:17.459 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:04:17.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:17.493 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:04:17.493 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:04:17.526 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:17.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:17.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:17.621 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:17 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/196700656' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:04:17.621 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:17 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/196700656' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:04:17.628 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:17.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:17.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:17.665 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 28 16 64608 64592 144.151 100.562 0.0162897 0.00693266 2024-09-21T10:04:17.762 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:17.800 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:17.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:17.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:17.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:17 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/196700656' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:04:17.903 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:17.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:17.944 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:17.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:17.953 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:18.045 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:18.085 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:18.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:18.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:18.187 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:18.223 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:18.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:18.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:18.325 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:18.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:18.360 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:18.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:18.461 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:18.495 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:18.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:18.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:18.597 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:18.630 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:18.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:18.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:18.665 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 29 16 66226 66210 142.667 101.125 0.00624889 0.00700468 2024-09-21T10:04:18.732 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:18.762 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:18.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:18.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:18 smithi098 ceph-mon[97580]: pgmap v263: 89 pgs: 89 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-09-21T10:04:18.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:18 smithi098 ceph-mon[100914]: pgmap v263: 89 pgs: 89 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-09-21T10:04:18.864 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:18.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:18 smithi196 ceph-mon[84784]: pgmap v263: 89 pgs: 89 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-09-21T10:04:18.898 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:18.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:18.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:19.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:19.021 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:19.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:19.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:19.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:19.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:19.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:19.259 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:19.290 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:19.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:19.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:19.394 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:19.429 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:19.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:19.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:19.531 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:19.571 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:19.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:19.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:19.666 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 30 16 67806 67790 141.203 98.75 0.0115596 0.00707704 2024-09-21T10:04:19.673 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:19.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:19.709 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:19.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:19.775 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:19.809 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:19.810 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:04:19.810 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:04:19.810 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-21T10:04:19.810 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:04:19.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:19.844 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:19.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:19.945 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:19.981 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:19.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:19.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:20.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:20.114 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:20.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:20.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:20.216 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:20.256 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:20.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:20.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:20.358 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:20.389 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:20.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:20.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:20.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:20.528 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:20.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:20.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:20.594 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:04:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:04:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:20.630 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:20.666 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 31 16 69360 69344 139.781 97.125 0.0162297 0.00714893 2024-09-21T10:04:20.666 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:20.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:20.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:20.768 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:20.808 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:20.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:20.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:20.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:20 smithi098 ceph-mon[97580]: pgmap v264: 89 pgs: 89 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-09-21T10:04:20.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:04:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:04:20] "GET /metrics HTTP/1.1" 200 36447 "" "Prometheus/2.51.0" 2024-09-21T10:04:20.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:20 smithi098 ceph-mon[100914]: pgmap v264: 89 pgs: 89 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-09-21T10:04:20.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:20 smithi196 ceph-mon[84784]: pgmap v264: 89 pgs: 89 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-09-21T10:04:20.910 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:20.946 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:20.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:20.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:21.048 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:21.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:21.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:21.181 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:21.213 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:04:21.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:21.225 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:21.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:21.326 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:21.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:21.369 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:21.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:21.470 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:21.509 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:21.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:21.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:21.610 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:21.648 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:21.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:21.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:21.666 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 32 16 70896 70880 138.413 96 0.0125168 0.00722024 2024-09-21T10:04:21.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:21.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:21.787 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:21.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:21.888 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:21.926 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:21.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:21.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:22.028 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:22.067 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:22.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:22.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:22.169 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:22.205 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:22.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:22.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:22.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:22.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:22.340 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:22.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:22.441 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:22.477 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:22.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:22.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:22.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:22.610 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:22.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:22.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:22.666 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 33 16 72296 72280 136.87 87.5 0.0105289 0.00728907 2024-09-21T10:04:22.666 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:22 smithi098 ceph-mon[97580]: pgmap v265: 89 pgs: 89 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-09-21T10:04:22.666 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:22.667 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:22 smithi098 ceph-mon[100914]: pgmap v265: 89 pgs: 89 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-09-21T10:04:22.667 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:22.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:22.740 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:22.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:22.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:22.842 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:22.877 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:22.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:22.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:22 smithi196 ceph-mon[84784]: pgmap v265: 89 pgs: 89 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-09-21T10:04:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:22.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:23.012 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:23.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:23.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:23.114 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:23.148 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:23.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:23.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:23.250 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:23.282 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:23.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:23.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:23.384 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:23.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:23.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:23.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:23.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:23.540 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:23.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:23.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:23.616 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T10:04:23.642 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:23.666 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 34 16 73670 73654 135.37 85.875 0.0134585 0.0073823 2024-09-21T10:04:23.677 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:23.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:23.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:23.779 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:23.815 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:23.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:23.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:23.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:23 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1283295130' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:04:23.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:23 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:04:23.851 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:23 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1283295130' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:04:23.851 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:23 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:04:23.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:23 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1283295130' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:04:23.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:23 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:04:23.917 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:23.954 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:23.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:23.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:24.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:24.093 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:24.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:24.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:24.195 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:24.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:24.227 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:24.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:24.328 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:24.367 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:24.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:24.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:24.469 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:24.503 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:24.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:24.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:24.605 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:24.640 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:24.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:24.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:24.666 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 35 16 75140 75124 134.127 91.875 0.0124112 0.00745176 2024-09-21T10:04:24.743 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:24.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:24.775 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:24.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:24 smithi098 ceph-mon[97580]: pgmap v266: 89 pgs: 89 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:04:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:24 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:04:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:24 smithi098 ceph-mon[97580]: osdmap e113: 8 total, 8 up, 8 in 2024-09-21T10:04:24.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:24 smithi098 ceph-mon[100914]: pgmap v266: 89 pgs: 89 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:04:24.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:24 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:04:24.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:24 smithi098 ceph-mon[100914]: osdmap e113: 8 total, 8 up, 8 in 2024-09-21T10:04:24.876 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:24 smithi196 ceph-mon[84784]: pgmap v266: 89 pgs: 89 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:04:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:24 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:04:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:24 smithi196 ceph-mon[84784]: osdmap e113: 8 total, 8 up, 8 in 2024-09-21T10:04:24.905 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:24.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:24.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:25.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:25.040 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:25.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:25.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:25.142 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:25.171 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:25.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:25.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:25.272 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:25.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:25.294 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:25.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:25.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:25.432 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:25.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:25.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:25.534 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:25.566 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:25.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:25.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:04:25.666 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 36 16 76536 76520 132.825 87.25 0.0251896 0.00752349 2024-09-21T10:04:25.668 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:25.704 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:25.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:25.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:25.806 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:25.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:25.842 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:25.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:25.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:25 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2903213716' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:04:25.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:25 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:04:25.943 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:25.980 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:25.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:25.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:26.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:26.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:25 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2903213716' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:04:26.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:25 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:04:26.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:25 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2903213716' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:04:26.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:25 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:04:26.117 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:26.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:26.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:26.219 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:26.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:26.257 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:26.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:26.358 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:26.386 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:26.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:26.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:26.488 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:26.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:26.527 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:26.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:26.628 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:26.666 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 37 16 77926 77910 131.583 86.875 0.0114236 0.00759512 2024-09-21T10:04:26.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:26.668 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:26.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:26.769 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:26.804 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:26.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:26.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:26 smithi196 ceph-mon[84784]: pgmap v268: 89 pgs: 89 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-21T10:04:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:26 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:04:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:26 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:04:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:26 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:04:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:26 smithi196 ceph-mon[84784]: osdmap e114: 8 total, 8 up, 8 in 2024-09-21T10:04:26.906 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:26.942 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:26.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:26.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:27.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:27.079 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:27.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:27.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:26 smithi098 ceph-mon[97580]: pgmap v268: 89 pgs: 89 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-21T10:04:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:26 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:04:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:26 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:04:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:26 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:04:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:26 smithi098 ceph-mon[97580]: osdmap e114: 8 total, 8 up, 8 in 2024-09-21T10:04:27.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:26 smithi098 ceph-mon[100914]: pgmap v268: 89 pgs: 89 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-21T10:04:27.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:26 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:04:27.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:26 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:04:27.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:26 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:04:27.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:26 smithi098 ceph-mon[100914]: osdmap e114: 8 total, 8 up, 8 in 2024-09-21T10:04:27.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:27.212 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:27.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:27.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:27.314 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:27.351 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:27.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:27.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:27.452 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:27.485 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:27.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:27.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:27.587 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:27.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:27.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:27.666 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 38 16 79299 79283 130.378 85.8125 0.0119574 0.00766537 2024-09-21T10:04:27.720 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:27.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:27.756 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:27.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:27.857 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:27.894 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:27.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:27.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:27.996 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:28.037 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:28.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:28.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:28.076 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:04:28.139 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:28.177 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:28.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:28.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:28.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T10:04:28.279 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:28.312 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:28.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:28.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:28.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:28.496 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:28.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:28.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:28.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:28.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:28.634 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:28.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:28.666 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 39 16 80646 80630 129.194 84.1875 0.00698015 0.0077363 2024-09-21T10:04:28.734 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:28.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:28.769 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:28.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:28.870 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:28 smithi196 ceph-mon[84784]: pgmap v270: 89 pgs: 89 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-21T10:04:28.904 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:28.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:28.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:28.909 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:28 smithi098 ceph-mon[97580]: pgmap v270: 89 pgs: 89 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-21T10:04:28.910 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:28 smithi098 ceph-mon[100914]: pgmap v270: 89 pgs: 89 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-21T10:04:29.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:29.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:29.041 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:29.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:29.142 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:29.180 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:29.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:29.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:29.282 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:29.309 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:29.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:29.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:29.411 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:29.450 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:29.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:29.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:29.552 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:29.582 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:29.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:29.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:29.667 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:04:29.665859+0000 min lat: 0.00106087 max lat: 0.125969 avg lat: 0.00780896 2024-09-21T10:04:29.667 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:04:29.667 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 40 16 81938 81922 127.982 80.75 0.0136944 0.00780896 2024-09-21T10:04:29.684 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:29.717 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:29.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:29.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:29.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:29.850 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:29.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:29.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:29.911 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:29.953 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:29.980 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:29.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:29.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:29.997 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:30.082 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:30.125 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:30.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:30.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:30.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:30.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:30.269 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:30.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:30.370 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:30.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:30.410 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:30.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:30.511 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:30.545 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:30.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:30.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:30.636 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:04:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:04:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:30.647 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:30.667 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 41 16 83266 83250 126.885 83 0.00488892 0.00787602 2024-09-21T10:04:30.681 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:30.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:30.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:30.734 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:30.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:30.831 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:30.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:30.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:30.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:04:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:04:30] "GET /metrics HTTP/1.1" 200 36453 "" "Prometheus/2.51.0" 2024-09-21T10:04:30.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:30 smithi098 ceph-mon[100914]: pgmap v271: 89 pgs: 89 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-21T10:04:30.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:30 smithi098 ceph-mon[97580]: pgmap v271: 89 pgs: 89 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-21T10:04:30.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:30 smithi196 ceph-mon[84784]: pgmap v271: 89 pgs: 89 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-21T10:04:30.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:30.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:30.977 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:30.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:31.078 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:31.112 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:31.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:31.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:31.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:31.246 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:31.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:31.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:31.347 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:31.382 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:31.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:31.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:31.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:31.524 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:31.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:31.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:31.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:31.668 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:31.668 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 42 16 84547 84531 125.77 80.0625 0.0189782 0.0079453 2024-09-21T10:04:31.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:31.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:31.770 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:31.802 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:31.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:31.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:31.903 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:31.934 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:31.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:31.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:31.957 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:04:32.036 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:32.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:32.071 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:32.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:32.172 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:32.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:32.232 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:32.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:32.333 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:32.376 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:32.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:32.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:32.478 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:32.515 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:32.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:32.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:32.617 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:32.638 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:32.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:32.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:32.667 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 43 16 85737 85721 124.575 74.375 0.0608109 0.00801482 2024-09-21T10:04:32.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:32.771 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:32.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:32.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:32.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:32 smithi098 ceph-mon[97580]: pgmap v272: 89 pgs: 89 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T10:04:32.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:32 smithi098 ceph-mon[100914]: pgmap v272: 89 pgs: 89 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T10:04:32.872 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:32.905 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:32.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:32.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:33.006 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:33.040 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:33.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:33.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:33.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:32 smithi196 ceph-mon[84784]: pgmap v272: 89 pgs: 89 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T10:04:33.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:33.167 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:33.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:33.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:33.269 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:33.302 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:33.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:33.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:33.404 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:33.424 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:33.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:33.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:33.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:33.565 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:33.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:33.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:33.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:33.668 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 44 16 86866 86850 123.347 70.5625 0.0163735 0.00810256 2024-09-21T10:04:33.700 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:33.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:33.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:33.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:33.835 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:33.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:33.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:33.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:33.970 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:33.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:33.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:34.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:34.103 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:34.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:34.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:34.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:34.239 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:34.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:34.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:34.341 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:34.381 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:34.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:34.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:34.483 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:34.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:34.520 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:34.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:34.621 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:34.654 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:34.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:34.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:34.667 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 45 16 88216 88200 122.481 84.375 0.0151906 0.00816015 2024-09-21T10:04:34.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:34.791 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:34.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:34.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:34.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:34 smithi098 ceph-mon[97580]: pgmap v273: 89 pgs: 89 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-21T10:04:34.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:34 smithi098 ceph-mon[100914]: pgmap v273: 89 pgs: 89 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-21T10:04:34.894 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:34.960 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:34.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:34.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:35.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:35.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:35.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:35.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:35.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:34 smithi196 ceph-mon[84784]: pgmap v273: 89 pgs: 89 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-21T10:04:35.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:35.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:35.244 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:35.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:35.345 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:35.381 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:35.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:35.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:35.483 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:35.520 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:35.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:35.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:35.622 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:35.665 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:35.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:35.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:35.667 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 46 16 89489 89473 121.548 79.5625 0.0300525 0.00822156 2024-09-21T10:04:35.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:35.789 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:04:35.789 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:04:35.800 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:35.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:35.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:35.820 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:04:35.820 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:04:35.857 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:04:35.857 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:04:35.885 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:04:35.886 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:04:35.901 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:35.917 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:04:35.917 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:04:35.937 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:35.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:35.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:35.955 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:04:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:04:35.988 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:04:35.988 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:04:36.025 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:04:36.025 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:04:36.039 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:36.082 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:36.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:36.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:36.184 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:36.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:36.223 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:36.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:36.324 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:36.354 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:36.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:36.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:36.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:36.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:36.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:36.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:36.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:36.627 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:36.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:36.667 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 47 16 90761 90745 120.653 79.5 0.0126694 0.00828364 2024-09-21T10:04:36.715 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:36.728 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:36.782 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:36.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:36.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:36.806 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:36 smithi098 ceph-mon[97580]: pgmap v274: 89 pgs: 89 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-21T10:04:36.806 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:36 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2237595225' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:04:36.807 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:36 smithi098 ceph-mon[100914]: pgmap v274: 89 pgs: 89 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-21T10:04:36.807 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:36 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2237595225' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:04:36.884 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:36.925 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:36.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:36.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:36.940 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:36 smithi196 ceph-mon[84784]: pgmap v274: 89 pgs: 89 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-21T10:04:36.940 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:36 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2237595225' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:04:37.027 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:37.060 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:37.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:37.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:37.162 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:37.196 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:37.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:37.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:37.298 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:37.330 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:37.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:37.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:37.432 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:37.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:37.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:37.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:37.599 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:37.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:37.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:37.667 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 48 16 92006 91990 119.761 77.8125 0.00935732 0.00834578 2024-09-21T10:04:37.689 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T10:04:37.701 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:37.737 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:37.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:37.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:37.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:37 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2204168561' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:04:37.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:37 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:04:37.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:04:37.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:37.741 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:37 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2204168561' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:04:37.741 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:37 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:04:37.741 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:04:37.741 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:37.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:37.872 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:37.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:37.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:37.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:37.996 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:37.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:37.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:38.098 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:38.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:38.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:38.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:37 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2204168561' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:04:38.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:37 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:04:38.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:04:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:38.232 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:38.266 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:38.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:38.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:38.368 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:38.399 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:38.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:38.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:38.501 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:38.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:38.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:38.637 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:38.667 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 49 16 93193 93177 118.831 74.1875 0.00665117 0.00841089 2024-09-21T10:04:38.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:38.670 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:38.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:38.771 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:38.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:38.799 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:38.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:38.900 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:38.940 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:38.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:38.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:39.042 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:39.075 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:39.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:39.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:39.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:38 smithi098 ceph-mon[97580]: pgmap v275: 89 pgs: 89 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-21T10:04:39.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:38 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:04:39.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:38 smithi098 ceph-mon[97580]: osdmap e115: 8 total, 8 up, 8 in 2024-09-21T10:04:39.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:38 smithi098 ceph-mon[100914]: pgmap v275: 89 pgs: 89 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-21T10:04:39.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:38 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:04:39.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:38 smithi098 ceph-mon[100914]: osdmap e115: 8 total, 8 up, 8 in 2024-09-21T10:04:39.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:38 smithi196 ceph-mon[84784]: pgmap v275: 89 pgs: 89 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-21T10:04:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:38 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:04:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:38 smithi196 ceph-mon[84784]: osdmap e115: 8 total, 8 up, 8 in 2024-09-21T10:04:39.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:39.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:39.224 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:39.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:39.292 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:04:39.293 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:04:39.325 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:39.361 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:39.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:39.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:39.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:39.492 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:39.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:39.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:39.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:04:39.594 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:39.640 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:39.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:39.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:39.667 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 50 16 94395 94379 117.956 75.125 0.010569 0.00847319 2024-09-21T10:04:39.743 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:39.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:39.780 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:39.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:39.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:39.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T10:04:39.918 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:39.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:39.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:40.019 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:40.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:40.070 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:40.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:40.171 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:40.206 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:40.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:40.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:40.308 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:40.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:40.427 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:40.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:40.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:40.564 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:40.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:40.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:40.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:04:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:04:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:40.666 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:40.668 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 51 16 95531 95515 117.036 71 0.00899745 0.00853939 2024-09-21T10:04:40.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:40.702 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:40.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:40.803 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:40.844 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:40.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:40.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:40.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:40 smithi098 ceph-mon[97580]: pgmap v277: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T10:04:40.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:40 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:04:40.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:04:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:04:40] "GET /metrics HTTP/1.1" 200 36454 "" "Prometheus/2.51.0" 2024-09-21T10:04:40.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:40 smithi098 ceph-mon[100914]: pgmap v277: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T10:04:40.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:40 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:04:40.946 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:40.993 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:40.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:40.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:41.024 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:40 smithi196 ceph-mon[84784]: pgmap v277: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T10:04:41.025 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:40 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:04:41.095 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:41.138 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:41.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:41.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:41.240 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:41.275 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:41.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:41.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:41.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:04:41.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:41.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:41.443 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:41.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:41.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:41.594 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:41.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:41.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:41.669 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 52 16 96677 96661 116.159 71.625 0.0194215 0.00860423 2024-09-21T10:04:41.696 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:41.732 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:41.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:41.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:41.833 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:41.872 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:41.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:41.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:41.977 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:42.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:42.019 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:42.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:42.058 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:42.062 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:42.120 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:42.163 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:42.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:42.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:42.266 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:42.310 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:42.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:42.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:42.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:42.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:42.469 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:42.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:42.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:42.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:42.616 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:42.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:42.671 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 53 16 97727 97711 115.202 65.625 0.0153852 0.00867538 2024-09-21T10:04:42.718 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:42.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:42.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:42.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:42.859 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:42.889 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:42.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:42.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:42.991 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:43.020 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:43.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:43.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:43.025 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:42 smithi196 ceph-mon[84784]: pgmap v278: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T10:04:43.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:42 smithi098 ceph-mon[97580]: pgmap v278: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T10:04:43.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:42 smithi098 ceph-mon[100914]: pgmap v278: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T10:04:43.122 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:43.159 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:43.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:43.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:43.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:43.295 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:43.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:43.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:43.397 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:43.436 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:43.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:43.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:43.537 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:43.586 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:43.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:43.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:43.671 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 54 16 98883 98867 114.406 72.25 0.00390982 0.00873565 2024-09-21T10:04:43.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:43.721 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:43.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:43.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:43.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:43.858 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:43.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:43.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:43.959 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:44.001 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:44.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:44.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:44.005 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:44.103 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:44.137 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:44.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:44.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:44.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:44.279 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:44.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:44.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:44.381 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:44.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:44.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:44.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:44.519 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:44.561 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:44.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:44.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:44.663 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:44.672 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 55 16 100017 100001 113.612 70.875 0.024937 0.0087968 2024-09-21T10:04:44.698 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:44.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:44.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:44.729 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:04:44.801 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:44.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:44.835 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:44.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:44.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:44 smithi098 ceph-mon[97580]: pgmap v279: 89 pgs: 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-21T10:04:44.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:44 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/18607737' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:04:44.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:44 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:04:44.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:44 smithi098 ceph-mon[100914]: pgmap v279: 89 pgs: 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-21T10:04:44.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:44 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/18607737' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:04:44.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:44 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:04:44.936 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:44.982 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:44.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:44.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:45.013 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:44 smithi196 ceph-mon[84784]: pgmap v279: 89 pgs: 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-21T10:04:45.013 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:44 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/18607737' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:04:45.013 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:44 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:04:45.084 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:45.121 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:45.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:45.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:45.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:45.254 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:45.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:45.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:45.356 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:45.389 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:45.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:45.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:45.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:45.521 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:45.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:45.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:45.623 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:45.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:45.666 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:45.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:45.672 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 56 16 101168 101152 112.868 71.9375 0.00748476 0.00885472 2024-09-21T10:04:45.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:45.803 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:45.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:45.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:45.805 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:45 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:04:45.805 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:45 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:04:45.805 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:45 smithi098 ceph-mon[97580]: osdmap e116: 8 total, 8 up, 8 in 2024-09-21T10:04:45.805 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:45 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:04:45.806 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:45 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:04:45.806 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:45 smithi098 ceph-mon[100914]: osdmap e116: 8 total, 8 up, 8 in 2024-09-21T10:04:45.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:45.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:45.942 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:45.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:46.044 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:46.072 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:46.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:46.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:45 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:04:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:45 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:04:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:45 smithi196 ceph-mon[84784]: osdmap e116: 8 total, 8 up, 8 in 2024-09-21T10:04:46.173 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:46.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:46.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:46.302 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:46.336 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:46.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:46.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:46.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:46.472 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:46.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:46.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:46.573 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:46.605 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:46.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:46.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:46.672 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 57 16 102259 102243 112.084 68.1875 0.0141695 0.00891665 2024-09-21T10:04:46.707 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:46.744 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:46.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:46.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:46.846 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:46.864 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:46 smithi098 ceph-mon[97580]: pgmap v281: 89 pgs: 89 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-21T10:04:46.865 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:04:46.865 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:04:46.865 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:46 smithi098 ceph-mon[100914]: pgmap v281: 89 pgs: 89 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-21T10:04:46.865 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:04:46.865 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:04:46.891 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:46.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:46.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:46.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:47.026 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:47.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:47.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:47.127 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:47.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:46 smithi196 ceph-mon[84784]: pgmap v281: 89 pgs: 89 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-21T10:04:47.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:04:47.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:04:47.165 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:47.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:47.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:47.267 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:47.302 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:47.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:47.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:47.404 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:47.437 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:47.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:47.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:47.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:47.576 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:47.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:47.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:47.673 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 58 16 103408 103392 111.39 71.8125 0.00698866 0.00897245 2024-09-21T10:04:47.677 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:47.712 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:47.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:47.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:47.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:47.823 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:04:47.824 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:04:47.845 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:47.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:47.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:47.855 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:04:47.856 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:04:47.891 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:04:47.891 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:04:47.926 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:04:47.926 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:04:47.947 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:47.956 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:04:47.956 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:04:47.981 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:47.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:47.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:47.982 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:04:47.983 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:04:48.012 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:04:48.012 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:04:48.045 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:04:48.045 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:04:48.082 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:48.120 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:48.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:48.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:48.182 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:04:48.221 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:48.265 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:48.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:48.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:48.368 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:48.398 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:48.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:48.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:48.500 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:48.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:48.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:48.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:48.673 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 59 16 104511 104495 110.67 68.9375 0.0111667 0.00903104 2024-09-21T10:04:48.684 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:48.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:48.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:48.786 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:48.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:48.822 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:48.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:48.906 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:48 smithi098 ceph-mon[97580]: pgmap v282: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T10:04:48.906 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:48 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1891727038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:04:48.906 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:48 smithi098 ceph-mon[100914]: pgmap v282: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T10:04:48.906 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:48 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1891727038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:04:48.924 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:48.962 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:48.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:48.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:49.064 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:49.108 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:49.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:49.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:49.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:48 smithi196 ceph-mon[84784]: pgmap v282: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T10:04:49.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:48 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1891727038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:04:49.211 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:49.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:49.247 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:49.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:49.348 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:49.385 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:49.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:49.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:49.487 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:49.516 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:49.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:49.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:49.618 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:49.650 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:49.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:49.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:49.674 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:04:49.672124+0000 min lat: 0.00106087 max lat: 0.125969 avg lat: 0.00908477 2024-09-21T10:04:49.674 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:04:49.674 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 60 16 105653 105637 110.015 71.375 0.013446 0.00908477 2024-09-21T10:04:49.751 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:49.783 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:49.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:49.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:49.885 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:49.931 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:49.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:49.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:50.033 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:50.073 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:50.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:50.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:50.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:50.213 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:50.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:50.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:50.316 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:50.354 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:50.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:50.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:50.455 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:50.487 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:50.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:50.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:50.522 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:04:50.522 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-21T10:04:50.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-21T10:04:50.522 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:04:50.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.522 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:64271 5.8 e112/105 5:13935367:::benchmark_data_smithi098_135187_object64270:head v 112'3955, mlcod=112'3948)", 2024-09-21T10:04:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:17.457184+0000", 2024-09-21T10:04:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.056719700999999, 2024-09-21T10:04:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.022284545999999999, 2024-09-21T10:04:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:04:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:04:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:04:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:04:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:04:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:17.457184+0000", 2024-09-21T10:04:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:04:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:17.457184+0000", 2024-09-21T10:04:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:04:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:17.457185+0000", 2024-09-21T10:04:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5499999999999998e-07 2024-09-21T10:04:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:04:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:17.457206+0000", 2024-09-21T10:04:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1495999999999999e-05 2024-09-21T10:04:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:04:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:17.457219+0000", 2024-09-21T10:04:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2473999999999999e-05 2024-09-21T10:04:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:04:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:17.457225+0000", 2024-09-21T10:04:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5690000000000001e-06 2024-09-21T10:04:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:04:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:17.462631+0000", 2024-09-21T10:04:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054056379999999999 2024-09-21T10:04:50.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:04:50.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:17.462648+0000", 2024-09-21T10:04:50.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7118000000000001e-05 2024-09-21T10:04:50.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:04:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:17.479249+0000", 2024-09-21T10:04:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016600319999999998 2024-09-21T10:04:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:04:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:17.479469+0000", 2024-09-21T10:04:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022027599999999999 2024-09-21T10:04:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:04:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:80271 5.7 e114/105 5:ead15099:::benchmark_data_smithi098_135187_object80270:head v 114'5049, mlcod=114'5045)", 2024-09-21T10:04:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:28.371121+0000", 2024-09-21T10:04:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 22.142783397999999, 2024-09-21T10:04:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.021826195999999999, 2024-09-21T10:04:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:04:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:04:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:04:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:04:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:04:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.371121+0000", 2024-09-21T10:04:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:04:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.371121+0000", 2024-09-21T10:04:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:04:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.371123+0000", 2024-09-21T10:04:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.03e-06 2024-09-21T10:04:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:04:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.371155+0000", 2024-09-21T10:04:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2079000000000002e-05 2024-09-21T10:04:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:04:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.371168+0000", 2024-09-21T10:04:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2850000000000001e-05 2024-09-21T10:04:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:04:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.371175+0000", 2024-09-21T10:04:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0700000000000001e-06 2024-09-21T10:04:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:04:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.373966+0000", 2024-09-21T10:04:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027912729999999999 2024-09-21T10:04:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:04:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.374013+0000", 2024-09-21T10:04:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6984000000000003e-05 2024-09-21T10:04:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:04:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.392900+0000", 2024-09-21T10:04:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018886888000000001 2024-09-21T10:04:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:04:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.392947+0000", 2024-09-21T10:04:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7021999999999998e-05 2024-09-21T10:04:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:04:50.533 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.533 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:80269 5.0 e114/105 5:0533addb:::benchmark_data_smithi098_135187_object80268:head v 114'5000, mlcod=114'4998)", 2024-09-21T10:04:50.533 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:28.371484+0000", 2024-09-21T10:04:50.533 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 22.142420215000001, 2024-09-21T10:04:50.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.022549762000000001, 2024-09-21T10:04:50.533 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:04:50.533 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:04:50.533 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:04:50.533 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:04:50.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:04:50.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.371484+0000", 2024-09-21T10:04:50.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:04:50.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.371484+0000", 2024-09-21T10:04:50.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:04:50.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.371485+0000", 2024-09-21T10:04:50.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.02e-07 2024-09-21T10:04:50.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:04:50.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.371510+0000", 2024-09-21T10:04:50.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4890999999999999e-05 2024-09-21T10:04:50.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:04:50.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.371524+0000", 2024-09-21T10:04:50.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4411e-05 2024-09-21T10:04:50.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:04:50.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.371531+0000", 2024-09-21T10:04:50.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.216e-06 2024-09-21T10:04:50.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:04:50.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.371596+0000", 2024-09-21T10:04:50.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5092000000000006e-05 2024-09-21T10:04:50.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:04:50.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.371619+0000", 2024-09-21T10:04:50.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2271000000000001e-05 2024-09-21T10:04:50.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:04:50.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.388522+0000", 2024-09-21T10:04:50.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016903485999999999 2024-09-21T10:04:50.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:04:50.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.394034+0000", 2024-09-21T10:04:50.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055114930000000001 2024-09-21T10:04:50.537 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.537 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:04:50.537 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.537 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:80283 5.7 e114/105 5:e7f80098:::benchmark_data_smithi098_135187_object80282:head v 114'5051, mlcod=114'5047)", 2024-09-21T10:04:50.537 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:28.380696+0000", 2024-09-21T10:04:50.538 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 22.133207466000002, 2024-09-21T10:04:50.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.021158039, 2024-09-21T10:04:50.538 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:04:50.538 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:04:50.538 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:04:50.538 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:04:50.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:04:50.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.380696+0000", 2024-09-21T10:04:50.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:04:50.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.380696+0000", 2024-09-21T10:04:50.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:04:50.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.380697+0000", 2024-09-21T10:04:50.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6499999999999999e-07 2024-09-21T10:04:50.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:04:50.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.383377+0000", 2024-09-21T10:04:50.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026800740000000002 2024-09-21T10:04:50.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:04:50.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.383389+0000", 2024-09-21T10:04:50.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2149e-05 2024-09-21T10:04:50.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:04:50.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.383397+0000", 2024-09-21T10:04:50.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4440000000000002e-06 2024-09-21T10:04:50.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:04:50.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.388355+0000", 2024-09-21T10:04:50.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0049583140000000001 2024-09-21T10:04:50.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:04:50.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.388420+0000", 2024-09-21T10:04:50.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4902999999999998e-05 2024-09-21T10:04:50.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:04:50.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.401555+0000", 2024-09-21T10:04:50.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013135294 2024-09-21T10:04:50.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:04:50.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.401855+0000", 2024-09-21T10:04:50.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000299196 2024-09-21T10:04:50.542 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.542 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:04:50.542 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.542 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:80490 5.7 e114/105 5:ea06f96c:::benchmark_data_smithi098_135187_object80489:head v 114'5064, mlcod=114'5060)", 2024-09-21T10:04:50.542 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:28.563329+0000", 2024-09-21T10:04:50.542 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 21.950575222000001, 2024-09-21T10:04:50.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028221629000000002, 2024-09-21T10:04:50.542 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:04:50.542 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:04:50.542 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:04:50.542 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:04:50.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:04:50.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.563329+0000", 2024-09-21T10:04:50.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.543 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:04:50.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.563329+0000", 2024-09-21T10:04:50.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.543 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:04:50.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.563330+0000", 2024-09-21T10:04:50.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0720000000000001e-06 2024-09-21T10:04:50.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:04:50.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.563364+0000", 2024-09-21T10:04:50.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4527000000000003e-05 2024-09-21T10:04:50.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:04:50.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.563515+0000", 2024-09-21T10:04:50.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000150786 2024-09-21T10:04:50.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:04:50.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.563523+0000", 2024-09-21T10:04:50.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0940000000000008e-06 2024-09-21T10:04:50.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:04:50.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.574505+0000", 2024-09-21T10:04:50.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010981835000000001 2024-09-21T10:04:50.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:04:50.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.574525+0000", 2024-09-21T10:04:50.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0404e-05 2024-09-21T10:04:50.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:04:50.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.591501+0000", 2024-09-21T10:04:50.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016975659000000001 2024-09-21T10:04:50.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:04:50.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:28.591550+0000", 2024-09-21T10:04:50.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9252000000000002e-05 2024-09-21T10:04:50.547 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.547 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:04:50.547 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.547 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:81416 5.0 e114/105 5:0cb8852c:::benchmark_data_smithi098_135187_object81415:head v 114'5063, mlcod=114'5061)", 2024-09-21T10:04:50.547 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:29.266858+0000", 2024-09-21T10:04:50.547 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 21.247045725, 2024-09-21T10:04:50.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.049670543999999997, 2024-09-21T10:04:50.548 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:04:50.548 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:04:50.548 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:04:50.548 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:04:50.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:04:50.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.266858+0000", 2024-09-21T10:04:50.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:04:50.549 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.266858+0000", 2024-09-21T10:04:50.549 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:04:50.549 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.266859+0000", 2024-09-21T10:04:50.549 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6400000000000002e-07 2024-09-21T10:04:50.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:04:50.549 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.266880+0000", 2024-09-21T10:04:50.549 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0866999999999999e-05 2024-09-21T10:04:50.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.550 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:04:50.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.266887+0000", 2024-09-21T10:04:50.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5050000000000001e-06 2024-09-21T10:04:50.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.550 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:04:50.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.266892+0000", 2024-09-21T10:04:50.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7369999999999997e-06 2024-09-21T10:04:50.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.550 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:04:50.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.266949+0000", 2024-09-21T10:04:50.551 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7469999999999997e-05 2024-09-21T10:04:50.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:04:50.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.266971+0000", 2024-09-21T10:04:50.551 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2011e-05 2024-09-21T10:04:50.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:04:50.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316486+0000", 2024-09-21T10:04:50.551 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.049514829000000003 2024-09-21T10:04:50.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.552 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:04:50.552 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316529+0000", 2024-09-21T10:04:50.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2561000000000001e-05 2024-09-21T10:04:50.552 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.552 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:04:50.552 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.552 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:81413 5.7 e114/105 5:eb012c7c:::benchmark_data_smithi098_135187_object81412:head v 114'5126, mlcod=114'5124)", 2024-09-21T10:04:50.552 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:29.268487+0000", 2024-09-21T10:04:50.552 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 21.245416964, 2024-09-21T10:04:50.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.048055411999999999, 2024-09-21T10:04:50.552 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:04:50.552 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:04:50.553 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:04:50.553 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:04:50.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:04:50.553 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.268487+0000", 2024-09-21T10:04:50.553 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:04:50.553 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.268487+0000", 2024-09-21T10:04:50.553 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:04:50.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.268488+0000", 2024-09-21T10:04:50.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4499999999999996e-07 2024-09-21T10:04:50.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:04:50.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.268534+0000", 2024-09-21T10:04:50.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6437000000000003e-05 2024-09-21T10:04:50.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:04:50.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.268547+0000", 2024-09-21T10:04:50.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2927e-05 2024-09-21T10:04:50.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:04:50.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.268554+0000", 2024-09-21T10:04:50.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7519999999999999e-06 2024-09-21T10:04:50.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:04:50.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.269782+0000", 2024-09-21T10:04:50.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001227943 2024-09-21T10:04:50.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.556 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:04:50.556 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.269800+0000", 2024-09-21T10:04:50.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8003e-05 2024-09-21T10:04:50.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.556 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:04:50.556 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316503+0000", 2024-09-21T10:04:50.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.046702746000000003 2024-09-21T10:04:50.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.556 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:04:50.556 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316542+0000", 2024-09-21T10:04:50.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9759000000000003e-05 2024-09-21T10:04:50.556 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.556 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:04:50.557 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.557 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:81410 5.8 e114/105 5:1918b719:::benchmark_data_smithi098_135187_object81409:head v 114'4993, mlcod=114'4991)", 2024-09-21T10:04:50.557 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:29.270376+0000", 2024-09-21T10:04:50.557 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 21.243527647000001, 2024-09-21T10:04:50.557 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.048632575999999997, 2024-09-21T10:04:50.557 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:04:50.557 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:04:50.557 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:04:50.557 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:04:50.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.557 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:04:50.558 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.270376+0000", 2024-09-21T10:04:50.558 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.558 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:04:50.558 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.270376+0000", 2024-09-21T10:04:50.558 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.558 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:04:50.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.270377+0000", 2024-09-21T10:04:50.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0750000000000001e-06 2024-09-21T10:04:50.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:04:50.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.270401+0000", 2024-09-21T10:04:50.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3357999999999999e-05 2024-09-21T10:04:50.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:04:50.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.270412+0000", 2024-09-21T10:04:50.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0903999999999999e-05 2024-09-21T10:04:50.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:04:50.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.270419+0000", 2024-09-21T10:04:50.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0829999999999998e-06 2024-09-21T10:04:50.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.565 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:04:50.565 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.270494+0000", 2024-09-21T10:04:50.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4870999999999995e-05 2024-09-21T10:04:50.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.566 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:04:50.566 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.270511+0000", 2024-09-21T10:04:50.566 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7158000000000001e-05 2024-09-21T10:04:50.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.566 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:04:50.566 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.318957+0000", 2024-09-21T10:04:50.567 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.048446217 2024-09-21T10:04:50.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.567 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:04:50.567 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.319009+0000", 2024-09-21T10:04:50.567 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1910000000000003e-05 2024-09-21T10:04:50.567 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.568 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:04:50.568 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.568 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:81417 5.8 e114/105 5:13f3ab1d:::benchmark_data_smithi098_135187_object81416:head v 114'4994, mlcod=114'4991)", 2024-09-21T10:04:50.568 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:29.271294+0000", 2024-09-21T10:04:50.568 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 21.242610352, 2024-09-21T10:04:50.568 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.047744399999999999, 2024-09-21T10:04:50.569 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:04:50.569 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:04:50.569 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:04:50.569 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:04:50.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.569 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:04:50.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.271294+0000", 2024-09-21T10:04:50.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.570 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:04:50.570 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.271294+0000", 2024-09-21T10:04:50.570 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.571 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:04:50.571 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.271294+0000", 2024-09-21T10:04:50.571 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0699999999999997e-07 2024-09-21T10:04:50.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.571 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:04:50.571 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.271315+0000", 2024-09-21T10:04:50.571 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0879000000000001e-05 2024-09-21T10:04:50.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.572 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.572 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:04:50.572 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.271325+0000", 2024-09-21T10:04:50.572 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4620000000000002e-06 2024-09-21T10:04:50.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.572 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.572 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:04:50.573 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.271331+0000", 2024-09-21T10:04:50.573 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1589999999999998e-06 2024-09-21T10:04:50.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.573 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:04:50.573 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.272973+0000", 2024-09-21T10:04:50.573 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016424409999999999 2024-09-21T10:04:50.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.573 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:04:50.574 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.272991+0000", 2024-09-21T10:04:50.574 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7657999999999999e-05 2024-09-21T10:04:50.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.574 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:04:50.574 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.319012+0000", 2024-09-21T10:04:50.574 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.046021199999999998 2024-09-21T10:04:50.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.575 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:04:50.575 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.319038+0000", 2024-09-21T10:04:50.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5993999999999999e-05 2024-09-21T10:04:50.575 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.575 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:04:50.575 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.576 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:81421 5.7 e114/105 5:e6e67dca:::benchmark_data_smithi098_135187_object81420:head v 114'5127, mlcod=114'5124)", 2024-09-21T10:04:50.576 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:29.272767+0000", 2024-09-21T10:04:50.576 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 21.241137394999999, 2024-09-21T10:04:50.576 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.046366598000000002, 2024-09-21T10:04:50.576 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:04:50.576 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:04:50.577 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:04:50.577 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:04:50.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.577 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:04:50.577 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.272767+0000", 2024-09-21T10:04:50.577 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.578 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:04:50.578 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.272767+0000", 2024-09-21T10:04:50.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.579 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:04:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:50.581 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.272767+0000", 2024-09-21T10:04:50.581 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3099999999999996e-07 2024-09-21T10:04:50.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.581 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:04:50.581 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.272802+0000", 2024-09-21T10:04:50.581 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4922999999999998e-05 2024-09-21T10:04:50.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:04:50.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.272814+0000", 2024-09-21T10:04:50.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1392e-05 2024-09-21T10:04:50.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:04:50.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.272821+0000", 2024-09-21T10:04:50.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.525e-06 2024-09-21T10:04:50.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:04:50.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.272870+0000", 2024-09-21T10:04:50.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9073999999999998e-05 2024-09-21T10:04:50.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.584 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:04:50.584 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.272888+0000", 2024-09-21T10:04:50.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8026999999999999e-05 2024-09-21T10:04:50.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.585 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:04:50.585 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.318793+0000", 2024-09-21T10:04:50.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.045904942999999997 2024-09-21T10:04:50.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.585 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:04:50.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.319133+0000", 2024-09-21T10:04:50.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00033978300000000002 2024-09-21T10:04:50.586 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.586 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:04:50.586 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.587 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:81419 5.0 e114/105 5:0b27c47c:::benchmark_data_smithi098_135187_object81418:head v 114'5064, mlcod=114'5061)", 2024-09-21T10:04:50.587 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:29.272855+0000", 2024-09-21T10:04:50.587 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 21.241048625000001, 2024-09-21T10:04:50.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.046216844, 2024-09-21T10:04:50.587 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:04:50.587 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:04:50.587 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:04:50.588 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:04:50.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.588 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:04:50.588 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.272855+0000", 2024-09-21T10:04:50.588 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.589 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:04:50.589 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.272855+0000", 2024-09-21T10:04:50.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:04:50.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.272856+0000", 2024-09-21T10:04:50.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5000000000000003e-07 2024-09-21T10:04:50.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:04:50.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.272881+0000", 2024-09-21T10:04:50.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5562999999999999e-05 2024-09-21T10:04:50.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.592 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:04:50.592 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.272890+0000", 2024-09-21T10:04:50.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5879999999999998e-06 2024-09-21T10:04:50.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.593 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:04:50.593 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.272896+0000", 2024-09-21T10:04:50.593 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8499999999999999e-06 2024-09-21T10:04:50.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.594 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:04:50.594 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.272962+0000", 2024-09-21T10:04:50.594 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5738e-05 2024-09-21T10:04:50.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.594 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:04:50.594 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.272980+0000", 2024-09-21T10:04:50.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8097999999999999e-05 2024-09-21T10:04:50.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:04:50.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.319040+0000", 2024-09-21T10:04:50.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.046060205999999999 2024-09-21T10:04:50.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.596 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:04:50.596 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.319072+0000", 2024-09-21T10:04:50.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2251e-05 2024-09-21T10:04:50.596 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.596 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:04:50.597 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.597 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:81425 5.7 e114/105 5:e0488b79:::benchmark_data_smithi098_135187_object81424:head v 114'5128, mlcod=114'5124)", 2024-09-21T10:04:50.597 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:29.277076+0000", 2024-09-21T10:04:50.597 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 21.236828295999999, 2024-09-21T10:04:50.598 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.046619946000000002, 2024-09-21T10:04:50.598 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:04:50.598 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:04:50.598 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:04:50.598 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:04:50.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:04:50.599 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.277076+0000", 2024-09-21T10:04:50.599 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:04:50.599 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.277076+0000", 2024-09-21T10:04:50.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.600 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:04:50.600 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.277077+0000", 2024-09-21T10:04:50.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8899999999999998e-07 2024-09-21T10:04:50.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.601 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:04:50.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.277147+0000", 2024-09-21T10:04:50.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0088999999999999e-05 2024-09-21T10:04:50.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.602 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:04:50.602 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316484+0000", 2024-09-21T10:04:50.602 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.039337142999999998 2024-09-21T10:04:50.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.603 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:04:50.603 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316498+0000", 2024-09-21T10:04:50.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3715e-05 2024-09-21T10:04:50.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.603 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:04:50.603 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316570+0000", 2024-09-21T10:04:50.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2628999999999999e-05 2024-09-21T10:04:50.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.604 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:04:50.604 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316589+0000", 2024-09-21T10:04:50.604 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8456000000000001e-05 2024-09-21T10:04:50.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:04:50.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.323585+0000", 2024-09-21T10:04:50.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0069967839999999998 2024-09-21T10:04:50.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.606 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:04:50.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.323696+0000", 2024-09-21T10:04:50.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000110141 2024-09-21T10:04:50.606 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.606 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:04:50.607 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.607 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:81429 5.8 e114/105 5:11aac4f3:::benchmark_data_smithi098_135187_object81428:head v 114'4995, mlcod=114'4991)", 2024-09-21T10:04:50.607 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:29.278922+0000", 2024-09-21T10:04:50.607 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 21.234981554000001, 2024-09-21T10:04:50.607 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.040247472999999999, 2024-09-21T10:04:50.608 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:04:50.608 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:04:50.608 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:04:50.608 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:04:50.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.608 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:04:50.608 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.278922+0000", 2024-09-21T10:04:50.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:04:50.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.278922+0000", 2024-09-21T10:04:50.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.610 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:04:50.610 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.278924+0000", 2024-09-21T10:04:50.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2810000000000001e-06 2024-09-21T10:04:50.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.611 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:04:50.611 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.278948+0000", 2024-09-21T10:04:50.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4207e-05 2024-09-21T10:04:50.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:04:50.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316515+0000", 2024-09-21T10:04:50.612 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037567494999999999 2024-09-21T10:04:50.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:04:50.613 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316522+0000", 2024-09-21T10:04:50.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6290000000000004e-06 2024-09-21T10:04:50.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.613 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:04:50.613 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316557+0000", 2024-09-21T10:04:50.614 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5407999999999997e-05 2024-09-21T10:04:50.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.614 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:04:50.614 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316575+0000", 2024-09-21T10:04:50.614 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7320999999999999e-05 2024-09-21T10:04:50.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.615 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:04:50.615 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.319137+0000", 2024-09-21T10:04:50.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002562188 2024-09-21T10:04:50.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.615 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:04:50.616 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.319170+0000", 2024-09-21T10:04:50.616 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2944000000000002e-05 2024-09-21T10:04:50.616 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.616 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:04:50.616 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.617 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:81434 5.c e114/105 5:31ef8da0:::benchmark_data_smithi098_135187_object81433:head v 114'4926, mlcod=114'4924)", 2024-09-21T10:04:50.617 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:29.281286+0000", 2024-09-21T10:04:50.617 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 21.232617705999999, 2024-09-21T10:04:50.617 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.051817101999999997, 2024-09-21T10:04:50.617 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:04:50.617 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:04:50.617 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:04:50.617 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:04:50.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.618 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:04:50.618 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.281286+0000", 2024-09-21T10:04:50.618 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.618 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:04:50.618 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.281286+0000", 2024-09-21T10:04:50.619 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:04:50.619 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.281287+0000", 2024-09-21T10:04:50.619 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0759999999999999e-06 2024-09-21T10:04:50.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.620 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:04:50.620 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.281370+0000", 2024-09-21T10:04:50.620 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2204000000000001e-05 2024-09-21T10:04:50.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.621 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:04:50.621 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316486+0000", 2024-09-21T10:04:50.621 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035116461000000002 2024-09-21T10:04:50.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.622 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:04:50.622 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316498+0000", 2024-09-21T10:04:50.622 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.216e-05 2024-09-21T10:04:50.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.622 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:04:50.622 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316657+0000", 2024-09-21T10:04:50.623 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015876 2024-09-21T10:04:50.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.623 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:04:50.623 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316676+0000", 2024-09-21T10:04:50.623 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9009999999999999e-05 2024-09-21T10:04:50.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.624 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:04:50.624 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.333051+0000", 2024-09-21T10:04:50.624 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016374667999999998 2024-09-21T10:04:50.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.625 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:04:50.625 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.333103+0000", 2024-09-21T10:04:50.625 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2763000000000003e-05 2024-09-21T10:04:50.625 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.625 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:04:50.625 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.626 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:82527 5.8 e114/105 5:188d4613:::benchmark_data_smithi098_135187_object82526:head v 114'5066, mlcod=114'5062)", 2024-09-21T10:04:50.626 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:30.109584+0000", 2024-09-21T10:04:50.626 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.404320187, 2024-09-21T10:04:50.626 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.022161592000000001, 2024-09-21T10:04:50.627 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:04:50.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:04:50.627 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:04:50.627 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:04:50.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.627 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:04:50.627 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:30.109584+0000", 2024-09-21T10:04:50.627 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.628 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:04:50.628 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:30.109584+0000", 2024-09-21T10:04:50.628 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.629 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:04:50.629 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:30.109585+0000", 2024-09-21T10:04:50.629 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.062e-06 2024-09-21T10:04:50.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.630 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:04:50.630 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:30.110420+0000", 2024-09-21T10:04:50.630 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00083562399999999998 2024-09-21T10:04:50.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.631 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:04:50.631 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:30.110434+0000", 2024-09-21T10:04:50.631 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3149999999999999e-05 2024-09-21T10:04:50.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.631 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:04:50.632 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:30.110441+0000", 2024-09-21T10:04:50.632 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.917e-06 2024-09-21T10:04:50.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.632 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:04:50.632 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:30.110585+0000", 2024-09-21T10:04:50.633 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014419899999999999 2024-09-21T10:04:50.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.633 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:04:50.633 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:30.110625+0000", 2024-09-21T10:04:50.633 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0559000000000002e-05 2024-09-21T10:04:50.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.634 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:04:50.634 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:30.131589+0000", 2024-09-21T10:04:50.634 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020963984000000001 2024-09-21T10:04:50.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.634 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:04:50.635 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:30.131745+0000", 2024-09-21T10:04:50.635 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015609700000000001 2024-09-21T10:04:50.635 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.635 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:04:50.635 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.635 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:84547 5.0 e114/105 5:07de3dcf:::benchmark_data_smithi098_135187_object84546:head v 114'5244, mlcod=114'5242)", 2024-09-21T10:04:50.636 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:31.670272+0000", 2024-09-21T10:04:50.636 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.843631832, 2024-09-21T10:04:50.636 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.025123901000000001, 2024-09-21T10:04:50.636 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:04:50.636 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:04:50.636 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:04:50.637 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:04:50.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.637 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:04:50.637 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:31.670272+0000", 2024-09-21T10:04:50.637 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.637 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:04:50.637 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:31.670272+0000", 2024-09-21T10:04:50.638 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.638 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:04:50.638 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:31.670274+0000", 2024-09-21T10:04:50.638 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.725e-06 2024-09-21T10:04:50.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.639 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:04:50.639 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:31.670305+0000", 2024-09-21T10:04:50.639 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1477999999999999e-05 2024-09-21T10:04:50.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.639 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:04:50.640 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:31.670326+0000", 2024-09-21T10:04:50.640 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0809e-05 2024-09-21T10:04:50.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.640 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:04:50.640 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:31.670336+0000", 2024-09-21T10:04:50.641 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8379999999999997e-06 2024-09-21T10:04:50.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.641 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:04:50.641 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:31.670436+0000", 2024-09-21T10:04:50.641 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010046 2024-09-21T10:04:50.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.642 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:04:50.642 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:31.670456+0000", 2024-09-21T10:04:50.642 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9459000000000001e-05 2024-09-21T10:04:50.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.643 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:04:50.643 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:31.695181+0000", 2024-09-21T10:04:50.643 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.02472529 2024-09-21T10:04:50.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.643 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:04:50.643 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:31.695396+0000", 2024-09-21T10:04:50.643 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000214842 2024-09-21T10:04:50.644 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.644 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:04:50.644 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.644 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:84552 5.7 e114/105 5:e6ead7be:::benchmark_data_smithi098_135187_object84551:head v 114'5314, mlcod=114'5312)", 2024-09-21T10:04:50.644 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:31.674047+0000", 2024-09-21T10:04:50.645 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.839856566999998, 2024-09-21T10:04:50.645 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.021203665999999999, 2024-09-21T10:04:50.645 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:04:50.645 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:04:50.645 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:04:50.645 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:04:50.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.646 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:04:50.646 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:31.674047+0000", 2024-09-21T10:04:50.646 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.646 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:04:50.646 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:31.674047+0000", 2024-09-21T10:04:50.647 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.647 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:04:50.647 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:31.674049+0000", 2024-09-21T10:04:50.647 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7829999999999999e-06 2024-09-21T10:04:50.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.648 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:04:50.648 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:31.674132+0000", 2024-09-21T10:04:50.648 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3159999999999997e-05 2024-09-21T10:04:50.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.648 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:04:50.649 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:31.674152+0000", 2024-09-21T10:04:50.649 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9437e-05 2024-09-21T10:04:50.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.649 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:04:50.649 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:31.674162+0000", 2024-09-21T10:04:50.650 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8379999999999997e-06 2024-09-21T10:04:50.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.650 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.650 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:04:50.650 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:31.675046+0000", 2024-09-21T10:04:50.650 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088407600000000002 2024-09-21T10:04:50.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.651 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:04:50.651 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:31.675090+0000", 2024-09-21T10:04:50.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3962999999999999e-05 2024-09-21T10:04:50.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.651 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:04:50.652 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:31.695197+0000", 2024-09-21T10:04:50.652 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020107531000000001 2024-09-21T10:04:50.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.652 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.652 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:04:50.652 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:31.695251+0000", 2024-09-21T10:04:50.652 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3878000000000002e-05 2024-09-21T10:04:50.653 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.653 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:04:50.653 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.654 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:85182 5.7 e114/105 5:ecd25ba4:::benchmark_data_smithi098_135187_object85181:head v 114'5346, mlcod=114'5343)", 2024-09-21T10:04:50.654 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:32.182953+0000", 2024-09-21T10:04:50.654 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.330950767000001, 2024-09-21T10:04:50.654 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.022001314000000001, 2024-09-21T10:04:50.654 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:04:50.654 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:04:50.654 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:04:50.655 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:04:50.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.655 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:04:50.655 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:32.182953+0000", 2024-09-21T10:04:50.655 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.656 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:04:50.656 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:32.182953+0000", 2024-09-21T10:04:50.656 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.656 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:04:50.656 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:32.182954+0000", 2024-09-21T10:04:50.656 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.009e-06 2024-09-21T10:04:50.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.657 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:04:50.657 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:32.182986+0000", 2024-09-21T10:04:50.657 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1322999999999998e-05 2024-09-21T10:04:50.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.657 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.657 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:04:50.657 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:32.182997+0000", 2024-09-21T10:04:50.657 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1216999999999999e-05 2024-09-21T10:04:50.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.658 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:04:50.658 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:32.183004+0000", 2024-09-21T10:04:50.658 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8499999999999996e-06 2024-09-21T10:04:50.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.659 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:04:50.659 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:32.183097+0000", 2024-09-21T10:04:50.659 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3580999999999997e-05 2024-09-21T10:04:50.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.659 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:04:50.660 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:32.183126+0000", 2024-09-21T10:04:50.660 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8464000000000001e-05 2024-09-21T10:04:50.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.660 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:04:50.660 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:32.204783+0000", 2024-09-21T10:04:50.660 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.021657592999999999 2024-09-21T10:04:50.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.661 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:04:50.661 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:32.204955+0000", 2024-09-21T10:04:50.661 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017127700000000001 2024-09-21T10:04:50.662 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.662 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:04:50.662 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.663 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:104121 5.0 e116/105 5:09dbf810:::benchmark_data_smithi098_135187_object104120:head v 116'6426, mlcod=116'6423)", 2024-09-21T10:04:50.663 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:48.324650+0000", 2024-09-21T10:04:50.663 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 2.1892537829999998, 2024-09-21T10:04:50.663 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.021821268000000001, 2024-09-21T10:04:50.663 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:04:50.663 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:04:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:04:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:04:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:04:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:48.324650+0000", 2024-09-21T10:04:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:04:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:48.324650+0000", 2024-09-21T10:04:50.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:04:50.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:48.324651+0000", 2024-09-21T10:04:50.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1850000000000001e-06 2024-09-21T10:04:50.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.666 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:04:50.666 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:48.324681+0000", 2024-09-21T10:04:50.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0074999999999999e-05 2024-09-21T10:04:50.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:04:50.667 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:48.324702+0000", 2024-09-21T10:04:50.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0590000000000001e-05 2024-09-21T10:04:50.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:04:50.668 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:48.324710+0000", 2024-09-21T10:04:50.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4749999999999996e-06 2024-09-21T10:04:50.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.668 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:04:50.668 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:48.327467+0000", 2024-09-21T10:04:50.669 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027573599999999999 2024-09-21T10:04:50.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.669 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:04:50.669 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:48.327522+0000", 2024-09-21T10:04:50.669 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5365000000000003e-05 2024-09-21T10:04:50.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:04:50.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:48.346428+0000", 2024-09-21T10:04:50.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018905317000000001 2024-09-21T10:04:50.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:04:50.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:48.346471+0000", 2024-09-21T10:04:50.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3900999999999999e-05 2024-09-21T10:04:50.671 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.671 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:04:50.671 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.671 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:105667 5.0 e116/105 5:06897a75:::benchmark_data_smithi098_135187_object105666:head v 116'6520, mlcod=116'6518)", 2024-09-21T10:04:50.672 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:49.693321+0000", 2024-09-21T10:04:50.672 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.82058262100000001, 2024-09-21T10:04:50.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.024503616999999998, 2024-09-21T10:04:50.672 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:04:50.672 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:04:50.672 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:04:50.672 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:04:50.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.673 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:04:50.673 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:49.693321+0000", 2024-09-21T10:04:50.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.674 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:04:50.674 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:49.693321+0000", 2024-09-21T10:04:50.674 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:04:50.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.674 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:04:50.675 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:49.693323+0000", 2024-09-21T10:04:50.675 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.269e-06 2024-09-21T10:04:50.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.675 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:04:50.675 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:49.693348+0000", 2024-09-21T10:04:50.675 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5670999999999999e-05 2024-09-21T10:04:50.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.676 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:04:50.676 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:49.699004+0000", 2024-09-21T10:04:50.676 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056553970000000004 2024-09-21T10:04:50.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.676 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:04:50.676 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:49.699020+0000", 2024-09-21T10:04:50.676 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5889999999999999e-05 2024-09-21T10:04:50.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:04:50.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:49.699356+0000", 2024-09-21T10:04:50.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00033636700000000001 2024-09-21T10:04:50.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:04:50.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:49.699407+0000", 2024-09-21T10:04:50.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1257000000000001e-05 2024-09-21T10:04:50.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:04:50.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:49.717785+0000", 2024-09-21T10:04:50.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018377655 2024-09-21T10:04:50.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:04:50.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:04:50.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:04:50.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:49.717825+0000", 2024-09-21T10:04:50.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0111000000000003e-05 2024-09-21T10:04:50.680 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.680 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:04:50.680 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.680 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:04:50.680 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:04:50.680 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:04:50.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:50.682 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:04:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:04:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:50.682 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 61 16 106700 106684 109.284 65.4375 0.00960489 0.0091458 2024-09-21T10:04:50.716 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:50.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:50.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:50.789 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T10:04:50.790 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:04:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:04:50] "GET /metrics HTTP/1.1" 200 36454 "" "Prometheus/2.51.0" 2024-09-21T10:04:50.817 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:50.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:50.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:50.947 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:04:50.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:50.989 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:50.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:50.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:51.091 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:51.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:50 smithi098 ceph-mon[97580]: pgmap v283: 89 pgs: 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T10:04:51.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:50 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2011080854' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:04:51.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:50 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:04:51.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:50 smithi098 ceph-mon[100914]: pgmap v283: 89 pgs: 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T10:04:51.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:50 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2011080854' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:04:51.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:50 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:04:51.125 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:51.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:51.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:51.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:50 smithi196 ceph-mon[84784]: pgmap v283: 89 pgs: 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T10:04:51.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:50 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2011080854' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:04:51.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:50 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:04:51.227 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:51.255 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:51.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:51.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:51.357 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:51.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:51.393 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:51.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:51.494 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:51.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:51.529 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:51.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:51.630 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:51.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:51.670 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:51.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:51.673 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 62 16 107765 107749 108.595 66.5625 0.0121566 0.00920287 2024-09-21T10:04:51.770 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:51.815 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:51.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:51.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:51.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:51.946 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:51.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:51.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:52.048 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:52.083 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:52.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:52.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:52.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:51 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:04:52.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:51 smithi098 ceph-mon[97580]: osdmap e117: 8 total, 8 up, 8 in 2024-09-21T10:04:52.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:51 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:04:52.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:51 smithi098 ceph-mon[100914]: osdmap e117: 8 total, 8 up, 8 in 2024-09-21T10:04:52.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:51 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:04:52.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:51 smithi196 ceph-mon[84784]: osdmap e117: 8 total, 8 up, 8 in 2024-09-21T10:04:52.185 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:52.218 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:52.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:52.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:52.320 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:52.354 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:52.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:52.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:52.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T10:04:52.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:52.498 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:52.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:52.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:52.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:52.633 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:52.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:52.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:52.674 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 63 16 108852 108836 107.948 67.9375 0.0150762 0.00925934 2024-09-21T10:04:52.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:52.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:52.768 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:52.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:52.869 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:52.902 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:52.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:52.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:52.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:04:53.004 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:53.036 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:53.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:53.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:52 smithi098 ceph-mon[97580]: pgmap v285: 89 pgs: 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 58 MiB/s wr, 925 op/s 2024-09-21T10:04:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:04:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:53.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:52 smithi098 ceph-mon[100914]: pgmap v285: 89 pgs: 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 58 MiB/s wr, 925 op/s 2024-09-21T10:04:53.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:04:53.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:53.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:52 smithi196 ceph-mon[84784]: pgmap v285: 89 pgs: 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 58 MiB/s wr, 925 op/s 2024-09-21T10:04:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:04:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:53.174 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:53.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:53.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:53.276 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:53.312 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:53.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:53.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:53.414 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:53.452 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:53.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:53.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:53.530 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:53.554 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:53.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:53.612 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:53.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:53.675 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 64 16 109958 109942 107.342 69.125 0.0110718 0.0093114 2024-09-21T10:04:53.713 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:53.751 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:53.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:53.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:53.852 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:53.886 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:53.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:53.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:53.988 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:54.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:54.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:54.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:54.191 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:54.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:54.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:54.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:54.324 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:54.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:54.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:54.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:54.465 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:54.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:54.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:54.567 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:54.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:54.601 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:54.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:54.674 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 65 16 111041 111025 106.731 67.6875 0.016689 0.009364 2024-09-21T10:04:54.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:54.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:54.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:54.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:54.869 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:54.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:54.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:54.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:55.010 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:55.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:55.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:55.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:54 smithi098 ceph-mon[97580]: pgmap v286: 89 pgs: 89 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-21T10:04:55.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:54 smithi098 ceph-mon[100914]: pgmap v286: 89 pgs: 89 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-21T10:04:55.112 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:55.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:54 smithi196 ceph-mon[84784]: pgmap v286: 89 pgs: 89 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-21T10:04:55.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:55.148 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:55.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:55.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:55.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:55.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:55.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:55.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:55.445 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:55.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:55.546 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:55.587 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:55.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:55.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:55.674 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 66 16 112135 112119 106.15 68.375 0.0162979 0.00941605 2024-09-21T10:04:55.688 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:55.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:55.718 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:55.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:55.819 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:55.851 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:55.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:55.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:55.953 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:55.991 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:55.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:55.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:56.093 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:56.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:56.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:56.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:56.262 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:56.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:56.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:56.363 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:56.399 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:56.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:56.501 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:56.542 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:56.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:56.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:56.644 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:56.675 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 67 16 113211 113195 105.57 67.25 0.00624043 0.00946729 2024-09-21T10:04:56.680 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:56.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:56.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:56.737 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:56.739 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:56.782 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:56.818 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:56.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:56.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:56.834 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:04:56.920 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:56.952 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:56.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:56.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:57.037 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:56 smithi098 ceph-mon[97580]: pgmap v287: 89 pgs: 89 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-21T10:04:57.037 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:56 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1058857668' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:04:57.037 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:56 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:04:57.037 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:56 smithi098 ceph-mon[100914]: pgmap v287: 89 pgs: 89 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-21T10:04:57.038 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:56 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1058857668' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:04:57.038 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:56 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:04:57.054 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:57.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:57.095 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:57.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:57.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:56 smithi196 ceph-mon[84784]: pgmap v287: 89 pgs: 89 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-21T10:04:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:56 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1058857668' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:04:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:56 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:04:57.197 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:57.228 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:57.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:57.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:57.330 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:57.367 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:57.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:57.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:57.469 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:57.500 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:57.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:57.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:57.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:57.632 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:57.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:57.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:57.675 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 68 16 114245 114229 104.967 64.625 0.0188136 0.00952216 2024-09-21T10:04:57.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:57.771 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:57.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:57.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:57.872 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:57.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:57.917 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:57.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:58.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:58.047 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:58.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:58.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:58.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:57 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:04:58.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:57 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:04:58.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:57 smithi098 ceph-mon[97580]: osdmap e118: 8 total, 8 up, 8 in 2024-09-21T10:04:58.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:57 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:04:58.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:57 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:04:58.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:57 smithi098 ceph-mon[100914]: osdmap e118: 8 total, 8 up, 8 in 2024-09-21T10:04:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:57 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:04:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:57 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:04:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:57 smithi196 ceph-mon[84784]: osdmap e118: 8 total, 8 up, 8 in 2024-09-21T10:04:58.148 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:58.193 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:58.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:58.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:58.295 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:58.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:58.336 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:58.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:58.437 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:58.466 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:58.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:58.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:58.568 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:58.597 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:58.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:58.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:58.675 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 69 15 115313 115298 104.414 66.8125 0.0066798 0.00957198 2024-09-21T10:04:58.698 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:58.731 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:58.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:58.833 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:58.865 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:58.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:58.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:58.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:59.025 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:59.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:59.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:59.095 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:58 smithi098 ceph-mon[97580]: pgmap v289: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 50 MiB/s wr, 807 op/s 2024-09-21T10:04:59.095 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:58 smithi098 ceph-mon[100914]: pgmap v289: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 50 MiB/s wr, 807 op/s 2024-09-21T10:04:59.128 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:59.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:58 smithi196 ceph-mon[84784]: pgmap v289: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 50 MiB/s wr, 807 op/s 2024-09-21T10:04:59.169 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:59.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:59.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:59.271 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:59.306 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:59.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:59.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:59.408 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:59.437 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:59.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:59.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:59.539 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:59.573 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:59.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:59.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:59.674 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:59.675 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 70 16 116377 116361 103.872 66.4375 0.024403 0.00962275 2024-09-21T10:04:59.698 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:04:59.698 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:04:59.698 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:04:59.698 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:04:59.698 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:04:59.698 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:04:59.698 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:04:59.698 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:04:59.698 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:04:59.698 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:04:59.698 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:04:59.698 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:04:59.699 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:04:59.699 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:04:59.699 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:04:59.699 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:04:59.709 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:59.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:59.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:59.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:59.844 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:04:59.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:59.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:59.945 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:59.981 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:04:59.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:59.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:00.083 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:00.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:04:59 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/779038080' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:00.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:04:59 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/779038080' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:00.115 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:00.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:00.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:00.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:04:59 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/779038080' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:00.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:00.256 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:00.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:00.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:00.357 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:00.385 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:00.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:00.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:00.486 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:00.520 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:00.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:00.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:00.622 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:00.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:05:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:05:00.215Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:00.657 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:00.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:00.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:00.675 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 71 15 117432 117417 103.338 66 0.00744065 0.0096721 2024-09-21T10:05:00.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:00.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:00.797 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:00.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:00.898 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:00.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:00.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:01.025 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:01.060 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:01.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:01.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:01.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:05:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:05:00] "GET /metrics HTTP/1.1" 200 36457 "" "Prometheus/2.51.0" 2024-09-21T10:05:01.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:00 smithi098 ceph-mon[97580]: pgmap v290: 89 pgs: 89 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-21T10:05:01.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:00 smithi098 ceph-mon[100914]: pgmap v290: 89 pgs: 89 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-21T10:05:01.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:00 smithi196 ceph-mon[84784]: pgmap v290: 89 pgs: 89 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-21T10:05:01.162 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:01.198 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:01.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:01.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:01.300 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:01.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:01.336 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:01.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:01.437 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:01.471 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:01.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:01.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:01.573 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:01.610 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:01.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:01.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:01.675 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 72 16 118416 118400 102.757 61.4375 0.0110225 0.00972683 2024-09-21T10:05:01.711 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:01.748 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:01.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:01.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:01.850 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:01.888 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:01.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:01.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:01.989 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:02.024 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:02.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:02.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:02.125 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:02.154 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:02.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:02.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:02.256 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:02.298 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:02.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:02.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:02.400 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:02.431 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:02.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:02.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:02.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:02.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:02.594 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:02.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:02.675 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 73 16 119404 119388 102.195 61.75 0.00809743 0.00978049 2024-09-21T10:05:02.694 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:02.721 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:02.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:02.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:02.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:02.858 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:02.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:02.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:02.960 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:02.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:02 smithi098 ceph-mon[97580]: pgmap v291: 89 pgs: 89 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-21T10:05:02.967 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:02 smithi098 ceph-mon[100914]: pgmap v291: 89 pgs: 89 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-21T10:05:02.995 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:02.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:02.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:03.096 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:03.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:02 smithi196 ceph-mon[84784]: pgmap v291: 89 pgs: 89 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-21T10:05:03.139 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:03.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:03.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:03.241 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:03.272 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:03.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:03.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:03.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:03.405 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:03.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:03.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:03.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:03.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:03.541 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:03.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:03.641 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:03.675 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 74 16 120397 120381 101.652 62.0625 0.00493976 0.00983239 2024-09-21T10:05:03.677 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:03.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:03.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:03.779 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:03.818 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:03.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:03.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:03.920 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:03.960 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:03.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:03.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:04.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:04.099 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:04.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:04.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:04.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:04.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:04.239 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:04.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:04.340 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:04.379 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:04.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:04.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:04.481 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:04.528 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:04.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:04.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:04.590 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:05:04.629 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:04.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:04.659 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:04.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:04.676 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 75 16 121373 121357 101.11 61 0.0102411 0.00988492 2024-09-21T10:05:04.761 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:04.792 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:04.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:04.834 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:05:04.835 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:05:04.854 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T10:05:04.894 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:04.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:04.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:05.043 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:05.084 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:05.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:05.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:05.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:04 smithi196 ceph-mon[84784]: pgmap v292: 89 pgs: 89 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:05:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:04 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2098308791' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:05:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:04 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:05:05.186 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:05.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:05.214 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:05.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:05.315 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:05.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:04 smithi098 ceph-mon[97580]: pgmap v292: 89 pgs: 89 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:05:05.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:04 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2098308791' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:05:05.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:04 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:05:05.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:04 smithi098 ceph-mon[100914]: pgmap v292: 89 pgs: 89 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:05:05.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:04 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2098308791' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:05:05.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:04 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:05:05.349 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:05.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:05.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:05.451 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:05.491 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:05.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:05.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:05.593 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:05.627 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:05.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:05.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:05.676 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 76 16 122346 122330 100.579 60.8125 0.00494177 0.00993555 2024-09-21T10:05:05.729 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:05.763 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:05.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:05.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:05.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:05:05.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:05.897 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:05.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:05.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:05.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:06.035 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:06.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:06.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:06.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:05 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:05:06.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:05 smithi098 ceph-mon[97580]: osdmap e119: 8 total, 8 up, 8 in 2024-09-21T10:05:06.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:05 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:05:06.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:05 smithi098 ceph-mon[100914]: osdmap e119: 8 total, 8 up, 8 in 2024-09-21T10:05:06.137 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:06.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:05 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:05:06.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:05 smithi196 ceph-mon[84784]: osdmap e119: 8 total, 8 up, 8 in 2024-09-21T10:05:06.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:06.177 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:06.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:06.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:06.317 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:06.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:06.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:06.419 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:06.459 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:06.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:06.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:06.561 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:06.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:06.597 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:06.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:06.676 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 77 16 123338 123322 100.078 62 0.0429078 0.00998723 2024-09-21T10:05:06.697 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:06.725 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:06.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:06.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:06.827 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:06.861 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:06.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:06.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:06.962 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:06.983 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:06.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:06.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:07.085 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:07.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:06 smithi196 ceph-mon[84784]: pgmap v294: 89 pgs: 89 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-21T10:05:07.128 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:07.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:07.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:07.229 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:07.258 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:07.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:07.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:07.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:06 smithi098 ceph-mon[97580]: pgmap v294: 89 pgs: 89 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-21T10:05:07.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:06 smithi098 ceph-mon[100914]: pgmap v294: 89 pgs: 89 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-21T10:05:07.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:07.392 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:07.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:07.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:07.494 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:07.520 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:07.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:07.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:07.621 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:07.657 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:07.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:07.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:07.676 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 78 16 124151 124135 99.4468 50.8125 0.0445494 0.0100507 2024-09-21T10:05:07.759 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:07.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:07.797 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:07.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:07.898 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:07.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:07.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:08.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:08.064 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:08.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:08.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:08.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:08.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:08.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:08.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:08.196 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:08.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:08.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:08.298 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:08.332 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:08.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:08.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:08.434 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:08.469 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:05:08.479 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:08.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:08.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:08.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:08.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:08.627 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:08.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:08.676 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 79 16 125047 125031 98.8968 56 0.02072 0.0101069 2024-09-21T10:05:08.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:08.755 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:08.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:08.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:08.856 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:08.886 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:08.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:08.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:08.987 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:09.026 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:09.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:09.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:09.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:08 smithi098 ceph-mon[97580]: pgmap v295: 89 pgs: 89 active+clean; 7.4 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T10:05:09.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:05:09.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:05:09.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:09.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:05:09.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:05:09.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:08 smithi098 ceph-mon[100914]: pgmap v295: 89 pgs: 89 active+clean; 7.4 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T10:05:09.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:05:09.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:05:09.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:09.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:05:09.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:05:09.128 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:09.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:08 smithi196 ceph-mon[84784]: pgmap v295: 89 pgs: 89 active+clean; 7.4 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T10:05:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:05:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:05:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:05:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:05:09.162 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:09.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:09.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:09.187 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T10:05:09.252 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:09.264 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:09.296 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:09.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:09.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:09.321 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:09.398 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:09.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:09.439 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:09.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:09.541 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:09.579 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:09.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:09.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:09.676 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:05:09.675536+0000 min lat: 0.00106087 max lat: 0.125969 avg lat: 0.0101567 2024-09-21T10:05:09.677 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:05:09.677 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 80 16 126007 125991 98.4106 60 0.0219192 0.0101567 2024-09-21T10:05:09.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:09.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:09.713 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:09.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:09.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:09.847 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:09.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:09.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:09.949 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:09.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:09.988 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:09.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:10.089 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:10.121 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:10.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:10.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:10.223 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:10.259 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:10.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:10.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:10.360 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:10.397 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:10.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:10.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:10.499 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:10.536 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:10.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:10.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:10.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:10.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:05:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:05:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:10.676 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 81 16 127016 127000 97.9742 63.0625 0.00762564 0.0102019 2024-09-21T10:05:10.680 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:10.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:10.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:10.753 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:10.782 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:10.841 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:10.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:10.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:10.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:05:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:05:10] "GET /metrics HTTP/1.1" 200 36452 "" "Prometheus/2.51.0" 2024-09-21T10:05:10.943 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:10.979 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:10.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:10.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:11.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:11.116 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:11.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:11.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:11.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:10 smithi196 ceph-mon[84784]: pgmap v296: 89 pgs: 89 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 62 MiB/s wr, 989 op/s 2024-09-21T10:05:11.218 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:11.232 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:10 smithi098 ceph-mon[97580]: pgmap v296: 89 pgs: 89 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 62 MiB/s wr, 989 op/s 2024-09-21T10:05:11.233 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:10 smithi098 ceph-mon[100914]: pgmap v296: 89 pgs: 89 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 62 MiB/s wr, 989 op/s 2024-09-21T10:05:11.249 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:11.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:11.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:11.350 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:11.400 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:11.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:11.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:11.502 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:11.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:11.538 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:11.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:11.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:11.676 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 82 16 128004 127988 97.5324 61.75 0.0129502 0.0102483 2024-09-21T10:05:11.681 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:11.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:11.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:11.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:11.821 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:11.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:11.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:11.923 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:11.964 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:11.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:11.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:12.066 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:12.102 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:12.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:12.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:12.158 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:12.205 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:12.240 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:12.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:12.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:12.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:12.379 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:12.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:12.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:12.481 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:12.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:12.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:12.617 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:12.657 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:12.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:12.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:12.676 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 83 16 128993 128977 97.1021 61.8125 0.00656701 0.0102939 2024-09-21T10:05:12.759 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:12.806 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:12.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:12.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:12.908 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:12.958 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:12.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:12.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:13.060 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:13.100 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:13.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:13.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:13.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:12 smithi196 ceph-mon[84784]: pgmap v297: 89 pgs: 89 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 62 MiB/s wr, 989 op/s 2024-09-21T10:05:13.141 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:12 smithi098 ceph-mon[97580]: pgmap v297: 89 pgs: 89 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 62 MiB/s wr, 989 op/s 2024-09-21T10:05:13.141 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:12 smithi098 ceph-mon[100914]: pgmap v297: 89 pgs: 89 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 62 MiB/s wr, 989 op/s 2024-09-21T10:05:13.203 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:13.206 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:05:13.206 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:05:13.233 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:05:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:05:13.245 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:13.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:13.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:13.267 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:05:13.267 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:05:13.295 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:05:13.296 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:05:13.331 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:05:13.331 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:05:13.346 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:13.365 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:05:13.366 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:05:13.382 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:13.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:13.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:13.400 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:05:13.400 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:05:13.431 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:05:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:05:13.484 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:13.525 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:13.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:13.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:13.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:13.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:13.663 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:13.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:13.678 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 84 16 129947 129931 96.6559 59.625 0.0166015 0.0103409 2024-09-21T10:05:13.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:13.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:13.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:13.895 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:13.931 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:13.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:13.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:14.033 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:14.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:14.069 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:14.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:14.154 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:13 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1428417795' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:14.155 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:13 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1428417795' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:14.170 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:14.199 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:14.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:14.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:14.300 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:14.335 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:14.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:14.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:14.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:13 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1428417795' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:14.437 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:14.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:14.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:14.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:14.596 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:05:14.596 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:05:14.596 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-21T10:05:14.596 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-21T10:05:14.596 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:05:14.602 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:14.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:14.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:14.677 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 85 16 130907 130891 96.2246 60 0.0184173 0.0103863 2024-09-21T10:05:14.703 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:14.728 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:14.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:14.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:14.830 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:14.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:14.870 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:14.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:14.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:15.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:15.015 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:15.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:15.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:14 smithi098 ceph-mon[97580]: pgmap v298: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 61 MiB/s wr, 968 op/s 2024-09-21T10:05:15.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:14 smithi098 ceph-mon[100914]: pgmap v298: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 61 MiB/s wr, 968 op/s 2024-09-21T10:05:15.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:15.145 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:15.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:15.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:15.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:15.284 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:15.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:15.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:15.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:15.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:14 smithi196 ceph-mon[84784]: pgmap v298: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 61 MiB/s wr, 968 op/s 2024-09-21T10:05:15.416 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:15.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:15.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:15.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:15.548 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:15.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:15.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:15.650 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:15.677 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 86 16 131861 131845 95.799 59.625 0.0151675 0.0104334 2024-09-21T10:05:15.681 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:15.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:15.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:15.782 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:15.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:15.821 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:15.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:15.913 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:05:15.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:15.964 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:15.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:15.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:16.066 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:16.099 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:16.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:16.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:16.174 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:15 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1970547743' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:05:16.175 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:15 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:05:16.175 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:15 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1970547743' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:05:16.175 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:15 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:05:16.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:16.211 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:05:16.232 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:16.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:16.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:16.334 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:16.368 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:16.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:16.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:16.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:15 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1970547743' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:05:16.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:15 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:05:16.470 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:16.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:16.512 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:16.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:16.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:16.651 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:16.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:16.677 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 87 16 132828 132812 95.3926 60.4375 0.0252165 0.010478 2024-09-21T10:05:16.752 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:16.787 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:16.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:16.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:16.889 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:16.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:16.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:17.034 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:17.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:17.073 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:17.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:17.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:17.206 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:17.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:17.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:17.305 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:16 smithi098 ceph-mon[97580]: pgmap v299: 89 pgs: 89 active+clean; 7.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 59 MiB/s wr, 938 op/s 2024-09-21T10:05:17.305 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:16 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:05:17.305 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:16 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:05:17.305 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:16 smithi098 ceph-mon[97580]: osdmap e120: 8 total, 8 up, 8 in 2024-09-21T10:05:17.306 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:16 smithi098 ceph-mon[100914]: pgmap v299: 89 pgs: 89 active+clean; 7.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 59 MiB/s wr, 938 op/s 2024-09-21T10:05:17.306 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:16 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:05:17.306 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:16 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:05:17.306 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:16 smithi098 ceph-mon[100914]: osdmap e120: 8 total, 8 up, 8 in 2024-09-21T10:05:17.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:17.309 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:05:17.339 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:17.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:17.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:17.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:16 smithi196 ceph-mon[84784]: pgmap v299: 89 pgs: 89 active+clean; 7.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 59 MiB/s wr, 938 op/s 2024-09-21T10:05:17.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:16 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:05:17.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:16 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:05:17.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:16 smithi196 ceph-mon[84784]: osdmap e120: 8 total, 8 up, 8 in 2024-09-21T10:05:17.441 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:17.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:17.510 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:17.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:17.611 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:17.642 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:17.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:17.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:17.677 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 88 16 133778 133762 94.9831 59.375 0.00772205 0.0105234 2024-09-21T10:05:17.744 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:17.788 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:17.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:17.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:17.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:17.921 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:17.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:17.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:18.022 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:18.058 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:18.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:18.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:18.160 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:18.197 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:18.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:18.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:18.299 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:18.325 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:18.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:18.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:18.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:18.470 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:18.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:18.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:18.539 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:18.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:18.624 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:18.632 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:18.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:18.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:18.678 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 89 16 134705 134689 94.566 57.9375 0.0210776 0.01057 2024-09-21T10:05:18.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:18.785 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:18.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:18.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:18.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:18.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:18.925 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:18.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:19.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:19.060 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:19.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:19.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:19.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:18 smithi098 ceph-mon[97580]: pgmap v301: 89 pgs: 89 active+clean; 8.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 60 MiB/s wr, 955 op/s 2024-09-21T10:05:19.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:18 smithi098 ceph-mon[100914]: pgmap v301: 89 pgs: 89 active+clean; 8.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 60 MiB/s wr, 955 op/s 2024-09-21T10:05:19.162 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:19.195 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:19.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:19.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:19.297 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:19.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:19.343 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:19.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:19.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:18 smithi196 ceph-mon[84784]: pgmap v301: 89 pgs: 89 active+clean; 8.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 60 MiB/s wr, 955 op/s 2024-09-21T10:05:19.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:19.481 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:19.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:19.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:19.583 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:19.620 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:19.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:19.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:19.678 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Total time run: 90.008 2024-09-21T10:05:19.678 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Total writes made: 135635 2024-09-21T10:05:19.679 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Write size: 65536 2024-09-21T10:05:19.679 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Object size: 65536 2024-09-21T10:05:19.679 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Bandwidth (MB/sec): 94.1826 2024-09-21T10:05:19.679 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Stddev Bandwidth: 43.0281 2024-09-21T10:05:19.679 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Max bandwidth (MB/sec): 248.312 2024-09-21T10:05:19.679 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Min bandwidth (MB/sec): 50.8125 2024-09-21T10:05:19.679 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Average IOPS: 1506 2024-09-21T10:05:19.679 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Stddev IOPS: 688.449 2024-09-21T10:05:19.679 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Max IOPS: 3973 2024-09-21T10:05:19.680 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Min IOPS: 813 2024-09-21T10:05:19.680 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Average Latency(s): 0.0106137 2024-09-21T10:05:19.680 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Stddev Latency(s): 0.00735654 2024-09-21T10:05:19.680 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Max latency(s): 0.125969 2024-09-21T10:05:19.680 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Min latency(s): 0.00106087 2024-09-21T10:05:19.685 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:05:19.723 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:19.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:19.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:19.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:19.859 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:19.896 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:19.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:19.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:19.923 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T10:05:19.998 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:20.025 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:20.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:20.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:20.127 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:20.156 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:20.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:20.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:20.208 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:05:20.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:19 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/918227920' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:05:20.248 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:19 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/918227920' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:05:20.250 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:19 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/918227920' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:05:20.258 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:20.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:20.292 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:20.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:20.393 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:20.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:20.431 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:20.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:20.532 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:20.562 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:20.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:20.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:20.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:05:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:05:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:20.664 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:20.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:20.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:20.818 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:20.854 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:20.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:20.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:20.912 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:05:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:05:20] "GET /metrics HTTP/1.1" 200 36452 "" "Prometheus/2.51.0" 2024-09-21T10:05:20.956 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:20.985 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:20.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:20.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:21.087 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:21.115 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:21.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:21.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:21.217 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:21.253 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:21.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:21.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:20 smithi098 ceph-mon[97580]: pgmap v302: 89 pgs: 89 active+clean; 8.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 51 MiB/s wr, 812 op/s 2024-09-21T10:05:21.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:20 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/918227920' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:05:21.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:20 smithi098 ceph-mon[97580]: osdmap e121: 8 total, 8 up, 8 in 2024-09-21T10:05:21.291 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:20 smithi098 ceph-mon[100914]: pgmap v302: 89 pgs: 89 active+clean; 8.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 51 MiB/s wr, 812 op/s 2024-09-21T10:05:21.291 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:20 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/918227920' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:05:21.291 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:20 smithi098 ceph-mon[100914]: osdmap e121: 8 total, 8 up, 8 in 2024-09-21T10:05:21.356 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:21.388 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:20 smithi196 ceph-mon[84784]: pgmap v302: 89 pgs: 89 active+clean; 8.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 51 MiB/s wr, 812 op/s 2024-09-21T10:05:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:20 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/918227920' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:05:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:20 smithi196 ceph-mon[84784]: osdmap e121: 8 total, 8 up, 8 in 2024-09-21T10:05:21.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:21.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:21.490 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:21.531 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:21.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:21.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:21.633 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:21.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:21.675 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:21.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:21.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:21.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T10:05:21.819 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:21.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:21.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:21.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:21.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:21.963 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:21.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:22.047 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:05:22.047 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-21T10:05:22.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-21T10:05:22.047 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:05:22.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.047 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:35209 5.b 5:d050c117:::benchmark_data_smithi098_135187_object35208:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:05:22.047 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.630227+0000", 2024-09-21T10:05:22.047 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 80.412016082999997, 2024-09-21T10:05:22.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.12529087799999999, 2024-09-21T10:05:22.047 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:05:22.047 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:05:22.048 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:22.048 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:05:22.048 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:05:22.048 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:05:22.048 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35209 2024-09-21T10:05:22.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.048 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:05:22.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:05:22.048 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.630227+0000", 2024-09-21T10:05:22.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:05:22.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.630227+0000", 2024-09-21T10:05:22.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:05:22.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.630227+0000", 2024-09-21T10:05:22.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5599999999999995e-07 2024-09-21T10:05:22.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:05:22.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.630247+0000", 2024-09-21T10:05:22.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.946e-05 2024-09-21T10:05:22.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:05:22.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.630254+0000", 2024-09-21T10:05:22.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4900000000000003e-06 2024-09-21T10:05:22.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:05:22.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.630259+0000", 2024-09-21T10:05:22.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1320000000000002e-06 2024-09-21T10:05:22.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:05:22.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.631477+0000", 2024-09-21T10:05:22.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001217641 2024-09-21T10:05:22.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.051 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:05:22.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.631596+0000", 2024-09-21T10:05:22.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000118804 2024-09-21T10:05:22.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.051 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:05:22.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.631725+0000", 2024-09-21T10:05:22.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012932699999999999 2024-09-21T10:05:22.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.051 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635243+0000", 2024-09-21T10:05:22.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035178549999999999 2024-09-21T10:05:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:05:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635565+0000", 2024-09-21T10:05:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000322412 2024-09-21T10:05:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.755430+0000", 2024-09-21T10:05:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.119864811 2024-09-21T10:05:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:05:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.755480+0000", 2024-09-21T10:05:22.053 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9493000000000002e-05 2024-09-21T10:05:22.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.053 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:05:22.053 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.755518+0000", 2024-09-21T10:05:22.053 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7897e-05 2024-09-21T10:05:22.053 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.053 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:05:22.053 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.053 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:35224 5.a 5:51fa5009:::benchmark_data_smithi098_135187_object35223:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:05:22.053 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.635579+0000", 2024-09-21T10:05:22.053 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 80.406663472000005, 2024-09-21T10:05:22.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11871541400000001, 2024-09-21T10:05:22.054 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:05:22.054 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:05:22.054 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:22.054 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:05:22.054 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:05:22.054 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:05:22.054 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35224 2024-09-21T10:05:22.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.054 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:05:22.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.054 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:05:22.054 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635579+0000", 2024-09-21T10:05:22.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:05:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635579+0000", 2024-09-21T10:05:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:05:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635581+0000", 2024-09-21T10:05:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3230000000000001e-06 2024-09-21T10:05:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:05:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635602+0000", 2024-09-21T10:05:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1001000000000001e-05 2024-09-21T10:05:22.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.056 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:05:22.056 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635610+0000", 2024-09-21T10:05:22.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1410000000000005e-06 2024-09-21T10:05:22.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.056 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:05:22.056 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635616+0000", 2024-09-21T10:05:22.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2570000000000004e-06 2024-09-21T10:05:22.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.057 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:05:22.057 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635659+0000", 2024-09-21T10:05:22.057 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2892999999999997e-05 2024-09-21T10:05:22.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.057 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:05:22.057 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635779+0000", 2024-09-21T10:05:22.057 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000120025 2024-09-21T10:05:22.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.057 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:05:22.057 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635905+0000", 2024-09-21T10:05:22.057 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000126382 2024-09-21T10:05:22.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.058 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:05:22.058 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754075+0000", 2024-09-21T10:05:22.058 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.118169815 2024-09-21T10:05:22.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.058 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.058 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754127+0000", 2024-09-21T10:05:22.058 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2080999999999998e-05 2024-09-21T10:05:22.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.058 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.058 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754200+0000", 2024-09-21T10:05:22.059 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3053999999999995e-05 2024-09-21T10:05:22.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.059 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:05:22.059 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754253+0000", 2024-09-21T10:05:22.059 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2377999999999999e-05 2024-09-21T10:05:22.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.059 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:05:22.059 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754295+0000", 2024-09-21T10:05:22.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2064000000000002e-05 2024-09-21T10:05:22.060 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.060 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:05:22.060 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.060 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:35226 5.a 5:577c758a:::benchmark_data_smithi098_135187_object35225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:05:22.060 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.635769+0000", 2024-09-21T10:05:22.060 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 80.406473579999997, 2024-09-21T10:05:22.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11862623799999999, 2024-09-21T10:05:22.060 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:05:22.060 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:05:22.061 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:22.061 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:05:22.061 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:05:22.062 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:05:22.062 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35226 2024-09-21T10:05:22.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.062 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:05:22.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.062 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:05:22.062 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635769+0000", 2024-09-21T10:05:22.062 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.062 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:05:22.062 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635769+0000", 2024-09-21T10:05:22.063 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.063 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:05:22.063 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635770+0000", 2024-09-21T10:05:22.063 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8199999999999999e-07 2024-09-21T10:05:22.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.063 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:05:22.063 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635788+0000", 2024-09-21T10:05:22.063 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8505999999999999e-05 2024-09-21T10:05:22.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.064 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:05:22.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635799+0000", 2024-09-21T10:05:22.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0173000000000001e-05 2024-09-21T10:05:22.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.064 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:05:22.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635805+0000", 2024-09-21T10:05:22.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2299999999999996e-06 2024-09-21T10:05:22.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.064 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:05:22.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637399+0000", 2024-09-21T10:05:22.065 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0015946249999999999 2024-09-21T10:05:22.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:05:22.065 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637531+0000", 2024-09-21T10:05:22.065 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013183800000000001 2024-09-21T10:05:22.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:05:22.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637674+0000", 2024-09-21T10:05:22.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014236000000000001 2024-09-21T10:05:22.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.066 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:05:22.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754081+0000", 2024-09-21T10:05:22.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.116406883 2024-09-21T10:05:22.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.066 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754168+0000", 2024-09-21T10:05:22.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7264999999999999e-05 2024-09-21T10:05:22.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.067 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.067 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754327+0000", 2024-09-21T10:05:22.067 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015965599999999999 2024-09-21T10:05:22.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.067 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:05:22.067 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754367+0000", 2024-09-21T10:05:22.067 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9221999999999998e-05 2024-09-21T10:05:22.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.068 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:05:22.068 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754395+0000", 2024-09-21T10:05:22.068 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8697999999999999e-05 2024-09-21T10:05:22.068 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.068 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:05:22.068 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.068 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.44894.0:35224 5.a e110/105 ondisk, result = 0)", 2024-09-21T10:05:22.068 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.637962+0000", 2024-09-21T10:05:22.068 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 80.404281045999994, 2024-09-21T10:05:22.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.116172572, 2024-09-21T10:05:22.069 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:05:22.069 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:05:22.069 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:05:22.069 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:05:22.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.069 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:05:22.069 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637962+0000", 2024-09-21T10:05:22.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:05:22.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637962+0000", 2024-09-21T10:05:22.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:05:22.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637963+0000", 2024-09-21T10:05:22.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1179999999999999e-06 2024-09-21T10:05:22.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:05:22.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637968+0000", 2024-09-21T10:05:22.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8910000000000003e-06 2024-09-21T10:05:22.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.071 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:05:22.071 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637976+0000", 2024-09-21T10:05:22.071 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6289999999999999e-06 2024-09-21T10:05:22.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.071 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:05:22.071 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637982+0000", 2024-09-21T10:05:22.071 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2299999999999999e-06 2024-09-21T10:05:22.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.071 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:05:22.071 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754114+0000", 2024-09-21T10:05:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.116132108 2024-09-21T10:05:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:05:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754122+0000", 2024-09-21T10:05:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1750000000000005e-06 2024-09-21T10:05:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:05:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754134+0000", 2024-09-21T10:05:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2421e-05 2024-09-21T10:05:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:05:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.073 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:35229 5.b 5:d08d135d:::benchmark_data_smithi098_135187_object35228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:05:22.073 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.638050+0000", 2024-09-21T10:05:22.073 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 80.404193086999996, 2024-09-21T10:05:22.073 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.117567405, 2024-09-21T10:05:22.073 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:05:22.073 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:05:22.073 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:22.073 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:05:22.073 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:05:22.074 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:05:22.074 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35229 2024-09-21T10:05:22.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.074 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:05:22.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.074 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:05:22.074 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638050+0000", 2024-09-21T10:05:22.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.075 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:05:22.075 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638050+0000", 2024-09-21T10:05:22.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.075 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:05:22.075 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638051+0000", 2024-09-21T10:05:22.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.502e-06 2024-09-21T10:05:22.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.075 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:05:22.075 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638072+0000", 2024-09-21T10:05:22.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0976e-05 2024-09-21T10:05:22.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:05:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638085+0000", 2024-09-21T10:05:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2988e-05 2024-09-21T10:05:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:05:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638092+0000", 2024-09-21T10:05:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1709999999999996e-06 2024-09-21T10:05:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:05:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638154+0000", 2024-09-21T10:05:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1567000000000002e-05 2024-09-21T10:05:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.077 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:05:22.077 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638258+0000", 2024-09-21T10:05:22.077 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000103959 2024-09-21T10:05:22.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.077 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:05:22.077 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638372+0000", 2024-09-21T10:05:22.077 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000113888 2024-09-21T10:05:22.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.077 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.077 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754684+0000", 2024-09-21T10:05:22.077 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11631245 2024-09-21T10:05:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:05:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754695+0000", 2024-09-21T10:05:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0969e-05 2024-09-21T10:05:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.755551+0000", 2024-09-21T10:05:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00085551900000000001 2024-09-21T10:05:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:05:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.755589+0000", 2024-09-21T10:05:22.079 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8213000000000002e-05 2024-09-21T10:05:22.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.079 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:05:22.079 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.755617+0000", 2024-09-21T10:05:22.079 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8203000000000001e-05 2024-09-21T10:05:22.079 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.079 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:05:22.079 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.079 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:35231 5.d 5:bb22eb7b:::benchmark_data_smithi098_135187_object35230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:05:22.079 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.638970+0000", 2024-09-21T10:05:22.079 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 80.403272705999996, 2024-09-21T10:05:22.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.12243952, 2024-09-21T10:05:22.080 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:05:22.080 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:05:22.080 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:22.080 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:05:22.080 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:05:22.080 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:05:22.080 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35231 2024-09-21T10:05:22.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.080 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:05:22.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.080 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:05:22.080 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638970+0000", 2024-09-21T10:05:22.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:05:22.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638970+0000", 2024-09-21T10:05:22.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:05:22.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638971+0000", 2024-09-21T10:05:22.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1030000000000001e-06 2024-09-21T10:05:22.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:05:22.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.639022+0000", 2024-09-21T10:05:22.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0949999999999998e-05 2024-09-21T10:05:22.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.082 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:05:22.082 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754209+0000", 2024-09-21T10:05:22.082 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11518701200000001 2024-09-21T10:05:22.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.082 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:05:22.082 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754219+0000", 2024-09-21T10:05:22.082 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8430000000000005e-06 2024-09-21T10:05:22.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.082 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:05:22.082 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754669+0000", 2024-09-21T10:05:22.083 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044998299999999999 2024-09-21T10:05:22.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.083 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:05:22.083 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754792+0000", 2024-09-21T10:05:22.083 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012265999999999999 2024-09-21T10:05:22.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.083 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,7", 2024-09-21T10:05:22.083 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754929+0000", 2024-09-21T10:05:22.083 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013770199999999999 2024-09-21T10:05:22.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.083 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.761225+0000", 2024-09-21T10:05:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.006295429 2024-09-21T10:05:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:05:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.761275+0000", 2024-09-21T10:05:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9883999999999999e-05 2024-09-21T10:05:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.761309+0000", 2024-09-21T10:05:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4767000000000001e-05 2024-09-21T10:05:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:05:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.761355+0000", 2024-09-21T10:05:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5918000000000003e-05 2024-09-21T10:05:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:05:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.761410+0000", 2024-09-21T10:05:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4268999999999998e-05 2024-09-21T10:05:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:05:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.44894.0:35226 5.a e110/105 ondisk, result = 0)", 2024-09-21T10:05:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.639410+0000", 2024-09-21T10:05:22.086 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 80.402832673000006, 2024-09-21T10:05:22.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.114763873, 2024-09-21T10:05:22.086 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:05:22.086 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:05:22.086 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:05:22.086 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:05:22.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.086 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:05:22.086 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.639410+0000", 2024-09-21T10:05:22.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.086 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:05:22.087 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.639410+0000", 2024-09-21T10:05:22.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.087 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:05:22.087 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.639411+0000", 2024-09-21T10:05:22.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-21T10:05:22.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.087 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:05:22.087 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.639481+0000", 2024-09-21T10:05:22.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0146000000000002e-05 2024-09-21T10:05:22.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.087 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:05:22.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754050+0000", 2024-09-21T10:05:22.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11456915300000001 2024-09-21T10:05:22.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:05:22.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754062+0000", 2024-09-21T10:05:22.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2055000000000001e-05 2024-09-21T10:05:22.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:05:22.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754158+0000", 2024-09-21T10:05:22.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5880999999999999e-05 2024-09-21T10:05:22.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.089 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:05:22.089 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754163+0000", 2024-09-21T10:05:22.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3909999999999997e-06 2024-09-21T10:05:22.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.089 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:05:22.089 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754174+0000", 2024-09-21T10:05:22.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0548e-05 2024-09-21T10:05:22.089 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.089 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:05:22.089 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.089 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.44894.0:35229 5.b e110/105 ondisk, result = 0)", 2024-09-21T10:05:22.089 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.640577+0000", 2024-09-21T10:05:22.090 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 80.401666249000002, 2024-09-21T10:05:22.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.114119283, 2024-09-21T10:05:22.090 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:05:22.090 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:05:22.090 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:05:22.090 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:05:22.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:05:22.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.640577+0000", 2024-09-21T10:05:22.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:05:22.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.640577+0000", 2024-09-21T10:05:22.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.091 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:05:22.091 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.640577+0000", 2024-09-21T10:05:22.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.51e-07 2024-09-21T10:05:22.091 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:22.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:05:22.092 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.640580+0000", 2024-09-21T10:05:22.092 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1099999999999999e-06 2024-09-21T10:05:22.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:05:22.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754580+0000", 2024-09-21T10:05:22.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.114000249 2024-09-21T10:05:22.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:05:22.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754587+0000", 2024-09-21T10:05:22.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3790000000000003e-06 2024-09-21T10:05:22.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:05:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754672+0000", 2024-09-21T10:05:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4889000000000007e-05 2024-09-21T10:05:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:05:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754679+0000", 2024-09-21T10:05:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1690000000000003e-06 2024-09-21T10:05:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:05:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754696+0000", 2024-09-21T10:05:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6935999999999998e-05 2024-09-21T10:05:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:05:22.095 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.095 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:35766 5.8 5:13b21daa:::benchmark_data_smithi098_135187_object35765:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:05:22.095 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.983865+0000", 2024-09-21T10:05:22.095 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 80.058377777000004, 2024-09-21T10:05:22.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.085926309000000006, 2024-09-21T10:05:22.095 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:05:22.095 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:05:22.095 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:22.096 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:05:22.096 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:05:22.096 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:05:22.096 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35766 2024-09-21T10:05:22.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.096 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:05:22.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.096 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:05:22.096 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.983865+0000", 2024-09-21T10:05:22.096 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.096 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:05:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.983865+0000", 2024-09-21T10:05:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:05:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.983866+0000", 2024-09-21T10:05:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0499999999999999e-06 2024-09-21T10:05:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:05:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.983886+0000", 2024-09-21T10:05:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0242999999999999e-05 2024-09-21T10:05:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:05:22.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.983894+0000", 2024-09-21T10:05:22.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3560000000000004e-06 2024-09-21T10:05:22.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:05:22.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.983898+0000", 2024-09-21T10:05:22.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8720000000000001e-06 2024-09-21T10:05:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:05:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.983960+0000", 2024-09-21T10:05:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1277000000000003e-05 2024-09-21T10:05:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:05:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:02.028580+0000", 2024-09-21T10:05:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.044620029999999998 2024-09-21T10:05:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:05:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:02.028728+0000", 2024-09-21T10:05:22.100 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000148331 2024-09-21T10:05:22.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.100 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.100 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:02.069558+0000", 2024-09-21T10:05:22.100 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.040829866999999999 2024-09-21T10:05:22.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.100 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.100 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:02.069658+0000", 2024-09-21T10:05:22.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9785999999999996e-05 2024-09-21T10:05:22.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:05:22.101 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:02.069710+0000", 2024-09-21T10:05:22.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2099999999999999e-05 2024-09-21T10:05:22.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:05:22.101 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:02.069751+0000", 2024-09-21T10:05:22.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1207e-05 2024-09-21T10:05:22.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:05:22.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:02.069791+0000", 2024-09-21T10:05:22.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.019e-05 2024-09-21T10:05:22.102 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.102 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:05:22.102 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.102 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:86398 5.d 5:b0bc3443:::benchmark_data_smithi098_135187_object86397:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T10:05:22.102 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:33.139751+0000", 2024-09-21T10:05:22.102 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.90249171, 2024-09-21T10:05:22.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.100480799, 2024-09-21T10:05:22.103 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:05:22.103 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:05:22.103 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:22.103 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:05:22.103 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:05:22.103 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:05:22.103 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 86398 2024-09-21T10:05:22.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.103 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:05:22.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.103 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:05:22.104 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.139751+0000", 2024-09-21T10:05:22.104 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.104 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:05:22.104 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.139751+0000", 2024-09-21T10:05:22.104 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.104 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:05:22.104 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.139752+0000", 2024-09-21T10:05:22.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7700000000000004e-07 2024-09-21T10:05:22.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.105 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:05:22.105 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.139770+0000", 2024-09-21T10:05:22.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8488999999999999e-05 2024-09-21T10:05:22.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.105 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:05:22.105 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.139779+0000", 2024-09-21T10:05:22.106 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.541e-06 2024-09-21T10:05:22.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.106 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:05:22.106 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.139783+0000", 2024-09-21T10:05:22.106 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6349999999999997e-06 2024-09-21T10:05:22.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.106 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:05:22.107 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.139823+0000", 2024-09-21T10:05:22.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9984999999999997e-05 2024-09-21T10:05:22.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.107 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:05:22.107 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.139907+0000", 2024-09-21T10:05:22.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3202999999999996e-05 2024-09-21T10:05:22.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.107 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,7", 2024-09-21T10:05:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.139991+0000", 2024-09-21T10:05:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4208000000000004e-05 2024-09-21T10:05:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:05:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.142889+0000", 2024-09-21T10:05:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028978710000000002 2024-09-21T10:05:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240055+0000", 2024-09-21T10:05:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.097166401999999999 2024-09-21T10:05:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.109 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.109 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240144+0000", 2024-09-21T10:05:22.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9240999999999995e-05 2024-09-21T10:05:22.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.109 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:05:22.109 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240193+0000", 2024-09-21T10:05:22.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8211999999999999e-05 2024-09-21T10:05:22.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.109 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:05:22.109 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240232+0000", 2024-09-21T10:05:22.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9235e-05 2024-09-21T10:05:22.110 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.110 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:05:22.110 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.110 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:86401 5.d 5:bfc65391:::benchmark_data_smithi098_135187_object86400:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T10:05:22.110 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:33.146216+0000", 2024-09-21T10:05:22.110 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.896027197000002, 2024-09-21T10:05:22.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.094144248999999999, 2024-09-21T10:05:22.110 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:05:22.110 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:05:22.110 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:05:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:05:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:05:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 86401 2024-09-21T10:05:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:05:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:05:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146216+0000", 2024-09-21T10:05:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:05:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146216+0000", 2024-09-21T10:05:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:05:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146217+0000", 2024-09-21T10:05:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.31e-06 2024-09-21T10:05:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:05:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146235+0000", 2024-09-21T10:05:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7642000000000002e-05 2024-09-21T10:05:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:05:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146244+0000", 2024-09-21T10:05:22.113 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8859999999999999e-06 2024-09-21T10:05:22.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.113 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:05:22.113 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146249+0000", 2024-09-21T10:05:22.113 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9599999999999999e-06 2024-09-21T10:05:22.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.113 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:05:22.113 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146295+0000", 2024-09-21T10:05:22.113 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5552e-05 2024-09-21T10:05:22.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.113 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:05:22.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146418+0000", 2024-09-21T10:05:22.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123144 2024-09-21T10:05:22.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.114 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,7", 2024-09-21T10:05:22.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146510+0000", 2024-09-21T10:05:22.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2324999999999994e-05 2024-09-21T10:05:22.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.114 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240099+0000", 2024-09-21T10:05:22.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.093588888999999995 2024-09-21T10:05:22.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.115 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.115 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240269+0000", 2024-09-21T10:05:22.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000170194 2024-09-21T10:05:22.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.115 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:05:22.115 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240280+0000", 2024-09-21T10:05:22.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0720000000000001e-05 2024-09-21T10:05:22.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.115 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:05:22.115 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240324+0000", 2024-09-21T10:05:22.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4285e-05 2024-09-21T10:05:22.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:05:22.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240360+0000", 2024-09-21T10:05:22.116 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5342000000000001e-05 2024-09-21T10:05:22.116 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.116 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:05:22.116 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.116 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:86402 5.8 5:1c1635b7:::benchmark_data_smithi098_135187_object86401:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T10:05:22.116 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:33.146388+0000", 2024-09-21T10:05:22.116 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.895854628000002, 2024-09-21T10:05:22.116 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.103009368, 2024-09-21T10:05:22.116 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:05:22.117 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:05:22.117 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:22.117 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:05:22.117 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:05:22.117 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:05:22.117 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 86402 2024-09-21T10:05:22.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.117 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:05:22.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:05:22.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146388+0000", 2024-09-21T10:05:22.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:05:22.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146388+0000", 2024-09-21T10:05:22.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:05:22.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146389+0000", 2024-09-21T10:05:22.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5000000000000002e-07 2024-09-21T10:05:22.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:05:22.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146406+0000", 2024-09-21T10:05:22.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6949e-05 2024-09-21T10:05:22.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.119 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:05:22.119 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146413+0000", 2024-09-21T10:05:22.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2810000000000003e-06 2024-09-21T10:05:22.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.119 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:05:22.119 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146417+0000", 2024-09-21T10:05:22.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.262e-06 2024-09-21T10:05:22.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.119 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:05:22.119 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146457+0000", 2024-09-21T10:05:22.120 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9858999999999999e-05 2024-09-21T10:05:22.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.120 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:05:22.120 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146533+0000", 2024-09-21T10:05:22.120 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6143000000000006e-05 2024-09-21T10:05:22.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.120 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:05:22.120 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146619+0000", 2024-09-21T10:05:22.120 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5394000000000003e-05 2024-09-21T10:05:22.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.120 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.120 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.245390+0000", 2024-09-21T10:05:22.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.098770872999999995 2024-09-21T10:05:22.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.245426+0000", 2024-09-21T10:05:22.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6656000000000003e-05 2024-09-21T10:05:22.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:05:22.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.249306+0000", 2024-09-21T10:05:22.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038799590000000001 2024-09-21T10:05:22.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:05:22.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.249358+0000", 2024-09-21T10:05:22.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1876999999999998e-05 2024-09-21T10:05:22.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:05:22.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.249397+0000", 2024-09-21T10:05:22.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9465000000000002e-05 2024-09-21T10:05:22.122 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.122 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:05:22.122 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.122 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.44894.0:86398 5.d e114/105 ondisk, result = 0)", 2024-09-21T10:05:22.122 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:33.146632+0000", 2024-09-21T10:05:22.123 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.895611035999998, 2024-09-21T10:05:22.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.093433566999999995, 2024-09-21T10:05:22.123 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:05:22.123 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:05:22.123 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:05:22.123 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:05:22.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:05:22.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146632+0000", 2024-09-21T10:05:22.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:05:22.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146632+0000", 2024-09-21T10:05:22.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:05:22.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146633+0000", 2024-09-21T10:05:22.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8100000000000001e-07 2024-09-21T10:05:22.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:05:22.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146637+0000", 2024-09-21T10:05:22.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.493e-06 2024-09-21T10:05:22.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:05:22.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146648+0000", 2024-09-21T10:05:22.125 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0521e-05 2024-09-21T10:05:22.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:05:22.125 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146654+0000", 2024-09-21T10:05:22.125 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4309999999999999e-06 2024-09-21T10:05:22.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:05:22.125 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240040+0000", 2024-09-21T10:05:22.125 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.093385900999999993 2024-09-21T10:05:22.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:05:22.125 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240048+0000", 2024-09-21T10:05:22.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1640000000000001e-06 2024-09-21T10:05:22.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.126 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:05:22.126 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240065+0000", 2024-09-21T10:05:22.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7176e-05 2024-09-21T10:05:22.126 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.126 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:05:22.126 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.126 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:86404 5.a 5:566c121b:::benchmark_data_smithi098_135187_object86403:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T10:05:22.126 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:33.147238+0000", 2024-09-21T10:05:22.126 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.895005247, 2024-09-21T10:05:22.127 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.092921171999999996, 2024-09-21T10:05:22.127 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:05:22.127 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:05:22.127 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:22.127 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:05:22.127 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:05:22.127 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:05:22.127 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 86404 2024-09-21T10:05:22.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.127 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:05:22.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.127 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:05:22.127 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.147238+0000", 2024-09-21T10:05:22.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:05:22.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.147238+0000", 2024-09-21T10:05:22.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:05:22.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.147239+0000", 2024-09-21T10:05:22.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2979999999999999e-06 2024-09-21T10:05:22.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:05:22.129 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.147261+0000", 2024-09-21T10:05:22.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2067000000000001e-05 2024-09-21T10:05:22.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:05:22.129 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.147272+0000", 2024-09-21T10:05:22.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1022000000000001e-05 2024-09-21T10:05:22.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:05:22.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.147279+0000", 2024-09-21T10:05:22.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0169999999999999e-06 2024-09-21T10:05:22.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:05:22.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.147324+0000", 2024-09-21T10:05:22.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5365999999999999e-05 2024-09-21T10:05:22.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:05:22.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.147437+0000", 2024-09-21T10:05:22.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000112553 2024-09-21T10:05:22.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.131 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:05:22.131 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.147556+0000", 2024-09-21T10:05:22.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000118887 2024-09-21T10:05:22.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.131 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.131 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150842+0000", 2024-09-21T10:05:22.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003285821 2024-09-21T10:05:22.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.131 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:05:22.132 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150852+0000", 2024-09-21T10:05:22.132 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0084e-05 2024-09-21T10:05:22.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.132 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.132 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150944+0000", 2024-09-21T10:05:22.132 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2577000000000005e-05 2024-09-21T10:05:22.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.132 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:05:22.132 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240100+0000", 2024-09-21T10:05:22.133 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.089155836000000002 2024-09-21T10:05:22.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.133 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:05:22.133 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240159+0000", 2024-09-21T10:05:22.133 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8644000000000003e-05 2024-09-21T10:05:22.133 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.133 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:05:22.133 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.133 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.44894.0:86401 5.d e114/105 ondisk, result = 0)", 2024-09-21T10:05:22.133 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:33.150446+0000", 2024-09-21T10:05:22.134 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.891796941000003, 2024-09-21T10:05:22.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.089670174000000005, 2024-09-21T10:05:22.134 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:05:22.134 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:05:22.134 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:05:22.134 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:05:22.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.134 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:05:22.134 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150446+0000", 2024-09-21T10:05:22.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.135 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:05:22.135 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150446+0000", 2024-09-21T10:05:22.135 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.135 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:05:22.135 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150447+0000", 2024-09-21T10:05:22.135 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2720000000000001e-06 2024-09-21T10:05:22.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.135 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:05:22.136 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150453+0000", 2024-09-21T10:05:22.136 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8710000000000002e-06 2024-09-21T10:05:22.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.136 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:05:22.136 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150465+0000", 2024-09-21T10:05:22.136 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1963e-05 2024-09-21T10:05:22.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.136 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:05:22.136 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150473+0000", 2024-09-21T10:05:22.136 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7090000000000002e-06 2024-09-21T10:05:22.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.137 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:05:22.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240088+0000", 2024-09-21T10:05:22.137 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.089615808000000005 2024-09-21T10:05:22.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.137 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:05:22.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240094+0000", 2024-09-21T10:05:22.138 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8629999999999996e-06 2024-09-21T10:05:22.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.139 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:05:22.139 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240116+0000", 2024-09-21T10:05:22.139 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1688e-05 2024-09-21T10:05:22.139 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.139 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:05:22.139 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.139 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:86407 5.8 5:17fee9e8:::benchmark_data_smithi098_135187_object86406:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T10:05:22.140 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:33.150648+0000", 2024-09-21T10:05:22.140 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.891594507000001, 2024-09-21T10:05:22.140 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.098833616999999999, 2024-09-21T10:05:22.140 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:05:22.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:05:22.140 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:22.140 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:05:22.140 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:05:22.140 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:05:22.140 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 86407 2024-09-21T10:05:22.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.140 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:05:22.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.140 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:05:22.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150648+0000", 2024-09-21T10:05:22.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.141 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:05:22.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150648+0000", 2024-09-21T10:05:22.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.141 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:05:22.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150650+0000", 2024-09-21T10:05:22.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5379999999999999e-06 2024-09-21T10:05:22.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:05:22.142 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150673+0000", 2024-09-21T10:05:22.142 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2742e-05 2024-09-21T10:05:22.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:05:22.142 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150685+0000", 2024-09-21T10:05:22.142 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2958e-05 2024-09-21T10:05:22.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.143 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:05:22.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150693+0000", 2024-09-21T10:05:22.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0099999999999995e-06 2024-09-21T10:05:22.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.143 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:05:22.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.239528+0000", 2024-09-21T10:05:22.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.088834479999999993 2024-09-21T10:05:22.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.143 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:05:22.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.239673+0000", 2024-09-21T10:05:22.144 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014516900000000001 2024-09-21T10:05:22.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.144 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:05:22.144 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.239814+0000", 2024-09-21T10:05:22.144 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014037699999999999 2024-09-21T10:05:22.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.144 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.144 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.245455+0000", 2024-09-21T10:05:22.144 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056417710000000003 2024-09-21T10:05:22.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.144 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.248791+0000", 2024-09-21T10:05:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00333526 2024-09-21T10:05:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:05:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.249411+0000", 2024-09-21T10:05:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00061998800000000003 2024-09-21T10:05:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:05:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.249450+0000", 2024-09-21T10:05:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9888000000000002e-05 2024-09-21T10:05:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.146 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:05:22.146 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.249482+0000", 2024-09-21T10:05:22.146 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1436000000000002e-05 2024-09-21T10:05:22.146 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.146 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:05:22.146 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.146 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.44894.0:86402 5.8 e114/105 ondisk, result = 0)", 2024-09-21T10:05:22.146 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:33.150733+0000", 2024-09-21T10:05:22.146 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.891509538999998, 2024-09-21T10:05:22.146 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.094663991000000003, 2024-09-21T10:05:22.146 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:05:22.146 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:05:22.147 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:05:22.147 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:05:22.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.147 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:05:22.147 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150733+0000", 2024-09-21T10:05:22.147 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.147 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:05:22.147 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150733+0000", 2024-09-21T10:05:22.147 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.147 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:05:22.147 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150734+0000", 2024-09-21T10:05:22.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5499999999999998e-07 2024-09-21T10:05:22.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:05:22.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150737+0000", 2024-09-21T10:05:22.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9809999999999999e-06 2024-09-21T10:05:22.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:05:22.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150745+0000", 2024-09-21T10:05:22.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2789999999999998e-06 2024-09-21T10:05:22.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:05:22.149 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150750+0000", 2024-09-21T10:05:22.149 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4620000000000004e-06 2024-09-21T10:05:22.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.149 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:05:22.149 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.245376+0000", 2024-09-21T10:05:22.149 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.094625090999999995 2024-09-21T10:05:22.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.149 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:05:22.149 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.245383+0000", 2024-09-21T10:05:22.149 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7409999999999992e-06 2024-09-21T10:05:22.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.150 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:05:22.150 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.245397+0000", 2024-09-21T10:05:22.150 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3882e-05 2024-09-21T10:05:22.150 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.150 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:05:22.150 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.150 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.44894.0:86404 5.a e114/105 ondisk, result = 0)", 2024-09-21T10:05:22.150 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:33.150867+0000", 2024-09-21T10:05:22.150 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.891375734999997, 2024-09-21T10:05:22.150 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.089299100000000006, 2024-09-21T10:05:22.150 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:05:22.150 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:05:22.151 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:05:22.151 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:05:22.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.151 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:05:22.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150867+0000", 2024-09-21T10:05:22.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.151 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:05:22.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150867+0000", 2024-09-21T10:05:22.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.151 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:05:22.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150868+0000", 2024-09-21T10:05:22.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2399999999999997e-07 2024-09-21T10:05:22.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:05:22.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150871+0000", 2024-09-21T10:05:22.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5760000000000002e-06 2024-09-21T10:05:22.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:05:22.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150879+0000", 2024-09-21T10:05:22.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9619999999999993e-06 2024-09-21T10:05:22.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:05:22.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150884+0000", 2024-09-21T10:05:22.153 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.152e-06 2024-09-21T10:05:22.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.153 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:05:22.153 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150920+0000", 2024-09-21T10:05:22.153 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5278999999999999e-05 2024-09-21T10:05:22.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.153 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:05:22.153 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.150927+0000", 2024-09-21T10:05:22.153 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4420000000000001e-06 2024-09-21T10:05:22.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.153 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:05:22.154 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240166+0000", 2024-09-21T10:05:22.154 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.089238965000000003 2024-09-21T10:05:22.154 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.154 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:05:22.154 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.154 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:86411 5.a 5:5f80647e:::benchmark_data_smithi098_135187_object86410:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T10:05:22.154 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:33.151250+0000", 2024-09-21T10:05:22.154 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.890992296999997, 2024-09-21T10:05:22.154 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.10103477399999999, 2024-09-21T10:05:22.154 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:05:22.154 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:05:22.154 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:22.155 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:05:22.155 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:05:22.155 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:05:22.155 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 86411 2024-09-21T10:05:22.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.155 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:05:22.155 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.155 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:05:22.155 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.151250+0000", 2024-09-21T10:05:22.155 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.155 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.155 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:05:22.155 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.151250+0000", 2024-09-21T10:05:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:05:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.151252+0000", 2024-09-21T10:05:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.096e-06 2024-09-21T10:05:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:05:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.151272+0000", 2024-09-21T10:05:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0937e-05 2024-09-21T10:05:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:05:22.157 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.239495+0000", 2024-09-21T10:05:22.157 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.088222574999999998 2024-09-21T10:05:22.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.157 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:05:22.157 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.239505+0000", 2024-09-21T10:05:22.157 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9510000000000001e-06 2024-09-21T10:05:22.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.157 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:05:22.157 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240198+0000", 2024-09-21T10:05:22.157 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00069264100000000002 2024-09-21T10:05:22.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.157 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:05:22.158 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240307+0000", 2024-09-21T10:05:22.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010927 2024-09-21T10:05:22.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.158 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:05:22.158 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240433+0000", 2024-09-21T10:05:22.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012565800000000001 2024-09-21T10:05:22.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.158 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.158 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.251994+0000", 2024-09-21T10:05:22.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011561068000000001 2024-09-21T10:05:22.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.159 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:22.159 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.252070+0000", 2024-09-21T10:05:22.159 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6796000000000001e-05 2024-09-21T10:05:22.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.159 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:05:22.159 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.252199+0000", 2024-09-21T10:05:22.159 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012868299999999999 2024-09-21T10:05:22.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.159 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:05:22.159 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.252243+0000", 2024-09-21T10:05:22.159 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3560999999999998e-05 2024-09-21T10:05:22.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:05:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.252285+0000", 2024-09-21T10:05:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2537999999999998e-05 2024-09-21T10:05:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:05:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.44894.0:86398 5.d e114/105 ondisk, result = 0)", 2024-09-21T10:05:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:33.152100+0000", 2024-09-21T10:05:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.890142705000002, 2024-09-21T10:05:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.088137498999999994, 2024-09-21T10:05:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:05:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:05:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:05:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:05:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:05:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.152100+0000", 2024-09-21T10:05:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:05:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.152100+0000", 2024-09-21T10:05:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:05:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:05:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.152101+0000", 2024-09-21T10:05:22.162 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0699999999999996e-07 2024-09-21T10:05:22.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.162 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:05:22.162 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.152125+0000", 2024-09-21T10:05:22.162 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4558000000000001e-05 2024-09-21T10:05:22.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.162 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:05:22.162 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.239400+0000", 2024-09-21T10:05:22.162 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.087274245 2024-09-21T10:05:22.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.163 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:05:22.163 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.239411+0000", 2024-09-21T10:05:22.163 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1090000000000001e-05 2024-09-21T10:05:22.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.163 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:05:22.163 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240134+0000", 2024-09-21T10:05:22.163 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00072358899999999996 2024-09-21T10:05:22.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.163 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:05:22.164 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240140+0000", 2024-09-21T10:05:22.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8699999999999997e-06 2024-09-21T10:05:22.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:05:22.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:05:22.164 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:05:22.164 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240238+0000", 2024-09-21T10:05:22.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.734e-05 2024-09-21T10:05:22.164 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.164 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:05:22.164 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.164 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:05:22.164 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:05:22.164 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:05:22.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:22.172 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:22.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:22.274 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:22.311 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:22.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:22.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:22.412 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:22.438 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:22.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:22.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:22.540 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:22.571 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:22.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:22.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:22.673 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:22.707 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:22.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:22.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:22.791 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:22.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:22.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:22.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:22.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:22.977 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:22.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:22.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:23.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:22 smithi098 ceph-mon[97580]: pgmap v304: 89 pgs: 89 active+clean; 8.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 49 MiB/s wr, 787 op/s 2024-09-21T10:05:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:23.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:22 smithi098 ceph-mon[100914]: pgmap v304: 89 pgs: 89 active+clean; 8.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 49 MiB/s wr, 787 op/s 2024-09-21T10:05:23.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:23.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:23.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:23.213 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:23.243 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:23.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:23.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:23.345 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:23.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:22 smithi196 ceph-mon[84784]: pgmap v304: 89 pgs: 89 active+clean; 8.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 49 MiB/s wr, 787 op/s 2024-09-21T10:05:23.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:23.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:23.396 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:23.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:23.396 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:05:23.396 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:05:23.421 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:05:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:05:23.447 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:05:23.447 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:05:23.473 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:05:23.473 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:05:23.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:23.507 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:05:23.507 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:05:23.533 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:23.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:23.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:23.536 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:05:23.536 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:05:23.568 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:05:23.568 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:05:23.592 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:05:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:05:23.635 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:23.670 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:23.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:23.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:23.772 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:23.804 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:23.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:23.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:23.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:05:23.906 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:23.936 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:23.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:23.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:24.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:23 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3173949781' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:24.037 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:23 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3173949781' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:24.037 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:24.112 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:24.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:24.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:24.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:24.244 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:24.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:24.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:24.346 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:24.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:23 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3173949781' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:24.400 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:24.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:24.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:24.502 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:24.533 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:24.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:24.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:24.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:24.671 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:24.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:24.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:24.772 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:24.815 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:24.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:24.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:24.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:24.949 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:24.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:24.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:25.050 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:25.091 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:25.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:25.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:25.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:24 smithi098 ceph-mon[97580]: pgmap v305: 89 pgs: 89 active+clean; 8.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 43 MiB/s wr, 816 op/s 2024-09-21T10:05:25.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:24 smithi098 ceph-mon[100914]: pgmap v305: 89 pgs: 89 active+clean; 8.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 43 MiB/s wr, 816 op/s 2024-09-21T10:05:25.193 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:25.222 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:25.226 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:25.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:25.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:25.328 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:25.364 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:25.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:25.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:25.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:24 smithi196 ceph-mon[84784]: pgmap v305: 89 pgs: 89 active+clean; 8.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 43 MiB/s wr, 816 op/s 2024-09-21T10:05:25.466 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:25.494 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:25.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:25.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:25.596 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:25.623 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:25.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:25.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:25.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:25.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:25.765 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:25.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:25.866 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:25.900 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:25.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:25.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:26.002 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:26.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:26.042 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:26.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:26.143 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:26.181 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:26.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:26.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:26.283 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:26.318 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:26.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:26.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:26.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:26.454 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:26.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:26.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:26.556 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:26.587 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:26.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:26.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:26.689 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:26.719 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:26.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:26.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:26.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:26.849 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:26.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:26.865 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:26.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:26.954 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:05:26.966 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:26.999 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:27.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:27.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:26 smithi098 ceph-mon[97580]: pgmap v306: 89 pgs: 89 active+clean; 8.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 111 B/s rd, 42 MiB/s wr, 1.04k op/s 2024-09-21T10:05:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:26 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2693306039' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:05:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:26 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:05:27.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:26 smithi098 ceph-mon[100914]: pgmap v306: 89 pgs: 89 active+clean; 8.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 111 B/s rd, 42 MiB/s wr, 1.04k op/s 2024-09-21T10:05:27.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:26 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2693306039' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:05:27.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:26 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:05:27.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:27.139 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:27.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:27.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:27.240 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:27.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:27.281 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:27.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:27.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:26 smithi196 ceph-mon[84784]: pgmap v306: 89 pgs: 89 active+clean; 8.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 111 B/s rd, 42 MiB/s wr, 1.04k op/s 2024-09-21T10:05:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:26 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2693306039' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:05:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:26 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:05:27.417 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:27.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:27.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:27.519 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:27.560 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:27.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:27.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:27.662 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:27.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:05:27.694 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:27.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:27.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:27.796 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:27.820 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:27.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:27.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:27.922 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:27.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:27.955 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:27.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:28.055 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:28.093 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:28.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:28.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:28.195 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:28.239 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:28.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:28.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:28.341 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:28.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:27 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:05:28.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:27 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:05:28.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:27 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:05:28.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:27 smithi098 ceph-mon[97580]: osdmap e122: 8 total, 8 up, 8 in 2024-09-21T10:05:28.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:27 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:05:28.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:27 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:05:28.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:27 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:05:28.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:27 smithi098 ceph-mon[100914]: osdmap e122: 8 total, 8 up, 8 in 2024-09-21T10:05:28.374 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:28.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:28.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:28.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:27 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:05:28.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:27 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:05:28.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:27 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:05:28.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:27 smithi196 ceph-mon[84784]: osdmap e122: 8 total, 8 up, 8 in 2024-09-21T10:05:28.476 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:28.504 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:28.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:28.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:28.605 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:28.633 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:28.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:28.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:28.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:28.763 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:28.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:28.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:28.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:28.887 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:28.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:28.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:28.988 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:29.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:29.020 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:29.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:29.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:29.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:29.147 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:29.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:29.248 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:29.278 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:29.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:29.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:29.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:28 smithi098 ceph-mon[97580]: pgmap v308: 89 pgs: 89 active+clean; 8.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 127 B/s rd, 23 MiB/s wr, 820 op/s 2024-09-21T10:05:29.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:28 smithi098 ceph-mon[100914]: pgmap v308: 89 pgs: 89 active+clean; 8.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 127 B/s rd, 23 MiB/s wr, 820 op/s 2024-09-21T10:05:29.380 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:29.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:28 smithi196 ceph-mon[84784]: pgmap v308: 89 pgs: 89 active+clean; 8.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 127 B/s rd, 23 MiB/s wr, 820 op/s 2024-09-21T10:05:29.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:29.411 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:29.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:29.512 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:29.545 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:29.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:29.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:29.647 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:29.684 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:29.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:29.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:29.786 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:29.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:29.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:29.917 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:29.946 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:29.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:29.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:30.048 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:30.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:30.085 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:30.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:30.185 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:30.211 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:30.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:30.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:30.313 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:30.342 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:30.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:30.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:30.444 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:30.477 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:30.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:30.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:30.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:30.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:30.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:30.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:05:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:05:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:30.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:05:30.705 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:30.786 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:30.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:30.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:30.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:05:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:05:30] "GET /metrics HTTP/1.1" 200 36449 "" "Prometheus/2.51.0" 2024-09-21T10:05:30.888 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:30.926 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:30.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:31.027 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:31.070 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:31.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:31.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:31.172 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:31.175 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:30 smithi098 ceph-mon[100914]: pgmap v309: 89 pgs: 89 active+clean; 7.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 111 B/s rd, 9.8 MiB/s wr, 886 op/s 2024-09-21T10:05:31.176 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:30 smithi098 ceph-mon[97580]: pgmap v309: 89 pgs: 89 active+clean; 7.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 111 B/s rd, 9.8 MiB/s wr, 886 op/s 2024-09-21T10:05:31.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:31.221 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:31.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:31.322 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:31.368 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:31.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:31.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:31.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:30 smithi196 ceph-mon[84784]: pgmap v309: 89 pgs: 89 active+clean; 7.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 111 B/s rd, 9.8 MiB/s wr, 886 op/s 2024-09-21T10:05:31.470 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:31.506 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:31.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:31.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:31.608 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:31.641 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:31.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:31.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:31.743 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:31.784 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:31.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:31.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:31.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:31.913 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:31.913 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:05:31.913 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:05:31.937 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:31.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:31.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:32.039 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:32.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:32.074 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:32.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:32.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:31 smithi098 ceph-mon[97580]: pgmap v310: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 102 B/s rd, 8.9 MiB/s wr, 956 op/s 2024-09-21T10:05:32.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:31 smithi098 ceph-mon[100914]: pgmap v310: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 102 B/s rd, 8.9 MiB/s wr, 956 op/s 2024-09-21T10:05:32.175 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:32.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:32.212 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:32.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:32.313 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:32.352 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:32.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:32.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:32.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:31 smithi196 ceph-mon[84784]: pgmap v310: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 102 B/s rd, 8.9 MiB/s wr, 956 op/s 2024-09-21T10:05:32.454 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:32.491 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:32.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:32.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:32.592 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:32.617 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:05:32.623 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:32.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:32.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:32.725 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:32.758 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:32.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:32.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:32.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:32.885 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:32.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:32.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:32.987 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:33.009 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T10:05:33.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:33.018 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:33.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:33.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:32 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2444414132' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:05:33.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:32 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:05:33.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:32 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2444414132' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:05:33.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:32 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:05:33.119 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:33.160 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:33.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:33.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:33.300 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:33.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:33.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:33.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:32 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2444414132' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:05:33.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:32 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:05:33.402 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:33.430 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:33.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:33.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:33.532 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:33.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:33.567 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:33.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:33.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:33.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:33.732 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:33.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:33.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:33.874 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:33.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:33.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:33.976 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:34.013 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:34.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:34.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:34.116 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:34.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:34.153 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:34.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:34.254 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:34.297 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:34.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:34.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:34.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:34 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:05:34.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:34 smithi098 ceph-mon[100914]: osdmap e123: 8 total, 8 up, 8 in 2024-09-21T10:05:34.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:34 smithi098 ceph-mon[100914]: pgmap v312: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 813 op/s 2024-09-21T10:05:34.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:34 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:05:34.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:34 smithi098 ceph-mon[97580]: osdmap e123: 8 total, 8 up, 8 in 2024-09-21T10:05:34.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:34 smithi098 ceph-mon[97580]: pgmap v312: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 813 op/s 2024-09-21T10:05:34.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:34 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:05:34.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:34 smithi196 ceph-mon[84784]: osdmap e123: 8 total, 8 up, 8 in 2024-09-21T10:05:34.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:34 smithi196 ceph-mon[84784]: pgmap v312: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 813 op/s 2024-09-21T10:05:34.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:34.444 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:34.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:34.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:34.546 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:34.578 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:34.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:34.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:34.680 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:34.716 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:34.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:34.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:34.818 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:34.852 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:34.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:34.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:34.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:34.977 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:34.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:34.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:35.078 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:35.113 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:35.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:35.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:35.215 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:35.246 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:35.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:35.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:35.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:35.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:35.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:35.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:35.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:35.518 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:35.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:35.619 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:35.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:35.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:35.758 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:35.791 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:35.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:35.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:35.892 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:35.918 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:35.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:35.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:36.020 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:36.051 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:36.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:36.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:36.152 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:36.173 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:36.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:36.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:36.275 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:36.307 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:36.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:36.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:36.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:36 smithi196 ceph-mon[84784]: pgmap v313: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-21T10:05:36.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:36.436 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:36.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:36.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:36.537 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:36.581 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:36.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:36.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:36.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:36 smithi098 ceph-mon[97580]: pgmap v313: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-21T10:05:36.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:36 smithi098 ceph-mon[100914]: pgmap v313: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-21T10:05:36.683 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:36.729 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:36.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:36.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:36.831 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:36.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:36.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:36.968 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:37.004 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:37.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:37.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:37.106 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:37.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:37.141 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:37.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:37.242 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:37.281 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:37.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:37.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:37.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T10:05:37.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:37.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:37.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:37.561 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:37.598 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:37.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:37.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:37.700 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:37.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:37.734 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:37.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:37.835 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:37.881 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:37.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:37.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:37.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:38.028 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:38.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:38.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:38.130 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:38.166 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:38.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:38.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:38.268 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:38.302 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:38.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:38.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:38.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:38 smithi196 ceph-mon[84784]: pgmap v314: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 922 op/s 2024-09-21T10:05:38.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:05:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:38.404 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:38.437 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:38.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:38.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:38.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:38.578 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:38.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:38.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:38.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:38 smithi098 ceph-mon[97580]: pgmap v314: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 922 op/s 2024-09-21T10:05:38.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:05:38.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:38.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:38 smithi098 ceph-mon[100914]: pgmap v314: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 922 op/s 2024-09-21T10:05:38.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:05:38.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:38.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:38.718 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:38.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:38.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:38.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:38.855 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:38.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:38.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:38.961 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:38.992 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:38.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:38.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:39.016 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:05:39.016 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:05:39.054 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:05:39.055 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:05:39.089 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:05:39.089 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:05:39.094 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:39.121 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:05:39.121 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:05:39.127 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:39.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:39.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:39.158 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:05:39.158 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:05:39.189 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:05:39.189 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:05:39.229 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:39.230 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:05:39.230 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:05:39.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:39.255 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:39.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:39.259 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:05:39.259 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:05:39.357 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:39.391 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:39.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:39.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:39.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:39 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1068986433' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:39.491 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:39 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1068986433' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:39.492 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:39.529 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:39.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:39.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:39.630 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:39.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:39 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1068986433' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:39.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:39.665 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:39.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:39.766 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:39.797 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:39.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:39.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:39.901 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:39.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:39.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:40.040 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:40.076 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:40.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:40.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:40.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:40.208 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:40.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:40.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:40.310 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:40.347 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:40.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:40.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:40.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:40.483 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:40.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:40.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:40.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:40 smithi098 ceph-mon[97580]: pgmap v315: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 882 op/s 2024-09-21T10:05:40.500 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:40 smithi098 ceph-mon[100914]: pgmap v315: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 882 op/s 2024-09-21T10:05:40.585 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:40.623 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:40.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:40.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:40.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:40 smithi196 ceph-mon[84784]: pgmap v315: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 882 op/s 2024-09-21T10:05:40.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:05:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:05:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:40.726 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:40.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:40.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:40.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:05:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:05:40] "GET /metrics HTTP/1.1" 200 36458 "" "Prometheus/2.51.0" 2024-09-21T10:05:40.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:40.900 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:40.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:40.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:41.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:41.036 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:41.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:41.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:41.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:41.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:41.175 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:41.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:41.278 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:41.313 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:41.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:41.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:41.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:41.446 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:41.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:41.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:41.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:41.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:41.585 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:41.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:41.638 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 10:05:41 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=sqlstore.transactions t=2024-09-21T10:05:41.365579257Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T10:05:41.639 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 10:05:41 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=cleanup t=2024-09-21T10:05:41.368163011Z level=info msg="Completed cleanup jobs" duration=13.131466ms 2024-09-21T10:05:41.686 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:41.723 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:41.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:41.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:41.780 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:41.824 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:41.865 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:41.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:41.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:41.966 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:41.998 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:41.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:41.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:42.100 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:42.132 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:42.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:42.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:42.234 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:42.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:05:42.269 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:42.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:42.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:42.335 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:05:42.336 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:05:42.336 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-21T10:05:42.336 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-21T10:05:42.336 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:05:42.370 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:42.422 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:42.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:42.511 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:05:42.522 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:43.061 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:42 smithi196 ceph-mon[84784]: pgmap v316: 89 pgs: 89 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 863 op/s 2024-09-21T10:05:43.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:42 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/62992054' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:05:43.062 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:42 smithi098 ceph-mon[97580]: pgmap v316: 89 pgs: 89 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 863 op/s 2024-09-21T10:05:43.062 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:42 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/62992054' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:05:43.062 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:42 smithi098 ceph-mon[100914]: pgmap v316: 89 pgs: 89 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 863 op/s 2024-09-21T10:05:43.062 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:42 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/62992054' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:05:43.100 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:43.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:43.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:43.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:43.233 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:43.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:43.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:43.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:43.361 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:43.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:43.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:43.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:43.496 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:43.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:43.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:43.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:43.629 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:43.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:43.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:43.731 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:43.763 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:43.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:43.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:43.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:43 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:05:43.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:43 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/62992054' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:05:43.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:43 smithi098 ceph-mon[97580]: osdmap e124: 8 total, 8 up, 8 in 2024-09-21T10:05:43.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:43 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:05:43.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:43 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/62992054' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:05:43.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:43 smithi098 ceph-mon[100914]: osdmap e124: 8 total, 8 up, 8 in 2024-09-21T10:05:43.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:43.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:43 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:05:43.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:43 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/62992054' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:05:43.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:43 smithi196 ceph-mon[84784]: osdmap e124: 8 total, 8 up, 8 in 2024-09-21T10:05:43.894 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:43.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:43.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:43.996 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:44.033 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:05:44.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:44.035 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:44.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:44.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:44.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:44.174 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:44.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:44.275 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:44.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:44.325 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:44.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:44.426 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:44.456 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:44.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:44.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:44.558 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:44.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:44.595 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:44.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:44.696 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:44.729 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:44.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:44.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:44.830 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:44.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:44 smithi098 ceph-mon[100914]: pgmap v318: 89 pgs: 89 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 855 op/s 2024-09-21T10:05:44.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:44 smithi098 ceph-mon[97580]: pgmap v318: 89 pgs: 89 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 855 op/s 2024-09-21T10:05:44.863 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:44.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:44.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:44.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:44 smithi196 ceph-mon[84784]: pgmap v318: 89 pgs: 89 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 855 op/s 2024-09-21T10:05:44.943 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:44.968 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:45.002 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:45.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:45.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:45.104 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:45.138 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:45.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:45.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:45.240 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:45.278 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:45.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:45.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:45.380 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:45.416 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:45.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:45.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:45.455 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:05:45.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:45.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:45.572 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:45.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:45.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:45.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:45.709 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:45.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:45.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:45.842 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:45.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:45.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:45.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:45.969 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:45.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:45.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:46.071 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:46.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:46.106 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:46.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:46.153 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:46.207 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:46.245 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:46.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:46.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:46.346 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:46.375 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:46.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:46.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:46.477 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:46.498 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:46.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:46.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:46.525 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T10:05:46.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:46.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:46.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:46.732 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:46.768 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:46.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:46.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:46.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:46 smithi098 ceph-mon[97580]: pgmap v319: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 865 op/s 2024-09-21T10:05:46.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:46 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3450943036' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:05:46.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:46 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:05:46.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:46 smithi098 ceph-mon[100914]: pgmap v319: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 865 op/s 2024-09-21T10:05:46.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:46 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3450943036' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:05:46.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:46 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:05:46.870 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:46.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:46 smithi196 ceph-mon[84784]: pgmap v319: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 865 op/s 2024-09-21T10:05:46.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:46 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3450943036' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:05:46.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:46 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:05:46.902 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:46.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:46.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:47.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:47.034 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:47.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:47.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:47.136 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:47.177 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:47.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:47.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:47.279 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:47.314 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:47.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:47.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:47.417 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:47.443 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:47.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:47.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:47.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:47.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:47.587 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:47.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:47.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:47 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:05:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:47 smithi196 ceph-mon[84784]: osdmap e125: 8 total, 8 up, 8 in 2024-09-21T10:05:47.689 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:47.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:47.723 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:47.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:47.822 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:47 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:05:47.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:47 smithi098 ceph-mon[97580]: osdmap e125: 8 total, 8 up, 8 in 2024-09-21T10:05:47.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:47 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:05:47.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:47 smithi098 ceph-mon[100914]: osdmap e125: 8 total, 8 up, 8 in 2024-09-21T10:05:47.825 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:47.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:47.855 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:47.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:47.873 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T10:05:47.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:05:47.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:47.992 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:47.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:47.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:48.094 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:48.128 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:48.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:48.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:48.230 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:48.264 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:48.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:48.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:48.366 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:48.397 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:48.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:48.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:48.499 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:48.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:48.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:48.636 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:48.663 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:48.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:48.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:48.765 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:48.803 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:48.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:48.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:48.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:48 smithi098 ceph-mon[97580]: pgmap v321: 89 pgs: 89 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 756 op/s 2024-09-21T10:05:48.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:48 smithi098 ceph-mon[100914]: pgmap v321: 89 pgs: 89 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 756 op/s 2024-09-21T10:05:48.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:48 smithi196 ceph-mon[84784]: pgmap v321: 89 pgs: 89 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 756 op/s 2024-09-21T10:05:48.905 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:48.939 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:48.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:48.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:49.041 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:49.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:49.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:49.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:49.079 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:49.079 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:05:49.080 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:05:49.123 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:05:49.123 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:05:49.145 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:05:49.145 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:05:49.172 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:05:49.172 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:05:49.173 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:49.200 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:49.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:49.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:49.204 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:05:49.205 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:05:49.234 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:05:49.234 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:05:49.268 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:05:49.269 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:05:49.297 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:05:49.297 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:05:49.302 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:49.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:49.339 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:49.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:49.440 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:49.472 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:49.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:49.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:49.573 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:49.604 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:49.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:49.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:49.706 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:49.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:49.751 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:49.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:49.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:49 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2213118190' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:49.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:49 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2213118190' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:49.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:49.880 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:49.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:49 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2213118190' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:49.914 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:49.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:49.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:50.016 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:50.049 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:50.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:50.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:50.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:50.183 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:50.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:50.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:50.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:50.325 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:50.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:50.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:50.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:50.460 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:50.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:50.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:50.554 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:05:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:05:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:50.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:50.588 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:50.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:50.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:50.690 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:50.727 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:50.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:50.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:50.748 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:05:50.748 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:05:50.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:50 smithi098 ceph-mon[97580]: pgmap v322: 89 pgs: 89 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 949 op/s 2024-09-21T10:05:50.748 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:05:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:05:50] "GET /metrics HTTP/1.1" 200 36458 "" "Prometheus/2.51.0" 2024-09-21T10:05:50.749 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:50 smithi098 ceph-mon[100914]: pgmap v322: 89 pgs: 89 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 949 op/s 2024-09-21T10:05:50.829 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:50.862 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:50.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:50.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:50 smithi196 ceph-mon[84784]: pgmap v322: 89 pgs: 89 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 949 op/s 2024-09-21T10:05:50.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:50.991 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:50.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:50.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:51.093 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:51.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:51.132 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:51.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:51.233 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:51.262 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:51.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:51.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:51.364 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:51.391 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:51.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:51.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:51.493 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:51.518 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:51.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:51.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:51.620 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:51.655 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:51.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:51.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:51.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:51.788 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:51.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:51.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:51.890 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:51.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:51.931 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:51.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:52.031 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:52.063 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:52.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:52.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:52.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:52.202 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:52.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:52.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:52.304 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:52.338 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:52.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:52.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:52.360 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:05:52.441 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:52.472 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:52.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:52.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:52.574 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:52.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:52 smithi098 ceph-mon[97580]: pgmap v323: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 945 op/s 2024-09-21T10:05:52.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:05:52.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:52.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:52 smithi098 ceph-mon[100914]: pgmap v323: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 945 op/s 2024-09-21T10:05:52.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:05:52.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:52.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:52.609 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:52.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:52.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:52.746 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:52.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:52.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:52.824 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:05:52.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:52.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:52.876 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:52.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:52 smithi196 ceph-mon[84784]: pgmap v323: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 945 op/s 2024-09-21T10:05:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:05:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:52.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:53.010 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:53.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:53.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:53.112 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:53.144 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:53.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:53.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:53.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:53.276 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:53.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:53.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:53.378 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:53.428 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:53.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:53.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:53.530 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:53.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:53.567 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:53.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:53.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:05:53.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:53.712 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:53.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:53.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:53.814 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:53.847 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:53.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:53.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:53.949 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:54.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:54.003 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:54.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:54.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:54.139 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:54.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:54.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:54.241 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:54.270 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:54.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:54.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:54.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:54.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:54.411 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:54.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:54.512 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:54.549 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:54.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:54.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:54.651 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:54.686 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:54.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:54.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:54.717 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:54 smithi196 ceph-mon[84784]: pgmap v324: 89 pgs: 89 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 899 op/s 2024-09-21T10:05:54.725 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:54 smithi098 ceph-mon[100914]: pgmap v324: 89 pgs: 89 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 899 op/s 2024-09-21T10:05:54.726 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:54 smithi098 ceph-mon[97580]: pgmap v324: 89 pgs: 89 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 899 op/s 2024-09-21T10:05:54.788 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:54.860 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:54.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:54.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:54.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:54.976 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:55.011 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:55.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:55.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:55.113 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:55.119 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:05:55.124 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:05:55.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:55.167 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:55.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:55.268 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:55.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:55.313 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:55.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:55.382 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:05:55.382 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:05:55.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:55.444 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:55.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:55.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:55.545 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:55.579 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:05:55.581 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:55.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:55.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:55.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:55 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1039698478' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:05:55.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:55.713 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:55.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:55.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:55.815 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:55.832 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:55 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1039698478' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:05:55.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:55 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1039698478' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:05:55.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:55.850 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:55.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:55.951 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:55.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:55.987 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:55.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:56.081 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-09-21T10:05:55.075126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6918051813,"num_objects":105517,"num_object_clones":0,"num_object_copies":316551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105517,"num_whiteouts":0,"num_read":6290,"num_read_kb":13766,"num_write":166839,"num_write_kb":8694892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147015,"ondisk_log_size":147015,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22013912,"kb_used_data":20426904,"kb_used_omap":35,"kb_used_meta":1586652,"kb_avail":727947304,"statfs":{"total":767960285184,"available":745418039296,"internally_reserved":0,"allocated":20917149696,"data_stored":20912089102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36353,"internal_metadata":1624732159},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-580714496,"num_objects":-8861,"num_object_clones":0,"num_object_copies":-26583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8861,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002530"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":116,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.592720+0000","last_change":"2024-09-21T10:02:14.097991+0000","last_active":"2024-09-21T10:05:46.592720+0000","last_peered":"2024-09-21T10:05:46.592720+0000","last_clean":"2024-09-21T10:05:46.592720+0000","last_became_active":"2024-09-21T10:02:14.097407+0000","last_became_peered":"2024-09-21T10:02:14.097407+0000","last_unstale":"2024-09-21T10:05:46.592720+0000","last_undegraded":"2024-09-21T10:05:46.592720+0000","last_fullsized":"2024-09-21T10:05:46.592720+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:18:42.098886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":92,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.513167+0000","last_change":"2024-09-21T10:01:41.360784+0000","last_active":"2024-09-21T10:05:42.513167+0000","last_peered":"2024-09-21T10:05:42.513167+0000","last_clean":"2024-09-21T10:05:42.513167+0000","last_became_active":"2024-09-21T10:01:41.360462+0000","last_became_peered":"2024-09-21T10:01:41.360462+0000","last_unstale":"2024-09-21T10:05:42.513167+0000","last_undegraded":"2024-09-21T10:05:42.513167+0000","last_fullsized":"2024-09-21T10:05:42.513167+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:11.867275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":88,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.518346+0000","last_change":"2024-09-21T10:02:14.103373+0000","last_active":"2024-09-21T10:05:42.518346+0000","last_peered":"2024-09-21T10:05:42.518346+0000","last_clean":"2024-09-21T10:05:42.518346+0000","last_became_active":"2024-09-21T10:02:14.102985+0000","last_became_peered":"2024-09-21T10:02:14.102985+0000","last_unstale":"2024-09-21T10:05:42.518346+0000","last_undegraded":"2024-09-21T10:05:42.518346+0000","last_fullsized":"2024-09-21T10:05:42.518346+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:03.418174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":112,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.577884+0000","last_change":"2024-09-21T10:02:14.098949+0000","last_active":"2024-09-21T10:05:46.577884+0000","last_peered":"2024-09-21T10:05:46.577884+0000","last_clean":"2024-09-21T10:05:46.577884+0000","last_became_active":"2024-09-21T10:02:14.098646+0000","last_became_peered":"2024-09-21T10:02:14.098646+0000","last_unstale":"2024-09-21T10:05:46.577884+0000","last_undegraded":"2024-09-21T10:05:46.577884+0000","last_fullsized":"2024-09-21T10:05:46.577884+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:50:56.862529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":112,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.578029+0000","last_change":"2024-09-21T10:02:14.098205+0000","last_active":"2024-09-21T10:05:46.578029+0000","last_peered":"2024-09-21T10:05:46.578029+0000","last_clean":"2024-09-21T10:05:46.578029+0000","last_became_active":"2024-09-21T10:02:14.097911+0000","last_became_peered":"2024-09-21T10:02:14.097911+0000","last_unstale":"2024-09-21T10:05:46.578029+0000","last_undegraded":"2024-09-21T10:05:46.578029+0000","last_fullsized":"2024-09-21T10:05:46.578029+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:55.971191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":73,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.514653+0000","last_change":"2024-09-21T09:52:20.371775+0000","last_active":"2024-09-21T10:05:42.514653+0000","last_peered":"2024-09-21T10:05:42.514653+0000","last_clean":"2024-09-21T10:05:42.514653+0000","last_became_active":"2024-09-21T09:52:20.371509+0000","last_became_peered":"2024-09-21T09:52:20.371509+0000","last_unstale":"2024-09-21T10:05:42.514653+0000","last_undegraded":"2024-09-21T10:05:42.514653+0000","last_fullsized":"2024-09-21T10:05:42.514653+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:43:21.822524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":116,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.592584+0000","last_change":"2024-09-21T10:02:14.097817+0000","last_active":"2024-09-21T10:05:46.592584+0000","last_peered":"2024-09-21T10:05:46.592584+0000","last_clean":"2024-09-21T10:05:46.592584+0000","last_became_active":"2024-09-21T10:02:14.097466+0000","last_became_peered":"2024-09-21T10:02:14.097466+0000","last_unstale":"2024-09-21T10:05:46.592584+0000","last_undegraded":"2024-09-21T10:05:46.592584+0000","last_fullsized":"2024-09-21T10:05:46.592584+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:57.883118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":98,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.518978+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:05:42.518978+0000","last_peered":"2024-09-21T10:05:42.518978+0000","last_clean":"2024-09-21T10:05:42.518978+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:05:42.518978+0000","last_undegraded":"2024-09-21T10:05:42.518978+0000","last_fullsized":"2024-09-21T10:05:42.518978+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"125'10313","reported_seq":10341,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:53.004746+0000","last_change":"2024-09-21T10:03:41.940189+0000","last_active":"2024-09-21T10:05:53.004746+0000","last_peered":"2024-09-21T10:05:53.004746+0000","last_clean":"2024-09-21T10:05:53.004746+0000","last_became_active":"2024-09-21T10:03:41.939900+0000","last_became_peered":"2024-09-21T10:03:41.939900+0000","last_unstale":"2024-09-21T10:05:53.004746+0000","last_undegraded":"2024-09-21T10:05:53.004746+0000","last_fullsized":"2024-09-21T10:05:53.004746+0000","mapping_epoch":105,"log_start":"109'1700","ondisk_log_start":"109'1700","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":8613,"log_dups_size":0,"ondisk_log_size":8613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:13:00.441130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429981696,"num_objects":6561,"num_object_clones":0,"num_object_copies":19683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10313,"num_write_kb":539968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":91,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.513661+0000","last_change":"2024-09-21T09:59:54.265455+0000","last_active":"2024-09-21T10:05:42.513661+0000","last_peered":"2024-09-21T10:05:42.513661+0000","last_clean":"2024-09-21T10:05:42.513661+0000","last_became_active":"2024-09-21T09:59:54.265110+0000","last_became_peered":"2024-09-21T09:59:54.265110+0000","last_unstale":"2024-09-21T10:05:42.513661+0000","last_undegraded":"2024-09-21T10:05:42.513661+0000","last_fullsized":"2024-09-21T10:05:42.513661+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:04:13.404874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":71,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.516925+0000","last_change":"2024-09-21T09:52:20.402753+0000","last_active":"2024-09-21T10:05:42.516925+0000","last_peered":"2024-09-21T10:05:42.516925+0000","last_clean":"2024-09-21T10:05:42.516925+0000","last_became_active":"2024-09-21T09:52:20.402462+0000","last_became_peered":"2024-09-21T09:52:20.402462+0000","last_unstale":"2024-09-21T10:05:42.516925+0000","last_undegraded":"2024-09-21T10:05:42.516925+0000","last_fullsized":"2024-09-21T10:05:42.516925+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:14.036869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"125'10354","reported_seq":10405,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:54.244925+0000","last_change":"2024-09-21T10:03:42.340396+0000","last_active":"2024-09-21T10:05:54.244925+0000","last_peered":"2024-09-21T10:05:54.244925+0000","last_clean":"2024-09-21T10:05:54.244925+0000","last_became_active":"2024-09-21T10:03:42.340086+0000","last_became_peered":"2024-09-21T10:03:42.340086+0000","last_unstale":"2024-09-21T10:05:54.244925+0000","last_undegraded":"2024-09-21T10:05:54.244925+0000","last_fullsized":"2024-09-21T10:05:54.244925+0000","mapping_epoch":105,"log_start":"109'300","ondisk_log_start":"109'300","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:35:46.959831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429916160,"num_objects":6560,"num_object_clones":0,"num_object_copies":19680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10354,"num_write_kb":541248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":92,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.515577+0000","last_change":"2024-09-21T10:01:07.420484+0000","last_active":"2024-09-21T10:05:42.515577+0000","last_peered":"2024-09-21T10:05:42.515577+0000","last_clean":"2024-09-21T10:05:42.515577+0000","last_became_active":"2024-09-21T10:01:07.420135+0000","last_became_peered":"2024-09-21T10:01:07.420135+0000","last_unstale":"2024-09-21T10:05:42.515577+0000","last_undegraded":"2024-09-21T10:05:42.515577+0000","last_fullsized":"2024-09-21T10:05:42.515577+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:33:10.241010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":106,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.516176+0000","last_change":"2024-09-21T10:02:14.098139+0000","last_active":"2024-09-21T10:05:42.516176+0000","last_peered":"2024-09-21T10:05:42.516176+0000","last_clean":"2024-09-21T10:05:42.516176+0000","last_became_active":"2024-09-21T10:02:14.097639+0000","last_became_peered":"2024-09-21T10:02:14.097639+0000","last_unstale":"2024-09-21T10:05:42.516176+0000","last_undegraded":"2024-09-21T10:05:42.516176+0000","last_fullsized":"2024-09-21T10:05:42.516176+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:11:26.925388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"125'10290","reported_seq":10340,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:54.132780+0000","last_change":"2024-09-21T10:03:41.941408+0000","last_active":"2024-09-21T10:05:54.132780+0000","last_peered":"2024-09-21T10:05:54.132780+0000","last_clean":"2024-09-21T10:05:54.132780+0000","last_became_active":"2024-09-21T10:03:41.941084+0000","last_became_peered":"2024-09-21T10:03:41.941084+0000","last_unstale":"2024-09-21T10:05:54.132780+0000","last_undegraded":"2024-09-21T10:05:54.132780+0000","last_fullsized":"2024-09-21T10:05:54.132780+0000","mapping_epoch":105,"log_start":"109'200","ondisk_log_start":"109'200","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:33:47.243861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426508288,"num_objects":6508,"num_object_clones":0,"num_object_copies":19524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10290,"num_write_kb":537536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":74,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.515716+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:05:42.515716+0000","last_peered":"2024-09-21T10:05:42.515716+0000","last_clean":"2024-09-21T10:05:42.515716+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:05:42.515716+0000","last_undegraded":"2024-09-21T10:05:42.515716+0000","last_fullsized":"2024-09-21T10:05:42.515716+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":89,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.523473+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:05:42.523473+0000","last_peered":"2024-09-21T10:05:42.523473+0000","last_clean":"2024-09-21T10:05:42.523473+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:05:42.523473+0000","last_undegraded":"2024-09-21T10:05:42.523473+0000","last_fullsized":"2024-09-21T10:05:42.523473+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"125'10410","reported_seq":10461,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:54.248079+0000","last_change":"2024-09-21T10:03:41.939951+0000","last_active":"2024-09-21T10:05:54.248079+0000","last_peered":"2024-09-21T10:05:54.248079+0000","last_clean":"2024-09-21T10:05:54.248079+0000","last_became_active":"2024-09-21T10:03:41.939644+0000","last_became_peered":"2024-09-21T10:03:41.939644+0000","last_unstale":"2024-09-21T10:05:54.248079+0000","last_undegraded":"2024-09-21T10:05:54.248079+0000","last_fullsized":"2024-09-21T10:05:54.248079+0000","mapping_epoch":105,"log_start":"109'400","ondisk_log_start":"109'400","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:28:55.215930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423624704,"num_objects":6464,"num_object_clones":0,"num_object_copies":19392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10410,"num_write_kb":539968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":89,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.517546+0000","last_change":"2024-09-21T10:02:14.104212+0000","last_active":"2024-09-21T10:05:42.517546+0000","last_peered":"2024-09-21T10:05:42.517546+0000","last_clean":"2024-09-21T10:05:42.517546+0000","last_became_active":"2024-09-21T10:02:14.103893+0000","last_became_peered":"2024-09-21T10:02:14.103893+0000","last_unstale":"2024-09-21T10:05:42.517546+0000","last_undegraded":"2024-09-21T10:05:42.517546+0000","last_fullsized":"2024-09-21T10:05:42.517546+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:59:21.191412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":112,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.578362+0000","last_change":"2024-09-21T10:01:07.419715+0000","last_active":"2024-09-21T10:05:46.578362+0000","last_peered":"2024-09-21T10:05:46.578362+0000","last_clean":"2024-09-21T10:05:46.578362+0000","last_became_active":"2024-09-21T10:01:07.419422+0000","last_became_peered":"2024-09-21T10:01:07.419422+0000","last_unstale":"2024-09-21T10:05:46.578362+0000","last_undegraded":"2024-09-21T10:05:46.578362+0000","last_fullsized":"2024-09-21T10:05:46.578362+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:36.424289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"125'10329","reported_seq":10380,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:54.216613+0000","last_change":"2024-09-21T10:03:41.942958+0000","last_active":"2024-09-21T10:05:54.216613+0000","last_peered":"2024-09-21T10:05:54.216613+0000","last_clean":"2024-09-21T10:05:54.216613+0000","last_became_active":"2024-09-21T10:03:41.942656+0000","last_became_peered":"2024-09-21T10:03:41.942656+0000","last_unstale":"2024-09-21T10:05:54.216613+0000","last_undegraded":"2024-09-21T10:05:54.216613+0000","last_fullsized":"2024-09-21T10:05:54.216613+0000","mapping_epoch":105,"log_start":"109'300","ondisk_log_start":"109'300","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:36:35.630043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427687960,"num_objects":6527,"num_object_clones":0,"num_object_copies":19581,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6527,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10329,"num_write_kb":539329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":114,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.592605+0000","last_change":"2024-09-21T10:02:14.096537+0000","last_active":"2024-09-21T10:05:46.592605+0000","last_peered":"2024-09-21T10:05:46.592605+0000","last_clean":"2024-09-21T10:05:46.592605+0000","last_became_active":"2024-09-21T10:02:14.096166+0000","last_became_peered":"2024-09-21T10:02:14.096166+0000","last_unstale":"2024-09-21T10:05:46.592605+0000","last_undegraded":"2024-09-21T10:05:46.592605+0000","last_fullsized":"2024-09-21T10:05:46.592605+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:54:18.927732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":89,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.514209+0000","last_change":"2024-09-21T10:02:14.099205+0000","last_active":"2024-09-21T10:05:42.514209+0000","last_peered":"2024-09-21T10:05:42.514209+0000","last_clean":"2024-09-21T10:05:42.514209+0000","last_became_active":"2024-09-21T10:02:14.098598+0000","last_became_peered":"2024-09-21T10:02:14.098598+0000","last_unstale":"2024-09-21T10:05:42.514209+0000","last_undegraded":"2024-09-21T10:05:42.514209+0000","last_fullsized":"2024-09-21T10:05:42.514209+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:46.424612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"125'10216","reported_seq":10244,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:53.083091+0000","last_change":"2024-09-21T10:03:42.340737+0000","last_active":"2024-09-21T10:05:53.083091+0000","last_peered":"2024-09-21T10:05:53.083091+0000","last_clean":"2024-09-21T10:05:53.083091+0000","last_became_active":"2024-09-21T10:03:42.340445+0000","last_became_peered":"2024-09-21T10:03:42.340445+0000","last_unstale":"2024-09-21T10:05:53.083091+0000","last_undegraded":"2024-09-21T10:05:53.083091+0000","last_fullsized":"2024-09-21T10:05:53.083091+0000","mapping_epoch":105,"log_start":"110'2800","ondisk_log_start":"110'2800","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":7416,"log_dups_size":0,"ondisk_log_size":7416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:18.067171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430047232,"num_objects":6562,"num_object_clones":0,"num_object_copies":19686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10216,"num_write_kb":536896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":90,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.523599+0000","last_change":"2024-09-21T10:01:07.424883+0000","last_active":"2024-09-21T10:05:42.523599+0000","last_peered":"2024-09-21T10:05:42.523599+0000","last_clean":"2024-09-21T10:05:42.523599+0000","last_became_active":"2024-09-21T10:01:07.424499+0000","last_became_peered":"2024-09-21T10:01:07.424499+0000","last_unstale":"2024-09-21T10:05:42.523599+0000","last_undegraded":"2024-09-21T10:05:42.523599+0000","last_fullsized":"2024-09-21T10:05:42.523599+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:46.101570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":106,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.513422+0000","last_change":"2024-09-21T10:01:41.362291+0000","last_active":"2024-09-21T10:05:42.513422+0000","last_peered":"2024-09-21T10:05:42.513422+0000","last_clean":"2024-09-21T10:05:42.513422+0000","last_became_active":"2024-09-21T10:01:41.361954+0000","last_became_peered":"2024-09-21T10:01:41.361954+0000","last_unstale":"2024-09-21T10:05:42.513422+0000","last_undegraded":"2024-09-21T10:05:42.513422+0000","last_fullsized":"2024-09-21T10:05:42.513422+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:57.669671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"125'10415","reported_seq":10442,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:53.085096+0000","last_change":"2024-09-21T10:03:41.942110+0000","last_active":"2024-09-21T10:05:53.085096+0000","last_peered":"2024-09-21T10:05:53.085096+0000","last_clean":"2024-09-21T10:05:53.085096+0000","last_became_active":"2024-09-21T10:03:41.941789+0000","last_became_peered":"2024-09-21T10:03:41.941789+0000","last_unstale":"2024-09-21T10:05:53.085096+0000","last_undegraded":"2024-09-21T10:05:53.085096+0000","last_fullsized":"2024-09-21T10:05:53.085096+0000","mapping_epoch":105,"log_start":"109'400","ondisk_log_start":"109'400","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:33:04.384003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434307072,"num_objects":6627,"num_object_clones":0,"num_object_copies":19881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10415,"num_write_kb":545344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":90,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.516278+0000","last_change":"2024-09-21T10:01:41.361244+0000","last_active":"2024-09-21T10:05:42.516278+0000","last_peered":"2024-09-21T10:05:42.516278+0000","last_clean":"2024-09-21T10:05:42.516278+0000","last_became_active":"2024-09-21T10:01:41.360645+0000","last_became_peered":"2024-09-21T10:01:41.360645+0000","last_unstale":"2024-09-21T10:05:42.516278+0000","last_undegraded":"2024-09-21T10:05:42.516278+0000","last_fullsized":"2024-09-21T10:05:42.516278+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:07.604991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":89,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.513675+0000","last_change":"2024-09-21T10:01:07.420824+0000","last_active":"2024-09-21T10:05:42.513675+0000","last_peered":"2024-09-21T10:05:42.513675+0000","last_clean":"2024-09-21T10:05:42.513675+0000","last_became_active":"2024-09-21T10:01:07.420387+0000","last_became_peered":"2024-09-21T10:01:07.420387+0000","last_unstale":"2024-09-21T10:05:42.513675+0000","last_undegraded":"2024-09-21T10:05:42.513675+0000","last_fullsized":"2024-09-21T10:05:42.513675+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:48.875842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"125'10542","reported_seq":10570,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:53.094410+0000","last_change":"2024-09-21T10:03:41.941262+0000","last_active":"2024-09-21T10:05:53.094410+0000","last_peered":"2024-09-21T10:05:53.094410+0000","last_clean":"2024-09-21T10:05:53.094410+0000","last_became_active":"2024-09-21T10:03:41.940968+0000","last_became_peered":"2024-09-21T10:03:41.940968+0000","last_unstale":"2024-09-21T10:05:53.094410+0000","last_undegraded":"2024-09-21T10:05:53.094410+0000","last_fullsized":"2024-09-21T10:05:53.094410+0000","mapping_epoch":105,"log_start":"110'3200","ondisk_log_start":"110'3200","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":7342,"log_dups_size":0,"ondisk_log_size":7342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:47.483647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439877632,"num_objects":6712,"num_object_clones":0,"num_object_copies":20136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10542,"num_write_kb":552128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":117,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.592556+0000","last_change":"2024-09-21T10:02:14.097462+0000","last_active":"2024-09-21T10:05:46.592556+0000","last_peered":"2024-09-21T10:05:46.592556+0000","last_clean":"2024-09-21T10:05:46.592556+0000","last_became_active":"2024-09-21T10:02:14.097167+0000","last_became_peered":"2024-09-21T10:02:14.097167+0000","last_unstale":"2024-09-21T10:05:46.592556+0000","last_undegraded":"2024-09-21T10:05:46.592556+0000","last_fullsized":"2024-09-21T10:05:46.592556+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:54.922731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":102,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.581283+0000","last_change":"2024-09-21T10:02:14.097676+0000","last_active":"2024-09-21T10:05:46.581283+0000","last_peered":"2024-09-21T10:05:46.581283+0000","last_clean":"2024-09-21T10:05:46.581283+0000","last_became_active":"2024-09-21T10:02:14.097391+0000","last_became_peered":"2024-09-21T10:02:14.097391+0000","last_unstale":"2024-09-21T10:05:46.581283+0000","last_undegraded":"2024-09-21T10:05:46.581283+0000","last_fullsized":"2024-09-21T10:05:46.581283+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:40:24.523178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"125'10576","reported_seq":10604,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:52.957768+0000","last_change":"2024-09-21T10:03:41.940655+0000","last_active":"2024-09-21T10:05:52.957768+0000","last_peered":"2024-09-21T10:05:52.957768+0000","last_clean":"2024-09-21T10:05:52.957768+0000","last_became_active":"2024-09-21T10:03:41.940058+0000","last_became_peered":"2024-09-21T10:03:41.940058+0000","last_unstale":"2024-09-21T10:05:52.957768+0000","last_undegraded":"2024-09-21T10:05:52.957768+0000","last_fullsized":"2024-09-21T10:05:52.957768+0000","mapping_epoch":105,"log_start":"110'2000","ondisk_log_start":"110'2000","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":8576,"log_dups_size":0,"ondisk_log_size":8576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:00:40.090535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448790528,"num_objects":6848,"num_object_clones":0,"num_object_copies":20544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10576,"num_write_kb":557568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":207,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.512888+0000","last_change":"2024-09-21T09:59:54.265556+0000","last_active":"2024-09-21T10:05:42.512888+0000","last_peered":"2024-09-21T10:05:42.512888+0000","last_clean":"2024-09-21T10:05:42.512888+0000","last_became_active":"2024-09-21T09:59:54.265033+0000","last_became_peered":"2024-09-21T09:59:54.265033+0000","last_unstale":"2024-09-21T10:05:42.512888+0000","last_undegraded":"2024-09-21T10:05:42.512888+0000","last_fullsized":"2024-09-21T10:05:42.512888+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T09:48:34.850872+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:40.449756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":513,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.517496+0000","last_change":"2024-09-21T10:01:41.361089+0000","last_active":"2024-09-21T10:05:42.517496+0000","last_peered":"2024-09-21T10:05:42.517496+0000","last_clean":"2024-09-21T10:05:42.517496+0000","last_became_active":"2024-09-21T10:01:41.360658+0000","last_became_peered":"2024-09-21T10:01:41.360658+0000","last_unstale":"2024-09-21T10:05:42.517496+0000","last_undegraded":"2024-09-21T10:05:42.517496+0000","last_fullsized":"2024-09-21T10:05:42.517496+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:58:06.840024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":482,"num_read_kb":1581,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":70,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.592451+0000","last_change":"2024-09-21T10:02:14.099961+0000","last_active":"2024-09-21T10:05:46.592451+0000","last_peered":"2024-09-21T10:05:46.592451+0000","last_clean":"2024-09-21T10:05:46.592451+0000","last_became_active":"2024-09-21T10:02:14.099067+0000","last_became_peered":"2024-09-21T10:02:14.099067+0000","last_unstale":"2024-09-21T10:05:46.592451+0000","last_undegraded":"2024-09-21T10:05:46.592451+0000","last_fullsized":"2024-09-21T10:05:46.592451+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:41.037532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":72,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.519335+0000","last_change":"2024-09-21T09:52:20.403534+0000","last_active":"2024-09-21T10:05:42.519335+0000","last_peered":"2024-09-21T10:05:42.519335+0000","last_clean":"2024-09-21T10:05:42.519335+0000","last_became_active":"2024-09-21T09:52:20.403258+0000","last_became_peered":"2024-09-21T09:52:20.403258+0000","last_unstale":"2024-09-21T10:05:42.519335+0000","last_undegraded":"2024-09-21T10:05:42.519335+0000","last_fullsized":"2024-09-21T10:05:42.519335+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:55.374343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"125'10272","reported_seq":10299,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:53.108465+0000","last_change":"2024-09-21T10:03:42.340526+0000","last_active":"2024-09-21T10:05:53.108465+0000","last_peered":"2024-09-21T10:05:53.108465+0000","last_clean":"2024-09-21T10:05:53.108465+0000","last_became_active":"2024-09-21T10:03:42.340207+0000","last_became_peered":"2024-09-21T10:03:42.340207+0000","last_unstale":"2024-09-21T10:05:53.108465+0000","last_undegraded":"2024-09-21T10:05:53.108465+0000","last_fullsized":"2024-09-21T10:05:53.108465+0000","mapping_epoch":105,"log_start":"109'200","ondisk_log_start":"109'200","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:02.279418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432930816,"num_objects":6606,"num_object_clones":0,"num_object_copies":19818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10272,"num_write_kb":540096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":470,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.512790+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:05:42.512790+0000","last_peered":"2024-09-21T10:05:42.512790+0000","last_clean":"2024-09-21T10:05:42.512790+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:05:42.512790+0000","last_undegraded":"2024-09-21T10:05:42.512790+0000","last_fullsized":"2024-09-21T10:05:42.512790+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":112,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.577812+0000","last_change":"2024-09-21T10:02:14.097097+0000","last_active":"2024-09-21T10:05:46.577812+0000","last_peered":"2024-09-21T10:05:46.577812+0000","last_clean":"2024-09-21T10:05:46.577812+0000","last_became_active":"2024-09-21T10:02:14.096788+0000","last_became_peered":"2024-09-21T10:02:14.096788+0000","last_unstale":"2024-09-21T10:05:46.577812+0000","last_undegraded":"2024-09-21T10:05:46.577812+0000","last_fullsized":"2024-09-21T10:05:46.577812+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:05:29.711020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":123,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.578115+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:05:46.578115+0000","last_peered":"2024-09-21T10:05:46.578115+0000","last_clean":"2024-09-21T10:05:46.578115+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:05:46.578115+0000","last_undegraded":"2024-09-21T10:05:46.578115+0000","last_fullsized":"2024-09-21T10:05:46.578115+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"125'10393","reported_seq":10444,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:54.240420+0000","last_change":"2024-09-21T10:03:41.943228+0000","last_active":"2024-09-21T10:05:54.240420+0000","last_peered":"2024-09-21T10:05:54.240420+0000","last_clean":"2024-09-21T10:05:54.240420+0000","last_became_active":"2024-09-21T10:03:41.942690+0000","last_became_peered":"2024-09-21T10:03:41.942690+0000","last_unstale":"2024-09-21T10:05:54.240420+0000","last_undegraded":"2024-09-21T10:05:54.240420+0000","last_fullsized":"2024-09-21T10:05:54.240420+0000","mapping_epoch":105,"log_start":"109'300","ondisk_log_start":"109'300","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:32:01.612946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422903808,"num_objects":6453,"num_object_clones":0,"num_object_copies":19359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10393,"num_write_kb":539072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":862,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.578816+0000","last_change":"2024-09-21T10:02:14.100143+0000","last_active":"2024-09-21T10:05:46.578816+0000","last_peered":"2024-09-21T10:05:46.578816+0000","last_clean":"2024-09-21T10:05:46.578816+0000","last_became_active":"2024-09-21T10:02:14.099586+0000","last_became_peered":"2024-09-21T10:02:14.099586+0000","last_unstale":"2024-09-21T10:05:46.578816+0000","last_undegraded":"2024-09-21T10:05:46.578816+0000","last_fullsized":"2024-09-21T10:05:46.578816+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:34.064652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":89,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.578658+0000","last_change":"2024-09-21T10:03:39.670431+0000","last_active":"2024-09-21T10:05:46.578658+0000","last_peered":"2024-09-21T10:05:46.578658+0000","last_clean":"2024-09-21T10:05:46.578658+0000","last_became_active":"2024-09-21T10:03:39.670088+0000","last_became_peered":"2024-09-21T10:03:39.670088+0000","last_unstale":"2024-09-21T10:05:46.578658+0000","last_undegraded":"2024-09-21T10:05:46.578658+0000","last_fullsized":"2024-09-21T10:05:46.578658+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:30:52.512603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":118,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.579007+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:05:46.579007+0000","last_peered":"2024-09-21T10:05:46.579007+0000","last_clean":"2024-09-21T10:05:46.579007+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:05:46.579007+0000","last_undegraded":"2024-09-21T10:05:46.579007+0000","last_fullsized":"2024-09-21T10:05:46.579007+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"125'10557","reported_seq":10609,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:52.036916+0000","last_change":"2024-09-21T10:03:42.341388+0000","last_active":"2024-09-21T10:05:52.036916+0000","last_peered":"2024-09-21T10:05:52.036916+0000","last_clean":"2024-09-21T10:05:52.036916+0000","last_became_active":"2024-09-21T10:03:42.340274+0000","last_became_peered":"2024-09-21T10:03:42.340274+0000","last_unstale":"2024-09-21T10:05:52.036916+0000","last_undegraded":"2024-09-21T10:05:52.036916+0000","last_fullsized":"2024-09-21T10:05:52.036916+0000","mapping_epoch":105,"log_start":"109'1700","ondisk_log_start":"109'1700","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":8857,"log_dups_size":0,"ondisk_log_size":8857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:47.666420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438108160,"num_objects":6685,"num_object_clones":0,"num_object_copies":20055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10557,"num_write_kb":551744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":697,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.517230+0000","last_change":"2024-09-21T10:01:07.424819+0000","last_active":"2024-09-21T10:05:42.517230+0000","last_peered":"2024-09-21T10:05:42.517230+0000","last_clean":"2024-09-21T10:05:42.517230+0000","last_became_active":"2024-09-21T10:01:07.424392+0000","last_became_peered":"2024-09-21T10:01:07.424392+0000","last_unstale":"2024-09-21T10:05:42.517230+0000","last_undegraded":"2024-09-21T10:05:42.517230+0000","last_fullsized":"2024-09-21T10:05:42.517230+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:27.266390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":90,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.513364+0000","last_change":"2024-09-21T09:59:54.267742+0000","last_active":"2024-09-21T10:05:42.513364+0000","last_peered":"2024-09-21T10:05:42.513364+0000","last_clean":"2024-09-21T10:05:42.513364+0000","last_became_active":"2024-09-21T09:59:54.267406+0000","last_became_peered":"2024-09-21T09:59:54.267406+0000","last_unstale":"2024-09-21T10:05:42.513364+0000","last_undegraded":"2024-09-21T10:05:42.513364+0000","last_fullsized":"2024-09-21T10:05:42.513364+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:25:35.968779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":102,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.517360+0000","last_change":"2024-09-21T10:02:14.103531+0000","last_active":"2024-09-21T10:05:42.517360+0000","last_peered":"2024-09-21T10:05:42.517360+0000","last_clean":"2024-09-21T10:05:42.517360+0000","last_became_active":"2024-09-21T10:02:14.103218+0000","last_became_peered":"2024-09-21T10:02:14.103218+0000","last_unstale":"2024-09-21T10:05:42.517360+0000","last_undegraded":"2024-09-21T10:05:42.517360+0000","last_fullsized":"2024-09-21T10:05:42.517360+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:40.014308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"125'10241","reported_seq":10290,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:52.050249+0000","last_change":"2024-09-21T10:03:41.941191+0000","last_active":"2024-09-21T10:05:52.050249+0000","last_peered":"2024-09-21T10:05:52.050249+0000","last_clean":"2024-09-21T10:05:52.050249+0000","last_became_active":"2024-09-21T10:03:41.940898+0000","last_became_peered":"2024-09-21T10:03:41.940898+0000","last_unstale":"2024-09-21T10:05:52.050249+0000","last_undegraded":"2024-09-21T10:05:52.050249+0000","last_fullsized":"2024-09-21T10:05:52.050249+0000","mapping_epoch":105,"log_start":"109'1400","ondisk_log_start":"109'1400","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":8841,"log_dups_size":0,"ondisk_log_size":8841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:53:35.519904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432603136,"num_objects":6601,"num_object_clones":0,"num_object_copies":19803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10241,"num_write_kb":538944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":106,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.520823+0000","last_change":"2024-09-21T10:02:14.103145+0000","last_active":"2024-09-21T10:05:42.520823+0000","last_peered":"2024-09-21T10:05:42.520823+0000","last_clean":"2024-09-21T10:05:42.520823+0000","last_became_active":"2024-09-21T10:02:14.102806+0000","last_became_peered":"2024-09-21T10:02:14.102806+0000","last_unstale":"2024-09-21T10:05:42.520823+0000","last_undegraded":"2024-09-21T10:05:42.520823+0000","last_fullsized":"2024-09-21T10:05:42.520823+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:24.348745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":763,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.578132+0000","last_change":"2024-09-21T10:03:39.669081+0000","last_active":"2024-09-21T10:05:46.578132+0000","last_peered":"2024-09-21T10:05:46.578132+0000","last_clean":"2024-09-21T10:05:46.578132+0000","last_became_active":"2024-09-21T10:03:39.668544+0000","last_became_peered":"2024-09-21T10:03:39.668544+0000","last_unstale":"2024-09-21T10:05:46.578132+0000","last_undegraded":"2024-09-21T10:05:46.578132+0000","last_fullsized":"2024-09-21T10:05:46.578132+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:48.735396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":46,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.517112+0000","last_change":"2024-09-21T10:03:39.668641+0000","last_active":"2024-09-21T10:05:42.517112+0000","last_peered":"2024-09-21T10:05:42.517112+0000","last_clean":"2024-09-21T10:05:42.517112+0000","last_became_active":"2024-09-21T10:03:39.668390+0000","last_became_peered":"2024-09-21T10:03:39.668390+0000","last_unstale":"2024-09-21T10:05:42.517112+0000","last_undegraded":"2024-09-21T10:05:42.517112+0000","last_fullsized":"2024-09-21T10:05:42.517112+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:07.211608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"125'10512","reported_seq":10540,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:53.011665+0000","last_change":"2024-09-21T10:03:41.941962+0000","last_active":"2024-09-21T10:05:53.011665+0000","last_peered":"2024-09-21T10:05:53.011665+0000","last_clean":"2024-09-21T10:05:53.011665+0000","last_became_active":"2024-09-21T10:03:41.941669+0000","last_became_peered":"2024-09-21T10:03:41.941669+0000","last_unstale":"2024-09-21T10:05:53.011665+0000","last_undegraded":"2024-09-21T10:05:53.011665+0000","last_fullsized":"2024-09-21T10:05:53.011665+0000","mapping_epoch":105,"log_start":"109'800","ondisk_log_start":"109'800","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":9712,"log_dups_size":0,"ondisk_log_size":9712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:52.959659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443416576,"num_objects":6766,"num_object_clones":0,"num_object_copies":20298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10512,"num_write_kb":552896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":89,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.578933+0000","last_change":"2024-09-21T10:03:39.670759+0000","last_active":"2024-09-21T10:05:46.578933+0000","last_peered":"2024-09-21T10:05:46.578933+0000","last_clean":"2024-09-21T10:05:46.578933+0000","last_became_active":"2024-09-21T10:03:39.670426+0000","last_became_peered":"2024-09-21T10:03:39.670426+0000","last_unstale":"2024-09-21T10:05:46.578933+0000","last_undegraded":"2024-09-21T10:05:46.578933+0000","last_fullsized":"2024-09-21T10:05:46.578933+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:03:22.478560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":436,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.515795+0000","last_change":"2024-09-21T09:59:54.265559+0000","last_active":"2024-09-21T10:05:42.515795+0000","last_peered":"2024-09-21T10:05:42.515795+0000","last_clean":"2024-09-21T10:05:42.515795+0000","last_became_active":"2024-09-21T09:59:54.264884+0000","last_became_peered":"2024-09-21T09:59:54.264884+0000","last_unstale":"2024-09-21T10:05:42.515795+0000","last_undegraded":"2024-09-21T10:05:42.515795+0000","last_fullsized":"2024-09-21T10:05:42.515795+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:24.544845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":101,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.578312+0000","last_change":"2024-09-21T09:59:54.267798+0000","last_active":"2024-09-21T10:05:46.578312+0000","last_peered":"2024-09-21T10:05:46.578312+0000","last_clean":"2024-09-21T10:05:46.578312+0000","last_became_active":"2024-09-21T09:59:54.265615+0000","last_became_peered":"2024-09-21T09:59:54.265615+0000","last_unstale":"2024-09-21T10:05:46.578312+0000","last_undegraded":"2024-09-21T10:05:46.578312+0000","last_fullsized":"2024-09-21T10:05:46.578312+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:30.363872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"125'10237","reported_seq":10265,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:53.085525+0000","last_change":"2024-09-21T10:03:41.939642+0000","last_active":"2024-09-21T10:05:53.085525+0000","last_peered":"2024-09-21T10:05:53.085525+0000","last_clean":"2024-09-21T10:05:53.085525+0000","last_became_active":"2024-09-21T10:03:41.939357+0000","last_became_peered":"2024-09-21T10:03:41.939357+0000","last_unstale":"2024-09-21T10:05:53.085525+0000","last_undegraded":"2024-09-21T10:05:53.085525+0000","last_fullsized":"2024-09-21T10:05:53.085525+0000","mapping_epoch":105,"log_start":"110'2900","ondisk_log_start":"110'2900","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":7337,"log_dups_size":0,"ondisk_log_size":7337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:54:51.943050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425918464,"num_objects":6499,"num_object_clones":0,"num_object_copies":19497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10237,"num_write_kb":535552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":114,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.592776+0000","last_change":"2024-09-21T10:02:14.101943+0000","last_active":"2024-09-21T10:05:46.592776+0000","last_peered":"2024-09-21T10:05:46.592776+0000","last_clean":"2024-09-21T10:05:46.592776+0000","last_became_active":"2024-09-21T10:02:14.101488+0000","last_became_peered":"2024-09-21T10:02:14.101488+0000","last_unstale":"2024-09-21T10:05:46.592776+0000","last_undegraded":"2024-09-21T10:05:46.592776+0000","last_fullsized":"2024-09-21T10:05:46.592776+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:52:27.866414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":424,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.519496+0000","last_change":"2024-09-21T10:03:39.670475+0000","last_active":"2024-09-21T10:05:42.519496+0000","last_peered":"2024-09-21T10:05:42.519496+0000","last_clean":"2024-09-21T10:05:42.519496+0000","last_became_active":"2024-09-21T10:03:39.670039+0000","last_became_peered":"2024-09-21T10:03:39.670039+0000","last_unstale":"2024-09-21T10:05:42.519496+0000","last_undegraded":"2024-09-21T10:05:42.519496+0000","last_fullsized":"2024-09-21T10:05:42.519496+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:55.384881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":111,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.592458+0000","last_change":"2024-09-21T10:02:14.095924+0000","last_active":"2024-09-21T10:05:46.592458+0000","last_peered":"2024-09-21T10:05:46.592458+0000","last_clean":"2024-09-21T10:05:46.592458+0000","last_became_active":"2024-09-21T10:02:14.095553+0000","last_became_peered":"2024-09-21T10:02:14.095553+0000","last_unstale":"2024-09-21T10:05:46.592458+0000","last_undegraded":"2024-09-21T10:05:46.592458+0000","last_fullsized":"2024-09-21T10:05:46.592458+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:08.155980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"125'10126","reported_seq":10178,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:51.665397+0000","last_change":"2024-09-21T10:03:42.340430+0000","last_active":"2024-09-21T10:05:51.665397+0000","last_peered":"2024-09-21T10:05:51.665397+0000","last_clean":"2024-09-21T10:05:51.665397+0000","last_became_active":"2024-09-21T10:03:42.340071+0000","last_became_peered":"2024-09-21T10:03:42.340071+0000","last_unstale":"2024-09-21T10:05:51.665397+0000","last_undegraded":"2024-09-21T10:05:51.665397+0000","last_fullsized":"2024-09-21T10:05:51.665397+0000","mapping_epoch":105,"log_start":"109'1300","ondisk_log_start":"109'1300","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":8826,"log_dups_size":0,"ondisk_log_size":8826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:42.171457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426639360,"num_objects":6510,"num_object_clones":0,"num_object_copies":19530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10126,"num_write_kb":532352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":116,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.577979+0000","last_change":"2024-09-21T10:01:07.426736+0000","last_active":"2024-09-21T10:05:46.577979+0000","last_peered":"2024-09-21T10:05:46.577979+0000","last_clean":"2024-09-21T10:05:46.577979+0000","last_became_active":"2024-09-21T10:01:07.426416+0000","last_became_peered":"2024-09-21T10:01:07.426416+0000","last_unstale":"2024-09-21T10:05:46.577979+0000","last_undegraded":"2024-09-21T10:05:46.577979+0000","last_fullsized":"2024-09-21T10:05:46.577979+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:59.784757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":352,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.522454+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:05:42.522454+0000","last_peered":"2024-09-21T10:05:42.522454+0000","last_clean":"2024-09-21T10:05:42.522454+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:05:42.522454+0000","last_undegraded":"2024-09-21T10:05:42.522454+0000","last_fullsized":"2024-09-21T10:05:42.522454+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":72,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.516098+0000","last_change":"2024-09-21T09:52:20.391439+0000","last_active":"2024-09-21T10:05:42.516098+0000","last_peered":"2024-09-21T10:05:42.516098+0000","last_clean":"2024-09-21T10:05:42.516098+0000","last_became_active":"2024-09-21T09:52:20.391157+0000","last_became_peered":"2024-09-21T09:52:20.391157+0000","last_unstale":"2024-09-21T10:05:42.516098+0000","last_undegraded":"2024-09-21T10:05:42.516098+0000","last_fullsized":"2024-09-21T10:05:42.516098+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:36.788330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":89,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.513300+0000","last_change":"2024-09-21T10:01:07.421150+0000","last_active":"2024-09-21T10:05:42.513300+0000","last_peered":"2024-09-21T10:05:42.513300+0000","last_clean":"2024-09-21T10:05:42.513300+0000","last_became_active":"2024-09-21T10:01:07.420830+0000","last_became_peered":"2024-09-21T10:01:07.420830+0000","last_unstale":"2024-09-21T10:05:42.513300+0000","last_undegraded":"2024-09-21T10:05:42.513300+0000","last_fullsized":"2024-09-21T10:05:42.513300+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:15:34.472626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":95,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.514290+0000","last_change":"2024-09-21T09:52:20.371668+0000","last_active":"2024-09-21T10:05:42.514290+0000","last_peered":"2024-09-21T10:05:42.514290+0000","last_clean":"2024-09-21T10:05:42.514290+0000","last_became_active":"2024-09-21T09:52:20.371389+0000","last_became_peered":"2024-09-21T09:52:20.371389+0000","last_unstale":"2024-09-21T10:05:42.514290+0000","last_undegraded":"2024-09-21T10:05:42.514290+0000","last_fullsized":"2024-09-21T10:05:42.514290+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:10.013020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":68,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.523534+0000","last_change":"2024-09-21T10:03:39.670792+0000","last_active":"2024-09-21T10:05:42.523534+0000","last_peered":"2024-09-21T10:05:42.523534+0000","last_clean":"2024-09-21T10:05:42.523534+0000","last_became_active":"2024-09-21T10:03:39.670506+0000","last_became_peered":"2024-09-21T10:03:39.670506+0000","last_unstale":"2024-09-21T10:05:42.523534+0000","last_undegraded":"2024-09-21T10:05:42.523534+0000","last_fullsized":"2024-09-21T10:05:42.523534+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:33:19.134786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":93,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.515587+0000","last_change":"2024-09-21T10:02:14.102896+0000","last_active":"2024-09-21T10:05:42.515587+0000","last_peered":"2024-09-21T10:05:42.515587+0000","last_clean":"2024-09-21T10:05:42.515587+0000","last_became_active":"2024-09-21T10:02:14.102493+0000","last_became_peered":"2024-09-21T10:02:14.102493+0000","last_unstale":"2024-09-21T10:05:42.515587+0000","last_undegraded":"2024-09-21T10:05:42.515587+0000","last_fullsized":"2024-09-21T10:05:42.515587+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:10:43.572005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":89,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.524379+0000","last_change":"2024-09-21T10:01:07.424062+0000","last_active":"2024-09-21T10:05:42.524379+0000","last_peered":"2024-09-21T10:05:42.524379+0000","last_clean":"2024-09-21T10:05:42.524379+0000","last_became_active":"2024-09-21T10:01:07.423629+0000","last_became_peered":"2024-09-21T10:01:07.423629+0000","last_unstale":"2024-09-21T10:05:42.524379+0000","last_undegraded":"2024-09-21T10:05:42.524379+0000","last_fullsized":"2024-09-21T10:05:42.524379+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:24.119642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":71,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.594603+0000","last_change":"2024-09-21T10:02:14.100451+0000","last_active":"2024-09-21T10:05:46.594603+0000","last_peered":"2024-09-21T10:05:46.594603+0000","last_clean":"2024-09-21T10:05:46.594603+0000","last_became_active":"2024-09-21T10:02:14.100160+0000","last_became_peered":"2024-09-21T10:02:14.100160+0000","last_unstale":"2024-09-21T10:05:46.594603+0000","last_undegraded":"2024-09-21T10:05:46.594603+0000","last_fullsized":"2024-09-21T10:05:46.594603+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:28.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":112,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.578210+0000","last_change":"2024-09-21T10:02:14.096098+0000","last_active":"2024-09-21T10:05:46.578210+0000","last_peered":"2024-09-21T10:05:46.578210+0000","last_clean":"2024-09-21T10:05:46.578210+0000","last_became_active":"2024-09-21T10:02:14.095811+0000","last_became_peered":"2024-09-21T10:02:14.095811+0000","last_unstale":"2024-09-21T10:05:46.578210+0000","last_undegraded":"2024-09-21T10:05:46.578210+0000","last_fullsized":"2024-09-21T10:05:46.578210+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:54.619333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":116,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.592359+0000","last_change":"2024-09-21T10:02:14.100712+0000","last_active":"2024-09-21T10:05:46.592359+0000","last_peered":"2024-09-21T10:05:46.592359+0000","last_clean":"2024-09-21T10:05:46.592359+0000","last_became_active":"2024-09-21T10:02:14.100427+0000","last_became_peered":"2024-09-21T10:02:14.100427+0000","last_unstale":"2024-09-21T10:05:46.592359+0000","last_undegraded":"2024-09-21T10:05:46.592359+0000","last_fullsized":"2024-09-21T10:05:46.592359+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:16:42.350642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":89,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.522383+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:05:42.522383+0000","last_peered":"2024-09-21T10:05:42.522383+0000","last_clean":"2024-09-21T10:05:42.522383+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:05:42.522383+0000","last_undegraded":"2024-09-21T10:05:42.522383+0000","last_fullsized":"2024-09-21T10:05:42.522383+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":134,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.592465+0000","last_change":"2024-09-21T10:02:14.097369+0000","last_active":"2024-09-21T10:05:46.592465+0000","last_peered":"2024-09-21T10:05:46.592465+0000","last_clean":"2024-09-21T10:05:46.592465+0000","last_became_active":"2024-09-21T10:02:14.097067+0000","last_became_peered":"2024-09-21T10:02:14.097067+0000","last_unstale":"2024-09-21T10:05:46.592465+0000","last_undegraded":"2024-09-21T10:05:46.592465+0000","last_fullsized":"2024-09-21T10:05:46.592465+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:50:48.123813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":113,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.578149+0000","last_change":"2024-09-21T10:02:14.098970+0000","last_active":"2024-09-21T10:05:46.578149+0000","last_peered":"2024-09-21T10:05:46.578149+0000","last_clean":"2024-09-21T10:05:46.578149+0000","last_became_active":"2024-09-21T10:02:14.098641+0000","last_became_peered":"2024-09-21T10:02:14.098641+0000","last_unstale":"2024-09-21T10:05:46.578149+0000","last_undegraded":"2024-09-21T10:05:46.578149+0000","last_fullsized":"2024-09-21T10:05:46.578149+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:25.215602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":101,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.580179+0000","last_change":"2024-09-21T10:02:14.097021+0000","last_active":"2024-09-21T10:05:46.580179+0000","last_peered":"2024-09-21T10:05:46.580179+0000","last_clean":"2024-09-21T10:05:46.580179+0000","last_became_active":"2024-09-21T10:02:14.096741+0000","last_became_peered":"2024-09-21T10:02:14.096741+0000","last_unstale":"2024-09-21T10:05:46.580179+0000","last_undegraded":"2024-09-21T10:05:46.580179+0000","last_fullsized":"2024-09-21T10:05:46.580179+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:18:20.631150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":116,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.592725+0000","last_change":"2024-09-21T10:02:14.099460+0000","last_active":"2024-09-21T10:05:46.592725+0000","last_peered":"2024-09-21T10:05:46.592725+0000","last_clean":"2024-09-21T10:05:46.592725+0000","last_became_active":"2024-09-21T10:02:14.099159+0000","last_became_peered":"2024-09-21T10:02:14.099159+0000","last_unstale":"2024-09-21T10:05:46.592725+0000","last_undegraded":"2024-09-21T10:05:46.592725+0000","last_fullsized":"2024-09-21T10:05:46.592725+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:23:47.599107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":108,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.513250+0000","last_change":"2024-09-21T10:01:41.360480+0000","last_active":"2024-09-21T10:05:42.513250+0000","last_peered":"2024-09-21T10:05:42.513250+0000","last_clean":"2024-09-21T10:05:42.513250+0000","last_became_active":"2024-09-21T10:01:41.359791+0000","last_became_peered":"2024-09-21T10:01:41.359791+0000","last_unstale":"2024-09-21T10:05:42.513250+0000","last_undegraded":"2024-09-21T10:05:42.513250+0000","last_fullsized":"2024-09-21T10:05:42.513250+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:42.812974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.579035+0000","last_change":"2024-09-21T10:02:14.102826+0000","last_active":"2024-09-21T10:05:46.579035+0000","last_peered":"2024-09-21T10:05:46.579035+0000","last_clean":"2024-09-21T10:05:46.579035+0000","last_became_active":"2024-09-21T10:02:14.102499+0000","last_became_peered":"2024-09-21T10:02:14.102499+0000","last_unstale":"2024-09-21T10:05:46.579035+0000","last_undegraded":"2024-09-21T10:05:46.579035+0000","last_fullsized":"2024-09-21T10:05:46.579035+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:30.924687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":105,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.578289+0000","last_change":"2024-09-21T09:59:54.268378+0000","last_active":"2024-09-21T10:05:46.578289+0000","last_peered":"2024-09-21T10:05:46.578289+0000","last_clean":"2024-09-21T10:05:46.578289+0000","last_became_active":"2024-09-21T09:59:54.267187+0000","last_became_peered":"2024-09-21T09:59:54.267187+0000","last_unstale":"2024-09-21T10:05:46.578289+0000","last_undegraded":"2024-09-21T10:05:46.578289+0000","last_fullsized":"2024-09-21T10:05:46.578289+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:12:06.054483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":106,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.513232+0000","last_change":"2024-09-21T10:02:14.100541+0000","last_active":"2024-09-21T10:05:42.513232+0000","last_peered":"2024-09-21T10:05:42.513232+0000","last_clean":"2024-09-21T10:05:42.513232+0000","last_became_active":"2024-09-21T10:02:14.100327+0000","last_became_peered":"2024-09-21T10:02:14.100327+0000","last_unstale":"2024-09-21T10:05:42.513232+0000","last_undegraded":"2024-09-21T10:05:42.513232+0000","last_fullsized":"2024-09-21T10:05:42.513232+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:34:34.304188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":91,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.517478+0000","last_change":"2024-09-21T10:01:07.424248+0000","last_active":"2024-09-21T10:05:42.517478+0000","last_peered":"2024-09-21T10:05:42.517478+0000","last_clean":"2024-09-21T10:05:42.517478+0000","last_became_active":"2024-09-21T10:01:07.423738+0000","last_became_peered":"2024-09-21T10:01:07.423738+0000","last_unstale":"2024-09-21T10:05:42.517478+0000","last_undegraded":"2024-09-21T10:05:42.517478+0000","last_fullsized":"2024-09-21T10:05:42.517478+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:44:35.663555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":106,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.513167+0000","last_change":"2024-09-21T10:02:14.096583+0000","last_active":"2024-09-21T10:05:42.513167+0000","last_peered":"2024-09-21T10:05:42.513167+0000","last_clean":"2024-09-21T10:05:42.513167+0000","last_became_active":"2024-09-21T10:02:14.096196+0000","last_became_peered":"2024-09-21T10:02:14.096196+0000","last_unstale":"2024-09-21T10:05:42.513167+0000","last_undegraded":"2024-09-21T10:05:42.513167+0000","last_fullsized":"2024-09-21T10:05:42.513167+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:07:01.888435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":74,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.524227+0000","last_change":"2024-09-21T09:52:20.371638+0000","last_active":"2024-09-21T10:05:42.524227+0000","last_peered":"2024-09-21T10:05:42.524227+0000","last_clean":"2024-09-21T10:05:42.524227+0000","last_became_active":"2024-09-21T09:52:20.371383+0000","last_became_peered":"2024-09-21T09:52:20.371383+0000","last_unstale":"2024-09-21T10:05:42.524227+0000","last_undegraded":"2024-09-21T10:05:42.524227+0000","last_fullsized":"2024-09-21T10:05:42.524227+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:33:45.384397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":137,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.594778+0000","last_change":"2024-09-21T10:02:14.099799+0000","last_active":"2024-09-21T10:05:46.594778+0000","last_peered":"2024-09-21T10:05:46.594778+0000","last_clean":"2024-09-21T10:05:46.594778+0000","last_became_active":"2024-09-21T10:02:14.099375+0000","last_became_peered":"2024-09-21T10:02:14.099375+0000","last_unstale":"2024-09-21T10:05:46.594778+0000","last_undegraded":"2024-09-21T10:05:46.594778+0000","last_fullsized":"2024-09-21T10:05:46.594778+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:13.407638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":122,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.592984+0000","last_change":"2024-09-21T10:02:14.099439+0000","last_active":"2024-09-21T10:05:46.592984+0000","last_peered":"2024-09-21T10:05:46.592984+0000","last_clean":"2024-09-21T10:05:46.592984+0000","last_became_active":"2024-09-21T10:02:14.099117+0000","last_became_peered":"2024-09-21T10:02:14.099117+0000","last_unstale":"2024-09-21T10:05:46.592984+0000","last_undegraded":"2024-09-21T10:05:46.592984+0000","last_fullsized":"2024-09-21T10:05:46.592984+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:39.681817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":99,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-21T10:05:42.512938+0000","last_change":"2024-09-21T10:01:41.360674+0000","last_active":"2024-09-21T10:05:42.512938+0000","last_peered":"2024-09-21T10:05:42.512938+0000","last_clean":"2024-09-21T10:05:42.512938+0000","last_became_active":"2024-09-21T10:01:41.360258+0000","last_became_peered":"2024-09-21T10:01:41.360258+0000","last_unstale":"2024-09-21T10:05:42.512938+0000","last_undegraded":"2024-09-21T10:05:42.512938+0000","last_fullsized":"2024-09-21T10:05:42.512938+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:06.200022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":116,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.578144+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:05:46.578144+0000","last_peered":"2024-09-21T10:05:46.578144+0000","last_clean":"2024-09-21T10:05:46.578144+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:05:46.578144+0000","last_undegraded":"2024-09-21T10:05:46.578144+0000","last_fullsized":"2024-09-21T10:05:46.578144+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":6913261592,"num_objects":105489,"num_object_clones":0,"num_object_copies":316467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105489,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165783,"num_write_kb":8680641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20819619840,"data_stored":20819607624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145883,"ondisk_log_size":145883,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5983,"num_read_kb":13006,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397693,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2432064,"kb_used_data":2122168,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91313088,"statfs":{"total":95995035648,"available":93504602112,"internally_reserved":0,"allocated":2173100032,"data_stored":2172449298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593921,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2013172,"kb_used_data":1703340,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91731980,"statfs":{"total":95995035648,"available":93933547520,"internally_reserved":0,"allocated":1744220160,"data_stored":1743581563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4154664,"kb_used_data":3857120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":89590488,"statfs":{"total":95995035648,"available":91740659712,"internally_reserved":0,"allocated":3949690880,"data_stored":3949059562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986380,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3691060,"kb_used_data":3385388,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":90054092,"statfs":{"total":95995035648,"available":92215390208,"internally_reserved":0,"allocated":3466637312,"data_stored":3465998220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":101,"seq":433791696942,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1328416,"kb_used_data":1260696,"kb_used_omap":5,"kb_used_meta":67706,"kb_avail":92416736,"statfs":{"total":95995035648,"available":94634737664,"internally_reserved":0,"allocated":1290952704,"data_stored":1290329312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":69331728},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860469,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2641384,"kb_used_data":2529184,"kb_used_omap":8,"kb_used_meta":112183,"kb_avail":91103768,"statfs":{"total":95995035648,"available":93290258432,"internally_reserved":0,"allocated":2589884416,"data_stored":2589253378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":114876268},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":91,"seq":390842023995,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2223192,"kb_used_data":2147088,"kb_used_omap":7,"kb_used_meta":76088,"kb_avail":91521960,"statfs":{"total":95995035648,"available":93718487040,"internally_reserved":0,"allocated":2198618112,"data_stored":2198002219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":77914674},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187530,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3529960,"kb_used_data":3421920,"kb_used_omap":8,"kb_used_meta":108023,"kb_avail":90215192,"statfs":{"total":95995035648,"available":92380356608,"internally_reserved":0,"allocated":3504046080,"data_stored":3503415550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":110616561},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3501785088,"data_stored":3501785088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2196701184,"data_stored":2196701184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2574192640,"data_stored":2574188568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1289224192,"data_stored":1289224192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3443523584,"data_stored":3443523584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3914403840,"data_stored":3914399768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1736638464,"data_stored":1736638464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2163150848,"data_stored":2163146776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:05:56.085 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:05:56.087 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:56.124 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:56.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:56.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:56.160 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:05:56.227 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:56.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:56.282 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:56.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:56.383 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:56.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:56.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:56.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:56.519 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:56.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:56.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:56.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:56.663 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:05:56.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:56.700 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:56.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:56.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:56.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:56.827 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:56 smithi196 ceph-mon[84784]: pgmap v325: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 885 op/s 2024-09-21T10:05:56.827 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:56 smithi196 ceph-mon[84784]: from='client.45032 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:05:56.827 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:56 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:05:56.827 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:56 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1039698478' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:05:56.827 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:56 smithi196 ceph-mon[84784]: osdmap e126: 8 total, 8 up, 8 in 2024-09-21T10:05:56.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:56.836 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:56.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:56.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:56 smithi098 ceph-mon[97580]: pgmap v325: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 885 op/s 2024-09-21T10:05:56.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:56 smithi098 ceph-mon[97580]: from='client.45032 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:05:56.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:56 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:05:56.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:56 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1039698478' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:05:56.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:56 smithi098 ceph-mon[97580]: osdmap e126: 8 total, 8 up, 8 in 2024-09-21T10:05:56.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:56 smithi098 ceph-mon[100914]: pgmap v325: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 885 op/s 2024-09-21T10:05:56.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:56 smithi098 ceph-mon[100914]: from='client.45032 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:05:56.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:56 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:05:56.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:56 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1039698478' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:05:56.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:56 smithi098 ceph-mon[100914]: osdmap e126: 8 total, 8 up, 8 in 2024-09-21T10:05:56.937 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:56.961 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:56.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:56.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:57.062 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:57.092 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:57.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:57.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:57.194 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:57.226 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:57.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:57.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:57.328 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:57.359 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:57.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:57.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:57.461 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:57.495 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:57.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:57.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:57.597 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:57.631 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:57.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:57.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:57.732 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:57.756 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:57.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:57.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:57.857 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:57.893 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:57.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:57.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:57.995 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:58.037 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:58.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:58.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:58.110 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:58.139 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:58.176 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:58.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:58.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:58.278 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:58.327 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:58.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:58.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:58.429 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:58.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:58.457 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:58.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:58.558 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:58.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:58.601 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:58.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:58.602 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:05:58.602 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:05:58.702 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:58.728 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:58.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:58.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:58.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:58.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:58 smithi098 ceph-mon[97580]: pgmap v327: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 886 op/s 2024-09-21T10:05:58.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:58 smithi098 ceph-mon[100914]: pgmap v327: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 886 op/s 2024-09-21T10:05:58.856 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:58.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:58.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:58 smithi196 ceph-mon[84784]: pgmap v327: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 886 op/s 2024-09-21T10:05:58.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:58.992 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:58.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:58.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:59.094 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:59.127 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:59.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:59.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:59.208 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:05:59.229 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:59.289 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:59.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:59.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:59.322 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:05:59.391 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:59.447 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:05:59.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:59.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:59.549 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:59.569 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:05:59.570 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:05:59.596 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:05:59.596 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:05:59.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:59.597 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:59.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:59.624 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:05:59.625 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:05:59.657 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:05:59.657 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:05:59.697 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:05:59.698 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:05:59.698 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:59.733 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:05:59.733 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:05:59.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:59.750 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:59.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:59.755 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:05:59.776 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:05:59.776 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:05:59.777 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:05:59 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/553562395' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:59.777 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:05:59 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/553562395' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:59.803 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:05:59.803 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:05:59.851 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:59.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:05:59 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/553562395' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:59.891 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:05:59.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:59.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:59.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:00.033 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:00.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:00.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:00.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:00.176 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:00.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:00.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:00.278 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:00.310 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:00.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:00.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:00.412 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:00.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:00.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:00.554 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:00.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:00.582 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:00.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:00.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:00 smithi196 ceph-mon[84784]: pgmap v328: 89 pgs: 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 862 op/s 2024-09-21T10:06:00.583 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:06:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:06:00.215Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:06:00.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:00.725 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:00.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:00.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:00.827 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:00.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:00 smithi098 ceph-mon[97580]: pgmap v328: 89 pgs: 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 862 op/s 2024-09-21T10:06:00.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:06:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:06:00] "GET /metrics HTTP/1.1" 200 36461 "" "Prometheus/2.51.0" 2024-09-21T10:06:00.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:00 smithi098 ceph-mon[100914]: pgmap v328: 89 pgs: 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 862 op/s 2024-09-21T10:06:00.865 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:00.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:00.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:00.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:00.999 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:01.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:01.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:01.101 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:01.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:01.139 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:01.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:01.240 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:01.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:01.275 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:01.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:01.375 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:01.416 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:01.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:01.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:01.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:01.553 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:01.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:01.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:01.646 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:06:01.646 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-21T10:06:01.647 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-21T10:06:01.647 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:06:01.647 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.647 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:31160 5.9 e110/105 5:969448b3:::benchmark_data_smithi098_135187_object31159:head v 110'1945, mlcod=110'1940)", 2024-09-21T10:06:01.647 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:03:59.625413+0000", 2024-09-21T10:06:01.647 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 122.015821526, 2024-09-21T10:06:01.647 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.090990823999999998, 2024-09-21T10:06:01.647 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:01.647 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:01.647 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:01.647 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.647 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:01.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.625413+0000", 2024-09-21T10:06:01.648 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.648 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.648 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:01.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.625413+0000", 2024-09-21T10:06:01.648 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.648 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.648 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:01.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.625412+0000", 2024-09-21T10:06:01.648 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:01.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:01.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.625435+0000", 2024-09-21T10:06:01.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2957000000000001e-05 2024-09-21T10:06:01.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:01.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.625446+0000", 2024-09-21T10:06:01.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0195e-05 2024-09-21T10:06:01.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:01.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.625452+0000", 2024-09-21T10:06:01.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8669999999999996e-06 2024-09-21T10:06:01.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.650 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.650 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:01.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.626635+0000", 2024-09-21T10:06:01.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001182324 2024-09-21T10:06:01.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.650 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.650 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:01.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.626651+0000", 2024-09-21T10:06:01.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5818999999999999e-05 2024-09-21T10:06:01.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.650 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.650 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:01.651 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.716336+0000", 2024-09-21T10:06:01.651 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.089685444000000003 2024-09-21T10:06:01.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.651 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:01.651 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.716404+0000", 2024-09-21T10:06:01.651 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7878000000000003e-05 2024-09-21T10:06:01.651 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.651 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:01.651 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.651 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:31161 5.0 e110/105 5:0ddaf0d6:::benchmark_data_smithi098_135187_object31160:head v 110'1943, mlcod=110'1941)", 2024-09-21T10:06:01.652 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:03:59.625599+0000", 2024-09-21T10:06:01.652 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 122.015635817, 2024-09-21T10:06:01.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.086761368000000005, 2024-09-21T10:06:01.652 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:01.652 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:01.652 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:01.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:01.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.625599+0000", 2024-09-21T10:06:01.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:01.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.625599+0000", 2024-09-21T10:06:01.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:01.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.625598+0000", 2024-09-21T10:06:01.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:01.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:01.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.625666+0000", 2024-09-21T10:06:01.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7271e-05 2024-09-21T10:06:01.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.654 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:01.654 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.625675+0000", 2024-09-21T10:06:01.654 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3389999999999999e-06 2024-09-21T10:06:01.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.654 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.654 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:01.654 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.625680+0000", 2024-09-21T10:06:01.654 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7990000000000001e-06 2024-09-21T10:06:01.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.654 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.654 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:01.654 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.626301+0000", 2024-09-21T10:06:01.654 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00062095599999999998 2024-09-21T10:06:01.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.655 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.655 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:01.655 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.626319+0000", 2024-09-21T10:06:01.655 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7921e-05 2024-09-21T10:06:01.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.655 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.655 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:01.655 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.712313+0000", 2024-09-21T10:06:01.655 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.085994838000000004 2024-09-21T10:06:01.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.655 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:01.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.712360+0000", 2024-09-21T10:06:01.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6816e-05 2024-09-21T10:06:01.656 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.656 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:01.656 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.656 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:31166 5.c 5:39818d95:::benchmark_data_smithi098_135187_object31165:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:06:01.656 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:03:59.625922+0000", 2024-09-21T10:06:01.656 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 122.015312725, 2024-09-21T10:06:01.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.084735654999999993, 2024-09-21T10:06:01.656 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:01.656 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:01.657 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:01.657 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:01.657 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:01.657 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 31166 2024-09-21T10:06:01.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.657 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:01.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.657 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:01.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.625922+0000", 2024-09-21T10:06:01.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.657 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:01.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.625922+0000", 2024-09-21T10:06:01.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:01.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.625920+0000", 2024-09-21T10:06:01.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:06:01.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:01.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.626038+0000", 2024-09-21T10:06:01.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000117334 2024-09-21T10:06:01.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.659 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:01.659 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.626051+0000", 2024-09-21T10:06:01.659 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2989999999999999e-05 2024-09-21T10:06:01.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.659 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.659 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:01.659 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.626056+0000", 2024-09-21T10:06:01.659 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9150000000000004e-06 2024-09-21T10:06:01.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.659 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.659 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:01.659 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.626092+0000", 2024-09-21T10:06:01.659 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6109000000000003e-05 2024-09-21T10:06:01.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.660 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.660 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:01.660 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.626197+0000", 2024-09-21T10:06:01.660 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104947 2024-09-21T10:06:01.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.660 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.660 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:06:01.660 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.626315+0000", 2024-09-21T10:06:01.660 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000118748 2024-09-21T10:06:01.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.660 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.660 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:01.660 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628156+0000", 2024-09-21T10:06:01.661 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001840342 2024-09-21T10:06:01.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.661 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:01.661 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710483+0000", 2024-09-21T10:06:01.661 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.082326954999999993 2024-09-21T10:06:01.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.661 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:01.661 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710566+0000", 2024-09-21T10:06:01.661 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3615999999999994e-05 2024-09-21T10:06:01.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.661 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:01.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710616+0000", 2024-09-21T10:06:01.662 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9505000000000001e-05 2024-09-21T10:06:01.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.662 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.662 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:01.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710658+0000", 2024-09-21T10:06:01.662 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1749000000000003e-05 2024-09-21T10:06:01.662 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.662 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:01.662 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.662 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.662 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:31162 5.0 e110/105 5:05fe675d:::benchmark_data_smithi098_135187_object31161:head v 110'1944, mlcod=110'1941)", 2024-09-21T10:06:01.663 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:03:59.627115+0000", 2024-09-21T10:06:01.663 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 122.014119565, 2024-09-21T10:06:01.663 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.085269972999999999, 2024-09-21T10:06:01.663 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:01.663 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:01.663 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:01.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:01.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.627115+0000", 2024-09-21T10:06:01.663 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:01.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.627115+0000", 2024-09-21T10:06:01.664 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.664 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.664 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:01.664 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.627114+0000", 2024-09-21T10:06:01.664 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:01.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.664 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.664 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:01.664 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.627150+0000", 2024-09-21T10:06:01.664 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5710000000000002e-05 2024-09-21T10:06:01.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.664 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.664 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:01.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.627160+0000", 2024-09-21T10:06:01.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5249999999999994e-06 2024-09-21T10:06:01.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.665 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.665 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:01.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.627169+0000", 2024-09-21T10:06:01.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1009999999999998e-06 2024-09-21T10:06:01.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.665 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.665 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:01.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.627382+0000", 2024-09-21T10:06:01.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00021277399999999999 2024-09-21T10:06:01.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.665 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.665 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:01.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.627397+0000", 2024-09-21T10:06:01.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5792999999999999e-05 2024-09-21T10:06:01.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.666 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:01.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.712364+0000", 2024-09-21T10:06:01.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.08496621 2024-09-21T10:06:01.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.666 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:01.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.712385+0000", 2024-09-21T10:06:01.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1525000000000001e-05 2024-09-21T10:06:01.666 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.667 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:01.667 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.667 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.667 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.667 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.44894.0:31166 5.c e110/105 ondisk, result = 0)", 2024-09-21T10:06:01.667 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:03:59.628022+0000", 2024-09-21T10:06:01.667 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 122.013212602, 2024-09-21T10:06:01.667 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.082463615000000004, 2024-09-21T10:06:01.667 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:01.667 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:06:01.667 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:01.667 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.667 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:01.667 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628022+0000", 2024-09-21T10:06:01.668 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.668 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.668 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:01.668 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628022+0000", 2024-09-21T10:06:01.668 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.668 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.668 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:01.668 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628021+0000", 2024-09-21T10:06:01.668 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:01.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.668 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.669 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:01.669 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628028+0000", 2024-09-21T10:06:01.669 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.002e-06 2024-09-21T10:06:01.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.669 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.669 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:01.669 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628049+0000", 2024-09-21T10:06:01.669 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0169e-05 2024-09-21T10:06:01.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.669 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.669 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:01.669 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628306+0000", 2024-09-21T10:06:01.669 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025782300000000001 2024-09-21T10:06:01.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.670 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.670 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:01.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710467+0000", 2024-09-21T10:06:01.670 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.082160772000000007 2024-09-21T10:06:01.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.670 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.670 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:01.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710476+0000", 2024-09-21T10:06:01.670 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3329999999999997e-06 2024-09-21T10:06:01.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.670 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.670 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:01.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710486+0000", 2024-09-21T10:06:01.670 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0149000000000001e-05 2024-09-21T10:06:01.671 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.671 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:01.671 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.671 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.671 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:31168 5.e 5:7adf1ceb:::benchmark_data_smithi098_135187_object31167:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:06:01.671 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:03:59.628358+0000", 2024-09-21T10:06:01.671 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 122.012876725, 2024-09-21T10:06:01.671 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.088497760999999994, 2024-09-21T10:06:01.671 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:01.671 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:01.671 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:01.671 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:01.671 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:01.672 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 31168 2024-09-21T10:06:01.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.672 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:01.672 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.672 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:01.672 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628358+0000", 2024-09-21T10:06:01.672 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.672 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.672 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:01.672 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628358+0000", 2024-09-21T10:06:01.672 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.673 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.673 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:01.673 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628357+0000", 2024-09-21T10:06:01.673 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:06:01.673 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.673 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.673 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:01.673 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628386+0000", 2024-09-21T10:06:01.673 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9292e-05 2024-09-21T10:06:01.673 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.673 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.673 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:01.673 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628396+0000", 2024-09-21T10:06:01.673 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0318e-05 2024-09-21T10:06:01.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.674 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:01.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628401+0000", 2024-09-21T10:06:01.674 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9509999999999997e-06 2024-09-21T10:06:01.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.674 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:01.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.712075+0000", 2024-09-21T10:06:01.674 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.083673917 2024-09-21T10:06:01.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.674 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:01.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.712188+0000", 2024-09-21T10:06:01.675 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000112854 2024-09-21T10:06:01.675 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.675 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.675 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:01.675 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.712309+0000", 2024-09-21T10:06:01.675 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000120741 2024-09-21T10:06:01.675 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:01.676 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:01.676 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.677 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.677 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:01.677 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.715891+0000", 2024-09-21T10:06:01.677 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003582146 2024-09-21T10:06:01.677 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.677 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.677 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:01.677 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.716474+0000", 2024-09-21T10:06:01.677 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00058251500000000003 2024-09-21T10:06:01.677 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.677 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.677 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:01.678 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.716784+0000", 2024-09-21T10:06:01.678 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000310847 2024-09-21T10:06:01.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.678 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.678 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:01.678 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.716823+0000", 2024-09-21T10:06:01.678 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8717999999999998e-05 2024-09-21T10:06:01.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.678 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.678 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:01.678 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.716856+0000", 2024-09-21T10:06:01.678 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2566e-05 2024-09-21T10:06:01.678 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.679 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:01.679 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.679 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.679 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.679 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:31164 5.0 e110/105 5:0afeb33f:::benchmark_data_smithi098_135187_object31163:head v 110'1945, mlcod=110'1941)", 2024-09-21T10:06:01.679 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:03:59.628368+0000", 2024-09-21T10:06:01.679 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 122.012866972, 2024-09-21T10:06:01.679 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.084189009999999995, 2024-09-21T10:06:01.679 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:01.679 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:01.679 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:01.679 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.679 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:01.679 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628368+0000", 2024-09-21T10:06:01.679 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.680 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.680 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.680 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:01.680 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628368+0000", 2024-09-21T10:06:01.680 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.680 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.680 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.680 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:01.680 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628367+0000", 2024-09-21T10:06:01.680 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:01.680 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.680 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.680 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:01.680 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628428+0000", 2024-09-21T10:06:01.681 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1124000000000006e-05 2024-09-21T10:06:01.681 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.681 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.681 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:01.681 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628438+0000", 2024-09-21T10:06:01.681 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0112000000000001e-05 2024-09-21T10:06:01.681 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.681 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.681 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:01.681 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628444+0000", 2024-09-21T10:06:01.681 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3079999999999998e-06 2024-09-21T10:06:01.681 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.681 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.681 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:01.682 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628663+0000", 2024-09-21T10:06:01.682 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00021982100000000001 2024-09-21T10:06:01.682 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.682 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.682 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:01.682 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628681+0000", 2024-09-21T10:06:01.682 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7379000000000001e-05 2024-09-21T10:06:01.682 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.682 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.682 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:01.682 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.712526+0000", 2024-09-21T10:06:01.682 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.083845722999999997 2024-09-21T10:06:01.682 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.682 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.683 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:01.683 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.712557+0000", 2024-09-21T10:06:01.683 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0247999999999999e-05 2024-09-21T10:06:01.683 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.683 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:01.683 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.683 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.683 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.683 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:31174 5.1 5:80c47547:::benchmark_data_smithi098_135187_object31173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:06:01.683 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:03:59.628729+0000", 2024-09-21T10:06:01.683 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 122.01250537599999, 2024-09-21T10:06:01.683 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.087766128999999998, 2024-09-21T10:06:01.683 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:01.684 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:01.684 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:01.684 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:01.684 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:01.684 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 31174 2024-09-21T10:06:01.684 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.684 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:01.684 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.684 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:01.684 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628729+0000", 2024-09-21T10:06:01.684 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.684 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.684 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.684 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:01.685 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628729+0000", 2024-09-21T10:06:01.685 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.685 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.685 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.685 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:01.685 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628729+0000", 2024-09-21T10:06:01.685 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:06:01.685 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.685 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.685 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:01.685 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.628789+0000", 2024-09-21T10:06:01.685 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0804999999999998e-05 2024-09-21T10:06:01.685 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.685 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.686 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:01.686 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710221+0000", 2024-09-21T10:06:01.686 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.081432168999999999 2024-09-21T10:06:01.686 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.686 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.686 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:01.686 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710233+0000", 2024-09-21T10:06:01.686 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1197e-05 2024-09-21T10:06:01.686 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.686 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.686 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:01.686 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710435+0000", 2024-09-21T10:06:01.686 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00020220399999999999 2024-09-21T10:06:01.687 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.687 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.687 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:01.687 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710610+0000", 2024-09-21T10:06:01.687 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017462000000000001 2024-09-21T10:06:01.687 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.687 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.687 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:06:01.687 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710732+0000", 2024-09-21T10:06:01.687 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012234600000000001 2024-09-21T10:06:01.687 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.687 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.687 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:01.687 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.716304+0000", 2024-09-21T10:06:01.688 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055718989999999999 2024-09-21T10:06:01.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.688 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.688 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:01.688 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.716333+0000", 2024-09-21T10:06:01.688 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8935e-05 2024-09-21T10:06:01.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.688 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.688 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:01.688 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.716412+0000", 2024-09-21T10:06:01.688 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9506999999999996e-05 2024-09-21T10:06:01.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.688 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.688 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:01.688 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.716456+0000", 2024-09-21T10:06:01.689 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4141999999999999e-05 2024-09-21T10:06:01.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.689 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.689 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:01.689 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.716495+0000", 2024-09-21T10:06:01.689 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9054000000000003e-05 2024-09-21T10:06:01.689 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.689 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:01.689 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.689 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.689 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:35235 5.c 5:3dae604c:::benchmark_data_smithi098_135187_object35234:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:06:01.689 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:01.640349+0000", 2024-09-21T10:06:01.689 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 120.000885819, 2024-09-21T10:06:01.690 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.100142382, 2024-09-21T10:06:01.690 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:01.690 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:01.690 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:01.690 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:01.690 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:01.690 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35235 2024-09-21T10:06:01.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.690 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:01.690 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.690 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:01.690 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.640349+0000", 2024-09-21T10:06:01.690 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.691 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.691 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.691 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:01.691 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.640349+0000", 2024-09-21T10:06:01.691 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.691 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.691 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.691 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:01.691 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.640347+0000", 2024-09-21T10:06:01.691 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:06:01.691 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.691 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.691 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:01.691 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.640402+0000", 2024-09-21T10:06:01.692 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4369000000000001e-05 2024-09-21T10:06:01.692 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.692 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.692 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:01.692 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.640417+0000", 2024-09-21T10:06:01.692 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5102e-05 2024-09-21T10:06:01.692 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.692 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.692 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:01.692 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.640423+0000", 2024-09-21T10:06:01.692 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7599999999999999e-06 2024-09-21T10:06:01.692 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.692 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.692 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:01.693 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.640465+0000", 2024-09-21T10:06:01.693 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2106e-05 2024-09-21T10:06:01.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.693 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.693 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:01.693 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.640578+0000", 2024-09-21T10:06:01.693 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011309 2024-09-21T10:06:01.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.693 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.693 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:06:01.693 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.640669+0000", 2024-09-21T10:06:01.693 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1698000000000002e-05 2024-09-21T10:06:01.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.693 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.694 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:01.694 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.641690+0000", 2024-09-21T10:06:01.694 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010200879999999999 2024-09-21T10:06:01.694 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.694 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.694 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:01.694 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.641969+0000", 2024-09-21T10:06:01.694 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00027928000000000002 2024-09-21T10:06:01.694 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.694 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.694 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:01.694 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.740358+0000", 2024-09-21T10:06:01.694 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.098388939999999994 2024-09-21T10:06:01.695 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.695 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.695 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:01.695 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.740421+0000", 2024-09-21T10:06:01.695 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3731000000000004e-05 2024-09-21T10:06:01.695 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.695 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.695 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:01.695 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.740491+0000", 2024-09-21T10:06:01.695 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9717999999999999e-05 2024-09-21T10:06:01.695 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.695 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:01.695 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.695 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.696 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.696 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:35236 5.e 5:7128b257:::benchmark_data_smithi098_135187_object35235:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:06:01.696 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:01.641433+0000", 2024-09-21T10:06:01.696 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 119.999801745, 2024-09-21T10:06:01.696 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.099285729000000003, 2024-09-21T10:06:01.696 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:01.696 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:01.696 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:01.696 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:01.696 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:01.696 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35236 2024-09-21T10:06:01.696 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.696 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:01.696 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.697 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:01.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.641433+0000", 2024-09-21T10:06:01.697 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.697 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.697 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:01.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.641433+0000", 2024-09-21T10:06:01.697 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.697 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.697 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:01.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.641432+0000", 2024-09-21T10:06:01.697 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:06:01.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:01.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.641488+0000", 2024-09-21T10:06:01.698 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5544999999999998e-05 2024-09-21T10:06:01.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:01.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.641498+0000", 2024-09-21T10:06:01.698 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0519999999999999e-05 2024-09-21T10:06:01.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:01.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.641502+0000", 2024-09-21T10:06:01.698 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2230000000000001e-06 2024-09-21T10:06:01.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.699 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.699 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:01.699 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.641538+0000", 2024-09-21T10:06:01.699 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5602999999999998e-05 2024-09-21T10:06:01.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.699 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.699 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:01.699 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.641611+0000", 2024-09-21T10:06:01.699 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3124e-05 2024-09-21T10:06:01.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.699 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.699 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:01.699 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.641686+0000", 2024-09-21T10:06:01.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4561000000000007e-05 2024-09-21T10:06:01.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.700 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.700 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:01.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.642686+0000", 2024-09-21T10:06:01.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010004409999999999 2024-09-21T10:06:01.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.700 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.700 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:01.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.642915+0000", 2024-09-21T10:06:01.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00022913400000000001 2024-09-21T10:06:01.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.700 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.700 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:01.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.740643+0000", 2024-09-21T10:06:01.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.097728231999999998 2024-09-21T10:06:01.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.701 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.701 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:01.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.740679+0000", 2024-09-21T10:06:01.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5722999999999997e-05 2024-09-21T10:06:01.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.701 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.701 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:01.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.740719+0000", 2024-09-21T10:06:01.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9399e-05 2024-09-21T10:06:01.701 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.701 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:01.702 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.702 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.702 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:72281 5.1 5:876e16d1:::benchmark_data_smithi098_135187_object72280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T10:06:01.702 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:22.600295+0000", 2024-09-21T10:06:01.702 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 99.040939707000007, 2024-09-21T10:06:01.702 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.071655091000000004, 2024-09-21T10:06:01.702 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:01.702 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:01.702 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:01.702 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:01.702 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:01.703 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 72281 2024-09-21T10:06:01.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.703 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:01.703 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.703 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:01.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.600295+0000", 2024-09-21T10:06:01.703 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.703 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.703 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:01.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.600295+0000", 2024-09-21T10:06:01.703 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.703 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.704 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:01.704 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.600293+0000", 2024-09-21T10:06:01.704 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999976 2024-09-21T10:06:01.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.704 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.704 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:01.704 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.600341+0000", 2024-09-21T10:06:01.704 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8205999999999999e-05 2024-09-21T10:06:01.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.704 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.704 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:01.704 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.600354+0000", 2024-09-21T10:06:01.704 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2856e-05 2024-09-21T10:06:01.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.705 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:01.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.600359+0000", 2024-09-21T10:06:01.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5949999999999998e-06 2024-09-21T10:06:01.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.705 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:01.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.601558+0000", 2024-09-21T10:06:01.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001199149 2024-09-21T10:06:01.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.705 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:01.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.601668+0000", 2024-09-21T10:06:01.706 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109215 2024-09-21T10:06:01.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.706 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.706 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:06:01.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.601754+0000", 2024-09-21T10:06:01.706 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6154000000000002e-05 2024-09-21T10:06:01.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.706 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.706 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:01.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671008+0000", 2024-09-21T10:06:01.706 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.069254223000000004 2024-09-21T10:06:01.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.706 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.706 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:01.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671014+0000", 2024-09-21T10:06:01.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5300000000000004e-06 2024-09-21T10:06:01.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.707 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.707 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:01.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671821+0000", 2024-09-21T10:06:01.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00080713300000000003 2024-09-21T10:06:01.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.707 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.707 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:01.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671909+0000", 2024-09-21T10:06:01.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8096999999999994e-05 2024-09-21T10:06:01.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.707 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.708 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:01.708 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671950+0000", 2024-09-21T10:06:01.708 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1288000000000002e-05 2024-09-21T10:06:01.708 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.708 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:01.708 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.708 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.708 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.708 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:72285 5.c 5:35ed1599:::benchmark_data_smithi098_135187_object72284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T10:06:01.708 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:22.602105+0000", 2024-09-21T10:06:01.708 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 99.039130055000001, 2024-09-21T10:06:01.708 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.070949458000000007, 2024-09-21T10:06:01.708 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:01.708 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:01.709 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:01.709 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:01.709 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:01.709 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 72285 2024-09-21T10:06:01.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.709 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:01.709 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.709 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:01.709 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602105+0000", 2024-09-21T10:06:01.709 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.709 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.709 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:01.709 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602105+0000", 2024-09-21T10:06:01.709 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.710 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.710 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:01.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602103+0000", 2024-09-21T10:06:01.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:06:01.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.710 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.710 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:01.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602166+0000", 2024-09-21T10:06:01.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2853000000000003e-05 2024-09-21T10:06:01.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.710 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.710 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:01.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602184+0000", 2024-09-21T10:06:01.711 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8277000000000001e-05 2024-09-21T10:06:01.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.711 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:01.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602190+0000", 2024-09-21T10:06:01.711 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.728e-06 2024-09-21T10:06:01.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.711 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:01.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602236+0000", 2024-09-21T10:06:01.711 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5731e-05 2024-09-21T10:06:01.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.712 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:01.712 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602347+0000", 2024-09-21T10:06:01.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000111406 2024-09-21T10:06:01.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.712 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.712 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:06:01.712 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602469+0000", 2024-09-21T10:06:01.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000121692 2024-09-21T10:06:01.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.712 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.712 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:01.712 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605799+0000", 2024-09-21T10:06:01.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003330614 2024-09-21T10:06:01.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:01.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.606938+0000", 2024-09-21T10:06:01.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011387299999999999 2024-09-21T10:06:01.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:01.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.672943+0000", 2024-09-21T10:06:01.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.066004878000000003 2024-09-21T10:06:01.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:01.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.673020+0000", 2024-09-21T10:06:01.714 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6600000000000005e-05 2024-09-21T10:06:01.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.714 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.714 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:01.714 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.673054+0000", 2024-09-21T10:06:01.714 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.447e-05 2024-09-21T10:06:01.714 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.714 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:01.714 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.714 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.714 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:72286 5.1 5:8e837b35:::benchmark_data_smithi098_135187_object72285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T10:06:01.714 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:22.602209+0000", 2024-09-21T10:06:01.715 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 99.039025726000006, 2024-09-21T10:06:01.715 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.072412780999999996, 2024-09-21T10:06:01.715 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:01.715 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:01.715 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:01.715 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:01.715 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:01.715 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 72286 2024-09-21T10:06:01.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.715 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:01.715 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.715 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:01.715 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602209+0000", 2024-09-21T10:06:01.715 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.716 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:01.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602209+0000", 2024-09-21T10:06:01.716 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.716 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:01.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602208+0000", 2024-09-21T10:06:01.716 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:01.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.716 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:01.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602247+0000", 2024-09-21T10:06:01.717 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8557999999999999e-05 2024-09-21T10:06:01.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.717 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.717 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:01.717 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602254+0000", 2024-09-21T10:06:01.717 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2269999999999996e-06 2024-09-21T10:06:01.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.717 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.717 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:01.717 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602258+0000", 2024-09-21T10:06:01.717 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2200000000000003e-06 2024-09-21T10:06:01.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.717 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.717 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:01.718 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.604534+0000", 2024-09-21T10:06:01.719 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022752480000000001 2024-09-21T10:06:01.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.719 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.719 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:01.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.604632+0000", 2024-09-21T10:06:01.719 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8021000000000002e-05 2024-09-21T10:06:01.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.719 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.719 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:06:01.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.604757+0000", 2024-09-21T10:06:01.719 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000124976 2024-09-21T10:06:01.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.719 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.720 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:01.720 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671018+0000", 2024-09-21T10:06:01.720 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.066261002999999999 2024-09-21T10:06:01.720 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.720 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.720 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:01.720 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671985+0000", 2024-09-21T10:06:01.720 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00096741700000000004 2024-09-21T10:06:01.720 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.720 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.720 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:01.720 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.674496+0000", 2024-09-21T10:06:01.720 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025111859999999999 2024-09-21T10:06:01.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.721 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.721 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:01.721 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.674585+0000", 2024-09-21T10:06:01.721 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8449999999999995e-05 2024-09-21T10:06:01.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.721 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.721 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:01.721 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.674622+0000", 2024-09-21T10:06:01.721 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6956999999999999e-05 2024-09-21T10:06:01.721 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.721 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:01.721 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.722 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.722 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:72291 5.9 e112/105 5:9924dcec:::benchmark_data_smithi098_135187_object72290:head v 112'4502, mlcod=112'4500)", 2024-09-21T10:06:01.722 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:22.602971+0000", 2024-09-21T10:06:01.722 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 99.038263342999997, 2024-09-21T10:06:01.722 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.068818678999999994, 2024-09-21T10:06:01.722 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:01.722 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:01.722 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:01.722 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.722 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:01.722 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602971+0000", 2024-09-21T10:06:01.722 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.722 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.723 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:01.723 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602971+0000", 2024-09-21T10:06:01.723 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.723 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.723 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.723 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:01.723 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602971+0000", 2024-09-21T10:06:01.723 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:01.723 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.723 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.723 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:01.723 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602995+0000", 2024-09-21T10:06:01.723 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4179999999999999e-05 2024-09-21T10:06:01.724 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.724 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.724 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:01.724 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.603005+0000", 2024-09-21T10:06:01.724 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7550000000000006e-06 2024-09-21T10:06:01.724 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.724 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.724 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:01.724 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.603010+0000", 2024-09-21T10:06:01.724 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2920000000000003e-06 2024-09-21T10:06:01.724 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.724 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.724 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:01.725 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.604043+0000", 2024-09-21T10:06:01.725 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010329320000000001 2024-09-21T10:06:01.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.725 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.725 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:01.725 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.604059+0000", 2024-09-21T10:06:01.725 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5687000000000001e-05 2024-09-21T10:06:01.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.725 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.725 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:01.725 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671616+0000", 2024-09-21T10:06:01.725 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.067557818000000006 2024-09-21T10:06:01.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.725 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.726 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:01.726 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671790+0000", 2024-09-21T10:06:01.726 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017359299999999999 2024-09-21T10:06:01.726 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.726 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:01.726 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.726 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.726 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.44894.0:72281 5.1 e112/105 ondisk, result = 0)", 2024-09-21T10:06:01.726 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:22.604691+0000", 2024-09-21T10:06:01.726 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 99.036543666, 2024-09-21T10:06:01.726 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.067102261999999996, 2024-09-21T10:06:01.726 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:01.727 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:06:01.727 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:01.727 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.727 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:01.727 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.604691+0000", 2024-09-21T10:06:01.727 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.727 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.727 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:01.727 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.604691+0000", 2024-09-21T10:06:01.727 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.727 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.727 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:01.728 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.604690+0000", 2024-09-21T10:06:01.728 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:06:01.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.728 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.728 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:01.728 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.604696+0000", 2024-09-21T10:06:01.728 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5300000000000004e-06 2024-09-21T10:06:01.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.728 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.728 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:01.728 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.604706+0000", 2024-09-21T10:06:01.728 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0754000000000001e-05 2024-09-21T10:06:01.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.728 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.729 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:01.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.604712+0000", 2024-09-21T10:06:01.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6169999999999998e-06 2024-09-21T10:06:01.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.729 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.729 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:01.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.670995+0000", 2024-09-21T10:06:01.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.066282674 2024-09-21T10:06:01.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.729 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.729 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:01.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671001+0000", 2024-09-21T10:06:01.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5320000000000003e-06 2024-09-21T10:06:01.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.730 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.730 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:01.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671793+0000", 2024-09-21T10:06:01.730 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00079192800000000001 2024-09-21T10:06:01.730 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.730 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:01.730 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.730 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.730 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:159902 5.e 5:78eb14c5:::benchmark_data_smithi098_135187_object24264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-21T10:06:01.730 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:05:46.468503+0000", 2024-09-21T10:06:01.730 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 15.172731441, 2024-09-21T10:06:01.730 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.101676892, 2024-09-21T10:06:01.731 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:01.731 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:01.731 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:01.731 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:01.731 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:01.731 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 159902 2024-09-21T10:06:01.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.731 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:01.731 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.731 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:01.731 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.468503+0000", 2024-09-21T10:06:01.731 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.731 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.732 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:01.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.468503+0000", 2024-09-21T10:06:01.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.732 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.732 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:01.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.468502+0000", 2024-09-21T10:06:01.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:06:01.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.732 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.732 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:01.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.468508+0000", 2024-09-21T10:06:01.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4579999999999998e-06 2024-09-21T10:06:01.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.733 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.733 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:01.733 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.468521+0000", 2024-09-21T10:06:01.733 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2959000000000001e-05 2024-09-21T10:06:01.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.733 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.733 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:01.733 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.468528+0000", 2024-09-21T10:06:01.733 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2779999999999999e-06 2024-09-21T10:06:01.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.733 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.733 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.470247+0000", 2024-09-21T10:06:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0017189250000000001 2024-09-21T10:06:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.470312+0000", 2024-09-21T10:06:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5331000000000002e-05 2024-09-21T10:06:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.470380+0000", 2024-09-21T10:06:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8127999999999996e-05 2024-09-21T10:06:01.734 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.476950+0000", 2024-09-21T10:06:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0065702089999999996 2024-09-21T10:06:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.477022+0000", 2024-09-21T10:06:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1369000000000001e-05 2024-09-21T10:06:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.570098+0000", 2024-09-21T10:06:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.093076753999999998 2024-09-21T10:06:01.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.736 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.736 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:01.736 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.570142+0000", 2024-09-21T10:06:01.736 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3690999999999999e-05 2024-09-21T10:06:01.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.736 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.736 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:01.736 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.570180+0000", 2024-09-21T10:06:01.736 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8009000000000001e-05 2024-09-21T10:06:01.736 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.736 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:01.736 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.737 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.737 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:159907 5.e 5:7ac47d85:::benchmark_data_smithi098_135187_object24269:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-21T10:06:01.737 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:05:46.470973+0000", 2024-09-21T10:06:01.737 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 15.170262064999999, 2024-09-21T10:06:01.737 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.099297015000000002, 2024-09-21T10:06:01.737 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:01.737 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:01.737 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:01.737 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:01.737 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:01.737 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 159907 2024-09-21T10:06:01.737 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.737 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:01.737 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.738 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:01.738 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.470973+0000", 2024-09-21T10:06:01.738 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.738 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.738 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.738 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:01.738 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.470973+0000", 2024-09-21T10:06:01.738 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.738 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.738 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.738 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:01.738 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.470971+0000", 2024-09-21T10:06:01.738 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:06:01.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.739 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.739 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:01.739 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.470977+0000", 2024-09-21T10:06:01.739 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6110000000000001e-06 2024-09-21T10:06:01.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.739 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.739 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:01.739 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.470984+0000", 2024-09-21T10:06:01.739 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7220000000000002e-06 2024-09-21T10:06:01.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.739 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.739 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:01.739 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.470988+0000", 2024-09-21T10:06:01.740 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1130000000000003e-06 2024-09-21T10:06:01.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.740 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.740 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:01.740 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.474027+0000", 2024-09-21T10:06:01.740 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030388160000000002 2024-09-21T10:06:01.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.740 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.740 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:01.740 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.474125+0000", 2024-09-21T10:06:01.740 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8542000000000006e-05 2024-09-21T10:06:01.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.740 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.740 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:01.741 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.474235+0000", 2024-09-21T10:06:01.741 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109357 2024-09-21T10:06:01.741 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.741 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.741 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:01.741 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.477089+0000", 2024-09-21T10:06:01.741 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0028548269999999999 2024-09-21T10:06:01.741 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.741 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.741 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:01.741 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.477702+0000", 2024-09-21T10:06:01.741 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00061254199999999999 2024-09-21T10:06:01.741 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.741 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.742 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:01.742 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.570211+0000", 2024-09-21T10:06:01.742 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.092508542999999999 2024-09-21T10:06:01.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.742 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.742 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:01.742 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.570243+0000", 2024-09-21T10:06:01.742 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1990999999999999e-05 2024-09-21T10:06:01.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.742 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.742 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:01.742 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.570270+0000", 2024-09-21T10:06:01.742 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7069000000000001e-05 2024-09-21T10:06:01.742 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.743 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:01.743 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.743 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.743 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:159910 5.0 e124/105 5:02d77b2f:::benchmark_data_smithi098_135187_object24272:head v 124'9833, mlcod=124'9831)", 2024-09-21T10:06:01.743 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:05:46.478655+0000", 2024-09-21T10:06:01.743 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 15.162579993, 2024-09-21T10:06:01.743 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.100181858, 2024-09-21T10:06:01.743 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:01.743 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:01.743 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:01.743 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.743 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:01.744 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.478655+0000", 2024-09-21T10:06:01.744 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.744 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.744 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.744 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:01.744 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.478655+0000", 2024-09-21T10:06:01.744 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.744 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.744 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.744 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:01.744 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.478654+0000", 2024-09-21T10:06:01.744 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:01.744 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.744 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.745 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:01.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.478661+0000", 2024-09-21T10:06:01.745 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1180000000000003e-06 2024-09-21T10:06:01.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.745 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.745 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:01.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.478669+0000", 2024-09-21T10:06:01.745 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1899999999999995e-06 2024-09-21T10:06:01.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.745 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.745 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:01.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.478673+0000", 2024-09-21T10:06:01.745 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.613e-06 2024-09-21T10:06:01.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.745 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.746 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:01.746 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.570051+0000", 2024-09-21T10:06:01.746 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.091378031999999998 2024-09-21T10:06:01.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.746 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.746 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:01.746 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.570085+0000", 2024-09-21T10:06:01.746 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3729000000000002e-05 2024-09-21T10:06:01.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.746 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.746 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:01.746 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.578794+0000", 2024-09-21T10:06:01.746 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0087097129999999991 2024-09-21T10:06:01.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.747 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.747 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:01.747 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.578837+0000", 2024-09-21T10:06:01.747 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2104000000000002e-05 2024-09-21T10:06:01.747 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.747 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:01.747 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.747 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.747 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.44894.0:159902 5.e e124/105 ondisk, result = 0)", 2024-09-21T10:06:01.747 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:05:46.480549+0000", 2024-09-21T10:06:01.747 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 15.160685368999999, 2024-09-21T10:06:01.747 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.089635825000000002, 2024-09-21T10:06:01.748 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:01.748 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:06:01.748 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:01.748 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.748 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:01.748 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.480549+0000", 2024-09-21T10:06:01.748 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.748 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.748 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:01.748 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.480549+0000", 2024-09-21T10:06:01.748 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.748 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.749 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:01.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.480549+0000", 2024-09-21T10:06:01.749 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:01.749 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.749 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.749 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:01.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.480554+0000", 2024-09-21T10:06:01.749 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0350000000000001e-06 2024-09-21T10:06:01.749 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.749 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.749 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:01.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.570029+0000", 2024-09-21T10:06:01.749 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.089475381000000007 2024-09-21T10:06:01.749 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.749 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.750 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:01.750 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.570041+0000", 2024-09-21T10:06:01.750 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1610000000000001e-05 2024-09-21T10:06:01.750 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.750 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.750 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:01.750 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.570084+0000", 2024-09-21T10:06:01.750 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3377999999999997e-05 2024-09-21T10:06:01.750 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.750 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.750 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:01.750 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.570092+0000", 2024-09-21T10:06:01.750 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6159999999999996e-06 2024-09-21T10:06:01.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.751 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.751 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:01.751 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.570185+0000", 2024-09-21T10:06:01.751 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3487999999999997e-05 2024-09-21T10:06:01.751 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.751 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:01.751 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.751 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.751 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:159918 5.9 e124/105 5:97164d6b:::benchmark_data_smithi098_135187_object24280:head v 124'9949, mlcod=124'9947)", 2024-09-21T10:06:01.751 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:05:46.486837+0000", 2024-09-21T10:06:01.751 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 15.154397612, 2024-09-21T10:06:01.751 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.10831666700000001, 2024-09-21T10:06:01.752 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:01.752 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:01.752 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:01.752 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.752 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:01.752 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.486837+0000", 2024-09-21T10:06:01.752 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.752 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.752 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:01.752 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.486837+0000", 2024-09-21T10:06:01.752 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:01.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.752 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.753 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:01.753 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.486836+0000", 2024-09-21T10:06:01.753 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:06:01.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.753 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.753 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:01.753 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.486843+0000", 2024-09-21T10:06:01.753 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9619999999999996e-06 2024-09-21T10:06:01.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.753 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.753 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:01.753 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.571521+0000", 2024-09-21T10:06:01.753 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.084678379999999998 2024-09-21T10:06:01.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.754 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.754 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:01.754 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.571529+0000", 2024-09-21T10:06:01.754 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8599999999999993e-06 2024-09-21T10:06:01.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.754 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.754 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:01.754 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.571941+0000", 2024-09-21T10:06:01.754 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00041216199999999999 2024-09-21T10:06:01.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.754 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.754 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:01.754 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.571962+0000", 2024-09-21T10:06:01.754 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0809e-05 2024-09-21T10:06:01.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.755 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.755 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:01.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.595119+0000", 2024-09-21T10:06:01.755 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.023157069999999998 2024-09-21T10:06:01.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:01.755 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:01.755 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:01.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.595154+0000", 2024-09-21T10:06:01.755 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4319999999999997e-05 2024-09-21T10:06:01.755 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.755 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:01.755 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.756 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:01.756 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:01.756 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:06:01.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:01.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:01.861 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:01.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:01.899 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:01.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:02.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:02.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:02.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:02.134 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:02.169 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:02.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:02.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:02.271 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:02.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:02.316 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:02.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:02.417 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:02.464 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:02.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:02.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:02.566 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:02.597 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:06:02.616 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:02.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:02.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:02.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:02.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:02.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:02.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:02 smithi098 ceph-mon[97580]: pgmap v329: 89 pgs: 89 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 854 op/s 2024-09-21T10:06:02.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:02 smithi098 ceph-mon[100914]: pgmap v329: 89 pgs: 89 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 854 op/s 2024-09-21T10:06:02.850 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:02.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:02 smithi196 ceph-mon[84784]: pgmap v329: 89 pgs: 89 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 854 op/s 2024-09-21T10:06:02.914 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:02.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:02.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:03.016 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:03.027 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:06:03.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:03.049 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:03.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:03.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:03.197 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:03.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:03.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:03.299 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:03.350 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:03.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:03.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:03.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:06:03.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:03.504 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:03.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:03.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:03.647 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:03.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:03.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:03.750 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:03.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:03.788 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:03.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:03.889 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:03.915 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:03.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:03.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:04.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:04.045 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:04.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:04.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:04.146 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:04.177 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:04.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:04.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:04.279 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:04.320 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:04.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:04.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:04.421 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:04.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:04.468 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:04.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:04.569 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:04.594 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:04.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:04.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:04.696 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:04.737 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:04.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:04.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:04.839 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:04.842 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:04 smithi196 ceph-mon[84784]: pgmap v330: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 788 op/s 2024-09-21T10:06:04.873 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:04.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:04.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:04.975 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:04.985 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:06:04.985 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:04 smithi098 ceph-mon[97580]: pgmap v330: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 788 op/s 2024-09-21T10:06:04.986 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:04 smithi098 ceph-mon[100914]: pgmap v330: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 788 op/s 2024-09-21T10:06:04.989 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:06:04.989 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:06:04.990 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:06:04.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.990 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:175444 5.3 e126/105 5:c7c8b4de:::benchmark_data_smithi098_135187_object39806:head v 126'10901, mlcod=126'10898)", 2024-09-21T10:06:04.990 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:06:04.968266+0000", 2024-09-21T10:06:04.990 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.016173758, 2024-09-21T10:06:04.990 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016204633999999999, 2024-09-21T10:06:04.990 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:04.990 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:06:04.990 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:04.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.990 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:04.990 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.968266+0000", 2024-09-21T10:06:04.990 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:04.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:04.991 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.991 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:04.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.968266+0000", 2024-09-21T10:06:04.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:04.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:04.991 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.991 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:04.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.968265+0000", 2024-09-21T10:06:04.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:06:04.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:04.991 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.991 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:04.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.968272+0000", 2024-09-21T10:06:04.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.759e-06 2024-09-21T10:06:04.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:04.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.992 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:04.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.968287+0000", 2024-09-21T10:06:04.992 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.596e-05 2024-09-21T10:06:04.992 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:04.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.992 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:04.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.968293+0000", 2024-09-21T10:06:04.992 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.772e-06 2024-09-21T10:06:04.992 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:04.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.992 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:04.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.969544+0000", 2024-09-21T10:06:04.993 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001250754 2024-09-21T10:06:04.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:04.993 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.993 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:04.993 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.969570+0000", 2024-09-21T10:06:04.993 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5531999999999999e-05 2024-09-21T10:06:04.993 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:04.993 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:04.993 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:04.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:04.993 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.993 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:175452 5.b e126/105 5:d0524228:::benchmark_data_smithi098_135187_object39814:head v 126'10852, mlcod=126'10850)", 2024-09-21T10:06:04.993 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:06:04.975969+0000", 2024-09-21T10:06:04.994 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0084702040000000003, 2024-09-21T10:06:04.994 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0085624120000000001, 2024-09-21T10:06:04.994 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:04.994 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:06:04.994 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:04.994 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.994 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:04.994 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.975969+0000", 2024-09-21T10:06:04.994 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:04.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:04.994 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.994 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:04.994 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.975969+0000", 2024-09-21T10:06:04.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:04.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:04.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.995 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:04.995 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.975968+0000", 2024-09-21T10:06:04.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:06:04.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:04.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.995 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:04.995 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.975979+0000", 2024-09-21T10:06:04.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0592e-05 2024-09-21T10:06:04.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:04.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.995 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:04.996 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.975988+0000", 2024-09-21T10:06:04.996 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0599999999999997e-06 2024-09-21T10:06:04.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:04.996 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.996 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:04.996 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.975995+0000", 2024-09-21T10:06:04.996 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4109999999999999e-06 2024-09-21T10:06:04.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:04.996 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.996 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:04.996 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.982941+0000", 2024-09-21T10:06:04.996 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0069455009999999998 2024-09-21T10:06:04.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:04.996 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.996 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:04.997 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.982986+0000", 2024-09-21T10:06:04.997 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4901000000000003e-05 2024-09-21T10:06:04.997 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:04.997 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:04.997 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:04.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:04.997 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.997 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:175451 5.d e126/105 5:b0e07427:::benchmark_data_smithi098_135187_object39813:head v 126'10887, mlcod=126'10884)", 2024-09-21T10:06:04.997 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:06:04.976006+0000", 2024-09-21T10:06:04.997 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0084336229999999995, 2024-09-21T10:06:04.997 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0085795699999999999, 2024-09-21T10:06:04.997 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:04.997 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:06:04.997 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:04.997 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.997 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:04.998 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.976006+0000", 2024-09-21T10:06:04.998 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:04.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:04.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.998 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:04.998 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.976006+0000", 2024-09-21T10:06:04.998 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:04.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:04.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.998 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:04.998 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.976005+0000", 2024-09-21T10:06:04.998 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:04.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:04.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:04.999 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.976009+0000", 2024-09-21T10:06:04.999 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4460000000000001e-06 2024-09-21T10:06:04.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:04.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:04.999 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.976014+0000", 2024-09-21T10:06:04.999 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0860000000000001e-06 2024-09-21T10:06:04.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:04.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:04.999 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.976018+0000", 2024-09-21T10:06:04.999 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7929999999999999e-06 2024-09-21T10:06:04.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:04.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:04.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:05.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.976074+0000", 2024-09-21T10:06:05.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6617999999999999e-05 2024-09-21T10:06:05.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:05.000 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:05.000 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:05.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.976094+0000", 2024-09-21T10:06:05.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9791000000000001e-05 2024-09-21T10:06:05.000 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:05.000 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:05.000 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:05.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:05.000 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:05.000 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:175461 5.f 5:f09954ae:::benchmark_data_smithi098_135187_object39823:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T10:06:05.000 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:06:04.980533+0000", 2024-09-21T10:06:05.000 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0039066470000000001, 2024-09-21T10:06:05.001 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041110840000000001, 2024-09-21T10:06:05.001 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:05.001 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:06:05.001 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:05.001 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:05.001 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:05.001 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 175461 2024-09-21T10:06:05.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:05.001 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:05.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:05.001 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:05.001 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.980533+0000", 2024-09-21T10:06:05.001 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:05.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:05.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:05.002 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:05.002 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.980533+0000", 2024-09-21T10:06:05.002 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:05.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:05.002 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:05.002 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:05.002 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.980531+0000", 2024-09-21T10:06:05.002 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:06:05.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:05.002 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:05.002 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:05.002 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.980538+0000", 2024-09-21T10:06:05.002 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6619999999999999e-06 2024-09-21T10:06:05.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:05.002 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:05.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:05.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.980550+0000", 2024-09-21T10:06:05.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2248e-05 2024-09-21T10:06:05.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:05.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:05.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:05.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.980556+0000", 2024-09-21T10:06:05.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8200000000000002e-06 2024-09-21T10:06:05.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:05.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:05.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:05.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.980717+0000", 2024-09-21T10:06:05.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016149400000000001 2024-09-21T10:06:05.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:05.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:05.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:05.004 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.980817+0000", 2024-09-21T10:06:05.004 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9780999999999999e-05 2024-09-21T10:06:05.004 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:05.004 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:05.004 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:06:05.004 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:04.980925+0000", 2024-09-21T10:06:05.004 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010764000000000001 2024-09-21T10:06:05.004 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:05.004 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:05.004 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:05.004 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:05.004 INFO:teuthology.orchestra.run.smithi196.stdout: ], 2024-09-21T10:06:05.004 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 4 2024-09-21T10:06:05.004 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:06:05.017 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:05.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:05.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:05.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:05.149 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:05.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:05.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:05.251 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:05.281 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:05.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:05.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:05.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:05.411 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:05.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:05.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:05.512 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:05.542 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:05.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:05.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:05.643 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:05.686 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:05.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:05.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:05.757 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:05 smithi196 ceph-mon[84784]: from='client.45047 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:06:05.788 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:05.798 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:06:05.823 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:05.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:05.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:05.925 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:05.954 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:05.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:05.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:06.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:06.080 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:06.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:06.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:06.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:05 smithi098 ceph-mon[97580]: from='client.45047 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:06:06.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:05 smithi098 ceph-mon[100914]: from='client.45047 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:06:06.182 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:06.227 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:06.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:06.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:06.256 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:06:06.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:06.371 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:06.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:06.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:06.473 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:06.524 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:06.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:06.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:06.626 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:06.654 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:06.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:06.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:06.694 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:06 smithi196 ceph-mon[84784]: pgmap v331: 89 pgs: 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 823 op/s 2024-09-21T10:06:06.756 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:06.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:06.798 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:06.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:06.810 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2024-09-21T10:06:03.077100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6580017125,"num_objects":100359,"num_object_clones":0,"num_object_copies":301077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100359,"num_whiteouts":0,"num_read":6290,"num_read_kb":13766,"num_write":171997,"num_write_kb":8694892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147273,"ondisk_log_size":147273,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":20883264,"kb_used_data":19294904,"kb_used_omap":35,"kb_used_meta":1588060,"kb_avail":729077952,"statfs":{"total":767960285184,"available":746575822848,"internally_reserved":0,"allocated":19757981696,"data_stored":19752889470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36353,"internal_metadata":1626173951},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-516816896,"num_objects":-7886,"num_object_clones":0,"num_object_copies":-23658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7886,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002504"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":118,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596725+0000","last_change":"2024-09-21T10:02:14.097991+0000","last_active":"2024-09-21T10:05:55.596725+0000","last_peered":"2024-09-21T10:05:55.596725+0000","last_clean":"2024-09-21T10:05:55.596725+0000","last_became_active":"2024-09-21T10:02:14.097407+0000","last_became_peered":"2024-09-21T10:02:14.097407+0000","last_unstale":"2024-09-21T10:05:55.596725+0000","last_undegraded":"2024-09-21T10:05:55.596725+0000","last_fullsized":"2024-09-21T10:05:55.596725+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:18:42.098886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":93,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.522521+0000","last_change":"2024-09-21T10:01:41.360784+0000","last_active":"2024-09-21T10:05:46.522521+0000","last_peered":"2024-09-21T10:05:46.522521+0000","last_clean":"2024-09-21T10:05:46.522521+0000","last_became_active":"2024-09-21T10:01:41.360462+0000","last_became_peered":"2024-09-21T10:01:41.360462+0000","last_unstale":"2024-09-21T10:05:46.522521+0000","last_undegraded":"2024-09-21T10:05:46.522521+0000","last_fullsized":"2024-09-21T10:05:46.522521+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:11.867275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":89,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.571725+0000","last_change":"2024-09-21T10:02:14.103373+0000","last_active":"2024-09-21T10:05:46.571725+0000","last_peered":"2024-09-21T10:05:46.571725+0000","last_clean":"2024-09-21T10:05:46.571725+0000","last_became_active":"2024-09-21T10:02:14.102985+0000","last_became_peered":"2024-09-21T10:02:14.102985+0000","last_unstale":"2024-09-21T10:05:46.571725+0000","last_undegraded":"2024-09-21T10:05:46.571725+0000","last_fullsized":"2024-09-21T10:05:46.571725+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:03.418174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":114,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.592903+0000","last_change":"2024-09-21T10:02:14.098949+0000","last_active":"2024-09-21T10:05:55.592903+0000","last_peered":"2024-09-21T10:05:55.592903+0000","last_clean":"2024-09-21T10:05:55.592903+0000","last_became_active":"2024-09-21T10:02:14.098646+0000","last_became_peered":"2024-09-21T10:02:14.098646+0000","last_unstale":"2024-09-21T10:05:55.592903+0000","last_undegraded":"2024-09-21T10:05:55.592903+0000","last_fullsized":"2024-09-21T10:05:55.592903+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:50:56.862529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":114,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.593285+0000","last_change":"2024-09-21T10:02:14.098205+0000","last_active":"2024-09-21T10:05:55.593285+0000","last_peered":"2024-09-21T10:05:55.593285+0000","last_clean":"2024-09-21T10:05:55.593285+0000","last_became_active":"2024-09-21T10:02:14.097911+0000","last_became_peered":"2024-09-21T10:02:14.097911+0000","last_unstale":"2024-09-21T10:05:55.593285+0000","last_undegraded":"2024-09-21T10:05:55.593285+0000","last_fullsized":"2024-09-21T10:05:55.593285+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:55.971191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":74,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.569667+0000","last_change":"2024-09-21T09:52:20.371775+0000","last_active":"2024-09-21T10:05:46.569667+0000","last_peered":"2024-09-21T10:05:46.569667+0000","last_clean":"2024-09-21T10:05:46.569667+0000","last_became_active":"2024-09-21T09:52:20.371509+0000","last_became_peered":"2024-09-21T09:52:20.371509+0000","last_unstale":"2024-09-21T10:05:46.569667+0000","last_undegraded":"2024-09-21T10:05:46.569667+0000","last_fullsized":"2024-09-21T10:05:46.569667+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:43:21.822524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":118,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596979+0000","last_change":"2024-09-21T10:02:14.097817+0000","last_active":"2024-09-21T10:05:55.596979+0000","last_peered":"2024-09-21T10:05:55.596979+0000","last_clean":"2024-09-21T10:05:55.596979+0000","last_became_active":"2024-09-21T10:02:14.097466+0000","last_became_peered":"2024-09-21T10:02:14.097466+0000","last_unstale":"2024-09-21T10:05:55.596979+0000","last_undegraded":"2024-09-21T10:05:55.596979+0000","last_fullsized":"2024-09-21T10:05:55.596979+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:57.883118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":99,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.576486+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:05:46.576486+0000","last_peered":"2024-09-21T10:05:46.576486+0000","last_clean":"2024-09-21T10:05:46.576486+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:05:46.576486+0000","last_undegraded":"2024-09-21T10:05:46.576486+0000","last_fullsized":"2024-09-21T10:05:46.576486+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"126'10551","reported_seq":10580,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:58.029928+0000","last_change":"2024-09-21T10:03:41.940189+0000","last_active":"2024-09-21T10:05:58.029928+0000","last_peered":"2024-09-21T10:05:58.029928+0000","last_clean":"2024-09-21T10:05:58.029928+0000","last_became_active":"2024-09-21T10:03:41.939900+0000","last_became_peered":"2024-09-21T10:03:41.939900+0000","last_unstale":"2024-09-21T10:05:58.029928+0000","last_undegraded":"2024-09-21T10:05:58.029928+0000","last_fullsized":"2024-09-21T10:05:58.029928+0000","mapping_epoch":105,"log_start":"110'1900","ondisk_log_start":"110'1900","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":8651,"log_dups_size":0,"ondisk_log_size":8651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:13:00.441130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414384128,"num_objects":6323,"num_object_clones":0,"num_object_copies":18969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10551,"num_write_kb":539968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":92,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.570723+0000","last_change":"2024-09-21T09:59:54.265455+0000","last_active":"2024-09-21T10:05:46.570723+0000","last_peered":"2024-09-21T10:05:46.570723+0000","last_clean":"2024-09-21T10:05:46.570723+0000","last_became_active":"2024-09-21T09:59:54.265110+0000","last_became_peered":"2024-09-21T09:59:54.265110+0000","last_unstale":"2024-09-21T10:05:46.570723+0000","last_undegraded":"2024-09-21T10:05:46.570723+0000","last_fullsized":"2024-09-21T10:05:46.570723+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:04:13.404874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":72,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.571155+0000","last_change":"2024-09-21T09:52:20.402753+0000","last_active":"2024-09-21T10:05:46.571155+0000","last_peered":"2024-09-21T10:05:46.571155+0000","last_clean":"2024-09-21T10:05:46.571155+0000","last_became_active":"2024-09-21T09:52:20.402462+0000","last_became_peered":"2024-09-21T09:52:20.402462+0000","last_unstale":"2024-09-21T10:05:46.571155+0000","last_undegraded":"2024-09-21T10:05:46.571155+0000","last_fullsized":"2024-09-21T10:05:46.571155+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:14.036869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"126'10615","reported_seq":10668,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:59.201170+0000","last_change":"2024-09-21T10:03:42.340396+0000","last_active":"2024-09-21T10:05:59.201170+0000","last_peered":"2024-09-21T10:05:59.201170+0000","last_clean":"2024-09-21T10:05:59.201170+0000","last_became_active":"2024-09-21T10:03:42.340086+0000","last_became_peered":"2024-09-21T10:03:42.340086+0000","last_unstale":"2024-09-21T10:05:59.201170+0000","last_undegraded":"2024-09-21T10:05:59.201170+0000","last_fullsized":"2024-09-21T10:05:59.201170+0000","mapping_epoch":105,"log_start":"109'600","ondisk_log_start":"109'600","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:35:46.959831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412811264,"num_objects":6299,"num_object_clones":0,"num_object_copies":18897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10615,"num_write_kb":541248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":93,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.570625+0000","last_change":"2024-09-21T10:01:07.420484+0000","last_active":"2024-09-21T10:05:46.570625+0000","last_peered":"2024-09-21T10:05:46.570625+0000","last_clean":"2024-09-21T10:05:46.570625+0000","last_became_active":"2024-09-21T10:01:07.420135+0000","last_became_peered":"2024-09-21T10:01:07.420135+0000","last_unstale":"2024-09-21T10:05:46.570625+0000","last_undegraded":"2024-09-21T10:05:46.570625+0000","last_fullsized":"2024-09-21T10:05:46.570625+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:33:10.241010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":107,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.595369+0000","last_change":"2024-09-21T10:02:14.098139+0000","last_active":"2024-09-21T10:05:46.595369+0000","last_peered":"2024-09-21T10:05:46.595369+0000","last_clean":"2024-09-21T10:05:46.595369+0000","last_became_active":"2024-09-21T10:02:14.097639+0000","last_became_peered":"2024-09-21T10:02:14.097639+0000","last_unstale":"2024-09-21T10:05:46.595369+0000","last_undegraded":"2024-09-21T10:05:46.595369+0000","last_fullsized":"2024-09-21T10:05:46.595369+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:11:26.925388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"126'10560","reported_seq":10613,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:59.250086+0000","last_change":"2024-09-21T10:03:41.941408+0000","last_active":"2024-09-21T10:05:59.250086+0000","last_peered":"2024-09-21T10:05:59.250086+0000","last_clean":"2024-09-21T10:05:59.250086+0000","last_became_active":"2024-09-21T10:03:41.941084+0000","last_became_peered":"2024-09-21T10:03:41.941084+0000","last_unstale":"2024-09-21T10:05:59.250086+0000","last_undegraded":"2024-09-21T10:05:59.250086+0000","last_fullsized":"2024-09-21T10:05:59.250086+0000","mapping_epoch":105,"log_start":"109'500","ondisk_log_start":"109'500","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:33:47.243861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408813568,"num_objects":6238,"num_object_clones":0,"num_object_copies":18714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10560,"num_write_kb":537536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":75,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.569897+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:05:46.569897+0000","last_peered":"2024-09-21T10:05:46.569897+0000","last_clean":"2024-09-21T10:05:46.569897+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:05:46.569897+0000","last_undegraded":"2024-09-21T10:05:46.569897+0000","last_fullsized":"2024-09-21T10:05:46.569897+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":90,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.576832+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:05:46.576832+0000","last_peered":"2024-09-21T10:05:46.576832+0000","last_clean":"2024-09-21T10:05:46.576832+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:05:46.576832+0000","last_undegraded":"2024-09-21T10:05:46.576832+0000","last_fullsized":"2024-09-21T10:05:46.576832+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"126'10637","reported_seq":10690,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:59.249070+0000","last_change":"2024-09-21T10:03:41.939951+0000","last_active":"2024-09-21T10:05:59.249070+0000","last_peered":"2024-09-21T10:05:59.249070+0000","last_clean":"2024-09-21T10:05:59.249070+0000","last_became_active":"2024-09-21T10:03:41.939644+0000","last_became_peered":"2024-09-21T10:03:41.939644+0000","last_unstale":"2024-09-21T10:05:59.249070+0000","last_undegraded":"2024-09-21T10:05:59.249070+0000","last_fullsized":"2024-09-21T10:05:59.249070+0000","mapping_epoch":105,"log_start":"109'600","ondisk_log_start":"109'600","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:28:55.215930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408748032,"num_objects":6237,"num_object_clones":0,"num_object_copies":18711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10637,"num_write_kb":539968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":90,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.571659+0000","last_change":"2024-09-21T10:02:14.104212+0000","last_active":"2024-09-21T10:05:46.571659+0000","last_peered":"2024-09-21T10:05:46.571659+0000","last_clean":"2024-09-21T10:05:46.571659+0000","last_became_active":"2024-09-21T10:02:14.103893+0000","last_became_peered":"2024-09-21T10:02:14.103893+0000","last_unstale":"2024-09-21T10:05:46.571659+0000","last_undegraded":"2024-09-21T10:05:46.571659+0000","last_fullsized":"2024-09-21T10:05:46.571659+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:59:21.191412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":114,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.600256+0000","last_change":"2024-09-21T10:01:07.419715+0000","last_active":"2024-09-21T10:05:55.600256+0000","last_peered":"2024-09-21T10:05:55.600256+0000","last_clean":"2024-09-21T10:05:55.600256+0000","last_became_active":"2024-09-21T10:01:07.419422+0000","last_became_peered":"2024-09-21T10:01:07.419422+0000","last_unstale":"2024-09-21T10:05:55.600256+0000","last_undegraded":"2024-09-21T10:05:55.600256+0000","last_fullsized":"2024-09-21T10:05:55.600256+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:36.424289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"126'10590","reported_seq":10644,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:59.250801+0000","last_change":"2024-09-21T10:03:41.942958+0000","last_active":"2024-09-21T10:05:59.250801+0000","last_peered":"2024-09-21T10:05:59.250801+0000","last_clean":"2024-09-21T10:05:59.250801+0000","last_became_active":"2024-09-21T10:03:41.942656+0000","last_became_peered":"2024-09-21T10:03:41.942656+0000","last_unstale":"2024-09-21T10:05:59.250801+0000","last_undegraded":"2024-09-21T10:05:59.250801+0000","last_fullsized":"2024-09-21T10:05:59.250801+0000","mapping_epoch":105,"log_start":"109'500","ondisk_log_start":"109'500","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:36:35.630043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410583064,"num_objects":6266,"num_object_clones":0,"num_object_copies":18798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6266,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10590,"num_write_kb":539329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":116,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596599+0000","last_change":"2024-09-21T10:02:14.096537+0000","last_active":"2024-09-21T10:05:55.596599+0000","last_peered":"2024-09-21T10:05:55.596599+0000","last_clean":"2024-09-21T10:05:55.596599+0000","last_became_active":"2024-09-21T10:02:14.096166+0000","last_became_peered":"2024-09-21T10:02:14.096166+0000","last_unstale":"2024-09-21T10:05:55.596599+0000","last_undegraded":"2024-09-21T10:05:55.596599+0000","last_fullsized":"2024-09-21T10:05:55.596599+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:54:18.927732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":90,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.590946+0000","last_change":"2024-09-21T10:02:14.099205+0000","last_active":"2024-09-21T10:05:46.590946+0000","last_peered":"2024-09-21T10:05:46.590946+0000","last_clean":"2024-09-21T10:05:46.590946+0000","last_became_active":"2024-09-21T10:02:14.098598+0000","last_became_peered":"2024-09-21T10:02:14.098598+0000","last_unstale":"2024-09-21T10:05:46.590946+0000","last_undegraded":"2024-09-21T10:05:46.590946+0000","last_fullsized":"2024-09-21T10:05:46.590946+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:46.424612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"126'10440","reported_seq":10469,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:58.083509+0000","last_change":"2024-09-21T10:03:42.340737+0000","last_active":"2024-09-21T10:05:58.083509+0000","last_peered":"2024-09-21T10:05:58.083509+0000","last_clean":"2024-09-21T10:05:58.083509+0000","last_became_active":"2024-09-21T10:03:42.340445+0000","last_became_peered":"2024-09-21T10:03:42.340445+0000","last_unstale":"2024-09-21T10:05:58.083509+0000","last_undegraded":"2024-09-21T10:05:58.083509+0000","last_fullsized":"2024-09-21T10:05:58.083509+0000","mapping_epoch":105,"log_start":"110'3100","ondisk_log_start":"110'3100","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":7340,"log_dups_size":0,"ondisk_log_size":7340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:18.067171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415367168,"num_objects":6338,"num_object_clones":0,"num_object_copies":19014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10440,"num_write_kb":536896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":91,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.576977+0000","last_change":"2024-09-21T10:01:07.424883+0000","last_active":"2024-09-21T10:05:46.576977+0000","last_peered":"2024-09-21T10:05:46.576977+0000","last_clean":"2024-09-21T10:05:46.576977+0000","last_became_active":"2024-09-21T10:01:07.424499+0000","last_became_peered":"2024-09-21T10:01:07.424499+0000","last_unstale":"2024-09-21T10:05:46.576977+0000","last_undegraded":"2024-09-21T10:05:46.576977+0000","last_fullsized":"2024-09-21T10:05:46.576977+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:46.101570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":107,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.597394+0000","last_change":"2024-09-21T10:01:41.362291+0000","last_active":"2024-09-21T10:05:46.597394+0000","last_peered":"2024-09-21T10:05:46.597394+0000","last_clean":"2024-09-21T10:05:46.597394+0000","last_became_active":"2024-09-21T10:01:41.361954+0000","last_became_peered":"2024-09-21T10:01:41.361954+0000","last_unstale":"2024-09-21T10:05:46.597394+0000","last_undegraded":"2024-09-21T10:05:46.597394+0000","last_fullsized":"2024-09-21T10:05:46.597394+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:57.669671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"126'10680","reported_seq":10705,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:58.119772+0000","last_change":"2024-09-21T10:03:41.942110+0000","last_active":"2024-09-21T10:05:58.119772+0000","last_peered":"2024-09-21T10:05:58.119772+0000","last_clean":"2024-09-21T10:05:58.119772+0000","last_became_active":"2024-09-21T10:03:41.941789+0000","last_became_peered":"2024-09-21T10:03:41.941789+0000","last_unstale":"2024-09-21T10:05:58.119772+0000","last_undegraded":"2024-09-21T10:05:58.119772+0000","last_fullsized":"2024-09-21T10:05:58.119772+0000","mapping_epoch":105,"log_start":"109'600","ondisk_log_start":"109'600","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:33:04.384003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416940032,"num_objects":6362,"num_object_clones":0,"num_object_copies":19086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10680,"num_write_kb":545344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":91,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.595430+0000","last_change":"2024-09-21T10:01:41.361244+0000","last_active":"2024-09-21T10:05:46.595430+0000","last_peered":"2024-09-21T10:05:46.595430+0000","last_clean":"2024-09-21T10:05:46.595430+0000","last_became_active":"2024-09-21T10:01:41.360645+0000","last_became_peered":"2024-09-21T10:01:41.360645+0000","last_unstale":"2024-09-21T10:05:46.595430+0000","last_undegraded":"2024-09-21T10:05:46.595430+0000","last_fullsized":"2024-09-21T10:05:46.595430+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:07.604991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":90,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.590848+0000","last_change":"2024-09-21T10:01:07.420824+0000","last_active":"2024-09-21T10:05:46.590848+0000","last_peered":"2024-09-21T10:05:46.590848+0000","last_clean":"2024-09-21T10:05:46.590848+0000","last_became_active":"2024-09-21T10:01:07.420387+0000","last_became_peered":"2024-09-21T10:01:07.420387+0000","last_unstale":"2024-09-21T10:05:46.590848+0000","last_undegraded":"2024-09-21T10:05:46.590848+0000","last_fullsized":"2024-09-21T10:05:46.590848+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:48.875842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"126'10799","reported_seq":10828,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:58.099748+0000","last_change":"2024-09-21T10:03:41.941262+0000","last_active":"2024-09-21T10:05:58.099748+0000","last_peered":"2024-09-21T10:05:58.099748+0000","last_clean":"2024-09-21T10:05:58.099748+0000","last_became_active":"2024-09-21T10:03:41.940968+0000","last_became_peered":"2024-09-21T10:03:41.940968+0000","last_unstale":"2024-09-21T10:05:58.099748+0000","last_undegraded":"2024-09-21T10:05:58.099748+0000","last_fullsized":"2024-09-21T10:05:58.099748+0000","mapping_epoch":105,"log_start":"111'3400","ondisk_log_start":"111'3400","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":7399,"log_dups_size":0,"ondisk_log_size":7399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:47.483647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423034880,"num_objects":6455,"num_object_clones":0,"num_object_copies":19365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10799,"num_write_kb":552128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":119,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596541+0000","last_change":"2024-09-21T10:02:14.097462+0000","last_active":"2024-09-21T10:05:55.596541+0000","last_peered":"2024-09-21T10:05:55.596541+0000","last_clean":"2024-09-21T10:05:55.596541+0000","last_became_active":"2024-09-21T10:02:14.097167+0000","last_became_peered":"2024-09-21T10:02:14.097167+0000","last_unstale":"2024-09-21T10:05:55.596541+0000","last_undegraded":"2024-09-21T10:05:55.596541+0000","last_fullsized":"2024-09-21T10:05:55.596541+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:54.922731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":104,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.599701+0000","last_change":"2024-09-21T10:02:14.097676+0000","last_active":"2024-09-21T10:05:55.599701+0000","last_peered":"2024-09-21T10:05:55.599701+0000","last_clean":"2024-09-21T10:05:55.599701+0000","last_became_active":"2024-09-21T10:02:14.097391+0000","last_became_peered":"2024-09-21T10:02:14.097391+0000","last_unstale":"2024-09-21T10:05:55.599701+0000","last_undegraded":"2024-09-21T10:05:55.599701+0000","last_fullsized":"2024-09-21T10:05:55.599701+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:40:24.523178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"126'10851","reported_seq":10880,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:58.035091+0000","last_change":"2024-09-21T10:03:41.940655+0000","last_active":"2024-09-21T10:05:58.035091+0000","last_peered":"2024-09-21T10:05:58.035091+0000","last_clean":"2024-09-21T10:05:58.035091+0000","last_became_active":"2024-09-21T10:03:41.940058+0000","last_became_peered":"2024-09-21T10:03:41.940058+0000","last_unstale":"2024-09-21T10:05:58.035091+0000","last_undegraded":"2024-09-21T10:05:58.035091+0000","last_fullsized":"2024-09-21T10:05:58.035091+0000","mapping_epoch":105,"log_start":"110'2200","ondisk_log_start":"110'2200","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":8651,"log_dups_size":0,"ondisk_log_size":8651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:00:40.090535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430768128,"num_objects":6573,"num_object_clones":0,"num_object_copies":19719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10851,"num_write_kb":557568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":208,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.570793+0000","last_change":"2024-09-21T09:59:54.265556+0000","last_active":"2024-09-21T10:05:46.570793+0000","last_peered":"2024-09-21T10:05:46.570793+0000","last_clean":"2024-09-21T10:05:46.570793+0000","last_became_active":"2024-09-21T09:59:54.265033+0000","last_became_peered":"2024-09-21T09:59:54.265033+0000","last_unstale":"2024-09-21T10:05:46.570793+0000","last_undegraded":"2024-09-21T10:05:46.570793+0000","last_fullsized":"2024-09-21T10:05:46.570793+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T09:48:34.850872+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:40.449756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":514,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.575591+0000","last_change":"2024-09-21T10:01:41.361089+0000","last_active":"2024-09-21T10:05:46.575591+0000","last_peered":"2024-09-21T10:05:46.575591+0000","last_clean":"2024-09-21T10:05:46.575591+0000","last_became_active":"2024-09-21T10:01:41.360658+0000","last_became_peered":"2024-09-21T10:01:41.360658+0000","last_unstale":"2024-09-21T10:05:46.575591+0000","last_undegraded":"2024-09-21T10:05:46.575591+0000","last_fullsized":"2024-09-21T10:05:46.575591+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:58:06.840024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":482,"num_read_kb":1581,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":72,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596635+0000","last_change":"2024-09-21T10:02:14.099961+0000","last_active":"2024-09-21T10:05:55.596635+0000","last_peered":"2024-09-21T10:05:55.596635+0000","last_clean":"2024-09-21T10:05:55.596635+0000","last_became_active":"2024-09-21T10:02:14.099067+0000","last_became_peered":"2024-09-21T10:02:14.099067+0000","last_unstale":"2024-09-21T10:05:55.596635+0000","last_undegraded":"2024-09-21T10:05:55.596635+0000","last_fullsized":"2024-09-21T10:05:55.596635+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:41.037532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":73,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.577049+0000","last_change":"2024-09-21T09:52:20.403534+0000","last_active":"2024-09-21T10:05:46.577049+0000","last_peered":"2024-09-21T10:05:46.577049+0000","last_clean":"2024-09-21T10:05:46.577049+0000","last_became_active":"2024-09-21T09:52:20.403258+0000","last_became_peered":"2024-09-21T09:52:20.403258+0000","last_unstale":"2024-09-21T10:05:46.577049+0000","last_undegraded":"2024-09-21T10:05:46.577049+0000","last_fullsized":"2024-09-21T10:05:46.577049+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:55.374343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"126'10544","reported_seq":10572,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:58.078606+0000","last_change":"2024-09-21T10:03:42.340526+0000","last_active":"2024-09-21T10:05:58.078606+0000","last_peered":"2024-09-21T10:05:58.078606+0000","last_clean":"2024-09-21T10:05:58.078606+0000","last_became_active":"2024-09-21T10:03:42.340207+0000","last_became_peered":"2024-09-21T10:03:42.340207+0000","last_unstale":"2024-09-21T10:05:58.078606+0000","last_undegraded":"2024-09-21T10:05:58.078606+0000","last_fullsized":"2024-09-21T10:05:58.078606+0000","mapping_epoch":105,"log_start":"109'500","ondisk_log_start":"109'500","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:02.279418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415105024,"num_objects":6334,"num_object_clones":0,"num_object_copies":19002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10544,"num_write_kb":540096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":471,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.522520+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:05:46.522520+0000","last_peered":"2024-09-21T10:05:46.522520+0000","last_clean":"2024-09-21T10:05:46.522520+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:05:46.522520+0000","last_undegraded":"2024-09-21T10:05:46.522520+0000","last_fullsized":"2024-09-21T10:05:46.522520+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":114,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.592813+0000","last_change":"2024-09-21T10:02:14.097097+0000","last_active":"2024-09-21T10:05:55.592813+0000","last_peered":"2024-09-21T10:05:55.592813+0000","last_clean":"2024-09-21T10:05:55.592813+0000","last_became_active":"2024-09-21T10:02:14.096788+0000","last_became_peered":"2024-09-21T10:02:14.096788+0000","last_unstale":"2024-09-21T10:05:55.592813+0000","last_undegraded":"2024-09-21T10:05:55.592813+0000","last_fullsized":"2024-09-21T10:05:55.592813+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:05:29.711020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":125,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.593357+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:05:55.593357+0000","last_peered":"2024-09-21T10:05:55.593357+0000","last_clean":"2024-09-21T10:05:55.593357+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:05:55.593357+0000","last_undegraded":"2024-09-21T10:05:55.593357+0000","last_fullsized":"2024-09-21T10:05:55.593357+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"126'10634","reported_seq":10687,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:59.196335+0000","last_change":"2024-09-21T10:03:41.943228+0000","last_active":"2024-09-21T10:05:59.196335+0000","last_peered":"2024-09-21T10:05:59.196335+0000","last_clean":"2024-09-21T10:05:59.196335+0000","last_became_active":"2024-09-21T10:03:41.942690+0000","last_became_peered":"2024-09-21T10:03:41.942690+0000","last_unstale":"2024-09-21T10:05:59.196335+0000","last_undegraded":"2024-09-21T10:05:59.196335+0000","last_fullsized":"2024-09-21T10:05:59.196335+0000","mapping_epoch":105,"log_start":"109'600","ondisk_log_start":"109'600","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:32:01.612946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407109632,"num_objects":6212,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10634,"num_write_kb":539072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":864,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.591975+0000","last_change":"2024-09-21T10:02:14.100143+0000","last_active":"2024-09-21T10:05:55.591975+0000","last_peered":"2024-09-21T10:05:55.591975+0000","last_clean":"2024-09-21T10:05:55.591975+0000","last_became_active":"2024-09-21T10:02:14.099586+0000","last_became_peered":"2024-09-21T10:02:14.099586+0000","last_unstale":"2024-09-21T10:05:55.591975+0000","last_undegraded":"2024-09-21T10:05:55.591975+0000","last_fullsized":"2024-09-21T10:05:55.591975+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:34.064652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":91,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.592144+0000","last_change":"2024-09-21T10:03:39.670431+0000","last_active":"2024-09-21T10:05:55.592144+0000","last_peered":"2024-09-21T10:05:55.592144+0000","last_clean":"2024-09-21T10:05:55.592144+0000","last_became_active":"2024-09-21T10:03:39.670088+0000","last_became_peered":"2024-09-21T10:03:39.670088+0000","last_unstale":"2024-09-21T10:05:55.592144+0000","last_undegraded":"2024-09-21T10:05:55.592144+0000","last_fullsized":"2024-09-21T10:05:55.592144+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:30:52.512603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":121,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.592021+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:05:55.592021+0000","last_peered":"2024-09-21T10:05:55.592021+0000","last_clean":"2024-09-21T10:05:55.592021+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:05:55.592021+0000","last_undegraded":"2024-09-21T10:05:55.592021+0000","last_fullsized":"2024-09-21T10:05:55.592021+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"126'11079","reported_seq":11133,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:06:02.109916+0000","last_change":"2024-09-21T10:03:42.341388+0000","last_active":"2024-09-21T10:06:02.109916+0000","last_peered":"2024-09-21T10:06:02.109916+0000","last_clean":"2024-09-21T10:06:02.109916+0000","last_became_active":"2024-09-21T10:03:42.340274+0000","last_became_peered":"2024-09-21T10:03:42.340274+0000","last_unstale":"2024-09-21T10:06:02.109916+0000","last_undegraded":"2024-09-21T10:06:02.109916+0000","last_fullsized":"2024-09-21T10:06:02.109916+0000","mapping_epoch":105,"log_start":"110'2200","ondisk_log_start":"110'2200","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":8879,"log_dups_size":0,"ondisk_log_size":8879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:47.666420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403898368,"num_objects":6163,"num_object_clones":0,"num_object_copies":18489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11079,"num_write_kb":551744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":698,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.571402+0000","last_change":"2024-09-21T10:01:07.424819+0000","last_active":"2024-09-21T10:05:46.571402+0000","last_peered":"2024-09-21T10:05:46.571402+0000","last_clean":"2024-09-21T10:05:46.571402+0000","last_became_active":"2024-09-21T10:01:07.424392+0000","last_became_peered":"2024-09-21T10:01:07.424392+0000","last_unstale":"2024-09-21T10:05:46.571402+0000","last_undegraded":"2024-09-21T10:05:46.571402+0000","last_fullsized":"2024-09-21T10:05:46.571402+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:27.266390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":91,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.578760+0000","last_change":"2024-09-21T09:59:54.267742+0000","last_active":"2024-09-21T10:05:46.578760+0000","last_peered":"2024-09-21T10:05:46.578760+0000","last_clean":"2024-09-21T10:05:46.578760+0000","last_became_active":"2024-09-21T09:59:54.267406+0000","last_became_peered":"2024-09-21T09:59:54.267406+0000","last_unstale":"2024-09-21T10:05:46.578760+0000","last_undegraded":"2024-09-21T10:05:46.578760+0000","last_fullsized":"2024-09-21T10:05:46.578760+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:25:35.968779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":103,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.571270+0000","last_change":"2024-09-21T10:02:14.103531+0000","last_active":"2024-09-21T10:05:46.571270+0000","last_peered":"2024-09-21T10:05:46.571270+0000","last_clean":"2024-09-21T10:05:46.571270+0000","last_became_active":"2024-09-21T10:02:14.103218+0000","last_became_peered":"2024-09-21T10:02:14.103218+0000","last_unstale":"2024-09-21T10:05:46.571270+0000","last_undegraded":"2024-09-21T10:05:46.571270+0000","last_fullsized":"2024-09-21T10:05:46.571270+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:40.014308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"126'10761","reported_seq":10815,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:06:02.020604+0000","last_change":"2024-09-21T10:03:41.941191+0000","last_active":"2024-09-21T10:06:02.020604+0000","last_peered":"2024-09-21T10:06:02.020604+0000","last_clean":"2024-09-21T10:06:02.020604+0000","last_became_active":"2024-09-21T10:03:41.940898+0000","last_became_peered":"2024-09-21T10:03:41.940898+0000","last_unstale":"2024-09-21T10:06:02.020604+0000","last_undegraded":"2024-09-21T10:06:02.020604+0000","last_fullsized":"2024-09-21T10:06:02.020604+0000","mapping_epoch":105,"log_start":"110'1900","ondisk_log_start":"110'1900","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":8861,"log_dups_size":0,"ondisk_log_size":8861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:53:35.519904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398524416,"num_objects":6081,"num_object_clones":0,"num_object_copies":18243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10761,"num_write_kb":538944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":107,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.571868+0000","last_change":"2024-09-21T10:02:14.103145+0000","last_active":"2024-09-21T10:05:46.571868+0000","last_peered":"2024-09-21T10:05:46.571868+0000","last_clean":"2024-09-21T10:05:46.571868+0000","last_became_active":"2024-09-21T10:02:14.102806+0000","last_became_peered":"2024-09-21T10:02:14.102806+0000","last_unstale":"2024-09-21T10:05:46.571868+0000","last_undegraded":"2024-09-21T10:05:46.571868+0000","last_fullsized":"2024-09-21T10:05:46.571868+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:24.348745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":765,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.593000+0000","last_change":"2024-09-21T10:03:39.669081+0000","last_active":"2024-09-21T10:05:55.593000+0000","last_peered":"2024-09-21T10:05:55.593000+0000","last_clean":"2024-09-21T10:05:55.593000+0000","last_became_active":"2024-09-21T10:03:39.668544+0000","last_became_peered":"2024-09-21T10:03:39.668544+0000","last_unstale":"2024-09-21T10:05:55.593000+0000","last_undegraded":"2024-09-21T10:05:55.593000+0000","last_fullsized":"2024-09-21T10:05:55.593000+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:48.735396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":47,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.571290+0000","last_change":"2024-09-21T10:03:39.668641+0000","last_active":"2024-09-21T10:05:46.571290+0000","last_peered":"2024-09-21T10:05:46.571290+0000","last_clean":"2024-09-21T10:05:46.571290+0000","last_became_active":"2024-09-21T10:03:39.668390+0000","last_became_peered":"2024-09-21T10:03:39.668390+0000","last_unstale":"2024-09-21T10:05:46.571290+0000","last_undegraded":"2024-09-21T10:05:46.571290+0000","last_fullsized":"2024-09-21T10:05:46.571290+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:07.211608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"126'11050","reported_seq":11079,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:06:03.000246+0000","last_change":"2024-09-21T10:03:41.941962+0000","last_active":"2024-09-21T10:06:03.000246+0000","last_peered":"2024-09-21T10:06:03.000246+0000","last_clean":"2024-09-21T10:06:03.000246+0000","last_became_active":"2024-09-21T10:03:41.941669+0000","last_became_peered":"2024-09-21T10:03:41.941669+0000","last_unstale":"2024-09-21T10:06:03.000246+0000","last_undegraded":"2024-09-21T10:06:03.000246+0000","last_fullsized":"2024-09-21T10:06:03.000246+0000","mapping_epoch":105,"log_start":"109'1300","ondisk_log_start":"109'1300","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":9750,"log_dups_size":0,"ondisk_log_size":9750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:52.959659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408158208,"num_objects":6228,"num_object_clones":0,"num_object_copies":18684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11050,"num_write_kb":552896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":91,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.592214+0000","last_change":"2024-09-21T10:03:39.670759+0000","last_active":"2024-09-21T10:05:55.592214+0000","last_peered":"2024-09-21T10:05:55.592214+0000","last_clean":"2024-09-21T10:05:55.592214+0000","last_became_active":"2024-09-21T10:03:39.670426+0000","last_became_peered":"2024-09-21T10:03:39.670426+0000","last_unstale":"2024-09-21T10:05:55.592214+0000","last_undegraded":"2024-09-21T10:05:55.592214+0000","last_fullsized":"2024-09-21T10:05:55.592214+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:03:22.478560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":437,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.569979+0000","last_change":"2024-09-21T09:59:54.265559+0000","last_active":"2024-09-21T10:05:46.569979+0000","last_peered":"2024-09-21T10:05:46.569979+0000","last_clean":"2024-09-21T10:05:46.569979+0000","last_became_active":"2024-09-21T09:59:54.264884+0000","last_became_peered":"2024-09-21T09:59:54.264884+0000","last_unstale":"2024-09-21T10:05:46.569979+0000","last_undegraded":"2024-09-21T10:05:46.569979+0000","last_fullsized":"2024-09-21T10:05:46.569979+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:24.544845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":103,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.600201+0000","last_change":"2024-09-21T09:59:54.267798+0000","last_active":"2024-09-21T10:05:55.600201+0000","last_peered":"2024-09-21T10:05:55.600201+0000","last_clean":"2024-09-21T10:05:55.600201+0000","last_became_active":"2024-09-21T09:59:54.265615+0000","last_became_peered":"2024-09-21T09:59:54.265615+0000","last_unstale":"2024-09-21T10:05:55.600201+0000","last_undegraded":"2024-09-21T10:05:55.600201+0000","last_fullsized":"2024-09-21T10:05:55.600201+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:30.363872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"126'10500","reported_seq":10529,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:58.055012+0000","last_change":"2024-09-21T10:03:41.939642+0000","last_active":"2024-09-21T10:05:58.055012+0000","last_peered":"2024-09-21T10:05:58.055012+0000","last_clean":"2024-09-21T10:05:58.055012+0000","last_became_active":"2024-09-21T10:03:41.939357+0000","last_became_peered":"2024-09-21T10:03:41.939357+0000","last_unstale":"2024-09-21T10:05:58.055012+0000","last_undegraded":"2024-09-21T10:05:58.055012+0000","last_fullsized":"2024-09-21T10:05:58.055012+0000","mapping_epoch":105,"log_start":"110'3100","ondisk_log_start":"110'3100","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":7400,"log_dups_size":0,"ondisk_log_size":7400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:54:51.943050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408682496,"num_objects":6236,"num_object_clones":0,"num_object_copies":18708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10500,"num_write_kb":535552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":116,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.597120+0000","last_change":"2024-09-21T10:02:14.101943+0000","last_active":"2024-09-21T10:05:55.597120+0000","last_peered":"2024-09-21T10:05:55.597120+0000","last_clean":"2024-09-21T10:05:55.597120+0000","last_became_active":"2024-09-21T10:02:14.101488+0000","last_became_peered":"2024-09-21T10:02:14.101488+0000","last_unstale":"2024-09-21T10:05:55.597120+0000","last_undegraded":"2024-09-21T10:05:55.597120+0000","last_fullsized":"2024-09-21T10:05:55.597120+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:52:27.866414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":425,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.577218+0000","last_change":"2024-09-21T10:03:39.670475+0000","last_active":"2024-09-21T10:05:46.577218+0000","last_peered":"2024-09-21T10:05:46.577218+0000","last_clean":"2024-09-21T10:05:46.577218+0000","last_became_active":"2024-09-21T10:03:39.670039+0000","last_became_peered":"2024-09-21T10:03:39.670039+0000","last_unstale":"2024-09-21T10:05:46.577218+0000","last_undegraded":"2024-09-21T10:05:46.577218+0000","last_fullsized":"2024-09-21T10:05:46.577218+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:55.384881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":113,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596452+0000","last_change":"2024-09-21T10:02:14.095924+0000","last_active":"2024-09-21T10:05:55.596452+0000","last_peered":"2024-09-21T10:05:55.596452+0000","last_clean":"2024-09-21T10:05:55.596452+0000","last_became_active":"2024-09-21T10:02:14.095553+0000","last_became_peered":"2024-09-21T10:02:14.095553+0000","last_unstale":"2024-09-21T10:05:55.596452+0000","last_undegraded":"2024-09-21T10:05:55.596452+0000","last_fullsized":"2024-09-21T10:05:55.596452+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:08.155980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"126'10650","reported_seq":10704,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:06:01.710450+0000","last_change":"2024-09-21T10:03:42.340430+0000","last_active":"2024-09-21T10:06:01.710450+0000","last_peered":"2024-09-21T10:06:01.710450+0000","last_clean":"2024-09-21T10:06:01.710450+0000","last_became_active":"2024-09-21T10:03:42.340071+0000","last_became_peered":"2024-09-21T10:03:42.340071+0000","last_unstale":"2024-09-21T10:06:01.710450+0000","last_undegraded":"2024-09-21T10:06:01.710450+0000","last_fullsized":"2024-09-21T10:06:01.710450+0000","mapping_epoch":105,"log_start":"110'1800","ondisk_log_start":"110'1800","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":8850,"log_dups_size":0,"ondisk_log_size":8850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:42.171457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392298496,"num_objects":5986,"num_object_clones":0,"num_object_copies":17958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10650,"num_write_kb":532352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":118,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.593180+0000","last_change":"2024-09-21T10:01:07.426736+0000","last_active":"2024-09-21T10:05:55.593180+0000","last_peered":"2024-09-21T10:05:55.593180+0000","last_clean":"2024-09-21T10:05:55.593180+0000","last_became_active":"2024-09-21T10:01:07.426416+0000","last_became_peered":"2024-09-21T10:01:07.426416+0000","last_unstale":"2024-09-21T10:05:55.593180+0000","last_undegraded":"2024-09-21T10:05:55.593180+0000","last_fullsized":"2024-09-21T10:05:55.593180+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:59.784757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":353,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.581138+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:05:46.581138+0000","last_peered":"2024-09-21T10:05:46.581138+0000","last_clean":"2024-09-21T10:05:46.581138+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:05:46.581138+0000","last_undegraded":"2024-09-21T10:05:46.581138+0000","last_fullsized":"2024-09-21T10:05:46.581138+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":73,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.595263+0000","last_change":"2024-09-21T09:52:20.391439+0000","last_active":"2024-09-21T10:05:46.595263+0000","last_peered":"2024-09-21T10:05:46.595263+0000","last_clean":"2024-09-21T10:05:46.595263+0000","last_became_active":"2024-09-21T09:52:20.391157+0000","last_became_peered":"2024-09-21T09:52:20.391157+0000","last_unstale":"2024-09-21T10:05:46.595263+0000","last_undegraded":"2024-09-21T10:05:46.595263+0000","last_fullsized":"2024-09-21T10:05:46.595263+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:36.788330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":90,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.578697+0000","last_change":"2024-09-21T10:01:07.421150+0000","last_active":"2024-09-21T10:05:46.578697+0000","last_peered":"2024-09-21T10:05:46.578697+0000","last_clean":"2024-09-21T10:05:46.578697+0000","last_became_active":"2024-09-21T10:01:07.420830+0000","last_became_peered":"2024-09-21T10:01:07.420830+0000","last_unstale":"2024-09-21T10:05:46.578697+0000","last_undegraded":"2024-09-21T10:05:46.578697+0000","last_fullsized":"2024-09-21T10:05:46.578697+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:15:34.472626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":96,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.591036+0000","last_change":"2024-09-21T09:52:20.371668+0000","last_active":"2024-09-21T10:05:46.591036+0000","last_peered":"2024-09-21T10:05:46.591036+0000","last_clean":"2024-09-21T10:05:46.591036+0000","last_became_active":"2024-09-21T09:52:20.371389+0000","last_became_peered":"2024-09-21T09:52:20.371389+0000","last_unstale":"2024-09-21T10:05:46.591036+0000","last_undegraded":"2024-09-21T10:05:46.591036+0000","last_fullsized":"2024-09-21T10:05:46.591036+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:10.013020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":69,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.576907+0000","last_change":"2024-09-21T10:03:39.670792+0000","last_active":"2024-09-21T10:05:46.576907+0000","last_peered":"2024-09-21T10:05:46.576907+0000","last_clean":"2024-09-21T10:05:46.576907+0000","last_became_active":"2024-09-21T10:03:39.670506+0000","last_became_peered":"2024-09-21T10:03:39.670506+0000","last_unstale":"2024-09-21T10:05:46.576907+0000","last_undegraded":"2024-09-21T10:05:46.576907+0000","last_fullsized":"2024-09-21T10:05:46.576907+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:33:19.134786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":94,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.570015+0000","last_change":"2024-09-21T10:02:14.102896+0000","last_active":"2024-09-21T10:05:46.570015+0000","last_peered":"2024-09-21T10:05:46.570015+0000","last_clean":"2024-09-21T10:05:46.570015+0000","last_became_active":"2024-09-21T10:02:14.102493+0000","last_became_peered":"2024-09-21T10:02:14.102493+0000","last_unstale":"2024-09-21T10:05:46.570015+0000","last_undegraded":"2024-09-21T10:05:46.570015+0000","last_fullsized":"2024-09-21T10:05:46.570015+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:10:43.572005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":90,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.571770+0000","last_change":"2024-09-21T10:01:07.424062+0000","last_active":"2024-09-21T10:05:46.571770+0000","last_peered":"2024-09-21T10:05:46.571770+0000","last_clean":"2024-09-21T10:05:46.571770+0000","last_became_active":"2024-09-21T10:01:07.423629+0000","last_became_peered":"2024-09-21T10:01:07.423629+0000","last_unstale":"2024-09-21T10:05:46.571770+0000","last_undegraded":"2024-09-21T10:05:46.571770+0000","last_fullsized":"2024-09-21T10:05:46.571770+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:24.119642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":73,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596431+0000","last_change":"2024-09-21T10:02:14.100451+0000","last_active":"2024-09-21T10:05:55.596431+0000","last_peered":"2024-09-21T10:05:55.596431+0000","last_clean":"2024-09-21T10:05:55.596431+0000","last_became_active":"2024-09-21T10:02:14.100160+0000","last_became_peered":"2024-09-21T10:02:14.100160+0000","last_unstale":"2024-09-21T10:05:55.596431+0000","last_undegraded":"2024-09-21T10:05:55.596431+0000","last_fullsized":"2024-09-21T10:05:55.596431+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:28.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":114,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.593147+0000","last_change":"2024-09-21T10:02:14.096098+0000","last_active":"2024-09-21T10:05:55.593147+0000","last_peered":"2024-09-21T10:05:55.593147+0000","last_clean":"2024-09-21T10:05:55.593147+0000","last_became_active":"2024-09-21T10:02:14.095811+0000","last_became_peered":"2024-09-21T10:02:14.095811+0000","last_unstale":"2024-09-21T10:05:55.593147+0000","last_undegraded":"2024-09-21T10:05:55.593147+0000","last_fullsized":"2024-09-21T10:05:55.593147+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:54.619333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":118,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596360+0000","last_change":"2024-09-21T10:02:14.100712+0000","last_active":"2024-09-21T10:05:55.596360+0000","last_peered":"2024-09-21T10:05:55.596360+0000","last_clean":"2024-09-21T10:05:55.596360+0000","last_became_active":"2024-09-21T10:02:14.100427+0000","last_became_peered":"2024-09-21T10:02:14.100427+0000","last_unstale":"2024-09-21T10:05:55.596360+0000","last_undegraded":"2024-09-21T10:05:55.596360+0000","last_fullsized":"2024-09-21T10:05:55.596360+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:16:42.350642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":90,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.581056+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:05:46.581056+0000","last_peered":"2024-09-21T10:05:46.581056+0000","last_clean":"2024-09-21T10:05:46.581056+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:05:46.581056+0000","last_undegraded":"2024-09-21T10:05:46.581056+0000","last_fullsized":"2024-09-21T10:05:46.581056+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":136,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596456+0000","last_change":"2024-09-21T10:02:14.097369+0000","last_active":"2024-09-21T10:05:55.596456+0000","last_peered":"2024-09-21T10:05:55.596456+0000","last_clean":"2024-09-21T10:05:55.596456+0000","last_became_active":"2024-09-21T10:02:14.097067+0000","last_became_peered":"2024-09-21T10:02:14.097067+0000","last_unstale":"2024-09-21T10:05:55.596456+0000","last_undegraded":"2024-09-21T10:05:55.596456+0000","last_fullsized":"2024-09-21T10:05:55.596456+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:50:48.123813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":115,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.593341+0000","last_change":"2024-09-21T10:02:14.098970+0000","last_active":"2024-09-21T10:05:55.593341+0000","last_peered":"2024-09-21T10:05:55.593341+0000","last_clean":"2024-09-21T10:05:55.593341+0000","last_became_active":"2024-09-21T10:02:14.098641+0000","last_became_peered":"2024-09-21T10:02:14.098641+0000","last_unstale":"2024-09-21T10:05:55.593341+0000","last_undegraded":"2024-09-21T10:05:55.593341+0000","last_fullsized":"2024-09-21T10:05:55.593341+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:25.215602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":104,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.598657+0000","last_change":"2024-09-21T10:02:14.097021+0000","last_active":"2024-09-21T10:05:55.598657+0000","last_peered":"2024-09-21T10:05:55.598657+0000","last_clean":"2024-09-21T10:05:55.598657+0000","last_became_active":"2024-09-21T10:02:14.096741+0000","last_became_peered":"2024-09-21T10:02:14.096741+0000","last_unstale":"2024-09-21T10:05:55.598657+0000","last_undegraded":"2024-09-21T10:05:55.598657+0000","last_fullsized":"2024-09-21T10:05:55.598657+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:18:20.631150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":118,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596838+0000","last_change":"2024-09-21T10:02:14.099460+0000","last_active":"2024-09-21T10:05:55.596838+0000","last_peered":"2024-09-21T10:05:55.596838+0000","last_clean":"2024-09-21T10:05:55.596838+0000","last_became_active":"2024-09-21T10:02:14.099159+0000","last_became_peered":"2024-09-21T10:02:14.099159+0000","last_unstale":"2024-09-21T10:05:55.596838+0000","last_undegraded":"2024-09-21T10:05:55.596838+0000","last_fullsized":"2024-09-21T10:05:55.596838+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:23:47.599107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":109,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.596955+0000","last_change":"2024-09-21T10:01:41.360480+0000","last_active":"2024-09-21T10:05:46.596955+0000","last_peered":"2024-09-21T10:05:46.596955+0000","last_clean":"2024-09-21T10:05:46.596955+0000","last_became_active":"2024-09-21T10:01:41.359791+0000","last_became_peered":"2024-09-21T10:01:41.359791+0000","last_unstale":"2024-09-21T10:05:46.596955+0000","last_undegraded":"2024-09-21T10:05:46.596955+0000","last_fullsized":"2024-09-21T10:05:46.596955+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:42.812974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.592260+0000","last_change":"2024-09-21T10:02:14.102826+0000","last_active":"2024-09-21T10:05:55.592260+0000","last_peered":"2024-09-21T10:05:55.592260+0000","last_clean":"2024-09-21T10:05:55.592260+0000","last_became_active":"2024-09-21T10:02:14.102499+0000","last_became_peered":"2024-09-21T10:02:14.102499+0000","last_unstale":"2024-09-21T10:05:55.592260+0000","last_undegraded":"2024-09-21T10:05:55.592260+0000","last_fullsized":"2024-09-21T10:05:55.592260+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:30.924687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":107,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.600167+0000","last_change":"2024-09-21T09:59:54.268378+0000","last_active":"2024-09-21T10:05:55.600167+0000","last_peered":"2024-09-21T10:05:55.600167+0000","last_clean":"2024-09-21T10:05:55.600167+0000","last_became_active":"2024-09-21T09:59:54.267187+0000","last_became_peered":"2024-09-21T09:59:54.267187+0000","last_unstale":"2024-09-21T10:05:55.600167+0000","last_undegraded":"2024-09-21T10:05:55.600167+0000","last_fullsized":"2024-09-21T10:05:55.600167+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:12:06.054483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":107,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.578635+0000","last_change":"2024-09-21T10:02:14.100541+0000","last_active":"2024-09-21T10:05:46.578635+0000","last_peered":"2024-09-21T10:05:46.578635+0000","last_clean":"2024-09-21T10:05:46.578635+0000","last_became_active":"2024-09-21T10:02:14.100327+0000","last_became_peered":"2024-09-21T10:02:14.100327+0000","last_unstale":"2024-09-21T10:05:46.578635+0000","last_undegraded":"2024-09-21T10:05:46.578635+0000","last_fullsized":"2024-09-21T10:05:46.578635+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:34:34.304188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":92,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.571584+0000","last_change":"2024-09-21T10:01:07.424248+0000","last_active":"2024-09-21T10:05:46.571584+0000","last_peered":"2024-09-21T10:05:46.571584+0000","last_clean":"2024-09-21T10:05:46.571584+0000","last_became_active":"2024-09-21T10:01:07.423738+0000","last_became_peered":"2024-09-21T10:01:07.423738+0000","last_unstale":"2024-09-21T10:05:46.571584+0000","last_undegraded":"2024-09-21T10:05:46.571584+0000","last_fullsized":"2024-09-21T10:05:46.571584+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:44:35.663555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":107,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.597051+0000","last_change":"2024-09-21T10:02:14.096583+0000","last_active":"2024-09-21T10:05:46.597051+0000","last_peered":"2024-09-21T10:05:46.597051+0000","last_clean":"2024-09-21T10:05:46.597051+0000","last_became_active":"2024-09-21T10:02:14.096196+0000","last_became_peered":"2024-09-21T10:02:14.096196+0000","last_unstale":"2024-09-21T10:05:46.597051+0000","last_undegraded":"2024-09-21T10:05:46.597051+0000","last_fullsized":"2024-09-21T10:05:46.597051+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:07:01.888435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":75,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.571799+0000","last_change":"2024-09-21T09:52:20.371638+0000","last_active":"2024-09-21T10:05:46.571799+0000","last_peered":"2024-09-21T10:05:46.571799+0000","last_clean":"2024-09-21T10:05:46.571799+0000","last_became_active":"2024-09-21T09:52:20.371383+0000","last_became_peered":"2024-09-21T09:52:20.371383+0000","last_unstale":"2024-09-21T10:05:46.571799+0000","last_undegraded":"2024-09-21T10:05:46.571799+0000","last_fullsized":"2024-09-21T10:05:46.571799+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:33:45.384397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":139,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596807+0000","last_change":"2024-09-21T10:02:14.099799+0000","last_active":"2024-09-21T10:05:55.596807+0000","last_peered":"2024-09-21T10:05:55.596807+0000","last_clean":"2024-09-21T10:05:55.596807+0000","last_became_active":"2024-09-21T10:02:14.099375+0000","last_became_peered":"2024-09-21T10:02:14.099375+0000","last_unstale":"2024-09-21T10:05:55.596807+0000","last_undegraded":"2024-09-21T10:05:55.596807+0000","last_fullsized":"2024-09-21T10:05:55.596807+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:13.407638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":124,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596950+0000","last_change":"2024-09-21T10:02:14.099439+0000","last_active":"2024-09-21T10:05:55.596950+0000","last_peered":"2024-09-21T10:05:55.596950+0000","last_clean":"2024-09-21T10:05:55.596950+0000","last_became_active":"2024-09-21T10:02:14.099117+0000","last_became_peered":"2024-09-21T10:02:14.099117+0000","last_unstale":"2024-09-21T10:05:55.596950+0000","last_undegraded":"2024-09-21T10:05:55.596950+0000","last_fullsized":"2024-09-21T10:05:55.596950+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:39.681817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":100,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.578809+0000","last_change":"2024-09-21T10:01:41.360674+0000","last_active":"2024-09-21T10:05:46.578809+0000","last_peered":"2024-09-21T10:05:46.578809+0000","last_clean":"2024-09-21T10:05:46.578809+0000","last_became_active":"2024-09-21T10:01:41.360258+0000","last_became_peered":"2024-09-21T10:01:41.360258+0000","last_unstale":"2024-09-21T10:05:46.578809+0000","last_undegraded":"2024-09-21T10:05:46.578809+0000","last_fullsized":"2024-09-21T10:05:46.578809+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:06.200022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":118,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.593159+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:05:55.593159+0000","last_peered":"2024-09-21T10:05:55.593159+0000","last_clean":"2024-09-21T10:05:55.593159+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:05:55.593159+0000","last_undegraded":"2024-09-21T10:05:55.593159+0000","last_fullsized":"2024-09-21T10:05:55.593159+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":6575226904,"num_objects":100331,"num_object_clones":0,"num_object_copies":300993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100331,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170941,"num_write_kb":8680641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19677786112,"data_stored":19677773896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146141,"ondisk_log_size":146141,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5983,"num_read_kb":13006,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397694,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2338448,"kb_used_data":2041092,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91406704,"statfs":{"total":95995035648,"available":93600464896,"internally_reserved":0,"allocated":2090078208,"data_stored":2089423520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593923,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1871616,"kb_used_data":1570104,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91873536,"statfs":{"total":95995035648,"available":94078500864,"internally_reserved":0,"allocated":1607786496,"data_stored":1607143945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790152,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3988596,"kb_used_data":3678764,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":89756556,"statfs":{"total":95995035648,"available":91910713344,"internally_reserved":0,"allocated":3767054336,"data_stored":3766419064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986381,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3570628,"kb_used_data":3260792,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":90174524,"statfs":{"total":95995035648,"available":92338712576,"internally_reserved":0,"allocated":3339051008,"data_stored":3338407962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":101,"seq":433791696944,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1248748,"kb_used_data":1172708,"kb_used_omap":5,"kb_used_meta":76026,"kb_avail":92496404,"statfs":{"total":95995035648,"available":94716317696,"internally_reserved":0,"allocated":1200852992,"data_stored":1200225646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":77851408},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860470,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2532212,"kb_used_data":2432364,"kb_used_omap":8,"kb_used_meta":99703,"kb_avail":91212940,"statfs":{"total":95995035648,"available":93402050560,"internally_reserved":0,"allocated":2490740736,"data_stored":2490105744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":102096748},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":91,"seq":390842023997,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2069412,"kb_used_data":1983580,"kb_used_omap":7,"kb_used_meta":85816,"kb_avail":91675740,"statfs":{"total":95995035648,"available":93875957760,"internally_reserved":0,"allocated":2031185920,"data_stored":2030566073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":87876146},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187532,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3263604,"kb_used_data":3155500,"kb_used_omap":8,"kb_used_meta":108023,"kb_avail":90481548,"statfs":{"total":95995035648,"available":92653105152,"internally_reserved":0,"allocated":3231232000,"data_stored":3230597516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":110616561},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3227779072,"data_stored":3227779072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2028601344,"data_stored":2028601344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2474971136,"data_stored":2474967064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1188823040,"data_stored":1188823040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3314352128,"data_stored":3314352128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3760984064,"data_stored":3760979992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1602289664,"data_stored":1602289664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2079985664,"data_stored":2079981592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:06:06.813 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T10:06:06.813 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:06:06.814 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-21T10:06:06.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:06:06.900 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:06.944 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:06.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:06.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:06.958 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:06 smithi098 ceph-mon[97580]: pgmap v331: 89 pgs: 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 823 op/s 2024-09-21T10:06:06.958 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:06 smithi098 ceph-mon[100914]: pgmap v331: 89 pgs: 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 823 op/s 2024-09-21T10:06:07.046 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:07.093 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:07.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:07.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:07.194 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:07.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:07.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:07.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:07.333 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:07.369 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:07.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:07.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:07.471 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:07.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:07.502 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:07.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:07.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:07.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:07.653 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:07.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:07.723 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:07.754 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:07.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:07.787 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:07.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:07.889 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:07.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:07.935 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:07.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:07.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:07.968 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:07.976 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:08.036 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:08.047 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:08.085 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:08.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:08.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:08.189 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:08.208 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:06:08.209 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:06:08.209 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-21T10:06:08.209 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-21T10:06:08.209 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:06:08.239 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:08.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:08.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:08.341 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:08.370 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:08.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:08.472 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:08.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:08.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:08.607 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:08.637 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:06:08.639 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:08.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:08.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:08.741 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:08.778 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:08.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:08.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:08.869 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:08 smithi196 ceph-mon[84784]: pgmap v332: 89 pgs: 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 739 op/s 2024-09-21T10:06:08.869 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:08 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2669107310' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:06:08.870 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:08 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:06:08.880 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:08.933 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:06:08.937 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:08.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:08.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:09.039 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:09.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:09.086 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:09.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:09.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:08 smithi098 ceph-mon[97580]: pgmap v332: 89 pgs: 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 739 op/s 2024-09-21T10:06:09.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:08 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2669107310' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:06:09.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:08 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:06:09.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:08 smithi098 ceph-mon[100914]: pgmap v332: 89 pgs: 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 739 op/s 2024-09-21T10:06:09.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:08 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2669107310' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:06:09.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:08 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:06:09.187 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:09.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:09.247 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:09.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:09.348 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:09.424 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:06:09.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:09.430 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:09.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:09.531 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:09.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:09.583 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:09.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:09.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:09.720 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:09.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:09.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:09.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:09.856 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:09.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:09.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:09.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:09 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:09 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:06:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:09 smithi196 ceph-mon[84784]: osdmap e127: 8 total, 8 up, 8 in 2024-09-21T10:06:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:09 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:09.917 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:09 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:09.917 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:09 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:06:09.917 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:09 smithi098 ceph-mon[97580]: osdmap e127: 8 total, 8 up, 8 in 2024-09-21T10:06:09.918 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:09 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:09.918 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:09 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:09.918 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:09 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:06:09.918 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:09 smithi098 ceph-mon[100914]: osdmap e127: 8 total, 8 up, 8 in 2024-09-21T10:06:09.918 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:09 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:09.958 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:09.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:09.995 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:09.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:10.096 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:10.149 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:10.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:10.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:10.250 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:10.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:10.280 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:10.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:10.380 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:10.413 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:10.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:10.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:10.515 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:10.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:10.555 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:10.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:10.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:06:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:06:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:06:10.656 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:10.685 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:10.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:10.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:10.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:10.822 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:10.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:10.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:10.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:10.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:10 smithi098 ceph-mon[97580]: pgmap v334: 89 pgs: 89 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 825 op/s 2024-09-21T10:06:10.932 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:06:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:06:10] "GET /metrics HTTP/1.1" 200 36458 "" "Prometheus/2.51.0" 2024-09-21T10:06:10.932 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:10 smithi098 ceph-mon[100914]: pgmap v334: 89 pgs: 89 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 825 op/s 2024-09-21T10:06:10.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:10.972 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:10.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:11.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:10 smithi196 ceph-mon[84784]: pgmap v334: 89 pgs: 89 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 825 op/s 2024-09-21T10:06:11.073 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:11.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:11.107 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:11.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:11.208 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:11.253 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:11.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:11.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:11.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:11.385 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:11.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:11.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:11.487 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:11.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:11.547 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:11.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:11.648 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:11.678 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:11.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:11.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:11.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:11.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:11.843 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:11.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:11.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:11.984 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:11.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:11.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:12.024 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:06:12.024 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:06:12.087 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:12.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:12.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:12.219 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:12.246 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:12.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:12.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:12.347 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:12.381 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:12.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:12.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:12.410 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:06:12.410 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-21T10:06:12.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-21T10:06:12.410 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:06:12.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.411 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:31182 5.7 5:ee38202f:::benchmark_data_smithi098_135187_object31181:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:06:12.411 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:03:59.633733+0000", 2024-09-21T10:06:12.411 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 132.77207210500001, 2024-09-21T10:06:12.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.076961207000000004, 2024-09-21T10:06:12.411 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:12.411 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:12.411 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:12.411 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:12.411 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:12.411 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 31182 2024-09-21T10:06:12.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.411 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:12.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:12.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.633733+0000", 2024-09-21T10:06:12.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:12.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.633733+0000", 2024-09-21T10:06:12.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:12.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.633733+0000", 2024-09-21T10:06:12.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:06:12.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:12.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.633790+0000", 2024-09-21T10:06:12.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7463999999999997e-05 2024-09-21T10:06:12.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:12.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.633800+0000", 2024-09-21T10:06:12.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0107e-05 2024-09-21T10:06:12.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:12.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.633804+0000", 2024-09-21T10:06:12.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9110000000000003e-06 2024-09-21T10:06:12.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:12.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.633838+0000", 2024-09-21T10:06:12.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4455999999999997e-05 2024-09-21T10:06:12.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:12.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.633917+0000", 2024-09-21T10:06:12.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8064000000000003e-05 2024-09-21T10:06:12.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:06:12.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.633997+0000", 2024-09-21T10:06:12.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0835999999999997e-05 2024-09-21T10:06:12.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:12.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710482+0000", 2024-09-21T10:06:12.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.076484416 2024-09-21T10:06:12.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:12.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710515+0000", 2024-09-21T10:06:12.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3464999999999999e-05 2024-09-21T10:06:12.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:12.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710610+0000", 2024-09-21T10:06:12.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4394000000000005e-05 2024-09-21T10:06:12.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:12.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710661+0000", 2024-09-21T10:06:12.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1320999999999999e-05 2024-09-21T10:06:12.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:12.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710695+0000", 2024-09-21T10:06:12.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3763999999999997e-05 2024-09-21T10:06:12.416 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.416 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:12.417 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.417 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.44894.0:31182 5.7 e110/105 ondisk, result = 0)", 2024-09-21T10:06:12.417 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:03:59.635169+0000", 2024-09-21T10:06:12.417 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 132.77063666399999, 2024-09-21T10:06:12.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.075348685999999998, 2024-09-21T10:06:12.417 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:12.417 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:06:12.417 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:12.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:12.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.635169+0000", 2024-09-21T10:06:12.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:12.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.635169+0000", 2024-09-21T10:06:12.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:12.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.635168+0000", 2024-09-21T10:06:12.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:12.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:12.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.635175+0000", 2024-09-21T10:06:12.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.199e-06 2024-09-21T10:06:12.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:12.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710027+0000", 2024-09-21T10:06:12.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.074851493000000005 2024-09-21T10:06:12.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:12.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710049+0000", 2024-09-21T10:06:12.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1976000000000001e-05 2024-09-21T10:06:12.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:12.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710502+0000", 2024-09-21T10:06:12.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00045345299999999998 2024-09-21T10:06:12.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:12.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710510+0000", 2024-09-21T10:06:12.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2919999999999998e-06 2024-09-21T10:06:12.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:12.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710518+0000", 2024-09-21T10:06:12.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9400000000000002e-06 2024-09-21T10:06:12.420 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.420 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:12.420 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.421 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.44894.0:31182 5.7 e110/105 ondisk, result = 0)", 2024-09-21T10:06:12.421 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:03:59.635173+0000", 2024-09-21T10:06:12.421 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 132.77063304399999, 2024-09-21T10:06:12.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.075527768999999995, 2024-09-21T10:06:12.421 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:12.421 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:06:12.421 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:12.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:12.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.635173+0000", 2024-09-21T10:06:12.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:12.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.635173+0000", 2024-09-21T10:06:12.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:12.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.635172+0000", 2024-09-21T10:06:12.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:12.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:12.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.635176+0000", 2024-09-21T10:06:12.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8800000000000001e-06 2024-09-21T10:06:12.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:12.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710538+0000", 2024-09-21T10:06:12.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.075362506999999995 2024-09-21T10:06:12.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:12.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710544+0000", 2024-09-21T10:06:12.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9490000000000001e-06 2024-09-21T10:06:12.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:12.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710599+0000", 2024-09-21T10:06:12.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4589000000000002e-05 2024-09-21T10:06:12.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:12.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710605+0000", 2024-09-21T10:06:12.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8889999999999999e-06 2024-09-21T10:06:12.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:12.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710700+0000", 2024-09-21T10:06:12.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5546999999999998e-05 2024-09-21T10:06:12.425 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.425 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:12.425 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.425 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:72280 5.d e112/105 5:b2fd55dd:::benchmark_data_smithi098_135187_object72279:head v 112'4469, mlcod=112'4466)", 2024-09-21T10:06:12.425 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:22.600438+0000", 2024-09-21T10:06:12.425 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 109.80536725899999, 2024-09-21T10:06:12.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.072455923000000005, 2024-09-21T10:06:12.425 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:12.425 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:12.425 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:12.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:12.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.600438+0000", 2024-09-21T10:06:12.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:12.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.600438+0000", 2024-09-21T10:06:12.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:12.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.600438+0000", 2024-09-21T10:06:12.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:12.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:12.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.600545+0000", 2024-09-21T10:06:12.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010741799999999999 2024-09-21T10:06:12.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:12.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.600555+0000", 2024-09-21T10:06:12.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0246e-05 2024-09-21T10:06:12.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:12.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.600560+0000", 2024-09-21T10:06:12.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9019999999999998e-06 2024-09-21T10:06:12.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:12.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.600612+0000", 2024-09-21T10:06:12.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1431999999999998e-05 2024-09-21T10:06:12.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:12.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.600631+0000", 2024-09-21T10:06:12.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8896999999999999e-05 2024-09-21T10:06:12.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:12.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.672852+0000", 2024-09-21T10:06:12.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.072221271000000004 2024-09-21T10:06:12.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:12.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.672894+0000", 2024-09-21T10:06:12.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2336999999999998e-05 2024-09-21T10:06:12.429 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.429 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:12.429 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.429 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:72288 5.2 e112/105 5:4f79b5b0:::benchmark_data_smithi098_135187_object72287:head v 112'4575, mlcod=112'4573)", 2024-09-21T10:06:12.429 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:22.602868+0000", 2024-09-21T10:06:12.429 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 109.802938052, 2024-09-21T10:06:12.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.072995333999999995, 2024-09-21T10:06:12.430 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:12.430 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:12.430 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:12.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:12.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602868+0000", 2024-09-21T10:06:12.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:12.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602868+0000", 2024-09-21T10:06:12.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:12.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602867+0000", 2024-09-21T10:06:12.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:12.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:12.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602900+0000", 2024-09-21T10:06:12.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3535999999999999e-05 2024-09-21T10:06:12.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:12.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602911+0000", 2024-09-21T10:06:12.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0604e-05 2024-09-21T10:06:12.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:12.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602917+0000", 2024-09-21T10:06:12.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7270000000000004e-06 2024-09-21T10:06:12.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:12.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.604496+0000", 2024-09-21T10:06:12.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015789479999999999 2024-09-21T10:06:12.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:12.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.604514+0000", 2024-09-21T10:06:12.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7889000000000001e-05 2024-09-21T10:06:12.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:12.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.675675+0000", 2024-09-21T10:06:12.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.071161665999999998 2024-09-21T10:06:12.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:12.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.675863+0000", 2024-09-21T10:06:12.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00018764000000000001 2024-09-21T10:06:12.433 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.433 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:12.433 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.433 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:72290 5.b e112/105 5:d51677e2:::benchmark_data_smithi098_135187_object72289:head v 112'4480, mlcod=112'4478)", 2024-09-21T10:06:12.433 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:22.602955+0000", 2024-09-21T10:06:12.433 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 109.80285026, 2024-09-21T10:06:12.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.068283890999999999, 2024-09-21T10:06:12.434 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:12.434 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:12.434 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:12.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:12.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602955+0000", 2024-09-21T10:06:12.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:12.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602955+0000", 2024-09-21T10:06:12.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:12.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602955+0000", 2024-09-21T10:06:12.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:12.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:12.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.603028+0000", 2024-09-21T10:06:12.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3514000000000001e-05 2024-09-21T10:06:12.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:12.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.603038+0000", 2024-09-21T10:06:12.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0149000000000001e-05 2024-09-21T10:06:12.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:12.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.603044+0000", 2024-09-21T10:06:12.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.429e-06 2024-09-21T10:06:12.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:12.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.603097+0000", 2024-09-21T10:06:12.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2882e-05 2024-09-21T10:06:12.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:12.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.603114+0000", 2024-09-21T10:06:12.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7462999999999999e-05 2024-09-21T10:06:12.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:12.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671172+0000", 2024-09-21T10:06:12.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.068057965999999998 2024-09-21T10:06:12.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:12.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671239+0000", 2024-09-21T10:06:12.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7144000000000006e-05 2024-09-21T10:06:12.437 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.437 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:12.437 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.438 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:72293 5.7 5:e294dae0:::benchmark_data_smithi098_135187_object72292:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T10:06:12.438 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:22.605214+0000", 2024-09-21T10:06:12.438 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 109.800591639, 2024-09-21T10:06:12.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.069737357999999999, 2024-09-21T10:06:12.438 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:12.438 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:12.438 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:12.438 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:12.438 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:12.438 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 72293 2024-09-21T10:06:12.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.438 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:12.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:12.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605214+0000", 2024-09-21T10:06:12.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:12.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605214+0000", 2024-09-21T10:06:12.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:12.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605212+0000", 2024-09-21T10:06:12.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:06:12.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:12.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605253+0000", 2024-09-21T10:06:12.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0808e-05 2024-09-21T10:06:12.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:12.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605267+0000", 2024-09-21T10:06:12.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3789e-05 2024-09-21T10:06:12.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:12.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605273+0000", 2024-09-21T10:06:12.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0349999999999998e-06 2024-09-21T10:06:12.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:12.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605322+0000", 2024-09-21T10:06:12.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9478000000000003e-05 2024-09-21T10:06:12.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:12.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.672025+0000", 2024-09-21T10:06:12.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.066702556999999996 2024-09-21T10:06:12.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:06:12.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.672152+0000", 2024-09-21T10:06:12.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000126744 2024-09-21T10:06:12.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:12.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.674802+0000", 2024-09-21T10:06:12.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026499610000000002 2024-09-21T10:06:12.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:12.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.674827+0000", 2024-09-21T10:06:12.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5786999999999999e-05 2024-09-21T10:06:12.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:12.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.674853+0000", 2024-09-21T10:06:12.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5184000000000001e-05 2024-09-21T10:06:12.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:12.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.674905+0000", 2024-09-21T10:06:12.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2868000000000003e-05 2024-09-21T10:06:12.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:12.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.674951+0000", 2024-09-21T10:06:12.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5840000000000002e-05 2024-09-21T10:06:12.443 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.443 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:12.443 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.444 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:72292 5.4 e112/105 5:2f3c813f:::benchmark_data_smithi098_135187_object72291:head v 112'4596, mlcod=112'4594)", 2024-09-21T10:06:12.444 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:22.605465+0000", 2024-09-21T10:06:12.444 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 109.80034049299999, 2024-09-21T10:06:12.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.068218862000000005, 2024-09-21T10:06:12.444 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:12.444 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:12.444 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:12.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:12.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605465+0000", 2024-09-21T10:06:12.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:12.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605465+0000", 2024-09-21T10:06:12.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:12.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605464+0000", 2024-09-21T10:06:12.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:06:12.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:12.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605499+0000", 2024-09-21T10:06:12.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5029e-05 2024-09-21T10:06:12.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.446 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:12.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.670988+0000", 2024-09-21T10:06:12.446 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.065489107000000005 2024-09-21T10:06:12.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.446 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:12.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.670998+0000", 2024-09-21T10:06:12.446 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5079999999999994e-06 2024-09-21T10:06:12.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.446 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:12.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671042+0000", 2024-09-21T10:06:12.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3835000000000003e-05 2024-09-21T10:06:12.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:12.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671063+0000", 2024-09-21T10:06:12.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0953000000000001e-05 2024-09-21T10:06:12.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:12.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.673594+0000", 2024-09-21T10:06:12.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00253104 2024-09-21T10:06:12.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:12.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.673684+0000", 2024-09-21T10:06:12.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0270000000000001e-05 2024-09-21T10:06:12.448 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.448 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:12.448 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.448 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:72289 5.2 e112/105 5:46d682ec:::benchmark_data_smithi098_135187_object72288:head v 112'4576, mlcod=112'4573)", 2024-09-21T10:06:12.448 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:22.605676+0000", 2024-09-21T10:06:12.448 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 109.800129273, 2024-09-21T10:06:12.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.070263872000000005, 2024-09-21T10:06:12.448 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:12.448 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:12.448 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:12.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:12.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605676+0000", 2024-09-21T10:06:12.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:12.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605676+0000", 2024-09-21T10:06:12.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:12.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605676+0000", 2024-09-21T10:06:12.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:12.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:12.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605698+0000", 2024-09-21T10:06:12.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2269e-05 2024-09-21T10:06:12.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:12.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.670902+0000", 2024-09-21T10:06:12.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.065204256000000002 2024-09-21T10:06:12.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:12.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.670916+0000", 2024-09-21T10:06:12.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3926e-05 2024-09-21T10:06:12.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:12.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671088+0000", 2024-09-21T10:06:12.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017191799999999999 2024-09-21T10:06:12.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:12.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671107+0000", 2024-09-21T10:06:12.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8627999999999999e-05 2024-09-21T10:06:12.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:12.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.675867+0000", 2024-09-21T10:06:12.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004760673 2024-09-21T10:06:12.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:12.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.675940+0000", 2024-09-21T10:06:12.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2798000000000003e-05 2024-09-21T10:06:12.452 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.452 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:12.452 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.452 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:85709 5.7 5:e34e5457:::benchmark_data_smithi098_135187_object85708:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T10:06:12.452 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:32.593458+0000", 2024-09-21T10:06:12.452 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 99.812347367000001, 2024-09-21T10:06:12.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.069336535000000005, 2024-09-21T10:06:12.453 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:12.453 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:12.453 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:12.453 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:12.453 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:12.453 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 85709 2024-09-21T10:06:12.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.453 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:12.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:12.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.593458+0000", 2024-09-21T10:06:12.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:12.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.593458+0000", 2024-09-21T10:06:12.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:12.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.593456+0000", 2024-09-21T10:06:12.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:06:12.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:12.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.593487+0000", 2024-09-21T10:06:12.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0490000000000001e-05 2024-09-21T10:06:12.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:12.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.593497+0000", 2024-09-21T10:06:12.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8290000000000003e-06 2024-09-21T10:06:12.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:12.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.593501+0000", 2024-09-21T10:06:12.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6310000000000002e-06 2024-09-21T10:06:12.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:12.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.594950+0000", 2024-09-21T10:06:12.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001449103 2024-09-21T10:06:12.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:12.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.595037+0000", 2024-09-21T10:06:12.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6341999999999994e-05 2024-09-21T10:06:12.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:06:12.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.595123+0000", 2024-09-21T10:06:12.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5798999999999996e-05 2024-09-21T10:06:12.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:12.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.597972+0000", 2024-09-21T10:06:12.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0028492550000000002 2024-09-21T10:06:12.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:12.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598495+0000", 2024-09-21T10:06:12.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00052359900000000003 2024-09-21T10:06:12.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:12.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.662715+0000", 2024-09-21T10:06:12.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.064219429999999994 2024-09-21T10:06:12.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:12.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.662762+0000", 2024-09-21T10:06:12.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7241999999999999e-05 2024-09-21T10:06:12.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:12.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.662795+0000", 2024-09-21T10:06:12.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2675999999999997e-05 2024-09-21T10:06:12.458 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.458 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:12.458 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.458 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:85719 5.4 e114/105 5:25a2b8e7:::benchmark_data_smithi098_135187_object85718:head v 114'5490, mlcod=114'5488)", 2024-09-21T10:06:12.458 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:32.596110+0000", 2024-09-21T10:06:12.458 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 99.809695118999997, 2024-09-21T10:06:12.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.065889160000000002, 2024-09-21T10:06:12.458 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:12.459 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:12.459 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:12.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:12.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.596110+0000", 2024-09-21T10:06:12.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:12.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.596110+0000", 2024-09-21T10:06:12.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:12.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.596110+0000", 2024-09-21T10:06:12.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:12.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:12.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.596131+0000", 2024-09-21T10:06:12.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1497000000000001e-05 2024-09-21T10:06:12.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:12.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.596139+0000", 2024-09-21T10:06:12.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0339999999999999e-06 2024-09-21T10:06:12.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:12.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.596142+0000", 2024-09-21T10:06:12.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6519999999999999e-06 2024-09-21T10:06:12.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:12.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.596189+0000", 2024-09-21T10:06:12.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6789000000000002e-05 2024-09-21T10:06:12.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:12.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.596201+0000", 2024-09-21T10:06:12.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1796e-05 2024-09-21T10:06:12.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:12.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.661962+0000", 2024-09-21T10:06:12.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.065761301999999994 2024-09-21T10:06:12.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:12.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.662000+0000", 2024-09-21T10:06:12.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7567999999999997e-05 2024-09-21T10:06:12.462 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.462 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:12.462 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.462 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:85713 5.b e114/105 5:d62e5ba1:::benchmark_data_smithi098_135187_object85712:head v 114'5314, mlcod=114'5312)", 2024-09-21T10:06:12.463 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:32.597344+0000", 2024-09-21T10:06:12.463 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 99.808461124999994, 2024-09-21T10:06:12.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.069982381999999996, 2024-09-21T10:06:12.463 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:12.463 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:12.463 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:12.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:12.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.597344+0000", 2024-09-21T10:06:12.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:12.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.597344+0000", 2024-09-21T10:06:12.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:12.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.597344+0000", 2024-09-21T10:06:12.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:12.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:12.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.597398+0000", 2024-09-21T10:06:12.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.469e-05 2024-09-21T10:06:12.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:12.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.597406+0000", 2024-09-21T10:06:12.465 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8779999999999998e-06 2024-09-21T10:06:12.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:12.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.597412+0000", 2024-09-21T10:06:12.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6300000000000003e-06 2024-09-21T10:06:12.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:12.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.597449+0000", 2024-09-21T10:06:12.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7066000000000001e-05 2024-09-21T10:06:12.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.467 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:12.467 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.597466+0000", 2024-09-21T10:06:12.467 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6684000000000002e-05 2024-09-21T10:06:12.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.467 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.467 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:12.467 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.667142+0000", 2024-09-21T10:06:12.467 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.069675916000000004 2024-09-21T10:06:12.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.467 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.467 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:12.467 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.667327+0000", 2024-09-21T10:06:12.467 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00018519399999999999 2024-09-21T10:06:12.467 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.467 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:12.468 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.468 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:85711 5.d e114/105 5:b14dccba:::benchmark_data_smithi098_135187_object85710:head v 114'5346, mlcod=114'5343)", 2024-09-21T10:06:12.468 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:32.598436+0000", 2024-09-21T10:06:12.468 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 99.807369225000002, 2024-09-21T10:06:12.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.076763213999999996, 2024-09-21T10:06:12.468 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:12.468 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:12.468 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:12.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:12.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598436+0000", 2024-09-21T10:06:12.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:12.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598436+0000", 2024-09-21T10:06:12.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:12.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598436+0000", 2024-09-21T10:06:12.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:12.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:12.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598512+0000", 2024-09-21T10:06:12.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5797000000000006e-05 2024-09-21T10:06:12.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:12.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598521+0000", 2024-09-21T10:06:12.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4040000000000001e-06 2024-09-21T10:06:12.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:12.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598526+0000", 2024-09-21T10:06:12.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6290000000000001e-06 2024-09-21T10:06:12.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:12.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598554+0000", 2024-09-21T10:06:12.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8401000000000001e-05 2024-09-21T10:06:12.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:12.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598565+0000", 2024-09-21T10:06:12.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0934000000000001e-05 2024-09-21T10:06:12.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:12.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.675162+0000", 2024-09-21T10:06:12.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.076597255000000003 2024-09-21T10:06:12.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:12.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.675200+0000", 2024-09-21T10:06:12.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7355999999999999e-05 2024-09-21T10:06:12.472 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.472 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:12.472 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.472 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:85715 5.3 e114/105 5:cce5decf:::benchmark_data_smithi098_135187_object85714:head v 114'5295, mlcod=114'5293)", 2024-09-21T10:06:12.472 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:32.598545+0000", 2024-09-21T10:06:12.472 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 99.807260517000003, 2024-09-21T10:06:12.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.074556140000000007, 2024-09-21T10:06:12.472 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:12.472 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:12.472 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:12.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.473 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:12.473 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598545+0000", 2024-09-21T10:06:12.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.473 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.473 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:12.473 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598545+0000", 2024-09-21T10:06:12.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.473 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.473 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:12.473 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598545+0000", 2024-09-21T10:06:12.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:12.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.474 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.474 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:12.474 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598576+0000", 2024-09-21T10:06:12.474 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1591000000000003e-05 2024-09-21T10:06:12.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.474 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.474 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:12.474 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598583+0000", 2024-09-21T10:06:12.474 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8700000000000003e-06 2024-09-21T10:06:12.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.474 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.474 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:12.474 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598587+0000", 2024-09-21T10:06:12.474 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5379999999999998e-06 2024-09-21T10:06:12.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:12.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.661994+0000", 2024-09-21T10:06:12.475 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.063406792000000003 2024-09-21T10:06:12.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:12.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.662013+0000", 2024-09-21T10:06:12.475 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9638000000000001e-05 2024-09-21T10:06:12.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:12.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.673004+0000", 2024-09-21T10:06:12.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010991288 2024-09-21T10:06:12.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:12.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.673101+0000", 2024-09-21T10:06:12.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6778000000000001e-05 2024-09-21T10:06:12.476 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.476 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:12.476 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.476 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:130868 5.7 5:ed4758c1:::benchmark_data_smithi098_135187_object130867:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-21T10:06:12.476 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:05:14.629322+0000", 2024-09-21T10:06:12.477 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 57.776483323999997, 2024-09-21T10:06:12.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.065226574999999995, 2024-09-21T10:06:12.477 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:12.477 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:12.477 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:12.477 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:12.477 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:12.477 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 130868 2024-09-21T10:06:12.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.477 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:12.477 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.477 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:12.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:14.629322+0000", 2024-09-21T10:06:12.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:12.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:14.629322+0000", 2024-09-21T10:06:12.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:12.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:14.629321+0000", 2024-09-21T10:06:12.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:06:12.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:12.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:14.629417+0000", 2024-09-21T10:06:12.479 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6582000000000005e-05 2024-09-21T10:06:12.479 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.479 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:12.479 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:14.629430+0000", 2024-09-21T10:06:12.479 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2486999999999999e-05 2024-09-21T10:06:12.479 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.479 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:12.479 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:14.629436+0000", 2024-09-21T10:06:12.479 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.908e-06 2024-09-21T10:06:12.479 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.479 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:12.480 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:14.633463+0000", 2024-09-21T10:06:12.480 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0040272179999999999 2024-09-21T10:06:12.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.480 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.480 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:12.480 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:14.633574+0000", 2024-09-21T10:06:12.480 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000110728 2024-09-21T10:06:12.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.480 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.480 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:06:12.480 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:14.633692+0000", 2024-09-21T10:06:12.480 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011845699999999999 2024-09-21T10:06:12.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.480 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:12.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:14.646070+0000", 2024-09-21T10:06:12.481 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012377634 2024-09-21T10:06:12.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:12.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:14.646289+0000", 2024-09-21T10:06:12.481 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00021901099999999999 2024-09-21T10:06:12.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:12.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:14.694423+0000", 2024-09-21T10:06:12.481 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.048134797 2024-09-21T10:06:12.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.482 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:12.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:14.694514+0000", 2024-09-21T10:06:12.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0716000000000004e-05 2024-09-21T10:06:12.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.482 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:12.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:14.694549+0000", 2024-09-21T10:06:12.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4687999999999997e-05 2024-09-21T10:06:12.482 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.482 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:12.482 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.483 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:156800 5.f 5:fc22f626:::benchmark_data_smithi098_135187_object21162:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-21T10:06:12.483 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:05:43.030539+0000", 2024-09-21T10:06:12.483 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 29.375266733, 2024-09-21T10:06:12.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.067001390999999993, 2024-09-21T10:06:12.483 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:12.483 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:12.483 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:12.483 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:12.483 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:12.483 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 156800 2024-09-21T10:06:12.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.483 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:12.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:12.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:43.030539+0000", 2024-09-21T10:06:12.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.484 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:12.484 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:43.030539+0000", 2024-09-21T10:06:12.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.484 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:12.484 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:43.030537+0000", 2024-09-21T10:06:12.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:06:12.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.484 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:12.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:43.030544+0000", 2024-09-21T10:06:12.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1800000000000001e-06 2024-09-21T10:06:12.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.485 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:12.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:43.030555+0000", 2024-09-21T10:06:12.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1507e-05 2024-09-21T10:06:12.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.485 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:12.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:43.030564+0000", 2024-09-21T10:06:12.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0040000000000005e-06 2024-09-21T10:06:12.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.485 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:12.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:43.058342+0000", 2024-09-21T10:06:12.486 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.027778363 2024-09-21T10:06:12.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.486 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.486 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:12.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:43.058454+0000", 2024-09-21T10:06:12.486 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000111857 2024-09-21T10:06:12.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.486 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.486 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:06:12.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:43.058803+0000", 2024-09-21T10:06:12.486 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00034898999999999998 2024-09-21T10:06:12.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.486 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:12.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:43.074757+0000", 2024-09-21T10:06:12.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015954117 2024-09-21T10:06:12.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:12.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:43.074763+0000", 2024-09-21T10:06:12.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.806e-06 2024-09-21T10:06:12.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:12.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:43.096904+0000", 2024-09-21T10:06:12.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.022140340000000001 2024-09-21T10:06:12.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:12.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:43.097242+0000", 2024-09-21T10:06:12.488 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00033794800000000002 2024-09-21T10:06:12.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:12.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:43.097540+0000", 2024-09-21T10:06:12.488 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00029873900000000001 2024-09-21T10:06:12.488 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.488 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:12.488 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.488 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:159919 5.3 e124/105 5:c765c100:::benchmark_data_smithi098_135187_object24281:head v 124'9916, mlcod=124'9913)", 2024-09-21T10:06:12.489 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:05:46.491193+0000", 2024-09-21T10:06:12.489 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.914613078999999, 2024-09-21T10:06:12.489 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.085250303999999999, 2024-09-21T10:06:12.489 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:12.489 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:12.489 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:12.489 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.489 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:12.489 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491193+0000", 2024-09-21T10:06:12.489 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.489 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.489 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.489 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:12.489 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491193+0000", 2024-09-21T10:06:12.490 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.490 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.490 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:12.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491192+0000", 2024-09-21T10:06:12.490 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:12.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.490 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.490 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:12.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491197+0000", 2024-09-21T10:06:12.490 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2580000000000003e-06 2024-09-21T10:06:12.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.490 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.490 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:12.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491204+0000", 2024-09-21T10:06:12.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7900000000000002e-06 2024-09-21T10:06:12.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.491 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.491 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:12.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491207+0000", 2024-09-21T10:06:12.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3220000000000001e-06 2024-09-21T10:06:12.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.491 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.491 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:12.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491237+0000", 2024-09-21T10:06:12.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9791999999999999e-05 2024-09-21T10:06:12.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.491 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.491 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:12.492 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491250+0000", 2024-09-21T10:06:12.492 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2775999999999999e-05 2024-09-21T10:06:12.492 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.492 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.492 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:12.492 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.576210+0000", 2024-09-21T10:06:12.492 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.084960182999999995 2024-09-21T10:06:12.492 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.492 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.492 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:12.492 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.576443+0000", 2024-09-21T10:06:12.492 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000232745 2024-09-21T10:06:12.492 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.492 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:12.492 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.493 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.493 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.493 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:159921 5.b e124/105 5:d56ee90f:::benchmark_data_smithi098_135187_object24283:head v 124'9893, mlcod=124'9891)", 2024-09-21T10:06:12.493 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:05:46.491221+0000", 2024-09-21T10:06:12.493 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.914584521999998, 2024-09-21T10:06:12.493 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.087278346000000007, 2024-09-21T10:06:12.493 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:12.493 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:12.493 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:12.493 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.493 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:12.493 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491221+0000", 2024-09-21T10:06:12.493 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.493 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.493 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:12.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491221+0000", 2024-09-21T10:06:12.494 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.494 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:12.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491221+0000", 2024-09-21T10:06:12.494 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:12.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.494 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:12.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491223+0000", 2024-09-21T10:06:12.494 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7209999999999999e-06 2024-09-21T10:06:12.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.495 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:12.495 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491230+0000", 2024-09-21T10:06:12.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5749999999999997e-06 2024-09-21T10:06:12.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.495 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:12.495 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491233+0000", 2024-09-21T10:06:12.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4020000000000002e-06 2024-09-21T10:06:12.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.495 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:12.495 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491261+0000", 2024-09-21T10:06:12.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7583e-05 2024-09-21T10:06:12.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.496 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.496 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:12.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491275+0000", 2024-09-21T10:06:12.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3601000000000001e-05 2024-09-21T10:06:12.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.496 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.496 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:12.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.578357+0000", 2024-09-21T10:06:12.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.087082076999999994 2024-09-21T10:06:12.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.496 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.496 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:12.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.578499+0000", 2024-09-21T10:06:12.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014272699999999999 2024-09-21T10:06:12.497 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.497 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:12.497 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.497 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.497 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.497 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:171352 5.f 5:fe452d91:::benchmark_data_smithi098_135187_object35714:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T10:06:12.497 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:06:00.008025+0000", 2024-09-21T10:06:12.497 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 12.397780601999999, 2024-09-21T10:06:12.497 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.065577942, 2024-09-21T10:06:12.497 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:12.497 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:12.497 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:12.497 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:12.497 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:12.497 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 171352 2024-09-21T10:06:12.498 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.498 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:12.498 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.498 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:12.498 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.008025+0000", 2024-09-21T10:06:12.498 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.498 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.498 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.498 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:12.498 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.008025+0000", 2024-09-21T10:06:12.498 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.498 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.498 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.498 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:12.499 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.008024+0000", 2024-09-21T10:06:12.499 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:06:12.499 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.499 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.499 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:12.499 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.008032+0000", 2024-09-21T10:06:12.499 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.413e-06 2024-09-21T10:06:12.499 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.499 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.499 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:12.499 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.008045+0000", 2024-09-21T10:06:12.499 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3138e-05 2024-09-21T10:06:12.499 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.499 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.500 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:12.500 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.008050+0000", 2024-09-21T10:06:12.500 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1610000000000002e-06 2024-09-21T10:06:12.500 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.500 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.500 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:12.500 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.033336+0000", 2024-09-21T10:06:12.500 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.025286093999999999 2024-09-21T10:06:12.500 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.500 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.500 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:12.500 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.033608+0000", 2024-09-21T10:06:12.500 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00027157599999999999 2024-09-21T10:06:12.500 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.501 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.501 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:06:12.501 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.033715+0000", 2024-09-21T10:06:12.501 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107129 2024-09-21T10:06:12.501 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.501 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.501 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:12.501 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.053450+0000", 2024-09-21T10:06:12.501 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019734958 2024-09-21T10:06:12.501 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.501 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.501 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:12.501 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.067555+0000", 2024-09-21T10:06:12.501 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014104993999999999 2024-09-21T10:06:12.501 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.502 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.502 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:12.502 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.073360+0000", 2024-09-21T10:06:12.502 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0058048960000000004 2024-09-21T10:06:12.502 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.502 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.502 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:12.502 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.073480+0000", 2024-09-21T10:06:12.502 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012013 2024-09-21T10:06:12.502 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.502 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.502 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:12.502 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.073603+0000", 2024-09-21T10:06:12.502 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012320300000000001 2024-09-21T10:06:12.503 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.503 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:12.503 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.503 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.503 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.503 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:171354 5.f 5:f899f6d2:::benchmark_data_smithi098_135187_object35716:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T10:06:12.503 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:06:00.009129+0000", 2024-09-21T10:06:12.503 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 12.396676438, 2024-09-21T10:06:12.503 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.064863272, 2024-09-21T10:06:12.503 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:12.503 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:12.503 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:12.503 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:12.503 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:12.504 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 171354 2024-09-21T10:06:12.504 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.504 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:12.504 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.504 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:12.504 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.009129+0000", 2024-09-21T10:06:12.504 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.504 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.504 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.504 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:12.504 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.009129+0000", 2024-09-21T10:06:12.504 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:12.504 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.504 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.504 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:12.505 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.009129+0000", 2024-09-21T10:06:12.505 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:12.505 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.505 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.505 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:12.505 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.009132+0000", 2024-09-21T10:06:12.505 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7110000000000001e-06 2024-09-21T10:06:12.505 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.505 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.505 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:12.505 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.009140+0000", 2024-09-21T10:06:12.505 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8329999999999994e-06 2024-09-21T10:06:12.505 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.505 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.506 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:12.506 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.009145+0000", 2024-09-21T10:06:12.506 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.498e-06 2024-09-21T10:06:12.506 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.506 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.506 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:12.506 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.038934+0000", 2024-09-21T10:06:12.506 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.029789000999999999 2024-09-21T10:06:12.506 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.506 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.506 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:12.506 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.039055+0000", 2024-09-21T10:06:12.506 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000121342 2024-09-21T10:06:12.506 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.507 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.507 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:06:12.507 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.039172+0000", 2024-09-21T10:06:12.507 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000116895 2024-09-21T10:06:12.507 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.507 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.507 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:12.507 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.053455+0000", 2024-09-21T10:06:12.507 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014282845000000001 2024-09-21T10:06:12.507 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.507 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.507 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:12.507 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.067578+0000", 2024-09-21T10:06:12.507 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014122836999999999 2024-09-21T10:06:12.507 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.508 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.508 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:12.508 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.073735+0000", 2024-09-21T10:06:12.508 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0061575739999999999 2024-09-21T10:06:12.508 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.508 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.508 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:12.508 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.073843+0000", 2024-09-21T10:06:12.508 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107645 2024-09-21T10:06:12.508 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:12.508 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:12.508 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:12.508 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:00.073992+0000", 2024-09-21T10:06:12.508 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000149629 2024-09-21T10:06:12.509 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.509 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:12.509 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.509 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:12.509 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:12.509 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:06:12.509 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:12.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:12.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:12.639 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:12.682 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:12.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:12.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:12.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:06:12.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:12 smithi098 ceph-mon[97580]: pgmap v335: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 828 op/s 2024-09-21T10:06:12.741 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:12 smithi098 ceph-mon[100914]: pgmap v335: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 828 op/s 2024-09-21T10:06:12.778 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-09-21T10:06:11.079467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6105143269,"num_objects":93113,"num_object_clones":0,"num_object_copies":279339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93113,"num_whiteouts":0,"num_read":6290,"num_read_kb":13766,"num_write":179243,"num_write_kb":8694892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147119,"ondisk_log_size":147119,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":19600148,"kb_used_data":17952080,"kb_used_omap":35,"kb_used_meta":1647708,"kb_avail":730361068,"statfs":{"total":767960285184,"available":747889733632,"internally_reserved":0,"allocated":18382929920,"data_stored":18377829786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36353,"internal_metadata":1687253503},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,7,6,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-543358976,"num_objects":-8291,"num_object_clones":0,"num_object_copies":-24873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8291,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002809"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":118,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596725+0000","last_change":"2024-09-21T10:02:14.097991+0000","last_active":"2024-09-21T10:05:55.596725+0000","last_peered":"2024-09-21T10:05:55.596725+0000","last_clean":"2024-09-21T10:05:55.596725+0000","last_became_active":"2024-09-21T10:02:14.097407+0000","last_became_peered":"2024-09-21T10:02:14.097407+0000","last_unstale":"2024-09-21T10:05:55.596725+0000","last_undegraded":"2024-09-21T10:05:55.596725+0000","last_fullsized":"2024-09-21T10:05:55.596725+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:18:42.098886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":93,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.522521+0000","last_change":"2024-09-21T10:01:41.360784+0000","last_active":"2024-09-21T10:05:46.522521+0000","last_peered":"2024-09-21T10:05:46.522521+0000","last_clean":"2024-09-21T10:05:46.522521+0000","last_became_active":"2024-09-21T10:01:41.360462+0000","last_became_peered":"2024-09-21T10:01:41.360462+0000","last_unstale":"2024-09-21T10:05:46.522521+0000","last_undegraded":"2024-09-21T10:05:46.522521+0000","last_fullsized":"2024-09-21T10:05:46.522521+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:11.867275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":89,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.571725+0000","last_change":"2024-09-21T10:02:14.103373+0000","last_active":"2024-09-21T10:05:46.571725+0000","last_peered":"2024-09-21T10:05:46.571725+0000","last_clean":"2024-09-21T10:05:46.571725+0000","last_became_active":"2024-09-21T10:02:14.102985+0000","last_became_peered":"2024-09-21T10:02:14.102985+0000","last_unstale":"2024-09-21T10:05:46.571725+0000","last_undegraded":"2024-09-21T10:05:46.571725+0000","last_fullsized":"2024-09-21T10:05:46.571725+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:03.418174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":114,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.592903+0000","last_change":"2024-09-21T10:02:14.098949+0000","last_active":"2024-09-21T10:05:55.592903+0000","last_peered":"2024-09-21T10:05:55.592903+0000","last_clean":"2024-09-21T10:05:55.592903+0000","last_became_active":"2024-09-21T10:02:14.098646+0000","last_became_peered":"2024-09-21T10:02:14.098646+0000","last_unstale":"2024-09-21T10:05:55.592903+0000","last_undegraded":"2024-09-21T10:05:55.592903+0000","last_fullsized":"2024-09-21T10:05:55.592903+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:50:56.862529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":114,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.593285+0000","last_change":"2024-09-21T10:02:14.098205+0000","last_active":"2024-09-21T10:05:55.593285+0000","last_peered":"2024-09-21T10:05:55.593285+0000","last_clean":"2024-09-21T10:05:55.593285+0000","last_became_active":"2024-09-21T10:02:14.097911+0000","last_became_peered":"2024-09-21T10:02:14.097911+0000","last_unstale":"2024-09-21T10:05:55.593285+0000","last_undegraded":"2024-09-21T10:05:55.593285+0000","last_fullsized":"2024-09-21T10:05:55.593285+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:55.971191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":74,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.569667+0000","last_change":"2024-09-21T09:52:20.371775+0000","last_active":"2024-09-21T10:05:46.569667+0000","last_peered":"2024-09-21T10:05:46.569667+0000","last_clean":"2024-09-21T10:05:46.569667+0000","last_became_active":"2024-09-21T09:52:20.371509+0000","last_became_peered":"2024-09-21T09:52:20.371509+0000","last_unstale":"2024-09-21T10:05:46.569667+0000","last_undegraded":"2024-09-21T10:05:46.569667+0000","last_fullsized":"2024-09-21T10:05:46.569667+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:43:21.822524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":118,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596979+0000","last_change":"2024-09-21T10:02:14.097817+0000","last_active":"2024-09-21T10:05:55.596979+0000","last_peered":"2024-09-21T10:05:55.596979+0000","last_clean":"2024-09-21T10:05:55.596979+0000","last_became_active":"2024-09-21T10:02:14.097466+0000","last_became_peered":"2024-09-21T10:02:14.097466+0000","last_unstale":"2024-09-21T10:05:55.596979+0000","last_undegraded":"2024-09-21T10:05:55.596979+0000","last_fullsized":"2024-09-21T10:05:55.596979+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:57.883118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":99,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.576486+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:05:46.576486+0000","last_peered":"2024-09-21T10:05:46.576486+0000","last_clean":"2024-09-21T10:05:46.576486+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:05:46.576486+0000","last_undegraded":"2024-09-21T10:05:46.576486+0000","last_fullsized":"2024-09-21T10:05:46.576486+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"126'11076","reported_seq":11103,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:06:08.009398+0000","last_change":"2024-09-21T10:03:41.940189+0000","last_active":"2024-09-21T10:06:08.009398+0000","last_peered":"2024-09-21T10:06:08.009398+0000","last_clean":"2024-09-21T10:06:08.009398+0000","last_became_active":"2024-09-21T10:03:41.939900+0000","last_became_peered":"2024-09-21T10:03:41.939900+0000","last_unstale":"2024-09-21T10:06:08.009398+0000","last_undegraded":"2024-09-21T10:06:08.009398+0000","last_fullsized":"2024-09-21T10:06:08.009398+0000","mapping_epoch":105,"log_start":"110'2500","ondisk_log_start":"110'2500","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":8576,"log_dups_size":0,"ondisk_log_size":8576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:13:00.441130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379977728,"num_objects":5798,"num_object_clones":0,"num_object_copies":17394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11076,"num_write_kb":539968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":92,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.570723+0000","last_change":"2024-09-21T09:59:54.265455+0000","last_active":"2024-09-21T10:05:46.570723+0000","last_peered":"2024-09-21T10:05:46.570723+0000","last_clean":"2024-09-21T10:05:46.570723+0000","last_became_active":"2024-09-21T09:59:54.265110+0000","last_became_peered":"2024-09-21T09:59:54.265110+0000","last_unstale":"2024-09-21T10:05:46.570723+0000","last_undegraded":"2024-09-21T10:05:46.570723+0000","last_fullsized":"2024-09-21T10:05:46.570723+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:04:13.404874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":72,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.571155+0000","last_change":"2024-09-21T09:52:20.402753+0000","last_active":"2024-09-21T10:05:46.571155+0000","last_peered":"2024-09-21T10:05:46.571155+0000","last_clean":"2024-09-21T10:05:46.571155+0000","last_became_active":"2024-09-21T09:52:20.402462+0000","last_became_peered":"2024-09-21T09:52:20.402462+0000","last_unstale":"2024-09-21T10:05:46.571155+0000","last_undegraded":"2024-09-21T10:05:46.571155+0000","last_fullsized":"2024-09-21T10:05:46.571155+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:14.036869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"127'11110","reported_seq":11165,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T10:06:09.277900+0000","last_change":"2024-09-21T10:03:42.340396+0000","last_active":"2024-09-21T10:06:09.277900+0000","last_peered":"2024-09-21T10:06:09.277900+0000","last_clean":"2024-09-21T10:06:09.277900+0000","last_became_active":"2024-09-21T10:03:42.340086+0000","last_became_peered":"2024-09-21T10:03:42.340086+0000","last_unstale":"2024-09-21T10:06:09.277900+0000","last_undegraded":"2024-09-21T10:06:09.277900+0000","last_fullsized":"2024-09-21T10:06:09.277900+0000","mapping_epoch":105,"log_start":"109'1100","ondisk_log_start":"109'1100","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:35:46.959831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380370944,"num_objects":5804,"num_object_clones":0,"num_object_copies":17412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11110,"num_write_kb":541248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":93,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.570625+0000","last_change":"2024-09-21T10:01:07.420484+0000","last_active":"2024-09-21T10:05:46.570625+0000","last_peered":"2024-09-21T10:05:46.570625+0000","last_clean":"2024-09-21T10:05:46.570625+0000","last_became_active":"2024-09-21T10:01:07.420135+0000","last_became_peered":"2024-09-21T10:01:07.420135+0000","last_unstale":"2024-09-21T10:05:46.570625+0000","last_undegraded":"2024-09-21T10:05:46.570625+0000","last_fullsized":"2024-09-21T10:05:46.570625+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:33:10.241010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":107,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.595369+0000","last_change":"2024-09-21T10:02:14.098139+0000","last_active":"2024-09-21T10:05:46.595369+0000","last_peered":"2024-09-21T10:05:46.595369+0000","last_clean":"2024-09-21T10:05:46.595369+0000","last_became_active":"2024-09-21T10:02:14.097639+0000","last_became_peered":"2024-09-21T10:02:14.097639+0000","last_unstale":"2024-09-21T10:05:46.595369+0000","last_undegraded":"2024-09-21T10:05:46.595369+0000","last_fullsized":"2024-09-21T10:05:46.595369+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:11:26.925388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"127'11054","reported_seq":11109,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T10:06:09.200662+0000","last_change":"2024-09-21T10:03:41.941408+0000","last_active":"2024-09-21T10:06:09.200662+0000","last_peered":"2024-09-21T10:06:09.200662+0000","last_clean":"2024-09-21T10:06:09.200662+0000","last_became_active":"2024-09-21T10:03:41.941084+0000","last_became_peered":"2024-09-21T10:03:41.941084+0000","last_unstale":"2024-09-21T10:06:09.200662+0000","last_undegraded":"2024-09-21T10:06:09.200662+0000","last_fullsized":"2024-09-21T10:06:09.200662+0000","mapping_epoch":105,"log_start":"109'1000","ondisk_log_start":"109'1000","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:33:47.243861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376438784,"num_objects":5744,"num_object_clones":0,"num_object_copies":17232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11054,"num_write_kb":537536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":75,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.569897+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:05:46.569897+0000","last_peered":"2024-09-21T10:05:46.569897+0000","last_clean":"2024-09-21T10:05:46.569897+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:05:46.569897+0000","last_undegraded":"2024-09-21T10:05:46.569897+0000","last_fullsized":"2024-09-21T10:05:46.569897+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":90,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.576832+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:05:46.576832+0000","last_peered":"2024-09-21T10:05:46.576832+0000","last_clean":"2024-09-21T10:05:46.576832+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:05:46.576832+0000","last_undegraded":"2024-09-21T10:05:46.576832+0000","last_fullsized":"2024-09-21T10:05:46.576832+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"127'11122","reported_seq":11177,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T10:06:09.274818+0000","last_change":"2024-09-21T10:03:41.939951+0000","last_active":"2024-09-21T10:06:09.274818+0000","last_peered":"2024-09-21T10:06:09.274818+0000","last_clean":"2024-09-21T10:06:09.274818+0000","last_became_active":"2024-09-21T10:03:41.939644+0000","last_became_peered":"2024-09-21T10:03:41.939644+0000","last_unstale":"2024-09-21T10:06:09.274818+0000","last_undegraded":"2024-09-21T10:06:09.274818+0000","last_fullsized":"2024-09-21T10:06:09.274818+0000","mapping_epoch":105,"log_start":"109'1100","ondisk_log_start":"109'1100","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:28:55.215930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376963072,"num_objects":5752,"num_object_clones":0,"num_object_copies":17256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11122,"num_write_kb":539968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":90,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.571659+0000","last_change":"2024-09-21T10:02:14.104212+0000","last_active":"2024-09-21T10:05:46.571659+0000","last_peered":"2024-09-21T10:05:46.571659+0000","last_clean":"2024-09-21T10:05:46.571659+0000","last_became_active":"2024-09-21T10:02:14.103893+0000","last_became_peered":"2024-09-21T10:02:14.103893+0000","last_unstale":"2024-09-21T10:05:46.571659+0000","last_undegraded":"2024-09-21T10:05:46.571659+0000","last_fullsized":"2024-09-21T10:05:46.571659+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:59:21.191412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":114,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.600256+0000","last_change":"2024-09-21T10:01:07.419715+0000","last_active":"2024-09-21T10:05:55.600256+0000","last_peered":"2024-09-21T10:05:55.600256+0000","last_clean":"2024-09-21T10:05:55.600256+0000","last_became_active":"2024-09-21T10:01:07.419422+0000","last_became_peered":"2024-09-21T10:01:07.419422+0000","last_unstale":"2024-09-21T10:05:55.600256+0000","last_undegraded":"2024-09-21T10:05:55.600256+0000","last_fullsized":"2024-09-21T10:05:55.600256+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:36.424289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"127'11127","reported_seq":11183,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T10:06:09.228050+0000","last_change":"2024-09-21T10:03:41.942958+0000","last_active":"2024-09-21T10:06:09.228050+0000","last_peered":"2024-09-21T10:06:09.228050+0000","last_clean":"2024-09-21T10:06:09.228050+0000","last_became_active":"2024-09-21T10:03:41.942656+0000","last_became_peered":"2024-09-21T10:03:41.942656+0000","last_unstale":"2024-09-21T10:06:09.228050+0000","last_undegraded":"2024-09-21T10:06:09.228050+0000","last_fullsized":"2024-09-21T10:06:09.228050+0000","mapping_epoch":105,"log_start":"109'1100","ondisk_log_start":"109'1100","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:36:35.630043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375390232,"num_objects":5729,"num_object_clones":0,"num_object_copies":17187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5729,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11127,"num_write_kb":539329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":116,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596599+0000","last_change":"2024-09-21T10:02:14.096537+0000","last_active":"2024-09-21T10:05:55.596599+0000","last_peered":"2024-09-21T10:05:55.596599+0000","last_clean":"2024-09-21T10:05:55.596599+0000","last_became_active":"2024-09-21T10:02:14.096166+0000","last_became_peered":"2024-09-21T10:02:14.096166+0000","last_unstale":"2024-09-21T10:05:55.596599+0000","last_undegraded":"2024-09-21T10:05:55.596599+0000","last_fullsized":"2024-09-21T10:05:55.596599+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:54:18.927732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":90,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.590946+0000","last_change":"2024-09-21T10:02:14.099205+0000","last_active":"2024-09-21T10:05:46.590946+0000","last_peered":"2024-09-21T10:05:46.590946+0000","last_clean":"2024-09-21T10:05:46.590946+0000","last_became_active":"2024-09-21T10:02:14.098598+0000","last_became_peered":"2024-09-21T10:02:14.098598+0000","last_unstale":"2024-09-21T10:05:46.590946+0000","last_undegraded":"2024-09-21T10:05:46.590946+0000","last_fullsized":"2024-09-21T10:05:46.590946+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:46.424612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"126'10945","reported_seq":10974,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:06:08.120745+0000","last_change":"2024-09-21T10:03:42.340737+0000","last_active":"2024-09-21T10:06:08.120745+0000","last_peered":"2024-09-21T10:06:08.120745+0000","last_clean":"2024-09-21T10:06:08.120745+0000","last_became_active":"2024-09-21T10:03:42.340445+0000","last_became_peered":"2024-09-21T10:03:42.340445+0000","last_unstale":"2024-09-21T10:06:08.120745+0000","last_undegraded":"2024-09-21T10:06:08.120745+0000","last_fullsized":"2024-09-21T10:06:08.120745+0000","mapping_epoch":105,"log_start":"111'3600","ondisk_log_start":"111'3600","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":7345,"log_dups_size":0,"ondisk_log_size":7345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:18.067171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382271488,"num_objects":5833,"num_object_clones":0,"num_object_copies":17499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10945,"num_write_kb":536896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":91,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.576977+0000","last_change":"2024-09-21T10:01:07.424883+0000","last_active":"2024-09-21T10:05:46.576977+0000","last_peered":"2024-09-21T10:05:46.576977+0000","last_clean":"2024-09-21T10:05:46.576977+0000","last_became_active":"2024-09-21T10:01:07.424499+0000","last_became_peered":"2024-09-21T10:01:07.424499+0000","last_unstale":"2024-09-21T10:05:46.576977+0000","last_undegraded":"2024-09-21T10:05:46.576977+0000","last_fullsized":"2024-09-21T10:05:46.576977+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:46.101570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":107,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.597394+0000","last_change":"2024-09-21T10:01:41.362291+0000","last_active":"2024-09-21T10:05:46.597394+0000","last_peered":"2024-09-21T10:05:46.597394+0000","last_clean":"2024-09-21T10:05:46.597394+0000","last_became_active":"2024-09-21T10:01:41.361954+0000","last_became_peered":"2024-09-21T10:01:41.361954+0000","last_unstale":"2024-09-21T10:05:46.597394+0000","last_undegraded":"2024-09-21T10:05:46.597394+0000","last_fullsized":"2024-09-21T10:05:46.597394+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:57.669671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"126'11185","reported_seq":11213,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:06:08.141757+0000","last_change":"2024-09-21T10:03:41.942110+0000","last_active":"2024-09-21T10:06:08.141757+0000","last_peered":"2024-09-21T10:06:08.141757+0000","last_clean":"2024-09-21T10:06:08.141757+0000","last_became_active":"2024-09-21T10:03:41.941789+0000","last_became_peered":"2024-09-21T10:03:41.941789+0000","last_unstale":"2024-09-21T10:06:08.141757+0000","last_undegraded":"2024-09-21T10:06:08.141757+0000","last_fullsized":"2024-09-21T10:06:08.141757+0000","mapping_epoch":105,"log_start":"109'1100","ondisk_log_start":"109'1100","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:33:04.384003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383844352,"num_objects":5857,"num_object_clones":0,"num_object_copies":17571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11185,"num_write_kb":545344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":91,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.595430+0000","last_change":"2024-09-21T10:01:41.361244+0000","last_active":"2024-09-21T10:05:46.595430+0000","last_peered":"2024-09-21T10:05:46.595430+0000","last_clean":"2024-09-21T10:05:46.595430+0000","last_became_active":"2024-09-21T10:01:41.360645+0000","last_became_peered":"2024-09-21T10:01:41.360645+0000","last_unstale":"2024-09-21T10:05:46.595430+0000","last_undegraded":"2024-09-21T10:05:46.595430+0000","last_fullsized":"2024-09-21T10:05:46.595430+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:07.604991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":90,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.590848+0000","last_change":"2024-09-21T10:01:07.420824+0000","last_active":"2024-09-21T10:05:46.590848+0000","last_peered":"2024-09-21T10:05:46.590848+0000","last_clean":"2024-09-21T10:05:46.590848+0000","last_became_active":"2024-09-21T10:01:07.420387+0000","last_became_peered":"2024-09-21T10:01:07.420387+0000","last_unstale":"2024-09-21T10:05:46.590848+0000","last_undegraded":"2024-09-21T10:05:46.590848+0000","last_fullsized":"2024-09-21T10:05:46.590848+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:48.875842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"126'11345","reported_seq":11373,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:06:08.137922+0000","last_change":"2024-09-21T10:03:41.941262+0000","last_active":"2024-09-21T10:06:08.137922+0000","last_peered":"2024-09-21T10:06:08.137922+0000","last_clean":"2024-09-21T10:06:08.137922+0000","last_became_active":"2024-09-21T10:03:41.940968+0000","last_became_peered":"2024-09-21T10:03:41.940968+0000","last_unstale":"2024-09-21T10:06:08.137922+0000","last_undegraded":"2024-09-21T10:06:08.137922+0000","last_fullsized":"2024-09-21T10:06:08.137922+0000","mapping_epoch":105,"log_start":"112'4000","ondisk_log_start":"112'4000","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":7345,"log_dups_size":0,"ondisk_log_size":7345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:47.483647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387252224,"num_objects":5909,"num_object_clones":0,"num_object_copies":17727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11345,"num_write_kb":552128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":119,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596541+0000","last_change":"2024-09-21T10:02:14.097462+0000","last_active":"2024-09-21T10:05:55.596541+0000","last_peered":"2024-09-21T10:05:55.596541+0000","last_clean":"2024-09-21T10:05:55.596541+0000","last_became_active":"2024-09-21T10:02:14.097167+0000","last_became_peered":"2024-09-21T10:02:14.097167+0000","last_unstale":"2024-09-21T10:05:55.596541+0000","last_undegraded":"2024-09-21T10:05:55.596541+0000","last_fullsized":"2024-09-21T10:05:55.596541+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:54.922731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":106,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T10:06:08.640361+0000","last_change":"2024-09-21T10:02:14.097676+0000","last_active":"2024-09-21T10:06:08.640361+0000","last_peered":"2024-09-21T10:06:08.640361+0000","last_clean":"2024-09-21T10:06:08.640361+0000","last_became_active":"2024-09-21T10:02:14.097391+0000","last_became_peered":"2024-09-21T10:02:14.097391+0000","last_unstale":"2024-09-21T10:06:08.640361+0000","last_undegraded":"2024-09-21T10:06:08.640361+0000","last_fullsized":"2024-09-21T10:06:08.640361+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:40:24.523178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"126'11368","reported_seq":11397,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:06:07.989141+0000","last_change":"2024-09-21T10:03:41.940655+0000","last_active":"2024-09-21T10:06:07.989141+0000","last_peered":"2024-09-21T10:06:07.989141+0000","last_clean":"2024-09-21T10:06:07.989141+0000","last_became_active":"2024-09-21T10:03:41.940058+0000","last_became_peered":"2024-09-21T10:03:41.940058+0000","last_unstale":"2024-09-21T10:06:07.989141+0000","last_undegraded":"2024-09-21T10:06:07.989141+0000","last_fullsized":"2024-09-21T10:06:07.989141+0000","mapping_epoch":105,"log_start":"110'2700","ondisk_log_start":"110'2700","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":8668,"log_dups_size":0,"ondisk_log_size":8668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:00:40.090535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396886016,"num_objects":6056,"num_object_clones":0,"num_object_copies":18168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11368,"num_write_kb":557568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":208,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.570793+0000","last_change":"2024-09-21T09:59:54.265556+0000","last_active":"2024-09-21T10:05:46.570793+0000","last_peered":"2024-09-21T10:05:46.570793+0000","last_clean":"2024-09-21T10:05:46.570793+0000","last_became_active":"2024-09-21T09:59:54.265033+0000","last_became_peered":"2024-09-21T09:59:54.265033+0000","last_unstale":"2024-09-21T10:05:46.570793+0000","last_undegraded":"2024-09-21T10:05:46.570793+0000","last_fullsized":"2024-09-21T10:05:46.570793+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T09:48:34.850872+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:40.449756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":514,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.575591+0000","last_change":"2024-09-21T10:01:41.361089+0000","last_active":"2024-09-21T10:05:46.575591+0000","last_peered":"2024-09-21T10:05:46.575591+0000","last_clean":"2024-09-21T10:05:46.575591+0000","last_became_active":"2024-09-21T10:01:41.360658+0000","last_became_peered":"2024-09-21T10:01:41.360658+0000","last_unstale":"2024-09-21T10:05:46.575591+0000","last_undegraded":"2024-09-21T10:05:46.575591+0000","last_fullsized":"2024-09-21T10:05:46.575591+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:58:06.840024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":482,"num_read_kb":1581,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":72,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596635+0000","last_change":"2024-09-21T10:02:14.099961+0000","last_active":"2024-09-21T10:05:55.596635+0000","last_peered":"2024-09-21T10:05:55.596635+0000","last_clean":"2024-09-21T10:05:55.596635+0000","last_became_active":"2024-09-21T10:02:14.099067+0000","last_became_peered":"2024-09-21T10:02:14.099067+0000","last_unstale":"2024-09-21T10:05:55.596635+0000","last_undegraded":"2024-09-21T10:05:55.596635+0000","last_fullsized":"2024-09-21T10:05:55.596635+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:41.037532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":73,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.577049+0000","last_change":"2024-09-21T09:52:20.403534+0000","last_active":"2024-09-21T10:05:46.577049+0000","last_peered":"2024-09-21T10:05:46.577049+0000","last_clean":"2024-09-21T10:05:46.577049+0000","last_became_active":"2024-09-21T09:52:20.403258+0000","last_became_peered":"2024-09-21T09:52:20.403258+0000","last_unstale":"2024-09-21T10:05:46.577049+0000","last_undegraded":"2024-09-21T10:05:46.577049+0000","last_fullsized":"2024-09-21T10:05:46.577049+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:55.374343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"126'11073","reported_seq":11101,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:06:08.066632+0000","last_change":"2024-09-21T10:03:42.340526+0000","last_active":"2024-09-21T10:06:08.066632+0000","last_peered":"2024-09-21T10:06:08.066632+0000","last_clean":"2024-09-21T10:06:08.066632+0000","last_became_active":"2024-09-21T10:03:42.340207+0000","last_became_peered":"2024-09-21T10:03:42.340207+0000","last_unstale":"2024-09-21T10:06:08.066632+0000","last_undegraded":"2024-09-21T10:06:08.066632+0000","last_fullsized":"2024-09-21T10:06:08.066632+0000","mapping_epoch":105,"log_start":"109'1000","ondisk_log_start":"109'1000","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:02.279418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380436480,"num_objects":5805,"num_object_clones":0,"num_object_copies":17415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11073,"num_write_kb":540096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":471,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.522520+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:05:46.522520+0000","last_peered":"2024-09-21T10:05:46.522520+0000","last_clean":"2024-09-21T10:05:46.522520+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:05:46.522520+0000","last_undegraded":"2024-09-21T10:05:46.522520+0000","last_fullsized":"2024-09-21T10:05:46.522520+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":115,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.592947+0000","last_change":"2024-09-21T10:02:14.097097+0000","last_active":"2024-09-21T10:05:55.592947+0000","last_peered":"2024-09-21T10:05:55.592947+0000","last_clean":"2024-09-21T10:05:55.592947+0000","last_became_active":"2024-09-21T10:02:14.096788+0000","last_became_peered":"2024-09-21T10:02:14.096788+0000","last_unstale":"2024-09-21T10:05:55.592947+0000","last_undegraded":"2024-09-21T10:05:55.592947+0000","last_fullsized":"2024-09-21T10:05:55.592947+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:05:29.711020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":125,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.593357+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:05:55.593357+0000","last_peered":"2024-09-21T10:05:55.593357+0000","last_clean":"2024-09-21T10:05:55.593357+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:05:55.593357+0000","last_undegraded":"2024-09-21T10:05:55.593357+0000","last_fullsized":"2024-09-21T10:05:55.593357+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"127'11171","reported_seq":11226,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T10:06:09.264205+0000","last_change":"2024-09-21T10:03:41.943228+0000","last_active":"2024-09-21T10:06:09.264205+0000","last_peered":"2024-09-21T10:06:09.264205+0000","last_clean":"2024-09-21T10:06:09.264205+0000","last_became_active":"2024-09-21T10:03:41.942690+0000","last_became_peered":"2024-09-21T10:03:41.942690+0000","last_unstale":"2024-09-21T10:06:09.264205+0000","last_undegraded":"2024-09-21T10:06:09.264205+0000","last_fullsized":"2024-09-21T10:06:09.264205+0000","mapping_epoch":105,"log_start":"109'1100","ondisk_log_start":"109'1100","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:32:01.612946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371916800,"num_objects":5675,"num_object_clones":0,"num_object_copies":17025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11171,"num_write_kb":539072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":866,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T10:06:08.639412+0000","last_change":"2024-09-21T10:02:14.100143+0000","last_active":"2024-09-21T10:06:08.639412+0000","last_peered":"2024-09-21T10:06:08.639412+0000","last_clean":"2024-09-21T10:06:08.639412+0000","last_became_active":"2024-09-21T10:02:14.099586+0000","last_became_peered":"2024-09-21T10:02:14.099586+0000","last_unstale":"2024-09-21T10:06:08.639412+0000","last_undegraded":"2024-09-21T10:06:08.639412+0000","last_fullsized":"2024-09-21T10:06:08.639412+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:34.064652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":93,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T10:06:08.636984+0000","last_change":"2024-09-21T10:03:39.670431+0000","last_active":"2024-09-21T10:06:08.636984+0000","last_peered":"2024-09-21T10:06:08.636984+0000","last_clean":"2024-09-21T10:06:08.636984+0000","last_became_active":"2024-09-21T10:03:39.670088+0000","last_became_peered":"2024-09-21T10:03:39.670088+0000","last_unstale":"2024-09-21T10:06:08.636984+0000","last_undegraded":"2024-09-21T10:06:08.636984+0000","last_fullsized":"2024-09-21T10:06:08.636984+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:30:52.512603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":123,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T10:06:08.639784+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:06:08.639784+0000","last_peered":"2024-09-21T10:06:08.639784+0000","last_clean":"2024-09-21T10:06:08.639784+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:06:08.639784+0000","last_undegraded":"2024-09-21T10:06:08.639784+0000","last_fullsized":"2024-09-21T10:06:08.639784+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"126'11334","reported_seq":11388,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:06:07.086312+0000","last_change":"2024-09-21T10:03:42.341388+0000","last_active":"2024-09-21T10:06:07.086312+0000","last_peered":"2024-09-21T10:06:07.086312+0000","last_clean":"2024-09-21T10:06:07.086312+0000","last_became_active":"2024-09-21T10:03:42.340274+0000","last_became_peered":"2024-09-21T10:03:42.340274+0000","last_unstale":"2024-09-21T10:06:07.086312+0000","last_undegraded":"2024-09-21T10:06:07.086312+0000","last_fullsized":"2024-09-21T10:06:07.086312+0000","mapping_epoch":105,"log_start":"110'2500","ondisk_log_start":"110'2500","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":8834,"log_dups_size":0,"ondisk_log_size":8834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:47.666420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387186688,"num_objects":5908,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11334,"num_write_kb":551744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":698,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.571402+0000","last_change":"2024-09-21T10:01:07.424819+0000","last_active":"2024-09-21T10:05:46.571402+0000","last_peered":"2024-09-21T10:05:46.571402+0000","last_clean":"2024-09-21T10:05:46.571402+0000","last_became_active":"2024-09-21T10:01:07.424392+0000","last_became_peered":"2024-09-21T10:01:07.424392+0000","last_unstale":"2024-09-21T10:05:46.571402+0000","last_undegraded":"2024-09-21T10:05:46.571402+0000","last_fullsized":"2024-09-21T10:05:46.571402+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:27.266390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":91,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.578760+0000","last_change":"2024-09-21T09:59:54.267742+0000","last_active":"2024-09-21T10:05:46.578760+0000","last_peered":"2024-09-21T10:05:46.578760+0000","last_clean":"2024-09-21T10:05:46.578760+0000","last_became_active":"2024-09-21T09:59:54.267406+0000","last_became_peered":"2024-09-21T09:59:54.267406+0000","last_unstale":"2024-09-21T10:05:46.578760+0000","last_undegraded":"2024-09-21T10:05:46.578760+0000","last_fullsized":"2024-09-21T10:05:46.578760+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:25:35.968779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":103,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.571270+0000","last_change":"2024-09-21T10:02:14.103531+0000","last_active":"2024-09-21T10:05:46.571270+0000","last_peered":"2024-09-21T10:05:46.571270+0000","last_clean":"2024-09-21T10:05:46.571270+0000","last_became_active":"2024-09-21T10:02:14.103218+0000","last_became_peered":"2024-09-21T10:02:14.103218+0000","last_unstale":"2024-09-21T10:05:46.571270+0000","last_undegraded":"2024-09-21T10:05:46.571270+0000","last_fullsized":"2024-09-21T10:05:46.571270+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:40.014308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"126'11005","reported_seq":11056,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:06:07.052400+0000","last_change":"2024-09-21T10:03:41.941191+0000","last_active":"2024-09-21T10:06:07.052400+0000","last_peered":"2024-09-21T10:06:07.052400+0000","last_clean":"2024-09-21T10:06:07.052400+0000","last_became_active":"2024-09-21T10:03:41.940898+0000","last_became_peered":"2024-09-21T10:03:41.940898+0000","last_unstale":"2024-09-21T10:06:07.052400+0000","last_undegraded":"2024-09-21T10:06:07.052400+0000","last_fullsized":"2024-09-21T10:06:07.052400+0000","mapping_epoch":105,"log_start":"110'2100","ondisk_log_start":"110'2100","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":8905,"log_dups_size":0,"ondisk_log_size":8905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:53:35.519904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382533632,"num_objects":5837,"num_object_clones":0,"num_object_copies":17511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11005,"num_write_kb":538944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":107,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.571868+0000","last_change":"2024-09-21T10:02:14.103145+0000","last_active":"2024-09-21T10:05:46.571868+0000","last_peered":"2024-09-21T10:05:46.571868+0000","last_clean":"2024-09-21T10:05:46.571868+0000","last_became_active":"2024-09-21T10:02:14.102806+0000","last_became_peered":"2024-09-21T10:02:14.102806+0000","last_unstale":"2024-09-21T10:05:46.571868+0000","last_undegraded":"2024-09-21T10:05:46.571868+0000","last_fullsized":"2024-09-21T10:05:46.571868+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:24.348745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":765,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.593000+0000","last_change":"2024-09-21T10:03:39.669081+0000","last_active":"2024-09-21T10:05:55.593000+0000","last_peered":"2024-09-21T10:05:55.593000+0000","last_clean":"2024-09-21T10:05:55.593000+0000","last_became_active":"2024-09-21T10:03:39.668544+0000","last_became_peered":"2024-09-21T10:03:39.668544+0000","last_unstale":"2024-09-21T10:05:55.593000+0000","last_undegraded":"2024-09-21T10:05:55.593000+0000","last_fullsized":"2024-09-21T10:05:55.593000+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:48.735396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":47,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.571290+0000","last_change":"2024-09-21T10:03:39.668641+0000","last_active":"2024-09-21T10:05:46.571290+0000","last_peered":"2024-09-21T10:05:46.571290+0000","last_clean":"2024-09-21T10:05:46.571290+0000","last_became_active":"2024-09-21T10:03:39.668390+0000","last_became_peered":"2024-09-21T10:03:39.668390+0000","last_unstale":"2024-09-21T10:05:46.571290+0000","last_undegraded":"2024-09-21T10:05:46.571290+0000","last_fullsized":"2024-09-21T10:05:46.571290+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:07.211608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"126'11331","reported_seq":11360,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:06:07.988140+0000","last_change":"2024-09-21T10:03:41.941962+0000","last_active":"2024-09-21T10:06:07.988140+0000","last_peered":"2024-09-21T10:06:07.988140+0000","last_clean":"2024-09-21T10:06:07.988140+0000","last_became_active":"2024-09-21T10:03:41.941669+0000","last_became_peered":"2024-09-21T10:03:41.941669+0000","last_unstale":"2024-09-21T10:06:07.988140+0000","last_undegraded":"2024-09-21T10:06:07.988140+0000","last_fullsized":"2024-09-21T10:06:07.988140+0000","mapping_epoch":105,"log_start":"109'1600","ondisk_log_start":"109'1600","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":9731,"log_dups_size":0,"ondisk_log_size":9731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:52.959659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389742592,"num_objects":5947,"num_object_clones":0,"num_object_copies":17841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11331,"num_write_kb":552896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":93,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T10:06:08.641222+0000","last_change":"2024-09-21T10:03:39.670759+0000","last_active":"2024-09-21T10:06:08.641222+0000","last_peered":"2024-09-21T10:06:08.641222+0000","last_clean":"2024-09-21T10:06:08.641222+0000","last_became_active":"2024-09-21T10:03:39.670426+0000","last_became_peered":"2024-09-21T10:03:39.670426+0000","last_unstale":"2024-09-21T10:06:08.641222+0000","last_undegraded":"2024-09-21T10:06:08.641222+0000","last_fullsized":"2024-09-21T10:06:08.641222+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:03:22.478560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":437,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.569979+0000","last_change":"2024-09-21T09:59:54.265559+0000","last_active":"2024-09-21T10:05:46.569979+0000","last_peered":"2024-09-21T10:05:46.569979+0000","last_clean":"2024-09-21T10:05:46.569979+0000","last_became_active":"2024-09-21T09:59:54.264884+0000","last_became_peered":"2024-09-21T09:59:54.264884+0000","last_unstale":"2024-09-21T10:05:46.569979+0000","last_undegraded":"2024-09-21T10:05:46.569979+0000","last_fullsized":"2024-09-21T10:05:46.569979+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:24.544845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":104,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.600237+0000","last_change":"2024-09-21T09:59:54.267798+0000","last_active":"2024-09-21T10:05:55.600237+0000","last_peered":"2024-09-21T10:05:55.600237+0000","last_clean":"2024-09-21T10:05:55.600237+0000","last_became_active":"2024-09-21T09:59:54.265615+0000","last_became_peered":"2024-09-21T09:59:54.265615+0000","last_unstale":"2024-09-21T10:05:55.600237+0000","last_undegraded":"2024-09-21T10:05:55.600237+0000","last_fullsized":"2024-09-21T10:05:55.600237+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:30.363872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"126'11028","reported_seq":11057,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:06:08.090946+0000","last_change":"2024-09-21T10:03:41.939642+0000","last_active":"2024-09-21T10:06:08.090946+0000","last_peered":"2024-09-21T10:06:08.090946+0000","last_clean":"2024-09-21T10:06:08.090946+0000","last_became_active":"2024-09-21T10:03:41.939357+0000","last_became_peered":"2024-09-21T10:03:41.939357+0000","last_unstale":"2024-09-21T10:06:08.090946+0000","last_undegraded":"2024-09-21T10:06:08.090946+0000","last_fullsized":"2024-09-21T10:06:08.090946+0000","mapping_epoch":105,"log_start":"111'3700","ondisk_log_start":"111'3700","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":7328,"log_dups_size":0,"ondisk_log_size":7328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:54:51.943050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374079488,"num_objects":5708,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11028,"num_write_kb":535552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":116,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.597120+0000","last_change":"2024-09-21T10:02:14.101943+0000","last_active":"2024-09-21T10:05:55.597120+0000","last_peered":"2024-09-21T10:05:55.597120+0000","last_clean":"2024-09-21T10:05:55.597120+0000","last_became_active":"2024-09-21T10:02:14.101488+0000","last_became_peered":"2024-09-21T10:02:14.101488+0000","last_unstale":"2024-09-21T10:05:55.597120+0000","last_undegraded":"2024-09-21T10:05:55.597120+0000","last_fullsized":"2024-09-21T10:05:55.597120+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:52:27.866414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":425,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.577218+0000","last_change":"2024-09-21T10:03:39.670475+0000","last_active":"2024-09-21T10:05:46.577218+0000","last_peered":"2024-09-21T10:05:46.577218+0000","last_clean":"2024-09-21T10:05:46.577218+0000","last_became_active":"2024-09-21T10:03:39.670039+0000","last_became_peered":"2024-09-21T10:03:39.670039+0000","last_unstale":"2024-09-21T10:05:46.577218+0000","last_undegraded":"2024-09-21T10:05:46.577218+0000","last_fullsized":"2024-09-21T10:05:46.577218+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:55.384881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":113,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596452+0000","last_change":"2024-09-21T10:02:14.095924+0000","last_active":"2024-09-21T10:05:55.596452+0000","last_peered":"2024-09-21T10:05:55.596452+0000","last_clean":"2024-09-21T10:05:55.596452+0000","last_became_active":"2024-09-21T10:02:14.095553+0000","last_became_peered":"2024-09-21T10:02:14.095553+0000","last_unstale":"2024-09-21T10:05:55.596452+0000","last_undegraded":"2024-09-21T10:05:55.596452+0000","last_fullsized":"2024-09-21T10:05:55.596452+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:08.155980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"126'10913","reported_seq":10967,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:06:06.720980+0000","last_change":"2024-09-21T10:03:42.340430+0000","last_active":"2024-09-21T10:06:06.720980+0000","last_peered":"2024-09-21T10:06:06.720980+0000","last_clean":"2024-09-21T10:06:06.720980+0000","last_became_active":"2024-09-21T10:03:42.340071+0000","last_became_peered":"2024-09-21T10:03:42.340071+0000","last_unstale":"2024-09-21T10:06:06.720980+0000","last_undegraded":"2024-09-21T10:06:06.720980+0000","last_fullsized":"2024-09-21T10:06:06.720980+0000","mapping_epoch":105,"log_start":"110'2000","ondisk_log_start":"110'2000","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:03:40.924093+0000","last_clean_scrub_stamp":"2024-09-21T10:03:40.924093+0000","objects_scrubbed":0,"log_size":8913,"log_dups_size":0,"ondisk_log_size":8913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:42.171457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375062528,"num_objects":5723,"num_object_clones":0,"num_object_copies":17169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10913,"num_write_kb":532352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":118,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.593180+0000","last_change":"2024-09-21T10:01:07.426736+0000","last_active":"2024-09-21T10:05:55.593180+0000","last_peered":"2024-09-21T10:05:55.593180+0000","last_clean":"2024-09-21T10:05:55.593180+0000","last_became_active":"2024-09-21T10:01:07.426416+0000","last_became_peered":"2024-09-21T10:01:07.426416+0000","last_unstale":"2024-09-21T10:05:55.593180+0000","last_undegraded":"2024-09-21T10:05:55.593180+0000","last_fullsized":"2024-09-21T10:05:55.593180+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:59.784757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":353,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.581138+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:05:46.581138+0000","last_peered":"2024-09-21T10:05:46.581138+0000","last_clean":"2024-09-21T10:05:46.581138+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:05:46.581138+0000","last_undegraded":"2024-09-21T10:05:46.581138+0000","last_fullsized":"2024-09-21T10:05:46.581138+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":73,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.595263+0000","last_change":"2024-09-21T09:52:20.391439+0000","last_active":"2024-09-21T10:05:46.595263+0000","last_peered":"2024-09-21T10:05:46.595263+0000","last_clean":"2024-09-21T10:05:46.595263+0000","last_became_active":"2024-09-21T09:52:20.391157+0000","last_became_peered":"2024-09-21T09:52:20.391157+0000","last_unstale":"2024-09-21T10:05:46.595263+0000","last_undegraded":"2024-09-21T10:05:46.595263+0000","last_fullsized":"2024-09-21T10:05:46.595263+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:36.788330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":90,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.578697+0000","last_change":"2024-09-21T10:01:07.421150+0000","last_active":"2024-09-21T10:05:46.578697+0000","last_peered":"2024-09-21T10:05:46.578697+0000","last_clean":"2024-09-21T10:05:46.578697+0000","last_became_active":"2024-09-21T10:01:07.420830+0000","last_became_peered":"2024-09-21T10:01:07.420830+0000","last_unstale":"2024-09-21T10:05:46.578697+0000","last_undegraded":"2024-09-21T10:05:46.578697+0000","last_fullsized":"2024-09-21T10:05:46.578697+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:15:34.472626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":96,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.591036+0000","last_change":"2024-09-21T09:52:20.371668+0000","last_active":"2024-09-21T10:05:46.591036+0000","last_peered":"2024-09-21T10:05:46.591036+0000","last_clean":"2024-09-21T10:05:46.591036+0000","last_became_active":"2024-09-21T09:52:20.371389+0000","last_became_peered":"2024-09-21T09:52:20.371389+0000","last_unstale":"2024-09-21T10:05:46.591036+0000","last_undegraded":"2024-09-21T10:05:46.591036+0000","last_fullsized":"2024-09-21T10:05:46.591036+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:10.013020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":69,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.576907+0000","last_change":"2024-09-21T10:03:39.670792+0000","last_active":"2024-09-21T10:05:46.576907+0000","last_peered":"2024-09-21T10:05:46.576907+0000","last_clean":"2024-09-21T10:05:46.576907+0000","last_became_active":"2024-09-21T10:03:39.670506+0000","last_became_peered":"2024-09-21T10:03:39.670506+0000","last_unstale":"2024-09-21T10:05:46.576907+0000","last_undegraded":"2024-09-21T10:05:46.576907+0000","last_fullsized":"2024-09-21T10:05:46.576907+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:33:19.134786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":94,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.570015+0000","last_change":"2024-09-21T10:02:14.102896+0000","last_active":"2024-09-21T10:05:46.570015+0000","last_peered":"2024-09-21T10:05:46.570015+0000","last_clean":"2024-09-21T10:05:46.570015+0000","last_became_active":"2024-09-21T10:02:14.102493+0000","last_became_peered":"2024-09-21T10:02:14.102493+0000","last_unstale":"2024-09-21T10:05:46.570015+0000","last_undegraded":"2024-09-21T10:05:46.570015+0000","last_fullsized":"2024-09-21T10:05:46.570015+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:10:43.572005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":90,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.571770+0000","last_change":"2024-09-21T10:01:07.424062+0000","last_active":"2024-09-21T10:05:46.571770+0000","last_peered":"2024-09-21T10:05:46.571770+0000","last_clean":"2024-09-21T10:05:46.571770+0000","last_became_active":"2024-09-21T10:01:07.423629+0000","last_became_peered":"2024-09-21T10:01:07.423629+0000","last_unstale":"2024-09-21T10:05:46.571770+0000","last_undegraded":"2024-09-21T10:05:46.571770+0000","last_fullsized":"2024-09-21T10:05:46.571770+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:24.119642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":73,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596431+0000","last_change":"2024-09-21T10:02:14.100451+0000","last_active":"2024-09-21T10:05:55.596431+0000","last_peered":"2024-09-21T10:05:55.596431+0000","last_clean":"2024-09-21T10:05:55.596431+0000","last_became_active":"2024-09-21T10:02:14.100160+0000","last_became_peered":"2024-09-21T10:02:14.100160+0000","last_unstale":"2024-09-21T10:05:55.596431+0000","last_undegraded":"2024-09-21T10:05:55.596431+0000","last_fullsized":"2024-09-21T10:05:55.596431+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:28.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":114,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.593147+0000","last_change":"2024-09-21T10:02:14.096098+0000","last_active":"2024-09-21T10:05:55.593147+0000","last_peered":"2024-09-21T10:05:55.593147+0000","last_clean":"2024-09-21T10:05:55.593147+0000","last_became_active":"2024-09-21T10:02:14.095811+0000","last_became_peered":"2024-09-21T10:02:14.095811+0000","last_unstale":"2024-09-21T10:05:55.593147+0000","last_undegraded":"2024-09-21T10:05:55.593147+0000","last_fullsized":"2024-09-21T10:05:55.593147+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:54.619333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":118,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596360+0000","last_change":"2024-09-21T10:02:14.100712+0000","last_active":"2024-09-21T10:05:55.596360+0000","last_peered":"2024-09-21T10:05:55.596360+0000","last_clean":"2024-09-21T10:05:55.596360+0000","last_became_active":"2024-09-21T10:02:14.100427+0000","last_became_peered":"2024-09-21T10:02:14.100427+0000","last_unstale":"2024-09-21T10:05:55.596360+0000","last_undegraded":"2024-09-21T10:05:55.596360+0000","last_fullsized":"2024-09-21T10:05:55.596360+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:16:42.350642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":90,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.581056+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:05:46.581056+0000","last_peered":"2024-09-21T10:05:46.581056+0000","last_clean":"2024-09-21T10:05:46.581056+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:05:46.581056+0000","last_undegraded":"2024-09-21T10:05:46.581056+0000","last_fullsized":"2024-09-21T10:05:46.581056+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":136,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596456+0000","last_change":"2024-09-21T10:02:14.097369+0000","last_active":"2024-09-21T10:05:55.596456+0000","last_peered":"2024-09-21T10:05:55.596456+0000","last_clean":"2024-09-21T10:05:55.596456+0000","last_became_active":"2024-09-21T10:02:14.097067+0000","last_became_peered":"2024-09-21T10:02:14.097067+0000","last_unstale":"2024-09-21T10:05:55.596456+0000","last_undegraded":"2024-09-21T10:05:55.596456+0000","last_fullsized":"2024-09-21T10:05:55.596456+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:50:48.123813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":115,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.593341+0000","last_change":"2024-09-21T10:02:14.098970+0000","last_active":"2024-09-21T10:05:55.593341+0000","last_peered":"2024-09-21T10:05:55.593341+0000","last_clean":"2024-09-21T10:05:55.593341+0000","last_became_active":"2024-09-21T10:02:14.098641+0000","last_became_peered":"2024-09-21T10:02:14.098641+0000","last_unstale":"2024-09-21T10:05:55.593341+0000","last_undegraded":"2024-09-21T10:05:55.593341+0000","last_fullsized":"2024-09-21T10:05:55.593341+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:25.215602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":106,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T10:06:08.640421+0000","last_change":"2024-09-21T10:02:14.097021+0000","last_active":"2024-09-21T10:06:08.640421+0000","last_peered":"2024-09-21T10:06:08.640421+0000","last_clean":"2024-09-21T10:06:08.640421+0000","last_became_active":"2024-09-21T10:02:14.096741+0000","last_became_peered":"2024-09-21T10:02:14.096741+0000","last_unstale":"2024-09-21T10:06:08.640421+0000","last_undegraded":"2024-09-21T10:06:08.640421+0000","last_fullsized":"2024-09-21T10:06:08.640421+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:18:20.631150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":118,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596838+0000","last_change":"2024-09-21T10:02:14.099460+0000","last_active":"2024-09-21T10:05:55.596838+0000","last_peered":"2024-09-21T10:05:55.596838+0000","last_clean":"2024-09-21T10:05:55.596838+0000","last_became_active":"2024-09-21T10:02:14.099159+0000","last_became_peered":"2024-09-21T10:02:14.099159+0000","last_unstale":"2024-09-21T10:05:55.596838+0000","last_undegraded":"2024-09-21T10:05:55.596838+0000","last_fullsized":"2024-09-21T10:05:55.596838+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:23:47.599107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":109,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.596955+0000","last_change":"2024-09-21T10:01:41.360480+0000","last_active":"2024-09-21T10:05:46.596955+0000","last_peered":"2024-09-21T10:05:46.596955+0000","last_clean":"2024-09-21T10:05:46.596955+0000","last_became_active":"2024-09-21T10:01:41.359791+0000","last_became_peered":"2024-09-21T10:01:41.359791+0000","last_unstale":"2024-09-21T10:05:46.596955+0000","last_undegraded":"2024-09-21T10:05:46.596955+0000","last_fullsized":"2024-09-21T10:05:46.596955+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:42.812974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T10:06:08.641308+0000","last_change":"2024-09-21T10:02:14.102826+0000","last_active":"2024-09-21T10:06:08.641308+0000","last_peered":"2024-09-21T10:06:08.641308+0000","last_clean":"2024-09-21T10:06:08.641308+0000","last_became_active":"2024-09-21T10:02:14.102499+0000","last_became_peered":"2024-09-21T10:02:14.102499+0000","last_unstale":"2024-09-21T10:06:08.641308+0000","last_undegraded":"2024-09-21T10:06:08.641308+0000","last_fullsized":"2024-09-21T10:06:08.641308+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:30.924687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":107,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.600167+0000","last_change":"2024-09-21T09:59:54.268378+0000","last_active":"2024-09-21T10:05:55.600167+0000","last_peered":"2024-09-21T10:05:55.600167+0000","last_clean":"2024-09-21T10:05:55.600167+0000","last_became_active":"2024-09-21T09:59:54.267187+0000","last_became_peered":"2024-09-21T09:59:54.267187+0000","last_unstale":"2024-09-21T10:05:55.600167+0000","last_undegraded":"2024-09-21T10:05:55.600167+0000","last_fullsized":"2024-09-21T10:05:55.600167+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:12:06.054483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":107,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.578635+0000","last_change":"2024-09-21T10:02:14.100541+0000","last_active":"2024-09-21T10:05:46.578635+0000","last_peered":"2024-09-21T10:05:46.578635+0000","last_clean":"2024-09-21T10:05:46.578635+0000","last_became_active":"2024-09-21T10:02:14.100327+0000","last_became_peered":"2024-09-21T10:02:14.100327+0000","last_unstale":"2024-09-21T10:05:46.578635+0000","last_undegraded":"2024-09-21T10:05:46.578635+0000","last_fullsized":"2024-09-21T10:05:46.578635+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:34:34.304188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":92,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.571584+0000","last_change":"2024-09-21T10:01:07.424248+0000","last_active":"2024-09-21T10:05:46.571584+0000","last_peered":"2024-09-21T10:05:46.571584+0000","last_clean":"2024-09-21T10:05:46.571584+0000","last_became_active":"2024-09-21T10:01:07.423738+0000","last_became_peered":"2024-09-21T10:01:07.423738+0000","last_unstale":"2024-09-21T10:05:46.571584+0000","last_undegraded":"2024-09-21T10:05:46.571584+0000","last_fullsized":"2024-09-21T10:05:46.571584+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:44:35.663555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":107,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.597051+0000","last_change":"2024-09-21T10:02:14.096583+0000","last_active":"2024-09-21T10:05:46.597051+0000","last_peered":"2024-09-21T10:05:46.597051+0000","last_clean":"2024-09-21T10:05:46.597051+0000","last_became_active":"2024-09-21T10:02:14.096196+0000","last_became_peered":"2024-09-21T10:02:14.096196+0000","last_unstale":"2024-09-21T10:05:46.597051+0000","last_undegraded":"2024-09-21T10:05:46.597051+0000","last_fullsized":"2024-09-21T10:05:46.597051+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:07:01.888435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":75,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.571799+0000","last_change":"2024-09-21T09:52:20.371638+0000","last_active":"2024-09-21T10:05:46.571799+0000","last_peered":"2024-09-21T10:05:46.571799+0000","last_clean":"2024-09-21T10:05:46.571799+0000","last_became_active":"2024-09-21T09:52:20.371383+0000","last_became_peered":"2024-09-21T09:52:20.371383+0000","last_unstale":"2024-09-21T10:05:46.571799+0000","last_undegraded":"2024-09-21T10:05:46.571799+0000","last_fullsized":"2024-09-21T10:05:46.571799+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:33:45.384397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":139,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596807+0000","last_change":"2024-09-21T10:02:14.099799+0000","last_active":"2024-09-21T10:05:55.596807+0000","last_peered":"2024-09-21T10:05:55.596807+0000","last_clean":"2024-09-21T10:05:55.596807+0000","last_became_active":"2024-09-21T10:02:14.099375+0000","last_became_peered":"2024-09-21T10:02:14.099375+0000","last_unstale":"2024-09-21T10:05:55.596807+0000","last_undegraded":"2024-09-21T10:05:55.596807+0000","last_fullsized":"2024-09-21T10:05:55.596807+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:13.407638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":124,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.596950+0000","last_change":"2024-09-21T10:02:14.099439+0000","last_active":"2024-09-21T10:05:55.596950+0000","last_peered":"2024-09-21T10:05:55.596950+0000","last_clean":"2024-09-21T10:05:55.596950+0000","last_became_active":"2024-09-21T10:02:14.099117+0000","last_became_peered":"2024-09-21T10:02:14.099117+0000","last_unstale":"2024-09-21T10:05:55.596950+0000","last_undegraded":"2024-09-21T10:05:55.596950+0000","last_fullsized":"2024-09-21T10:05:55.596950+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:39.681817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":100,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T10:05:46.578809+0000","last_change":"2024-09-21T10:01:41.360674+0000","last_active":"2024-09-21T10:05:46.578809+0000","last_peered":"2024-09-21T10:05:46.578809+0000","last_clean":"2024-09-21T10:05:46.578809+0000","last_became_active":"2024-09-21T10:01:41.360258+0000","last_became_peered":"2024-09-21T10:01:41.360258+0000","last_unstale":"2024-09-21T10:05:46.578809+0000","last_undegraded":"2024-09-21T10:05:46.578809+0000","last_fullsized":"2024-09-21T10:05:46.578809+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:06.200022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":119,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-21T10:05:55.593208+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:05:55.593208+0000","last_peered":"2024-09-21T10:05:55.593208+0000","last_clean":"2024-09-21T10:05:55.593208+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:05:55.593208+0000","last_undegraded":"2024-09-21T10:05:55.593208+0000","last_fullsized":"2024-09-21T10:05:55.593208+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":6100353048,"num_objects":93085,"num_object_clones":0,"num_object_copies":279255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93085,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178187,"num_write_kb":8680641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18301333504,"data_stored":18301321288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145987,"ondisk_log_size":145987,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5983,"num_read_kb":13006,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397696,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2178892,"kb_used_data":1873220,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91566260,"statfs":{"total":95995035648,"available":93763850240,"internally_reserved":0,"allocated":1918177280,"data_stored":1917522592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593924,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1808640,"kb_used_data":1502968,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91936512,"statfs":{"total":95995035648,"available":94142988288,"internally_reserved":0,"allocated":1539039232,"data_stored":1538396681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790154,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3688244,"kb_used_data":3378412,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":90056908,"statfs":{"total":95995035648,"available":92218273792,"internally_reserved":0,"allocated":3459493888,"data_stored":3458858616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986383,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3284160,"kb_used_data":2974136,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":90460992,"statfs":{"total":95995035648,"available":92632055808,"internally_reserved":0,"allocated":3045515264,"data_stored":3044872218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":101,"seq":433791696945,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1198124,"kb_used_data":1122084,"kb_used_omap":5,"kb_used_meta":76026,"kb_avail":92547028,"statfs":{"total":95995035648,"available":94768156672,"internally_reserved":0,"allocated":1149014016,"data_stored":1148386670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":77851408},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860472,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2351300,"kb_used_data":2239096,"kb_used_omap":8,"kb_used_meta":112183,"kb_avail":91393852,"statfs":{"total":95995035648,"available":93587304448,"internally_reserved":0,"allocated":2292834304,"data_stored":2292195358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":114876268},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":91,"seq":390842023999,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1915568,"kb_used_data":1817192,"kb_used_omap":7,"kb_used_meta":98296,"kb_avail":91829584,"statfs":{"total":95995035648,"available":94033494016,"internally_reserved":0,"allocated":1860804608,"data_stored":1860180807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":100655666},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187533,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3175220,"kb_used_data":3044972,"kb_used_omap":8,"kb_used_meta":130231,"kb_avail":90569932,"statfs":{"total":95995035648,"available":92743610368,"internally_reserved":0,"allocated":3118051328,"data_stored":3117416844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":133357553},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3091070976,"data_stored":3091070976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1858797568,"data_stored":1858797568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2273710080,"data_stored":2273706008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1138622464,"data_stored":1138622464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3043426304,"data_stored":3043426304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3453620224,"data_stored":3453616152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1534132224,"data_stored":1534132224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1907953664,"data_stored":1907949592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:06:12.782 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pgp_num 1 2024-09-21T10:06:12.783 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:12.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:12.837 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:12.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:12.938 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:12.977 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:12.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:12.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:13.079 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:13.106 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:13.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:13.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:13.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:12 smithi196 ceph-mon[84784]: pgmap v335: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 828 op/s 2024-09-21T10:06:13.208 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:13.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:13.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:13.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:13.405 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:06:13.405 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:06:13.427 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:13.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:13.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:13.448 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:06:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:06:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:13.500 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:06:13.500 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:06:13.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:13.542 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:06:13.542 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:06:13.550 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:13.574 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:13.574 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:06:13.574 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:06:13.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:13.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:13.605 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:06:13.605 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:06:13.640 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:06:13.640 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:06:13.671 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:06:13.671 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:06:13.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:13.716 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:13.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:13.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:13.818 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:13.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:14.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:14.476 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:13 smithi098 ceph-mon[97580]: from='client.45059 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:06:14.476 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:13 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3217199235' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:14.476 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:13 smithi098 ceph-mon[100914]: from='client.45059 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:06:14.476 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:13 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3217199235' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:14.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:13 smithi196 ceph-mon[84784]: from='client.45059 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:06:14.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:13 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3217199235' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:14.478 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:06:14.577 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:14.649 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:14.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:14.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:14.751 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:14.775 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:14.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:14.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:14.877 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:14.912 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:14.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:14.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:15.011 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:14 smithi098 ceph-mon[97580]: pgmap v336: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 873 op/s 2024-09-21T10:06:15.011 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:14 smithi098 ceph-mon[100914]: pgmap v336: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 873 op/s 2024-09-21T10:06:15.014 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:15.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:15.044 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:15.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:15.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:14 smithi196 ceph-mon[84784]: pgmap v336: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 873 op/s 2024-09-21T10:06:15.146 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:15.189 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:15.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:15.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:15.290 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:15.340 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:15.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:15.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:15.444 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:15.487 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:15.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:15.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:15.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:15.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:15.623 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:15.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:15.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:15.757 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:15.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:15.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:15.860 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:15.880 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:15.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:15.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:15.982 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:16.040 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:16.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:16.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:16.051 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:06:16.142 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:16.177 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:16.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:16.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:16.279 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:16.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:16.316 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:16.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:16.417 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:16.444 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:16.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:16.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:16.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:06:16.546 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:16.580 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:16.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:16.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:16.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:16.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:16.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:16.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:16.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:16 smithi098 ceph-mon[97580]: pgmap v337: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 823 op/s 2024-09-21T10:06:16.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:16 smithi098 ceph-mon[100914]: pgmap v337: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 823 op/s 2024-09-21T10:06:16.866 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:16.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:16.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:16 smithi196 ceph-mon[84784]: pgmap v337: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 823 op/s 2024-09-21T10:06:16.969 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:17.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:17.014 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:17.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:17.115 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:17.155 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:17.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:17.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:17.258 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:17.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:17.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:17.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:17.422 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:17.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:17.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:17.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:17.567 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:17.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:17.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:17.669 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:17.727 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:17.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:17.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:17.828 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:17.865 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:17.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:17.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:17.967 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:18.003 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:18.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:18.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:18.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:18.147 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:18.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:18.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:18.248 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:18.280 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:18.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:18.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:18.382 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:18.418 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:18.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:18.446 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:18.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:18.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:18.597 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:18.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:18.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:18.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:18.713 INFO:teuthology.orchestra.run.smithi098.stderr:set pool 1 pgp_num to 1 2024-09-21T10:06:18.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:18.747 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:18.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:18 smithi098 ceph-mon[97580]: pgmap v338: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 824 op/s 2024-09-21T10:06:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:18 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1756202697' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-21T10:06:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:18 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-21T10:06:18.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:18 smithi098 ceph-mon[100914]: pgmap v338: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 824 op/s 2024-09-21T10:06:18.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:18 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1756202697' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-21T10:06:18.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:18 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-21T10:06:18.848 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:18.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:18.888 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:18.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:18.989 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:19.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:19.028 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:19.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:19.129 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:19.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:18 smithi196 ceph-mon[84784]: pgmap v338: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 824 op/s 2024-09-21T10:06:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:18 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1756202697' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-21T10:06:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:18 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-21T10:06:19.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:19.162 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:19.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:19.263 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:19.308 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:19.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:19.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:19.410 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:19.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:19.435 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:19.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:19.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:19.570 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:19.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:19.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:19.671 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:19.700 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:19.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:19.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:19.707 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:06:19.801 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:19.822 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:19.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:19.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:19.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:19 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-21T10:06:19.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:19 smithi098 ceph-mon[100914]: osdmap e128: 8 total, 8 up, 8 in 2024-09-21T10:06:19.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:19 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/291998707' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:06:19.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:19 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-21T10:06:19.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:19 smithi098 ceph-mon[97580]: osdmap e128: 8 total, 8 up, 8 in 2024-09-21T10:06:19.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:19 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/291998707' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:06:19.924 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:19.962 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:19.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:19.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:20.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:20.097 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:20.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:20.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:20.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:19 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-21T10:06:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:19 smithi196 ceph-mon[84784]: osdmap e128: 8 total, 8 up, 8 in 2024-09-21T10:06:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:19 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/291998707' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:06:20.198 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:20.223 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:20.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:20.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:20.325 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:20.353 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:20.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:20.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:20.455 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:20.493 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:20.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:20.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:20.595 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:20.636 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:20.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:20.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:20.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:06:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:06:20.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:06:20.738 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:20.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:20.773 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:20.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:20.803 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:06:20.804 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:06:20.823 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:06:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:06:20] "GET /metrics HTTP/1.1" 200 36458 "" "Prometheus/2.51.0" 2024-09-21T10:06:20.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:20 smithi098 ceph-mon[100914]: pgmap v340: 89 pgs: 89 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 824 op/s 2024-09-21T10:06:20.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:20 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:06:20.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:20 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:06:20.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:20 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/291998707' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:06:20.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:20 smithi098 ceph-mon[100914]: osdmap e129: 8 total, 8 up, 8 in 2024-09-21T10:06:20.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:20 smithi098 ceph-mon[97580]: pgmap v340: 89 pgs: 89 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 824 op/s 2024-09-21T10:06:20.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:20 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:06:20.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:20 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:06:20.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:20 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/291998707' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:06:20.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:20 smithi098 ceph-mon[97580]: osdmap e129: 8 total, 8 up, 8 in 2024-09-21T10:06:20.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:20.905 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:20.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:20.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:21.007 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:21.042 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:21.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:21.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:20 smithi196 ceph-mon[84784]: pgmap v340: 89 pgs: 89 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 824 op/s 2024-09-21T10:06:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:20 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:06:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:20 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:06:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:20 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/291998707' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:06:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:20 smithi196 ceph-mon[84784]: osdmap e129: 8 total, 8 up, 8 in 2024-09-21T10:06:21.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:21.182 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:21.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:21.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:21.217 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:06:21.217 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:06:21.251 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:06:21.251 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:06:21.282 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:06:21.282 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:06:21.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:21.309 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:06:21.309 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:06:21.318 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:21.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:21.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:21.345 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:06:21.345 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:06:21.377 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:06:21.377 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:06:21.413 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:06:21.414 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:06:21.420 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:21.446 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:06:21.447 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:06:21.448 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:21.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:21.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:21.550 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:21.582 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:21.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:21.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:21.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:21.712 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:21.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:21.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:21.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:21.842 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:21.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:21.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:21.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:21.977 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:21.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:21.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:22.078 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:22.092 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:21 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3694954171' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:22.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:21 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3694954171' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:22.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:21 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3694954171' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:22.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:22.121 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:22.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:22.129 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:06:22.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:22.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:22.254 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:22.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:22.356 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:22.379 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:22.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:22.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:22.481 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:22.517 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:22.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:22.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:22.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:22.648 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:22.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:22.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:22.652 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:06:22.750 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:22.811 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:22.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:22.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:22.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:22 smithi098 ceph-mon[100914]: pgmap v342: 89 pgs: 89 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 835 op/s 2024-09-21T10:06:22.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:06:22.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:22.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:06:22.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:06:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:22 smithi098 ceph-mon[97580]: pgmap v342: 89 pgs: 89 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 835 op/s 2024-09-21T10:06:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:06:22.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:22.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:06:22.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:06:22.913 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:22.943 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:22.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:22.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:22.974 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:22 smithi196 ceph-mon[84784]: pgmap v342: 89 pgs: 89 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 835 op/s 2024-09-21T10:06:22.975 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:06:22.975 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:22.975 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:06:22.975 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:06:23.044 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:23.078 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:23.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:23.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:23.180 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:23.204 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:23.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:23.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:23.307 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:23.341 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:23.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:23.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:23.443 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:23.470 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:23.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:23.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:23.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:23.605 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:23.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:23.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:23.707 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:23.745 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:23.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:23.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:23.846 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:23.873 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:23.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:23.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:23.975 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:24.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:24.022 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:24.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:24.123 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:24.157 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:24.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:24.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:24.258 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:24.291 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:24.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:24.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:24.393 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:24.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:24.442 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:24.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:24.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:06:24.543 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:24.591 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:24.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:24.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:24.617 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:06:24.618 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:06:24.618 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-21T10:06:24.618 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-21T10:06:24.618 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:06:24.692 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:24.725 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:24.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:24.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:24.826 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:24 smithi098 ceph-mon[97580]: pgmap v343: 89 pgs: 89 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 704 op/s 2024-09-21T10:06:24.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:24 smithi098 ceph-mon[100914]: pgmap v343: 89 pgs: 89 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 704 op/s 2024-09-21T10:06:24.867 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:24.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:24.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:24.969 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:25.002 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:25.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:25.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:25.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:25.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:24 smithi196 ceph-mon[84784]: pgmap v343: 89 pgs: 89 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 704 op/s 2024-09-21T10:06:25.144 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:25.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:25.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:25.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:25.271 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:25.274 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:06:25.298 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:25.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:25.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:25.400 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:25.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:25.431 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:25.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:25.522 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:06:25.532 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:25.582 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:25.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:25.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:25.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:25.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:25.711 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:25.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:25.812 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:25.839 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:25.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:25.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:25.861 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:06:25.862 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:06:25.862 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T10:06:25.862 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.846500 2024-09-21T10:06:25.862 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 7 0.8464998125120695 2024-09-21T10:06:25.941 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:25.949 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:06:25.974 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:25.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:25.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:26.076 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:26.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:26.114 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:26.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:26.215 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:26.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:26.263 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:26.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:26.364 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:26.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:26.397 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:26.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:26.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:26.530 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:26.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:26.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:26.632 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:26.668 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:26.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:26.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:26.770 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:26.814 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:26.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:26.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:26.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:26.956 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:26.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:26.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:27.058 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:27.087 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:27.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:27.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:26 smithi098 ceph-mon[97580]: pgmap v344: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 990 op/s 2024-09-21T10:06:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:26 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:06:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:26 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:06:27.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:26 smithi098 ceph-mon[100914]: pgmap v344: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 990 op/s 2024-09-21T10:06:27.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:26 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:06:27.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:26 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:06:27.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:26 smithi196 ceph-mon[84784]: pgmap v344: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 990 op/s 2024-09-21T10:06:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:26 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:06:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:26 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:06:27.189 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:27.220 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:27.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:27.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:27.322 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:27.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:27.365 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:27.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:27.466 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:27.499 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:27.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:27.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:27.601 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:27.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:27.650 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:27.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:27.750 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:27.758 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:27.773 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:27.822 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:27.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:27.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:27.834 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:27 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:06:27.834 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:27 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:06:27.834 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:27 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:06:27.905 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:27 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:06:27.906 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:27 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:06:27.906 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:27 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:06:27.906 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:27 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:06:27.906 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:27 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:06:27.906 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:27 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:06:27.923 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:27.955 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:27.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:27.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:28.057 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:28.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:28.090 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:28.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:28.190 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:28.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:28.235 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:28.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:28.276 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:06:28.276 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-21T10:06:28.276 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-21T10:06:28.276 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:06:28.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.277 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:35225 5.2 5:444626bb:::benchmark_data_smithi098_135187_object35224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:06:28.277 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:01.635693+0000", 2024-09-21T10:06:28.277 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 146.63615479699999, 2024-09-21T10:06:28.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.10436506800000001, 2024-09-21T10:06:28.277 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:28.277 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:28.277 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:28.277 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:28.277 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:28.277 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35225 2024-09-21T10:06:28.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.277 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:28.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.278 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:28.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.635693+0000", 2024-09-21T10:06:28.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.278 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:28.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.635693+0000", 2024-09-21T10:06:28.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.278 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:28.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.635691+0000", 2024-09-21T10:06:28.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:06:28.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:28.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.635732+0000", 2024-09-21T10:06:28.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1350000000000002e-05 2024-09-21T10:06:28.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:28.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.635744+0000", 2024-09-21T10:06:28.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1780000000000001e-05 2024-09-21T10:06:28.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:28.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.635751+0000", 2024-09-21T10:06:28.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1509999999999998e-06 2024-09-21T10:06:28.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:28.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.635791+0000", 2024-09-21T10:06:28.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9376999999999999e-05 2024-09-21T10:06:28.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:28.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.635898+0000", 2024-09-21T10:06:28.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010759 2024-09-21T10:06:28.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:28.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.636020+0000", 2024-09-21T10:06:28.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000121445 2024-09-21T10:06:28.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:28.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.637439+0000", 2024-09-21T10:06:28.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014191309999999999 2024-09-21T10:06:28.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.638516+0000", 2024-09-21T10:06:28.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001077644 2024-09-21T10:06:28.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.739996+0000", 2024-09-21T10:06:28.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.101479032 2024-09-21T10:06:28.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:28.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.740033+0000", 2024-09-21T10:06:28.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7373999999999998e-05 2024-09-21T10:06:28.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:28.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.740058+0000", 2024-09-21T10:06:28.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4746e-05 2024-09-21T10:06:28.283 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.283 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:28.283 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.283 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:35230 5.2 5:495d761a:::benchmark_data_smithi098_135187_object35229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:06:28.284 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:01.638968+0000", 2024-09-21T10:06:28.284 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 146.63287937000001, 2024-09-21T10:06:28.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.101622658, 2024-09-21T10:06:28.284 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:28.284 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:28.284 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:28.284 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:28.284 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:28.284 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35230 2024-09-21T10:06:28.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.284 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:28.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:28.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.638968+0000", 2024-09-21T10:06:28.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.285 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:28.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.638968+0000", 2024-09-21T10:06:28.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.285 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:28.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.638966+0000", 2024-09-21T10:06:28.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:06:28.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.286 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:28.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.639039+0000", 2024-09-21T10:06:28.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2706000000000005e-05 2024-09-21T10:06:28.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.286 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:28.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.639054+0000", 2024-09-21T10:06:28.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4598e-05 2024-09-21T10:06:28.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.286 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:28.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.639060+0000", 2024-09-21T10:06:28.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0299999999999999e-06 2024-09-21T10:06:28.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:28.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.639109+0000", 2024-09-21T10:06:28.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9636999999999999e-05 2024-09-21T10:06:28.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:28.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.639196+0000", 2024-09-21T10:06:28.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6482000000000003e-05 2024-09-21T10:06:28.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:28.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.639317+0000", 2024-09-21T10:06:28.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000121459 2024-09-21T10:06:28.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:28.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.640629+0000", 2024-09-21T10:06:28.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001312159 2024-09-21T10:06:28.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.640820+0000", 2024-09-21T10:06:28.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00019025800000000001 2024-09-21T10:06:28.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.740510+0000", 2024-09-21T10:06:28.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.099690117999999994 2024-09-21T10:06:28.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:28.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.740557+0000", 2024-09-21T10:06:28.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7058000000000002e-05 2024-09-21T10:06:28.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:28.290 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.740591+0000", 2024-09-21T10:06:28.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3918000000000003e-05 2024-09-21T10:06:28.290 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.290 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:28.290 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.290 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:40340 5.2 5:4371f465:::benchmark_data_smithi098_135187_object40339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:06:28.290 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:04.253586+0000", 2024-09-21T10:06:28.290 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 144.01826149499999, 2024-09-21T10:06:28.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.056361873999999999, 2024-09-21T10:06:28.290 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:28.290 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:28.290 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:28.290 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:28.291 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:28.291 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 40340 2024-09-21T10:06:28.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.291 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:28.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.292 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:28.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:04.253586+0000", 2024-09-21T10:06:28.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.292 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:28.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:04.253586+0000", 2024-09-21T10:06:28.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.292 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:28.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:04.253584+0000", 2024-09-21T10:06:28.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:06:28.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:28.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:04.253628+0000", 2024-09-21T10:06:28.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3405999999999997e-05 2024-09-21T10:06:28.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:28.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:04.253639+0000", 2024-09-21T10:06:28.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1117e-05 2024-09-21T10:06:28.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:28.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:04.253643+0000", 2024-09-21T10:06:28.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2799999999999997e-06 2024-09-21T10:06:28.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:28.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:04.253684+0000", 2024-09-21T10:06:28.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1282000000000002e-05 2024-09-21T10:06:28.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:28.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:04.253815+0000", 2024-09-21T10:06:28.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013091099999999999 2024-09-21T10:06:28.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:28.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:04.253961+0000", 2024-09-21T10:06:28.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000145983 2024-09-21T10:06:28.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:28.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:04.255466+0000", 2024-09-21T10:06:28.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001504451 2024-09-21T10:06:28.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:04.255566+0000", 2024-09-21T10:06:28.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100647 2024-09-21T10:06:28.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:04.309854+0000", 2024-09-21T10:06:28.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.054287955999999998 2024-09-21T10:06:28.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:28.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:04.309911+0000", 2024-09-21T10:06:28.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7176000000000003e-05 2024-09-21T10:06:28.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:28.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:04.309948+0000", 2024-09-21T10:06:28.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6331999999999997e-05 2024-09-21T10:06:28.297 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.297 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:28.297 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.298 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:72288 5.2 5:4f79b5b0:::benchmark_data_smithi098_135187_object72287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T10:06:28.298 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:22.602287+0000", 2024-09-21T10:06:28.298 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 125.669560108, 2024-09-21T10:06:28.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.073969280999999998, 2024-09-21T10:06:28.298 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:28.298 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:28.298 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:28.298 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:28.299 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:28.299 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 72288 2024-09-21T10:06:28.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.299 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:28.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.299 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:28.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602287+0000", 2024-09-21T10:06:28.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.299 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:28.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602287+0000", 2024-09-21T10:06:28.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:28.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602285+0000", 2024-09-21T10:06:28.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999976 2024-09-21T10:06:28.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:28.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602326+0000", 2024-09-21T10:06:28.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1257999999999997e-05 2024-09-21T10:06:28.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:28.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602343+0000", 2024-09-21T10:06:28.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.632e-05 2024-09-21T10:06:28.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:28.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602350+0000", 2024-09-21T10:06:28.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8339999999999999e-06 2024-09-21T10:06:28.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:28.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602445+0000", 2024-09-21T10:06:28.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5016e-05 2024-09-21T10:06:28.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:28.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602554+0000", 2024-09-21T10:06:28.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108857 2024-09-21T10:06:28.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:28.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602686+0000", 2024-09-21T10:06:28.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000131376 2024-09-21T10:06:28.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:28.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.607760+0000", 2024-09-21T10:06:28.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0050746979999999999 2024-09-21T10:06:28.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.610206+0000", 2024-09-21T10:06:28.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002446045 2024-09-21T10:06:28.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.676164+0000", 2024-09-21T10:06:28.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.065957872000000001 2024-09-21T10:06:28.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:28.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.676219+0000", 2024-09-21T10:06:28.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4969999999999997e-05 2024-09-21T10:06:28.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.304 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:28.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.676257+0000", 2024-09-21T10:06:28.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7335000000000001e-05 2024-09-21T10:06:28.304 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.304 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:28.304 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.304 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:72289 5.2 5:46d682ec:::benchmark_data_smithi098_135187_object72288:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T10:06:28.304 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:22.602365+0000", 2024-09-21T10:06:28.304 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 125.669482489, 2024-09-21T10:06:28.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.074037214000000004, 2024-09-21T10:06:28.305 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:28.305 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:28.305 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:28.305 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:28.305 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:28.305 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 72289 2024-09-21T10:06:28.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.305 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:28.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:28.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602365+0000", 2024-09-21T10:06:28.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:28.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602365+0000", 2024-09-21T10:06:28.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:28.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602364+0000", 2024-09-21T10:06:28.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:28.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:28.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602397+0000", 2024-09-21T10:06:28.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2181000000000002e-05 2024-09-21T10:06:28.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:28.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602404+0000", 2024-09-21T10:06:28.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0829999999999998e-06 2024-09-21T10:06:28.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:28.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602407+0000", 2024-09-21T10:06:28.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8800000000000001e-06 2024-09-21T10:06:28.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:28.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605159+0000", 2024-09-21T10:06:28.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027516979999999999 2024-09-21T10:06:28.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:28.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605271+0000", 2024-09-21T10:06:28.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011178100000000001 2024-09-21T10:06:28.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:28.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605402+0000", 2024-09-21T10:06:28.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013075700000000001 2024-09-21T10:06:28.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:28.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.607870+0000", 2024-09-21T10:06:28.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002467959 2024-09-21T10:06:28.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.610271+0000", 2024-09-21T10:06:28.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024008850000000002 2024-09-21T10:06:28.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.676294+0000", 2024-09-21T10:06:28.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.066023803000000006 2024-09-21T10:06:28.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:28.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.676369+0000", 2024-09-21T10:06:28.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4213000000000002e-05 2024-09-21T10:06:28.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:28.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.676402+0000", 2024-09-21T10:06:28.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3491000000000002e-05 2024-09-21T10:06:28.311 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.311 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:28.311 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.311 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:76268 5.2 5:45057f13:::benchmark_data_smithi098_135187_object76267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-21T10:06:28.311 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:25.433893+0000", 2024-09-21T10:06:28.311 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 122.837953988, 2024-09-21T10:06:28.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.049789925999999998, 2024-09-21T10:06:28.311 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:28.311 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:28.311 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:28.311 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:28.311 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:28.312 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 76268 2024-09-21T10:06:28.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.312 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:28.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:28.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:25.433893+0000", 2024-09-21T10:06:28.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:28.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:25.433893+0000", 2024-09-21T10:06:28.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:28.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:25.433892+0000", 2024-09-21T10:06:28.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:06:28.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:28.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:25.433938+0000", 2024-09-21T10:06:28.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6767999999999997e-05 2024-09-21T10:06:28.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:28.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:25.433956+0000", 2024-09-21T10:06:28.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8068e-05 2024-09-21T10:06:28.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.314 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:28.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:25.433963+0000", 2024-09-21T10:06:28.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9059999999999996e-06 2024-09-21T10:06:28.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.314 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:28.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:25.434005+0000", 2024-09-21T10:06:28.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1399000000000001e-05 2024-09-21T10:06:28.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.314 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:28.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:25.434137+0000", 2024-09-21T10:06:28.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013222599999999999 2024-09-21T10:06:28.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:28.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:25.434287+0000", 2024-09-21T10:06:28.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015050799999999999 2024-09-21T10:06:28.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:28.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:25.436904+0000", 2024-09-21T10:06:28.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026168680000000001 2024-09-21T10:06:28.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:25.438039+0000", 2024-09-21T10:06:28.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001134892 2024-09-21T10:06:28.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:25.483627+0000", 2024-09-21T10:06:28.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.045587849999999999 2024-09-21T10:06:28.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:28.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:25.483660+0000", 2024-09-21T10:06:28.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2521999999999998e-05 2024-09-21T10:06:28.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:28.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:25.483683+0000", 2024-09-21T10:06:28.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3666e-05 2024-09-21T10:06:28.317 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.317 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:28.317 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.317 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:77448 5.2 5:43ba49b2:::benchmark_data_smithi098_135187_object77447:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T10:06:28.317 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:26.308703+0000", 2024-09-21T10:06:28.317 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 121.963144402, 2024-09-21T10:06:28.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.052944816999999998, 2024-09-21T10:06:28.317 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:28.318 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:28.318 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:28.318 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:28.318 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:28.318 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 77448 2024-09-21T10:06:28.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.318 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:28.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:28.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.308703+0000", 2024-09-21T10:06:28.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.319 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:28.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.308703+0000", 2024-09-21T10:06:28.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.319 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:28.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.308701+0000", 2024-09-21T10:06:28.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:06:28.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:28.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.308741+0000", 2024-09-21T10:06:28.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9675000000000002e-05 2024-09-21T10:06:28.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:28.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.308758+0000", 2024-09-21T10:06:28.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6756e-05 2024-09-21T10:06:28.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:28.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.308764+0000", 2024-09-21T10:06:28.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6320000000000002e-06 2024-09-21T10:06:28.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:28.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.308902+0000", 2024-09-21T10:06:28.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013771599999999999 2024-09-21T10:06:28.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:28.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.309180+0000", 2024-09-21T10:06:28.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000277877 2024-09-21T10:06:28.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:28.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.309468+0000", 2024-09-21T10:06:28.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00028824299999999998 2024-09-21T10:06:28.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:28.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.312618+0000", 2024-09-21T10:06:28.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003150038 2024-09-21T10:06:28.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.314139+0000", 2024-09-21T10:06:28.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015210099999999999 2024-09-21T10:06:28.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.361509+0000", 2024-09-21T10:06:28.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.047369561999999997 2024-09-21T10:06:28.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:28.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.361610+0000", 2024-09-21T10:06:28.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000101271 2024-09-21T10:06:28.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:28.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.361648+0000", 2024-09-21T10:06:28.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7861000000000002e-05 2024-09-21T10:06:28.324 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.324 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:28.324 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.324 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:77457 5.2 5:43161979:::benchmark_data_smithi098_135187_object77456:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T10:06:28.324 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:26.313733+0000", 2024-09-21T10:06:28.324 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 121.958114143, 2024-09-21T10:06:28.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.054544395000000002, 2024-09-21T10:06:28.324 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:28.324 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:28.324 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:28.325 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:28.325 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:28.325 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 77457 2024-09-21T10:06:28.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.325 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:28.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:28.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.313733+0000", 2024-09-21T10:06:28.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:28.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.313733+0000", 2024-09-21T10:06:28.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.326 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:28.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.313731+0000", 2024-09-21T10:06:28.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:06:28.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.326 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:28.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.313782+0000", 2024-09-21T10:06:28.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0983999999999998e-05 2024-09-21T10:06:28.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.326 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:28.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.313804+0000", 2024-09-21T10:06:28.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1701000000000001e-05 2024-09-21T10:06:28.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:28.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.313811+0000", 2024-09-21T10:06:28.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4150000000000004e-06 2024-09-21T10:06:28.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:28.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.314246+0000", 2024-09-21T10:06:28.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00043543599999999998 2024-09-21T10:06:28.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:28.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.314550+0000", 2024-09-21T10:06:28.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00030448399999999999 2024-09-21T10:06:28.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:28.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.314920+0000", 2024-09-21T10:06:28.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00036947500000000002 2024-09-21T10:06:28.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:28.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.321513+0000", 2024-09-21T10:06:28.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006593336 2024-09-21T10:06:28.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.321647+0000", 2024-09-21T10:06:28.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013336399999999999 2024-09-21T10:06:28.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.368180+0000", 2024-09-21T10:06:28.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.046533839 2024-09-21T10:06:28.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:28.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.368237+0000", 2024-09-21T10:06:28.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6242999999999997e-05 2024-09-21T10:06:28.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:28.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.368278+0000", 2024-09-21T10:06:28.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0877999999999997e-05 2024-09-21T10:06:28.330 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.330 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:28.330 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.331 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:77460 5.2 5:4cb7e3ec:::benchmark_data_smithi098_135187_object77459:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T10:06:28.331 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:26.313823+0000", 2024-09-21T10:06:28.331 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 121.95802438, 2024-09-21T10:06:28.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.056905582000000003, 2024-09-21T10:06:28.331 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:28.331 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:28.331 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:28.331 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:28.331 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:28.331 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 77460 2024-09-21T10:06:28.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.331 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:28.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:28.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.313823+0000", 2024-09-21T10:06:28.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:28.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.313823+0000", 2024-09-21T10:06:28.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:28.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.313823+0000", 2024-09-21T10:06:28.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:28.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:28.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.313852+0000", 2024-09-21T10:06:28.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9706999999999999e-05 2024-09-21T10:06:28.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:28.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.313860+0000", 2024-09-21T10:06:28.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3730000000000004e-06 2024-09-21T10:06:28.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:28.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.313863+0000", 2024-09-21T10:06:28.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.839e-06 2024-09-21T10:06:28.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:28.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.317883+0000", 2024-09-21T10:06:28.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004019557 2024-09-21T10:06:28.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:28.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.318142+0000", 2024-09-21T10:06:28.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025887499999999999 2024-09-21T10:06:28.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:28.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.318741+0000", 2024-09-21T10:06:28.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000599494 2024-09-21T10:06:28.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:28.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.321696+0000", 2024-09-21T10:06:28.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029548090000000001 2024-09-21T10:06:28.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.321810+0000", 2024-09-21T10:06:28.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114252 2024-09-21T10:06:28.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.370640+0000", 2024-09-21T10:06:28.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.048830042999999997 2024-09-21T10:06:28.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:28.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.370693+0000", 2024-09-21T10:06:28.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2459e-05 2024-09-21T10:06:28.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:28.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:26.370729+0000", 2024-09-21T10:06:28.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5644e-05 2024-09-21T10:06:28.337 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.337 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:28.337 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.337 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:85729 5.2 5:468120ce:::benchmark_data_smithi098_135187_object85728:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T10:06:28.337 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:32.605052+0000", 2024-09-21T10:06:28.337 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 115.666795296, 2024-09-21T10:06:28.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.060664740000000002, 2024-09-21T10:06:28.338 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:28.338 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:28.338 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:28.338 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:28.338 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:28.338 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 85729 2024-09-21T10:06:28.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.338 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:28.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:28.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.605052+0000", 2024-09-21T10:06:28.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:28.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.605052+0000", 2024-09-21T10:06:28.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:28.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.605050+0000", 2024-09-21T10:06:28.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:06:28.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:28.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.605148+0000", 2024-09-21T10:06:28.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7705000000000001e-05 2024-09-21T10:06:28.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:28.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.605161+0000", 2024-09-21T10:06:28.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3123e-05 2024-09-21T10:06:28.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:28.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.605168+0000", 2024-09-21T10:06:28.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.268e-06 2024-09-21T10:06:28.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:28.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.605216+0000", 2024-09-21T10:06:28.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8912000000000002e-05 2024-09-21T10:06:28.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:28.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.605318+0000", 2024-09-21T10:06:28.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010147800000000001 2024-09-21T10:06:28.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:28.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.605417+0000", 2024-09-21T10:06:28.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9208000000000003e-05 2024-09-21T10:06:28.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:28.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.607507+0000", 2024-09-21T10:06:28.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020899109999999999 2024-09-21T10:06:28.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.608427+0000", 2024-09-21T10:06:28.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00091966400000000005 2024-09-21T10:06:28.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.665628+0000", 2024-09-21T10:06:28.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.057201211000000002 2024-09-21T10:06:28.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:28.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.665676+0000", 2024-09-21T10:06:28.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7997000000000001e-05 2024-09-21T10:06:28.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:28.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.665717+0000", 2024-09-21T10:06:28.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0911000000000002e-05 2024-09-21T10:06:28.344 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.344 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:28.344 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.344 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:86786 5.2 5:48b080e7:::benchmark_data_smithi098_135187_object86785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T10:06:28.344 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:33.562433+0000", 2024-09-21T10:06:28.344 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 114.709414322, 2024-09-21T10:06:28.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.064391162000000002, 2024-09-21T10:06:28.345 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:28.345 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:28.345 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:28.345 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:28.345 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:28.345 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 86786 2024-09-21T10:06:28.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.345 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:28.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.345 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:28.345 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.562433+0000", 2024-09-21T10:06:28.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:28.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.562433+0000", 2024-09-21T10:06:28.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:28.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.562432+0000", 2024-09-21T10:06:28.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:28.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:28.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.562473+0000", 2024-09-21T10:06:28.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0929000000000001e-05 2024-09-21T10:06:28.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:28.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.562485+0000", 2024-09-21T10:06:28.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1787000000000001e-05 2024-09-21T10:06:28.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:28.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.562490+0000", 2024-09-21T10:06:28.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9599999999999999e-06 2024-09-21T10:06:28.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:28.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.566147+0000", 2024-09-21T10:06:28.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036568030000000001 2024-09-21T10:06:28.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.348 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:28.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.566277+0000", 2024-09-21T10:06:28.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000129727 2024-09-21T10:06:28.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.348 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:28.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.566412+0000", 2024-09-21T10:06:28.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013521900000000001 2024-09-21T10:06:28.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.570734+0000", 2024-09-21T10:06:28.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043224680000000003 2024-09-21T10:06:28.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:28.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.570754+0000", 2024-09-21T10:06:28.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9507000000000001e-05 2024-09-21T10:06:28.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.626767+0000", 2024-09-21T10:06:28.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.056013737000000001 2024-09-21T10:06:28.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:28.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.626801+0000", 2024-09-21T10:06:28.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3399999999999999e-05 2024-09-21T10:06:28.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:28.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.626824+0000", 2024-09-21T10:06:28.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3311000000000001e-05 2024-09-21T10:06:28.350 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.350 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:28.350 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.351 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:86796 5.2 5:44bfc3f9:::benchmark_data_smithi098_135187_object86795:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T10:06:28.351 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:33.575451+0000", 2024-09-21T10:06:28.351 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 114.696396286, 2024-09-21T10:06:28.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.054220861000000002, 2024-09-21T10:06:28.351 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:28.351 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:28.351 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:28.351 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:28.351 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:28.351 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 86796 2024-09-21T10:06:28.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.352 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:28.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:28.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.575451+0000", 2024-09-21T10:06:28.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:28.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.575451+0000", 2024-09-21T10:06:28.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:28.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.575450+0000", 2024-09-21T10:06:28.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:06:28.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:28.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.575495+0000", 2024-09-21T10:06:28.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5328999999999999e-05 2024-09-21T10:06:28.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:28.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.575508+0000", 2024-09-21T10:06:28.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2381999999999999e-05 2024-09-21T10:06:28.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:28.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.575513+0000", 2024-09-21T10:06:28.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5199999999999997e-06 2024-09-21T10:06:28.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:28.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.575550+0000", 2024-09-21T10:06:28.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7212000000000002e-05 2024-09-21T10:06:28.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:28.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.575635+0000", 2024-09-21T10:06:28.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4685999999999995e-05 2024-09-21T10:06:28.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:28.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.575723+0000", 2024-09-21T10:06:28.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8531999999999999e-05 2024-09-21T10:06:28.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:28.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.577763+0000", 2024-09-21T10:06:28.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020391350000000001 2024-09-21T10:06:28.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.578004+0000", 2024-09-21T10:06:28.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00024160300000000001 2024-09-21T10:06:28.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.356 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.629558+0000", 2024-09-21T10:06:28.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.051554042000000001 2024-09-21T10:06:28.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.356 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:28.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.629618+0000", 2024-09-21T10:06:28.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9509000000000002e-05 2024-09-21T10:06:28.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.356 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:28.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:33.629672+0000", 2024-09-21T10:06:28.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4184000000000002e-05 2024-09-21T10:06:28.357 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.357 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:28.357 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.357 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:135354 5.2 5:46c59891:::benchmark_data_smithi098_135187_object135353:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-21T10:06:28.357 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:05:19.364242+0000", 2024-09-21T10:06:28.357 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 68.907605560999997, 2024-09-21T10:06:28.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.056530357000000003, 2024-09-21T10:06:28.357 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:28.357 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:28.357 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:28.358 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:28.358 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:28.358 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 135354 2024-09-21T10:06:28.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.358 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:28.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:28.358 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.364242+0000", 2024-09-21T10:06:28.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:28.358 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.364242+0000", 2024-09-21T10:06:28.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:28.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.364240+0000", 2024-09-21T10:06:28.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:06:28.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:28.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.364356+0000", 2024-09-21T10:06:28.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011585 2024-09-21T10:06:28.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:28.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.364385+0000", 2024-09-21T10:06:28.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8648000000000001e-05 2024-09-21T10:06:28.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:28.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.364391+0000", 2024-09-21T10:06:28.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1380000000000004e-06 2024-09-21T10:06:28.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:28.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.364544+0000", 2024-09-21T10:06:28.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015288699999999999 2024-09-21T10:06:28.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:28.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.364806+0000", 2024-09-21T10:06:28.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000262277 2024-09-21T10:06:28.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:28.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.365181+0000", 2024-09-21T10:06:28.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000374342 2024-09-21T10:06:28.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:28.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.370145+0000", 2024-09-21T10:06:28.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049643389999999999 2024-09-21T10:06:28.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.370351+0000", 2024-09-21T10:06:28.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00020561000000000001 2024-09-21T10:06:28.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.420443+0000", 2024-09-21T10:06:28.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.050092429000000001 2024-09-21T10:06:28.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:28.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.420663+0000", 2024-09-21T10:06:28.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00021991600000000001 2024-09-21T10:06:28.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:28.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.420772+0000", 2024-09-21T10:06:28.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0001093 2024-09-21T10:06:28.363 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.363 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:28.363 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.364 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:135358 5.2 5:4ad46ff3:::benchmark_data_smithi098_135187_object135357:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-21T10:06:28.364 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:05:19.370644+0000", 2024-09-21T10:06:28.364 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 68.901203397000003, 2024-09-21T10:06:28.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.051200778000000002, 2024-09-21T10:06:28.364 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:28.364 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:28.364 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:28.364 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:28.364 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:28.364 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 135358 2024-09-21T10:06:28.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.364 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:28.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:28.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.370644+0000", 2024-09-21T10:06:28.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:28.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.370644+0000", 2024-09-21T10:06:28.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:28.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.370639+0000", 2024-09-21T10:06:28.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999957 2024-09-21T10:06:28.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:28.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.370682+0000", 2024-09-21T10:06:28.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2599999999999999e-05 2024-09-21T10:06:28.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:28.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.370698+0000", 2024-09-21T10:06:28.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5753e-05 2024-09-21T10:06:28.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:28.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.370704+0000", 2024-09-21T10:06:28.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6960000000000002e-06 2024-09-21T10:06:28.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:28.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.370741+0000", 2024-09-21T10:06:28.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7456000000000002e-05 2024-09-21T10:06:28.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:28.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.370855+0000", 2024-09-21T10:06:28.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000113566 2024-09-21T10:06:28.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:28.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.370986+0000", 2024-09-21T10:06:28.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013105199999999999 2024-09-21T10:06:28.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:28.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.376529+0000", 2024-09-21T10:06:28.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055430890000000002 2024-09-21T10:06:28.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.387858+0000", 2024-09-21T10:06:28.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011329805 2024-09-21T10:06:28.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.421393+0000", 2024-09-21T10:06:28.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.033534863999999998 2024-09-21T10:06:28.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:28.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.421569+0000", 2024-09-21T10:06:28.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017531099999999999 2024-09-21T10:06:28.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:28.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:19.421845+0000", 2024-09-21T10:06:28.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00027606899999999999 2024-09-21T10:06:28.371 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.371 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:28.371 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.371 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:176308 5.2 5:4fe381d9:::benchmark_data_smithi098_135187_object40670:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T10:06:28.371 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:06:05.936258+0000", 2024-09-21T10:06:28.371 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 22.335589567, 2024-09-21T10:06:28.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.050052571999999997, 2024-09-21T10:06:28.371 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:28.372 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:28.372 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:28.372 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:28.372 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:28.372 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 176308 2024-09-21T10:06:28.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.372 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:28.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:28.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:05.936258+0000", 2024-09-21T10:06:28.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:28.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:05.936258+0000", 2024-09-21T10:06:28.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:28.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:05.936257+0000", 2024-09-21T10:06:28.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:06:28.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:28.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:05.936262+0000", 2024-09-21T10:06:28.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6879999999999996e-06 2024-09-21T10:06:28.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:28.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:05.936286+0000", 2024-09-21T10:06:28.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4190999999999999e-05 2024-09-21T10:06:28.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:28.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:05.936292+0000", 2024-09-21T10:06:28.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4550000000000003e-06 2024-09-21T10:06:28.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:28.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:05.947109+0000", 2024-09-21T10:06:28.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010817482999999999 2024-09-21T10:06:28.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:28.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:05.947254+0000", 2024-09-21T10:06:28.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000144225 2024-09-21T10:06:28.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:28.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:05.947373+0000", 2024-09-21T10:06:28.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119048 2024-09-21T10:06:28.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:05.956446+0000", 2024-09-21T10:06:28.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0090734779999999994 2024-09-21T10:06:28.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:28.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:05.970558+0000", 2024-09-21T10:06:28.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014112347000000001 2024-09-21T10:06:28.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:05.986026+0000", 2024-09-21T10:06:28.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015467829000000001 2024-09-21T10:06:28.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:28.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:05.986201+0000", 2024-09-21T10:06:28.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017451599999999999 2024-09-21T10:06:28.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:28.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:05.986310+0000", 2024-09-21T10:06:28.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010955000000000001 2024-09-21T10:06:28.378 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.378 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:28.379 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.379 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:191444 5.2 5:4dac7e1b:::benchmark_data_smithi098_135187_object55806:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-21T10:06:28.379 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:06:24.463945+0000", 2024-09-21T10:06:28.379 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 3.8079020529999998, 2024-09-21T10:06:28.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.065793538999999998, 2024-09-21T10:06:28.379 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:28.379 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:28.379 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:28.379 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:28.380 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:28.380 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 191444 2024-09-21T10:06:28.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.380 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:28.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:28.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.463945+0000", 2024-09-21T10:06:28.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:28.380 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:28.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.463945+0000", 2024-09-21T10:06:28.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:28.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.463945+0000", 2024-09-21T10:06:28.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:28.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:28.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.463949+0000", 2024-09-21T10:06:28.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4259999999999996e-06 2024-09-21T10:06:28.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:28.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.463959+0000", 2024-09-21T10:06:28.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0438999999999999e-05 2024-09-21T10:06:28.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:28.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.463965+0000", 2024-09-21T10:06:28.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7939999999999999e-06 2024-09-21T10:06:28.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:28.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.468539+0000", 2024-09-21T10:06:28.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045740579999999998 2024-09-21T10:06:28.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:28.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.468664+0000", 2024-09-21T10:06:28.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012446899999999999 2024-09-21T10:06:28.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:28.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.468781+0000", 2024-09-21T10:06:28.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011671 2024-09-21T10:06:28.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.478478+0000", 2024-09-21T10:06:28.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0096974009999999996 2024-09-21T10:06:28.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:28.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.504401+0000", 2024-09-21T10:06:28.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.025922962000000001 2024-09-21T10:06:28.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.529646+0000", 2024-09-21T10:06:28.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.025245588999999999 2024-09-21T10:06:28.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:28.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.529697+0000", 2024-09-21T10:06:28.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0519000000000002e-05 2024-09-21T10:06:28.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:28.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.529739+0000", 2024-09-21T10:06:28.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1859e-05 2024-09-21T10:06:28.387 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.387 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:28.387 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.387 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:191450 5.2 5:4d167040:::benchmark_data_smithi098_135187_object55812:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-21T10:06:28.387 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:06:24.477375+0000", 2024-09-21T10:06:28.387 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 3.7944726090000001, 2024-09-21T10:06:28.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.052473302999999999, 2024-09-21T10:06:28.387 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:28.387 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:28.387 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:28.388 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:28.388 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:28.388 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 191450 2024-09-21T10:06:28.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.388 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:28.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:28.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.477375+0000", 2024-09-21T10:06:28.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:28.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.477375+0000", 2024-09-21T10:06:28.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:28.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.477365+0000", 2024-09-21T10:06:28.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.99999 2024-09-21T10:06:28.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:28.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.477382+0000", 2024-09-21T10:06:28.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7693000000000002e-05 2024-09-21T10:06:28.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:28.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.477398+0000", 2024-09-21T10:06:28.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6192999999999999e-05 2024-09-21T10:06:28.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:28.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.477406+0000", 2024-09-21T10:06:28.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2599999999999999e-06 2024-09-21T10:06:28.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:28.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.478494+0000", 2024-09-21T10:06:28.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010885630000000001 2024-09-21T10:06:28.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:28.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.478611+0000", 2024-09-21T10:06:28.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000117222 2024-09-21T10:06:28.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:28.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.478726+0000", 2024-09-21T10:06:28.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114938 2024-09-21T10:06:28.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.489453+0000", 2024-09-21T10:06:28.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010726584000000001 2024-09-21T10:06:28.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:28.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.504417+0000", 2024-09-21T10:06:28.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014964299 2024-09-21T10:06:28.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.529774+0000", 2024-09-21T10:06:28.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0253566 2024-09-21T10:06:28.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:28.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.529817+0000", 2024-09-21T10:06:28.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3498999999999998e-05 2024-09-21T10:06:28.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:28.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.529848+0000", 2024-09-21T10:06:28.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0645999999999998e-05 2024-09-21T10:06:28.393 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.393 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:28.393 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.394 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:191454 5.2 5:49c7e277:::benchmark_data_smithi098_135187_object55816:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-21T10:06:28.394 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:06:24.480015+0000", 2024-09-21T10:06:28.394 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 3.7918319110000001, 2024-09-21T10:06:28.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.049936191999999997, 2024-09-21T10:06:28.394 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:28.394 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:28.394 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:28.394 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:28.394 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:28.394 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 191454 2024-09-21T10:06:28.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.395 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:28.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:28.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.480015+0000", 2024-09-21T10:06:28.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:28.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.480015+0000", 2024-09-21T10:06:28.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:28.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.480014+0000", 2024-09-21T10:06:28.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:06:28.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:28.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.480023+0000", 2024-09-21T10:06:28.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3959999999999995e-06 2024-09-21T10:06:28.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:28.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.480040+0000", 2024-09-21T10:06:28.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6875999999999999e-05 2024-09-21T10:06:28.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:28.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.480046+0000", 2024-09-21T10:06:28.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2740000000000004e-06 2024-09-21T10:06:28.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:28.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.484144+0000", 2024-09-21T10:06:28.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0040976720000000001 2024-09-21T10:06:28.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:28.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.484564+0000", 2024-09-21T10:06:28.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00041962199999999997 2024-09-21T10:06:28.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:28.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.484810+0000", 2024-09-21T10:06:28.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00024614400000000001 2024-09-21T10:06:28.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.494286+0000", 2024-09-21T10:06:28.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0094761729999999992 2024-09-21T10:06:28.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:28.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.504434+0000", 2024-09-21T10:06:28.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010147741 2024-09-21T10:06:28.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.529885+0000", 2024-09-21T10:06:28.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.025451403000000001 2024-09-21T10:06:28.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:28.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.529923+0000", 2024-09-21T10:06:28.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8383000000000002e-05 2024-09-21T10:06:28.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:28.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:24.529952+0000", 2024-09-21T10:06:28.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8235999999999999e-05 2024-09-21T10:06:28.399 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.400 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:28.400 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.400 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:194354 5.2 5:4b7967e5:::benchmark_data_smithi098_135187_object58716:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-21T10:06:28.400 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:06:27.968059+0000", 2024-09-21T10:06:28.400 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.303788314, 2024-09-21T10:06:28.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.048999388999999997, 2024-09-21T10:06:28.400 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:28.400 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:28.400 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:28.400 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:28.400 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:28.401 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 194354 2024-09-21T10:06:28.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.401 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:28.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:28.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.968059+0000", 2024-09-21T10:06:28.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:28.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.968059+0000", 2024-09-21T10:06:28.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:28.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.968058+0000", 2024-09-21T10:06:28.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:06:28.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:28.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.968064+0000", 2024-09-21T10:06:28.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.682e-06 2024-09-21T10:06:28.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:28.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.968073+0000", 2024-09-21T10:06:28.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3039999999999994e-06 2024-09-21T10:06:28.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:28.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.968079+0000", 2024-09-21T10:06:28.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7389999999999996e-06 2024-09-21T10:06:28.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:28.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.968119+0000", 2024-09-21T10:06:28.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9866000000000001e-05 2024-09-21T10:06:28.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:28.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.968338+0000", 2024-09-21T10:06:28.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000219934 2024-09-21T10:06:28.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:28.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.969307+0000", 2024-09-21T10:06:28.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00096811399999999995 2024-09-21T10:06:28.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.977000+0000", 2024-09-21T10:06:28.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0076929449999999996 2024-09-21T10:06:28.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:28.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.978420+0000", 2024-09-21T10:06:28.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014204210000000001 2024-09-21T10:06:28.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.016958+0000", 2024-09-21T10:06:28.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.038537977000000001 2024-09-21T10:06:28.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:28.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.017024+0000", 2024-09-21T10:06:28.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6216000000000004e-05 2024-09-21T10:06:28.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:28.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.017058+0000", 2024-09-21T10:06:28.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4301000000000002e-05 2024-09-21T10:06:28.406 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.406 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:28.406 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.406 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:194494 5.2 5:401a0bd7:::benchmark_data_smithi098_135187_object58856:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-21T10:06:28.406 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:06:28.142985+0000", 2024-09-21T10:06:28.407 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.128861899, 2024-09-21T10:06:28.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.056907646999999999, 2024-09-21T10:06:28.407 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:28.407 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:28.407 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:28.407 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:28.407 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:28.407 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 194494 2024-09-21T10:06:28.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.407 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:28.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:28.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.142985+0000", 2024-09-21T10:06:28.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:28.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.142985+0000", 2024-09-21T10:06:28.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:28.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:28.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.142985+0000", 2024-09-21T10:06:28.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:28.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:28.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.142989+0000", 2024-09-21T10:06:28.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1239999999999998e-06 2024-09-21T10:06:28.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:28.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.142997+0000", 2024-09-21T10:06:28.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.507e-06 2024-09-21T10:06:28.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:28.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.143002+0000", 2024-09-21T10:06:28.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3590000000000001e-06 2024-09-21T10:06:28.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:28.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.164142+0000", 2024-09-21T10:06:28.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.021140085999999999 2024-09-21T10:06:28.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:28.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.164254+0000", 2024-09-21T10:06:28.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011191799999999999 2024-09-21T10:06:28.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:28.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.164370+0000", 2024-09-21T10:06:28.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000116072 2024-09-21T10:06:28.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:06:28.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.179036+0000", 2024-09-21T10:06:28.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014665905999999999 2024-09-21T10:06:28.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.179208+0000", 2024-09-21T10:06:28.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017258400000000001 2024-09-21T10:06:28.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:28.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.199555+0000", 2024-09-21T10:06:28.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.020346418000000002 2024-09-21T10:06:28.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:06:28.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.199746+0000", 2024-09-21T10:06:28.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000191714 2024-09-21T10:06:28.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:28.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:28.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:06:28.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.199893+0000", 2024-09-21T10:06:28.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000146639 2024-09-21T10:06:28.412 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.412 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:28.412 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.413 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:28.413 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:28.413 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:06:28.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:28.414 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:28.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:28.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:28.539 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:28.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:28.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:28.641 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:28.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:28.669 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:28.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:28.746 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:06:28.770 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:28.815 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:28.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:28.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:28.917 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:28.955 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:28.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:28.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:29.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:29.084 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:29.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:29.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:29.096 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:28 smithi098 ceph-mon[97580]: pgmap v345: 89 pgs: 89 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 663 op/s 2024-09-21T10:06:29.096 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:28 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/89595505' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:06:29.096 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:28 smithi098 ceph-mon[100914]: pgmap v345: 89 pgs: 89 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 663 op/s 2024-09-21T10:06:29.096 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:28 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/89595505' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:06:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:28 smithi196 ceph-mon[84784]: pgmap v345: 89 pgs: 89 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 663 op/s 2024-09-21T10:06:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:28 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/89595505' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:06:29.186 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:29.228 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:29.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:29.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:29.278 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:06:29.330 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:29.384 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:29.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:29.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:29.487 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:29.515 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:29.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:29.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:29.617 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:29.647 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:29.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:29.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:29.721 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:06:29.750 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:29.802 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:29.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:29.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:29.906 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:29.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:29.947 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:29.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:30.038 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:29 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:30.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:29 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/89595505' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:06:30.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:29 smithi196 ceph-mon[84784]: osdmap e130: 8 total, 8 up, 8 in 2024-09-21T10:06:30.048 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:30.080 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:30.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:30.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:30.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:29 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:30.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:29 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/89595505' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:06:30.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:29 smithi098 ceph-mon[97580]: osdmap e130: 8 total, 8 up, 8 in 2024-09-21T10:06:30.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:29 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:30.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:29 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/89595505' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:06:30.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:29 smithi098 ceph-mon[100914]: osdmap e130: 8 total, 8 up, 8 in 2024-09-21T10:06:30.182 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:30.218 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:30.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:30.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:30.319 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:30.352 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:30.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:30.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:30.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:06:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:06:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:06:30.455 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:30.488 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:30.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:30.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:30.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:30.625 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:30.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:30.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:30.727 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:30.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:30.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:30.830 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:06:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:06:30] "GET /metrics HTTP/1.1" 200 36449 "" "Prometheus/2.51.0" 2024-09-21T10:06:30.830 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:30 smithi098 ceph-mon[97580]: pgmap v347: 89 pgs: 89 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 743 op/s 2024-09-21T10:06:30.830 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:30 smithi098 ceph-mon[100914]: pgmap v347: 89 pgs: 89 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 743 op/s 2024-09-21T10:06:30.861 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:30.899 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:30.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:30.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:31.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:31.031 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:31.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:31.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:31.133 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:31.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:30 smithi196 ceph-mon[84784]: pgmap v347: 89 pgs: 89 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 743 op/s 2024-09-21T10:06:31.169 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:31.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:31.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:31.271 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:31.319 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:31.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:31.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:31.421 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:31.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:31.464 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:31.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:31.565 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:31.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:31.610 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:31.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:31.711 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:31.753 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:31.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:31.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:31.855 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:31.884 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:06:31.884 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:06:31.884 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.884 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:197386 5.4 5:21d58124:::benchmark_data_smithi098_135187_object61748:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T10:06:31.884 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:06:31.870715+0000", 2024-09-21T10:06:31.884 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0031217879999999999, 2024-09-21T10:06:31.884 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003163515, 2024-09-21T10:06:31.884 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:31.884 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:06:31.885 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:31.885 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:31.885 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:31.885 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 197386 2024-09-21T10:06:31.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.885 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:31.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.885 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:31.885 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.870715+0000", 2024-09-21T10:06:31.885 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:31.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.885 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:31.885 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.870715+0000", 2024-09-21T10:06:31.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:31.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.886 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:31.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.870713+0000", 2024-09-21T10:06:31.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999976 2024-09-21T10:06:31.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.886 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:31.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.870725+0000", 2024-09-21T10:06:31.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.199e-05 2024-09-21T10:06:31.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:31.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.870748+0000", 2024-09-21T10:06:31.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3651000000000001e-05 2024-09-21T10:06:31.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:31.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.870757+0000", 2024-09-21T10:06:31.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2579999999999995e-06 2024-09-21T10:06:31.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:31.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.870925+0000", 2024-09-21T10:06:31.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000168834 2024-09-21T10:06:31.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:31.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.871047+0000", 2024-09-21T10:06:31.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012204199999999999 2024-09-21T10:06:31.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:06:31.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.871166+0000", 2024-09-21T10:06:31.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000118768 2024-09-21T10:06:31.888 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:31.888 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:31.888 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:31.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.889 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:197389 5.9 5.99f04459 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T10:06:31.889 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:06:31.870768+0000", 2024-09-21T10:06:31.889 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.003068351, 2024-09-21T10:06:31.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0031928970000000001, 2024-09-21T10:06:31.889 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:31.889 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "queued for pg", 2024-09-21T10:06:31.889 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:06:31.889 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:06:31.889 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:31.889 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 197389 2024-09-21T10:06:31.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.890 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:31.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.890 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:31.890 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.870768+0000", 2024-09-21T10:06:31.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:31.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.890 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:31.890 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.870768+0000", 2024-09-21T10:06:31.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:31.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.890 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:31.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.870768+0000", 2024-09-21T10:06:31.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:31.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:31.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.870771+0000", 2024-09-21T10:06:31.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.841e-06 2024-09-21T10:06:31.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:31.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.870776+0000", 2024-09-21T10:06:31.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6590000000000004e-06 2024-09-21T10:06:31.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:31.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.870780+0000", 2024-09-21T10:06:31.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5389999999999999e-06 2024-09-21T10:06:31.892 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:31.892 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:31.892 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:31.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.892 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:197388 5.d e130/105 5:bce88d5d:::benchmark_data_smithi098_135187_object61750:head v 130'12218, mlcod=130'12216)", 2024-09-21T10:06:31.892 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:06:31.870803+0000", 2024-09-21T10:06:31.892 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0030335259999999999, 2024-09-21T10:06:31.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032246430000000001, 2024-09-21T10:06:31.892 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:06:31.893 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:06:31.893 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:06:31.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:06:31.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.870803+0000", 2024-09-21T10:06:31.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:31.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:06:31.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.870803+0000", 2024-09-21T10:06:31.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:06:31.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:06:31.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.870803+0000", 2024-09-21T10:06:31.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:31.894 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:06:31.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.870808+0000", 2024-09-21T10:06:31.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5160000000000002e-06 2024-09-21T10:06:31.894 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:06:31.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.870818+0000", 2024-09-21T10:06:31.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9939999999999995e-06 2024-09-21T10:06:31.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.895 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:06:31.895 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.870822+0000", 2024-09-21T10:06:31.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.929e-06 2024-09-21T10:06:31.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.895 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:06:31.895 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.870929+0000", 2024-09-21T10:06:31.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107064 2024-09-21T10:06:31.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:06:31.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:06:31.896 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:06:31.896 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:31.870947+0000", 2024-09-21T10:06:31.896 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7978999999999998e-05 2024-09-21T10:06:31.896 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:31.896 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:06:31.896 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:31.896 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:06:31.896 INFO:teuthology.orchestra.run.smithi196.stdout: ], 2024-09-21T10:06:31.896 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 3 2024-09-21T10:06:31.896 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:06:31.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:31.897 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:31.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:31.998 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:32.053 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:32.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:32.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:32.154 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:32.195 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:32.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:32.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:32.296 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:32.341 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:32.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:32.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:32.443 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:32.472 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:32.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:32.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:32.574 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:32.606 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:32.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:32.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:32.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:06:32.689 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:06:32.708 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:32.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:32.794 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:32.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:32.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:32 smithi098 ceph-mon[97580]: pgmap v348: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 818 op/s 2024-09-21T10:06:32.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:32 smithi098 ceph-mon[100914]: pgmap v348: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 818 op/s 2024-09-21T10:06:32.895 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:32.930 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:32.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:32.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:32.982 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:06:32.982 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:06:33.013 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:06:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:06:33.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:33.037 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:06:33.038 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:06:33.064 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:06:33.064 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:06:33.079 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:33.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:33.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:33.100 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:06:33.100 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:06:33.122 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:06:33.122 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:06:33.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:32 smithi196 ceph-mon[84784]: pgmap v348: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 818 op/s 2024-09-21T10:06:33.148 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:06:33.148 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:06:33.163 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:06:33.177 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:06:33.178 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:06:33.181 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:33.222 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:33.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:33.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:33.324 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:33.367 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:33.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:33.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:33.469 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:33.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:33.503 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:33.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:33.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:33.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:33.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:33.742 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:33.771 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:33.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:33.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:33.872 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:33.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:33 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/372268136' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:33.906 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:33.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:33.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:34.008 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:34.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:34.034 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:34.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:34.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:33 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/372268136' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:34.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:33 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/372268136' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:34.134 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:34.175 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:34.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:34.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:34.276 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:34.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:34.320 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:34.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:34.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:34.422 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:34.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:34.488 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:34.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:34.589 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:34.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:34.625 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:34.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:34.726 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:34.749 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:34.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:34.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:34.793 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.7 primary-affinity to 0.8465 (8554762) 2024-09-21T10:06:34.851 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:34 smithi196 ceph-mon[84784]: pgmap v349: 89 pgs: 89 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 809 op/s 2024-09-21T10:06:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:34 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2762974021' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.8464998125120695} : dispatch 2024-09-21T10:06:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:34 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.8464998125120695} : dispatch 2024-09-21T10:06:34.893 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:34.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:34.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:34.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:34 smithi098 ceph-mon[97580]: pgmap v349: 89 pgs: 89 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 809 op/s 2024-09-21T10:06:34.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:34 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2762974021' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.8464998125120695} : dispatch 2024-09-21T10:06:34.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:34 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.8464998125120695} : dispatch 2024-09-21T10:06:34.993 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:34 smithi098 ceph-mon[100914]: pgmap v349: 89 pgs: 89 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 809 op/s 2024-09-21T10:06:34.993 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:34 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2762974021' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.8464998125120695} : dispatch 2024-09-21T10:06:34.993 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:34 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.8464998125120695} : dispatch 2024-09-21T10:06:34.996 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:35.031 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:06:35.031 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:06:35.035 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:35.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:35.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:35.137 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:35.165 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:35.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:35.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:35.267 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:35.301 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:35.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:35.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:35.403 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:35.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:35.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:35.538 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:35.571 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:35.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:35.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:35.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:35.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:35.706 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:35.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:35.723 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:06:35.807 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:35.838 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:35.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:35.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:35.940 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:35.987 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:35.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:35.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:36.089 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:36.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:35 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.8464998125120695}]': finished 2024-09-21T10:06:36.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:35 smithi098 ceph-mon[97580]: osdmap e131: 8 total, 8 up, 8 in 2024-09-21T10:06:36.090 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:35 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.8464998125120695}]': finished 2024-09-21T10:06:36.090 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:35 smithi098 ceph-mon[100914]: osdmap e131: 8 total, 8 up, 8 in 2024-09-21T10:06:36.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:36.137 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:36.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:35 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.8464998125120695}]': finished 2024-09-21T10:06:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:35 smithi196 ceph-mon[84784]: osdmap e131: 8 total, 8 up, 8 in 2024-09-21T10:06:36.139 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:06:36.196 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:06:36.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:36.287 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:36.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:36.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:36.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:36.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:36.439 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:36.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:36.540 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:36.568 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:36.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:36.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:36.670 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:36.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:36.704 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:36.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:36.805 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:36.813 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:36.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:36.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:36.962 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:36.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:36.992 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:36.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:37.093 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:37.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:36 smithi098 ceph-mon[97580]: pgmap v351: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 981 op/s 2024-09-21T10:06:37.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:36 smithi098 ceph-mon[97580]: osdmap e132: 8 total, 8 up, 8 in 2024-09-21T10:06:37.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:36 smithi098 ceph-mon[100914]: pgmap v351: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 981 op/s 2024-09-21T10:06:37.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:36 smithi098 ceph-mon[100914]: osdmap e132: 8 total, 8 up, 8 in 2024-09-21T10:06:37.127 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:37.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:37.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:37.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:36 smithi196 ceph-mon[84784]: pgmap v351: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 981 op/s 2024-09-21T10:06:37.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:36 smithi196 ceph-mon[84784]: osdmap e132: 8 total, 8 up, 8 in 2024-09-21T10:06:37.229 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:37.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:37.263 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:37.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:37.364 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:37.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:37.405 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:37.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:37.506 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:37.558 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:37.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:37.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:37.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:37.697 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:37.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:37.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:37.799 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:37.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:37.830 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:37.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:37.830 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:06:37.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:37 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3863598841' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:06:37.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:06:37.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:37.931 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:37.964 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:37.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:37.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:38.064 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:06:38.064 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:06:38.066 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:38.095 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:38.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:38.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:38.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:37 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3863598841' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:06:38.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:06:38.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:38.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:37 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3863598841' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:06:38.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:06:38.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:38.197 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:38.239 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:38.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:38.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:38.340 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:38.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:38.371 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:38.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:38.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:38.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:38.514 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:38.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:38.615 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:38.642 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:38.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:38.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:38.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:38.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:06:38.770 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:38.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:38.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:38.873 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:38.909 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:38.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:38.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:39.011 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:39.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:39.045 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:39.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:39.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:38 smithi098 ceph-mon[97580]: pgmap v353: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 683 op/s 2024-09-21T10:06:39.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:38 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:39.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:38 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3863598841' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:06:39.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:38 smithi098 ceph-mon[97580]: osdmap e133: 8 total, 8 up, 8 in 2024-09-21T10:06:39.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:38 smithi098 ceph-mon[100914]: pgmap v353: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 683 op/s 2024-09-21T10:06:39.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:38 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:39.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:38 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3863598841' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:06:39.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:38 smithi098 ceph-mon[100914]: osdmap e133: 8 total, 8 up, 8 in 2024-09-21T10:06:39.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:38 smithi196 ceph-mon[84784]: pgmap v353: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 683 op/s 2024-09-21T10:06:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:38 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:38 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3863598841' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:06:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:38 smithi196 ceph-mon[84784]: osdmap e133: 8 total, 8 up, 8 in 2024-09-21T10:06:39.146 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:39.179 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:39.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:39.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:39.280 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:39.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:39.314 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:39.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:39.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:39.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:39.458 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:39.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:39.559 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:39.588 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:39.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:39.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:39.602 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:39.690 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:39.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:39.743 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:39.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:39.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:39.877 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:39.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:39.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:39.979 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:40.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:40.014 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:40.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:40.116 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:40.164 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:40.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:40.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:40.266 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:40.304 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:40.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:40.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:40.406 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:40.436 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:40.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:40.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:40.538 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:40.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:40.569 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:40.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:40.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:06:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:06:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:06:40.670 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:40.700 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:40.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:40.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:40.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:40.843 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:40.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:40.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:40.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:06:40.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:40.983 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:40.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:40.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:41.057 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:40 smithi098 ceph-mon[97580]: pgmap v355: 89 pgs: 89 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 910 op/s 2024-09-21T10:06:41.058 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:06:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:06:40] "GET /metrics HTTP/1.1" 200 36453 "" "Prometheus/2.51.0" 2024-09-21T10:06:41.058 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:40 smithi098 ceph-mon[100914]: pgmap v355: 89 pgs: 89 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 910 op/s 2024-09-21T10:06:41.085 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:41.115 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:41.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:41.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:41.118 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:06:41.118 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:06:41.118 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:06:41.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd out 4 2024-09-21T10:06:41.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:40 smithi196 ceph-mon[84784]: pgmap v355: 89 pgs: 89 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 910 op/s 2024-09-21T10:06:41.218 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:41.243 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:41.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:41.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:41.258 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:06:41.258 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:06:41.275 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:06:41.275 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:06:41.310 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:06:41.310 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:06:41.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:41.349 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:06:41.350 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:06:41.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:41.375 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:41.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:41.388 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:06:41.388 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:06:41.428 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:06:41.428 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:06:41.456 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:06:41.456 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:06:41.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:41.488 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:06:41.488 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:06:41.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:41.507 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:41.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:41.608 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:41.642 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:41.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:41.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:41.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:41.775 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:41.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:41.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:41.877 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:41.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:41.915 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:41.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:42.016 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:42.049 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:42.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:42.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:42.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:41 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1693594937' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:42.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:41 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1693594937' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:42.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:41 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1693594937' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:42.151 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:42.185 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:42.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:42.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:42.286 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:42.328 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:42.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:42.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:42.429 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:42.465 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:42.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:42.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:42.567 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:42.598 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:42.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:42.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:42.700 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:42.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:42.735 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:42.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:42.836 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:42.873 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:42.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:42.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:42.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:43.017 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:43.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:43.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:43.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:42 smithi098 ceph-mon[97580]: pgmap v356: 89 pgs: 89 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 674 op/s 2024-09-21T10:06:43.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:42 smithi098 ceph-mon[100914]: pgmap v356: 89 pgs: 89 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 674 op/s 2024-09-21T10:06:43.119 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:42 smithi196 ceph-mon[84784]: pgmap v356: 89 pgs: 89 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 674 op/s 2024-09-21T10:06:43.150 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:43.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:43.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:43.252 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:43.291 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:43.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:43.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:43.393 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:43.428 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:43.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:43.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:43.530 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:43.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:43.565 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:43.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:43.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:43.700 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:43.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:43.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:43.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:43.835 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:43.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:43.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:43.936 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:43.964 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:43.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:43.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:44.065 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:44.101 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:44.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:44.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:44.169 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:44.203 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:44.237 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:44.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:44.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:44.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:44.388 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:44.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:44.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:44.490 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:44.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:44.532 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:44.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:44.617 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:06:44.633 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:44.660 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:44.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:44.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:44.762 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:44.786 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:44.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:44.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:44.788 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:06:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:06:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:208412 5.3 5:c61f481e:::benchmark_data_smithi098_135187_object72774:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-21T10:06:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:06:44.742571+0000", 2024-09-21T10:06:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.036750205000000001, 2024-09-21T10:06:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036783903999999999, 2024-09-21T10:06:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:06:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:06:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:06:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:06:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:06:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 208412 2024-09-21T10:06:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:06:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:06:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.742571+0000", 2024-09-21T10:06:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:06:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:06:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.742571+0000", 2024-09-21T10:06:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:06:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:06:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.742572+0000", 2024-09-21T10:06:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4529999999999999e-06 2024-09-21T10:06:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:06:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.742577+0000", 2024-09-21T10:06:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7609999999999998e-06 2024-09-21T10:06:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:06:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.742588+0000", 2024-09-21T10:06:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0825999999999999e-05 2024-09-21T10:06:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:06:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.742595+0000", 2024-09-21T10:06:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9090000000000003e-06 2024-09-21T10:06:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:06:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.742774+0000", 2024-09-21T10:06:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017976299999999999 2024-09-21T10:06:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:06:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.742919+0000", 2024-09-21T10:06:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014437799999999999 2024-09-21T10:06:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:06:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.743051+0000", 2024-09-21T10:06:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013237599999999999 2024-09-21T10:06:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.752781+0000", 2024-09-21T10:06:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.009729985 2024-09-21T10:06:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:06:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.753933+0000", 2024-09-21T10:06:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011517680000000001 2024-09-21T10:06:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:06:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:06:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:06:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:208422 5.8 e133/105 5:1b84a33b:::benchmark_data_smithi098_135187_object72784:head v 133'12905, mlcod=133'12903)", 2024-09-21T10:06:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:06:44.758707+0000", 2024-09-21T10:06:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.020613718999999999, 2024-09-21T10:06:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020774277000000001, 2024-09-21T10:06:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:06:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:06:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:06:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:06:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:06:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.758707+0000", 2024-09-21T10:06:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:06:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:06:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.758707+0000", 2024-09-21T10:06:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:06:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:06:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.758709+0000", 2024-09-21T10:06:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1339999999999998e-06 2024-09-21T10:06:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:06:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.758723+0000", 2024-09-21T10:06:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3689999999999999e-05 2024-09-21T10:06:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:06:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.758746+0000", 2024-09-21T10:06:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2962000000000001e-05 2024-09-21T10:06:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:06:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.758757+0000", 2024-09-21T10:06:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0997e-05 2024-09-21T10:06:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:06:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.758820+0000", 2024-09-21T10:06:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2648000000000001e-05 2024-09-21T10:06:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:06:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.758857+0000", 2024-09-21T10:06:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7370999999999998e-05 2024-09-21T10:06:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:06:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:06:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:06:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:208414 5.b e133/105 5:df7395fd:::benchmark_data_smithi098_135187_object72776:head v 133'12917, mlcod=133'12915)", 2024-09-21T10:06:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:06:44.758989+0000", 2024-09-21T10:06:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.020331657, 2024-09-21T10:06:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020542123999999998, 2024-09-21T10:06:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:06:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:06:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:06:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:06:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:06:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.758989+0000", 2024-09-21T10:06:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:06:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:06:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.758989+0000", 2024-09-21T10:06:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:06:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:06:44.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.758990+0000", 2024-09-21T10:06:44.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3799999999999997e-07 2024-09-21T10:06:44.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:06:44.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.758995+0000", 2024-09-21T10:06:44.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0200000000000002e-06 2024-09-21T10:06:44.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:06:44.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.759004+0000", 2024-09-21T10:06:44.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2e-06 2024-09-21T10:06:44.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:06:44.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.759010+0000", 2024-09-21T10:06:44.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8880000000000002e-06 2024-09-21T10:06:44.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:06:44.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.763896+0000", 2024-09-21T10:06:44.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048860320000000002 2024-09-21T10:06:44.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:06:44.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.763921+0000", 2024-09-21T10:06:44.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4598000000000001e-05 2024-09-21T10:06:44.801 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:06:44.801 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:06:44.801 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:06:44.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.801 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:208426 5.8 e133/105 5:1d531b6b:::benchmark_data_smithi098_135187_object72788:head v 133'12906, mlcod=133'12903)", 2024-09-21T10:06:44.801 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:06:44.767030+0000", 2024-09-21T10:06:44.801 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.012291318000000001, 2024-09-21T10:06:44.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01254835, 2024-09-21T10:06:44.801 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:06:44.801 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:06:44.801 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:06:44.802 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:06:44.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:06:44.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767030+0000", 2024-09-21T10:06:44.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:06:44.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:06:44.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767030+0000", 2024-09-21T10:06:44.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:06:44.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:06:44.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767032+0000", 2024-09-21T10:06:44.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3020000000000002e-06 2024-09-21T10:06:44.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:06:44.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767043+0000", 2024-09-21T10:06:44.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0831e-05 2024-09-21T10:06:44.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:06:44.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767059+0000", 2024-09-21T10:06:44.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6470999999999999e-05 2024-09-21T10:06:44.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:06:44.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767069+0000", 2024-09-21T10:06:44.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9119999999999993e-06 2024-09-21T10:06:44.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:06:44.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767475+0000", 2024-09-21T10:06:44.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00040566400000000002 2024-09-21T10:06:44.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:06:44.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767497+0000", 2024-09-21T10:06:44.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1974e-05 2024-09-21T10:06:44.805 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:06:44.805 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:06:44.805 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:06:44.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.805 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:208427 5.b e133/105)", 2024-09-21T10:06:44.805 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:06:44.767090+0000", 2024-09-21T10:06:44.805 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.012230829, 2024-09-21T10:06:44.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012531426999999999, 2024-09-21T10:06:44.805 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:06:44.805 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:06:44.806 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-21T10:06:44.806 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:06:44.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:06:44.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767090+0000", 2024-09-21T10:06:44.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:06:44.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:06:44.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767090+0000", 2024-09-21T10:06:44.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:06:44.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:06:44.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767090+0000", 2024-09-21T10:06:44.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4299999999999999e-07 2024-09-21T10:06:44.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:06:44.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767093+0000", 2024-09-21T10:06:44.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0489999999999999e-06 2024-09-21T10:06:44.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:06:44.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767099+0000", 2024-09-21T10:06:44.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5400000000000003e-06 2024-09-21T10:06:44.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:06:44.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767103+0000", 2024-09-21T10:06:44.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3309999999999997e-06 2024-09-21T10:06:44.808 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:06:44.808 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:06:44.808 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:06:44.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.808 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:208423 5.3 5.7c542123 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-21T10:06:44.808 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:06:44.767496+0000", 2024-09-21T10:06:44.808 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.011824771, 2024-09-21T10:06:44.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012163208999999999, 2024-09-21T10:06:44.808 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:06:44.809 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:06:44.809 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-21T10:06:44.809 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:06:44.809 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:06:44.809 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:06:44.809 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 208423 2024-09-21T10:06:44.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.809 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:06:44.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:06:44.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767496+0000", 2024-09-21T10:06:44.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:06:44.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:06:44.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767496+0000", 2024-09-21T10:06:44.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:06:44.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:06:44.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767498+0000", 2024-09-21T10:06:44.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.784e-06 2024-09-21T10:06:44.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:06:44.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767504+0000", 2024-09-21T10:06:44.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8340000000000004e-06 2024-09-21T10:06:44.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:06:44.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767516+0000", 2024-09-21T10:06:44.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2711000000000001e-05 2024-09-21T10:06:44.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:06:44.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767524+0000", 2024-09-21T10:06:44.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.374e-06 2024-09-21T10:06:44.811 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:06:44.811 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:06:44.811 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:06:44.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.812 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.44894.0:208412 5.3 e133/105)", 2024-09-21T10:06:44.812 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:06:44.767589+0000", 2024-09-21T10:06:44.812 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.01173185, 2024-09-21T10:06:44.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012113011, 2024-09-21T10:06:44.812 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:06:44.812 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:06:44.812 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-21T10:06:44.812 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:06:44.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:06:44.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767589+0000", 2024-09-21T10:06:44.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:06:44.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:06:44.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767589+0000", 2024-09-21T10:06:44.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:06:44.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:06:44.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767589+0000", 2024-09-21T10:06:44.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-21T10:06:44.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:06:44.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767592+0000", 2024-09-21T10:06:44.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9100000000000001e-06 2024-09-21T10:06:44.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:06:44.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767599+0000", 2024-09-21T10:06:44.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8179999999999998e-06 2024-09-21T10:06:44.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:06:44.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.767603+0000", 2024-09-21T10:06:44.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.245e-06 2024-09-21T10:06:44.814 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:06:44.814 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:06:44.814 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:06:44.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.815 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:208433 5.e e133/105 5:7b4fe06a:::benchmark_data_smithi098_135187_object72795:head v 133'13291, mlcod=133'13289)", 2024-09-21T10:06:44.815 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:06:44.778952+0000", 2024-09-21T10:06:44.815 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.000368685, 2024-09-21T10:06:44.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00078658899999999997, 2024-09-21T10:06:44.815 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:06:44.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:06:44.815 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:06:44.815 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:06:44.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:06:44.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.778952+0000", 2024-09-21T10:06:44.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:06:44.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:06:44.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.778952+0000", 2024-09-21T10:06:44.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:06:44.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:06:44.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.778953+0000", 2024-09-21T10:06:44.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1400000000000001e-06 2024-09-21T10:06:44.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:06:44.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.778963+0000", 2024-09-21T10:06:44.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3859999999999996e-06 2024-09-21T10:06:44.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:06:44.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.778979+0000", 2024-09-21T10:06:44.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6633000000000002e-05 2024-09-21T10:06:44.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:06:44.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.778987+0000", 2024-09-21T10:06:44.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9189999999999999e-06 2024-09-21T10:06:44.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:06:44.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.779054+0000", 2024-09-21T10:06:44.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7107000000000006e-05 2024-09-21T10:06:44.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:06:44.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:06:44.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:06:44.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:44.779074+0000", 2024-09-21T10:06:44.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9221000000000001e-05 2024-09-21T10:06:44.818 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:06:44.818 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:06:44.818 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:06:44.818 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:06:44.818 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-21T10:06:44.818 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 8 2024-09-21T10:06:44.818 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:06:44.869 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:44 smithi098 ceph-mon[97580]: pgmap v357: 89 pgs: 89 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 667 op/s 2024-09-21T10:06:44.870 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:44 smithi098 ceph-mon[100914]: pgmap v357: 89 pgs: 89 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 667 op/s 2024-09-21T10:06:44.887 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:44.926 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:44.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:44.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:45.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:45.065 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:45.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:45.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:45.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:44 smithi196 ceph-mon[84784]: pgmap v357: 89 pgs: 89 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 667 op/s 2024-09-21T10:06:45.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:45.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:45.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:45.297 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:45.324 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:45.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:45.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:45.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:45.473 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:45.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:45.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:45.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:45.613 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:45.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:45.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:45.714 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:45.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:06:45.751 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:45.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:45.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:45.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:45.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:45.917 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:45.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:46.017 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:46.048 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:46.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:46.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:46.150 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:46.187 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:46.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:46.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:46.289 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:46.319 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:46.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:46.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:46.421 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:46.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:46.466 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:46.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:46.567 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:46.606 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:46.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:46.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:46.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:46.747 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:46.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:46.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:46.849 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:46.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:46.881 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:46.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:46.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:47.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:47.019 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:47.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:47.049 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:47.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:46 smithi098 ceph-mon[100914]: pgmap v358: 89 pgs: 89 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 930 op/s 2024-09-21T10:06:47.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:46 smithi098 ceph-mon[97580]: pgmap v358: 89 pgs: 89 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 930 op/s 2024-09-21T10:06:47.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:47.137 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:47.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:46 smithi196 ceph-mon[84784]: pgmap v358: 89 pgs: 89 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 930 op/s 2024-09-21T10:06:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:47.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:47.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:47.288 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:47.328 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:47.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:47.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:47.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:47.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:47.461 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:47.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:47.562 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:47.596 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:47.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:47.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:47.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:47.727 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:47.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:47.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:47.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:47.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:47.863 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:47.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:47.964 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:48.001 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:48.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:48.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:48.103 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:48.107 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:48.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:48.135 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:48.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:48.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:48.273 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:48.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:48.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:48.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:48.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:48.427 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:48.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:48.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:48.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:48.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:48.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:48.698 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:48.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:48.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:48.800 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:48.832 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:48.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:48.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:48.934 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:48.966 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:48.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:48.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:49.068 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:49.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:48 smithi098 ceph-mon[97580]: pgmap v359: 89 pgs: 89 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 864 op/s 2024-09-21T10:06:49.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:48 smithi098 ceph-mon[100914]: pgmap v359: 89 pgs: 89 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 864 op/s 2024-09-21T10:06:49.102 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:49.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:49.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:49.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:48 smithi196 ceph-mon[84784]: pgmap v359: 89 pgs: 89 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 864 op/s 2024-09-21T10:06:49.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:49.234 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:49.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:49.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:49.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:49.367 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:49.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:49.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:49.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:49.505 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:49.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:49.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:49.607 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:49.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:49.654 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:49.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:49.755 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:49.793 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:49.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:49.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:49.865 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:06:49.865 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:49 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2281042699' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:06:49.866 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:49 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:06:49.866 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:49 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2281042699' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:06:49.866 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:49 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:06:49.895 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:49.935 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:49.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:49.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:50.037 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:50.070 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:50.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:50.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:49 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2281042699' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:06:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:49 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:06:50.172 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:50.206 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:50.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:50.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:50.309 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:50.347 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:50.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:50.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:50.450 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:50.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:50.490 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:50.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:50.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:50.631 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:50.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:50.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:50.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:06:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:06:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:06:50.733 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:50.769 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:50.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:50.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:50.857 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:06:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:06:50] "GET /metrics HTTP/1.1" 200 36453 "" "Prometheus/2.51.0" 2024-09-21T10:06:50.871 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:50.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:50.908 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:50.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:51.009 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:51.046 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:51.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:51.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:51.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:50 smithi196 ceph-mon[84784]: pgmap v360: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 962 op/s 2024-09-21T10:06:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:50 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:50 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:06:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:50 smithi196 ceph-mon[84784]: osdmap e134: 8 total, 8 up, 8 in 2024-09-21T10:06:51.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:51.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:51.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:51.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:50 smithi098 ceph-mon[97580]: pgmap v360: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 962 op/s 2024-09-21T10:06:51.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:50 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:51.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:50 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:06:51.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:50 smithi098 ceph-mon[97580]: osdmap e134: 8 total, 8 up, 8 in 2024-09-21T10:06:51.227 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:50 smithi098 ceph-mon[100914]: pgmap v360: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 962 op/s 2024-09-21T10:06:51.228 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:50 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:51.228 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:50 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:06:51.228 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:50 smithi098 ceph-mon[100914]: osdmap e134: 8 total, 8 up, 8 in 2024-09-21T10:06:51.286 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:51.316 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:51.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:51.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:51.418 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:51.451 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:51.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:51.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:51.553 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:51.598 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:51.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:51.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:51.699 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:51.740 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:51.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:51.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:51.842 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:51.882 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:51.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:51.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:51.983 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:52.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:52.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:52.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:52.156 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:52.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:52.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:52.171 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:06:52.171 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:06:52.172 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-21T10:06:52.172 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-21T10:06:52.172 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:06:52.258 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:52.299 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:52.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:52.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:52.401 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:52.433 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:52.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:52.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:52.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:52.575 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:52.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:52.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:52.677 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:52.725 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:52.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:52.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:52.827 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:52.859 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:52.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:52.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:52.904 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:06:52.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:52.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:52 smithi098 ceph-mon[97580]: pgmap v362: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 864 op/s 2024-09-21T10:06:52.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:52.976 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:52 smithi098 ceph-mon[100914]: pgmap v362: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 864 op/s 2024-09-21T10:06:52.977 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:52.991 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:52.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:52.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:53.093 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:53.137 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:53.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:53.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:52 smithi196 ceph-mon[84784]: pgmap v362: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 864 op/s 2024-09-21T10:06:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:53.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:53.267 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:06:53.275 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:53.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:53.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:53.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:53.411 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:53.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:53.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:53.513 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:53.551 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:53.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:53.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:53.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:53.710 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:53.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:53.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:53.812 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:53.849 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:53.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:53.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:53.871 INFO:teuthology.orchestra.run.smithi098.stderr:marked out osd.4. 2024-09-21T10:06:53.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:53 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a[97576]: 2024-09-21T10:06:53.803+0000 7f65561f9640 -1 mon.a@0(leader).osd e134 definitely_dead 0 2024-09-21T10:06:53.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:53 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/322605195' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:06:53.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:53 smithi098 ceph-mon[97580]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T10:06:53.935 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:53 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/322605195' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:06:53.935 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:53 smithi098 ceph-mon[100914]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T10:06:53.951 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:53.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:53.972 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:53.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:54.073 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:54.094 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:54.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:54.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:54.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:53 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/322605195' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:06:54.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:53 smithi196 ceph-mon[84784]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T10:06:54.196 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:54.217 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:54.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:54.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:54.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:54.339 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:54.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:54.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:54.440 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:54.461 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:54.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:54.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:54.563 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:54.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:54.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:54.689 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:54.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:54.712 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:54.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:54.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:54.833 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:54.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:54.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:54.935 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:54.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:54.957 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:54.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:55.058 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:55.080 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:55.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:55.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:55.123 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:54 smithi098 ceph-mon[97580]: pgmap v363: 89 pgs: 89 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 869 op/s 2024-09-21T10:06:55.123 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:54 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/322605195' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T10:06:55.123 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:54 smithi098 ceph-mon[97580]: osdmap e135: 8 total, 8 up, 7 in 2024-09-21T10:06:55.123 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:54 smithi098 ceph-mon[100914]: pgmap v363: 89 pgs: 89 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 869 op/s 2024-09-21T10:06:55.123 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:54 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/322605195' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T10:06:55.123 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:54 smithi098 ceph-mon[100914]: osdmap e135: 8 total, 8 up, 7 in 2024-09-21T10:06:55.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:54 smithi196 ceph-mon[84784]: pgmap v363: 89 pgs: 89 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 869 op/s 2024-09-21T10:06:55.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:54 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/322605195' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T10:06:55.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:54 smithi196 ceph-mon[84784]: osdmap e135: 8 total, 8 up, 7 in 2024-09-21T10:06:55.182 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:55.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:55.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:55.306 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:55.328 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:55.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:55.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:55.429 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:55.453 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:55.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:55.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:55.554 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:55.577 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:55.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:55.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:55.679 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:55.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:55.701 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:55.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:55.741 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:06:55.741 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:06:55.765 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:06:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:06:55.767 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:55.783 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:06:55.784 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:06:55.800 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:06:55.801 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:06:55.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:55.816 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:06:55.816 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:06:55.827 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:55.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:55.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:55.831 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:06:55.831 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:06:55.847 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:06:55.847 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:06:55.862 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:06:55.863 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:06:55.892 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:55 smithi098 ceph-mon[97580]: osdmap e136: 8 total, 8 up, 7 in 2024-09-21T10:06:55.892 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:55 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1250001450' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:55.892 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:55 smithi098 ceph-mon[97580]: osdmap e137: 8 total, 8 up, 7 in 2024-09-21T10:06:55.929 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:55.951 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:55.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:55.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:56.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:56.089 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:56.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:56.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:56.155 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:55 smithi098 ceph-mon[100914]: osdmap e136: 8 total, 8 up, 7 in 2024-09-21T10:06:56.155 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:55 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1250001450' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:56.155 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:55 smithi098 ceph-mon[100914]: osdmap e137: 8 total, 8 up, 7 in 2024-09-21T10:06:56.190 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:56.224 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:56.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:56.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:56.326 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:56.363 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:56.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:56.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:56.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:55 smithi196 ceph-mon[84784]: osdmap e136: 8 total, 8 up, 7 in 2024-09-21T10:06:56.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:55 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1250001450' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:55 smithi196 ceph-mon[84784]: osdmap e137: 8 total, 8 up, 7 in 2024-09-21T10:06:56.465 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:56.505 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:56.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:56.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:56.607 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:56.646 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:56.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:56.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:56.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:56.787 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:56.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:56.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:56.889 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:56.927 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:56.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:56.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:57.012 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:56 smithi098 ceph-mon[97580]: pgmap v366: 89 pgs: 4 peering, 85 active+clean; 3.4 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 938 op/s 2024-09-21T10:06:57.012 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:56 smithi098 ceph-mon[97580]: 5.9 starting backfill to osd.1 from (0'0,0'0] MAX to 134'13460 2024-09-21T10:06:57.012 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:56 smithi098 ceph-mon[97580]: 5.1 starting backfill to osd.0 from (0'0,0'0] MAX to 134'13420 2024-09-21T10:06:57.012 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:56 smithi098 ceph-mon[97580]: 5.0 starting backfill to osd.0 from (0'0,0'0] MAX to 134'13342 2024-09-21T10:06:57.012 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:56 smithi098 ceph-mon[97580]: 5.e starting backfill to osd.2 from (0'0,0'0] MAX to 134'13803 2024-09-21T10:06:57.013 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:56 smithi098 ceph-mon[97580]: 5.f starting backfill to osd.2 from (0'0,0'0] MAX to 134'13571 2024-09-21T10:06:57.013 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:56 smithi098 ceph-mon[97580]: 5.c starting backfill to osd.6 from (0'0,0'0] MAX to 134'13260 2024-09-21T10:06:57.013 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:56 smithi098 ceph-mon[97580]: 5.a starting backfill to osd.0 from (0'0,0'0] MAX to 134'13427 2024-09-21T10:06:57.013 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:56 smithi098 ceph-mon[97580]: 5.5 starting backfill to osd.6 from (0'0,0'0] MAX to 134'13849 2024-09-21T10:06:57.013 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:56 smithi098 ceph-mon[97580]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T10:06:57.013 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:56 smithi098 ceph-mon[100914]: pgmap v366: 89 pgs: 4 peering, 85 active+clean; 3.4 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 938 op/s 2024-09-21T10:06:57.013 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:56 smithi098 ceph-mon[100914]: 5.9 starting backfill to osd.1 from (0'0,0'0] MAX to 134'13460 2024-09-21T10:06:57.013 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:56 smithi098 ceph-mon[100914]: 5.1 starting backfill to osd.0 from (0'0,0'0] MAX to 134'13420 2024-09-21T10:06:57.013 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:56 smithi098 ceph-mon[100914]: 5.0 starting backfill to osd.0 from (0'0,0'0] MAX to 134'13342 2024-09-21T10:06:57.013 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:56 smithi098 ceph-mon[100914]: 5.e starting backfill to osd.2 from (0'0,0'0] MAX to 134'13803 2024-09-21T10:06:57.013 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:56 smithi098 ceph-mon[100914]: 5.f starting backfill to osd.2 from (0'0,0'0] MAX to 134'13571 2024-09-21T10:06:57.013 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:56 smithi098 ceph-mon[100914]: 5.c starting backfill to osd.6 from (0'0,0'0] MAX to 134'13260 2024-09-21T10:06:57.014 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:56 smithi098 ceph-mon[100914]: 5.a starting backfill to osd.0 from (0'0,0'0] MAX to 134'13427 2024-09-21T10:06:57.014 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:56 smithi098 ceph-mon[100914]: 5.5 starting backfill to osd.6 from (0'0,0'0] MAX to 134'13849 2024-09-21T10:06:57.014 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:56 smithi098 ceph-mon[100914]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T10:06:57.029 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:57.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:57.067 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:57.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:57.168 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:57.218 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:57.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:57.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:57.320 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:57.366 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:57.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:57.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:57.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:56 smithi196 ceph-mon[84784]: pgmap v366: 89 pgs: 4 peering, 85 active+clean; 3.4 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 938 op/s 2024-09-21T10:06:57.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:56 smithi196 ceph-mon[84784]: 5.9 starting backfill to osd.1 from (0'0,0'0] MAX to 134'13460 2024-09-21T10:06:57.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:56 smithi196 ceph-mon[84784]: 5.1 starting backfill to osd.0 from (0'0,0'0] MAX to 134'13420 2024-09-21T10:06:57.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:56 smithi196 ceph-mon[84784]: 5.0 starting backfill to osd.0 from (0'0,0'0] MAX to 134'13342 2024-09-21T10:06:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:56 smithi196 ceph-mon[84784]: 5.e starting backfill to osd.2 from (0'0,0'0] MAX to 134'13803 2024-09-21T10:06:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:56 smithi196 ceph-mon[84784]: 5.f starting backfill to osd.2 from (0'0,0'0] MAX to 134'13571 2024-09-21T10:06:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:56 smithi196 ceph-mon[84784]: 5.c starting backfill to osd.6 from (0'0,0'0] MAX to 134'13260 2024-09-21T10:06:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:56 smithi196 ceph-mon[84784]: 5.a starting backfill to osd.0 from (0'0,0'0] MAX to 134'13427 2024-09-21T10:06:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:56 smithi196 ceph-mon[84784]: 5.5 starting backfill to osd.6 from (0'0,0'0] MAX to 134'13849 2024-09-21T10:06:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:56 smithi196 ceph-mon[84784]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T10:06:57.467 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:57.497 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:57.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:57.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:57.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:57.638 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:57.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:57.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:57.740 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:57.770 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:57.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:57.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:57.871 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:57.907 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:57.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:57.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:58.008 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:58.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:58.146 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:58.208 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:58.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:58.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:58.275 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:57 smithi098 ceph-mon[97580]: osdmap e138: 8 total, 8 up, 7 in 2024-09-21T10:06:58.275 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:57 smithi098 ceph-mon[100914]: osdmap e138: 8 total, 8 up, 7 in 2024-09-21T10:06:58.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:58.319 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:06:58.319 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:06:58.339 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:58.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:58.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:58.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:57 smithi196 ceph-mon[84784]: osdmap e138: 8 total, 8 up, 7 in 2024-09-21T10:06:58.441 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:58.472 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:58.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:58.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:58.574 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:58.616 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:58.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:58.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:58.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:58.746 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:06:58.755 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:58.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:58.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:58.858 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:58.885 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:58.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:58.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:58.987 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:59.033 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:59.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:59.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:59.060 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:58 smithi098 ceph-mon[100914]: pgmap v369: 89 pgs: 2 active+remapped+backfill_wait, 4 peering, 83 active+clean; 3.4 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 840 op/s; 6554/166005 objects misplaced (3.948%); 0 B/s, 0 objects/s recovering 2024-09-21T10:06:59.060 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:06:58 smithi098 ceph-mon[100914]: osdmap e139: 8 total, 8 up, 7 in 2024-09-21T10:06:59.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:58 smithi098 ceph-mon[97580]: pgmap v369: 89 pgs: 2 active+remapped+backfill_wait, 4 peering, 83 active+clean; 3.4 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 840 op/s; 6554/166005 objects misplaced (3.948%); 0 B/s, 0 objects/s recovering 2024-09-21T10:06:59.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:06:58 smithi098 ceph-mon[97580]: osdmap e139: 8 total, 8 up, 7 in 2024-09-21T10:06:59.135 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:59.166 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:59.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:59.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:59.268 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:59.306 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:59.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:59.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:59.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:58 smithi196 ceph-mon[84784]: pgmap v369: 89 pgs: 2 active+remapped+backfill_wait, 4 peering, 83 active+clean; 3.4 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 840 op/s; 6554/166005 objects misplaced (3.948%); 0 B/s, 0 objects/s recovering 2024-09-21T10:06:59.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:06:58 smithi196 ceph-mon[84784]: osdmap e139: 8 total, 8 up, 7 in 2024-09-21T10:06:59.408 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:59.451 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:59.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:59.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:59.552 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:59.579 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:06:59.579 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:06:59.579 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-21T10:06:59.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd in 4 2024-09-21T10:06:59.581 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:59.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:59.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:59.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:59.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:06:59.756 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:06:59.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:59.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:59.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:59.888 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:06:59.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:59.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:59.990 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:00.019 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:00.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:00.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:00.121 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:00.147 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:00.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:00.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:00.248 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:00.291 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:00.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:00.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:00.392 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:00.432 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:00.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:00.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:00.534 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:00.562 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:00.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:00.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:00.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:07:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:07:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:07:00.664 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:00.692 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:00.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:00.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:00.703 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:00.704 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:00.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:00.835 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:00.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:00.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:00.895 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:07:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:07:00] "GET /metrics HTTP/1.1" 200 36455 "" "Prometheus/2.51.0" 2024-09-21T10:07:00.937 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:00.963 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:00.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:00.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:01.065 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:01.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:01.111 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:01.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:01.156 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:00 smithi098 ceph-mon[97580]: pgmap v371: 89 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_wait, 4 peering, 78 active+clean; 3.3 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 337 op/s; 22829/162339 objects misplaced (14.063%); 7.9 MiB/s, 2 keys/s, 134 objects/s recovering 2024-09-21T10:07:01.157 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:00 smithi098 ceph-mon[100914]: pgmap v371: 89 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_wait, 4 peering, 78 active+clean; 3.3 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 337 op/s; 22829/162339 objects misplaced (14.063%); 7.9 MiB/s, 2 keys/s, 134 objects/s recovering 2024-09-21T10:07:01.211 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:01.249 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:01.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:01.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:01.351 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:01.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:00 smithi196 ceph-mon[84784]: pgmap v371: 89 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_wait, 4 peering, 78 active+clean; 3.3 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 337 op/s; 22829/162339 objects misplaced (14.063%); 7.9 MiB/s, 2 keys/s, 134 objects/s recovering 2024-09-21T10:07:01.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:01.390 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:01.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:01.491 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:01.520 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:01.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:01.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:01.621 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:01.650 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:01.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:01.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:01.714 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:01.751 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:01.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:01.782 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:01.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:01.883 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:01.920 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:01.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:01.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:01.948 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:07:02.022 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:02.072 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:02.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:02.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:02.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:01 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/453295497' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:02.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:01 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:02.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:01 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/453295497' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:02.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:01 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:02.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:02.201 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:02.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:02.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:02.302 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:02.337 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:02.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:02.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:02.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:01 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/453295497' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:01 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:02.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:02.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:02.480 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:02.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:02.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:02.622 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:02.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:02.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:02.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:02.763 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:02.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:02.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:02.864 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:02.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:02.905 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:02.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:03.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:03.034 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:03.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:03.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:03.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:02 smithi098 ceph-mon[97580]: pgmap v372: 89 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_wait, 81 active+clean; 3.3 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 373 op/s; 25814/159879 objects misplaced (16.146%); 8.7 MiB/s, 1 keys/s, 143 objects/s recovering 2024-09-21T10:07:03.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:02 smithi098 ceph-mon[97580]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-21T10:07:03.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:02 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:07:03.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:02 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:07:03.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:02 smithi098 ceph-mon[97580]: osdmap e140: 8 total, 8 up, 7 in 2024-09-21T10:07:03.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:02 smithi098 ceph-mon[100914]: pgmap v372: 89 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_wait, 81 active+clean; 3.3 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 373 op/s; 25814/159879 objects misplaced (16.146%); 8.7 MiB/s, 1 keys/s, 143 objects/s recovering 2024-09-21T10:07:03.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:02 smithi098 ceph-mon[100914]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-21T10:07:03.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:02 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:07:03.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:02 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:07:03.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:02 smithi098 ceph-mon[100914]: osdmap e140: 8 total, 8 up, 7 in 2024-09-21T10:07:03.136 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:03.167 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:03.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:03.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:03.268 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:03.310 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:03.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:03.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:03.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:02 smithi196 ceph-mon[84784]: pgmap v372: 89 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_wait, 81 active+clean; 3.3 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 373 op/s; 25814/159879 objects misplaced (16.146%); 8.7 MiB/s, 1 keys/s, 143 objects/s recovering 2024-09-21T10:07:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:02 smithi196 ceph-mon[84784]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-21T10:07:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:02 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:07:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:02 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:07:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:02 smithi196 ceph-mon[84784]: osdmap e140: 8 total, 8 up, 7 in 2024-09-21T10:07:03.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:03.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:03.444 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:03.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:03.545 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:03.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:03.585 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:03.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:03.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:03.728 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:03.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:03.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:03.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:03.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:03.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:03.990 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:04.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:04.028 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:04.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:04.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:04.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:04.170 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:04.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:04.271 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:04.303 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:04.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:04.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:04.404 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:04.446 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:04.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:04.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:04.548 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:04.580 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:04.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:04.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:04.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:04.732 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:04.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:04.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:04.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:04.873 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:04.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:04.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:04.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:04 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a[97576]: 2024-09-21T10:07:04.807+0000 7f65561f9640 -1 mon.a@0(leader).osd e140 definitely_dead 0 2024-09-21T10:07:04.945 INFO:teuthology.orchestra.run.smithi098.stderr:marked in osd.4. 2024-09-21T10:07:04.975 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:04.998 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:04.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:04.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:05.100 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:05.121 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:05.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:05.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:05.193 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:04 smithi098 ceph-mon[97580]: pgmap v374: 89 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_wait, 81 active+clean; 3.2 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 542 op/s; 24448/155931 objects misplaced (15.679%); 14 MiB/s, 1 keys/s, 225 objects/s recovering 2024-09-21T10:07:05.193 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:04 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1891176711' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:07:05.193 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:04 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:07:05.194 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:04 smithi098 ceph-mon[100914]: pgmap v374: 89 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_wait, 81 active+clean; 3.2 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 542 op/s; 24448/155931 objects misplaced (15.679%); 14 MiB/s, 1 keys/s, 225 objects/s recovering 2024-09-21T10:07:05.194 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:04 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1891176711' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:07:05.194 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:04 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:07:05.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:05.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:05.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:05.345 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:05.367 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:05.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:05.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:05.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:04 smithi196 ceph-mon[84784]: pgmap v374: 89 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_wait, 81 active+clean; 3.2 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 542 op/s; 24448/155931 objects misplaced (15.679%); 14 MiB/s, 1 keys/s, 225 objects/s recovering 2024-09-21T10:07:05.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:04 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1891176711' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:07:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:04 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:07:05.468 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:05.490 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:05.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:05.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:05.592 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:05.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:05.614 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:05.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:05.714 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:05.736 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:05.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:05.838 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:05.859 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:05.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:05.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:05.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:05.982 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:05.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:05.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:06.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:06.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:06.107 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:06.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:06.208 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:06.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:06.232 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:06.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:06.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:05 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T10:07:06.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:05 smithi196 ceph-mon[84784]: osdmap e141: 8 total, 8 up, 8 in 2024-09-21T10:07:06.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:05 smithi196 ceph-mon[84784]: osdmap e142: 8 total, 8 up, 8 in 2024-09-21T10:07:06.333 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:06.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:05 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T10:07:06.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:05 smithi098 ceph-mon[97580]: osdmap e141: 8 total, 8 up, 8 in 2024-09-21T10:07:06.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:05 smithi098 ceph-mon[97580]: osdmap e142: 8 total, 8 up, 8 in 2024-09-21T10:07:06.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:05 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T10:07:06.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:05 smithi098 ceph-mon[100914]: osdmap e141: 8 total, 8 up, 8 in 2024-09-21T10:07:06.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:05 smithi098 ceph-mon[100914]: osdmap e142: 8 total, 8 up, 8 in 2024-09-21T10:07:06.354 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:06.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:06.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:06.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:06.479 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:06.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:06.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:06.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:06.611 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:06.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:06.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:06.617 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-21T10:07:06.713 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:06.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:06.747 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:06.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:06.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:06.878 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:06.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:06.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:06.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:07.013 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:07.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:07.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:07.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:07.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:07.157 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:07.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:07.214 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:06 smithi098 ceph-mon[97580]: pgmap v376: 89 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_wait, 81 active+clean; 3.0 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 749 op/s; 22215/148722 objects misplaced (14.937%); 23 MiB/s, 1 keys/s, 372 objects/s recovering 2024-09-21T10:07:07.214 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:06 smithi098 ceph-mon[97580]: osdmap e143: 8 total, 8 up, 8 in 2024-09-21T10:07:07.214 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:06 smithi098 ceph-mon[100914]: pgmap v376: 89 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_wait, 81 active+clean; 3.0 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 749 op/s; 22215/148722 objects misplaced (14.937%); 23 MiB/s, 1 keys/s, 372 objects/s recovering 2024-09-21T10:07:07.214 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:06 smithi098 ceph-mon[100914]: osdmap e143: 8 total, 8 up, 8 in 2024-09-21T10:07:07.259 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:07.289 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:07.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:07.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:07.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:06 smithi196 ceph-mon[84784]: pgmap v376: 89 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_wait, 81 active+clean; 3.0 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 749 op/s; 22215/148722 objects misplaced (14.937%); 23 MiB/s, 1 keys/s, 372 objects/s recovering 2024-09-21T10:07:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:06 smithi196 ceph-mon[84784]: osdmap e143: 8 total, 8 up, 8 in 2024-09-21T10:07:07.391 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:07.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:07.422 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:07.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:07.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:07.569 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:07.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:07.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:07.671 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:07.704 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:07.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:07.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:07.806 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:07.851 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:07.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:07.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:07.952 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:07.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:07.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:08.095 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:08.130 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:08.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:08.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:08.232 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:08.266 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:08.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:08.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:08.367 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:08.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:08 smithi196 ceph-mon[84784]: pgmap v379: 89 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_wait, 82 active+clean; 3.0 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 670 op/s; 19161/148239 objects misplaced (12.926%); 23 MiB/s, 374 objects/s recovering 2024-09-21T10:07:08.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:08.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:08 smithi196 ceph-mon[84784]: osdmap e144: 8 total, 8 up, 8 in 2024-09-21T10:07:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:08 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4199245596' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:08.405 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:08.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:08.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:08.507 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:08.550 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:08.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:08.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:08 smithi098 ceph-mon[97580]: pgmap v379: 89 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_wait, 82 active+clean; 3.0 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 670 op/s; 19161/148239 objects misplaced (12.926%); 23 MiB/s, 374 objects/s recovering 2024-09-21T10:07:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:08 smithi098 ceph-mon[97580]: osdmap e144: 8 total, 8 up, 8 in 2024-09-21T10:07:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:08 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4199245596' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:08.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:08 smithi098 ceph-mon[100914]: pgmap v379: 89 pgs: 4 active+remapped+backfilling, 3 active+remapped+backfill_wait, 82 active+clean; 3.0 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 670 op/s; 19161/148239 objects misplaced (12.926%); 23 MiB/s, 374 objects/s recovering 2024-09-21T10:07:08.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:08.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:08.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:08 smithi098 ceph-mon[100914]: osdmap e144: 8 total, 8 up, 8 in 2024-09-21T10:07:08.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:08 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4199245596' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:08.651 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:08.683 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:08.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:08.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:08.785 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:08.820 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:08.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:08.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:08.861 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:07:08.861 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:07:08.861 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:07:08.861 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:07:08.861 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:07:08.861 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:07:08.862 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:07:08.862 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:07:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:07:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:07:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:07:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:07:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:07:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:07:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:07:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:07:08.922 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:08.943 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:08.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:08.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:09.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:07:09.045 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:09.079 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:09.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:09.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:09.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:09.213 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:09.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:09.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:09.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:09.340 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:09.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:09.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:09.442 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:09.472 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:09.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:09.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:09.574 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:09.613 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:09.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:09.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:09.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:09.742 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:09.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:09.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:09.804 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:09 smithi098 ceph-mon[97580]: osdmap e145: 8 total, 8 up, 8 in 2024-09-21T10:07:09.806 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:09 smithi098 ceph-mon[100914]: osdmap e145: 8 total, 8 up, 8 in 2024-09-21T10:07:09.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:09.875 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:09.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:09.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:09.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:09 smithi196 ceph-mon[84784]: osdmap e145: 8 total, 8 up, 8 in 2024-09-21T10:07:09.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:10.009 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:10.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:10.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:10.110 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:10.147 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:10.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:10.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:10.248 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:10.282 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:10.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:10.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:10.384 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:10.416 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:10.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:10.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:10.518 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:10.536 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:07:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:07:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:07:10.555 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:10.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:10.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:10.657 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:10.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:10.696 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:10.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:10.788 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:10 smithi098 ceph-mon[97580]: pgmap v382: 89 pgs: 1 peering, 1 active+remapped+backfilling, 87 active+clean; 2.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 483 op/s; 2385/142920 objects misplaced (1.669%); 11 MiB/s, 2 keys/s, 175 objects/s recovering 2024-09-21T10:07:10.788 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:07:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:07:10] "GET /metrics HTTP/1.1" 200 36472 "" "Prometheus/2.51.0" 2024-09-21T10:07:10.789 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:10 smithi098 ceph-mon[100914]: pgmap v382: 89 pgs: 1 peering, 1 active+remapped+backfilling, 87 active+clean; 2.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 483 op/s; 2385/142920 objects misplaced (1.669%); 11 MiB/s, 2 keys/s, 175 objects/s recovering 2024-09-21T10:07:10.797 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:10.831 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:10.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:10.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:10 smithi196 ceph-mon[84784]: pgmap v382: 89 pgs: 1 peering, 1 active+remapped+backfilling, 87 active+clean; 2.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 483 op/s; 2385/142920 objects misplaced (1.669%); 11 MiB/s, 2 keys/s, 175 objects/s recovering 2024-09-21T10:07:10.934 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:10.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:10.981 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:10.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:11.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:11.118 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:11.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:11.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:11.150 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:07:11.150 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:07:11.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.151 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:226892 5.7 e145/105 5:e7e7ad69:::benchmark_data_smithi098_135187_object91254:head v 145'14158, mlcod=145'14153)", 2024-09-21T10:07:11.151 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:10.919073+0000", 2024-09-21T10:07:11.151 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0096115509999999994, 2024-09-21T10:07:11.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0096376949999999999, 2024-09-21T10:07:11.151 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:11.151 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:11.151 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:07:11.151 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:11.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.151 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:11.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.919073+0000", 2024-09-21T10:07:11.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:11.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:11.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.919073+0000", 2024-09-21T10:07:11.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:11.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:11.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.919074+0000", 2024-09-21T10:07:11.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3650000000000001e-06 2024-09-21T10:07:11.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:11.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.919083+0000", 2024-09-21T10:07:11.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1279999999999997e-06 2024-09-21T10:07:11.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.153 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:11.153 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.919094+0000", 2024-09-21T10:07:11.153 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0778000000000001e-05 2024-09-21T10:07:11.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.153 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:11.153 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.919102+0000", 2024-09-21T10:07:11.153 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.678e-06 2024-09-21T10:07:11.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.153 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:11.153 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.922624+0000", 2024-09-21T10:07:11.153 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035225690000000001 2024-09-21T10:07:11.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.154 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:11.154 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.922646+0000", 2024-09-21T10:07:11.154 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1841e-05 2024-09-21T10:07:11.154 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:11.154 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:11.154 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:11.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.154 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:226897 5.8 e145/105 5:15b35852:::benchmark_data_smithi098_135187_object91259:head v 145'14103, mlcod=145'14101)", 2024-09-21T10:07:11.154 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:10.924273+0000", 2024-09-21T10:07:11.154 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0044110319999999996, 2024-09-21T10:07:11.155 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044904400000000001, 2024-09-21T10:07:11.155 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:11.155 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:11.155 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:07:11.155 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:11.155 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.155 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:11.155 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.924273+0000", 2024-09-21T10:07:11.155 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:11.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.155 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.155 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:11.155 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.924273+0000", 2024-09-21T10:07:11.155 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:11.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.156 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:11.156 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.924274+0000", 2024-09-21T10:07:11.156 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1200000000000001e-06 2024-09-21T10:07:11.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.156 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:11.156 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.924367+0000", 2024-09-21T10:07:11.156 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2956999999999998e-05 2024-09-21T10:07:11.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.156 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:11.156 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.924382+0000", 2024-09-21T10:07:11.156 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4289e-05 2024-09-21T10:07:11.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.157 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:11.157 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.924389+0000", 2024-09-21T10:07:11.157 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5650000000000004e-06 2024-09-21T10:07:11.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.157 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:11.157 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.924537+0000", 2024-09-21T10:07:11.157 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014789 2024-09-21T10:07:11.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.157 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:11.157 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.924588+0000", 2024-09-21T10:07:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0643999999999998e-05 2024-09-21T10:07:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:226901 5.0 e145/141 5:0aa2d6e8:::benchmark_data_smithi098_135187_object91263:head v 145'13954, mlcod=145'13952)", 2024-09-21T10:07:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:10.925722+0000", 2024-09-21T10:07:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.002961927, 2024-09-21T10:07:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003088613, 2024-09-21T10:07:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:07:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.925722+0000", 2024-09-21T10:07:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.925722+0000", 2024-09-21T10:07:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.925723+0000", 2024-09-21T10:07:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6599999999999995e-07 2024-09-21T10:07:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.925729+0000", 2024-09-21T10:07:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1750000000000002e-06 2024-09-21T10:07:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.925740+0000", 2024-09-21T10:07:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0541e-05 2024-09-21T10:07:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.925746+0000", 2024-09-21T10:07:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.353e-06 2024-09-21T10:07:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.925821+0000", 2024-09-21T10:07:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4863999999999993e-05 2024-09-21T10:07:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.925839+0000", 2024-09-21T10:07:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7439999999999999e-05 2024-09-21T10:07:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:226898 5.7 e145/105 5:e27990ca:::benchmark_data_smithi098_135187_object91260:head v 145'14159, mlcod=145'14155)", 2024-09-21T10:07:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:10.928174+0000", 2024-09-21T10:07:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.00051064000000000003, 2024-09-21T10:07:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000682283, 2024-09-21T10:07:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:07:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.928174+0000", 2024-09-21T10:07:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:11.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.163 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:11.163 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.928174+0000", 2024-09-21T10:07:11.163 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:11.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.163 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:11.163 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.928175+0000", 2024-09-21T10:07:11.163 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8100000000000001e-07 2024-09-21T10:07:11.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.928182+0000", 2024-09-21T10:07:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4490000000000002e-06 2024-09-21T10:07:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.928193+0000", 2024-09-21T10:07:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1173e-05 2024-09-21T10:07:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.928200+0000", 2024-09-21T10:07:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.438e-06 2024-09-21T10:07:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.928369+0000", 2024-09-21T10:07:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000169369 2024-09-21T10:07:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:10.928433+0000", 2024-09-21T10:07:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3961999999999995e-05 2024-09-21T10:07:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-21T10:07:11.166 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 4 2024-09-21T10:07:11.166 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:07:11.220 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:11.257 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:11.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:11.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:11.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:11.397 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:11.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:11.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:11.499 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:11.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:11.544 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:11.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:11.618 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:07:11.618 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:07:11.619 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-21T10:07:11.619 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:07:11.619 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:07:11.619 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-21T10:07:11.645 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:11.682 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:11.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:11.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:11.784 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:11.824 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:11.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:11.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:11.926 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:11.968 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:11.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:11.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:12.070 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:12.113 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:12.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:12.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:12.215 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:12.250 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:12.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:12.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:12.352 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:12.387 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:12.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:12.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:12.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:12.531 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:12.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:12.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:12.620 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:07:12.632 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:12.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:12.669 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:12.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:12.769 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:12.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:12.816 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:12.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:12.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:12 smithi098 ceph-mon[97580]: pgmap v383: 89 pgs: 1 peering, 88 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 511 op/s; 7.9 MiB/s, 1 keys/s, 131 objects/s recovering 2024-09-21T10:07:12.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:12 smithi098 ceph-mon[100914]: pgmap v383: 89 pgs: 1 peering, 88 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 511 op/s; 7.9 MiB/s, 1 keys/s, 131 objects/s recovering 2024-09-21T10:07:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:12 smithi196 ceph-mon[84784]: pgmap v383: 89 pgs: 1 peering, 88 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 511 op/s; 7.9 MiB/s, 1 keys/s, 131 objects/s recovering 2024-09-21T10:07:12.917 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:12.933 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:12.945 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:12.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:12.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:13.047 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:13.087 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:13.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:13.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:13.189 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:13.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:13.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:13.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:13.332 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:13.367 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:13.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:13.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:13.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:13.506 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:13.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:13.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:13.608 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:13.640 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:13.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:13.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:13.742 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:13.768 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:13.769 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:13.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:13.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:13.841 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:13.871 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:13.909 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:13.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:13.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:14.011 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:14.037 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:14.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:14.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:14.139 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:14.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:14.171 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:14.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:14.272 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:14.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:14.310 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:14.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:14.410 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:14.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:07:14.448 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:14.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:14.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:14.550 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:14.587 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:14.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:14.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:14.689 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:14.739 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:14.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:14.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:14.779 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:14 smithi098 ceph-mon[97580]: pgmap v384: 89 pgs: 89 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 597 op/s; 6.7 MiB/s, 1 keys/s, 111 objects/s recovering 2024-09-21T10:07:14.779 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:14 smithi098 ceph-mon[100914]: pgmap v384: 89 pgs: 89 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 597 op/s; 6.7 MiB/s, 1 keys/s, 111 objects/s recovering 2024-09-21T10:07:14.841 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:14.884 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:14.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:14.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:14 smithi196 ceph-mon[84784]: pgmap v384: 89 pgs: 89 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 597 op/s; 6.7 MiB/s, 1 keys/s, 111 objects/s recovering 2024-09-21T10:07:14.987 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:15.020 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:15.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:15.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:15.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:15.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:15.158 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:15.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:15.259 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:15.296 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:15.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:15.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:15.397 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:15.424 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:15.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:15.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:15.526 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:15.567 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:15.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:15.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:15.669 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:15.719 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:15.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:15.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:15.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:15.859 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:15.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:15.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:15.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:16.001 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:16.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:16.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:16.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:16.137 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:16.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:16.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:16.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:16.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:16.293 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:16.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:16.394 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:16.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:16.440 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:16.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:16.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:16.585 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:16.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:16.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:16.687 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:16.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:16.728 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:16.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:16.805 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:16 smithi098 ceph-mon[97580]: pgmap v385: 89 pgs: 89 active+clean; 2.6 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 903 op/s; 5.5 MiB/s, 1 keys/s, 91 objects/s recovering 2024-09-21T10:07:16.805 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:16 smithi098 ceph-mon[100914]: pgmap v385: 89 pgs: 89 active+clean; 2.6 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 903 op/s; 5.5 MiB/s, 1 keys/s, 91 objects/s recovering 2024-09-21T10:07:16.830 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:16.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:16.870 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:16.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:16 smithi196 ceph-mon[84784]: pgmap v385: 89 pgs: 89 active+clean; 2.6 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 903 op/s; 5.5 MiB/s, 1 keys/s, 91 objects/s recovering 2024-09-21T10:07:16.971 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:17.025 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:17.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:17.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:17.127 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:17.171 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:17.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:17.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:17.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:17.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:17.313 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:17.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:17.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:17.452 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:17.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:17.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:17.554 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:17.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:17.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:17.692 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:17.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:17.730 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:17.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:17.831 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:17.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:17.864 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:17.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:17.965 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:18.012 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:18.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:18.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:18.114 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:18.145 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:18.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:18.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:18.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:18.279 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:18.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:18.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:18.381 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:18.410 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:18.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:18.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:18.513 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:18.548 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:18.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:18.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:18.650 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:18.688 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:07:18.688 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:07:18.688 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:07:18.697 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:18.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:18.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:18.798 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:18.829 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:18.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:18.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:18 smithi098 ceph-mon[97580]: pgmap v386: 89 pgs: 89 active+clean; 2.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 750 op/s; 4.6 MiB/s, 1 keys/s, 76 objects/s recovering 2024-09-21T10:07:18.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:18 smithi098 ceph-mon[100914]: pgmap v386: 89 pgs: 89 active+clean; 2.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 750 op/s; 4.6 MiB/s, 1 keys/s, 76 objects/s recovering 2024-09-21T10:07:18.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:18 smithi196 ceph-mon[84784]: pgmap v386: 89 pgs: 89 active+clean; 2.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 750 op/s; 4.6 MiB/s, 1 keys/s, 76 objects/s recovering 2024-09-21T10:07:18.931 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:18.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:18.970 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:18.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:19.070 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:19.112 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:19.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:19.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:19.214 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:19.248 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:19.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:19.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:19.350 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:19.370 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:19.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:19.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:19.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:19.505 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:19.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:19.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:19.607 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:19.639 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:19.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:19.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:19.740 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:19.771 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:19.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:19.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:19.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:19.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:19.921 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:19.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:20.022 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:20.050 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:20.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:20.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:20.105 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:07:20.106 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:07:20.152 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:20.159 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:07:20.161 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:07:20.192 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:20.210 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:20.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:20.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:20.214 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:07:20.214 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:07:20.261 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:07:20.261 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:07:20.293 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:07:20.294 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:07:20.312 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:20.324 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:07:20.324 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:07:20.345 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:20.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:20.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:20.357 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:07:20.357 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:07:20.392 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:07:20.393 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:07:20.447 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:20.481 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:20.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:20.562 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:07:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:07:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:07:20.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:20.587 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:07:20.628 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:20.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:20.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:20.729 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:20.764 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:20.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:20.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:20.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:07:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:07:20] "GET /metrics HTTP/1.1" 200 36472 "" "Prometheus/2.51.0" 2024-09-21T10:07:20.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:20 smithi098 ceph-mon[97580]: pgmap v387: 89 pgs: 89 active+clean; 2.4 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 806 op/s; 175 KiB/s, 2 objects/s recovering 2024-09-21T10:07:20.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:20 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3891813260' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:20.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:20 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1888474630' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:07:20.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:20 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:07:20.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:20 smithi098 ceph-mon[100914]: pgmap v387: 89 pgs: 89 active+clean; 2.4 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 806 op/s; 175 KiB/s, 2 objects/s recovering 2024-09-21T10:07:20.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:20 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3891813260' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:20.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:20 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1888474630' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:07:20.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:20 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:07:20.866 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:20 smithi196 ceph-mon[84784]: pgmap v387: 89 pgs: 89 active+clean; 2.4 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 806 op/s; 175 KiB/s, 2 objects/s recovering 2024-09-21T10:07:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:20 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3891813260' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:20 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1888474630' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:07:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:20 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:07:20.905 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:20.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:20.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:21.007 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:21.038 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:21.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:21.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:21.140 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:21.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:21.183 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:21.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:21.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:21.324 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:21.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:21.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:21.426 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:21.463 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:21.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:21.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:21.565 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:21.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:21.607 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:21.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:21.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:21.740 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:21.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:21.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:21.842 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:21.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:21 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:07:21.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:21 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:07:21.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:21 smithi098 ceph-mon[97580]: osdmap e146: 8 total, 8 up, 8 in 2024-09-21T10:07:21.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:21 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:07:21.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:21 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:07:21.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:21 smithi098 ceph-mon[100914]: osdmap e146: 8 total, 8 up, 8 in 2024-09-21T10:07:21.872 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:21.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:21.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:21 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:07:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:21 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:07:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:21 smithi196 ceph-mon[84784]: osdmap e146: 8 total, 8 up, 8 in 2024-09-21T10:07:21.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:22.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:22.009 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:22.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:22.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:22.139 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:22.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:22.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:22.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:22.285 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:22.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:22.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:22.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:22.421 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:22.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:22.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:22.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:22.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:22.559 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:22.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:22.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:22.689 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:22.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:22.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:22.791 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:22.836 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:22.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:22.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:22 smithi098 ceph-mon[97580]: pgmap v389: 89 pgs: 89 active+clean; 2.3 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 895 op/s 2024-09-21T10:07:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:22.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:22 smithi098 ceph-mon[100914]: pgmap v389: 89 pgs: 89 active+clean; 2.3 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 895 op/s 2024-09-21T10:07:22.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:22.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:22.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:22 smithi196 ceph-mon[84784]: pgmap v389: 89 pgs: 89 active+clean; 2.3 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 895 op/s 2024-09-21T10:07:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:22.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:22.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:22.969 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:22.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:23.070 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:23.101 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:23.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:23.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:23.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:23.233 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:23.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:23.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:23.328 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:07:23.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:23.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:23.417 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:23.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:23.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:23.556 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:23.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:23.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:23.657 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:23.688 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:23.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:23.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:23.790 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:23.834 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:23.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:23.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:23.937 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:23.968 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:23.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:23.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:24.069 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:24.101 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:24.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:24.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:24.202 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:24.233 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:24.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:24.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:24.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:24.367 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:24.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:24.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:24.465 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:07:24.466 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:07:24.467 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:07:24.467 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:07:24.467 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-21T10:07:24.467 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:07:24.467 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:07:24.468 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-21T10:07:24.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:24.501 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:24.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:24.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:24.604 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:24.641 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:24.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:24.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:24.743 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:24.743 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:24.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:24.794 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:24.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:24 smithi098 ceph-mon[97580]: pgmap v390: 89 pgs: 89 active+clean; 2.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 900 op/s 2024-09-21T10:07:24.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:24 smithi098 ceph-mon[100914]: pgmap v390: 89 pgs: 89 active+clean; 2.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 900 op/s 2024-09-21T10:07:24.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:24 smithi196 ceph-mon[84784]: pgmap v390: 89 pgs: 89 active+clean; 2.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 900 op/s 2024-09-21T10:07:24.895 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:24.914 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:07:24.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:24.931 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:24.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:25.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:25.062 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:25.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:25.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:25.164 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:25.168 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:07:25.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:25.199 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:25.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:25.300 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:25.342 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:25.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:25.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:25.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:07:25.443 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:25.481 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:25.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:25.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:25.583 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:25.613 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:25.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:25.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:25.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:25.769 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:25.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:25.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:25.871 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:25.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:25.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:26.007 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:26.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:26.042 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:26.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:26.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:26.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:26.182 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:26.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:26.283 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:26.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:26.327 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:26.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:26.428 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:26.457 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:26.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:26.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:26.560 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:26.594 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:26.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:26.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:26.696 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:26.730 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:26.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:26.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:26.783 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:26.794 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:26.832 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:26.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:26 smithi098 ceph-mon[97580]: pgmap v391: 89 pgs: 89 active+clean; 2.0 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 884 op/s 2024-09-21T10:07:26.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:26 smithi098 ceph-mon[100914]: pgmap v391: 89 pgs: 89 active+clean; 2.0 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 884 op/s 2024-09-21T10:07:26.862 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:26.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:26.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:26 smithi196 ceph-mon[84784]: pgmap v391: 89 pgs: 89 active+clean; 2.0 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 884 op/s 2024-09-21T10:07:26.964 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:26.997 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:26.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:26.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:27.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:27.133 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:27.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:27.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:27.235 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:27.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:27.278 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:27.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:27.358 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:07:27.358 INFO:teuthology.orchestra.run.smithi196.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:07:27.358 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:07:27.378 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:27.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:27.431 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:27.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:27.532 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:27.558 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:27.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:27.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:27.660 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:27.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:27.708 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:27.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:27.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:27.854 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:27.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:27.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:27.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:27.998 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:27.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:27.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:28.100 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:28.149 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:28.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:28.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:28.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:28.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:28.298 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:28.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:28.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:28.431 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:28.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:28.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:28.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:28.558 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:28.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:28.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:28.660 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:28.715 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:28.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:28.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:28.743 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:28 smithi098 ceph-mon[97580]: pgmap v392: 89 pgs: 89 active+clean; 2.0 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 882 op/s 2024-09-21T10:07:28.743 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:28 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:28.743 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:28 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:28.743 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:28 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:28.743 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:28 smithi098 ceph-mon[100914]: pgmap v392: 89 pgs: 89 active+clean; 2.0 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 882 op/s 2024-09-21T10:07:28.743 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:28 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:28.743 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:28 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:28.743 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:28 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:28.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:28.868 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:28.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:28.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:28 smithi196 ceph-mon[84784]: pgmap v392: 89 pgs: 89 active+clean; 2.0 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 882 op/s 2024-09-21T10:07:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:28 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:28 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:28 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:28.893 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:07:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:07:28.929 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:07:28.929 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:07:28.965 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:07:28.965 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:07:28.969 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:28.996 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:07:28.996 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:07:29.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:29.008 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:29.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:29.032 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:07:29.032 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:07:29.074 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:07:29.074 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:07:29.102 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:07:29.102 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:07:29.108 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:29.128 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:07:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:07:29.149 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:29.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:29.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:29.251 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:29.283 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:29.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:29.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:29.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:29.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:29.416 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:29.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:29.517 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:29.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:29.579 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:29.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:29.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:29.707 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:29.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:29.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:29.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:29.842 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:29.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:29.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:29.943 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:29.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:29.974 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:29.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:29 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:29 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:29 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/573313373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:30.075 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:30.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:29 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:30.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:29 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:30.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:29 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/573313373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:30.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:29 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:30.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:29 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:30.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:29 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/573313373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:30.104 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:30.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:30.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:30.206 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:30.232 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:30.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:30.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:30.334 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:30.358 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:30.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:30.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:30.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:07:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:07:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:07:30.460 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:30.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:30.515 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:30.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:30.616 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:30.652 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:30.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:30.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:30.754 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:30.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:30.816 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:30.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:30.893 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:07:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:07:30] "GET /metrics HTTP/1.1" 200 36448 "" "Prometheus/2.51.0" 2024-09-21T10:07:30.894 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:30 smithi098 ceph-mon[97580]: pgmap v393: 89 pgs: 89 active+clean; 1.9 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 871 op/s 2024-09-21T10:07:30.894 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:30 smithi098 ceph-mon[100914]: pgmap v393: 89 pgs: 89 active+clean; 1.9 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 871 op/s 2024-09-21T10:07:30.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:30.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:30.951 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:30.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:31.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:31.096 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:31.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:31.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:31.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:30 smithi196 ceph-mon[84784]: pgmap v393: 89 pgs: 89 active+clean; 1.9 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 871 op/s 2024-09-21T10:07:31.198 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:31.227 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:31.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:31.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:31.329 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:31.361 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:31.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:31.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:31.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:31.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:31.522 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:31.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:31.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:31.661 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:31.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:31.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:31.762 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:31.806 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:31.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:31.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:31.908 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:31.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:31.947 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:31.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:32.048 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:32.085 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:32.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:32.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:32.187 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:32.228 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:32.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:32.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:32.291 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:07:32.291 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-21T10:07:32.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-21T10:07:32.291 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:07:32.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:81416 5.0 e114/105 5:0cb8852c:::benchmark_data_smithi098_135187_object81415:head v 114'5063, mlcod=114'5061)", 2024-09-21T10:07:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:29.266858+0000", 2024-09-21T10:07:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 183.01857468200001, 2024-09-21T10:07:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.049670543999999997, 2024-09-21T10:07:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.266858+0000", 2024-09-21T10:07:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.266858+0000", 2024-09-21T10:07:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.266859+0000", 2024-09-21T10:07:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6400000000000002e-07 2024-09-21T10:07:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.266880+0000", 2024-09-21T10:07:32.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0866999999999999e-05 2024-09-21T10:07:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.266887+0000", 2024-09-21T10:07:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5050000000000001e-06 2024-09-21T10:07:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.266892+0000", 2024-09-21T10:07:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7369999999999997e-06 2024-09-21T10:07:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.266949+0000", 2024-09-21T10:07:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7469999999999997e-05 2024-09-21T10:07:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.266971+0000", 2024-09-21T10:07:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2011e-05 2024-09-21T10:07:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:07:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316486+0000", 2024-09-21T10:07:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.049514829000000003 2024-09-21T10:07:32.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:07:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316529+0000", 2024-09-21T10:07:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2561000000000001e-05 2024-09-21T10:07:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:81413 5.7 e114/105 5:eb012c7c:::benchmark_data_smithi098_135187_object81412:head v 114'5126, mlcod=114'5124)", 2024-09-21T10:07:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:29.268487+0000", 2024-09-21T10:07:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 183.016945921, 2024-09-21T10:07:32.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.048055411999999999, 2024-09-21T10:07:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.268487+0000", 2024-09-21T10:07:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.268487+0000", 2024-09-21T10:07:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.268488+0000", 2024-09-21T10:07:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4499999999999996e-07 2024-09-21T10:07:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.268534+0000", 2024-09-21T10:07:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6437000000000003e-05 2024-09-21T10:07:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:32.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.268547+0000", 2024-09-21T10:07:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2927e-05 2024-09-21T10:07:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.268554+0000", 2024-09-21T10:07:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7519999999999999e-06 2024-09-21T10:07:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.269782+0000", 2024-09-21T10:07:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001227943 2024-09-21T10:07:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.269800+0000", 2024-09-21T10:07:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8003e-05 2024-09-21T10:07:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:07:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316503+0000", 2024-09-21T10:07:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.046702746000000003 2024-09-21T10:07:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:07:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316542+0000", 2024-09-21T10:07:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9759000000000003e-05 2024-09-21T10:07:32.300 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:81410 5.8 e114/105 5:1918b719:::benchmark_data_smithi098_135187_object81409:head v 114'4993, mlcod=114'4991)", 2024-09-21T10:07:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:29.270376+0000", 2024-09-21T10:07:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 183.01505660399999, 2024-09-21T10:07:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.048632575999999997, 2024-09-21T10:07:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.301 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.270376+0000", 2024-09-21T10:07:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.270376+0000", 2024-09-21T10:07:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.270377+0000", 2024-09-21T10:07:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0750000000000001e-06 2024-09-21T10:07:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.270401+0000", 2024-09-21T10:07:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3357999999999999e-05 2024-09-21T10:07:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.270412+0000", 2024-09-21T10:07:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0903999999999999e-05 2024-09-21T10:07:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.270419+0000", 2024-09-21T10:07:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0829999999999998e-06 2024-09-21T10:07:32.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.270494+0000", 2024-09-21T10:07:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4870999999999995e-05 2024-09-21T10:07:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.270511+0000", 2024-09-21T10:07:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7158000000000001e-05 2024-09-21T10:07:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:07:32.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.318957+0000", 2024-09-21T10:07:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.048446217 2024-09-21T10:07:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:07:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.319009+0000", 2024-09-21T10:07:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1910000000000003e-05 2024-09-21T10:07:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:81417 5.8 e114/105 5:13f3ab1d:::benchmark_data_smithi098_135187_object81416:head v 114'4994, mlcod=114'4991)", 2024-09-21T10:07:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:29.271294+0000", 2024-09-21T10:07:32.305 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 183.014139309, 2024-09-21T10:07:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.047744399999999999, 2024-09-21T10:07:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.271294+0000", 2024-09-21T10:07:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.271294+0000", 2024-09-21T10:07:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.271294+0000", 2024-09-21T10:07:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0699999999999997e-07 2024-09-21T10:07:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.271315+0000", 2024-09-21T10:07:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0879000000000001e-05 2024-09-21T10:07:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.271325+0000", 2024-09-21T10:07:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4620000000000002e-06 2024-09-21T10:07:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.271331+0000", 2024-09-21T10:07:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1589999999999998e-06 2024-09-21T10:07:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.272973+0000", 2024-09-21T10:07:32.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016424409999999999 2024-09-21T10:07:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.272991+0000", 2024-09-21T10:07:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7657999999999999e-05 2024-09-21T10:07:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:07:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.319012+0000", 2024-09-21T10:07:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.046021199999999998 2024-09-21T10:07:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:07:32.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.319038+0000", 2024-09-21T10:07:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5993999999999999e-05 2024-09-21T10:07:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:81425 5.7 e114/105 5:e0488b79:::benchmark_data_smithi098_135187_object81424:head v 114'5128, mlcod=114'5124)", 2024-09-21T10:07:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:29.277076+0000", 2024-09-21T10:07:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 183.00835725300001, 2024-09-21T10:07:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.046619946000000002, 2024-09-21T10:07:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:32.310 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.277076+0000", 2024-09-21T10:07:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.277076+0000", 2024-09-21T10:07:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.277077+0000", 2024-09-21T10:07:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8899999999999998e-07 2024-09-21T10:07:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.277147+0000", 2024-09-21T10:07:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0088999999999999e-05 2024-09-21T10:07:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316484+0000", 2024-09-21T10:07:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.039337142999999998 2024-09-21T10:07:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316498+0000", 2024-09-21T10:07:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3715e-05 2024-09-21T10:07:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316570+0000", 2024-09-21T10:07:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2628999999999999e-05 2024-09-21T10:07:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316589+0000", 2024-09-21T10:07:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8456000000000001e-05 2024-09-21T10:07:32.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:07:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.323585+0000", 2024-09-21T10:07:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0069967839999999998 2024-09-21T10:07:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:07:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.323696+0000", 2024-09-21T10:07:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000110141 2024-09-21T10:07:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:81434 5.c e114/105 5:31ef8da0:::benchmark_data_smithi098_135187_object81433:head v 114'4926, mlcod=114'4924)", 2024-09-21T10:07:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:29.281286+0000", 2024-09-21T10:07:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 183.004146663, 2024-09-21T10:07:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.051817101999999997, 2024-09-21T10:07:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.281286+0000", 2024-09-21T10:07:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.281286+0000", 2024-09-21T10:07:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.281287+0000", 2024-09-21T10:07:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0759999999999999e-06 2024-09-21T10:07:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.281370+0000", 2024-09-21T10:07:32.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2204000000000001e-05 2024-09-21T10:07:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316486+0000", 2024-09-21T10:07:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035116461000000002 2024-09-21T10:07:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316498+0000", 2024-09-21T10:07:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.216e-05 2024-09-21T10:07:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:32.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316657+0000", 2024-09-21T10:07:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015876 2024-09-21T10:07:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.316676+0000", 2024-09-21T10:07:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9009999999999999e-05 2024-09-21T10:07:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:07:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.333051+0000", 2024-09-21T10:07:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016374667999999998 2024-09-21T10:07:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:07:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:29.333103+0000", 2024-09-21T10:07:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2763000000000003e-05 2024-09-21T10:07:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:153443 5.0 e123/105 5:01cb1c52:::benchmark_data_smithi098_135187_object17805:head v 123'9410, mlcod=123'9408)", 2024-09-21T10:07:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:05:39.354291+0000", 2024-09-21T10:07:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 112.93114179299999, 2024-09-21T10:07:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.064051306000000002, 2024-09-21T10:07:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:32.319 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:32.320 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:32.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:32.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.354291+0000", 2024-09-21T10:07:32.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.354291+0000", 2024-09-21T10:07:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.354292+0000", 2024-09-21T10:07:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7000000000000003e-07 2024-09-21T10:07:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.354298+0000", 2024-09-21T10:07:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8150000000000002e-06 2024-09-21T10:07:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.354309+0000", 2024-09-21T10:07:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0727000000000001e-05 2024-09-21T10:07:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.354315+0000", 2024-09-21T10:07:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5649999999999998e-06 2024-09-21T10:07:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.354404+0000", 2024-09-21T10:07:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8894999999999995e-05 2024-09-21T10:07:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.354422+0000", 2024-09-21T10:07:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8077999999999999e-05 2024-09-21T10:07:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:07:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.418309+0000", 2024-09-21T10:07:32.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.063886942000000002 2024-09-21T10:07:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:07:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.418342+0000", 2024-09-21T10:07:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3414000000000002e-05 2024-09-21T10:07:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:153444 5.0 e123/105 5:03342715:::benchmark_data_smithi098_135187_object17806:head v 123'9411, mlcod=123'9408)", 2024-09-21T10:07:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:05:39.358226+0000", 2024-09-21T10:07:32.324 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 112.927207079, 2024-09-21T10:07:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.068639825000000002, 2024-09-21T10:07:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.358226+0000", 2024-09-21T10:07:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.358226+0000", 2024-09-21T10:07:32.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.358227+0000", 2024-09-21T10:07:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4900000000000005e-07 2024-09-21T10:07:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.358232+0000", 2024-09-21T10:07:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.604e-06 2024-09-21T10:07:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.358242+0000", 2024-09-21T10:07:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8889999999999997e-06 2024-09-21T10:07:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.358256+0000", 2024-09-21T10:07:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3828999999999999e-05 2024-09-21T10:07:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.363071+0000", 2024-09-21T10:07:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048150069999999996 2024-09-21T10:07:32.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.363090+0000", 2024-09-21T10:07:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9344999999999999e-05 2024-09-21T10:07:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:07:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.426826+0000", 2024-09-21T10:07:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.063735498000000002 2024-09-21T10:07:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:07:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.426866+0000", 2024-09-21T10:07:32.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9904000000000003e-05 2024-09-21T10:07:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:153446 5.c e123/105 5:38941910:::benchmark_data_smithi098_135187_object17808:head v 123'9475, mlcod=123'9473)", 2024-09-21T10:07:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:05:39.358418+0000", 2024-09-21T10:07:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 112.927015285, 2024-09-21T10:07:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.056098497999999997, 2024-09-21T10:07:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:32.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.358418+0000", 2024-09-21T10:07:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.358418+0000", 2024-09-21T10:07:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.358418+0000", 2024-09-21T10:07:32.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1600000000000001e-07 2024-09-21T10:07:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.358424+0000", 2024-09-21T10:07:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7339999999999996e-06 2024-09-21T10:07:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.358432+0000", 2024-09-21T10:07:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9230000000000002e-06 2024-09-21T10:07:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:32.331 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.358436+0000", 2024-09-21T10:07:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6700000000000002e-06 2024-09-21T10:07:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.358557+0000", 2024-09-21T10:07:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000120283 2024-09-21T10:07:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.358574+0000", 2024-09-21T10:07:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7127999999999999e-05 2024-09-21T10:07:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.332 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:07:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.414460+0000", 2024-09-21T10:07:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.055886586000000002 2024-09-21T10:07:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:07:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.414516+0000", 2024-09-21T10:07:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5658000000000001e-05 2024-09-21T10:07:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:153447 5.0 e123/105 5:051f2a8b:::benchmark_data_smithi098_135187_object17809:head v 123'9412, mlcod=123'9408)", 2024-09-21T10:07:32.333 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:05:39.363444+0000", 2024-09-21T10:07:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 112.92198882, 2024-09-21T10:07:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.067554385999999994, 2024-09-21T10:07:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.363444+0000", 2024-09-21T10:07:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.363444+0000", 2024-09-21T10:07:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.363445+0000", 2024-09-21T10:07:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.64e-07 2024-09-21T10:07:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.363452+0000", 2024-09-21T10:07:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.528e-06 2024-09-21T10:07:32.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.414508+0000", 2024-09-21T10:07:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.051056490000000003 2024-09-21T10:07:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.414515+0000", 2024-09-21T10:07:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7619999999999998e-06 2024-09-21T10:07:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:32.336 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.418358+0000", 2024-09-21T10:07:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00384337 2024-09-21T10:07:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.418371+0000", 2024-09-21T10:07:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2653999999999999e-05 2024-09-21T10:07:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:07:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.430908+0000", 2024-09-21T10:07:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012537609 2024-09-21T10:07:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.337 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:07:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.430998+0000", 2024-09-21T10:07:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0008999999999998e-05 2024-09-21T10:07:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:153451 5.7 e123/105 5:e59497ee:::benchmark_data_smithi098_135187_object17813:head v 123'9533, mlcod=123'9530)", 2024-09-21T10:07:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:05:39.365134+0000", 2024-09-21T10:07:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 112.920299361, 2024-09-21T10:07:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.057072396999999997, 2024-09-21T10:07:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:32.338 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.365134+0000", 2024-09-21T10:07:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.365134+0000", 2024-09-21T10:07:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.365135+0000", 2024-09-21T10:07:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1119999999999999e-06 2024-09-21T10:07:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.365144+0000", 2024-09-21T10:07:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7510000000000003e-06 2024-09-21T10:07:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.414434+0000", 2024-09-21T10:07:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.049289263 2024-09-21T10:07:32.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.414448+0000", 2024-09-21T10:07:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4338999999999999e-05 2024-09-21T10:07:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.414502+0000", 2024-09-21T10:07:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3643999999999997e-05 2024-09-21T10:07:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:32.341 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.414534+0000", 2024-09-21T10:07:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.235e-05 2024-09-21T10:07:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:07:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.422134+0000", 2024-09-21T10:07:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0076001590000000004 2024-09-21T10:07:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:07:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.422206+0000", 2024-09-21T10:07:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1779000000000005e-05 2024-09-21T10:07:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.342 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:153463 5.8 e123/105 5:191b2f14:::benchmark_data_smithi098_135187_object17825:head v 123'9552, mlcod=123'9550)", 2024-09-21T10:07:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:05:39.371974+0000", 2024-09-21T10:07:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 112.913459003, 2024-09-21T10:07:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.063803158999999998, 2024-09-21T10:07:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:32.343 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.371974+0000", 2024-09-21T10:07:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.371974+0000", 2024-09-21T10:07:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.371975+0000", 2024-09-21T10:07:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.164e-06 2024-09-21T10:07:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.371983+0000", 2024-09-21T10:07:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0190000000000006e-06 2024-09-21T10:07:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.414454+0000", 2024-09-21T10:07:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.042470542 2024-09-21T10:07:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.414464+0000", 2024-09-21T10:07:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0716999999999999e-05 2024-09-21T10:07:32.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.418329+0000", 2024-09-21T10:07:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038647859999999998 2024-09-21T10:07:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.418343+0000", 2024-09-21T10:07:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4113e-05 2024-09-21T10:07:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:07:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.435643+0000", 2024-09-21T10:07:32.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017299361999999999 2024-09-21T10:07:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:07:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.435777+0000", 2024-09-21T10:07:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000134456 2024-09-21T10:07:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:153475 5.8 e123/105 5:1a0e5318:::benchmark_data_smithi098_135187_object17837:head v 123'9556, mlcod=123'9550)", 2024-09-21T10:07:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:05:39.414553+0000", 2024-09-21T10:07:32.347 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 112.87088037300001, 2024-09-21T10:07:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.047174779, 2024-09-21T10:07:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.414553+0000", 2024-09-21T10:07:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.414553+0000", 2024-09-21T10:07:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:32.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.414553+0000", 2024-09-21T10:07:32.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5400000000000002e-07 2024-09-21T10:07:32.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:32.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.414555+0000", 2024-09-21T10:07:32.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4609999999999998e-06 2024-09-21T10:07:32.349 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:32.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.414560+0000", 2024-09-21T10:07:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5669999999999996e-06 2024-09-21T10:07:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.414564+0000", 2024-09-21T10:07:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9319999999999998e-06 2024-09-21T10:07:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:32.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.436442+0000", 2024-09-21T10:07:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.021877880999999998 2024-09-21T10:07:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.436547+0000", 2024-09-21T10:07:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105604 2024-09-21T10:07:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:07:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.461684+0000", 2024-09-21T10:07:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.025136261 2024-09-21T10:07:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:07:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.461727+0000", 2024-09-21T10:07:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3819000000000002e-05 2024-09-21T10:07:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:210366 5.0 e133/105 5:0cae2c65:::benchmark_data_smithi098_135187_object74728:head v 133'12958, mlcod=133'12955)", 2024-09-21T10:07:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:06:47.092570+0000", 2024-09-21T10:07:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 45.192862841999997, 2024-09-21T10:07:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.052983740000000001, 2024-09-21T10:07:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.092570+0000", 2024-09-21T10:07:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.092570+0000", 2024-09-21T10:07:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.092572+0000", 2024-09-21T10:07:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.747e-06 2024-09-21T10:07:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.092583+0000", 2024-09-21T10:07:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1381e-05 2024-09-21T10:07:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:32.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.092598+0000", 2024-09-21T10:07:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5193e-05 2024-09-21T10:07:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.092608+0000", 2024-09-21T10:07:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.679e-06 2024-09-21T10:07:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.092827+0000", 2024-09-21T10:07:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00021895199999999999 2024-09-21T10:07:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.092846+0000", 2024-09-21T10:07:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.908e-05 2024-09-21T10:07:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:07:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.145404+0000", 2024-09-21T10:07:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.05255758 2024-09-21T10:07:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:07:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.145554+0000", 2024-09-21T10:07:32.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000150128 2024-09-21T10:07:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:232825 5.0 e145/141 5:002604e9:::benchmark_data_smithi098_135187_object97187:head v 145'14325, mlcod=145'14322)", 2024-09-21T10:07:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:17.557986+0000", 2024-09-21T10:07:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.727447214, 2024-09-21T10:07:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.050056731, 2024-09-21T10:07:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.557986+0000", 2024-09-21T10:07:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.557986+0000", 2024-09-21T10:07:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.557987+0000", 2024-09-21T10:07:32.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.164e-06 2024-09-21T10:07:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.557996+0000", 2024-09-21T10:07:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7239999999999998e-06 2024-09-21T10:07:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.558011+0000", 2024-09-21T10:07:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.535e-05 2024-09-21T10:07:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.558019+0000", 2024-09-21T10:07:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0450000000000001e-06 2024-09-21T10:07:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.565891+0000", 2024-09-21T10:07:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.007872301 2024-09-21T10:07:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.566051+0000", 2024-09-21T10:07:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015985399999999999 2024-09-21T10:07:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:07:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.607849+0000", 2024-09-21T10:07:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.041798271999999997 2024-09-21T10:07:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:07:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.608042+0000", 2024-09-21T10:07:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019302100000000001 2024-09-21T10:07:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:232839 5.8 e145/105 5:18f22f5f:::benchmark_data_smithi098_135187_object97201:head v 145'14472, mlcod=145'14470)", 2024-09-21T10:07:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:17.568234+0000", 2024-09-21T10:07:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.717198673, 2024-09-21T10:07:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.048211739000000003, 2024-09-21T10:07:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.568234+0000", 2024-09-21T10:07:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.568234+0000", 2024-09-21T10:07:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.568235+0000", 2024-09-21T10:07:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1459999999999999e-06 2024-09-21T10:07:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.568243+0000", 2024-09-21T10:07:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2910000000000002e-06 2024-09-21T10:07:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.568260+0000", 2024-09-21T10:07:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7295999999999998e-05 2024-09-21T10:07:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.568268+0000", 2024-09-21T10:07:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9500000000000001e-06 2024-09-21T10:07:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.571246+0000", 2024-09-21T10:07:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029784009999999999 2024-09-21T10:07:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.571296+0000", 2024-09-21T10:07:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9651999999999998e-05 2024-09-21T10:07:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:07:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.616358+0000", 2024-09-21T10:07:32.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.045061563999999998 2024-09-21T10:07:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:07:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.616446+0000", 2024-09-21T10:07:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8438999999999998e-05 2024-09-21T10:07:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:232834 5.0 e145/141 5:0bae319f:::benchmark_data_smithi098_135187_object97196:head v 145'14326, mlcod=145'14322)", 2024-09-21T10:07:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:17.569105+0000", 2024-09-21T10:07:32.367 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.716328272, 2024-09-21T10:07:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.047426096000000001, 2024-09-21T10:07:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.569105+0000", 2024-09-21T10:07:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:32.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.569105+0000", 2024-09-21T10:07:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.569112+0000", 2024-09-21T10:07:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8889999999999996e-06 2024-09-21T10:07:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.569119+0000", 2024-09-21T10:07:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9240000000000007e-06 2024-09-21T10:07:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.569139+0000", 2024-09-21T10:07:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9046999999999999e-05 2024-09-21T10:07:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.569147+0000", 2024-09-21T10:07:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9549999999999998e-06 2024-09-21T10:07:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.576278+0000", 2024-09-21T10:07:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0071306640000000001 2024-09-21T10:07:32.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.576329+0000", 2024-09-21T10:07:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0571000000000001e-05 2024-09-21T10:07:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:07:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.616453+0000", 2024-09-21T10:07:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.040123961 2024-09-21T10:07:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:07:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.616531+0000", 2024-09-21T10:07:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8084999999999995e-05 2024-09-21T10:07:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:232840 5.8 e145/105 5:1496c6e5:::benchmark_data_smithi098_135187_object97202:head v 145'14473, mlcod=145'14470)", 2024-09-21T10:07:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:17.576622+0000", 2024-09-21T10:07:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.708811277000001, 2024-09-21T10:07:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.051329975999999999, 2024-09-21T10:07:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:32.372 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.576622+0000", 2024-09-21T10:07:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.576622+0000", 2024-09-21T10:07:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.576623+0000", 2024-09-21T10:07:32.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8599999999999996e-07 2024-09-21T10:07:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.576629+0000", 2024-09-21T10:07:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8630000000000002e-06 2024-09-21T10:07:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.576641+0000", 2024-09-21T10:07:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1838000000000001e-05 2024-09-21T10:07:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.576648+0000", 2024-09-21T10:07:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8399999999999997e-06 2024-09-21T10:07:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.581774+0000", 2024-09-21T10:07:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051259679999999998 2024-09-21T10:07:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.581797+0000", 2024-09-21T10:07:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3258999999999999e-05 2024-09-21T10:07:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:07:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.627764+0000", 2024-09-21T10:07:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.045966184 2024-09-21T10:07:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:07:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.627952+0000", 2024-09-21T10:07:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018803799999999999 2024-09-21T10:07:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:232847 5.0 e145/141 5:096d31e7:::benchmark_data_smithi098_135187_object97209:head v 145'14327, mlcod=145'14323)", 2024-09-21T10:07:32.376 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:17.581608+0000", 2024-09-21T10:07:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.703824581999999, 2024-09-21T10:07:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.046537598999999999, 2024-09-21T10:07:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.581608+0000", 2024-09-21T10:07:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.581608+0000", 2024-09-21T10:07:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.581610+0000", 2024-09-21T10:07:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3030000000000001e-06 2024-09-21T10:07:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.581618+0000", 2024-09-21T10:07:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4440000000000008e-06 2024-09-21T10:07:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.581637+0000", 2024-09-21T10:07:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9185e-05 2024-09-21T10:07:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.581646+0000", 2024-09-21T10:07:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7679999999999997e-06 2024-09-21T10:07:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.590569+0000", 2024-09-21T10:07:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0089232189999999996 2024-09-21T10:07:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.590587+0000", 2024-09-21T10:07:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8104000000000001e-05 2024-09-21T10:07:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:07:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.627956+0000", 2024-09-21T10:07:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037368230000000002 2024-09-21T10:07:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:07:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:17.628146+0000", 2024-09-21T10:07:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019034600000000001 2024-09-21T10:07:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.44894.0:234077 5.8 e145/105 5:11b6eaa2:::benchmark_data_smithi098_135187_object98439:head v 145'14546, mlcod=145'14542)", 2024-09-21T10:07:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:18.994996+0000", 2024-09-21T10:07:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.290436808999999, 2024-09-21T10:07:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.050464243999999998, 2024-09-21T10:07:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:07:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:07:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:07:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:07:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:18.994996+0000", 2024-09-21T10:07:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:07:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:18.994996+0000", 2024-09-21T10:07:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:07:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:07:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:18.994997+0000", 2024-09-21T10:07:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8899999999999998e-07 2024-09-21T10:07:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:07:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:18.995003+0000", 2024-09-21T10:07:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2960000000000004e-06 2024-09-21T10:07:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:07:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:18.995019+0000", 2024-09-21T10:07:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5967000000000001e-05 2024-09-21T10:07:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:07:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:18.995027+0000", 2024-09-21T10:07:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4220000000000003e-06 2024-09-21T10:07:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:07:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:19.005697+0000", 2024-09-21T10:07:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010670750999999999 2024-09-21T10:07:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:07:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:19.005741+0000", 2024-09-21T10:07:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3668999999999998e-05 2024-09-21T10:07:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:07:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:19.045320+0000", 2024-09-21T10:07:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.039578557 2024-09-21T10:07:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:07:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:07:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:07:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:19.045460+0000", 2024-09-21T10:07:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000140693 2024-09-21T10:07:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:07:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:07:32.385 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:07:32.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:32.435 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:32.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:32.536 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:32.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:32.571 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:32.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:32.653 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:07:32.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:32.717 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:32.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:32.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:32.761 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:07:32.818 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:32.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:32.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:32.963 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:32 smithi098 ceph-mon[97580]: pgmap v394: 89 pgs: 89 active+clean; 1.8 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 813 op/s 2024-09-21T10:07:32.963 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:32 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1878125185' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:32.963 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:32 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:32.964 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:32 smithi098 ceph-mon[100914]: pgmap v394: 89 pgs: 89 active+clean; 1.8 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 813 op/s 2024-09-21T10:07:32.964 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:32 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1878125185' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:32.964 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:32 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:32.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:33.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:33.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:33.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:32 smithi196 ceph-mon[84784]: pgmap v394: 89 pgs: 89 active+clean; 1.8 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 813 op/s 2024-09-21T10:07:33.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:32 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1878125185' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:33.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:32 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:33.057 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:07:33.057 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:07:33.057 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T10:07:33.057 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.000000 2024-09-21T10:07:33.058 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 5 0 2024-09-21T10:07:33.125 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:33.159 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:33.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:33.262 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:33.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:33.307 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:33.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:33.408 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:33.446 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:33.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:33.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:33.547 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:33.610 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:33.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:33.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:33.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:33.747 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:33.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:33.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:33.849 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:33.907 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:33.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:33.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:34.009 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:34.058 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:34.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:34.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:34.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:33 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:07:34.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:33 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:07:34.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:33 smithi098 ceph-mon[97580]: osdmap e147: 8 total, 8 up, 8 in 2024-09-21T10:07:34.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:33 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:34.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:33 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:34.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:33 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:07:34.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:33 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:07:34.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:33 smithi098 ceph-mon[100914]: osdmap e147: 8 total, 8 up, 8 in 2024-09-21T10:07:34.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:33 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:34.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:33 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:34.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:33 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:07:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:33 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:07:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:33 smithi196 ceph-mon[84784]: osdmap e147: 8 total, 8 up, 8 in 2024-09-21T10:07:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:33 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:33 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:34.160 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:34.197 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:34.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:34.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:34.299 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:34.334 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:34.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:34.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:34.435 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:34.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:34.464 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:34.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:34.565 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:34.599 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:34.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:34.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:34.701 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:34.734 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:34.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:34.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:34.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:34.862 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:34.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:34.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:34.964 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:35.000 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:35.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:35.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:35.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:34 smithi098 ceph-mon[97580]: pgmap v396: 89 pgs: 89 active+clean; 1.7 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 815 op/s 2024-09-21T10:07:35.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:34 smithi098 ceph-mon[100914]: pgmap v396: 89 pgs: 89 active+clean; 1.7 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 815 op/s 2024-09-21T10:07:35.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:35.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:35.133 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:35.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:35.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:34 smithi196 ceph-mon[84784]: pgmap v396: 89 pgs: 89 active+clean; 1.7 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 815 op/s 2024-09-21T10:07:35.198 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:07:35.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:35.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:35.283 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:35.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:35.384 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:35.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:35.411 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:35.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:35.512 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:35.546 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:35.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:35.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:35.648 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:35.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:35.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:35.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:35.830 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:35.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:35.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:35.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:07:35.932 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:36.023 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:36.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:36.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:36.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:36.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:36.172 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:36.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:36.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:36.298 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:36.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:36.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:36.400 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:36.423 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:36.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:36.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:36.425 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:36.525 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:36.552 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:36.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:36.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:36.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:36.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:36.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:36.788 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:36.817 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:36.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:36.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:36.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:36 smithi098 ceph-mon[97580]: pgmap v397: 89 pgs: 89 active+clean; 1.5 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 808 op/s 2024-09-21T10:07:36.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:36 smithi098 ceph-mon[100914]: pgmap v397: 89 pgs: 89 active+clean; 1.5 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 808 op/s 2024-09-21T10:07:36.919 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:36.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:36.966 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:36.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:37.067 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:37.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:37.102 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:37.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:37.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:36 smithi196 ceph-mon[84784]: pgmap v397: 89 pgs: 89 active+clean; 1.5 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 808 op/s 2024-09-21T10:07:37.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:37.239 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:37.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:37.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:37.248 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:37.341 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:37.367 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:37.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:37.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:37.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:37.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:37.514 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:37.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:37.615 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:37.648 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:37.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:37.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:37.750 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:37.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:37.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:37.891 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:37.922 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:37.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:37.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:38.023 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:38.061 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:38.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:38.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:38.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:38.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:38.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:38.163 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:38.191 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:38.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:38.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:38.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:38.323 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:38.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:38.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:38.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:38.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:38.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:38.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:38.596 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:38.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:38.696 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:38.725 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:38.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:38.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:38.826 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:38.863 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:38.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:38.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:38.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:38 smithi098 ceph-mon[97580]: pgmap v398: 89 pgs: 89 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 806 op/s 2024-09-21T10:07:38.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:38 smithi098 ceph-mon[100914]: pgmap v398: 89 pgs: 89 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 806 op/s 2024-09-21T10:07:38.965 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:39.013 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:39.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:39.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:39.116 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:39.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:38 smithi196 ceph-mon[84784]: pgmap v398: 89 pgs: 89 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 806 op/s 2024-09-21T10:07:39.149 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:39.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:39.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:39.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:39.282 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:39.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:39.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:39.384 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:39.422 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:39.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:39.434 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:39.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:39.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:39.577 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:39.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:39.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:39.679 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:39.699 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:39.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:39.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:39.801 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:39.835 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:39.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:39.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:39.937 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:39.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:39.978 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:39.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:40.079 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:40.113 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:40.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:40.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:40.215 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:40.251 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:40.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:40.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:40.352 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:40.388 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:40.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:40.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:40.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:40.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:40.525 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:40.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:40.626 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:40.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:07:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:07:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:07:40.663 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:40.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:40.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:40.765 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:40.793 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:40.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:40.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:40.895 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:40.933 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:40.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:40.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:41.034 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:41.075 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:41.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:41.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:41.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:40 smithi098 ceph-mon[97580]: pgmap v399: 89 pgs: 89 active+clean; 1.4 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 808 op/s 2024-09-21T10:07:41.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:07:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:07:40] "GET /metrics HTTP/1.1" 200 36446 "" "Prometheus/2.51.0" 2024-09-21T10:07:41.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:40 smithi098 ceph-mon[100914]: pgmap v399: 89 pgs: 89 active+clean; 1.4 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 808 op/s 2024-09-21T10:07:41.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:40 smithi196 ceph-mon[84784]: pgmap v399: 89 pgs: 89 active+clean; 1.4 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 808 op/s 2024-09-21T10:07:41.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:41.210 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:41.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:41.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:41.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:41.351 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:41.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:41.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:41.457 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:41.498 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:41.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:41.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:41.600 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:41.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:41.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:41.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:41.768 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:41.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:41.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:41.869 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:41.900 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:41.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:41.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:42.002 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:42.032 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:42.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:42.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:42.134 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:42.180 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:42.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:42.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:42.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:42.321 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:42.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:42.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:42.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:42.465 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:42.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:42.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:42.537 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:07:42.537 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:07:42.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:42.575 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:07:42.575 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:07:42.612 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:07:42.612 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:07:42.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:42.617 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:42.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:42.645 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:07:42.645 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:07:42.677 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:07:42.677 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:07:42.712 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:07:42.713 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:07:42.718 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:42.747 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:07:42.747 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:07:42.748 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:42.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:42.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:42.787 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.5 primary-affinity to 0 (802) 2024-09-21T10:07:42.790 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:07:42.790 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:07:42.850 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:42.866 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:42 smithi098 ceph-mon[97580]: pgmap v400: 89 pgs: 89 active+clean; 1.3 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 823 op/s 2024-09-21T10:07:42.866 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:42 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1313371157' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:42.866 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:42 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3005124951' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.0} : dispatch 2024-09-21T10:07:42.867 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:42 smithi098 ceph-mon[100914]: pgmap v400: 89 pgs: 89 active+clean; 1.3 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 823 op/s 2024-09-21T10:07:42.867 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:42 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1313371157' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:42.867 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:42 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3005124951' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.0} : dispatch 2024-09-21T10:07:42.880 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:42.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:42.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:42.981 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:43.003 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:43.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:43.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:43.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:43.126 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:43.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:43.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:42 smithi196 ceph-mon[84784]: pgmap v400: 89 pgs: 89 active+clean; 1.3 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 823 op/s 2024-09-21T10:07:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:42 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1313371157' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:42 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3005124951' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.0} : dispatch 2024-09-21T10:07:43.228 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:43.250 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:43.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:43.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:43.351 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:43.373 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:43.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:43.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:43.475 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:43.496 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:43.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:43.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:43.598 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:43.622 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:43.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:43.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:43.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:43.746 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:43.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:43.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:43.847 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:43.877 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:43.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:43.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:43.978 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:44.006 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:44.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:44.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:44.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:43 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3005124951' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-09-21T10:07:44.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:43 smithi098 ceph-mon[97580]: osdmap e148: 8 total, 8 up, 8 in 2024-09-21T10:07:44.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:43 smithi098 ceph-mon[97580]: osdmap e149: 8 total, 8 up, 8 in 2024-09-21T10:07:44.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:43 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3005124951' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-09-21T10:07:44.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:43 smithi098 ceph-mon[100914]: osdmap e148: 8 total, 8 up, 8 in 2024-09-21T10:07:44.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:43 smithi098 ceph-mon[100914]: osdmap e149: 8 total, 8 up, 8 in 2024-09-21T10:07:44.108 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:44.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:43 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3005124951' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-09-21T10:07:44.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:43 smithi196 ceph-mon[84784]: osdmap e148: 8 total, 8 up, 8 in 2024-09-21T10:07:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:43 smithi196 ceph-mon[84784]: osdmap e149: 8 total, 8 up, 8 in 2024-09-21T10:07:44.144 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:44.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:44.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:44.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:44.273 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:44.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:44.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:44.375 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:44.413 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:44.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:44.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:44.515 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:44.554 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:44.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:44.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:44.656 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:44.688 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:44.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:44.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:44.789 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:44.828 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:44.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:44.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:44.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:44.960 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:44.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:44.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:45.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:45.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:44 smithi098 ceph-mon[97580]: pgmap v402: 89 pgs: 89 active+clean; 1.2 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 835 op/s 2024-09-21T10:07:45.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:44 smithi098 ceph-mon[100914]: pgmap v402: 89 pgs: 89 active+clean; 1.2 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 835 op/s 2024-09-21T10:07:45.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:45.099 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:45.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:45.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:44 smithi196 ceph-mon[84784]: pgmap v402: 89 pgs: 89 active+clean; 1.2 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 835 op/s 2024-09-21T10:07:45.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:45.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:45.239 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:45.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:45.340 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:45.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:45.379 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:45.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:45.480 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:45.518 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:45.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:45.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:45.620 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:45.650 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:45.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:45.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:45.752 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:45.791 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:45.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:45.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:45.893 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:45.921 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:45.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:45.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:46.023 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:46.063 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:46.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:46.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:46.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:45 smithi196 ceph-mon[84784]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T10:07:46.165 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:46.177 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:45 smithi098 ceph-mon[97580]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T10:07:46.178 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:45 smithi098 ceph-mon[100914]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T10:07:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:46.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:46.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:46.310 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:46.344 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:46.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:46.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:46.446 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:46.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:46.477 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:46.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:46.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:46.608 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:46.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:46.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:46.648 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:07:46.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:46.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:46.776 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:46.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:46.849 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:07:46.849 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:07:46.877 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:46.908 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:46.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:46.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:47.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:47.043 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:47.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:47.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:47.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:46 smithi098 ceph-mon[97580]: pgmap v404: 89 pgs: 7 peering, 82 active+clean; 1.1 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 970 op/s 2024-09-21T10:07:47.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:46 smithi098 ceph-mon[100914]: pgmap v404: 89 pgs: 7 peering, 82 active+clean; 1.1 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 970 op/s 2024-09-21T10:07:47.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:46 smithi196 ceph-mon[84784]: pgmap v404: 89 pgs: 7 peering, 82 active+clean; 1.1 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 970 op/s 2024-09-21T10:07:47.145 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:47.179 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:47.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:47.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:47.281 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:47.315 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:47.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:47.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:47.417 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:47.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:47.467 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:47.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:47.568 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:47.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:47.611 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:47.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:47.712 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:47.741 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:47.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:47.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:47.844 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:47.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:47.877 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:47.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:47.978 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:48.010 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:48.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:48.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:48.100 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:07:48.111 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:48.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:48.194 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:48.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:48.295 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:48.327 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:48.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:48.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:48.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:48.455 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:48.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:48.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:48.557 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:48.606 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:48.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:48.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:48.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:48.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:48.758 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:48.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:48.849 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:48.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:48.867 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:48.870 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:07:48.870 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:07:48.870 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-21T10:07:48.871 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd out 4 2024-09-21T10:07:48.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:48.887 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:48.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:48.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:48 smithi098 ceph-mon[97580]: pgmap v405: 89 pgs: 7 peering, 82 active+clean; 1.0 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 657 op/s 2024-09-21T10:07:48.935 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:48 smithi098 ceph-mon[100914]: pgmap v405: 89 pgs: 7 peering, 82 active+clean; 1.0 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 657 op/s 2024-09-21T10:07:48.988 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:49.043 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:49.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:49.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:49.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:48 smithi196 ceph-mon[84784]: pgmap v405: 89 pgs: 7 peering, 82 active+clean; 1.0 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 657 op/s 2024-09-21T10:07:49.145 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:49.176 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:49.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:49.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:49.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:49.320 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:49.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:49.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:49.422 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:49.455 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:49.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:49.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:49.557 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:49.588 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:49.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:49.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:49.690 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:49.722 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:49.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:49.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:49.824 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:49.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:49.860 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:49.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:49.870 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:07:49.946 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:49 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1733736774' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:07:49.946 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:49 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1733736774' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:07:49.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:49.998 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:49.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:49.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:50.100 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:50.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:49 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1733736774' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:07:50.143 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:50.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:50.245 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:50.280 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:50.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:50.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:50.382 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:50.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:50.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:50.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:50.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:50.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:50.550 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:50.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:50.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:07:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:07:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:07:50.651 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:50.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:50.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:50.788 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:50.814 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:50.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:50.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:50.916 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:50.958 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:50.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:50.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:51.060 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:51.088 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:51.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:51.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:51.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:07:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:07:50] "GET /metrics HTTP/1.1" 200 36446 "" "Prometheus/2.51.0" 2024-09-21T10:07:51.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:50 smithi098 ceph-mon[100914]: pgmap v406: 89 pgs: 89 active+clean; 907 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 809 op/s 2024-09-21T10:07:51.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:50 smithi098 ceph-mon[100914]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-21T10:07:51.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:50 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:07:51.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:50 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1733736774' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:07:51.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:50 smithi098 ceph-mon[100914]: osdmap e150: 8 total, 8 up, 8 in 2024-09-21T10:07:51.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:50 smithi098 ceph-mon[97580]: pgmap v406: 89 pgs: 89 active+clean; 907 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 809 op/s 2024-09-21T10:07:51.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:50 smithi098 ceph-mon[97580]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-21T10:07:51.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:50 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:07:51.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:50 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1733736774' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:07:51.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:50 smithi098 ceph-mon[97580]: osdmap e150: 8 total, 8 up, 8 in 2024-09-21T10:07:51.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:50 smithi196 ceph-mon[84784]: pgmap v406: 89 pgs: 89 active+clean; 907 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 809 op/s 2024-09-21T10:07:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:50 smithi196 ceph-mon[84784]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-21T10:07:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:50 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:07:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:50 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1733736774' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:07:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:50 smithi196 ceph-mon[84784]: osdmap e150: 8 total, 8 up, 8 in 2024-09-21T10:07:51.189 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:51.220 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:51.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:51.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:51.322 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:51.355 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:51.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:51.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:51.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:51.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:51.501 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:51.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:51.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:51.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:51.653 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:51.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:51.754 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:51.783 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:51.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:51.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:51.885 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:51.923 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:51.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:51.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:52.025 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:52.069 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:52.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:52.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:52.170 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:52.204 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:52.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:52.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:52.306 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:52.338 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:52.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:52.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:52.440 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:52.471 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:52.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:52.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:52.573 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:52.610 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:52.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:52.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:52.712 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:52.747 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:52.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:52.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:52.849 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:52.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:52.879 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:52.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:52.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:53.008 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:53.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:53.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:52 smithi098 ceph-mon[97580]: pgmap v408: 89 pgs: 89 active+clean; 824 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 871 op/s 2024-09-21T10:07:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:53.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:52 smithi098 ceph-mon[100914]: pgmap v408: 89 pgs: 89 active+clean; 824 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 871 op/s 2024-09-21T10:07:53.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:53.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:53.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:52 smithi196 ceph-mon[84784]: pgmap v408: 89 pgs: 89 active+clean; 824 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 871 op/s 2024-09-21T10:07:53.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:53.147 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:53.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:53.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:53.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:53.285 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:53.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:53.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:53.387 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:53.423 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:53.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:53.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:53.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:53.564 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:53.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:53.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:53.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:53.694 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:53.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:53.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:53.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:53.842 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:53.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:53.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:53.945 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:53.979 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:53.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:53.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:54.081 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:54.161 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:54.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:54.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:54.263 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:54.299 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:54.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:54.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:54.381 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:07:54.382 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:07:54.401 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:54.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:54.450 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:54.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:54.551 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:54.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:54.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:54.670 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:07:54.671 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:07:54.692 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:54.723 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:07:54.723 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:07:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:54.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:54.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:54.753 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:07:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:07:54.784 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:07:54.785 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:07:54.814 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:07:54.815 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:07:54.834 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:54.849 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:07:54.849 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:07:54.856 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:54.874 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:54.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:54.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:54.882 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:07:54.882 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:07:54.916 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:07:54.916 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:07:54.976 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:55.011 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:55.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:55.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:55.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:54 smithi098 ceph-mon[97580]: pgmap v409: 89 pgs: 89 active+clean; 762 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 667 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:07:55.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:54 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:55.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:54 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:55.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:54 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:55.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:54 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:07:55.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:54 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:55.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:54 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/575293935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:55.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:54 smithi098 ceph-mon[100914]: pgmap v409: 89 pgs: 89 active+clean; 762 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 667 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:07:55.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:54 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:55.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:54 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:55.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:54 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:55.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:54 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:07:55.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:54 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:55.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:54 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/575293935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:55.113 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:54 smithi196 ceph-mon[84784]: pgmap v409: 89 pgs: 89 active+clean; 762 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 667 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:07:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:54 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:54 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:54 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:54 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:07:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:54 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:54 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/575293935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:55.146 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:55.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:55.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:55.187 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:07:55.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:55.286 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:55.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:55.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:55.387 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:55.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:55.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:55.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:55.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:07:55.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:55.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:55.561 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:55.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:55.661 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:55.693 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:55.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:55.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:55.794 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:55.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:55.822 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:55.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:55.923 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:55.961 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:55.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:55.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:56.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:56.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:56.099 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:56.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:56.200 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:56.223 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:56.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:56.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:56.325 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:56.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:56.360 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:56.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:56.460 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:56.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:56.497 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:56.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:56.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:56.637 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:56.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:56.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:56.739 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:56.779 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:56.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:56.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:56.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:56.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:56.913 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:56.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:57.014 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:57.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:57.051 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:57.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:57.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:56 smithi098 ceph-mon[97580]: pgmap v410: 89 pgs: 89 active+clean; 551 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:07:57.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:56 smithi098 ceph-mon[100914]: pgmap v410: 89 pgs: 89 active+clean; 551 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:07:57.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:56 smithi196 ceph-mon[84784]: pgmap v410: 89 pgs: 89 active+clean; 551 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:07:57.153 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:57.188 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:57.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:57.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:57.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:57.322 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:57.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:57.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:57.424 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:57.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:57.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:57.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:57.595 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:57.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:57.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:57.697 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:57.717 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:57.722 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:57.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:57.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:57.824 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:57.866 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:57.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:57.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:57.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:58.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:58.003 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:58.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:58.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:58.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:58.135 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:58.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:58.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:58.275 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:58.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:58.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:58.377 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:58.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:58.421 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:58.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:58.522 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:58.554 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:58.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:58.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:58.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:58 smithi098 ceph-mon[97580]: pgmap v411: 89 pgs: 89 active+clean; 534 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 854 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:07:58.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:58 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:58.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:58 smithi098 ceph-mon[100914]: pgmap v411: 89 pgs: 89 active+clean; 534 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 854 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:07:58.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:58 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:58.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:58 smithi196 ceph-mon[84784]: pgmap v411: 89 pgs: 89 active+clean; 534 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 854 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:07:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:58 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:07:58.657 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:58.702 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:58.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:58.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:58.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:58.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:58.842 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:58.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:58.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:07:58.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:07:58.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:58.981 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:58.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:07:59.082 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:59.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:59.127 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:59.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:59.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:59 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a[97576]: 2024-09-21T10:07:59.161+0000 7f65561f9640 -1 mon.a@0(leader).osd e150 definitely_dead 0 2024-09-21T10:07:59.228 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:59.268 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:59.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:59.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:59.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:59.373 INFO:teuthology.orchestra.run.smithi098.stderr:marked out osd.4. 2024-09-21T10:07:59.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:59.404 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:59.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:59.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:59.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:59.529 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:59.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:59.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:59 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2668680805' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:07:59.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:59 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:07:59.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:07:59 smithi098 ceph-mon[97580]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T10:07:59.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:59 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2668680805' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:07:59.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:59 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:07:59.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:07:59 smithi098 ceph-mon[100914]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T10:07:59.630 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:59 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2668680805' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:07:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:59 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:07:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:07:59 smithi196 ceph-mon[84784]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T10:07:59.651 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:59.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:59.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:59.753 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:59.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:59.777 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:07:59.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:59.877 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:59.899 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:07:59.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:59.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:00.001 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:00.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:00.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:00.125 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:00.148 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:00.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:00.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:00.250 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:00.272 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:00.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:00.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:00.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:00.405 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:00.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:00.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:00.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:00.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:00.531 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:00.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:00.632 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:00 smithi196 ceph-mon[84784]: pgmap v412: 89 pgs: 89 active+clean; 365 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 867 op/s 2024-09-21T10:08:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:00 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T10:08:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:00 smithi196 ceph-mon[84784]: osdmap e151: 8 total, 8 up, 7 in 2024-09-21T10:08:00.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:08:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:08:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:08:00.654 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:00.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:00.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:00.657 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:00 smithi098 ceph-mon[97580]: pgmap v412: 89 pgs: 89 active+clean; 365 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 867 op/s 2024-09-21T10:08:00.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:00 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T10:08:00.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:00 smithi098 ceph-mon[97580]: osdmap e151: 8 total, 8 up, 7 in 2024-09-21T10:08:00.658 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:00 smithi098 ceph-mon[100914]: pgmap v412: 89 pgs: 89 active+clean; 365 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 867 op/s 2024-09-21T10:08:00.658 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:00 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T10:08:00.658 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:00 smithi098 ceph-mon[100914]: osdmap e151: 8 total, 8 up, 7 in 2024-09-21T10:08:00.756 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:00.781 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:00.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:00.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:00.884 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:00.904 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:00.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:00.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:01.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:01.032 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:01.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:01.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:01.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:08:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:08:00] "GET /metrics HTTP/1.1" 200 36448 "" "Prometheus/2.51.0" 2024-09-21T10:08:01.133 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:01.159 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:01.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:01.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:01.261 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:01.286 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:01.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:01.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:01.388 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:01.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:01.442 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:01.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:01.481 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:01 smithi098 ceph-mon[97580]: osdmap e152: 8 total, 8 up, 7 in 2024-09-21T10:08:01.481 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:01 smithi098 ceph-mon[97580]: osdmap e153: 8 total, 8 up, 7 in 2024-09-21T10:08:01.482 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:01 smithi098 ceph-mon[100914]: osdmap e152: 8 total, 8 up, 7 in 2024-09-21T10:08:01.482 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:01 smithi098 ceph-mon[100914]: osdmap e153: 8 total, 8 up, 7 in 2024-09-21T10:08:01.542 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:01.565 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:01.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:01.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:01.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:01 smithi196 ceph-mon[84784]: osdmap e152: 8 total, 8 up, 7 in 2024-09-21T10:08:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:01 smithi196 ceph-mon[84784]: osdmap e153: 8 total, 8 up, 7 in 2024-09-21T10:08:01.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:01.709 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:01.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:01.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:01.811 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:01.842 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:01.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:01.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:01.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:01.972 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:01.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:01.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:02.074 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:02.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:02.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:02.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:02.264 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:02.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:02.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:02.366 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:02.396 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:02.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:02.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:02.494 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:02 smithi098 ceph-mon[100914]: pgmap v415: 89 pgs: 1 peering, 88 active+clean; 284 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 955 op/s 2024-09-21T10:08:02.494 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:02 smithi098 ceph-mon[100914]: 5.9 starting backfill to osd.1 from (0'0,0'0] MAX to 150'16658 2024-09-21T10:08:02.494 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:02 smithi098 ceph-mon[100914]: 5.1 starting backfill to osd.0 from (0'0,0'0] MAX to 150'16524 2024-09-21T10:08:02.494 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:02 smithi098 ceph-mon[100914]: 5.0 starting backfill to osd.0 from (0'0,0'0] MAX to 150'16422 2024-09-21T10:08:02.494 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:02 smithi098 ceph-mon[100914]: 5.a starting backfill to osd.0 from (0'0,0'0] MAX to 150'16662 2024-09-21T10:08:02.494 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:02 smithi098 ceph-mon[100914]: 5.f starting backfill to osd.2 from (0'0,0'0] MAX to 150'16829 2024-09-21T10:08:02.494 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:02 smithi098 ceph-mon[100914]: 5.e starting backfill to osd.2 from (0'0,0'0] MAX to 150'17023 2024-09-21T10:08:02.494 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:02 smithi098 ceph-mon[100914]: 5.c starting backfill to osd.6 from (0'0,0'0] MAX to 150'16555 2024-09-21T10:08:02.495 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:02 smithi098 ceph-mon[100914]: 5.5 starting backfill to osd.6 from (0'0,0'0] MAX to 150'17024 2024-09-21T10:08:02.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:02 smithi098 ceph-mon[97580]: pgmap v415: 89 pgs: 1 peering, 88 active+clean; 284 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 955 op/s 2024-09-21T10:08:02.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:02 smithi098 ceph-mon[97580]: 5.9 starting backfill to osd.1 from (0'0,0'0] MAX to 150'16658 2024-09-21T10:08:02.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:02 smithi098 ceph-mon[97580]: 5.1 starting backfill to osd.0 from (0'0,0'0] MAX to 150'16524 2024-09-21T10:08:02.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:02 smithi098 ceph-mon[97580]: 5.0 starting backfill to osd.0 from (0'0,0'0] MAX to 150'16422 2024-09-21T10:08:02.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:02 smithi098 ceph-mon[97580]: 5.a starting backfill to osd.0 from (0'0,0'0] MAX to 150'16662 2024-09-21T10:08:02.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:02 smithi098 ceph-mon[97580]: 5.f starting backfill to osd.2 from (0'0,0'0] MAX to 150'16829 2024-09-21T10:08:02.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:02 smithi098 ceph-mon[97580]: 5.e starting backfill to osd.2 from (0'0,0'0] MAX to 150'17023 2024-09-21T10:08:02.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:02 smithi098 ceph-mon[97580]: 5.c starting backfill to osd.6 from (0'0,0'0] MAX to 150'16555 2024-09-21T10:08:02.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:02 smithi098 ceph-mon[97580]: 5.5 starting backfill to osd.6 from (0'0,0'0] MAX to 150'17024 2024-09-21T10:08:02.498 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:02.526 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:02.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:02.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:02.628 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:02.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:02 smithi196 ceph-mon[84784]: pgmap v415: 89 pgs: 1 peering, 88 active+clean; 284 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 955 op/s 2024-09-21T10:08:02.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:02 smithi196 ceph-mon[84784]: 5.9 starting backfill to osd.1 from (0'0,0'0] MAX to 150'16658 2024-09-21T10:08:02.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:02 smithi196 ceph-mon[84784]: 5.1 starting backfill to osd.0 from (0'0,0'0] MAX to 150'16524 2024-09-21T10:08:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:02 smithi196 ceph-mon[84784]: 5.0 starting backfill to osd.0 from (0'0,0'0] MAX to 150'16422 2024-09-21T10:08:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:02 smithi196 ceph-mon[84784]: 5.a starting backfill to osd.0 from (0'0,0'0] MAX to 150'16662 2024-09-21T10:08:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:02 smithi196 ceph-mon[84784]: 5.f starting backfill to osd.2 from (0'0,0'0] MAX to 150'16829 2024-09-21T10:08:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:02 smithi196 ceph-mon[84784]: 5.e starting backfill to osd.2 from (0'0,0'0] MAX to 150'17023 2024-09-21T10:08:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:02 smithi196 ceph-mon[84784]: 5.c starting backfill to osd.6 from (0'0,0'0] MAX to 150'16555 2024-09-21T10:08:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:02 smithi196 ceph-mon[84784]: 5.5 starting backfill to osd.6 from (0'0,0'0] MAX to 150'17024 2024-09-21T10:08:02.663 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:02.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:02.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:02.764 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:02.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:02.797 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:02.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:02.898 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:02.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:02.927 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:02.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:03.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:03.064 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:03.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:03.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:03.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:03.196 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:03.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:03.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:03.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:03.346 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:03.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:03.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:08:03.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:03.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:03.493 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:03.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:03.594 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:03.621 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:03.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:03.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:03.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:03.745 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:03.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:03.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:03.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:03 smithi098 ceph-mon[97580]: osdmap e154: 8 total, 8 up, 7 in 2024-09-21T10:08:03.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:03 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3910923218' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:03.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:03 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:03.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:03 smithi098 ceph-mon[100914]: osdmap e154: 8 total, 8 up, 7 in 2024-09-21T10:08:03.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:03 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3910923218' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:03.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:03 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:03.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:03.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:03.870 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:03.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:03 smithi196 ceph-mon[84784]: osdmap e154: 8 total, 8 up, 7 in 2024-09-21T10:08:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:03 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3910923218' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:03 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:03.971 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:03.994 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:03.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:03.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:04.096 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:04.120 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:04.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:04.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:04.222 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:04.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:04.244 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:04.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:04.345 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:04.366 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:04.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:04.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:04.468 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:04.496 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:04.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:04.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:04.598 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:04.619 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:04.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:04.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:04.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:04.744 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:04.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:04.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:04.846 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:04.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:04 smithi098 ceph-mon[100914]: pgmap v418: 89 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 1 peering, 83 active+clean; 234 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 523 op/s; 991/11094 objects misplaced (8.933%); 128 KiB/s, 2 objects/s recovering 2024-09-21T10:08:04.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:04 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:08:04.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:04 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:08:04.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:04 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:08:04.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:04 smithi098 ceph-mon[100914]: osdmap e155: 8 total, 8 up, 7 in 2024-09-21T10:08:04.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:04 smithi098 ceph-mon[97580]: pgmap v418: 89 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 1 peering, 83 active+clean; 234 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 523 op/s; 991/11094 objects misplaced (8.933%); 128 KiB/s, 2 objects/s recovering 2024-09-21T10:08:04.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:04 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:08:04.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:04 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:08:04.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:04 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:08:04.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:04 smithi098 ceph-mon[97580]: osdmap e155: 8 total, 8 up, 7 in 2024-09-21T10:08:04.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:04.868 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:04.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:04 smithi196 ceph-mon[84784]: pgmap v418: 89 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 1 peering, 83 active+clean; 234 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 523 op/s; 991/11094 objects misplaced (8.933%); 128 KiB/s, 2 objects/s recovering 2024-09-21T10:08:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:04 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:08:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:04 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:08:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:04 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:08:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:04 smithi196 ceph-mon[84784]: osdmap e155: 8 total, 8 up, 7 in 2024-09-21T10:08:04.969 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:04.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:04.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:05.093 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:05.115 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:05.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:05.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:05.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:05.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:05.240 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:05.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:05.341 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:05.363 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:05.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:05.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:05.465 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:05.498 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:05.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:05.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:05.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:05 smithi098 ceph-mon[97580]: osdmap e156: 8 total, 8 up, 7 in 2024-09-21T10:08:05.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:05 smithi098 ceph-mon[100914]: osdmap e156: 8 total, 8 up, 7 in 2024-09-21T10:08:05.600 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:05.639 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:05.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:05.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:05.741 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:05.772 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:05.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:05.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:05.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:05.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:05 smithi196 ceph-mon[84784]: osdmap e156: 8 total, 8 up, 7 in 2024-09-21T10:08:05.900 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:05.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:05.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:06.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:06.041 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:06.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:06.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:06.143 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:06.178 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:06.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:06.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:06.212 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:08:06.212 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:08:06.213 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T10:08:06.213 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2024-09-21T10:08:06.213 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 1 1 2024-09-21T10:08:06.281 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:06.306 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:06.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:06.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:06.408 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:06.431 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:06.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:06.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:06.470 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:08:06.470 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-21T10:08:06.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-21T10:08:06.470 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:08:06.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.470 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:35209 5.b 5:d050c117:::benchmark_data_smithi098_135187_object35208:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:08:06.470 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.630227+0000", 2024-09-21T10:08:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 244.83556840200001, 2024-09-21T10:08:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.12529087799999999, 2024-09-21T10:08:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:06.471 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35209 2024-09-21T10:08:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.630227+0000", 2024-09-21T10:08:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:06.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.630227+0000", 2024-09-21T10:08:06.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:06.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.630227+0000", 2024-09-21T10:08:06.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5599999999999995e-07 2024-09-21T10:08:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.630247+0000", 2024-09-21T10:08:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.946e-05 2024-09-21T10:08:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.630254+0000", 2024-09-21T10:08:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4900000000000003e-06 2024-09-21T10:08:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:06.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.630259+0000", 2024-09-21T10:08:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1320000000000002e-06 2024-09-21T10:08:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.631477+0000", 2024-09-21T10:08:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001217641 2024-09-21T10:08:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.631596+0000", 2024-09-21T10:08:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000118804 2024-09-21T10:08:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:08:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.631725+0000", 2024-09-21T10:08:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012932699999999999 2024-09-21T10:08:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635243+0000", 2024-09-21T10:08:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035178549999999999 2024-09-21T10:08:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635565+0000", 2024-09-21T10:08:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000322412 2024-09-21T10:08:06.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.755430+0000", 2024-09-21T10:08:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.119864811 2024-09-21T10:08:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.755480+0000", 2024-09-21T10:08:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9493000000000002e-05 2024-09-21T10:08:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:06.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.755518+0000", 2024-09-21T10:08:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7897e-05 2024-09-21T10:08:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:35224 5.a 5:51fa5009:::benchmark_data_smithi098_135187_object35223:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:08:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.635579+0000", 2024-09-21T10:08:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 244.830215791, 2024-09-21T10:08:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11871541400000001, 2024-09-21T10:08:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:06.478 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35224 2024-09-21T10:08:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635579+0000", 2024-09-21T10:08:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635579+0000", 2024-09-21T10:08:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635581+0000", 2024-09-21T10:08:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3230000000000001e-06 2024-09-21T10:08:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635602+0000", 2024-09-21T10:08:06.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1001000000000001e-05 2024-09-21T10:08:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635610+0000", 2024-09-21T10:08:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1410000000000005e-06 2024-09-21T10:08:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635616+0000", 2024-09-21T10:08:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2570000000000004e-06 2024-09-21T10:08:06.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635659+0000", 2024-09-21T10:08:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2892999999999997e-05 2024-09-21T10:08:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635779+0000", 2024-09-21T10:08:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000120025 2024-09-21T10:08:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:08:06.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635905+0000", 2024-09-21T10:08:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000126382 2024-09-21T10:08:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754075+0000", 2024-09-21T10:08:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.118169815 2024-09-21T10:08:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754127+0000", 2024-09-21T10:08:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2080999999999998e-05 2024-09-21T10:08:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754200+0000", 2024-09-21T10:08:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3053999999999995e-05 2024-09-21T10:08:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754253+0000", 2024-09-21T10:08:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2377999999999999e-05 2024-09-21T10:08:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754295+0000", 2024-09-21T10:08:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2064000000000002e-05 2024-09-21T10:08:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:35226 5.a 5:577c758a:::benchmark_data_smithi098_135187_object35225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:08:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.635769+0000", 2024-09-21T10:08:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 244.83002589899999, 2024-09-21T10:08:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11862623799999999, 2024-09-21T10:08:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:06.485 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:06.486 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:06.486 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:06.486 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:06.486 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35226 2024-09-21T10:08:06.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635769+0000", 2024-09-21T10:08:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635769+0000", 2024-09-21T10:08:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:06.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635770+0000", 2024-09-21T10:08:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8199999999999999e-07 2024-09-21T10:08:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635788+0000", 2024-09-21T10:08:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8505999999999999e-05 2024-09-21T10:08:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635799+0000", 2024-09-21T10:08:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0173000000000001e-05 2024-09-21T10:08:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635805+0000", 2024-09-21T10:08:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2299999999999996e-06 2024-09-21T10:08:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637399+0000", 2024-09-21T10:08:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0015946249999999999 2024-09-21T10:08:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637531+0000", 2024-09-21T10:08:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013183800000000001 2024-09-21T10:08:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:08:06.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637674+0000", 2024-09-21T10:08:06.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014236000000000001 2024-09-21T10:08:06.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:06.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754081+0000", 2024-09-21T10:08:06.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.116406883 2024-09-21T10:08:06.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754168+0000", 2024-09-21T10:08:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7264999999999999e-05 2024-09-21T10:08:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754327+0000", 2024-09-21T10:08:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015965599999999999 2024-09-21T10:08:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754367+0000", 2024-09-21T10:08:06.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9221999999999998e-05 2024-09-21T10:08:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754395+0000", 2024-09-21T10:08:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8697999999999999e-05 2024-09-21T10:08:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.44894.0:35224 5.a e110/105 ondisk, result = 0)", 2024-09-21T10:08:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.637962+0000", 2024-09-21T10:08:06.493 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 244.827833365, 2024-09-21T10:08:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.116172572, 2024-09-21T10:08:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:08:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637962+0000", 2024-09-21T10:08:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637962+0000", 2024-09-21T10:08:06.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637963+0000", 2024-09-21T10:08:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1179999999999999e-06 2024-09-21T10:08:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637968+0000", 2024-09-21T10:08:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8910000000000003e-06 2024-09-21T10:08:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637976+0000", 2024-09-21T10:08:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6289999999999999e-06 2024-09-21T10:08:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637982+0000", 2024-09-21T10:08:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2299999999999999e-06 2024-09-21T10:08:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754114+0000", 2024-09-21T10:08:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.116132108 2024-09-21T10:08:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:06.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754122+0000", 2024-09-21T10:08:06.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1750000000000005e-06 2024-09-21T10:08:06.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:06.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754134+0000", 2024-09-21T10:08:06.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2421e-05 2024-09-21T10:08:06.498 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.498 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:06.498 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.498 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:35229 5.b 5:d08d135d:::benchmark_data_smithi098_135187_object35228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:08:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.638050+0000", 2024-09-21T10:08:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 244.82774540599999, 2024-09-21T10:08:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.117567405, 2024-09-21T10:08:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35229 2024-09-21T10:08:06.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638050+0000", 2024-09-21T10:08:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638050+0000", 2024-09-21T10:08:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.500 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638051+0000", 2024-09-21T10:08:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.502e-06 2024-09-21T10:08:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638072+0000", 2024-09-21T10:08:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0976e-05 2024-09-21T10:08:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638085+0000", 2024-09-21T10:08:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2988e-05 2024-09-21T10:08:06.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638092+0000", 2024-09-21T10:08:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1709999999999996e-06 2024-09-21T10:08:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638154+0000", 2024-09-21T10:08:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1567000000000002e-05 2024-09-21T10:08:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638258+0000", 2024-09-21T10:08:06.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000103959 2024-09-21T10:08:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:08:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638372+0000", 2024-09-21T10:08:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000113888 2024-09-21T10:08:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754684+0000", 2024-09-21T10:08:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11631245 2024-09-21T10:08:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754695+0000", 2024-09-21T10:08:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0969e-05 2024-09-21T10:08:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.755551+0000", 2024-09-21T10:08:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00085551900000000001 2024-09-21T10:08:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.755589+0000", 2024-09-21T10:08:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8213000000000002e-05 2024-09-21T10:08:06.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.755617+0000", 2024-09-21T10:08:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8203000000000001e-05 2024-09-21T10:08:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:35231 5.d 5:bb22eb7b:::benchmark_data_smithi098_135187_object35230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:08:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.638970+0000", 2024-09-21T10:08:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 244.82682502500001, 2024-09-21T10:08:06.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.12243952, 2024-09-21T10:08:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35231 2024-09-21T10:08:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:06.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638970+0000", 2024-09-21T10:08:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638970+0000", 2024-09-21T10:08:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638971+0000", 2024-09-21T10:08:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1030000000000001e-06 2024-09-21T10:08:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.639022+0000", 2024-09-21T10:08:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0949999999999998e-05 2024-09-21T10:08:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754209+0000", 2024-09-21T10:08:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11518701200000001 2024-09-21T10:08:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:06.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754219+0000", 2024-09-21T10:08:06.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8430000000000005e-06 2024-09-21T10:08:06.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754669+0000", 2024-09-21T10:08:06.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044998299999999999 2024-09-21T10:08:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754792+0000", 2024-09-21T10:08:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012265999999999999 2024-09-21T10:08:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,7", 2024-09-21T10:08:06.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754929+0000", 2024-09-21T10:08:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013770199999999999 2024-09-21T10:08:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.761225+0000", 2024-09-21T10:08:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.006295429 2024-09-21T10:08:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.761275+0000", 2024-09-21T10:08:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9883999999999999e-05 2024-09-21T10:08:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.761309+0000", 2024-09-21T10:08:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4767000000000001e-05 2024-09-21T10:08:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.761355+0000", 2024-09-21T10:08:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5918000000000003e-05 2024-09-21T10:08:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:06.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.761410+0000", 2024-09-21T10:08:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4268999999999998e-05 2024-09-21T10:08:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.44894.0:35226 5.a e110/105 ondisk, result = 0)", 2024-09-21T10:08:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.639410+0000", 2024-09-21T10:08:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 244.82638499199999, 2024-09-21T10:08:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.114763873, 2024-09-21T10:08:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:06.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:08:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.639410+0000", 2024-09-21T10:08:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.639410+0000", 2024-09-21T10:08:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.639411+0000", 2024-09-21T10:08:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-21T10:08:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:06.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.639481+0000", 2024-09-21T10:08:06.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0146000000000002e-05 2024-09-21T10:08:06.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:06.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754050+0000", 2024-09-21T10:08:06.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11456915300000001 2024-09-21T10:08:06.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:06.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754062+0000", 2024-09-21T10:08:06.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2055000000000001e-05 2024-09-21T10:08:06.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.517 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754158+0000", 2024-09-21T10:08:06.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5880999999999999e-05 2024-09-21T10:08:06.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.517 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:06.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754163+0000", 2024-09-21T10:08:06.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3909999999999997e-06 2024-09-21T10:08:06.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:06.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754174+0000", 2024-09-21T10:08:06.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0548e-05 2024-09-21T10:08:06.518 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.518 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:06.518 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.518 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.44894.0:35229 5.b e110/105 ondisk, result = 0)", 2024-09-21T10:08:06.518 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.640577+0000", 2024-09-21T10:08:06.518 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 244.825218568, 2024-09-21T10:08:06.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.114119283, 2024-09-21T10:08:06.519 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:06.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:06.519 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:08:06.519 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:06.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:06.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.640577+0000", 2024-09-21T10:08:06.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:06.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.640577+0000", 2024-09-21T10:08:06.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:06.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.640577+0000", 2024-09-21T10:08:06.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.51e-07 2024-09-21T10:08:06.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:06.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.640580+0000", 2024-09-21T10:08:06.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1099999999999999e-06 2024-09-21T10:08:06.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:06.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754580+0000", 2024-09-21T10:08:06.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.114000249 2024-09-21T10:08:06.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:06.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754587+0000", 2024-09-21T10:08:06.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3790000000000003e-06 2024-09-21T10:08:06.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754672+0000", 2024-09-21T10:08:06.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4889000000000007e-05 2024-09-21T10:08:06.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:06.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754679+0000", 2024-09-21T10:08:06.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1690000000000003e-06 2024-09-21T10:08:06.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:06.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754696+0000", 2024-09-21T10:08:06.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6935999999999998e-05 2024-09-21T10:08:06.524 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.524 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:06.524 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.524 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:86402 5.8 5:1c1635b7:::benchmark_data_smithi098_135187_object86401:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T10:08:06.524 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:33.146388+0000", 2024-09-21T10:08:06.524 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 213.319406947, 2024-09-21T10:08:06.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.103009368, 2024-09-21T10:08:06.524 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:06.525 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:06.525 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:06.525 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:06.525 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:06.525 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:06.525 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 86402 2024-09-21T10:08:06.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.525 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:06.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:06.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146388+0000", 2024-09-21T10:08:06.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:06.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146388+0000", 2024-09-21T10:08:06.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:06.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146389+0000", 2024-09-21T10:08:06.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5000000000000002e-07 2024-09-21T10:08:06.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:06.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146406+0000", 2024-09-21T10:08:06.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6949e-05 2024-09-21T10:08:06.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:06.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146413+0000", 2024-09-21T10:08:06.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2810000000000003e-06 2024-09-21T10:08:06.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:06.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146417+0000", 2024-09-21T10:08:06.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.262e-06 2024-09-21T10:08:06.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146457+0000", 2024-09-21T10:08:06.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9858999999999999e-05 2024-09-21T10:08:06.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:06.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146533+0000", 2024-09-21T10:08:06.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6143000000000006e-05 2024-09-21T10:08:06.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:08:06.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.146619+0000", 2024-09-21T10:08:06.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5394000000000003e-05 2024-09-21T10:08:06.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.245390+0000", 2024-09-21T10:08:06.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.098770872999999995 2024-09-21T10:08:06.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.245426+0000", 2024-09-21T10:08:06.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6656000000000003e-05 2024-09-21T10:08:06.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:06.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.249306+0000", 2024-09-21T10:08:06.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038799590000000001 2024-09-21T10:08:06.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:06.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.249358+0000", 2024-09-21T10:08:06.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1876999999999998e-05 2024-09-21T10:08:06.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:06.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.249397+0000", 2024-09-21T10:08:06.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9465000000000002e-05 2024-09-21T10:08:06.531 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.531 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:06.531 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.531 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:86411 5.a 5:5f80647e:::benchmark_data_smithi098_135187_object86410:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T10:08:06.531 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:33.151250+0000", 2024-09-21T10:08:06.531 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 213.31454461600001, 2024-09-21T10:08:06.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.10103477399999999, 2024-09-21T10:08:06.531 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:06.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:06.531 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:06.532 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:06.532 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:06.532 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:06.532 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 86411 2024-09-21T10:08:06.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.532 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:06.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:06.532 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.151250+0000", 2024-09-21T10:08:06.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:06.532 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.151250+0000", 2024-09-21T10:08:06.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:06.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.151252+0000", 2024-09-21T10:08:06.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.096e-06 2024-09-21T10:08:06.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:06.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.151272+0000", 2024-09-21T10:08:06.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0937e-05 2024-09-21T10:08:06.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:06.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.239495+0000", 2024-09-21T10:08:06.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.088222574999999998 2024-09-21T10:08:06.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:06.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.239505+0000", 2024-09-21T10:08:06.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9510000000000001e-06 2024-09-21T10:08:06.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240198+0000", 2024-09-21T10:08:06.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00069264100000000002 2024-09-21T10:08:06.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:06.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240307+0000", 2024-09-21T10:08:06.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010927 2024-09-21T10:08:06.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:08:06.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.240433+0000", 2024-09-21T10:08:06.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012565800000000001 2024-09-21T10:08:06.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.251994+0000", 2024-09-21T10:08:06.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011561068000000001 2024-09-21T10:08:06.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.252070+0000", 2024-09-21T10:08:06.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6796000000000001e-05 2024-09-21T10:08:06.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:06.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.252199+0000", 2024-09-21T10:08:06.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012868299999999999 2024-09-21T10:08:06.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:06.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.252243+0000", 2024-09-21T10:08:06.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3560999999999998e-05 2024-09-21T10:08:06.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:06.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:33.252285+0000", 2024-09-21T10:08:06.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2537999999999998e-05 2024-09-21T10:08:06.537 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.537 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:06.537 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.537 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:159909 5.a 5:589715df:::benchmark_data_smithi098_135187_object24271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-21T10:08:06.538 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:05:46.477125+0000", 2024-09-21T10:08:06.538 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 139.98867021699999, 2024-09-21T10:08:06.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.119837549, 2024-09-21T10:08:06.538 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:06.538 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:06.538 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:06.538 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:06.538 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:06.538 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:06.538 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 159909 2024-09-21T10:08:06.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.538 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:06.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:06.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.477125+0000", 2024-09-21T10:08:06.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:06.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.477125+0000", 2024-09-21T10:08:06.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:06.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.477127+0000", 2024-09-21T10:08:06.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.826e-06 2024-09-21T10:08:06.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:06.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.477132+0000", 2024-09-21T10:08:06.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4709999999999998e-06 2024-09-21T10:08:06.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:06.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.477148+0000", 2024-09-21T10:08:06.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5449000000000001e-05 2024-09-21T10:08:06.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:06.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.477155+0000", 2024-09-21T10:08:06.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2869999999999999e-06 2024-09-21T10:08:06.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.478078+0000", 2024-09-21T10:08:06.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00092277099999999996 2024-09-21T10:08:06.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:06.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.478316+0000", 2024-09-21T10:08:06.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023818600000000001 2024-09-21T10:08:06.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:08:06.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.478412+0000", 2024-09-21T10:08:06.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6436999999999998e-05 2024-09-21T10:08:06.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:06.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.491697+0000", 2024-09-21T10:08:06.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013284374 2024-09-21T10:08:06.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.496201+0000", 2024-09-21T10:08:06.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0045046119999999999 2024-09-21T10:08:06.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.543 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.596874+0000", 2024-09-21T10:08:06.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.100672711 2024-09-21T10:08:06.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.543 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:06.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.596922+0000", 2024-09-21T10:08:06.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.774e-05 2024-09-21T10:08:06.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:06.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.596962+0000", 2024-09-21T10:08:06.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0685000000000001e-05 2024-09-21T10:08:06.544 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.544 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:06.544 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.544 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:159911 5.a 5:51b41833:::benchmark_data_smithi098_135187_object24273:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-21T10:08:06.544 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:05:46.481149+0000", 2024-09-21T10:08:06.544 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 139.98464629200001, 2024-09-21T10:08:06.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.115993818, 2024-09-21T10:08:06.544 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:06.544 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:06.545 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:06.545 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:06.545 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:06.545 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:06.545 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 159911 2024-09-21T10:08:06.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.545 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:06.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:06.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.481149+0000", 2024-09-21T10:08:06.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:06.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.481149+0000", 2024-09-21T10:08:06.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:06.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.481151+0000", 2024-09-21T10:08:06.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.739e-06 2024-09-21T10:08:06.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:06.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.481157+0000", 2024-09-21T10:08:06.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2720000000000003e-06 2024-09-21T10:08:06.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.547 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:06.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.481170+0000", 2024-09-21T10:08:06.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3508e-05 2024-09-21T10:08:06.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.547 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:06.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.481178+0000", 2024-09-21T10:08:06.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7929999999999998e-06 2024-09-21T10:08:06.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.485802+0000", 2024-09-21T10:08:06.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046242030000000003 2024-09-21T10:08:06.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:06.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.485909+0000", 2024-09-21T10:08:06.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106229 2024-09-21T10:08:06.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:08:06.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.486553+0000", 2024-09-21T10:08:06.549 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00064429199999999998 2024-09-21T10:08:06.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:06.549 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.496219+0000", 2024-09-21T10:08:06.549 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0096666579999999998 2024-09-21T10:08:06.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.549 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.496256+0000", 2024-09-21T10:08:06.549 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6879999999999999e-05 2024-09-21T10:08:06.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.597046+0000", 2024-09-21T10:08:06.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.100789183 2024-09-21T10:08:06.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.550 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:06.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.597089+0000", 2024-09-21T10:08:06.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3092e-05 2024-09-21T10:08:06.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.550 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:06.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.597143+0000", 2024-09-21T10:08:06.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3968999999999998e-05 2024-09-21T10:08:06.550 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.551 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:06.551 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.551 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:159913 5.a 5:5de12077:::benchmark_data_smithi098_135187_object24275:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-21T10:08:06.551 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:05:46.485073+0000", 2024-09-21T10:08:06.551 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 139.98072246999999, 2024-09-21T10:08:06.551 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.112177163, 2024-09-21T10:08:06.551 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:06.551 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:06.551 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:06.551 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:06.551 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:06.552 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:06.552 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 159913 2024-09-21T10:08:06.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.552 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:06.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.552 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:06.552 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.485073+0000", 2024-09-21T10:08:06.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.552 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:06.552 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.485073+0000", 2024-09-21T10:08:06.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:06.553 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.485074+0000", 2024-09-21T10:08:06.553 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1519999999999999e-06 2024-09-21T10:08:06.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:06.553 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.485079+0000", 2024-09-21T10:08:06.553 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7609999999999998e-06 2024-09-21T10:08:06.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:06.553 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.485089+0000", 2024-09-21T10:08:06.553 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0949e-05 2024-09-21T10:08:06.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:06.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.485096+0000", 2024-09-21T10:08:06.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9929999999999998e-06 2024-09-21T10:08:06.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.491743+0000", 2024-09-21T10:08:06.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0066466249999999998 2024-09-21T10:08:06.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:06.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.491855+0000", 2024-09-21T10:08:06.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000112169 2024-09-21T10:08:06.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:08:06.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.491962+0000", 2024-09-21T10:08:06.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106592 2024-09-21T10:08:06.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:06.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.496224+0000", 2024-09-21T10:08:06.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0042622240000000002 2024-09-21T10:08:06.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.556 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.556 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.496947+0000", 2024-09-21T10:08:06.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00072243800000000005 2024-09-21T10:08:06.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.556 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.556 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.597180+0000", 2024-09-21T10:08:06.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.100233746 2024-09-21T10:08:06.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.556 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:06.556 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.597219+0000", 2024-09-21T10:08:06.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9029999999999997e-05 2024-09-21T10:08:06.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.557 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:06.557 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.597250+0000", 2024-09-21T10:08:06.557 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0484000000000002e-05 2024-09-21T10:08:06.557 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.557 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:06.557 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.557 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:175155 5.b 5:d239a07f:::benchmark_data_smithi098_135187_object39517:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T10:08:06.557 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:06:04.521939+0000", 2024-09-21T10:08:06.557 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 121.94385585000001, 2024-09-21T10:08:06.557 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.102614697, 2024-09-21T10:08:06.557 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:06.558 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:06.558 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:06.558 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:06.558 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:06.558 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:06.558 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 175155 2024-09-21T10:08:06.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.558 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:06.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.558 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:06.558 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.521939+0000", 2024-09-21T10:08:06.558 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:06.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.521939+0000", 2024-09-21T10:08:06.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:06.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.521941+0000", 2024-09-21T10:08:06.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.494e-06 2024-09-21T10:08:06.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:06.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.521946+0000", 2024-09-21T10:08:06.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.412e-06 2024-09-21T10:08:06.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:06.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.521958+0000", 2024-09-21T10:08:06.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2063e-05 2024-09-21T10:08:06.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:06.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.521965+0000", 2024-09-21T10:08:06.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1799999999999999e-06 2024-09-21T10:08:06.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.522021+0000", 2024-09-21T10:08:06.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6076000000000003e-05 2024-09-21T10:08:06.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.561 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:06.561 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.522130+0000", 2024-09-21T10:08:06.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010877799999999999 2024-09-21T10:08:06.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.561 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:08:06.561 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.582908+0000", 2024-09-21T10:08:06.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.060778128000000001 2024-09-21T10:08:06.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.561 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:06.562 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.587772+0000", 2024-09-21T10:08:06.562 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048639920000000001 2024-09-21T10:08:06.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.562 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.562 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.607624+0000", 2024-09-21T10:08:06.562 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019851429 2024-09-21T10:08:06.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.562 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.562 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.624365+0000", 2024-09-21T10:08:06.562 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016740874999999999 2024-09-21T10:08:06.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.563 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:06.563 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.624460+0000", 2024-09-21T10:08:06.563 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4853999999999997e-05 2024-09-21T10:08:06.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.563 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:06.563 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.624554+0000", 2024-09-21T10:08:06.563 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4415999999999999e-05 2024-09-21T10:08:06.563 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.563 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:06.563 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.563 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:175292 5.b 5:dbabbd04:::benchmark_data_smithi098_135187_object39654:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T10:08:06.564 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:06:04.751439+0000", 2024-09-21T10:08:06.564 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 121.71435601899999, 2024-09-21T10:08:06.564 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.101317119, 2024-09-21T10:08:06.564 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:06.564 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:06.564 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:06.564 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:06.564 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:06.564 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:06.564 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 175292 2024-09-21T10:08:06.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.564 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:06.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.564 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:06.565 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:06.566 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.751439+0000", 2024-09-21T10:08:06.566 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.566 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:06.566 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.751439+0000", 2024-09-21T10:08:06.566 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.567 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:06.567 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.751440+0000", 2024-09-21T10:08:06.567 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.09e-07 2024-09-21T10:08:06.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.567 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:06.567 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.751444+0000", 2024-09-21T10:08:06.567 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0409999999999999e-06 2024-09-21T10:08:06.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.567 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:06.567 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.751455+0000", 2024-09-21T10:08:06.567 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1386000000000001e-05 2024-09-21T10:08:06.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.568 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:06.568 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.751463+0000", 2024-09-21T10:08:06.568 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1919999999999999e-06 2024-09-21T10:08:06.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.568 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.568 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.755683+0000", 2024-09-21T10:08:06.568 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0042200880000000003 2024-09-21T10:08:06.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.568 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:06.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.755805+0000", 2024-09-21T10:08:06.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000122008 2024-09-21T10:08:06.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.569 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:08:06.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.756528+0000", 2024-09-21T10:08:06.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00072333100000000002 2024-09-21T10:08:06.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.569 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:06.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.761217+0000", 2024-09-21T10:08:06.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046894199999999997 2024-09-21T10:08:06.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.570 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.570 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.762128+0000", 2024-09-21T10:08:06.570 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00091013400000000005 2024-09-21T10:08:06.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.570 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.570 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.852666+0000", 2024-09-21T10:08:06.570 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.090538059000000004 2024-09-21T10:08:06.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.570 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:06.570 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.852713+0000", 2024-09-21T10:08:06.570 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7509000000000002e-05 2024-09-21T10:08:06.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.571 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:06.571 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:04.852756+0000", 2024-09-21T10:08:06.571 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3041999999999999e-05 2024-09-21T10:08:06.571 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.571 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:06.571 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.571 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:267699 5.a 5:51b91a91:::benchmark_data_smithi098_135187_object132061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:08:06.571 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:59.409137+0000", 2024-09-21T10:08:06.571 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.0566585100000001, 2024-09-21T10:08:06.571 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3530456719999999, 2024-09-21T10:08:06.572 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:06.572 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:06.572 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:06.572 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:06.572 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:06.572 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:06.572 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 267699 2024-09-21T10:08:06.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.572 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:06.572 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.572 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:06.572 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409137+0000", 2024-09-21T10:08:06.572 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.573 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:06.573 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409137+0000", 2024-09-21T10:08:06.573 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.573 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:06.573 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409138+0000", 2024-09-21T10:08:06.573 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.331e-06 2024-09-21T10:08:06.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.573 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:06.573 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409143+0000", 2024-09-21T10:08:06.574 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0309999999999998e-06 2024-09-21T10:08:06.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.574 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:06.574 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409160+0000", 2024-09-21T10:08:06.574 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6694e-05 2024-09-21T10:08:06.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.574 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:06.574 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409168+0000", 2024-09-21T10:08:06.574 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2230000000000007e-06 2024-09-21T10:08:06.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.574 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.575 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409241+0000", 2024-09-21T10:08:06.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2818999999999995e-05 2024-09-21T10:08:06.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.575 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:08:06.575 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409253+0000", 2024-09-21T10:08:06.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1914e-05 2024-09-21T10:08:06.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.575 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.575 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.370041+0000", 2024-09-21T10:08:06.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.96078831799999997 2024-09-21T10:08:06.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.576 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:08:06.576 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.370053+0000", 2024-09-21T10:08:06.576 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2306999999999999e-05 2024-09-21T10:08:06.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.576 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.576 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.724460+0000", 2024-09-21T10:08:06.576 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3544063770000001 2024-09-21T10:08:06.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.576 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:06.576 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.724591+0000", 2024-09-21T10:08:06.576 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013132199999999999 2024-09-21T10:08:06.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.577 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3,4", 2024-09-21T10:08:06.577 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.724747+0000", 2024-09-21T10:08:06.577 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000156553 2024-09-21T10:08:06.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.577 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.577 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.742698+0000", 2024-09-21T10:08:06.577 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017950560000000001 2024-09-21T10:08:06.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.578 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:06.578 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.747832+0000", 2024-09-21T10:08:06.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051342310000000004 2024-09-21T10:08:06.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.578 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.578 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.755583+0000", 2024-09-21T10:08:06.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0077510879999999997 2024-09-21T10:08:06.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.579 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.579 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.761982+0000", 2024-09-21T10:08:06.579 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0063988789999999997 2024-09-21T10:08:06.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.579 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:06.579 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.762068+0000", 2024-09-21T10:08:06.579 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5698000000000005e-05 2024-09-21T10:08:06.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.580 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:06.580 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.762182+0000", 2024-09-21T10:08:06.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011432700000000001 2024-09-21T10:08:06.580 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.580 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:06.580 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.580 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:268317 5.8 5:14612c8f:::benchmark_data_smithi098_135187_object132679:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-21T10:08:06.581 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:02.604493+0000", 2024-09-21T10:08:06.581 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.8613019820000001, 2024-09-21T10:08:06.581 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.102379497, 2024-09-21T10:08:06.581 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:06.581 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:06.581 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:06.581 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:06.581 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:06.581 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:06.582 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 268317 2024-09-21T10:08:06.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.582 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:06.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:06.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:02.604493+0000", 2024-09-21T10:08:06.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:06.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:02.604493+0000", 2024-09-21T10:08:06.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:06.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:02.604494+0000", 2024-09-21T10:08:06.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7899999999999997e-07 2024-09-21T10:08:06.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:06.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:02.604498+0000", 2024-09-21T10:08:06.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6830000000000001e-06 2024-09-21T10:08:06.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.584 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:06.584 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:02.604508+0000", 2024-09-21T10:08:06.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.027e-05 2024-09-21T10:08:06.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.584 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:06.584 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:02.604514+0000", 2024-09-21T10:08:06.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.596e-06 2024-09-21T10:08:06.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.584 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.584 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:02.659859+0000", 2024-09-21T10:08:06.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.055344328999999998 2024-09-21T10:08:06.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.585 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:06.585 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:02.659971+0000", 2024-09-21T10:08:06.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000112744 2024-09-21T10:08:06.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.585 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:08:06.585 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:02.660087+0000", 2024-09-21T10:08:06.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000115605 2024-09-21T10:08:06.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.585 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:02.695089+0000", 2024-09-21T10:08:06.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035002196999999999 2024-09-21T10:08:06.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.586 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:02.695267+0000", 2024-09-21T10:08:06.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017807000000000001 2024-09-21T10:08:06.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.586 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:06.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:02.706726+0000", 2024-09-21T10:08:06.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011458628 2024-09-21T10:08:06.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.587 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:06.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:02.706832+0000", 2024-09-21T10:08:06.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010596500000000001 2024-09-21T10:08:06.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.587 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:06.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:02.706873+0000", 2024-09-21T10:08:06.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0630999999999998e-05 2024-09-21T10:08:06.587 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.587 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:06.588 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.588 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269037 5.a 5:5ae457a8:::benchmark_data_smithi098_135187_object133399:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:08:06.588 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.480193+0000", 2024-09-21T10:08:06.588 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.9856020650000001, 2024-09-21T10:08:06.588 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0309086750000001, 2024-09-21T10:08:06.588 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:06.588 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:06.588 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:06.588 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:06.589 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:06.589 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:06.589 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269037 2024-09-21T10:08:06.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.589 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:06.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.589 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:06.589 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480193+0000", 2024-09-21T10:08:06.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:06.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480193+0000", 2024-09-21T10:08:06.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:06.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480195+0000", 2024-09-21T10:08:06.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6479999999999999e-06 2024-09-21T10:08:06.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:06.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480200+0000", 2024-09-21T10:08:06.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7420000000000003e-06 2024-09-21T10:08:06.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:06.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480215+0000", 2024-09-21T10:08:06.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4293e-05 2024-09-21T10:08:06.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:06.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480222+0000", 2024-09-21T10:08:06.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3640000000000001e-06 2024-09-21T10:08:06.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.592 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480271+0000", 2024-09-21T10:08:06.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8466e-05 2024-09-21T10:08:06.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.592 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:08:06.592 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480276+0000", 2024-09-21T10:08:06.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9240000000000004e-06 2024-09-21T10:08:06.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.592 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.592 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.490543+0000", 2024-09-21T10:08:06.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0102668859999999 2024-09-21T10:08:06.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.593 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:06.593 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.490646+0000", 2024-09-21T10:08:06.593 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010246099999999999 2024-09-21T10:08:06.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.593 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:08:06.593 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.490749+0000", 2024-09-21T10:08:06.593 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000102839 2024-09-21T10:08:06.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.593 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.594 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510822+0000", 2024-09-21T10:08:06.594 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020072995999999999 2024-09-21T10:08:06.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.594 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:06.594 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510836+0000", 2024-09-21T10:08:06.594 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4445e-05 2024-09-21T10:08:06.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.594 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.594 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510993+0000", 2024-09-21T10:08:06.594 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000157207 2024-09-21T10:08:06.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:06.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511062+0000", 2024-09-21T10:08:06.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8923000000000004e-05 2024-09-21T10:08:06.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:06.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511102+0000", 2024-09-21T10:08:06.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9480999999999997e-05 2024-09-21T10:08:06.595 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.595 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:06.595 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.596 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269043 5.a 5:5d7883f5:::benchmark_data_smithi098_135187_object133405:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:08:06.596 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.487457+0000", 2024-09-21T10:08:06.596 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.9783375890000001, 2024-09-21T10:08:06.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.040401159, 2024-09-21T10:08:06.596 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:06.596 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:06.596 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:06.596 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:06.596 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:06.596 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:06.596 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269043 2024-09-21T10:08:06.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.597 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:06.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.597 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:06.597 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487457+0000", 2024-09-21T10:08:06.597 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.597 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:06.597 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487457+0000", 2024-09-21T10:08:06.597 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.597 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:06.598 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487458+0000", 2024-09-21T10:08:06.598 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8599999999999996e-07 2024-09-21T10:08:06.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.598 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:06.598 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487463+0000", 2024-09-21T10:08:06.598 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2830000000000003e-06 2024-09-21T10:08:06.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.598 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:06.598 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487476+0000", 2024-09-21T10:08:06.598 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2819e-05 2024-09-21T10:08:06.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:06.599 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487481+0000", 2024-09-21T10:08:06.599 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2519999999999999e-06 2024-09-21T10:08:06.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.599 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487524+0000", 2024-09-21T10:08:06.599 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2701000000000003e-05 2024-09-21T10:08:06.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:08:06.599 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487529+0000", 2024-09-21T10:08:06.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0690000000000001e-06 2024-09-21T10:08:06.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.600 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.600 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.500694+0000", 2024-09-21T10:08:06.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0131655589999999 2024-09-21T10:08:06.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.600 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:06.600 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.500787+0000", 2024-09-21T10:08:06.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2785e-05 2024-09-21T10:08:06.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.601 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:08:06.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.500887+0000", 2024-09-21T10:08:06.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9599000000000006e-05 2024-09-21T10:08:06.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.601 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:06.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510920+0000", 2024-09-21T10:08:06.602 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010033686999999999 2024-09-21T10:08:06.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.602 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.602 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.517569+0000", 2024-09-21T10:08:06.602 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0066490730000000001 2024-09-21T10:08:06.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.602 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.602 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.527765+0000", 2024-09-21T10:08:06.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010196197000000001 2024-09-21T10:08:06.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.603 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:06.603 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.527818+0000", 2024-09-21T10:08:06.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2401000000000002e-05 2024-09-21T10:08:06.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.603 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:06.603 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.527859+0000", 2024-09-21T10:08:06.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0747999999999997e-05 2024-09-21T10:08:06.603 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.604 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:06.604 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.604 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269059 5.a 5:530cb704:::benchmark_data_smithi098_135187_object133421:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:08:06.604 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.508524+0000", 2024-09-21T10:08:06.604 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.9572705930000001, 2024-09-21T10:08:06.604 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0242946340000001, 2024-09-21T10:08:06.604 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:06.604 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:06.604 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:06.604 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:06.604 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:06.604 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:06.605 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269059 2024-09-21T10:08:06.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.605 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:06.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:06.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508524+0000", 2024-09-21T10:08:06.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:06.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508524+0000", 2024-09-21T10:08:06.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:06.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.606 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:06.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508526+0000", 2024-09-21T10:08:06.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.387e-06 2024-09-21T10:08:06.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.606 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:06.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508530+0000", 2024-09-21T10:08:06.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.437e-06 2024-09-21T10:08:06.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.606 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:06.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508542+0000", 2024-09-21T10:08:06.607 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1249e-05 2024-09-21T10:08:06.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.607 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:06.607 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508547+0000", 2024-09-21T10:08:06.607 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5809999999999996e-06 2024-09-21T10:08:06.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.607 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.607 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508596+0000", 2024-09-21T10:08:06.607 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8641999999999999e-05 2024-09-21T10:08:06.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.608 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:08:06.608 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508602+0000", 2024-09-21T10:08:06.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8100000000000003e-06 2024-09-21T10:08:06.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.608 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:06.608 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.505838+0000", 2024-09-21T10:08:06.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.99723621699999998 2024-09-21T10:08:06.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:06.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.505943+0000", 2024-09-21T10:08:06.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010536500000000001 2024-09-21T10:08:06.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:08:06.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.506048+0000", 2024-09-21T10:08:06.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105221 2024-09-21T10:08:06.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:06.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.522571+0000", 2024-09-21T10:08:06.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016522236999999999 2024-09-21T10:08:06.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.610 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.610 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.523904+0000", 2024-09-21T10:08:06.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001332986 2024-09-21T10:08:06.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.610 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:06.610 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.532731+0000", 2024-09-21T10:08:06.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0088274440000000003 2024-09-21T10:08:06.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.610 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:06.611 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.532778+0000", 2024-09-21T10:08:06.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7413000000000002e-05 2024-09-21T10:08:06.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:06.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:06.611 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:06.611 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.532819+0000", 2024-09-21T10:08:06.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0645000000000001e-05 2024-09-21T10:08:06.611 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.611 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:06.611 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.611 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:06.611 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:06.612 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:08:06.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:06.656 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:06.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:06.757 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:06.786 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:06.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:06.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:06.789 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:06 smithi196 ceph-mon[84784]: pgmap v421: 89 pgs: 2 active+clean+remapped, 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 2 peering, 80 active+clean; 176 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 432 op/s; 991/8295 objects misplaced (11.947%); 8.0 MiB/s, 2 keys/s, 134 objects/s recovering 2024-09-21T10:08:06.789 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:06 smithi196 ceph-mon[84784]: osdmap e157: 8 total, 8 up, 7 in 2024-09-21T10:08:06.845 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:06 smithi098 ceph-mon[97580]: pgmap v421: 89 pgs: 2 active+clean+remapped, 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 2 peering, 80 active+clean; 176 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 432 op/s; 991/8295 objects misplaced (11.947%); 8.0 MiB/s, 2 keys/s, 134 objects/s recovering 2024-09-21T10:08:06.845 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:06 smithi098 ceph-mon[97580]: osdmap e157: 8 total, 8 up, 7 in 2024-09-21T10:08:06.846 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:06 smithi098 ceph-mon[100914]: pgmap v421: 89 pgs: 2 active+clean+remapped, 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 2 peering, 80 active+clean; 176 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 432 op/s; 991/8295 objects misplaced (11.947%); 8.0 MiB/s, 2 keys/s, 134 objects/s recovering 2024-09-21T10:08:06.846 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:06 smithi098 ceph-mon[100914]: osdmap e157: 8 total, 8 up, 7 in 2024-09-21T10:08:06.888 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:06.923 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:06.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:06.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:07.025 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:07.034 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:08:07.035 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:08:07.051 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:07.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:07.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:07.075 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:08:07.075 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:08:07.111 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:08:07.112 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:08:07.139 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:08:07.139 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:08:07.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:07.171 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:08:07.171 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:08:07.195 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:07.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:07.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:07.210 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:08:07.210 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:08:07.244 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:08:07.244 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:08:07.277 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:08:07.278 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:08:07.297 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:07.320 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:07.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:07.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:07.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:07.472 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:07.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:07.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:07.573 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:07.605 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:07.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:07.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:07.707 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:07.732 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:07.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:07.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:07.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:08:07.833 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:07.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:07 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1771287191' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:07.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:08:07.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:07.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:07 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1771287191' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:07.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:08:07.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:07.854 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:07.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:07.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:07.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:07 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1771287191' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:08:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:07.955 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:07.981 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:07.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:07.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:08.083 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:08.108 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:08.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:08.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:08.117 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Removed 135635 objects 2024-09-21T10:08:08.117 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Clean up completed and total clean up time :168.432 2024-09-21T10:08:08.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:08.232 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:08.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:08.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:08.333 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:08.354 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:08.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:08.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:08.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:08.477 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:08.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:08.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:08.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:08.601 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:08.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:08.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:08.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:08.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:08.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:08.827 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:08.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:08 smithi098 ceph-mon[97580]: pgmap v423: 89 pgs: 2 active+clean+remapped, 3 active+remapped+backfill_wait, 2 peering, 82 active+clean; 157 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 389 op/s; 586/7410 objects misplaced (7.908%); 10 MiB/s, 2 keys/s, 173 objects/s recovering 2024-09-21T10:08:08.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:08 smithi098 ceph-mon[100914]: pgmap v423: 89 pgs: 2 active+clean+remapped, 3 active+remapped+backfill_wait, 2 peering, 82 active+clean; 157 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 389 op/s; 586/7410 objects misplaced (7.908%); 10 MiB/s, 2 keys/s, 173 objects/s recovering 2024-09-21T10:08:08.849 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:08.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:08.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:08 smithi196 ceph-mon[84784]: pgmap v423: 89 pgs: 2 active+clean+remapped, 3 active+remapped+backfill_wait, 2 peering, 82 active+clean; 157 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 389 op/s; 586/7410 objects misplaced (7.908%); 10 MiB/s, 2 keys/s, 173 objects/s recovering 2024-09-21T10:08:08.951 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:08.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:08.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:09.076 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:09.098 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:09.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:09.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:09.163 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:08:09.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:09.229 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:09.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:09.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:09.331 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:09.355 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:09.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:09.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:09.456 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:09.480 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:09.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:09.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:09.581 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:09.603 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:09.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:09.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:09.705 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:09.727 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:09.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:09.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:09.829 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:09.850 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:09.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:09.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:09.952 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:09.974 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:09.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:09.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:10.076 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:10.102 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:10.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:10.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:10.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:10.227 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:10.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:10.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:10.329 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:10.353 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:10.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:10.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:10.455 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:10.481 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:10.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:10.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:10.583 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:10.605 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:10.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:10.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:10.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:10 smithi196 ceph-mon[84784]: pgmap v424: 89 pgs: 1 peering, 88 active+clean; 67 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 444 op/s; 13 MiB/s, 1 keys/s, 207 objects/s recovering 2024-09-21T10:08:10.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:08:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:08:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:08:10.706 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:10.728 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:10.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:10.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:10.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:10.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:10 smithi098 ceph-mon[97580]: pgmap v424: 89 pgs: 1 peering, 88 active+clean; 67 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 444 op/s; 13 MiB/s, 1 keys/s, 207 objects/s recovering 2024-09-21T10:08:10.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:08:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:08:10] "GET /metrics HTTP/1.1" 200 36474 "" "Prometheus/2.51.0" 2024-09-21T10:08:10.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:10 smithi098 ceph-mon[100914]: pgmap v424: 89 pgs: 1 peering, 88 active+clean; 67 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 444 op/s; 13 MiB/s, 1 keys/s, 207 objects/s recovering 2024-09-21T10:08:10.856 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:10.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:10.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:10.958 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:10.979 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:10.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:10.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:11.038 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:11.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:11.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:11.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:11.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:11.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:11.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:11.226 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:11.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:11.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:11.350 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:11.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:11.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:11.452 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:11.473 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:11.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:11.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:11.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:11.596 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:11.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:11.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:11.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:11.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:11.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:11.821 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:11.844 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:11.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:11.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:11.946 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:11.967 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:11.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:11.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:12.057 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-21T10:08:12.057 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-21T10:08:12.069 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:12.122 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:12.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:12.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:12.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:12.246 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:12.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:12.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:12.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:12.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:12.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:12.470 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:12.494 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:12.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:12.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:12.595 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:12.617 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:12.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:12.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:12.718 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:12.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:12.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:12.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:12.844 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:12.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:12 smithi098 ceph-mon[97580]: pgmap v425: 89 pgs: 89 active+clean; 24 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 368 op/s; 6.9 MiB/s, 110 objects/s recovering 2024-09-21T10:08:12.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:12 smithi098 ceph-mon[100914]: pgmap v425: 89 pgs: 89 active+clean; 24 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 368 op/s; 6.9 MiB/s, 110 objects/s recovering 2024-09-21T10:08:12.866 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:12.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:12.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:12 smithi196 ceph-mon[84784]: pgmap v425: 89 pgs: 89 active+clean; 24 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 368 op/s; 6.9 MiB/s, 110 objects/s recovering 2024-09-21T10:08:12.967 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:12.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:08:12.992 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:12.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:12.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:13.094 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:13.119 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:13.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:13.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:13.220 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:13.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:13.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:13.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:13.370 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:13.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:13.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:13.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:13.496 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:13.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:13.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:13.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:13.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:13.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:13.721 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:13.743 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:13.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:13.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:13.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:13.866 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:13.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:13.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:13.968 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:13.993 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:13.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:13.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:14.095 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:14.120 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:14.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:14.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:14.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:14.243 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:14.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:14.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:14.345 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:14.371 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:14.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:14.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:14.472 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:14.493 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:14.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:14.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:14.595 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:14.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:14 smithi098 ceph-mon[97580]: pgmap v426: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 362 op/s; 6.1 MiB/s, 97 objects/s recovering 2024-09-21T10:08:14.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:14 smithi098 ceph-mon[100914]: pgmap v426: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 362 op/s; 6.1 MiB/s, 97 objects/s recovering 2024-09-21T10:08:14.616 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:14.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:14.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:14.718 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:14.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:14.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:14.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:14.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:14.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:14.867 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:14.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:14 smithi196 ceph-mon[84784]: pgmap v426: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 362 op/s; 6.1 MiB/s, 97 objects/s recovering 2024-09-21T10:08:14.968 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:14.993 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:14.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:14.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:15.095 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:15.117 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:15.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:15.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:15.219 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:15.245 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:15.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:15.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:15.346 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:15.372 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:15.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:15.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:15.377 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:08:15.377 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:08:15.475 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:15.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:15.498 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:15.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:15.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:15.620 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:15.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:15.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:15.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:15.742 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:15.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:15.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:15.844 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:15.866 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:15.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:15.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:15.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:15.990 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:15.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:15.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:16.092 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:16.113 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:16.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:16.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:16.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:16.236 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:16.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:16.306 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:16.337 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:16.358 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:16.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:16.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:16.446 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:08:16.460 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:16.485 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:16.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:16.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:16.587 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:16.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:16 smithi098 ceph-mon[97580]: pgmap v427: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 273 op/s; 4.5 MiB/s, 71 objects/s recovering 2024-09-21T10:08:16.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:16 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2819715805' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:08:16.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:16 smithi098 ceph-mon[100914]: pgmap v427: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 273 op/s; 4.5 MiB/s, 71 objects/s recovering 2024-09-21T10:08:16.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:16 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2819715805' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:08:16.608 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:16.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:16.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:16.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:16.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:16.732 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:16.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:16.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:08:16.834 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:16.859 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:16.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:16.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:16 smithi196 ceph-mon[84784]: pgmap v427: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 273 op/s; 4.5 MiB/s, 71 objects/s recovering 2024-09-21T10:08:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:16 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2819715805' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:08:16.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:16.987 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:16.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:16.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:17.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:17.111 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:17.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:17.212 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:17.238 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:17.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:17.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:17.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:17.364 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:17.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:17.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:17.458 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-21T10:08:17.466 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:17.488 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:17.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:17.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:17.591 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:17.612 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:17.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:17.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:17.713 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:17.738 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:17.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:17.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:17.830 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:17 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:08:17.830 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:17 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2819715805' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:08:17.830 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:17 smithi098 ceph-mon[97580]: osdmap e158: 8 total, 8 up, 7 in 2024-09-21T10:08:17.830 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:17 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1767799891' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:08:17.830 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:17 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:08:17.830 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:17 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:08:17.830 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:17 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2819715805' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:08:17.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:17 smithi098 ceph-mon[100914]: osdmap e158: 8 total, 8 up, 7 in 2024-09-21T10:08:17.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:17 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1767799891' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:08:17.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:17 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:08:17.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:17.863 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:17.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:17.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:17.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:17 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:08:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:17 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2819715805' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:08:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:17 smithi196 ceph-mon[84784]: osdmap e158: 8 total, 8 up, 7 in 2024-09-21T10:08:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:17 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1767799891' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:08:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:17 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:08:17.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:17.986 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:17.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:17.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:18.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:18.110 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:18.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:18.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:18.212 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:18.238 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:18.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:18.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:18.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:18.361 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:18.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:18.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:18.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:18.466 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:18 smithi098 ceph-mon[97580]: pgmap v429: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 244 op/s; 2.7 MiB/s, 42 objects/s recovering 2024-09-21T10:08:18.466 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:18 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1767799891' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:08:18.466 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:18 smithi098 ceph-mon[97580]: osdmap e159: 8 total, 8 up, 7 in 2024-09-21T10:08:18.489 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:18.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:18.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:18.591 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:18.617 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:18.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:18.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:18.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:18.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:18.742 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:18.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:18.838 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:18.841 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:18.843 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:18.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:18 smithi098 ceph-mon[100914]: pgmap v429: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 244 op/s; 2.7 MiB/s, 42 objects/s recovering 2024-09-21T10:08:18.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:18 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1767799891' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:08:18.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:18 smithi098 ceph-mon[100914]: osdmap e159: 8 total, 8 up, 7 in 2024-09-21T10:08:18.869 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:18.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:18.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:18.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:18 smithi196 ceph-mon[84784]: pgmap v429: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 244 op/s; 2.7 MiB/s, 42 objects/s recovering 2024-09-21T10:08:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:18 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1767799891' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:08:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:18 smithi196 ceph-mon[84784]: osdmap e159: 8 total, 8 up, 7 in 2024-09-21T10:08:18.970 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:18.992 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:18.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:18.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:19.094 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:19.115 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:19.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:19.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:19.216 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:19.242 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:19.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:19.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:19.343 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:19.369 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:19.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:19.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:19.470 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:19.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:08:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:19.496 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:19.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:19.585 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:19 smithi098 ceph-mon[100914]: osdmap e160: 8 total, 8 up, 7 in 2024-09-21T10:08:19.585 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:19 smithi098 ceph-mon[97580]: osdmap e160: 8 total, 8 up, 7 in 2024-09-21T10:08:19.597 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:19.620 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:19.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:19.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:19.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:19.743 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:19.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:19.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:19.844 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:19.865 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:19.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:19.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:19 smithi196 ceph-mon[84784]: osdmap e160: 8 total, 8 up, 7 in 2024-09-21T10:08:19.952 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:08:19.952 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:08:19.956 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:08:19.956 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:08:19.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:19.972 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:08:19.972 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:08:19.988 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:19.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:19.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:19.990 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:08:19.990 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:08:20.009 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:08:20.009 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:08:20.025 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:08:20.026 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:08:20.041 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:08:20.042 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:08:20.057 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:08:20.058 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:08:20.090 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:20.112 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:20.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:20.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:20.213 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:20.239 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:20.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:20.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:20.341 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:20.364 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:20.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:20.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:20.466 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:20.471 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:08:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:08:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:08:20.490 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:20.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:20.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:20.591 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:20.613 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:20.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:20.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:20.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:20.737 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:20.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:20.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:20.838 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:20.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:08:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:08:20] "GET /metrics HTTP/1.1" 200 36474 "" "Prometheus/2.51.0" 2024-09-21T10:08:20.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:20 smithi098 ceph-mon[97580]: pgmap v432: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:20.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:20 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/94376802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:20.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:20 smithi098 ceph-mon[100914]: pgmap v432: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:20.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:20 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/94376802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:20.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:20.861 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:20.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:20 smithi196 ceph-mon[84784]: pgmap v432: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:20 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/94376802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:20.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:20.987 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:20.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:20.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:21.088 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:21.113 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:21.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:21.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:21.215 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:21.240 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:21.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:21.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:21.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:21.364 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:21.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:21.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:21.466 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:21.492 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:21.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:21.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:21.594 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:21.618 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:21.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:21.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:21.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:21.742 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:21.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:21.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:21.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:21.869 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:21.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:21.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:21.971 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:21.993 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:21.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:21.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:22.094 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:22.116 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:22.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:22.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:22.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:22.238 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:22.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:22.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:22.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:22.361 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:22.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:22.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:22.463 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:22.488 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:22.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:22.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:22.590 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:22.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:22.616 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:22.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:22.717 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:22.726 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:22 smithi098 ceph-mon[97580]: pgmap v433: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:22.727 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:08:22.727 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:22.727 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:22 smithi098 ceph-mon[100914]: pgmap v433: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:22.727 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:08:22.727 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:22.742 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:22.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:22.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:22.869 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:22.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:22.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:22 smithi196 ceph-mon[84784]: pgmap v433: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:08:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:22.912 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:08:22.912 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:08:22.912 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-21T10:08:22.912 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-21T10:08:22.912 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:08:22.971 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:22.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:22.994 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:22.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:23.094 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:23.116 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:23.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:23.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:23.173 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:08:23.173 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:08:23.173 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-21T10:08:23.173 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-21T10:08:23.217 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:23.242 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:23.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:23.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:23.343 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:23.365 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:23.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:23.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:23.467 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:23.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:23.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:23.509 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:08:23.592 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:23.618 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:23.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:23.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:23.720 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:23.745 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:23.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:23.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:23.847 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:23.869 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:23.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:23.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:23.970 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:23.981 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:23.988 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:23.998 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:23.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:23.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:24.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:24.125 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:24.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:24.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:24.227 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:24.253 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:24.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:24.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:24.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:24.376 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:24.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:24.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:24.478 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:24.503 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:24.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:24.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:24.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:08:24.526 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:24 smithi098 ceph-mon[97580]: pgmap v434: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:24.527 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:24 smithi098 ceph-mon[100914]: pgmap v434: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:24.605 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:24.630 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:24.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:24.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:24.732 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:24.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:24.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:24.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:24.859 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:24.880 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:24.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:24.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:24 smithi196 ceph-mon[84784]: pgmap v434: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:24.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:25.003 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:25.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:25.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:25.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:25.126 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:25.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:25.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:25.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:25.253 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:25.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:25.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:25.355 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:25.381 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:25.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:25.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:25.483 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:25.506 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:25.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:25.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:25.609 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:25.633 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:25.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:25.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:25.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:25.755 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:25.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:25.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:25.857 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:25.879 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:25.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:25.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:25.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:26.006 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:26.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:26.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:26.108 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:26.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:26.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:26.231 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:26.257 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:26.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:26.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:26.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:26.379 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:26.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:26.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:26.481 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:26.507 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:26.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:26.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:26.542 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'unique_pool_0' removed 2024-09-21T10:08:26.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:26.637 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:26.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:26.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:26.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:26.761 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:26.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:26.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:26.788 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:26 smithi098 ceph-mon[97580]: pgmap v435: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:26.788 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:26 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3692402251' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:08:26.788 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:26 smithi098 ceph-mon[100914]: pgmap v435: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:26.788 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:26 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3692402251' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:08:26.863 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:26.884 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:26.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:26.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:26 smithi196 ceph-mon[84784]: pgmap v435: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:26 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3692402251' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:08:26.986 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:27.006 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:27.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:27.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:27.108 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:27.129 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:27.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:27.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:27.231 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:27.252 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:27.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:27.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:27.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:27.381 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:27.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:27.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:27.483 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:27.505 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:27.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:27.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:27.513 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:08:27.514 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:08:27.514 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-21T10:08:27.514 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_1 16 2024-09-21T10:08:27.607 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:27.628 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:27.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:27.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:27.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:27.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:27.753 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:27.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:27.779 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:27 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3692402251' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:08:27.779 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:27 smithi098 ceph-mon[97580]: osdmap e161: 8 total, 8 up, 7 in 2024-09-21T10:08:27.779 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:27 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3692402251' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:08:27.779 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:27 smithi098 ceph-mon[100914]: osdmap e161: 8 total, 8 up, 7 in 2024-09-21T10:08:27.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:27.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:27.877 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:27.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:27 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3692402251' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:08:27.929 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:27 smithi196 ceph-mon[84784]: osdmap e161: 8 total, 8 up, 7 in 2024-09-21T10:08:27.978 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:28.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:28.000 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:28.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:28.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:28.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:28.123 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:28.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:28.224 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:28.247 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:28.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:28.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:28.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:28.372 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:28.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:28.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:28.386 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:28.473 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:28.494 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:28.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:28.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:28.595 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:28.618 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:28.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:28.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:28.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:28.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:28.742 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:28.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:28.843 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:28.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:28 smithi098 ceph-mon[97580]: pgmap v437: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:28.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:28 smithi098 ceph-mon[100914]: pgmap v437: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:28.864 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:28.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:28.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:28 smithi196 ceph-mon[84784]: pgmap v437: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:28.966 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:28.988 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:28.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:28.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:29.089 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:29.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:29.111 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:29.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:29.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:29.234 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:29.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:29.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:29.335 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:29.361 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:29.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:29.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:29.463 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:29.489 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:29.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:29.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:29.555 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:08:29.591 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:29.616 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:29.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:29.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:29.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:29.740 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:29.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:29.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:29.842 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:29.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:29 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3349586232' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:29.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:29 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3349586232' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:29.863 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:29.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:29.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:29.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:29 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3349586232' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:29.965 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:29.991 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:29.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:29.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:30.093 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:30.118 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:30.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:30.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:30.220 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:30.246 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:30.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:30.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:30.347 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:30.370 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:30.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:30.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:30.471 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:30.496 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:30.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:30.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:30.557 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:08:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:08:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:08:30.557 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:30 smithi196 ceph-mon[84784]: pgmap v438: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:30.557 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:30 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:08:30.557 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:30 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3349586232' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:08:30.557 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:30 smithi196 ceph-mon[84784]: osdmap e162: 8 total, 8 up, 7 in 2024-09-21T10:08:30.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:30.624 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:30.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:30.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:30.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:30 smithi098 ceph-mon[97580]: pgmap v438: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:30.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:30 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:08:30.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:30 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3349586232' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:08:30.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:30 smithi098 ceph-mon[97580]: osdmap e162: 8 total, 8 up, 7 in 2024-09-21T10:08:30.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:30 smithi098 ceph-mon[100914]: pgmap v438: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:30.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:30 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:08:30.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:30 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3349586232' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:08:30.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:30 smithi098 ceph-mon[100914]: osdmap e162: 8 total, 8 up, 7 in 2024-09-21T10:08:30.726 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:30.750 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:30.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:30.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:30.853 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:30.877 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:30.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:30.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:30.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:31.001 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:31.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:31.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:31.026 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:08:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:08:30] "GET /metrics HTTP/1.1" 200 36438 "" "Prometheus/2.51.0" 2024-09-21T10:08:31.103 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:31.128 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:31.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:31.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:31.230 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:31.256 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:31.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:31.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:31.357 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:31.382 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:31.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:31.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:31.484 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:31.509 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:31.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:31.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:31.611 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:31.634 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:31.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:31.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:31.661 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:08:31.661 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-21T10:08:31.661 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-21T10:08:31.661 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:08:31.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.661 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:31161 5.0 5:0ddaf0d6:::benchmark_data_smithi098_135187_object31160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:08:31.661 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:03:59.624988+0000", 2024-09-21T10:08:31.661 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 272.03268620699998, 2024-09-21T10:08:31.661 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.087531240999999996, 2024-09-21T10:08:31.662 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:31.662 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:31.662 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:31.662 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:31.662 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:31.662 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:31.662 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 31161 2024-09-21T10:08:31.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.662 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:31.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.662 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:31.662 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.624988+0000", 2024-09-21T10:08:31.662 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.663 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:31.663 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.624988+0000", 2024-09-21T10:08:31.663 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.663 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:31.663 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.624990+0000", 2024-09-21T10:08:31.663 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7689999999999999e-06 2024-09-21T10:08:31.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.663 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:31.663 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.625015+0000", 2024-09-21T10:08:31.663 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5292000000000001e-05 2024-09-21T10:08:31.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:31.664 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.625032+0000", 2024-09-21T10:08:31.664 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7414999999999998e-05 2024-09-21T10:08:31.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:31.664 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.625041+0000", 2024-09-21T10:08:31.664 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8610000000000004e-06 2024-09-21T10:08:31.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.625125+0000", 2024-09-21T10:08:31.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4332000000000005e-05 2024-09-21T10:08:31.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:31.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.625255+0000", 2024-09-21T10:08:31.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000129708 2024-09-21T10:08:31.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.666 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:08:31.666 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.625419+0000", 2024-09-21T10:08:31.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000163697 2024-09-21T10:08:31.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.666 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.666 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.629206+0000", 2024-09-21T10:08:31.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037873199999999998 2024-09-21T10:08:31.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.666 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:31.666 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.629218+0000", 2024-09-21T10:08:31.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2223e-05 2024-09-21T10:08:31.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.667 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.712446+0000", 2024-09-21T10:08:31.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.083227470999999997 2024-09-21T10:08:31.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:31.667 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.712484+0000", 2024-09-21T10:08:31.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8058000000000001e-05 2024-09-21T10:08:31.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.668 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:31.668 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.712519+0000", 2024-09-21T10:08:31.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5095000000000002e-05 2024-09-21T10:08:31.668 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.668 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:31.668 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.668 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:31162 5.0 5:05fe675d:::benchmark_data_smithi098_135187_object31161:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:08:31.668 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:03:59.625190+0000", 2024-09-21T10:08:31.668 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 272.032484318, 2024-09-21T10:08:31.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.087436227000000005, 2024-09-21T10:08:31.668 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:31.669 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:31.669 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:31.669 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:31.669 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:31.669 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:31.669 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 31162 2024-09-21T10:08:31.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.669 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:31.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.669 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:31.669 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.625190+0000", 2024-09-21T10:08:31.669 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:31.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.625190+0000", 2024-09-21T10:08:31.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:31.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.625191+0000", 2024-09-21T10:08:31.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.207e-06 2024-09-21T10:08:31.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:31.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.625215+0000", 2024-09-21T10:08:31.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4332e-05 2024-09-21T10:08:31.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.671 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:31.671 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.625228+0000", 2024-09-21T10:08:31.671 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2874000000000001e-05 2024-09-21T10:08:31.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.671 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:31.671 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.625236+0000", 2024-09-21T10:08:31.671 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3849999999999996e-06 2024-09-21T10:08:31.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.671 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.672 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.626471+0000", 2024-09-21T10:08:31.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001235426 2024-09-21T10:08:31.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:31.672 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.626614+0000", 2024-09-21T10:08:31.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014273899999999999 2024-09-21T10:08:31.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:08:31.672 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.626760+0000", 2024-09-21T10:08:31.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000146067 2024-09-21T10:08:31.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:31.673 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.629223+0000", 2024-09-21T10:08:31.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024634460000000002 2024-09-21T10:08:31.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.673 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.673 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.629257+0000", 2024-09-21T10:08:31.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4269e-05 2024-09-21T10:08:31.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.673 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.673 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.712562+0000", 2024-09-21T10:08:31.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.083304897000000003 2024-09-21T10:08:31.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.674 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:31.674 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.712595+0000", 2024-09-21T10:08:31.674 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2586000000000003e-05 2024-09-21T10:08:31.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.674 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:31.674 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.712626+0000", 2024-09-21T10:08:31.674 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0998999999999999e-05 2024-09-21T10:08:31.674 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.674 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:31.674 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.674 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:31164 5.0 5:0afeb33f:::benchmark_data_smithi098_135187_object31163:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:08:31.675 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:03:59.625616+0000", 2024-09-21T10:08:31.675 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 272.03205795399998, 2024-09-21T10:08:31.675 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.087150432, 2024-09-21T10:08:31.675 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:31.675 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:31.675 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:31.675 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:31.676 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:31.676 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:31.676 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 31164 2024-09-21T10:08:31.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.676 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:31.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.676 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:31.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.625616+0000", 2024-09-21T10:08:31.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:31.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.625616+0000", 2024-09-21T10:08:31.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:31.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.625617+0000", 2024-09-21T10:08:31.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.054e-06 2024-09-21T10:08:31.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:31.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.625639+0000", 2024-09-21T10:08:31.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1514000000000001e-05 2024-09-21T10:08:31.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:31.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.625663+0000", 2024-09-21T10:08:31.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4525e-05 2024-09-21T10:08:31.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:31.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.625672+0000", 2024-09-21T10:08:31.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3610000000000001e-06 2024-09-21T10:08:31.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.627970+0000", 2024-09-21T10:08:31.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022986450000000002 2024-09-21T10:08:31.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:31.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.628082+0000", 2024-09-21T10:08:31.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000112046 2024-09-21T10:08:31.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:08:31.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.628236+0000", 2024-09-21T10:08:31.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000153349 2024-09-21T10:08:31.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.680 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:31.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.629355+0000", 2024-09-21T10:08:31.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001119377 2024-09-21T10:08:31.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.680 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.629524+0000", 2024-09-21T10:08:31.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000168806 2024-09-21T10:08:31.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.680 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.712674+0000", 2024-09-21T10:08:31.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.083150289000000002 2024-09-21T10:08:31.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:31.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.712728+0000", 2024-09-21T10:08:31.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3808999999999999e-05 2024-09-21T10:08:31.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:31.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:03:59.712766+0000", 2024-09-21T10:08:31.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8656999999999999e-05 2024-09-21T10:08:31.681 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.681 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:31.681 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.682 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:153444 5.0 5:03342715:::benchmark_data_smithi098_135187_object17806:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T10:08:31.682 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:05:39.355062+0000", 2024-09-21T10:08:31.682 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 172.30261206899999, 2024-09-21T10:08:31.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.072473161999999994, 2024-09-21T10:08:31.682 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:31.682 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:31.682 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:31.682 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:31.682 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:31.682 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:31.682 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 153444 2024-09-21T10:08:31.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.683 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:31.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:31.683 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.355062+0000", 2024-09-21T10:08:31.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:31.683 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.355062+0000", 2024-09-21T10:08:31.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:31.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.355063+0000", 2024-09-21T10:08:31.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.128e-06 2024-09-21T10:08:31.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:31.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.355067+0000", 2024-09-21T10:08:31.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3900000000000003e-06 2024-09-21T10:08:31.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:31.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.355082+0000", 2024-09-21T10:08:31.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4375999999999999e-05 2024-09-21T10:08:31.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:31.685 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.355089+0000", 2024-09-21T10:08:31.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5299999999999999e-06 2024-09-21T10:08:31.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.685 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.357807+0000", 2024-09-21T10:08:31.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027172979999999999 2024-09-21T10:08:31.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:31.685 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.357922+0000", 2024-09-21T10:08:31.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000115319 2024-09-21T10:08:31.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:08:31.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.358037+0000", 2024-09-21T10:08:31.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000115128 2024-09-21T10:08:31.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:31.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.365596+0000", 2024-09-21T10:08:31.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0075588850000000004 2024-09-21T10:08:31.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.377449+0000", 2024-09-21T10:08:31.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011852763000000001 2024-09-21T10:08:31.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.427177+0000", 2024-09-21T10:08:31.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.049728466999999998 2024-09-21T10:08:31.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:31.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.427468+0000", 2024-09-21T10:08:31.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00029042500000000003 2024-09-21T10:08:31.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:31.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.427535+0000", 2024-09-21T10:08:31.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7453000000000006e-05 2024-09-21T10:08:31.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.688 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:31.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.688 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:153447 5.0 5:051f2a8b:::benchmark_data_smithi098_135187_object17809:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T10:08:31.688 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:05:39.357243+0000", 2024-09-21T10:08:31.688 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 172.300430654, 2024-09-21T10:08:31.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.074790989000000002, 2024-09-21T10:08:31.688 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:31.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:31.688 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:31.688 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:31.688 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:31.689 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:31.689 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 153447 2024-09-21T10:08:31.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.689 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:31.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:31.689 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.357243+0000", 2024-09-21T10:08:31.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:31.689 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.357243+0000", 2024-09-21T10:08:31.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:31.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.357245+0000", 2024-09-21T10:08:31.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.618e-06 2024-09-21T10:08:31.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:31.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.357250+0000", 2024-09-21T10:08:31.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3650000000000003e-06 2024-09-21T10:08:31.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:31.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.357267+0000", 2024-09-21T10:08:31.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6209999999999999e-05 2024-09-21T10:08:31.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:31.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.357274+0000", 2024-09-21T10:08:31.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7959999999999996e-06 2024-09-21T10:08:31.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.361878+0000", 2024-09-21T10:08:31.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046039619999999996 2024-09-21T10:08:31.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:31.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.361973+0000", 2024-09-21T10:08:31.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4897999999999998e-05 2024-09-21T10:08:31.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:08:31.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.362061+0000", 2024-09-21T10:08:31.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8276999999999995e-05 2024-09-21T10:08:31.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:31.693 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.370332+0000", 2024-09-21T10:08:31.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0082706310000000005 2024-09-21T10:08:31.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.693 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.693 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.381854+0000", 2024-09-21T10:08:31.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011521718 2024-09-21T10:08:31.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.693 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.693 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.431935+0000", 2024-09-21T10:08:31.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.050081036000000002 2024-09-21T10:08:31.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.694 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:31.694 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.431991+0000", 2024-09-21T10:08:31.694 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5760000000000001e-05 2024-09-21T10:08:31.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.694 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:31.694 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.432034+0000", 2024-09-21T10:08:31.694 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3717999999999997e-05 2024-09-21T10:08:31.694 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.694 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:31.694 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.695 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:153458 5.0 5:0034f328:::benchmark_data_smithi098_135187_object17820:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T10:08:31.695 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:05:39.365772+0000", 2024-09-21T10:08:31.695 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 172.291902209, 2024-09-21T10:08:31.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.078983768999999995, 2024-09-21T10:08:31.695 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:31.695 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:31.695 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:31.695 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:31.695 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:31.695 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:31.695 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 153458 2024-09-21T10:08:31.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.695 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:31.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.696 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:31.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.365772+0000", 2024-09-21T10:08:31.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.696 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:31.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.365772+0000", 2024-09-21T10:08:31.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.696 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:31.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.365774+0000", 2024-09-21T10:08:31.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7400000000000001e-06 2024-09-21T10:08:31.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.697 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:31.697 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.365779+0000", 2024-09-21T10:08:31.697 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8969999999999996e-06 2024-09-21T10:08:31.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.697 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:31.697 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.365794+0000", 2024-09-21T10:08:31.697 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4611e-05 2024-09-21T10:08:31.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.697 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:31.697 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.365802+0000", 2024-09-21T10:08:31.697 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.559e-06 2024-09-21T10:08:31.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.698 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.698 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.365946+0000", 2024-09-21T10:08:31.698 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014420399999999999 2024-09-21T10:08:31.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.698 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:31.698 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.366231+0000", 2024-09-21T10:08:31.698 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000285245 2024-09-21T10:08:31.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.698 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:08:31.698 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.366562+0000", 2024-09-21T10:08:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000330544 2024-09-21T10:08:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.377265+0000", 2024-09-21T10:08:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010703104999999999 2024-09-21T10:08:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.381977+0000", 2024-09-21T10:08:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0047120529999999999 2024-09-21T10:08:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.444639+0000", 2024-09-21T10:08:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.062662151999999999 2024-09-21T10:08:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.444711+0000", 2024-09-21T10:08:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2455999999999999e-05 2024-09-21T10:08:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.444756+0000", 2024-09-21T10:08:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4202999999999997e-05 2024-09-21T10:08:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:153459 5.0 5:01aedc36:::benchmark_data_smithi098_135187_object17821:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T10:08:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:05:39.369483+0000", 2024-09-21T10:08:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 172.288190872, 2024-09-21T10:08:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.079166963000000007, 2024-09-21T10:08:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 153459 2024-09-21T10:08:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.369483+0000", 2024-09-21T10:08:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.369483+0000", 2024-09-21T10:08:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.369485+0000", 2024-09-21T10:08:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5570000000000001e-06 2024-09-21T10:08:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.369490+0000", 2024-09-21T10:08:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2449999999999998e-06 2024-09-21T10:08:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.369505+0000", 2024-09-21T10:08:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4639e-05 2024-09-21T10:08:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.369512+0000", 2024-09-21T10:08:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1729999999999998e-06 2024-09-21T10:08:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.370394+0000", 2024-09-21T10:08:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088272199999999996 2024-09-21T10:08:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.370466+0000", 2024-09-21T10:08:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1340999999999993e-05 2024-09-21T10:08:31.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.705 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:08:31.705 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.370536+0000", 2024-09-21T10:08:31.705 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0259000000000006e-05 2024-09-21T10:08:31.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.705 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:31.705 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.377270+0000", 2024-09-21T10:08:31.705 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0067338559999999999 2024-09-21T10:08:31.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.705 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.705 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.382025+0000", 2024-09-21T10:08:31.706 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0047552100000000002 2024-09-21T10:08:31.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.706 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.706 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.448516+0000", 2024-09-21T10:08:31.706 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.066490651999999997 2024-09-21T10:08:31.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.706 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:31.706 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.448604+0000", 2024-09-21T10:08:31.706 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8398000000000003e-05 2024-09-21T10:08:31.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.448650+0000", 2024-09-21T10:08:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5911000000000001e-05 2024-09-21T10:08:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:153461 5.0 5:061ee0b2:::benchmark_data_smithi098_135187_object17823:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T10:08:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:05:39.370063+0000", 2024-09-21T10:08:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 172.28761055000001, 2024-09-21T10:08:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.088657066000000007, 2024-09-21T10:08:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 153461 2024-09-21T10:08:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.370063+0000", 2024-09-21T10:08:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.370063+0000", 2024-09-21T10:08:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.370064+0000", 2024-09-21T10:08:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0999999999999997e-07 2024-09-21T10:08:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.370068+0000", 2024-09-21T10:08:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9219999999999999e-06 2024-09-21T10:08:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.370082+0000", 2024-09-21T10:08:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3345e-05 2024-09-21T10:08:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.370088+0000", 2024-09-21T10:08:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6610000000000003e-06 2024-09-21T10:08:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.373490+0000", 2024-09-21T10:08:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034016879999999999 2024-09-21T10:08:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.373759+0000", 2024-09-21T10:08:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00026883300000000001 2024-09-21T10:08:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:08:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.374039+0000", 2024-09-21T10:08:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00028027299999999999 2024-09-21T10:08:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.377470+0000", 2024-09-21T10:08:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034308569999999998 2024-09-21T10:08:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.386479+0000", 2024-09-21T10:08:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0090093110000000007 2024-09-21T10:08:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.458617+0000", 2024-09-21T10:08:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.072137801000000001 2024-09-21T10:08:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.458687+0000", 2024-09-21T10:08:31.713 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9679999999999997e-05 2024-09-21T10:08:31.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.713 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:31.713 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:39.458721+0000", 2024-09-21T10:08:31.713 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3884999999999998e-05 2024-09-21T10:08:31.713 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.713 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:31.713 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.713 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:159910 5.0 5:02d77b2f:::benchmark_data_smithi098_135187_object24272:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-21T10:08:31.713 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:05:46.478049+0000", 2024-09-21T10:08:31.713 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 165.17962452500001, 2024-09-21T10:08:31.714 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11365120200000001, 2024-09-21T10:08:31.714 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:31.714 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:31.714 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:31.714 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:31.714 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:31.714 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:31.714 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 159910 2024-09-21T10:08:31.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.714 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:31.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.714 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:31.714 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.478049+0000", 2024-09-21T10:08:31.714 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.715 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:31.715 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.478049+0000", 2024-09-21T10:08:31.715 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.715 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:31.715 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.478051+0000", 2024-09-21T10:08:31.715 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7710000000000001e-06 2024-09-21T10:08:31.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.715 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:31.715 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.478058+0000", 2024-09-21T10:08:31.716 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.263e-06 2024-09-21T10:08:31.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.716 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:31.716 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.478071+0000", 2024-09-21T10:08:31.716 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3608000000000001e-05 2024-09-21T10:08:31.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.716 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:31.716 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.478081+0000", 2024-09-21T10:08:31.716 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4579999999999999e-06 2024-09-21T10:08:31.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.716 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.717 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.478153+0000", 2024-09-21T10:08:31.717 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2008999999999994e-05 2024-09-21T10:08:31.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.717 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:31.717 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.478268+0000", 2024-09-21T10:08:31.717 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011513799999999999 2024-09-21T10:08:31.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.717 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:08:31.717 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.478389+0000", 2024-09-21T10:08:31.717 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012151599999999999 2024-09-21T10:08:31.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.718 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:31.718 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.483937+0000", 2024-09-21T10:08:31.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055475589999999997 2024-09-21T10:08:31.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.718 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.718 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.484426+0000", 2024-09-21T10:08:31.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048927300000000003 2024-09-21T10:08:31.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.718 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.718 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.591632+0000", 2024-09-21T10:08:31.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.10720616099999999 2024-09-21T10:08:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.591672+0000", 2024-09-21T10:08:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0027999999999997e-05 2024-09-21T10:08:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.591701+0000", 2024-09-21T10:08:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8418000000000001e-05 2024-09-21T10:08:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:210366 5.0 5:0cae2c65:::benchmark_data_smithi098_135187_object74728:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-21T10:08:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:06:47.073750+0000", 2024-09-21T10:08:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 104.58392421000001, 2024-09-21T10:08:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.072871722, 2024-09-21T10:08:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 210366 2024-09-21T10:08:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.720 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:31.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.721 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:31.721 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.073750+0000", 2024-09-21T10:08:31.721 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.721 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:31.721 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.073750+0000", 2024-09-21T10:08:31.721 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.721 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:31.721 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.073750+0000", 2024-09-21T10:08:31.721 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6400000000000002e-07 2024-09-21T10:08:31.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.722 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:31.722 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.073754+0000", 2024-09-21T10:08:31.722 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4429999999999999e-06 2024-09-21T10:08:31.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.722 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:31.722 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.073761+0000", 2024-09-21T10:08:31.722 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5070000000000003e-06 2024-09-21T10:08:31.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.722 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:31.722 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.073767+0000", 2024-09-21T10:08:31.722 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4659999999999998e-06 2024-09-21T10:08:31.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.723 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.723 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.081578+0000", 2024-09-21T10:08:31.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0078108759999999996 2024-09-21T10:08:31.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.723 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:31.723 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.081844+0000", 2024-09-21T10:08:31.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00026675400000000001 2024-09-21T10:08:31.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.723 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:08:31.723 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.082191+0000", 2024-09-21T10:08:31.724 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00034658399999999999 2024-09-21T10:08:31.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.724 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:31.724 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.088518+0000", 2024-09-21T10:08:31.724 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0063274200000000003 2024-09-21T10:08:31.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.724 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.724 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.093924+0000", 2024-09-21T10:08:31.724 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054057460000000003 2024-09-21T10:08:31.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.724 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.725 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.146246+0000", 2024-09-21T10:08:31.725 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.052321925999999998 2024-09-21T10:08:31.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.725 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:31.725 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.146479+0000", 2024-09-21T10:08:31.725 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023303899999999999 2024-09-21T10:08:31.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.725 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:31.725 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:06:47.146622+0000", 2024-09-21T10:08:31.725 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000142397 2024-09-21T10:08:31.725 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.725 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:31.725 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.726 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:267693 5.0 5:02a5fe86:::benchmark_data_smithi098_135187_object132055:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:08:31.726 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:59.398968+0000", 2024-09-21T10:08:31.726 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.258705734000003, 2024-09-21T10:08:31.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.363874005, 2024-09-21T10:08:31.726 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:31.726 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:31.726 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:31.726 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:31.726 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:31.726 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:31.726 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 267693 2024-09-21T10:08:31.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.727 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:31.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:31.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.398968+0000", 2024-09-21T10:08:31.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:31.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.398968+0000", 2024-09-21T10:08:31.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:31.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.398970+0000", 2024-09-21T10:08:31.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0310000000000001e-06 2024-09-21T10:08:31.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:31.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.398977+0000", 2024-09-21T10:08:31.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1559999999999998e-06 2024-09-21T10:08:31.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:31.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.398999+0000", 2024-09-21T10:08:31.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1103000000000001e-05 2024-09-21T10:08:31.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:31.729 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399008+0000", 2024-09-21T10:08:31.729 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6350000000000001e-06 2024-09-21T10:08:31.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.729 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.729 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399115+0000", 2024-09-21T10:08:31.729 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106959 2024-09-21T10:08:31.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.729 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:08:31.729 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399130+0000", 2024-09-21T10:08:31.729 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4918e-05 2024-09-21T10:08:31.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.372566+0000", 2024-09-21T10:08:31.730 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.97343550599999995 2024-09-21T10:08:31.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:08:31.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.372578+0000", 2024-09-21T10:08:31.730 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2269000000000001e-05 2024-09-21T10:08:31.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.705405+0000", 2024-09-21T10:08:31.730 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3328273150000001 2024-09-21T10:08:31.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.731 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:31.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.705537+0000", 2024-09-21T10:08:31.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000131891 2024-09-21T10:08:31.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,1,4", 2024-09-21T10:08:31.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.706214+0000", 2024-09-21T10:08:31.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00067717799999999996 2024-09-21T10:08:31.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.733 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.740214+0000", 2024-09-21T10:08:31.733 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034000056000000001 2024-09-21T10:08:31.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.733 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.733 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.740297+0000", 2024-09-21T10:08:31.733 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2787999999999994e-05 2024-09-21T10:08:31.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.733 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:31.733 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.741535+0000", 2024-09-21T10:08:31.733 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001237602 2024-09-21T10:08:31.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.734 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.734 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.762486+0000", 2024-09-21T10:08:31.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020951805 2024-09-21T10:08:31.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.734 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:31.734 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.762689+0000", 2024-09-21T10:08:31.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020263099999999999 2024-09-21T10:08:31.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.734 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:31.734 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.762842+0000", 2024-09-21T10:08:31.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015316200000000001 2024-09-21T10:08:31.734 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.735 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:31.735 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.735 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:267694 5.1 5:88acc045:::benchmark_data_smithi098_135187_object132056:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:08:31.735 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:59.399033+0000", 2024-09-21T10:08:31.735 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.258641040999997, 2024-09-21T10:08:31.735 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2681572970000001, 2024-09-21T10:08:31.735 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:31.735 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:31.735 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:31.735 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:31.735 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:31.735 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:31.736 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 267694 2024-09-21T10:08:31.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.736 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:31.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.736 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:31.736 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399033+0000", 2024-09-21T10:08:31.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.736 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:31.736 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399033+0000", 2024-09-21T10:08:31.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.736 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:31.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399034+0000", 2024-09-21T10:08:31.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7800000000000001e-07 2024-09-21T10:08:31.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:31.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399036+0000", 2024-09-21T10:08:31.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8590000000000001e-06 2024-09-21T10:08:31.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:31.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399045+0000", 2024-09-21T10:08:31.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8950000000000004e-06 2024-09-21T10:08:31.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.738 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:31.738 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399051+0000", 2024-09-21T10:08:31.738 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6540000000000004e-06 2024-09-21T10:08:31.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.738 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.738 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399172+0000", 2024-09-21T10:08:31.738 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000121465 2024-09-21T10:08:31.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.738 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:08:31.738 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399182+0000", 2024-09-21T10:08:31.738 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9199999999999999e-06 2024-09-21T10:08:31.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.739 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.739 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.377429+0000", 2024-09-21T10:08:31.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.97824708500000002 2024-09-21T10:08:31.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.739 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:08:31.739 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.377440+0000", 2024-09-21T10:08:31.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0424999999999999e-05 2024-09-21T10:08:31.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.739 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.739 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.646211+0000", 2024-09-21T10:08:31.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2687708520000001 2024-09-21T10:08:31.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:31.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.646342+0000", 2024-09-21T10:08:31.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013128700000000001 2024-09-21T10:08:31.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4,7", 2024-09-21T10:08:31.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.646466+0000", 2024-09-21T10:08:31.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012438799999999999 2024-09-21T10:08:31.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.652333+0000", 2024-09-21T10:08:31.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058668069999999999 2024-09-21T10:08:31.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:31.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.661482+0000", 2024-09-21T10:08:31.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0091486180000000007 2024-09-21T10:08:31.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.662947+0000", 2024-09-21T10:08:31.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001465638 2024-09-21T10:08:31.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.742 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.666986+0000", 2024-09-21T10:08:31.742 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040387859999999999 2024-09-21T10:08:31.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.742 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:31.742 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.667053+0000", 2024-09-21T10:08:31.742 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6304999999999996e-05 2024-09-21T10:08:31.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.742 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:31.742 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.667190+0000", 2024-09-21T10:08:31.742 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000137735 2024-09-21T10:08:31.742 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.743 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:31.743 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.743 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:267704 5.1 5:8e7993d3:::benchmark_data_smithi098_135187_object132066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:08:31.743 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:59.421065+0000", 2024-09-21T10:08:31.743 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.236609454000003, 2024-09-21T10:08:31.743 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.250816414, 2024-09-21T10:08:31.743 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:31.743 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:31.743 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:31.743 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:31.743 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:31.743 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:31.744 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 267704 2024-09-21T10:08:31.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.744 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:31.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:31.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421065+0000", 2024-09-21T10:08:31.744 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:31.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421065+0000", 2024-09-21T10:08:31.744 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:31.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421066+0000", 2024-09-21T10:08:31.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.871e-06 2024-09-21T10:08:31.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:31.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421073+0000", 2024-09-21T10:08:31.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9059999999999996e-06 2024-09-21T10:08:31.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:31.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421091+0000", 2024-09-21T10:08:31.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7703e-05 2024-09-21T10:08:31.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:31.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421099+0000", 2024-09-21T10:08:31.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.464e-06 2024-09-21T10:08:31.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421173+0000", 2024-09-21T10:08:31.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4025999999999999e-05 2024-09-21T10:08:31.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:08:31.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421180+0000", 2024-09-21T10:08:31.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9240000000000007e-06 2024-09-21T10:08:31.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.747 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.747 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.380372+0000", 2024-09-21T10:08:31.747 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.95919200400000004 2024-09-21T10:08:31.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.747 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:08:31.747 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.380384+0000", 2024-09-21T10:08:31.747 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1955e-05 2024-09-21T10:08:31.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.747 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.747 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.652368+0000", 2024-09-21T10:08:31.748 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2719833110000001 2024-09-21T10:08:31.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.748 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:31.748 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.652497+0000", 2024-09-21T10:08:31.748 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012972 2024-09-21T10:08:31.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.748 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4,7", 2024-09-21T10:08:31.748 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.652632+0000", 2024-09-21T10:08:31.748 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013412799999999999 2024-09-21T10:08:31.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.749 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:31.749 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.661488+0000", 2024-09-21T10:08:31.749 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0088560349999999999 2024-09-21T10:08:31.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.749 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.749 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.661596+0000", 2024-09-21T10:08:31.749 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108306 2024-09-21T10:08:31.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.749 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.749 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.667234+0000", 2024-09-21T10:08:31.749 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056377440000000001 2024-09-21T10:08:31.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.750 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.671567+0000", 2024-09-21T10:08:31.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043329320000000003 2024-09-21T10:08:31.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.750 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:31.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.671746+0000", 2024-09-21T10:08:31.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017953199999999999 2024-09-21T10:08:31.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.750 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:31.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.671881+0000", 2024-09-21T10:08:31.751 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013485299999999999 2024-09-21T10:08:31.751 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.751 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:31.751 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.751 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:267713 5.0 5:0b4fa71e:::benchmark_data_smithi098_135187_object132075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-09-21T10:08:31.751 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:01.631012+0000", 2024-09-21T10:08:31.751 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 30.026662057999999, 2024-09-21T10:08:31.751 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.134078592, 2024-09-21T10:08:31.751 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:31.751 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:31.751 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:31.751 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:31.751 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:31.752 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:31.752 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 267713 2024-09-21T10:08:31.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.752 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:31.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.752 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:31.752 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.631012+0000", 2024-09-21T10:08:31.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.752 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:31.752 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.631012+0000", 2024-09-21T10:08:31.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:31.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.631014+0000", 2024-09-21T10:08:31.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0370000000000001e-06 2024-09-21T10:08:31.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:31.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.631020+0000", 2024-09-21T10:08:31.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4559999999999997e-06 2024-09-21T10:08:31.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:31.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.631041+0000", 2024-09-21T10:08:31.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0927000000000001e-05 2024-09-21T10:08:31.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:31.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.631050+0000", 2024-09-21T10:08:31.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9099999999999994e-06 2024-09-21T10:08:31.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.725855+0000", 2024-09-21T10:08:31.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.094804567000000006 2024-09-21T10:08:31.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:31.755 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.725980+0000", 2024-09-21T10:08:31.755 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012554800000000001 2024-09-21T10:08:31.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.755 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,1,4", 2024-09-21T10:08:31.755 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.726142+0000", 2024-09-21T10:08:31.755 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016126899999999999 2024-09-21T10:08:31.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.755 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.755 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.740256+0000", 2024-09-21T10:08:31.755 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014114732 2024-09-21T10:08:31.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.756 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.740335+0000", 2024-09-21T10:08:31.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8700000000000002e-05 2024-09-21T10:08:31.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:31.756 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.741540+0000", 2024-09-21T10:08:31.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0012051760000000001 2024-09-21T10:08:31.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.756 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.764833+0000", 2024-09-21T10:08:31.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.023293008 2024-09-21T10:08:31.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.757 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:31.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.764979+0000", 2024-09-21T10:08:31.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014574499999999999 2024-09-21T10:08:31.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.757 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:31.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.765091+0000", 2024-09-21T10:08:31.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000111517 2024-09-21T10:08:31.757 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.757 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:31.757 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.758 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269031 5.0 5:098330c7:::benchmark_data_smithi098_135187_object133393:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:08:31.758 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.462832+0000", 2024-09-21T10:08:31.758 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.194841688, 2024-09-21T10:08:31.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.042491235, 2024-09-21T10:08:31.758 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:31.758 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:31.758 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:31.758 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:31.758 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:31.758 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:31.758 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269031 2024-09-21T10:08:31.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.759 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:31.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:31.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462832+0000", 2024-09-21T10:08:31.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:31.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462832+0000", 2024-09-21T10:08:31.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:31.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462834+0000", 2024-09-21T10:08:31.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.68e-06 2024-09-21T10:08:31.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:31.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462840+0000", 2024-09-21T10:08:31.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6670000000000001e-06 2024-09-21T10:08:31.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:31.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462857+0000", 2024-09-21T10:08:31.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7028999999999999e-05 2024-09-21T10:08:31.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.761 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:31.761 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462866+0000", 2024-09-21T10:08:31.761 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9080000000000001e-06 2024-09-21T10:08:31.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.761 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.761 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462933+0000", 2024-09-21T10:08:31.761 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7100999999999993e-05 2024-09-21T10:08:31.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.761 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:08:31.761 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462943+0000", 2024-09-21T10:08:31.761 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0098999999999999e-05 2024-09-21T10:08:31.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.762 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.762 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.489116+0000", 2024-09-21T10:08:31.762 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0261731869999999 2024-09-21T10:08:31.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.762 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:31.762 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.489223+0000", 2024-09-21T10:08:31.762 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106709 2024-09-21T10:08:31.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.762 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:08:31.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.489331+0000", 2024-09-21T10:08:31.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010831399999999999 2024-09-21T10:08:31.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.763 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:31.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.498893+0000", 2024-09-21T10:08:31.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0095622350000000005 2024-09-21T10:08:31.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.763 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.500295+0000", 2024-09-21T10:08:31.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0014013249999999999 2024-09-21T10:08:31.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.505231+0000", 2024-09-21T10:08:31.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0049369000000000001 2024-09-21T10:08:31.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:31.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.505284+0000", 2024-09-21T10:08:31.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2510999999999999e-05 2024-09-21T10:08:31.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:31.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.505324+0000", 2024-09-21T10:08:31.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9570000000000002e-05 2024-09-21T10:08:31.765 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.765 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:31.765 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.765 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269041 5.1 5:89c4bb2f:::benchmark_data_smithi098_135187_object133403:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:08:31.765 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.486118+0000", 2024-09-21T10:08:31.765 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.171555764000001, 2024-09-21T10:08:31.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0245291729999999, 2024-09-21T10:08:31.765 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:31.765 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:31.765 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:31.765 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:31.765 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:31.766 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:31.766 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269041 2024-09-21T10:08:31.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.766 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:31.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.766 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:31.766 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.486118+0000", 2024-09-21T10:08:31.766 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.766 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:31.766 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.486118+0000", 2024-09-21T10:08:31.766 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.767 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:31.767 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.486121+0000", 2024-09-21T10:08:31.767 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2539999999999999e-06 2024-09-21T10:08:31.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.767 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:31.767 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.486128+0000", 2024-09-21T10:08:31.767 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4880000000000001e-06 2024-09-21T10:08:31.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.767 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:31.767 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.486145+0000", 2024-09-21T10:08:31.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7200000000000001e-05 2024-09-21T10:08:31.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:31.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.486154+0000", 2024-09-21T10:08:31.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1379999999999996e-06 2024-09-21T10:08:31.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.486211+0000", 2024-09-21T10:08:31.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6759000000000003e-05 2024-09-21T10:08:31.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:08:31.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.486217+0000", 2024-09-21T10:08:31.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3520000000000003e-06 2024-09-21T10:08:31.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.488241+0000", 2024-09-21T10:08:31.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.002023962 2024-09-21T10:08:31.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:31.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.488411+0000", 2024-09-21T10:08:31.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000169951 2024-09-21T10:08:31.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:08:31.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.488526+0000", 2024-09-21T10:08:31.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114225 2024-09-21T10:08:31.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.504331+0000", 2024-09-21T10:08:31.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015805092999999999 2024-09-21T10:08:31.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510365+0000", 2024-09-21T10:08:31.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0060348060000000002 2024-09-21T10:08:31.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.771 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:31.771 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510540+0000", 2024-09-21T10:08:31.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000174258 2024-09-21T10:08:31.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.771 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:31.771 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510601+0000", 2024-09-21T10:08:31.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1678000000000001e-05 2024-09-21T10:08:31.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.771 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510647+0000", 2024-09-21T10:08:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6008999999999999e-05 2024-09-21T10:08:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269050 5.1 5:811e02ab:::benchmark_data_smithi098_135187_object133412:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:08:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.496839+0000", 2024-09-21T10:08:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.160835186, 2024-09-21T10:08:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0140269550000001, 2024-09-21T10:08:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269050 2024-09-21T10:08:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496839+0000", 2024-09-21T10:08:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496839+0000", 2024-09-21T10:08:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496840+0000", 2024-09-21T10:08:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4899999999999999e-06 2024-09-21T10:08:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496848+0000", 2024-09-21T10:08:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2490000000000004e-06 2024-09-21T10:08:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496867+0000", 2024-09-21T10:08:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8949000000000001e-05 2024-09-21T10:08:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496877+0000", 2024-09-21T10:08:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0536e-05 2024-09-21T10:08:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496948+0000", 2024-09-21T10:08:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1159000000000001e-05 2024-09-21T10:08:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:08:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496958+0000", 2024-09-21T10:08:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0169e-05 2024-09-21T10:08:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.493320+0000", 2024-09-21T10:08:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.99636132300000002 2024-09-21T10:08:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.493421+0000", 2024-09-21T10:08:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010134899999999999 2024-09-21T10:08:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:08:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.493525+0000", 2024-09-21T10:08:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000103624 2024-09-21T10:08:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.504366+0000", 2024-09-21T10:08:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010841151 2024-09-21T10:08:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510409+0000", 2024-09-21T10:08:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0060434240000000004 2024-09-21T10:08:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510741+0000", 2024-09-21T10:08:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00033127500000000002 2024-09-21T10:08:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510811+0000", 2024-09-21T10:08:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0066000000000003e-05 2024-09-21T10:08:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510866+0000", 2024-09-21T10:08:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5191000000000001e-05 2024-09-21T10:08:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269053 5.1 5:8f082889:::benchmark_data_smithi098_135187_object133415:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:08:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.502117+0000", 2024-09-21T10:08:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.155556791999999, 2024-09-21T10:08:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0088990069999999, 2024-09-21T10:08:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269053 2024-09-21T10:08:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502117+0000", 2024-09-21T10:08:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502117+0000", 2024-09-21T10:08:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502119+0000", 2024-09-21T10:08:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.561e-06 2024-09-21T10:08:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502125+0000", 2024-09-21T10:08:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8619999999999999e-06 2024-09-21T10:08:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502138+0000", 2024-09-21T10:08:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3349e-05 2024-09-21T10:08:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502145+0000", 2024-09-21T10:08:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6309999999999997e-06 2024-09-21T10:08:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502203+0000", 2024-09-21T10:08:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8208999999999998e-05 2024-09-21T10:08:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:08:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502214+0000", 2024-09-21T10:08:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1096e-05 2024-09-21T10:08:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.497567+0000", 2024-09-21T10:08:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.99535269800000004 2024-09-21T10:08:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.497669+0000", 2024-09-21T10:08:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000102094 2024-09-21T10:08:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:08:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.497772+0000", 2024-09-21T10:08:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000103498 2024-09-21T10:08:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.504400+0000", 2024-09-21T10:08:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0066274569999999998 2024-09-21T10:08:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510446+0000", 2024-09-21T10:08:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0060458229999999996 2024-09-21T10:08:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510873+0000", 2024-09-21T10:08:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042730600000000002 2024-09-21T10:08:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510977+0000", 2024-09-21T10:08:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010447599999999999 2024-09-21T10:08:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511016+0000", 2024-09-21T10:08:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8946999999999998e-05 2024-09-21T10:08:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269054 5.1 5:8b4ad385:::benchmark_data_smithi098_135187_object133416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:08:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.502494+0000", 2024-09-21T10:08:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.155179833999998, 2024-09-21T10:08:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0086457820000001, 2024-09-21T10:08:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269054 2024-09-21T10:08:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502494+0000", 2024-09-21T10:08:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502494+0000", 2024-09-21T10:08:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502495+0000", 2024-09-21T10:08:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7899999999999998e-07 2024-09-21T10:08:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502499+0000", 2024-09-21T10:08:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7220000000000001e-06 2024-09-21T10:08:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502507+0000", 2024-09-21T10:08:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8659999999999995e-06 2024-09-21T10:08:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502512+0000", 2024-09-21T10:08:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5820000000000003e-06 2024-09-21T10:08:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502561+0000", 2024-09-21T10:08:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8516000000000001e-05 2024-09-21T10:08:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:08:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502569+0000", 2024-09-21T10:08:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6300000000000004e-06 2024-09-21T10:08:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.504428+0000", 2024-09-21T10:08:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.001859013 2024-09-21T10:08:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.504525+0000", 2024-09-21T10:08:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6582999999999993e-05 2024-09-21T10:08:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:08:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.504620+0000", 2024-09-21T10:08:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5313e-05 2024-09-21T10:08:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510486+0000", 2024-09-21T10:08:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058654859999999996 2024-09-21T10:08:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510519+0000", 2024-09-21T10:08:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3717999999999998e-05 2024-09-21T10:08:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511023+0000", 2024-09-21T10:08:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00050323200000000005 2024-09-21T10:08:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511094+0000", 2024-09-21T10:08:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1662000000000006e-05 2024-09-21T10:08:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511140+0000", 2024-09-21T10:08:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5779999999999999e-05 2024-09-21T10:08:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269056 5.0 5:097ecfd1:::benchmark_data_smithi098_135187_object133418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:08:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.505026+0000", 2024-09-21T10:08:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.152648405000001, 2024-09-21T10:08:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0033554609999999, 2024-09-21T10:08:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:08:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:08:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:08:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:08:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269056 2024-09-21T10:08:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:08:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:08:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505026+0000", 2024-09-21T10:08:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:08:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505026+0000", 2024-09-21T10:08:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:08:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:08:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505027+0000", 2024-09-21T10:08:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6300000000000001e-06 2024-09-21T10:08:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:08:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505035+0000", 2024-09-21T10:08:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5700000000000004e-06 2024-09-21T10:08:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:08:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505062+0000", 2024-09-21T10:08:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7098999999999999e-05 2024-09-21T10:08:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:08:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505080+0000", 2024-09-21T10:08:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7603000000000001e-05 2024-09-21T10:08:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505219+0000", 2024-09-21T10:08:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000139732 2024-09-21T10:08:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:08:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505229+0000", 2024-09-21T10:08:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7310000000000005e-06 2024-09-21T10:08:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:08:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.493545+0000", 2024-09-21T10:08:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.98831621599999997 2024-09-21T10:08:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:08:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.493636+0000", 2024-09-21T10:08:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0378999999999996e-05 2024-09-21T10:08:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:08:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.493732+0000", 2024-09-21T10:08:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6831999999999997e-05 2024-09-21T10:08:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:08:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.504194+0000", 2024-09-21T10:08:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010461981 2024-09-21T10:08:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.506161+0000", 2024-09-21T10:08:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0019666000000000002 2024-09-21T10:08:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.508250+0000", 2024-09-21T10:08:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002089025 2024-09-21T10:08:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:08:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.508337+0000", 2024-09-21T10:08:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7369999999999999e-05 2024-09-21T10:08:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:08:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:08:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:08:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.508381+0000", 2024-09-21T10:08:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3693000000000003e-05 2024-09-21T10:08:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:08:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:08:31.800 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:08:31.801 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:31.826 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:31.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:31.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:31.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:31.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:31.951 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:31.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:32.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:32.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:32.074 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:32.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:32.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:32.195 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:32.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:32.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:32.296 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:32.322 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:32.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:32.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:32.424 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:32.448 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:32.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:32.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:32.549 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:32.571 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:32.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:32.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:32.596 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:08:32.596 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:08:32.596 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:08:32.596 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:08:32.597 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:08:32.597 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:08:32.597 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:08:32.597 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:08:32.605 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:08:32.605 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:08:32.621 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:08:32.622 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:08:32.637 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:08:32.637 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:08:32.654 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:08:32.654 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:08:32.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:32.696 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:32.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:32.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:32.701 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:32 smithi098 ceph-mon[97580]: pgmap v440: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:32.701 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:32 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/953583993' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:32.701 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:32 smithi098 ceph-mon[100914]: pgmap v440: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:32.701 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:32 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/953583993' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:32.774 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:08:32.798 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:32.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:32.823 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:32.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:32 smithi196 ceph-mon[84784]: pgmap v440: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:32 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/953583993' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:32.924 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:32.948 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:32.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:32.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:33.049 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:33.073 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:33.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:33.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:33.175 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:33.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:33.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:33.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:33.319 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:33.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:33.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:33.421 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:33.442 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:33.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:33.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:33.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:33.552 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:08:33.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:33.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:33.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:33.673 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:33.695 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:33.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:33.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:33.797 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:33.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:33.819 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:33.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:33.824 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:08:33.920 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:33.941 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:33.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:33.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:34.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:34.064 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:34.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:34.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:34.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:34.187 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:34.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:34.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:34.257 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:34.289 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:34.310 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:34.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:34.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:34.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:34.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:34.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:34.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:34.536 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:34.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:34.558 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:34.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:34.659 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:34.680 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:34.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:34.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:34.782 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:34.804 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:34.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:34.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:34.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:34 smithi098 ceph-mon[97580]: pgmap v441: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:34.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:34 smithi098 ceph-mon[100914]: pgmap v441: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:34.868 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:34 smithi196 ceph-mon[84784]: pgmap v441: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:34.906 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:34.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:34.929 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:34.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:35.029 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:35.051 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:35.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:35.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:35.152 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:35.174 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:35.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:35.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:35.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-21T10:08:35.276 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:35.299 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:35.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:35.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:35.339 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:08:35.339 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:08:35.401 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:35.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:35.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:35.526 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:35.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:35.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:35.650 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:35.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:35.674 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:35.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:35.775 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:35.796 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:35.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:35.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:35.874 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:35.898 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:35.901 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:35.922 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:35.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:35.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:35.988 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:08:36.009 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:08:36.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:36.051 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:36.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:36.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:36.153 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:36.174 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:36.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:36.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:36.253 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:08:36.276 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:36.298 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:36.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:36.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:36.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:36.421 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:36.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:36.522 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:36.544 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:36.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:36.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:36.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:36 smithi196 ceph-mon[84784]: pgmap v442: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:36.646 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:36.667 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:36.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:36.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:36.769 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:36.791 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:36.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:36.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:36.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:36 smithi098 ceph-mon[100914]: pgmap v442: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:36.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:36 smithi098 ceph-mon[97580]: pgmap v442: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:36.892 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:36.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:36.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:37.016 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:37.038 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:37.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:37.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:37.140 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:37.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:37.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:37.263 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:37.285 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:37.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:37.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:37.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:37.409 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:37.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:37.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:37.511 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:37.533 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:37.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:37.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:37.634 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]} : dispatch 2024-09-21T10:08:37.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 5]} : dispatch 2024-09-21T10:08:37.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:37.635 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:37.657 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:37.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:37.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:37.759 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:37.781 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:37.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:37.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:37.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]} : dispatch 2024-09-21T10:08:37.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 5]} : dispatch 2024-09-21T10:08:37.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:37.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]} : dispatch 2024-09-21T10:08:37.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 5]} : dispatch 2024-09-21T10:08:37.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:37.854 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:08:37.854 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:08:37.883 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:37.905 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:37.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:37.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:38.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:38.028 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:38.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:38.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:38.130 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:38.152 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:38.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:38.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:38.253 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:38.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:38.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:38.377 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:38.399 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:38.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:38.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:38.500 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:38.522 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:38.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:38.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:38.564 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:08:38.624 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:38.645 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:38.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:38.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:38.747 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:38.768 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:38.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:38.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:38.812 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:08:38.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:38 smithi098 ceph-mon[97580]: pgmap v443: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:38.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]': finished 2024-09-21T10:08:38.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 5]}]': finished 2024-09-21T10:08:38.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:38 smithi098 ceph-mon[97580]: osdmap e163: 8 total, 8 up, 7 in 2024-09-21T10:08:38.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:38 smithi098 ceph-mon[100914]: pgmap v443: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:38.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]': finished 2024-09-21T10:08:38.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 5]}]': finished 2024-09-21T10:08:38.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:38 smithi098 ceph-mon[100914]: osdmap e163: 8 total, 8 up, 7 in 2024-09-21T10:08:38.870 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:38.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:38 smithi196 ceph-mon[84784]: pgmap v443: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]': finished 2024-09-21T10:08:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 5]}]': finished 2024-09-21T10:08:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:38 smithi196 ceph-mon[84784]: osdmap e163: 8 total, 8 up, 7 in 2024-09-21T10:08:38.892 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:38.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:38.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:38.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:39.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:39.016 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:39.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:39.117 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:39.139 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:39.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:39.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:39.240 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:39.262 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:39.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:39.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:39.364 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:39.385 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:39.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:39.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:39.487 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:39.509 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:39.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:39.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:39.586 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:39.593 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'unique_pool_1' created 2024-09-21T10:08:39.611 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:39.632 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:39.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:39.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:39.734 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:39.755 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:39.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:39.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:39.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:39 smithi098 ceph-mon[100914]: osdmap e164: 8 total, 8 up, 7 in 2024-09-21T10:08:39.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:39 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/449616247' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T10:08:39.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:39 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T10:08:39.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:39 smithi098 ceph-mon[97580]: osdmap e164: 8 total, 8 up, 7 in 2024-09-21T10:08:39.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:39 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/449616247' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T10:08:39.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:39 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T10:08:39.856 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:39.878 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:39.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:39.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:39 smithi196 ceph-mon[84784]: osdmap e164: 8 total, 8 up, 7 in 2024-09-21T10:08:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:39 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/449616247' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T10:08:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:39 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T10:08:39.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:40.004 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:40.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:40.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:40.106 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:40.128 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:40.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:40.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:40.229 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:40.252 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:40.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:40.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:40.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:40.366 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-21T10:08:40.375 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:40.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:40.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:40.385 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:08:40.386 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:08:40.478 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:40.501 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:40.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:40.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:40.597 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:08:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:08:40.215Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:08:40.598 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:08:40.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:40 smithi098 ceph-mon[97580]: pgmap v446: 73 pgs: 1 peering, 72 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:40.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:40 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-21T10:08:40.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:40 smithi098 ceph-mon[97580]: osdmap e165: 8 total, 8 up, 7 in 2024-09-21T10:08:40.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:40 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3245249298' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:08:40.601 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:40 smithi098 ceph-mon[100914]: pgmap v446: 73 pgs: 1 peering, 72 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:40.601 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:40 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-21T10:08:40.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:40.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:40.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:40.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:40.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:40.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:40.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:40.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:40 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:08:40.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:40 smithi098 ceph-mon[100914]: osdmap e165: 8 total, 8 up, 7 in 2024-09-21T10:08:40.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:40 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3245249298' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:08:40.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:40 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:08:40.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:08:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:08:40] "GET /metrics HTTP/1.1" 200 34092 "" "Prometheus/2.51.0" 2024-09-21T10:08:40.851 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:40.872 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:40.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:40.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:40.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:40 smithi196 ceph-mon[84784]: pgmap v446: 73 pgs: 1 peering, 72 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:40.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:40 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-21T10:08:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:40 smithi196 ceph-mon[84784]: osdmap e165: 8 total, 8 up, 7 in 2024-09-21T10:08:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:40 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3245249298' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:08:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:40 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:08:40.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:40.996 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:40.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:40.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:41.022 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:08:41.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:41.128 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:41.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:41.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:41.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:41.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:41.253 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:41.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:41.308 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:08:41.353 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:41.374 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:41.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:41.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:41.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:41.497 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:41.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:41.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:41.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:41.620 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:41.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:41.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:41.685 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:41 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:08:41.685 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:41 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:08:41.685 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:41 smithi196 ceph-mon[84784]: osdmap e166: 8 total, 8 up, 7 in 2024-09-21T10:08:41.721 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:41.742 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:41.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:41.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:41.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:41.854 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:41 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:08:41.854 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:41 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:08:41.854 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:41 smithi098 ceph-mon[97580]: osdmap e166: 8 total, 8 up, 7 in 2024-09-21T10:08:41.854 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:41 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:08:41.854 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:41 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:08:41.854 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:41 smithi098 ceph-mon[100914]: osdmap e166: 8 total, 8 up, 7 in 2024-09-21T10:08:41.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:41.866 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:41.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:41.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:41.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:41.990 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:41.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:42.091 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:42.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:42.113 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:42.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:42.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:42.235 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:42.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:42.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:42.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:42.357 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:42.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:42.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:42.459 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:42.481 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:42.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:42.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:42.583 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:42.604 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:42.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:42.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:42.705 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:42.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:42.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:42.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:42.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:42.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:42 smithi098 ceph-mon[97580]: pgmap v449: 89 pgs: 3 creating+activating, 1 creating+peering, 11 unknown, 1 peering, 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:42.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:42 smithi098 ceph-mon[97580]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:08:42.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:42 smithi098 ceph-mon[97580]: osdmap e167: 8 total, 8 up, 7 in 2024-09-21T10:08:42.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:42 smithi098 ceph-mon[100914]: pgmap v449: 89 pgs: 3 creating+activating, 1 creating+peering, 11 unknown, 1 peering, 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:42.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:42 smithi098 ceph-mon[100914]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:08:42.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:42 smithi098 ceph-mon[100914]: osdmap e167: 8 total, 8 up, 7 in 2024-09-21T10:08:42.851 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:42.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:42.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:42.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:42 smithi196 ceph-mon[84784]: pgmap v449: 89 pgs: 3 creating+activating, 1 creating+peering, 11 unknown, 1 peering, 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:42.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:42 smithi196 ceph-mon[84784]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:08:42.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:42 smithi196 ceph-mon[84784]: osdmap e167: 8 total, 8 up, 7 in 2024-09-21T10:08:42.953 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:42.960 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:42.974 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:42.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:42.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:43.078 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:43.095 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:08:43.095 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:08:43.100 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:43.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:43.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:43.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:43.223 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:43.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:43.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:43.325 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:43.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:43.347 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:43.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:43.448 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:43.470 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:43.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:43.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:43.571 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:43.592 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:43.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:43.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:43.694 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:43.715 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:43.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:43.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:43.815 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:08:43.817 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:43.838 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:43.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:43.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:43.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:08:43.940 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:43.961 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:43.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:43.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:44.061 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:08:44.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:44.083 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:44.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:44.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:44.187 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:44.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:44.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:44.311 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:44.332 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:44.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:44.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:44.434 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:44.456 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:44.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:44.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:44.558 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:44.579 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:44.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:44.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:44.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:44.702 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:44.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:44.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:44.803 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:44.825 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:44.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:44.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:44.846 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:44 smithi098 ceph-mon[97580]: pgmap v451: 89 pgs: 3 creating+activating, 1 creating+peering, 4 unknown, 1 peering, 80 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:44.846 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:44 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1458493721' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:44.846 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:08:44.846 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:08:44.847 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:08:44.847 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:08:44.847 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:08:44.847 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:08:44.847 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:08:44.847 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:08:44.847 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:08:44.847 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:08:44.847 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:08:44.847 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:08:44.848 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:08:44.848 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:08:44.848 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:08:44.848 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:08:44.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:44 smithi098 ceph-mon[100914]: pgmap v451: 89 pgs: 3 creating+activating, 1 creating+peering, 4 unknown, 1 peering, 80 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:44.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:44 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1458493721' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:44.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:44 smithi196 ceph-mon[84784]: pgmap v451: 89 pgs: 3 creating+activating, 1 creating+peering, 4 unknown, 1 peering, 80 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:44.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:44 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1458493721' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:44.927 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:44.949 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:44.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:44.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:45.051 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:45.073 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:45.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:45.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:45.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:45.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:45.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:45.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:45.319 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:45.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:45.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:45.421 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:45.443 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:45.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:45.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:45.544 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:45.566 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:45.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:45.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:45.628 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:08:45.628 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:08:45.628 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-21T10:08:45.628 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-21T10:08:45.628 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:08:45.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:45.689 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:45.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:45.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:45.790 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:45.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:45.813 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:45.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:45.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-21T10:08:45.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:45.935 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:45.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:45.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:46.037 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:46.058 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:46.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:46.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:46.159 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:46.181 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:46.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:46.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:46.254 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:08:46.282 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:46.318 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:46.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:46.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:46.380 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:46.419 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:46.441 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:46.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:46.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:46.498 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:08:46.543 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:46.563 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:46.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:46.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:46.610 INFO:teuthology.orchestra.run.smithi098.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-21T10:08:46.665 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:46.687 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:46.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:46.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:46.789 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:46.811 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:46.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:46.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:46.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:46 smithi098 ceph-mon[100914]: pgmap v452: 89 pgs: 3 creating+activating, 86 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:46.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:46 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1313765036' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:08:46.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:46 smithi098 ceph-mon[97580]: pgmap v452: 89 pgs: 3 creating+activating, 86 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:46.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:46 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1313765036' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:08:46.858 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:46 smithi196 ceph-mon[84784]: pgmap v452: 89 pgs: 3 creating+activating, 86 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:46.858 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:46 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1313765036' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:08:46.880 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:46.912 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:46.933 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:46.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:46.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:47.035 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:47.056 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:47.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:47.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:47.158 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:47.180 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:47.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:47.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:47.281 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:47.303 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:47.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:47.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:47.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:47.427 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:47.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:47.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:47.433 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:08:47.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:47.550 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:47.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:47.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:47.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:47.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:47.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:47.684 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:47 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1313765036' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:08:47.684 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:47 smithi098 ceph-mon[97580]: osdmap e168: 8 total, 8 up, 7 in 2024-09-21T10:08:47.684 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:47 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1313765036' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:08:47.684 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:47 smithi098 ceph-mon[100914]: osdmap e168: 8 total, 8 up, 7 in 2024-09-21T10:08:47.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:47.796 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:47.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:47.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:47.860 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:47 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1313765036' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:08:47.860 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:47 smithi196 ceph-mon[84784]: osdmap e168: 8 total, 8 up, 7 in 2024-09-21T10:08:47.898 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:47.919 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:47.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:47.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:48.020 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:48.041 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:48.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:48.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:48.070 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:08:48.070 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-21T10:08:48.071 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-21T10:08:48.071 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:08:48.071 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.071 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:31182 5.7 5:ee38202f:::benchmark_data_smithi098_135187_object31181:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:08:48.071 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:03:59.633733+0000", 2024-09-21T10:08:48.071 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 288.43357658299999, 2024-09-21T10:08:48.071 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.076961207000000004, 2024-09-21T10:08:48.071 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:08:48.071 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:48.071 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:08:48.071 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:08:48.072 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:48.072 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 31182 2024-09-21T10:08:48.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.072 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:08:48.072 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.072 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:08:48.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.633733+0000", 2024-09-21T10:08:48.072 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.072 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.072 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:08:48.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.633733+0000", 2024-09-21T10:08:48.072 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.073 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.073 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:08:48.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.633733+0000", 2024-09-21T10:08:48.073 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:08:48.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.073 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.073 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:08:48.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.633790+0000", 2024-09-21T10:08:48.073 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7463999999999997e-05 2024-09-21T10:08:48.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.073 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.073 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:08:48.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.633800+0000", 2024-09-21T10:08:48.073 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0107e-05 2024-09-21T10:08:48.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.074 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:08:48.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.633804+0000", 2024-09-21T10:08:48.074 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9110000000000003e-06 2024-09-21T10:08:48.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.074 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:08:48.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.633838+0000", 2024-09-21T10:08:48.074 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4455999999999997e-05 2024-09-21T10:08:48.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.074 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:08:48.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.633917+0000", 2024-09-21T10:08:48.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8064000000000003e-05 2024-09-21T10:08:48.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.075 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.075 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:08:48.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.633997+0000", 2024-09-21T10:08:48.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0835999999999997e-05 2024-09-21T10:08:48.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.075 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.075 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:08:48.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710482+0000", 2024-09-21T10:08:48.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.076484416 2024-09-21T10:08:48.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.075 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.075 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:48.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710515+0000", 2024-09-21T10:08:48.076 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3464999999999999e-05 2024-09-21T10:08:48.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.076 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.076 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:48.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710610+0000", 2024-09-21T10:08:48.076 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4394000000000005e-05 2024-09-21T10:08:48.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.076 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.076 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:08:48.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710661+0000", 2024-09-21T10:08:48.076 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1320999999999999e-05 2024-09-21T10:08:48.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.077 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.077 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:08:48.077 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710695+0000", 2024-09-21T10:08:48.077 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3763999999999997e-05 2024-09-21T10:08:48.077 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.077 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:08:48.077 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.077 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.077 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.44894.0:31182 5.7 e110/105 ondisk, result = 0)", 2024-09-21T10:08:48.077 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:03:59.635169+0000", 2024-09-21T10:08:48.077 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 288.43214114199998, 2024-09-21T10:08:48.077 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.075348685999999998, 2024-09-21T10:08:48.078 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:08:48.078 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:08:48.078 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:08:48.078 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.078 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:08:48.078 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.635169+0000", 2024-09-21T10:08:48.078 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.078 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.078 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:08:48.078 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.635169+0000", 2024-09-21T10:08:48.078 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.079 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.079 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:08:48.079 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.635168+0000", 2024-09-21T10:08:48.079 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:48.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.079 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.079 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:08:48.079 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.635175+0000", 2024-09-21T10:08:48.079 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.199e-06 2024-09-21T10:08:48.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.079 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.079 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:08:48.079 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710027+0000", 2024-09-21T10:08:48.079 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.074851493000000005 2024-09-21T10:08:48.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.080 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.080 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:08:48.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710049+0000", 2024-09-21T10:08:48.080 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1976000000000001e-05 2024-09-21T10:08:48.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.080 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.080 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:08:48.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710502+0000", 2024-09-21T10:08:48.080 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00045345299999999998 2024-09-21T10:08:48.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.080 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.080 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:08:48.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710510+0000", 2024-09-21T10:08:48.080 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2919999999999998e-06 2024-09-21T10:08:48.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.081 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.081 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:08:48.081 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710518+0000", 2024-09-21T10:08:48.081 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9400000000000002e-06 2024-09-21T10:08:48.081 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.081 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:08:48.081 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.081 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.081 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.44894.0:31182 5.7 e110/105 ondisk, result = 0)", 2024-09-21T10:08:48.081 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:03:59.635173+0000", 2024-09-21T10:08:48.081 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 288.432137522, 2024-09-21T10:08:48.081 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.075527768999999995, 2024-09-21T10:08:48.081 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:08:48.082 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:08:48.082 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:08:48.082 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.082 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:08:48.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.635173+0000", 2024-09-21T10:08:48.082 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.082 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.082 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:08:48.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.635173+0000", 2024-09-21T10:08:48.082 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.082 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.082 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:08:48.083 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.635172+0000", 2024-09-21T10:08:48.083 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:48.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.083 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.083 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:08:48.083 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.635176+0000", 2024-09-21T10:08:48.083 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8800000000000001e-06 2024-09-21T10:08:48.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.083 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.083 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:08:48.083 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710538+0000", 2024-09-21T10:08:48.083 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.075362506999999995 2024-09-21T10:08:48.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.083 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.084 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:08:48.084 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710544+0000", 2024-09-21T10:08:48.084 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9490000000000001e-06 2024-09-21T10:08:48.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.084 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.084 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:08:48.084 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710599+0000", 2024-09-21T10:08:48.084 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4589000000000002e-05 2024-09-21T10:08:48.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.084 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.084 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:08:48.084 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710605+0000", 2024-09-21T10:08:48.084 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8889999999999999e-06 2024-09-21T10:08:48.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.084 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.085 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:08:48.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:03:59.710700+0000", 2024-09-21T10:08:48.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5546999999999998e-05 2024-09-21T10:08:48.086 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.086 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:08:48.086 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.086 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:72280 5.d e112/105 5:b2fd55dd:::benchmark_data_smithi098_135187_object72279:head v 112'4469, mlcod=112'4466)", 2024-09-21T10:08:48.086 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:22.600438+0000", 2024-09-21T10:08:48.086 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 265.46687173700002, 2024-09-21T10:08:48.086 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.072455923000000005, 2024-09-21T10:08:48.086 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:08:48.086 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:48.086 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:08:48.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.087 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:08:48.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.600438+0000", 2024-09-21T10:08:48.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.087 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.087 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:08:48.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.600438+0000", 2024-09-21T10:08:48.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.087 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.087 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:08:48.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.600438+0000", 2024-09-21T10:08:48.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:48.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:08:48.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.600545+0000", 2024-09-21T10:08:48.088 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010741799999999999 2024-09-21T10:08:48.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:08:48.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.600555+0000", 2024-09-21T10:08:48.088 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0246e-05 2024-09-21T10:08:48.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:08:48.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.600560+0000", 2024-09-21T10:08:48.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9019999999999998e-06 2024-09-21T10:08:48.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.089 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.089 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:08:48.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.600612+0000", 2024-09-21T10:08:48.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1431999999999998e-05 2024-09-21T10:08:48.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.089 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.089 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:08:48.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.600631+0000", 2024-09-21T10:08:48.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8896999999999999e-05 2024-09-21T10:08:48.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.089 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.089 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:08:48.090 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.672852+0000", 2024-09-21T10:08:48.090 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.072221271000000004 2024-09-21T10:08:48.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.090 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.090 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:08:48.090 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.672894+0000", 2024-09-21T10:08:48.090 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2336999999999998e-05 2024-09-21T10:08:48.090 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.090 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:08:48.090 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.090 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.090 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:72288 5.2 e112/105 5:4f79b5b0:::benchmark_data_smithi098_135187_object72287:head v 112'4575, mlcod=112'4573)", 2024-09-21T10:08:48.090 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:22.602868+0000", 2024-09-21T10:08:48.091 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 265.46444252999999, 2024-09-21T10:08:48.091 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.072995333999999995, 2024-09-21T10:08:48.091 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:08:48.091 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:48.091 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:08:48.091 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.091 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:08:48.091 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602868+0000", 2024-09-21T10:08:48.091 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.091 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.091 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:08:48.091 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602868+0000", 2024-09-21T10:08:48.091 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.092 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.092 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:08:48.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602867+0000", 2024-09-21T10:08:48.092 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:48.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.092 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.092 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:08:48.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602900+0000", 2024-09-21T10:08:48.092 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3535999999999999e-05 2024-09-21T10:08:48.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.092 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.092 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:08:48.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602911+0000", 2024-09-21T10:08:48.092 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0604e-05 2024-09-21T10:08:48.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:08:48.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602917+0000", 2024-09-21T10:08:48.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7270000000000004e-06 2024-09-21T10:08:48.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:08:48.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.604496+0000", 2024-09-21T10:08:48.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015789479999999999 2024-09-21T10:08:48.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:08:48.094 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.604514+0000", 2024-09-21T10:08:48.094 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7889000000000001e-05 2024-09-21T10:08:48.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.094 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.094 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:08:48.094 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.675675+0000", 2024-09-21T10:08:48.094 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.071161665999999998 2024-09-21T10:08:48.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.094 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.094 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:08:48.094 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.675863+0000", 2024-09-21T10:08:48.094 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00018764000000000001 2024-09-21T10:08:48.094 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.094 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:08:48.095 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.095 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.095 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:72290 5.b e112/105 5:d51677e2:::benchmark_data_smithi098_135187_object72289:head v 112'4480, mlcod=112'4478)", 2024-09-21T10:08:48.095 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:22.602955+0000", 2024-09-21T10:08:48.095 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 265.464354738, 2024-09-21T10:08:48.095 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.068283890999999999, 2024-09-21T10:08:48.095 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:08:48.095 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:48.095 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:08:48.095 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.095 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:08:48.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602955+0000", 2024-09-21T10:08:48.095 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.096 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:08:48.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602955+0000", 2024-09-21T10:08:48.096 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.096 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:08:48.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602955+0000", 2024-09-21T10:08:48.096 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:48.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.096 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:08:48.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.603028+0000", 2024-09-21T10:08:48.097 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3514000000000001e-05 2024-09-21T10:08:48.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.097 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:08:48.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.603038+0000", 2024-09-21T10:08:48.097 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0149000000000001e-05 2024-09-21T10:08:48.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.097 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:08:48.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.603044+0000", 2024-09-21T10:08:48.097 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.429e-06 2024-09-21T10:08:48.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.097 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:08:48.098 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.603097+0000", 2024-09-21T10:08:48.098 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2882e-05 2024-09-21T10:08:48.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.098 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.098 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:08:48.098 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.603114+0000", 2024-09-21T10:08:48.098 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7462999999999999e-05 2024-09-21T10:08:48.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.098 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.098 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:08:48.098 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671172+0000", 2024-09-21T10:08:48.098 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.068057965999999998 2024-09-21T10:08:48.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.098 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.098 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:08:48.099 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671239+0000", 2024-09-21T10:08:48.099 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7144000000000006e-05 2024-09-21T10:08:48.099 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.099 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:08:48.099 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.099 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.099 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:72293 5.7 5:e294dae0:::benchmark_data_smithi098_135187_object72292:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T10:08:48.099 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:22.605214+0000", 2024-09-21T10:08:48.099 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 265.46209611699999, 2024-09-21T10:08:48.099 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.069737357999999999, 2024-09-21T10:08:48.099 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:08:48.099 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:48.100 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:08:48.100 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:08:48.100 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:48.100 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 72293 2024-09-21T10:08:48.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.100 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:08:48.100 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.100 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:08:48.100 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605214+0000", 2024-09-21T10:08:48.100 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.100 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.100 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:08:48.100 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605214+0000", 2024-09-21T10:08:48.100 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.101 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.101 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:08:48.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605212+0000", 2024-09-21T10:08:48.101 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:08:48.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.101 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.101 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:08:48.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605253+0000", 2024-09-21T10:08:48.101 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0808e-05 2024-09-21T10:08:48.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.101 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.101 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:08:48.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605267+0000", 2024-09-21T10:08:48.102 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3789e-05 2024-09-21T10:08:48.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.102 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:08:48.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605273+0000", 2024-09-21T10:08:48.102 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0349999999999998e-06 2024-09-21T10:08:48.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.102 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:08:48.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605322+0000", 2024-09-21T10:08:48.102 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9478000000000003e-05 2024-09-21T10:08:48.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.103 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:08:48.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.672025+0000", 2024-09-21T10:08:48.103 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.066702556999999996 2024-09-21T10:08:48.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.103 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.103 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:08:48.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.672152+0000", 2024-09-21T10:08:48.103 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000126744 2024-09-21T10:08:48.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.103 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.103 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:48.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.674802+0000", 2024-09-21T10:08:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026499610000000002 2024-09-21T10:08:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.674827+0000", 2024-09-21T10:08:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5786999999999999e-05 2024-09-21T10:08:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:08:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.674853+0000", 2024-09-21T10:08:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5184000000000001e-05 2024-09-21T10:08:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:08:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.674905+0000", 2024-09-21T10:08:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2868000000000003e-05 2024-09-21T10:08:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:08:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.674951+0000", 2024-09-21T10:08:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5840000000000002e-05 2024-09-21T10:08:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:08:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:72292 5.4 e112/105 5:2f3c813f:::benchmark_data_smithi098_135187_object72291:head v 112'4596, mlcod=112'4594)", 2024-09-21T10:08:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:22.605465+0000", 2024-09-21T10:08:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 265.461844971, 2024-09-21T10:08:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.068218862000000005, 2024-09-21T10:08:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:08:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:08:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:08:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605465+0000", 2024-09-21T10:08:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:08:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605465+0000", 2024-09-21T10:08:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:08:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605464+0000", 2024-09-21T10:08:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:08:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:08:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605499+0000", 2024-09-21T10:08:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5029e-05 2024-09-21T10:08:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:08:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.670988+0000", 2024-09-21T10:08:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.065489107000000005 2024-09-21T10:08:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:08:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.670998+0000", 2024-09-21T10:08:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5079999999999994e-06 2024-09-21T10:08:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:08:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671042+0000", 2024-09-21T10:08:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3835000000000003e-05 2024-09-21T10:08:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:08:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671063+0000", 2024-09-21T10:08:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0953000000000001e-05 2024-09-21T10:08:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:08:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.673594+0000", 2024-09-21T10:08:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00253104 2024-09-21T10:08:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:08:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.673684+0000", 2024-09-21T10:08:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0270000000000001e-05 2024-09-21T10:08:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:08:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:72289 5.2 e112/105 5:46d682ec:::benchmark_data_smithi098_135187_object72288:head v 112'4576, mlcod=112'4573)", 2024-09-21T10:08:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:22.605676+0000", 2024-09-21T10:08:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 265.46163375100002, 2024-09-21T10:08:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.070263872000000005, 2024-09-21T10:08:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:08:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:08:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:08:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605676+0000", 2024-09-21T10:08:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:08:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605676+0000", 2024-09-21T10:08:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:08:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605676+0000", 2024-09-21T10:08:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:08:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605698+0000", 2024-09-21T10:08:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2269e-05 2024-09-21T10:08:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:08:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.670902+0000", 2024-09-21T10:08:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.065204256000000002 2024-09-21T10:08:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:08:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.670916+0000", 2024-09-21T10:08:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3926e-05 2024-09-21T10:08:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:08:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671088+0000", 2024-09-21T10:08:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017191799999999999 2024-09-21T10:08:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:08:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.671107+0000", 2024-09-21T10:08:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8627999999999999e-05 2024-09-21T10:08:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:08:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.675867+0000", 2024-09-21T10:08:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004760673 2024-09-21T10:08:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:08:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.675940+0000", 2024-09-21T10:08:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2798000000000003e-05 2024-09-21T10:08:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:08:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:85709 5.7 5:e34e5457:::benchmark_data_smithi098_135187_object85708:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T10:08:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:32.593458+0000", 2024-09-21T10:08:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 255.47385184500001, 2024-09-21T10:08:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.069336535000000005, 2024-09-21T10:08:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:08:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:48.117 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:08:48.117 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:08:48.117 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:48.117 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 85709 2024-09-21T10:08:48.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:08:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:08:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.593458+0000", 2024-09-21T10:08:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:08:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.593458+0000", 2024-09-21T10:08:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:08:48.119 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.593456+0000", 2024-09-21T10:08:48.119 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:08:48.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.119 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.119 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:08:48.119 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.593487+0000", 2024-09-21T10:08:48.119 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0490000000000001e-05 2024-09-21T10:08:48.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.119 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.119 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:08:48.119 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.593497+0000", 2024-09-21T10:08:48.119 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8290000000000003e-06 2024-09-21T10:08:48.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.119 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.120 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:08:48.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.593501+0000", 2024-09-21T10:08:48.120 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6310000000000002e-06 2024-09-21T10:08:48.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.120 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.120 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:08:48.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.594950+0000", 2024-09-21T10:08:48.120 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001449103 2024-09-21T10:08:48.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.120 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.120 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:08:48.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.595037+0000", 2024-09-21T10:08:48.120 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6341999999999994e-05 2024-09-21T10:08:48.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.120 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:08:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.595123+0000", 2024-09-21T10:08:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5798999999999996e-05 2024-09-21T10:08:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:08:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.597972+0000", 2024-09-21T10:08:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0028492550000000002 2024-09-21T10:08:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598495+0000", 2024-09-21T10:08:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00052359900000000003 2024-09-21T10:08:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.662715+0000", 2024-09-21T10:08:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.064219429999999994 2024-09-21T10:08:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:08:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.662762+0000", 2024-09-21T10:08:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7241999999999999e-05 2024-09-21T10:08:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:08:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.662795+0000", 2024-09-21T10:08:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2675999999999997e-05 2024-09-21T10:08:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:08:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:85713 5.b e114/105 5:d62e5ba1:::benchmark_data_smithi098_135187_object85712:head v 114'5314, mlcod=114'5312)", 2024-09-21T10:08:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:32.597344+0000", 2024-09-21T10:08:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 255.46996560299999, 2024-09-21T10:08:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.069982381999999996, 2024-09-21T10:08:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:08:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:08:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:08:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.597344+0000", 2024-09-21T10:08:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:08:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.597344+0000", 2024-09-21T10:08:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:08:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.597344+0000", 2024-09-21T10:08:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:08:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.597398+0000", 2024-09-21T10:08:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.469e-05 2024-09-21T10:08:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:08:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.597406+0000", 2024-09-21T10:08:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8779999999999998e-06 2024-09-21T10:08:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:08:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.597412+0000", 2024-09-21T10:08:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6300000000000003e-06 2024-09-21T10:08:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:08:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.597449+0000", 2024-09-21T10:08:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7066000000000001e-05 2024-09-21T10:08:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:08:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.597466+0000", 2024-09-21T10:08:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6684000000000002e-05 2024-09-21T10:08:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:08:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.667142+0000", 2024-09-21T10:08:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.069675916000000004 2024-09-21T10:08:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:08:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.667327+0000", 2024-09-21T10:08:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00018519399999999999 2024-09-21T10:08:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:08:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:85711 5.d e114/105 5:b14dccba:::benchmark_data_smithi098_135187_object85710:head v 114'5346, mlcod=114'5343)", 2024-09-21T10:08:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:32.598436+0000", 2024-09-21T10:08:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 255.46887370300001, 2024-09-21T10:08:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.076763213999999996, 2024-09-21T10:08:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:08:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:08:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:08:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598436+0000", 2024-09-21T10:08:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:08:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598436+0000", 2024-09-21T10:08:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:08:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598436+0000", 2024-09-21T10:08:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:08:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598512+0000", 2024-09-21T10:08:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5797000000000006e-05 2024-09-21T10:08:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:08:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598521+0000", 2024-09-21T10:08:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4040000000000001e-06 2024-09-21T10:08:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:08:48.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598526+0000", 2024-09-21T10:08:48.130 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6290000000000001e-06 2024-09-21T10:08:48.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.130 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:08:48.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598554+0000", 2024-09-21T10:08:48.130 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8401000000000001e-05 2024-09-21T10:08:48.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.130 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:08:48.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598565+0000", 2024-09-21T10:08:48.130 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0934000000000001e-05 2024-09-21T10:08:48.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.131 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:08:48.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.675162+0000", 2024-09-21T10:08:48.131 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.076597255000000003 2024-09-21T10:08:48.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.131 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.131 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:08:48.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.675200+0000", 2024-09-21T10:08:48.131 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7355999999999999e-05 2024-09-21T10:08:48.131 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.132 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:08:48.132 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.132 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.132 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:85715 5.3 e114/105 5:cce5decf:::benchmark_data_smithi098_135187_object85714:head v 114'5295, mlcod=114'5293)", 2024-09-21T10:08:48.132 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:32.598545+0000", 2024-09-21T10:08:48.132 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 255.46876499499999, 2024-09-21T10:08:48.132 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.074556140000000007, 2024-09-21T10:08:48.132 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:08:48.132 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:48.132 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:08:48.132 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.132 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:08:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598545+0000", 2024-09-21T10:08:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:08:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598545+0000", 2024-09-21T10:08:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:08:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598545+0000", 2024-09-21T10:08:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:08:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598576+0000", 2024-09-21T10:08:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1591000000000003e-05 2024-09-21T10:08:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:08:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598583+0000", 2024-09-21T10:08:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8700000000000003e-06 2024-09-21T10:08:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:08:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.598587+0000", 2024-09-21T10:08:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5379999999999998e-06 2024-09-21T10:08:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:08:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.661994+0000", 2024-09-21T10:08:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.063406792000000003 2024-09-21T10:08:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:08:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.662013+0000", 2024-09-21T10:08:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9638000000000001e-05 2024-09-21T10:08:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:08:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.673004+0000", 2024-09-21T10:08:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010991288 2024-09-21T10:08:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:08:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:32.673101+0000", 2024-09-21T10:08:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6778000000000001e-05 2024-09-21T10:08:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:08:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:159919 5.3 e124/105 5:c765c100:::benchmark_data_smithi098_135187_object24281:head v 124'9916, mlcod=124'9913)", 2024-09-21T10:08:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:05:46.491193+0000", 2024-09-21T10:08:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 181.576117557, 2024-09-21T10:08:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.085250303999999999, 2024-09-21T10:08:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:08:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:08:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:08:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491193+0000", 2024-09-21T10:08:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:08:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491193+0000", 2024-09-21T10:08:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:08:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491192+0000", 2024-09-21T10:08:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:08:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491197+0000", 2024-09-21T10:08:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2580000000000003e-06 2024-09-21T10:08:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:08:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491204+0000", 2024-09-21T10:08:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7900000000000002e-06 2024-09-21T10:08:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:08:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491207+0000", 2024-09-21T10:08:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3220000000000001e-06 2024-09-21T10:08:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:08:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491237+0000", 2024-09-21T10:08:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9791999999999999e-05 2024-09-21T10:08:48.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.140 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.140 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:08:48.140 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491250+0000", 2024-09-21T10:08:48.140 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2775999999999999e-05 2024-09-21T10:08:48.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.140 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.140 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:08:48.140 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.576210+0000", 2024-09-21T10:08:48.140 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.084960182999999995 2024-09-21T10:08:48.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.140 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.140 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:08:48.140 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.576443+0000", 2024-09-21T10:08:48.140 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000232745 2024-09-21T10:08:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:08:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.44894.0:159921 5.b e124/105 5:d56ee90f:::benchmark_data_smithi098_135187_object24283:head v 124'9893, mlcod=124'9891)", 2024-09-21T10:08:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:05:46.491221+0000", 2024-09-21T10:08:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 181.576089, 2024-09-21T10:08:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.087278346000000007, 2024-09-21T10:08:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:08:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:08:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:08:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491221+0000", 2024-09-21T10:08:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:08:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491221+0000", 2024-09-21T10:08:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:08:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491221+0000", 2024-09-21T10:08:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:08:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491223+0000", 2024-09-21T10:08:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7209999999999999e-06 2024-09-21T10:08:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:08:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491230+0000", 2024-09-21T10:08:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5749999999999997e-06 2024-09-21T10:08:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:08:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491233+0000", 2024-09-21T10:08:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4020000000000002e-06 2024-09-21T10:08:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:48.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.145 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:08:48.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491261+0000", 2024-09-21T10:08:48.145 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7583e-05 2024-09-21T10:08:48.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.145 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:08:48.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.491275+0000", 2024-09-21T10:08:48.145 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3601000000000001e-05 2024-09-21T10:08:48.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.146 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.146 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:08:48.146 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.578357+0000", 2024-09-21T10:08:48.146 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.087082076999999994 2024-09-21T10:08:48.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.146 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.146 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:08:48.146 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:05:46.578499+0000", 2024-09-21T10:08:48.146 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014272699999999999 2024-09-21T10:08:48.146 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.146 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:08:48.146 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.147 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.147 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:194056 5.7 5:ec9f14de:::benchmark_data_smithi098_135187_object58418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-21T10:08:48.147 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:06:27.585042+0000", 2024-09-21T10:08:48.147 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 140.48226849299999, 2024-09-21T10:08:48.147 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.068117258999999999, 2024-09-21T10:08:48.147 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:08:48.147 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:48.147 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:08:48.147 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:08:48.147 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:48.147 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 194056 2024-09-21T10:08:48.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.147 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:08:48.148 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.148 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:08:48.148 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.585042+0000", 2024-09-21T10:08:48.148 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.148 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.148 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:08:48.148 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.585042+0000", 2024-09-21T10:08:48.148 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.148 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.148 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:08:48.148 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.585041+0000", 2024-09-21T10:08:48.148 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:48.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.149 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:08:48.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.585044+0000", 2024-09-21T10:08:48.149 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4650000000000001e-06 2024-09-21T10:08:48.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.149 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:08:48.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.585050+0000", 2024-09-21T10:08:48.149 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0889999999999996e-06 2024-09-21T10:08:48.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.149 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:08:48.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.585053+0000", 2024-09-21T10:08:48.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5659999999999998e-06 2024-09-21T10:08:48.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:08:48.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.612253+0000", 2024-09-21T10:08:48.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.027199244000000001 2024-09-21T10:08:48.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:08:48.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.612602+0000", 2024-09-21T10:08:48.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000349701 2024-09-21T10:08:48.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:08:48.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.613553+0000", 2024-09-21T10:08:48.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00095090300000000001 2024-09-21T10:08:48.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:48.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.633846+0000", 2024-09-21T10:08:48.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.020293234 2024-09-21T10:08:48.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:48.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.634289+0000", 2024-09-21T10:08:48.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00044277599999999998 2024-09-21T10:08:48.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.152 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:08:48.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.653086+0000", 2024-09-21T10:08:48.152 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018796694999999999 2024-09-21T10:08:48.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.152 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.152 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:08:48.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.653120+0000", 2024-09-21T10:08:48.152 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4156000000000003e-05 2024-09-21T10:08:48.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.152 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.152 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:08:48.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:27.653159+0000", 2024-09-21T10:08:48.152 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8853000000000002e-05 2024-09-21T10:08:48.152 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.153 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:08:48.153 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.153 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.153 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:194896 5.f 5:f58092b2:::benchmark_data_smithi098_135187_object59258:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-21T10:08:48.153 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:06:28.722935+0000", 2024-09-21T10:08:48.153 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 139.344374905, 2024-09-21T10:08:48.153 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.071991587999999995, 2024-09-21T10:08:48.153 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:08:48.153 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:48.153 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:08:48.153 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:08:48.153 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 194896 2024-09-21T10:08:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:08:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:08:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.722935+0000", 2024-09-21T10:08:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:08:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.722935+0000", 2024-09-21T10:08:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:08:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.722935+0000", 2024-09-21T10:08:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:08:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.722937+0000", 2024-09-21T10:08:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5490000000000001e-06 2024-09-21T10:08:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:08:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.722943+0000", 2024-09-21T10:08:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6110000000000001e-06 2024-09-21T10:08:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:08:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.722946+0000", 2024-09-21T10:08:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3510000000000002e-06 2024-09-21T10:08:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:08:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.730512+0000", 2024-09-21T10:08:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0075656780000000002 2024-09-21T10:08:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:08:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.730626+0000", 2024-09-21T10:08:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114048 2024-09-21T10:08:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:08:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.730741+0000", 2024-09-21T10:08:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000115182 2024-09-21T10:08:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:08:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.762402+0000", 2024-09-21T10:08:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.031660961000000001 2024-09-21T10:08:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.778010+0000", 2024-09-21T10:08:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015607789 2024-09-21T10:08:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.794789+0000", 2024-09-21T10:08:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016779565 2024-09-21T10:08:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:08:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.794884+0000", 2024-09-21T10:08:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4233000000000004e-05 2024-09-21T10:08:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:08:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.794927+0000", 2024-09-21T10:08:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3043000000000001e-05 2024-09-21T10:08:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:08:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:194901 5.f 5:f81e5796:::benchmark_data_smithi098_135187_object59263:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-21T10:08:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:06:28.722950+0000", 2024-09-21T10:08:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 139.34435989799999, 2024-09-21T10:08:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.072126886000000001, 2024-09-21T10:08:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:08:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:08:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:08:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 194901 2024-09-21T10:08:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:08:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:08:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.722950+0000", 2024-09-21T10:08:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:08:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.722950+0000", 2024-09-21T10:08:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:08:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.722950+0000", 2024-09-21T10:08:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:08:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.722952+0000", 2024-09-21T10:08:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0559999999999999e-06 2024-09-21T10:08:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:08:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.722957+0000", 2024-09-21T10:08:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9130000000000002e-06 2024-09-21T10:08:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:08:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.722960+0000", 2024-09-21T10:08:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9900000000000002e-06 2024-09-21T10:08:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:08:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.735822+0000", 2024-09-21T10:08:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012861895 2024-09-21T10:08:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:08:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.735940+0000", 2024-09-21T10:08:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000117831 2024-09-21T10:08:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:08:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.736049+0000", 2024-09-21T10:08:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109731 2024-09-21T10:08:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:08:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.762447+0000", 2024-09-21T10:08:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.026398049999999999 2024-09-21T10:08:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.778138+0000", 2024-09-21T10:08:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015690780000000001 2024-09-21T10:08:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.794967+0000", 2024-09-21T10:08:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016828803999999999 2024-09-21T10:08:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:08:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.795041+0000", 2024-09-21T10:08:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4259999999999997e-05 2024-09-21T10:08:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:08:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:28.795077+0000", 2024-09-21T10:08:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5868000000000003e-05 2024-09-21T10:08:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:08:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:251633 5.f 5:f4226135:::benchmark_data_smithi098_135187_object115995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:08:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:07:39.913062+0000", 2024-09-21T10:08:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 68.154248401999993, 2024-09-21T10:08:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.068570831999999998, 2024-09-21T10:08:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:08:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:08:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:08:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 251633 2024-09-21T10:08:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:08:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:08:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.913062+0000", 2024-09-21T10:08:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:08:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.913062+0000", 2024-09-21T10:08:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:08:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.913061+0000", 2024-09-21T10:08:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:08:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.913064+0000", 2024-09-21T10:08:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.796e-06 2024-09-21T10:08:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:08:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.913070+0000", 2024-09-21T10:08:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8889999999999999e-06 2024-09-21T10:08:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:08:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.913073+0000", 2024-09-21T10:08:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5389999999999999e-06 2024-09-21T10:08:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:08:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.926553+0000", 2024-09-21T10:08:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013479461 2024-09-21T10:08:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:08:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.926657+0000", 2024-09-21T10:08:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010414800000000001 2024-09-21T10:08:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:08:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.926760+0000", 2024-09-21T10:08:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000102905 2024-09-21T10:08:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.948196+0000", 2024-09-21T10:08:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.021436060999999999 2024-09-21T10:08:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:08:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.961034+0000", 2024-09-21T10:08:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0128378 2024-09-21T10:08:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.981578+0000", 2024-09-21T10:08:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.020543817999999998 2024-09-21T10:08:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:08:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.981609+0000", 2024-09-21T10:08:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1609999999999997e-05 2024-09-21T10:08:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:08:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.981633+0000", 2024-09-21T10:08:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3224999999999999e-05 2024-09-21T10:08:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:08:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:251635 5.f 5:fbaa8734:::benchmark_data_smithi098_135187_object115997:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:08:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:07:39.913080+0000", 2024-09-21T10:08:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 68.154230233000007, 2024-09-21T10:08:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.068608767000000001, 2024-09-21T10:08:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:08:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:08:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:08:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:08:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 251635 2024-09-21T10:08:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:08:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:08:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.913080+0000", 2024-09-21T10:08:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:08:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.913080+0000", 2024-09-21T10:08:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:08:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.913080+0000", 2024-09-21T10:08:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:08:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.913082+0000", 2024-09-21T10:08:48.174 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0609999999999998e-06 2024-09-21T10:08:48.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.174 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.174 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:08:48.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.913086+0000", 2024-09-21T10:08:48.174 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8060000000000002e-06 2024-09-21T10:08:48.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.174 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.174 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:08:48.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.913089+0000", 2024-09-21T10:08:48.174 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8899999999999999e-06 2024-09-21T10:08:48.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.174 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.175 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:08:48.175 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.932333+0000", 2024-09-21T10:08:48.175 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019243448 2024-09-21T10:08:48.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.175 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.175 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:08:48.175 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.932437+0000", 2024-09-21T10:08:48.175 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010468599999999999 2024-09-21T10:08:48.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.175 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.175 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:08:48.175 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.932544+0000", 2024-09-21T10:08:48.175 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010611 2024-09-21T10:08:48.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.176 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.176 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:48.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.948236+0000", 2024-09-21T10:08:48.176 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015692676999999999 2024-09-21T10:08:48.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.176 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.176 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:08:48.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.961159+0000", 2024-09-21T10:08:48.176 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012922942 2024-09-21T10:08:48.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.176 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.176 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:48.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.981651+0000", 2024-09-21T10:08:48.176 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.020492291999999999 2024-09-21T10:08:48.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.177 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.177 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:08:48.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.981672+0000", 2024-09-21T10:08:48.177 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0741e-05 2024-09-21T10:08:48.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:48.177 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:48.177 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:08:48.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:39.981689+0000", 2024-09-21T10:08:48.177 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6435000000000001e-05 2024-09-21T10:08:48.177 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.177 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:08:48.177 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.177 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:48.177 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:08:48.178 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:08:48.198 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:48.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:48.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:48.300 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:48.321 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:48.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:48.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:48.332 DEBUG:teuthology.orchestra.run.smithi098:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-21T10:08:48.334 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:08:48.334 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:08:48.379 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:hints = 1 2024-09-21T10:08:48.414 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:08:48.414 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Object prefix: benchmark_data_smithi098_169033 2024-09-21T10:08:48.414 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:08:48.415 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:08:48.423 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:48.459 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:48.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:48.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:48.563 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:48.595 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:48.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:48.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:48.697 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:48.736 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:48.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:48.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:48.838 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:48.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:48 smithi098 ceph-mon[97580]: pgmap v454: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:48.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:48 smithi098 ceph-mon[97580]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:08:48.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:48 smithi098 ceph-mon[100914]: pgmap v454: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:48.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:48 smithi098 ceph-mon[100914]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:08:48.876 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:48.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:48.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:48.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:48 smithi196 ceph-mon[84784]: pgmap v454: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:48 smithi196 ceph-mon[84784]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:08:48.978 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:49.005 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:49.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:49.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:49.101 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:08:49.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:49.144 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:49.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:49.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:49.247 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:49.276 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:49.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:49.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:49.378 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:49.379 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 1 16 3418 3402 212.622 212.625 0.00398292 0.00468807 2024-09-21T10:08:49.416 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:49.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:49.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:49.432 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:08:49.519 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:49.548 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:49.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:49.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:49.650 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:49.681 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:49.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:49.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:49.783 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:49.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:49.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:49.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:49.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:49.942 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:49.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:49.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:50.044 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:50.070 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:50.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:50.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:50.075 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:50.172 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:50.215 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:50.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:50.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:50.317 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:50.338 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:50.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:50.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:50.379 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 2 16 6383 6367 198.958 185.312 0.082277 0.00488797 2024-09-21T10:08:50.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:50.477 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:50.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:50.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:50.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:50.584 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:08:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:08:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:08:50.608 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:50.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:50.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:50.629 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:08:50.710 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:50.738 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:50.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:50.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:50.840 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:50.865 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:50.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:50.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:50 smithi196 ceph-mon[84784]: pgmap v455: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:50 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1093088946' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:50 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:50.967 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:50.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:50.997 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:50.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:51.068 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:08:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:08:50] "GET /metrics HTTP/1.1" 200 34092 "" "Prometheus/2.51.0" 2024-09-21T10:08:51.069 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:50 smithi098 ceph-mon[97580]: pgmap v455: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:51.069 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:50 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1093088946' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:51.069 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:50 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:51.069 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:50 smithi098 ceph-mon[100914]: pgmap v455: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:08:51.069 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:50 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1093088946' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:51.069 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:50 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:51.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:51.132 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:51.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:51.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:51.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:51.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:51.271 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:51.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:51.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:51.379 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 3 16 9301 9285 193.425 182.375 0.00200047 0.00516007 2024-09-21T10:08:51.401 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:51.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:51.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:51.448 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:08:51.448 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:08:51.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:51.448 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45263.0:8736 6.6 e169/165 6:61785d55:::benchmark_data_smithi098_169033_object8735:head v 169'531, mlcod=169'529)", 2024-09-21T10:08:51.448 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:08:51.195884+0000", 2024-09-21T10:08:51.448 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.00022286300000000001, 2024-09-21T10:08:51.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00024917400000000001, 2024-09-21T10:08:51.448 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:08:51.448 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:08:51.448 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:08:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:08:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:51.195884+0000", 2024-09-21T10:08:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:08:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:51.195884+0000", 2024-09-21T10:08:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:08:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:08:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:51.195884+0000", 2024-09-21T10:08:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:08:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:51.195994+0000", 2024-09-21T10:08:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000110586 2024-09-21T10:08:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:08:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:51.196006+0000", 2024-09-21T10:08:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2247e-05 2024-09-21T10:08:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:08:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:51.196027+0000", 2024-09-21T10:08:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0120999999999999e-05 2024-09-21T10:08:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:08:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:51.196073+0000", 2024-09-21T10:08:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5973999999999997e-05 2024-09-21T10:08:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:08:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:08:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:08:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:51.196094+0000", 2024-09-21T10:08:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1246e-05 2024-09-21T10:08:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:08:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:51.452 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:08:51.452 INFO:teuthology.orchestra.run.smithi196.stdout: ], 2024-09-21T10:08:51.452 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 1 2024-09-21T10:08:51.452 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:08:51.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:51.538 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:51.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:51.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:51.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:51.675 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:51.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:51.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:51.777 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:51.804 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:51.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:51.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:51.906 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:51.941 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:51.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:51.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:52.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:52.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:51 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:08:52.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:51 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:08:52.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:51 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:08:52.052 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:51 smithi098 ceph-mon[97580]: osdmap e169: 8 total, 8 up, 7 in 2024-09-21T10:08:52.052 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:51 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:08:52.052 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:51 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:08:52.052 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:51 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:08:52.052 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:51 smithi098 ceph-mon[100914]: osdmap e169: 8 total, 8 up, 7 in 2024-09-21T10:08:52.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:52.084 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:52.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:52.087 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:51 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:08:52.087 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:51 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:08:52.087 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:51 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:08:52.087 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:51 smithi196 ceph-mon[84784]: osdmap e169: 8 total, 8 up, 7 in 2024-09-21T10:08:52.123 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:08:52.185 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:52.223 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:52.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:52.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:52.325 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:52.364 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:52.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:52.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:52.379 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 4 16 12060 12044 188.174 172.438 0.00736228 0.00530664 2024-09-21T10:08:52.466 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:52.491 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:08:52.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:52.502 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:52.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:52.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:52.642 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:52.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:52.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:52.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:52.783 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:52.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:52.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:52.874 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:52 smithi196 ceph-mon[84784]: pgmap v457: 89 pgs: 89 active+clean; 137 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 17 MiB/s wr, 265 op/s 2024-09-21T10:08:52.874 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:08:52.874 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:52.874 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:08:52.887 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:52.914 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:52 smithi098 ceph-mon[97580]: pgmap v457: 89 pgs: 89 active+clean; 137 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 17 MiB/s wr, 265 op/s 2024-09-21T10:08:52.915 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:08:52.915 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:52.915 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:08:52.915 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:52 smithi098 ceph-mon[100914]: pgmap v457: 89 pgs: 89 active+clean; 137 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 17 MiB/s wr, 265 op/s 2024-09-21T10:08:52.915 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:08:52.915 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:52.915 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:08:52.922 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:52.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:52.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:53.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:53.063 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:53.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:53.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:53.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:53.193 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:53.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:53.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:53.295 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:53.331 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:53.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:53.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:53.379 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 5 16 14831 14815 185.174 173.188 0.00733591 0.00539392 2024-09-21T10:08:53.433 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:53.463 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:53.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:53.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:53.565 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:53.593 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:53.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:53.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:53.694 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:53.707 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-21T10:08:53.734 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:53.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:53.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:53.836 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:53.863 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:53.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:53.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:53.965 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:53.994 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:53.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:53.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:54.095 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:54.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:08:54.136 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:54.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:54.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:54.238 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:54.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:54.279 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:54.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:54.380 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 6 16 17365 17349 180.698 158.375 0.00929767 0.00552738 2024-09-21T10:08:54.380 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:54.421 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:54.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:54.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:54.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:54.561 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:54.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:54.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:54.663 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:54.689 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:54.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:54.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:54.703 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:54.791 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:54.817 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:54.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:54.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:54.919 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:54.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:54.951 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:54.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:54.992 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:08:54.993 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:08:54.993 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-21T10:08:54.993 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-21T10:08:54.993 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:08:54.993 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:54 smithi196 ceph-mon[84784]: pgmap v458: 89 pgs: 89 active+clean; 493 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 61 MiB/s wr, 976 op/s 2024-09-21T10:08:54.993 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:54 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:54.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:54 smithi098 ceph-mon[97580]: pgmap v458: 89 pgs: 89 active+clean; 493 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 61 MiB/s wr, 976 op/s 2024-09-21T10:08:54.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:54 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:54.995 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:54 smithi098 ceph-mon[100914]: pgmap v458: 89 pgs: 89 active+clean; 493 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 61 MiB/s wr, 976 op/s 2024-09-21T10:08:54.995 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:54 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:55.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:55.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:55.099 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:55.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:55.200 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:55.234 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:55.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:55.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:55.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:55.372 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:55.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:55.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:55.380 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 7 16 19652 19636 175.302 142.938 0.00273901 0.00569796 2024-09-21T10:08:55.473 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:55.505 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:55.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:55.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:55.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:55.642 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:55.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:55.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:55.743 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:08:55.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:55.782 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:55.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:55.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:55.884 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:55.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:55.921 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:55.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:56.022 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:56.054 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:56.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:56.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:56.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:56.198 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:56.207 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:56.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:56.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:56.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:56.339 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:56.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:56.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:56.380 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 8 16 22047 22031 172.098 149.688 0.00193994 0.00580357 2024-09-21T10:08:56.441 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:56.468 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:56.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:56.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:56.570 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:56.610 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:56.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:56.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:56.645 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:08:56.709 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:08:56.709 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:08:56.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:56.745 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:08:56.746 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:08:56.752 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:56.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:56.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:56.778 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:08:56.778 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:08:56.779 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:56 smithi098 ceph-mon[97580]: pgmap v459: 89 pgs: 89 active+clean; 714 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-21T10:08:56.779 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:56 smithi098 ceph-mon[100914]: pgmap v459: 89 pgs: 89 active+clean; 714 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-21T10:08:56.814 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:08:56.814 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:08:56.846 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:08:56.846 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:08:56.854 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:56.878 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:08:56.878 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:08:56.883 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:56.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:56.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:56.912 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:08:56.912 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:08:56.943 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:08:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:08:56.985 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:57.013 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:57.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:57.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:57.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:57.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:56 smithi196 ceph-mon[84784]: pgmap v459: 89 pgs: 89 active+clean; 714 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-21T10:08:57.143 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:57.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:57.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:57.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:57.272 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:57.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:57.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:57.373 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:57.380 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 9 16 24273 24257 168.433 139.125 0.00529646 0.00593127 2024-09-21T10:08:57.402 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:57.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:57.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:57.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:57.537 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:57.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:57.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:57.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:57.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:57.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:57.780 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:57.819 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:57.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:57.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:57.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:57 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/850401468' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:57.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:57 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/850401468' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:57.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:57.961 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:57.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:57.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:58.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:58.098 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:58.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:58.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:58.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:57 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/850401468' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:58.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:58.235 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:58.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:58.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:58.337 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:58.365 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:58.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:58.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:58.380 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 10 16 26339 26323 164.501 129.125 0.014199 0.00607259 2024-09-21T10:08:58.466 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:58.502 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:58.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:58.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:58.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:08:58.644 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:58.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:58.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:08:58.746 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:58.779 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:58.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:58.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:58.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:08:58 smithi098 ceph-mon[97580]: pgmap v460: 89 pgs: 89 active+clean; 1.1 GiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-09-21T10:08:58.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:08:58 smithi098 ceph-mon[100914]: pgmap v460: 89 pgs: 89 active+clean; 1.1 GiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-09-21T10:08:58.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:08:58 smithi196 ceph-mon[84784]: pgmap v460: 89 pgs: 89 active+clean; 1.1 GiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-09-21T10:08:58.907 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:58.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:58.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:59.008 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:59.038 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:59.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:59.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:59.140 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:59.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:59.161 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:59.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:59.262 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:59.290 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:59.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:59.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:59.380 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 11 16 28187 28171 160.045 115.5 0.00469507 0.00624277 2024-09-21T10:08:59.392 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:59.429 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:59.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:59.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:59.531 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:59.566 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:59.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:59.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:59.668 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:59.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:59.707 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:08:59.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:59.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:59.844 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:59.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:59.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:59.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:59.984 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:08:59.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:59.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:00.086 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:00.116 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:00.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:00.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:00.199 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:09:00.199 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:09:00.218 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:00.242 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:00.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:00.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:00.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:00.370 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:00.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:00.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:00.380 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 12 16 30103 30087 156.687 119.75 0.00538125 0.00637733 2024-09-21T10:09:00.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:09:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:09:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:09:00.473 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:00.498 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:00.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:00.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:00.600 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:00.635 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:00.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:00.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:00.737 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:00.770 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:00.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:00.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:00.872 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:00.901 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:00.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:00.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:01.003 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:01.025 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:01.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:01.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:01.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:00 smithi098 ceph-mon[97580]: pgmap v461: 89 pgs: 89 active+clean; 1.4 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-09-21T10:09:01.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:09:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:09:00] "GET /metrics HTTP/1.1" 200 36438 "" "Prometheus/2.51.0" 2024-09-21T10:09:01.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:00 smithi098 ceph-mon[100914]: pgmap v461: 89 pgs: 89 active+clean; 1.4 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-09-21T10:09:01.126 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:01.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:00 smithi196 ceph-mon[84784]: pgmap v461: 89 pgs: 89 active+clean; 1.4 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-09-21T10:09:01.156 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:01.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:01.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:01.259 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:01.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:01.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:01.380 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 13 16 31607 31591 151.864 94 0.00174184 0.00654773 2024-09-21T10:09:01.395 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:01.423 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:01.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:01.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:01.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:01.561 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:01.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:01.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:01.663 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:01.695 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:01.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:01.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:01.797 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:01.826 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:01.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:01.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:01.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:01.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:09:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:01.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:01.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:02.060 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:02.096 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:02.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:02.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:02.198 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:02.237 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:02.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:02.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:02.339 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:02.376 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:02.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:02.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:02.380 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 14 16 33520 33504 149.556 119.562 0.00944038 0.00668046 2024-09-21T10:09:02.478 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:02.499 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:02.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:02.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:02.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:02.635 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:02.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:02.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:02.712 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:09:02.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:02.769 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:02.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:02.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:02.844 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:02 smithi196 ceph-mon[84784]: pgmap v462: 89 pgs: 89 active+clean; 1.6 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-09-21T10:09:02.871 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:02.899 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:02.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:02.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:03.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:03.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:03.026 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:03.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:03.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:02 smithi098 ceph-mon[97580]: pgmap v462: 89 pgs: 89 active+clean; 1.6 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-09-21T10:09:03.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:02 smithi098 ceph-mon[100914]: pgmap v462: 89 pgs: 89 active+clean; 1.6 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-09-21T10:09:03.127 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:03.161 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:03.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:03.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:03.263 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:03.295 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:03.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:03.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:03.381 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 15 16 35290 35274 146.959 110.625 0.00772811 0.00679845 2024-09-21T10:09:03.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:03.419 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:03.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:03.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:03.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:03.554 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:03.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:03.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:03.656 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:03.691 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:03.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:03.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:03.793 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:03.819 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:03.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:03.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:03.921 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:03.949 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:03.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:03.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:04.051 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:04.093 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:04.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:04.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:04.194 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:04.224 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:04.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:04.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:04.326 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:04.354 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:04.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:04.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:04.381 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 16 16 37285 37269 145.566 124.688 0.0066692 0.00686495 2024-09-21T10:09:04.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:04.536 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:04.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:04.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:04.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:04.674 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:04.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:04.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:04.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:04.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:04.821 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:04.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:04.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:04 smithi196 ceph-mon[84784]: pgmap v463: 89 pgs: 89 active+clean; 1.9 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 147 MiB/s wr, 2.34k op/s 2024-09-21T10:09:04.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:04.960 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:04.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:04.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:05.062 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:05.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:04 smithi098 ceph-mon[97580]: pgmap v463: 89 pgs: 89 active+clean; 1.9 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 147 MiB/s wr, 2.34k op/s 2024-09-21T10:09:05.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:04 smithi098 ceph-mon[100914]: pgmap v463: 89 pgs: 89 active+clean; 1.9 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 147 MiB/s wr, 2.34k op/s 2024-09-21T10:09:05.103 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:05.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:05.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:05.205 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:05.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:05.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:05.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:05.374 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:05.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:05.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:05.381 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 17 16 39144 39128 143.838 116.188 0.00650142 0.00694657 2024-09-21T10:09:05.476 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:05.504 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:05.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:05.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:05.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-21T10:09:05.605 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:05.633 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:05.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:05.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:05.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:05.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:05.774 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:05.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:05.875 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:05.902 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:05.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:05.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:05.944 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:09:05.944 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:09:06.004 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:06.039 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:06.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:06.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:06.141 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:06.171 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:06.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:06.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:06.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:06.311 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:06.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:06.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:06.381 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 18 16 40743 40727 141.397 99.9375 0.00489771 0.00706618 2024-09-21T10:09:06.412 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:06.459 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:09:06.480 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:06.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:06.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:06.583 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:06.617 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:06.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:06.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:06.635 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:09:06.719 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:06.754 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:09:06.756 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:06.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:06.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:06.859 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:06 smithi196 ceph-mon[84784]: pgmap v464: 89 pgs: 89 active+clean; 2.0 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 132 MiB/s wr, 2.11k op/s 2024-09-21T10:09:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:06 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:09:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:06 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:09:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:06 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1948959594' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:09:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:06 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:09:06.909 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:06.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:06.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:07.001 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:09:07.010 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:07.042 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:07.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:07.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:07.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:06 smithi098 ceph-mon[97580]: pgmap v464: 89 pgs: 89 active+clean; 2.0 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 132 MiB/s wr, 2.11k op/s 2024-09-21T10:09:07.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:06 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:09:07.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:06 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:09:07.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:06 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1948959594' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:09:07.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:06 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:09:07.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:06 smithi098 ceph-mon[100914]: pgmap v464: 89 pgs: 89 active+clean; 2.0 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 132 MiB/s wr, 2.11k op/s 2024-09-21T10:09:07.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:06 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:09:07.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:06 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:09:07.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:06 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1948959594' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:09:07.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:06 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:09:07.144 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:07.176 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:07.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:07.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:07.278 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:07.313 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:07.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:07.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:07.381 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 19 16 42157 42141 138.606 88.375 0.0119515 0.00720841 2024-09-21T10:09:07.415 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:07.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:07.450 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:07.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:07.551 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:07.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:07.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:07.690 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:07.721 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:07.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:07.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:07.822 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:07.849 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:07.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:07.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:07.951 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:07.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:07.990 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:07.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:08.091 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:08.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:07 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:09:08.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:07 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:09:08.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:07 smithi098 ceph-mon[97580]: osdmap e170: 8 total, 8 up, 7 in 2024-09-21T10:09:08.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:09:08.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:08.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:07 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:09:08.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:07 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:09:08.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:07 smithi098 ceph-mon[100914]: osdmap e170: 8 total, 8 up, 7 in 2024-09-21T10:09:08.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:09:08.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:08.121 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:08.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:08.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:08.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:07 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:09:08.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:07 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:09:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:07 smithi196 ceph-mon[84784]: osdmap e170: 8 total, 8 up, 7 in 2024-09-21T10:09:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:09:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:08.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:08.262 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:08.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:08.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:08.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:08.384 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:09:08.383593+0000 min lat: 0.00114388 max lat: 0.178407 avg lat: 0.00734009 2024-09-21T10:09:08.384 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:09:08.385 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 20 16 43596 43580 136.15 89.9375 0.0039956 0.00734009 2024-09-21T10:09:08.401 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:08.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:08.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:08.503 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:08.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:08.534 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:08.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:08.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:08.675 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:08.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:08.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:08.777 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:08.814 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:08.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:08.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:08.862 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:09:08.862 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:09:08.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:08.951 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:08.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:08.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:09.053 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:09.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:09.090 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:09.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:09.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:08 smithi098 ceph-mon[97580]: pgmap v466: 89 pgs: 89 active+clean; 2.3 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-21T10:09:09.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:08 smithi098 ceph-mon[100914]: pgmap v466: 89 pgs: 89 active+clean; 2.3 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-21T10:09:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:08 smithi196 ceph-mon[84784]: pgmap v466: 89 pgs: 89 active+clean; 2.3 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-21T10:09:09.191 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:09.218 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:09.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:09.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:09.279 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:09.280 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:09.319 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:09.345 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:09.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:09.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:09.385 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 21 16 45346 45330 134.874 109.375 0.00605321 0.00740852 2024-09-21T10:09:09.447 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:09.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:09.494 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:09.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:09.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:09:09.595 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:09.626 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:09.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:09.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:09.728 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:09.763 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:09.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:09.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:09.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:09.893 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:09:09.905 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:09.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:09.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:10.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:10.043 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:10.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:10.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:10.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:10.183 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:10.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:10.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:10.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:10.319 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:10.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:10.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:10.356 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:09:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:09:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:09:10.386 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 22 16 47084 47068 133.671 108.625 0.00904743 0.00747519 2024-09-21T10:09:10.421 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:10.449 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:10.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:10.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:10.551 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:10.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:10.592 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:10.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:10.693 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:10.733 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:10.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:10.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:10.834 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:10.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:09:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:09:10] "GET /metrics HTTP/1.1" 200 36440 "" "Prometheus/2.51.0" 2024-09-21T10:09:10.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:10 smithi098 ceph-mon[97580]: pgmap v467: 89 pgs: 89 active+clean; 2.5 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 114 MiB/s wr, 1.83k op/s 2024-09-21T10:09:10.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:10 smithi098 ceph-mon[100914]: pgmap v467: 89 pgs: 89 active+clean; 2.5 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 114 MiB/s wr, 1.83k op/s 2024-09-21T10:09:10.876 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:10.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:10.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:10.978 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:11.006 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:11.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:11.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:11.020 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:10 smithi196 ceph-mon[84784]: pgmap v467: 89 pgs: 89 active+clean; 2.5 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 114 MiB/s wr, 1.83k op/s 2024-09-21T10:09:11.108 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:11.145 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:11.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:11.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:11.206 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:09:11.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:11.295 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:11.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:11.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:11.387 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 23 16 48779 48763 132.465 105.938 0.0102769 0.00754363 2024-09-21T10:09:11.397 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:11.441 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:11.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:11.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:11.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:11.575 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:11.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:11.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:11.635 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:09:11.635 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:09:11.677 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:11.708 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:11.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:11.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:11.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:11.846 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:11.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:11.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:11.948 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:11.975 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:11.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:11.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:12.077 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:12.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:12.107 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:12.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:12.208 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:12.238 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:12.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:12.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:12.287 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:09:12.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:12.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:12.387 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:12.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:12.388 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 24 16 50328 50312 130.98 96.8125 0.017702 0.00762939 2024-09-21T10:09:12.488 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:12.512 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:12.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:12.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:12.613 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:12.637 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:12.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:12.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:12.739 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:12.768 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:12.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:12.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:12.870 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:12.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:12.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:13.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:13.030 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:12 smithi098 ceph-mon[97580]: pgmap v468: 89 pgs: 89 active+clean; 2.7 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-21T10:09:13.031 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:12 smithi098 ceph-mon[100914]: pgmap v468: 89 pgs: 89 active+clean; 2.7 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-21T10:09:13.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:13.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:13.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:12 smithi196 ceph-mon[84784]: pgmap v468: 89 pgs: 89 active+clean; 2.7 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-21T10:09:13.147 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:13.173 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:13.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:13.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:13.275 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:13.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:13.322 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:13.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:13.387 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 25 16 51847 51831 129.536 94.9375 0.0174597 0.00771431 2024-09-21T10:09:13.424 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:13.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:13.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:13.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:13.581 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:13.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:13.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:13.682 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:13.709 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:13.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:13.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:13.811 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:13.845 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:13.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:13.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:13.947 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:13.974 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:13.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:13.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:14.076 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:14.104 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:14.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:14.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:14.206 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:14.237 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:14.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:14.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:14.338 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:14.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:14.360 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:14.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:14.387 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 26 16 53253 53237 127.934 87.875 0.011186 0.00781133 2024-09-21T10:09:14.462 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:14.492 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:14.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:14.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:14.594 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:14.619 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:14.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:14.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:14.720 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:14.746 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:14.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:14.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:14.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:14 smithi098 ceph-mon[97580]: pgmap v469: 89 pgs: 89 active+clean; 2.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-21T10:09:14.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:14 smithi098 ceph-mon[100914]: pgmap v469: 89 pgs: 89 active+clean; 2.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-21T10:09:14.847 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:14.884 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:14.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:14.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:14.966 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:09:14.966 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:09:14.970 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:09:14.971 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:09:14.988 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:15.010 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:09:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:09:15.027 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:15.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:15.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:15.062 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:09:15.062 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:09:15.101 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:09:15.102 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:09:15.129 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:15.132 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:09:15.132 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:09:15.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:14 smithi196 ceph-mon[84784]: pgmap v469: 89 pgs: 89 active+clean; 2.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-21T10:09:15.156 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:15.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:15.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:15.165 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:09:15.165 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:09:15.196 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:09:15.196 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:09:15.258 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:15.287 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:15.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:15.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:15.387 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 27 16 54709 54693 126.566 91 0.00399255 0.00789557 2024-09-21T10:09:15.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:15.425 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:15.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:15.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:15.526 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:15.566 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:15.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:15.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:15.668 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:15.695 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:15.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:15.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:15.797 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:15.824 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:15.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:15.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:15.926 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:15.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:15.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:15.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:16.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:16.095 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:16.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:16.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:16.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:15 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4282879586' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:16.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:15 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4282879586' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:15 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4282879586' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:16.197 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:16.221 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:16.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:16.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:16.231 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-21T10:09:16.323 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:16.391 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 28 16 56233 56217 125.448 95.25 0.00543803 0.00796641 2024-09-21T10:09:16.398 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:16.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:16.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:16.500 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:16.527 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:16.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:16.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:16.571 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:16.572 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:16.629 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:16.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:16.684 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:16.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:16.785 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:16.832 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:16.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:16.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:16.934 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:16.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:16 smithi098 ceph-mon[97580]: pgmap v470: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-21T10:09:16.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:16 smithi098 ceph-mon[100914]: pgmap v470: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-21T10:09:16.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:16.970 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:16.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:17.071 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:17.103 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:17.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:17.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:16 smithi196 ceph-mon[84784]: pgmap v470: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-21T10:09:17.206 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:17.249 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:17.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:17.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:17.351 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:17.389 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 29 16 57690 57674 124.255 91.0625 0.0112971 0.00804179 2024-09-21T10:09:17.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:17.408 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:17.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:17.509 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:17.537 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:17.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:17.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:17.639 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:17.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:17.670 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:17.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:17.772 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:17.811 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:17.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:17.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:17.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:17.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:17.943 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:17.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:18.044 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:18.069 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:18.069 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:18.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:18.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:18.171 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:18.195 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:18.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:18.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:18.297 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:18.325 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:18.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:18.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:18.389 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 30 16 59174 59158 123.204 92.75 0.0137763 0.00811034 2024-09-21T10:09:18.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:18.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:18.463 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:18.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:18.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:18.601 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:18.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:18.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:18.705 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:18.730 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:18.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:18.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:18.832 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:18.873 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:18.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:18.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:18.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:19.009 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:19.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:19.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:19.014 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:09:19.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:18 smithi098 ceph-mon[97580]: pgmap v471: 89 pgs: 89 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-09-21T10:09:19.100 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:18 smithi098 ceph-mon[100914]: pgmap v471: 89 pgs: 89 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-09-21T10:09:19.111 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:19.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:18 smithi196 ceph-mon[84784]: pgmap v471: 89 pgs: 89 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-09-21T10:09:19.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:19.152 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:19.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:19.253 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:19.296 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:19.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:19.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:19.389 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 31 16 60601 60585 122.107 89.1875 0.00813514 0.00818293 2024-09-21T10:09:19.398 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:19.425 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:19.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:19.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:19.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:19.552 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:19.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:19.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:19.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:19.694 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:19.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:19.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:19.796 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:19.825 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:19.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:19.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:19.927 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:19.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:19.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:19.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:20.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:20.088 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:20.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:20.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:20.190 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:20.215 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:20.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:20.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:20.317 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:20.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:20.364 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:20.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:20.389 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 32 16 62004 61988 121.031 87.6875 0.0113577 0.00825546 2024-09-21T10:09:20.465 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:20.489 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:20.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:20.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:20.591 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:20.619 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:20.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:20.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:20.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:09:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:09:20.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:09:20.721 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:20.760 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:20.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:20.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:20.810 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:09:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:09:20] "GET /metrics HTTP/1.1" 200 36440 "" "Prometheus/2.51.0" 2024-09-21T10:09:20.862 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:20.888 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:20.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:20.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:20.990 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:21.031 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:21.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:21.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:21.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:20 smithi098 ceph-mon[97580]: pgmap v472: 89 pgs: 89 active+clean; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-21T10:09:21.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:20 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:09:21.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:20 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:09:21.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:20 smithi098 ceph-mon[100914]: pgmap v472: 89 pgs: 89 active+clean; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-21T10:09:21.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:20 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:09:21.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:20 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:09:21.132 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:21.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:20 smithi196 ceph-mon[84784]: pgmap v472: 89 pgs: 89 active+clean; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-21T10:09:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:20 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:09:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:20 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:09:21.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:21.173 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:21.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:21.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:21.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:21.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:21.389 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 33 16 63457 63441 120.115 90.8125 0.00597186 0.00832091 2024-09-21T10:09:21.413 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:21.440 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:21.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:21.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:21.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:21.586 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:21.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:21.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:21.688 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:21.714 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:21.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:21.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:21.816 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:21.851 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:21.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:21.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:21.952 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:21.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:21.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:22.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:22.126 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:22.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:22.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:22.229 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:22.252 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:22.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:22.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:22.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:22.390 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 34 16 64821 64805 119.09 85.25 0.0130182 0.00839209 2024-09-21T10:09:22.392 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:22.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:22.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:22.494 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:22.529 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:22.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:22.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:22.631 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:22.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:22.683 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:22.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:22.784 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:22.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:22.811 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:22.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:22.912 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:22.913 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:22.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:22.962 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:22.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:23.038 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:22 smithi098 ceph-mon[100914]: pgmap v473: 89 pgs: 89 active+clean; 3.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-21T10:09:23.038 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:23.038 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:22 smithi098 ceph-mon[97580]: pgmap v473: 89 pgs: 89 active+clean; 3.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-21T10:09:23.038 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:23.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:23.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:23.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:23.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:22 smithi196 ceph-mon[84784]: pgmap v473: 89 pgs: 89 active+clean; 3.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-21T10:09:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:23.208 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:23.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:23.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:23.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:23.332 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:23.367 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:23.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:23.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:23.393 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 35 16 66011 65995 117.813 74.375 0.0110441 0.00848277 2024-09-21T10:09:23.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:23.504 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:23.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:23.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:23.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:23.639 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:23.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:23.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:23.741 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:23.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:23.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:23.882 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:23.915 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:23.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:23.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:23.948 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:09:24.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:24.053 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:24.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:24.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:24.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:23 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:09:24.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:23 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:09:24.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:23 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:09:24.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:23 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1793049518' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:09:24.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:23 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:09:24.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:23 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:09:24.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:23 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:09:24.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:23 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:09:24.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:23 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1793049518' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:09:24.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:23 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:09:24.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:23 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:09:24.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:23 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:09:24.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:23 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:09:24.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:23 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1793049518' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:09:24.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:23 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:09:24.154 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:24.190 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:24.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:24.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:24.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:24.329 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:24.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:24.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:24.390 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 36 16 67284 67268 116.75 79.5625 0.0142692 0.00856049 2024-09-21T10:09:24.431 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:24.460 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:24.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:24.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:24.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:24.583 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:24.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:24.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:24.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:24.723 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:24.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:24.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:24.825 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:24.854 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:24.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:24.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:24.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:24.983 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:24.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:24.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:25.085 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:25.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:24 smithi098 ceph-mon[97580]: pgmap v474: 89 pgs: 89 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-21T10:09:25.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:24 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:09:25.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:24 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:09:25.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:24 smithi098 ceph-mon[97580]: osdmap e171: 8 total, 8 up, 7 in 2024-09-21T10:09:25.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:24 smithi098 ceph-mon[100914]: pgmap v474: 89 pgs: 89 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-21T10:09:25.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:24 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:09:25.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:24 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:09:25.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:24 smithi098 ceph-mon[100914]: osdmap e171: 8 total, 8 up, 7 in 2024-09-21T10:09:25.106 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:25.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:25.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:25.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:24 smithi196 ceph-mon[84784]: pgmap v474: 89 pgs: 89 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-21T10:09:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:24 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:09:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:24 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:09:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:24 smithi196 ceph-mon[84784]: osdmap e171: 8 total, 8 up, 7 in 2024-09-21T10:09:25.208 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:25.237 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:25.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:25.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:25.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:25.360 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:25.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:25.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:25.390 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 37 16 68413 68397 115.502 70.5625 0.00251812 0.00863093 2024-09-21T10:09:25.462 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:25.489 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:25.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:25.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:25.591 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:25.616 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:25.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:25.718 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:25.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:25.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:25.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:25.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:25.886 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:25.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:25.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:25.987 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:26.020 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:26.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:26.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:26.122 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:26.149 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:26.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:26.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:26.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:26.279 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:26.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:26.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:26.381 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:26.390 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 38 16 69650 69634 114.497 77.3125 0.0071749 0.00872479 2024-09-21T10:09:26.402 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:26.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:26.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:26.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:26.541 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:26.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:26.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:26.642 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:26.671 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:26.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:26.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:26.772 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:26.801 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:26.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:26.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:26.902 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:26.930 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:26.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:26.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:27.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:27.070 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:27.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:27.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:26 smithi098 ceph-mon[97580]: pgmap v476: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-21T10:09:27.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:26 smithi098 ceph-mon[100914]: pgmap v476: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-21T10:09:27.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:26 smithi196 ceph-mon[84784]: pgmap v476: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-21T10:09:27.172 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:27.213 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:27.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:27.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:27.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:27.354 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:27.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:27.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:27.390 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 39 16 70878 70862 113.529 76.75 0.00457794 0.00880313 2024-09-21T10:09:27.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:27.488 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:27.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:27.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:27.590 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:27.619 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:27.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:27.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:27.721 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:27.758 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:27.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:27.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:27.860 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:27.898 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:27.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:27.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:28.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:28.041 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:28.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:28.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:28.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:28.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:28.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:28.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:28.320 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:28.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:28.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:28.390 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:09:28.389287+0000 min lat: 0.00114388 max lat: 0.178407 avg lat: 0.00887198 2024-09-21T10:09:28.390 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:09:28.391 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 40 16 72130 72114 112.647 78.25 0.00787689 0.00887198 2024-09-21T10:09:28.422 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:28.443 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:28.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:28.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:28.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:28.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:28.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:28.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:28.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:28.693 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:28.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:28.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:28.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:28.829 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:28.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:28.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:28.931 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:28.966 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:28.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:28.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:29.068 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:29.096 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:29.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:29.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:29.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:28 smithi098 ceph-mon[97580]: pgmap v477: 89 pgs: 89 active+clean; 4.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-21T10:09:29.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:28 smithi098 ceph-mon[100914]: pgmap v477: 89 pgs: 89 active+clean; 4.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-21T10:09:29.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:28 smithi196 ceph-mon[84784]: pgmap v477: 89 pgs: 89 active+clean; 4.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-21T10:09:29.198 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:29.227 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:29.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:29.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:29.330 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:29.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:29.368 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:29.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:29.390 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 41 16 73299 73283 111.681 73.0625 0.00354663 0.00894909 2024-09-21T10:09:29.469 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:29.502 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:29.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:29.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:29.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:29.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:29.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:29.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:29.781 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:29.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:29.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:29.883 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:29.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-21T10:09:29.928 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:29.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:29.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:30.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:30.084 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:30.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:30.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:30.186 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:30.233 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:30.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:30.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:30.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:30.377 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:30.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:30.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:30.390 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 42 16 74525 74509 110.846 76.625 0.00760563 0.0090168 2024-09-21T10:09:30.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:30.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:30.517 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:30.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:30.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:30.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:09:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:09:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:09:30.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:30.664 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:30.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:30.725 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:09:30.725 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:09:30.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:30.725 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.45263.0:74930 6.a e171/165 6:54d1baca:::benchmark_data_smithi098_169033_object74929:head v 171'4650, mlcod=171'4645)", 2024-09-21T10:09:30.726 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:09:30.716599+0000", 2024-09-21T10:09:30.726 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.005831976, 2024-09-21T10:09:30.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058601679999999998, 2024-09-21T10:09:30.726 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:09:30.726 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:09:30.726 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:09:30.726 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:09:30.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:30.726 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:09:30.726 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:30.716599+0000", 2024-09-21T10:09:30.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:30.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:30.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:30.726 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:09:30.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:30.716599+0000", 2024-09-21T10:09:30.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:30.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:30.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:30.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:09:30.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:30.716600+0000", 2024-09-21T10:09:30.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5199999999999995e-07 2024-09-21T10:09:30.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:30.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:30.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:09:30.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:30.716641+0000", 2024-09-21T10:09:30.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1140000000000003e-05 2024-09-21T10:09:30.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:30.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:30.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:09:30.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:30.716663+0000", 2024-09-21T10:09:30.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1727999999999999e-05 2024-09-21T10:09:30.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:30.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:30.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:09:30.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:30.716671+0000", 2024-09-21T10:09:30.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4710000000000001e-06 2024-09-21T10:09:30.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:30.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:30.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:30.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:30.720641+0000", 2024-09-21T10:09:30.729 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039701459999999999 2024-09-21T10:09:30.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:30.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:30.729 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:09:30.729 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:30.720695+0000", 2024-09-21T10:09:30.729 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4039999999999998e-05 2024-09-21T10:09:30.729 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:30.729 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:30.729 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:30.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:30.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:30.729 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:74937 6.9 6:9ee347a0:::benchmark_data_smithi098_169033_object74936:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T10:09:30.729 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:09:30.721415+0000", 2024-09-21T10:09:30.729 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0010160799999999999, 2024-09-21T10:09:30.730 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001103461, 2024-09-21T10:09:30.730 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:09:30.730 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:09:30.730 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:09:30.730 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:09:30.730 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:09:30.730 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:09:30.730 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 74937 2024-09-21T10:09:30.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:30.730 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:09:30.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:30.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:09:30.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:30.721415+0000", 2024-09-21T10:09:30.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:30.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:30.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:30.731 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:09:30.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:30.721415+0000", 2024-09-21T10:09:30.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:30.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:30.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:30.731 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:09:30.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:30.721417+0000", 2024-09-21T10:09:30.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.669e-06 2024-09-21T10:09:30.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:30.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:30.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:09:30.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:30.721438+0000", 2024-09-21T10:09:30.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0869e-05 2024-09-21T10:09:30.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:30.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:30.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:09:30.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:30.721455+0000", 2024-09-21T10:09:30.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7150999999999999e-05 2024-09-21T10:09:30.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:30.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:30.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:09:30.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:30.721463+0000", 2024-09-21T10:09:30.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0930000000000003e-06 2024-09-21T10:09:30.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:30.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:30.733 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:30.733 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:30.721526+0000", 2024-09-21T10:09:30.733 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3357999999999999e-05 2024-09-21T10:09:30.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:30.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:30.733 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:09:30.733 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:30.721663+0000", 2024-09-21T10:09:30.733 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000136475 2024-09-21T10:09:30.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:30.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:30.733 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:09:30.733 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:30.721791+0000", 2024-09-21T10:09:30.733 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012813899999999999 2024-09-21T10:09:30.734 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:30.734 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:30.734 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:30.734 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:30.734 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-21T10:09:30.734 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 2 2024-09-21T10:09:30.734 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:09:30.765 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:30.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:30.815 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:30.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:30.835 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:09:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:09:30] "GET /metrics HTTP/1.1" 200 36451 "" "Prometheus/2.51.0" 2024-09-21T10:09:30.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:30.956 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:30.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:30.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:31.058 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:31.084 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:31.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:31.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:31.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:30 smithi098 ceph-mon[97580]: pgmap v478: 89 pgs: 89 active+clean; 4.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-21T10:09:31.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:30 smithi098 ceph-mon[100914]: pgmap v478: 89 pgs: 89 active+clean; 4.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-21T10:09:31.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:30 smithi196 ceph-mon[84784]: pgmap v478: 89 pgs: 89 active+clean; 4.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-21T10:09:31.186 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:31.213 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:31.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:31.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:31.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:31.360 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:31.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:31.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:31.390 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 43 16 75782 75766 110.096 78.5625 0.00799453 0.00907844 2024-09-21T10:09:31.461 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:31.488 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:31.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:31.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:31.589 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:31.619 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:31.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:31.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:31.721 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:31.747 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:31.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:31.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:31.795 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:09:31.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:31.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:31.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:31.995 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:32.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:32.034 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:32.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:32.107 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:09:32.134 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:32.190 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:32.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:32.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:32.292 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:32.320 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:32.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:32.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:32.391 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 44 16 76973 76957 109.285 74.4375 0.00678721 0.00914458 2024-09-21T10:09:32.421 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:32.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:32.468 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:32.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:32.569 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:32.651 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:32.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:32.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:32.655 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:32.753 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:32.779 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:32.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:32.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:32.880 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:32.909 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:32.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:32.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:33.011 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:33.035 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:33.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:33.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:33.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:32 smithi098 ceph-mon[97580]: pgmap v479: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-21T10:09:33.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:32 smithi098 ceph-mon[100914]: pgmap v479: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-21T10:09:33.137 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:32 smithi196 ceph-mon[84784]: pgmap v479: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-21T10:09:33.167 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:33.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:33.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:33.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:33.303 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:33.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:33.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:33.391 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 45 16 78113 78097 108.44 71.25 0.0150493 0.00921553 2024-09-21T10:09:33.404 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:33.429 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:33.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:33.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:33.531 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:33.534 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:09:33.534 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:09:33.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:33.571 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:33.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:33.571 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:09:33.572 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:09:33.614 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:09:33.614 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:09:33.655 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:09:33.655 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:09:33.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:33.684 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:09:33.684 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:09:33.713 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:09:33.713 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:09:33.717 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:33.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:33.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:33.749 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:09:33.750 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:09:33.780 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:09:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:09:33.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:33.857 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:33.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:33.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:33.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:33.998 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:33.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:33.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:34.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:33 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/672724220' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:34.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:33 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/672724220' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:34.101 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:34.126 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:34.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:34.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:34.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:33 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/672724220' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:34.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:34.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:34.264 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:34.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:34.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:34.391 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 46 16 79310 79294 107.709 74.8125 0.0133448 0.00927807 2024-09-21T10:09:34.404 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:34.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:34.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:34.506 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:34.536 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:34.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:34.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:34.637 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:34.670 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:34.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:34.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:34.772 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:34.795 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:34.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:34.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:34.897 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:34.925 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:34.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:34.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:35.027 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:35.055 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:35.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:35.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:35.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:34 smithi196 ceph-mon[84784]: pgmap v480: 89 pgs: 89 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-21T10:09:35.157 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:35.183 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:35.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:35.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:35.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:34 smithi098 ceph-mon[97580]: pgmap v480: 89 pgs: 89 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-21T10:09:35.236 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:34 smithi098 ceph-mon[100914]: pgmap v480: 89 pgs: 89 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-21T10:09:35.285 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:35.308 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:35.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:35.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:35.391 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 47 16 80409 80393 106.879 68.6875 0.0969512 0.00934724 2024-09-21T10:09:35.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:35.444 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:35.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:35.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:35.545 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:35.581 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:35.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:35.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:35.683 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:35.712 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:35.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:35.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:35.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:35.852 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:35.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:35.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:35.953 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:35.979 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:35.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:35.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:36.080 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:36.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:36.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:36.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:36.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:36.237 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:36.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:36.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:36.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:36.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:36.365 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:36.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:36.391 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 48 16 81571 81555 106.165 72.625 0.012584 0.009414 2024-09-21T10:09:36.465 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:36.491 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:36.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:36.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:36.593 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:36.646 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:36.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:36.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:36.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:36.773 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:36.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:36.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:36.875 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:36.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:36.913 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:36.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:37.014 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:37.055 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:37.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:37.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:36 smithi196 ceph-mon[84784]: pgmap v481: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-21T10:09:37.156 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:37.172 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:36 smithi098 ceph-mon[97580]: pgmap v481: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-21T10:09:37.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:36 smithi098 ceph-mon[100914]: pgmap v481: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-21T10:09:37.186 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:37.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:37.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:37.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:37.325 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:37.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:37.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:37.391 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 49 16 82613 82597 105.328 65.125 0.00505876 0.00948817 2024-09-21T10:09:37.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:37.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:37.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:37.563 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:37.590 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:37.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:37.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:37.692 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:37.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:37.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:37.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:37.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:37.862 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:37.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:37.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:37.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:38.035 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:38.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:38.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:38.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:38.094 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:38.137 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:38.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:38.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:38.174 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:38.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:38.275 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:38.312 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:38.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:38.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:38.391 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 50 16 83756 83740 104.65 71.4375 0.00704117 0.00955026 2024-09-21T10:09:38.414 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:38.450 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:38.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:38.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:38.554 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:38.596 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:38.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:38.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:38.697 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:38.727 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:38.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:38.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:38.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:38.872 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:38.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:38.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:38.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:38 smithi196 ceph-mon[84784]: pgmap v482: 89 pgs: 89 active+clean; 4.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-21T10:09:38.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:39.019 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:39.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:39.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:39.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:38 smithi098 ceph-mon[97580]: pgmap v482: 89 pgs: 89 active+clean; 4.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-21T10:09:39.090 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:38 smithi098 ceph-mon[100914]: pgmap v482: 89 pgs: 89 active+clean; 4.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-21T10:09:39.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:39.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:39.182 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:39.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:39.283 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:39.311 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:39.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:39.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:39.391 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 51 16 84925 84909 104.03 73.0625 0.0318019 0.00960692 2024-09-21T10:09:39.414 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:39.440 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:39.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:39.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:39.542 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:39.568 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:39.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:39.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:39.670 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:39.720 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:39.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:39.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:39.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:39.862 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:39.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:39.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:39.964 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:39.992 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:39.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:39.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:40.093 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:40.121 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:40.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:40.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:40.223 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:40.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:40.251 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:40.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:40.352 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:40.391 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 52 16 85991 85975 103.311 66.625 0.012473 0.00967315 2024-09-21T10:09:40.392 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:40.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:40.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:40.494 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:40.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:40.541 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:40.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:40.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:09:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:09:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:09:40.642 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:40.668 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:40.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:40.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:40.769 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:40.811 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:40.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:40.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:40.895 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:09:40.914 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:40.917 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:40 smithi098 ceph-mon[97580]: pgmap v483: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-21T10:09:40.917 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:40 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2624647444' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:09:40.917 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:40 smithi098 ceph-mon[100914]: pgmap v483: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-21T10:09:40.917 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:40 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2624647444' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:09:40.918 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:09:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:09:40] "GET /metrics HTTP/1.1" 200 36450 "" "Prometheus/2.51.0" 2024-09-21T10:09:40.954 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:40.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:40.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:41.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:41.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:41.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:41.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:40 smithi196 ceph-mon[84784]: pgmap v483: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-21T10:09:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:40 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2624647444' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:09:41.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:41.230 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:41.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:41.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:41.332 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:41.368 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:41.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:41.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:41.391 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 53 16 87145 87129 102.723 72.125 0.0139897 0.00973037 2024-09-21T10:09:41.470 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:41.497 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:41.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:41.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:41.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:41.629 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:41.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:41.731 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:41.759 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:41.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:41.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:41.861 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:41.889 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:41.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:41.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:41.990 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:42.015 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:42.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:42.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:42.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:41 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:09:42.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:41 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2624647444' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:09:42.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:41 smithi098 ceph-mon[97580]: osdmap e172: 8 total, 8 up, 7 in 2024-09-21T10:09:42.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:41 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:09:42.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:41 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2624647444' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:09:42.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:41 smithi098 ceph-mon[100914]: osdmap e172: 8 total, 8 up, 7 in 2024-09-21T10:09:42.116 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:41 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:09:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:41 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2624647444' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:09:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:41 smithi196 ceph-mon[84784]: osdmap e172: 8 total, 8 up, 7 in 2024-09-21T10:09:42.143 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:42.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:42.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:42.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:42.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:42.291 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:42.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:42.392 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:42.393 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 54 16 88329 88313 102.191 74 0.00641289 0.00978013 2024-09-21T10:09:42.430 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:42.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:42.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:42.532 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:42.563 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:42.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:42.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:42.665 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:42.690 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:42.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:42.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:42.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:42.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:42.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:42.928 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:42.954 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:42.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:42.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:42.980 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:42 smithi098 ceph-mon[97580]: pgmap v485: 89 pgs: 89 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T10:09:42.980 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:42 smithi098 ceph-mon[100914]: pgmap v485: 89 pgs: 89 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T10:09:43.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:43.100 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:43.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:43.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:43.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:43.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:43.239 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:43.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:43.295 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:09:43.295 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:09:43.340 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:43.367 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:43.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:43.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:42 smithi196 ceph-mon[84784]: pgmap v485: 89 pgs: 89 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T10:09:43.392 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 55 16 89438 89422 101.593 69.3125 0.00831057 0.00983784 2024-09-21T10:09:43.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:43.497 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:43.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:43.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:43.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:43.629 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:43.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:43.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:43.731 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:43.771 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:43.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:43.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:43.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:09:43.873 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:43.896 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:43.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:43.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:43.998 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:44.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:44.037 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:44.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:44.134 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:09:44.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:44.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:44.246 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:44.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:44.286 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:44.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:44.393 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 56 16 90485 90469 100.945 65.4375 0.0262278 0.00990059 2024-09-21T10:09:44.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:44.407 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:44.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:44.508 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:44.536 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:44.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:44.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:44.637 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:44.666 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:44.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:44.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:44.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:09:44.767 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:44.793 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:44.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:44.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:44.895 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:44.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:44.949 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:44.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:45.023 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:44 smithi098 ceph-mon[97580]: pgmap v486: 89 pgs: 89 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T10:09:45.024 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:44 smithi098 ceph-mon[100914]: pgmap v486: 89 pgs: 89 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T10:09:45.050 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:45.076 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:45.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:45.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:45.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:45.216 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:45.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:45.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:45.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:45.356 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:45.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:45.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:45.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:44 smithi196 ceph-mon[84784]: pgmap v486: 89 pgs: 89 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T10:09:45.393 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 57 16 91532 91516 100.322 65.4375 0.0116542 0.00996325 2024-09-21T10:09:45.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:45.503 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:45.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:45.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:45.605 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:45.639 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:45.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:45.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:45.741 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:45.766 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:45.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:45.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:45.868 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:45.895 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:45.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:45.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:45.997 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:46.014 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:46.022 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:46.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:46.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:46.124 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:09:46.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T10:09:46.124 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:46.150 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:46.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:46.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:46.187 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:46.252 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:46.278 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:46.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:46.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:46.380 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:46.395 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 58 16 92592 92576 99.7322 66.25 0.0127779 0.0100216 2024-09-21T10:09:46.456 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:46.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:46.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:46.558 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:46.587 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:46.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:46.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:46.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:46.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:46.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:46.831 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:46.859 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:46.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:46.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:46.962 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:46.991 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:46.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:46.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:47.093 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:47.122 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:47.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:47.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:47.224 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:47.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:46 smithi098 ceph-mon[97580]: pgmap v487: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T10:09:47.237 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:46 smithi098 ceph-mon[100914]: pgmap v487: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T10:09:47.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:47.274 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:47.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:47.375 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:47.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:46 smithi196 ceph-mon[84784]: pgmap v487: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T10:09:47.394 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 59 16 93623 93607 99.1338 64.4375 0.0137291 0.0100813 2024-09-21T10:09:47.401 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:47.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:47.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:47.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:47.539 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:47.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:47.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:47.641 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:47.669 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:47.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:47.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:47.771 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:47.798 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:47.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:47.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:47.900 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:47.929 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:47.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:47.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:48.030 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:48.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:48.059 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:48.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:48.160 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:48.195 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:48.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:48.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:48.297 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:48.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:48.340 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:48.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:48.395 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:09:48.393785+0000 min lat: 0.00114388 max lat: 0.233826 avg lat: 0.0101438 2024-09-21T10:09:48.395 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:09:48.395 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 60 16 94633 94617 98.5336 63.125 0.00801941 0.0101438 2024-09-21T10:09:48.441 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:48.479 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:48.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:48.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:48.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:48.625 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:48.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:48.727 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:48.763 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:48.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:48.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:48.865 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:48.891 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:48.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:48.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:48.992 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:49.016 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:49.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:49.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:49.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:48 smithi098 ceph-mon[97580]: pgmap v488: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-21T10:09:49.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:48 smithi098 ceph-mon[100914]: pgmap v488: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-21T10:09:49.118 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:49.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:48 smithi196 ceph-mon[84784]: pgmap v488: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-21T10:09:49.146 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:49.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:49.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:49.247 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:49.275 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:49.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:49.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:49.377 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:49.395 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 61 16 95677 95661 97.988 65.25 0.00770702 0.0102 2024-09-21T10:09:49.403 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:49.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:49.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:49.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:49.544 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:49.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:49.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:49.646 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:49.675 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:49.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:49.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:49.777 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:49.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:49.898 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:49.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:09:49.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:50.042 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:50.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:50.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:50.144 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:50.165 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:50.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:50.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:50.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:49 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1375227144' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:09:50.267 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:50.293 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:50.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:50.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:50.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:49 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1375227144' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:09:50.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:49 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1375227144' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:09:50.395 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 62 16 96651 96635 97.3894 60.875 0.0108506 0.0102628 2024-09-21T10:09:50.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:50.431 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:50.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:50.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:50.532 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:50.576 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:50.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:50.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:50.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:09:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:09:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:09:50.678 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:50.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:50.711 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:50.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:50.812 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:50.840 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:50.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:50.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:50.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:09:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:09:50] "GET /metrics HTTP/1.1" 200 36450 "" "Prometheus/2.51.0" 2024-09-21T10:09:50.941 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:50.967 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:50.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:50.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:51.069 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:51.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:51.104 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:51.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:51.205 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:51.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:51.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:51.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:51.333 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:51.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:50 smithi098 ceph-mon[100914]: pgmap v489: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-21T10:09:51.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:50 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:09:51.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:50 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:09:51.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:50 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1375227144' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:09:51.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:50 smithi098 ceph-mon[100914]: osdmap e173: 8 total, 8 up, 7 in 2024-09-21T10:09:51.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:50 smithi098 ceph-mon[97580]: pgmap v489: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-21T10:09:51.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:50 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:09:51.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:50 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:09:51.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:50 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1375227144' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:09:51.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:50 smithi098 ceph-mon[97580]: osdmap e173: 8 total, 8 up, 7 in 2024-09-21T10:09:51.359 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:51.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:51.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:51.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:50 smithi196 ceph-mon[84784]: pgmap v489: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-21T10:09:51.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:50 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:09:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:50 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:09:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:50 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1375227144' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:09:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:50 smithi196 ceph-mon[84784]: osdmap e173: 8 total, 8 up, 7 in 2024-09-21T10:09:51.395 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 63 16 97658 97642 96.8425 62.9375 0.0156811 0.0103208 2024-09-21T10:09:51.461 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:51.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:51.495 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:51.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:51.596 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:51.622 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:51.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:51.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:51.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:51.764 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:51.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:51.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:51.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:51.904 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:51.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:51.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:52.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:52.038 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:52.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:52.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:52.140 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:52.179 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:52.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:52.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:52.280 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:52.302 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:52.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:52.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:52.395 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 64 16 98571 98555 96.221 57.0625 0.00444556 0.0103741 2024-09-21T10:09:52.404 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:52.438 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:52.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:52.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:52.540 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:52.568 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:52.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:52.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:52.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:52.695 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:52.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:52.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:52.797 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:52.830 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:52.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:52.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:52.932 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:52.966 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:52.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:52.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:53.067 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:53.108 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:53.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:53.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:53.210 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:53.239 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:53.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:53.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:53.341 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:53.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:52 smithi098 ceph-mon[97580]: pgmap v491: 89 pgs: 89 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-21T10:09:53.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:09:53.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:53.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:52 smithi098 ceph-mon[100914]: pgmap v491: 89 pgs: 89 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-21T10:09:53.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:09:53.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:53.382 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:53.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:53.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:53.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:52 smithi196 ceph-mon[84784]: pgmap v491: 89 pgs: 89 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-21T10:09:53.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:09:53.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:53.395 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 65 15 99590 99575 95.7215 63.75 0.0180096 0.0104419 2024-09-21T10:09:53.483 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:53.521 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:53.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:53.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:53.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:53.657 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:53.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:53.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:53.721 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:09:53.722 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:09:53.759 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:09:53.759 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:09:53.759 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:53.788 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:53.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:53.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:53.794 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:09:53.795 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:09:53.831 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:09:53.831 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:09:53.878 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:09:53.879 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:09:53.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:53.910 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:09:53.910 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:09:53.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:53.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:53.939 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:09:53.939 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:09:53.968 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:09:53.969 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:09:54.038 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:54.078 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:54.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:54.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:54.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:53 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3544037054' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:54.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:53 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3544037054' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:54.180 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:54.208 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:54.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:54.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:54.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:54.335 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:54.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:54.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:54.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:53 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3544037054' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:54.395 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 66 16 100559 100543 95.1878 60.5 0.0275205 0.0105005 2024-09-21T10:09:54.436 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:54.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:54.482 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:54.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:54.583 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:54.610 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:54.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:54.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:54.637 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:54.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:54.764 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:54.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:54.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:54.866 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:54.893 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:54.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:54.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:54.995 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:55.026 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:55.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:55.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:55.040 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:54 smithi098 ceph-mon[97580]: pgmap v492: 89 pgs: 89 active+clean; 5.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:09:55.042 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:54 smithi098 ceph-mon[100914]: pgmap v492: 89 pgs: 89 active+clean; 5.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:09:55.128 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:55.154 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:55.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:55.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:55.256 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:55.283 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:55.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:55.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:55.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:55.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:54 smithi196 ceph-mon[84784]: pgmap v492: 89 pgs: 89 active+clean; 5.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:09:55.395 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 67 16 101608 101592 94.7456 65.5625 0.020947 0.0105491 2024-09-21T10:09:55.431 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:55.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:55.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:55.532 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:55.557 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:55.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:55.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:55.659 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:55.700 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:55.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:55.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:55.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:55.827 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:55.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:55.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:55.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:09:55.888 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:09:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-21T10:09:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-21T10:09:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:09:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:159908 5.5 5:a6d09669:::benchmark_data_smithi098_135187_object24270:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-21T10:09:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:05:46.477242+0000", 2024-09-21T10:09:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 249.40504906499999, 2024-09-21T10:09:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.120525016, 2024-09-21T10:09:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:09:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:09:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:09:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:09:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:09:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 159908 2024-09-21T10:09:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:09:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:09:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.477242+0000", 2024-09-21T10:09:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:09:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.477242+0000", 2024-09-21T10:09:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:09:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.477244+0000", 2024-09-21T10:09:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8530000000000001e-06 2024-09-21T10:09:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:09:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.477248+0000", 2024-09-21T10:09:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.371e-06 2024-09-21T10:09:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:09:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.477263+0000", 2024-09-21T10:09:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5180999999999999e-05 2024-09-21T10:09:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:09:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.477270+0000", 2024-09-21T10:09:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0899999999999999e-06 2024-09-21T10:09:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.477366+0000", 2024-09-21T10:09:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6495000000000004e-05 2024-09-21T10:09:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:09:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.481511+0000", 2024-09-21T10:09:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0041445709999999997 2024-09-21T10:09:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:09:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.481943+0000", 2024-09-21T10:09:55.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000432473 2024-09-21T10:09:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.487353+0000", 2024-09-21T10:09:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054094390000000003 2024-09-21T10:09:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:09:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.487363+0000", 2024-09-21T10:09:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8290000000000003e-06 2024-09-21T10:09:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.893 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.597665+0000", 2024-09-21T10:09:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.110302076 2024-09-21T10:09:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:09:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.597717+0000", 2024-09-21T10:09:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2305999999999997e-05 2024-09-21T10:09:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:09:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.597767+0000", 2024-09-21T10:09:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0331999999999999e-05 2024-09-21T10:09:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:55.894 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:159917 5.5 5:a0cfe005:::benchmark_data_smithi098_135187_object24279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-21T10:09:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:05:46.486343+0000", 2024-09-21T10:09:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 249.39594815300001, 2024-09-21T10:09:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11158973799999999, 2024-09-21T10:09:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:09:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:09:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:09:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:09:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:09:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 159917 2024-09-21T10:09:55.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:09:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:09:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.486343+0000", 2024-09-21T10:09:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:09:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.486343+0000", 2024-09-21T10:09:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:09:55.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.486344+0000", 2024-09-21T10:09:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8200000000000008e-07 2024-09-21T10:09:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:09:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.486348+0000", 2024-09-21T10:09:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.907e-06 2024-09-21T10:09:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:09:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.486363+0000", 2024-09-21T10:09:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5014999999999999e-05 2024-09-21T10:09:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:09:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.486370+0000", 2024-09-21T10:09:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9689999999999997e-06 2024-09-21T10:09:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.487399+0000", 2024-09-21T10:09:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010287 2024-09-21T10:09:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:09:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.487508+0000", 2024-09-21T10:09:55.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010877199999999999 2024-09-21T10:09:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:09:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.487631+0000", 2024-09-21T10:09:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012340100000000001 2024-09-21T10:09:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.497276+0000", 2024-09-21T10:09:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.009644629 2024-09-21T10:09:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.899 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:09:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.497324+0000", 2024-09-21T10:09:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8715999999999999e-05 2024-09-21T10:09:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.597810+0000", 2024-09-21T10:09:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.100485939 2024-09-21T10:09:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:09:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.597897+0000", 2024-09-21T10:09:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7017999999999999e-05 2024-09-21T10:09:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:09:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.597933+0000", 2024-09-21T10:09:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5689999999999999e-05 2024-09-21T10:09:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:159920 5.5 5:a2cf2f9b:::benchmark_data_smithi098_135187_object24282:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-21T10:09:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:05:46.490507+0000", 2024-09-21T10:09:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 249.391784933, 2024-09-21T10:09:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.107792094, 2024-09-21T10:09:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:09:55.901 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:09:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:09:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:09:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:09:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 159920 2024-09-21T10:09:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:09:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:09:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.490507+0000", 2024-09-21T10:09:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:09:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.490507+0000", 2024-09-21T10:09:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:09:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.490508+0000", 2024-09-21T10:09:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7199999999999997e-07 2024-09-21T10:09:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:09:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.490510+0000", 2024-09-21T10:09:55.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8229999999999999e-06 2024-09-21T10:09:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:09:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.490517+0000", 2024-09-21T10:09:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.618e-06 2024-09-21T10:09:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:09:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.490521+0000", 2024-09-21T10:09:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2869999999999998e-06 2024-09-21T10:09:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.492208+0000", 2024-09-21T10:09:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016866629999999999 2024-09-21T10:09:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:09:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.492439+0000", 2024-09-21T10:09:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023144699999999999 2024-09-21T10:09:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:09:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.492772+0000", 2024-09-21T10:09:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00033275299999999998 2024-09-21T10:09:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.497311+0000", 2024-09-21T10:09:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0045389710000000002 2024-09-21T10:09:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:09:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.497361+0000", 2024-09-21T10:09:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9960000000000003e-05 2024-09-21T10:09:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.598116+0000", 2024-09-21T10:09:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.100755231 2024-09-21T10:09:55.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:09:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.598258+0000", 2024-09-21T10:09:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014189300000000001 2024-09-21T10:09:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:09:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.598299+0000", 2024-09-21T10:09:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0475999999999997e-05 2024-09-21T10:09:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:55.907 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:267668 5.5 5:a5d33d04:::benchmark_data_smithi098_135187_object132030:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-21T10:09:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:59.352446+0000", 2024-09-21T10:09:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 116.529845309, 2024-09-21T10:09:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.275669706, 2024-09-21T10:09:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:09:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:09:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-21T10:09:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:09:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:09:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:09:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 267668 2024-09-21T10:09:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.908 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:09:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:09:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.352446+0000", 2024-09-21T10:09:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:09:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.352446+0000", 2024-09-21T10:09:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:09:55.909 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.352448+0000", 2024-09-21T10:09:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4479999999999999e-06 2024-09-21T10:09:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:09:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.352452+0000", 2024-09-21T10:09:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0799999999999999e-06 2024-09-21T10:09:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:09:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.352461+0000", 2024-09-21T10:09:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2129999999999997e-06 2024-09-21T10:09:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:09:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.352468+0000", 2024-09-21T10:09:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7270000000000001e-06 2024-09-21T10:09:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.357875+0000", 2024-09-21T10:09:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005407439 2024-09-21T10:09:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:09:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.358048+0000", 2024-09-21T10:09:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000172547 2024-09-21T10:09:55.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:09:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.358165+0000", 2024-09-21T10:09:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011770899999999999 2024-09-21T10:09:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:09:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.393683+0000", 2024-09-21T10:09:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035517497000000002 2024-09-21T10:09:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.397071+0000", 2024-09-21T10:09:55.912 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033885159999999998 2024-09-21T10:09:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.397082+0000", 2024-09-21T10:09:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0651e-05 2024-09-21T10:09:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.370728+0000", 2024-09-21T10:09:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.97364649400000003 2024-09-21T10:09:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.370741+0000", 2024-09-21T10:09:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2497000000000001e-05 2024-09-21T10:09:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.628039+0000", 2024-09-21T10:09:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.257298408 2024-09-21T10:09:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:09:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.628116+0000", 2024-09-21T10:09:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.648e-05 2024-09-21T10:09:55.914 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:267675 5.5 5:ad0bc8a7:::benchmark_data_smithi098_135187_object132037:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-21T10:09:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:59.361649+0000", 2024-09-21T10:09:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 116.52064242599999, 2024-09-21T10:09:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2667899550000001, 2024-09-21T10:09:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:09:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:09:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-21T10:09:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:09:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:09:55.915 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:09:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 267675 2024-09-21T10:09:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:09:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:09:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.361649+0000", 2024-09-21T10:09:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:09:55.917 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.361649+0000", 2024-09-21T10:09:55.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.917 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:09:55.917 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.361649+0000", 2024-09-21T10:09:55.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.08e-07 2024-09-21T10:09:55.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:09:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.361652+0000", 2024-09-21T10:09:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1969999999999999e-06 2024-09-21T10:09:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:09:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.361658+0000", 2024-09-21T10:09:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8299999999999998e-06 2024-09-21T10:09:55.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:09:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.361664+0000", 2024-09-21T10:09:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8239999999999996e-06 2024-09-21T10:09:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.363513+0000", 2024-09-21T10:09:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001848274 2024-09-21T10:09:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:09:55.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.363624+0000", 2024-09-21T10:09:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000111054 2024-09-21T10:09:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:09:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.363741+0000", 2024-09-21T10:09:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000117742 2024-09-21T10:09:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:09:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.393688+0000", 2024-09-21T10:09:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029946719 2024-09-21T10:09:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.397320+0000", 2024-09-21T10:09:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003632372 2024-09-21T10:09:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.397329+0000", 2024-09-21T10:09:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4160000000000004e-06 2024-09-21T10:09:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.370980+0000", 2024-09-21T10:09:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.97365082300000005 2024-09-21T10:09:55.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.370989+0000", 2024-09-21T10:09:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6399999999999992e-06 2024-09-21T10:09:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.628380+0000", 2024-09-21T10:09:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.25739081 2024-09-21T10:09:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:09:55.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.628439+0000", 2024-09-21T10:09:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8845999999999999e-05 2024-09-21T10:09:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:267677 5.5 5:a8337b7d:::benchmark_data_smithi098_135187_object132039:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-21T10:09:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:59.364087+0000", 2024-09-21T10:09:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 116.51820419400001, 2024-09-21T10:09:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3165017990000001, 2024-09-21T10:09:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:09:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:09:55.923 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-21T10:09:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:09:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:09:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:09:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 267677 2024-09-21T10:09:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:09:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:09:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.364087+0000", 2024-09-21T10:09:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:09:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.364087+0000", 2024-09-21T10:09:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:09:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.364088+0000", 2024-09-21T10:09:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.19e-06 2024-09-21T10:09:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:09:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.364093+0000", 2024-09-21T10:09:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.053e-06 2024-09-21T10:09:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:09:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.364101+0000", 2024-09-21T10:09:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2600000000000005e-06 2024-09-21T10:09:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:09:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.364113+0000", 2024-09-21T10:09:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1762e-05 2024-09-21T10:09:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.388373+0000", 2024-09-21T10:09:55.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.024260223000000001 2024-09-21T10:09:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:09:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.388486+0000", 2024-09-21T10:09:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000112761 2024-09-21T10:09:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:09:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.388606+0000", 2024-09-21T10:09:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000120222 2024-09-21T10:09:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.397358+0000", 2024-09-21T10:09:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0087526070000000008 2024-09-21T10:09:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.397366+0000", 2024-09-21T10:09:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4950000000000002e-06 2024-09-21T10:09:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.371022+0000", 2024-09-21T10:09:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.97365609399999997 2024-09-21T10:09:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.371030+0000", 2024-09-21T10:09:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2230000000000007e-06 2024-09-21T10:09:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.628470+0000", 2024-09-21T10:09:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.25743984 2024-09-21T10:09:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for missing object", 2024-09-21T10:09:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.628608+0000", 2024-09-21T10:09:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013848500000000001 2024-09-21T10:09:55.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.680505+0000", 2024-09-21T10:09:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.051896389000000001 2024-09-21T10:09:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:09:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.680589+0000", 2024-09-21T10:09:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4195000000000003e-05 2024-09-21T10:09:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:55.930 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:267679 5.e 5:7bcc2abb:::benchmark_data_smithi098_135187_object132041:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:09:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:59.386526+0000", 2024-09-21T10:09:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 116.49576527799999, 2024-09-21T10:09:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2384244130000002, 2024-09-21T10:09:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:09:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:09:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-21T10:09:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:09:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:09:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:09:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 267679 2024-09-21T10:09:55.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:09:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:09:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.386526+0000", 2024-09-21T10:09:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:09:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.386526+0000", 2024-09-21T10:09:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:09:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.386528+0000", 2024-09-21T10:09:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3170000000000001e-06 2024-09-21T10:09:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:09:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.386531+0000", 2024-09-21T10:09:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.659e-06 2024-09-21T10:09:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:09:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.386541+0000", 2024-09-21T10:09:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8889999999999997e-06 2024-09-21T10:09:55.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:09:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.386554+0000", 2024-09-21T10:09:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2652e-05 2024-09-21T10:09:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.386650+0000", 2024-09-21T10:09:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5895000000000003e-05 2024-09-21T10:09:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-09-21T10:09:55.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.386657+0000", 2024-09-21T10:09:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5329999999999997e-06 2024-09-21T10:09:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.390670+0000", 2024-09-21T10:09:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040127299999999999 2024-09-21T10:09:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.390679+0000", 2024-09-21T10:09:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0529999999999996e-06 2024-09-21T10:09:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.375786+0000", 2024-09-21T10:09:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.98510710999999995 2024-09-21T10:09:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.375798+0000", 2024-09-21T10:09:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2119e-05 2024-09-21T10:09:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.623183+0000", 2024-09-21T10:09:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.247384549 2024-09-21T10:09:55.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:09:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.624951+0000", 2024-09-21T10:09:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001767907 2024-09-21T10:09:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:267689 5.e 5:7ba73ab0:::benchmark_data_smithi098_135187_object132051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:09:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:59.399429+0000", 2024-09-21T10:09:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 116.48286285099999, 2024-09-21T10:09:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2613909689999998, 2024-09-21T10:09:55.937 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:09:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:09:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:09:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:09:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:09:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 267689 2024-09-21T10:09:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:09:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:09:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399429+0000", 2024-09-21T10:09:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:09:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399429+0000", 2024-09-21T10:09:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:09:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399429+0000", 2024-09-21T10:09:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6499999999999999e-07 2024-09-21T10:09:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:09:55.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399431+0000", 2024-09-21T10:09:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9539999999999998e-06 2024-09-21T10:09:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:09:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399438+0000", 2024-09-21T10:09:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1189999999999999e-06 2024-09-21T10:09:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:09:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399444+0000", 2024-09-21T10:09:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5269999999999998e-06 2024-09-21T10:09:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399500+0000", 2024-09-21T10:09:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6301999999999997e-05 2024-09-21T10:09:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399509+0000", 2024-09-21T10:09:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0289999999999994e-06 2024-09-21T10:09:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.375824+0000", 2024-09-21T10:09:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.97631503500000005 2024-09-21T10:09:55.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.375842+0000", 2024-09-21T10:09:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8420999999999999e-05 2024-09-21T10:09:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.625192+0000", 2024-09-21T10:09:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2493493739999999 2024-09-21T10:09:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:09:55.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.625895+0000", 2024-09-21T10:09:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000702727 2024-09-21T10:09:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,4,5", 2024-09-21T10:09:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.626182+0000", 2024-09-21T10:09:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00028767899999999999 2024-09-21T10:09:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.642462+0000", 2024-09-21T10:09:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016279708 2024-09-21T10:09:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:09:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.650708+0000", 2024-09-21T10:09:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0082457369999999995 2024-09-21T10:09:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.651949+0000", 2024-09-21T10:09:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0012408160000000001 2024-09-21T10:09:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.660641+0000", 2024-09-21T10:09:55.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0086921540000000005 2024-09-21T10:09:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:09:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.660704+0000", 2024-09-21T10:09:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3021000000000005e-05 2024-09-21T10:09:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:09:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.660820+0000", 2024-09-21T10:09:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000115901 2024-09-21T10:09:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:267691 5.5 5:a3a772a8:::benchmark_data_smithi098_135187_object132053:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:09:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:59.399455+0000", 2024-09-21T10:09:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 116.482836714, 2024-09-21T10:09:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2940255839999999, 2024-09-21T10:09:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:09:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:09:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:09:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:09:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:09:55.946 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 267691 2024-09-21T10:09:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:09:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:09:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399455+0000", 2024-09-21T10:09:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:09:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399455+0000", 2024-09-21T10:09:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:09:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399455+0000", 2024-09-21T10:09:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0999999999999999e-07 2024-09-21T10:09:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:09:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399457+0000", 2024-09-21T10:09:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.841e-06 2024-09-21T10:09:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:09:55.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399463+0000", 2024-09-21T10:09:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0209999999999996e-06 2024-09-21T10:09:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:09:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399468+0000", 2024-09-21T10:09:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9219999999999996e-06 2024-09-21T10:09:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399505+0000", 2024-09-21T10:09:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7392999999999999e-05 2024-09-21T10:09:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:55.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399514+0000", 2024-09-21T10:09:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9809999999999992e-06 2024-09-21T10:09:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.371060+0000", 2024-09-21T10:09:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.97154581799999995 2024-09-21T10:09:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.371068+0000", 2024-09-21T10:09:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6909999999999997e-06 2024-09-21T10:09:55.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.629377+0000", 2024-09-21T10:09:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.258309079 2024-09-21T10:09:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:09:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.629473+0000", 2024-09-21T10:09:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6223000000000004e-05 2024-09-21T10:09:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4,6", 2024-09-21T10:09:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.629999+0000", 2024-09-21T10:09:55.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00052632899999999997 2024-09-21T10:09:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.671975+0000", 2024-09-21T10:09:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.041975483000000001 2024-09-21T10:09:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.672210+0000", 2024-09-21T10:09:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000235393 2024-09-21T10:09:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.686921+0000", 2024-09-21T10:09:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014710938 2024-09-21T10:09:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:09:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.693241+0000", 2024-09-21T10:09:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0063199600000000003 2024-09-21T10:09:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:09:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.693356+0000", 2024-09-21T10:09:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000115003 2024-09-21T10:09:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:09:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.693480+0000", 2024-09-21T10:09:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012409899999999999 2024-09-21T10:09:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:267695 5.c 5:39d3fd47:::benchmark_data_smithi098_135187_object132057:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:09:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:59.399499+0000", 2024-09-21T10:09:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 116.482792836, 2024-09-21T10:09:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2242494829999999, 2024-09-21T10:09:55.954 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:09:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:09:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:09:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:09:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:09:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 267695 2024-09-21T10:09:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:09:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:09:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399499+0000", 2024-09-21T10:09:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:09:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399499+0000", 2024-09-21T10:09:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:09:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399499+0000", 2024-09-21T10:09:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7300000000000002e-07 2024-09-21T10:09:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:09:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399501+0000", 2024-09-21T10:09:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8920000000000001e-06 2024-09-21T10:09:55.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:09:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399506+0000", 2024-09-21T10:09:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3600000000000004e-06 2024-09-21T10:09:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:09:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399511+0000", 2024-09-21T10:09:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8169999999999998e-06 2024-09-21T10:09:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399557+0000", 2024-09-21T10:09:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5571000000000001e-05 2024-09-21T10:09:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399565+0000", 2024-09-21T10:09:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.96e-06 2024-09-21T10:09:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.373376+0000", 2024-09-21T10:09:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.973810972 2024-09-21T10:09:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.373386+0000", 2024-09-21T10:09:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0835999999999999e-05 2024-09-21T10:09:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.604903+0000", 2024-09-21T10:09:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2315162989999999 2024-09-21T10:09:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:09:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.605037+0000", 2024-09-21T10:09:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013456500000000001 2024-09-21T10:09:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4,6", 2024-09-21T10:09:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.605584+0000", 2024-09-21T10:09:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00054704600000000003 2024-09-21T10:09:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.618865+0000", 2024-09-21T10:09:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013280955000000001 2024-09-21T10:09:55.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:55.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.618913+0000", 2024-09-21T10:09:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8068999999999997e-05 2024-09-21T10:09:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.618956+0000", 2024-09-21T10:09:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2277000000000002e-05 2024-09-21T10:09:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:09:55.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.623242+0000", 2024-09-21T10:09:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0042863010000000002 2024-09-21T10:09:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:09:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.623639+0000", 2024-09-21T10:09:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000397141 2024-09-21T10:09:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:09:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.623748+0000", 2024-09-21T10:09:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010904900000000001 2024-09-21T10:09:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:55.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:267702 5.c 5:38c772c1:::benchmark_data_smithi098_135187_object132064:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:09:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:59.417278+0000", 2024-09-21T10:09:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 116.465013763, 2024-09-21T10:09:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2094516300000002, 2024-09-21T10:09:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:09:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:09:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:09:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:09:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:09:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 267702 2024-09-21T10:09:55.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:09:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:09:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.417278+0000", 2024-09-21T10:09:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:09:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.417278+0000", 2024-09-21T10:09:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:09:55.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.417279+0000", 2024-09-21T10:09:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5030000000000001e-06 2024-09-21T10:09:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:09:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.417285+0000", 2024-09-21T10:09:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4249999999999997e-06 2024-09-21T10:09:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:09:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.417298+0000", 2024-09-21T10:09:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2925000000000001e-05 2024-09-21T10:09:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:09:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.417305+0000", 2024-09-21T10:09:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4490000000000002e-06 2024-09-21T10:09:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.417361+0000", 2024-09-21T10:09:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5587000000000002e-05 2024-09-21T10:09:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.417368+0000", 2024-09-21T10:09:55.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7710000000000006e-06 2024-09-21T10:09:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.373598+0000", 2024-09-21T10:09:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.95622994900000002 2024-09-21T10:09:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.373607+0000", 2024-09-21T10:09:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7399999999999993e-06 2024-09-21T10:09:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.619151+0000", 2024-09-21T10:09:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2455438510000001 2024-09-21T10:09:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:09:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.619286+0000", 2024-09-21T10:09:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000135504 2024-09-21T10:09:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,4,6", 2024-09-21T10:09:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.619441+0000", 2024-09-21T10:09:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015505900000000001 2024-09-21T10:09:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.623787+0000", 2024-09-21T10:09:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043455109999999998 2024-09-21T10:09:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:09:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.623872+0000", 2024-09-21T10:09:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4754999999999997e-05 2024-09-21T10:09:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.624239+0000", 2024-09-21T10:09:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000367528 2024-09-21T10:09:55.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.626336+0000", 2024-09-21T10:09:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002096667 2024-09-21T10:09:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:09:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.626390+0000", 2024-09-21T10:09:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4320999999999997e-05 2024-09-21T10:09:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:09:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.626729+0000", 2024-09-21T10:09:55.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00033908499999999998 2024-09-21T10:09:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269032 5.5 5:a165fcd9:::benchmark_data_smithi098_135187_object133394:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:09:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.462887+0000", 2024-09-21T10:09:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 111.419404528, 2024-09-21T10:09:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0369348970000001, 2024-09-21T10:09:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:09:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:09:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:09:55.972 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:09:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:09:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269032 2024-09-21T10:09:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:09:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:09:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462887+0000", 2024-09-21T10:09:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:09:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462887+0000", 2024-09-21T10:09:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:09:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462889+0000", 2024-09-21T10:09:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0219999999999999e-06 2024-09-21T10:09:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:09:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462895+0000", 2024-09-21T10:09:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8279999999999999e-06 2024-09-21T10:09:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:09:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462910+0000", 2024-09-21T10:09:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5257e-05 2024-09-21T10:09:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:09:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462919+0000", 2024-09-21T10:09:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2939999999999994e-06 2024-09-21T10:09:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462969+0000", 2024-09-21T10:09:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9202000000000001e-05 2024-09-21T10:09:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462974+0000", 2024-09-21T10:09:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6459999999999998e-06 2024-09-21T10:09:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.482308+0000", 2024-09-21T10:09:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0193338759999999 2024-09-21T10:09:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:09:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.482437+0000", 2024-09-21T10:09:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000128768 2024-09-21T10:09:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:09:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.482565+0000", 2024-09-21T10:09:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000127737 2024-09-21T10:09:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.486667+0000", 2024-09-21T10:09:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0041028239999999997 2024-09-21T10:09:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.499478+0000", 2024-09-21T10:09:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012810103 2024-09-21T10:09:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:09:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.499735+0000", 2024-09-21T10:09:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000257264 2024-09-21T10:09:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:09:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.499782+0000", 2024-09-21T10:09:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7305999999999997e-05 2024-09-21T10:09:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:09:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.499822+0000", 2024-09-21T10:09:55.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.977e-05 2024-09-21T10:09:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269034 5.e 5:77cf2ade:::benchmark_data_smithi098_135187_object133396:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:09:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.470275+0000", 2024-09-21T10:09:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 111.412016165, 2024-09-21T10:09:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.040110731, 2024-09-21T10:09:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:09:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:09:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:09:55.979 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:09:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:09:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269034 2024-09-21T10:09:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:09:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:09:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.470275+0000", 2024-09-21T10:09:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:09:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.470275+0000", 2024-09-21T10:09:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:09:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.470277+0000", 2024-09-21T10:09:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5030000000000001e-06 2024-09-21T10:09:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:09:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.470281+0000", 2024-09-21T10:09:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2899999999999996e-06 2024-09-21T10:09:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:09:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.470292+0000", 2024-09-21T10:09:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1195e-05 2024-09-21T10:09:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:09:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.470297+0000", 2024-09-21T10:09:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1399999999999999e-06 2024-09-21T10:09:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.470344+0000", 2024-09-21T10:09:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6499000000000003e-05 2024-09-21T10:09:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.470351+0000", 2024-09-21T10:09:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1629999999999999e-06 2024-09-21T10:09:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.491909+0000", 2024-09-21T10:09:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.021558033 2024-09-21T10:09:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:09:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.492008+0000", 2024-09-21T10:09:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8487999999999996e-05 2024-09-21T10:09:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:09:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.492120+0000", 2024-09-21T10:09:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000112193 2024-09-21T10:09:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:09:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.503657+0000", 2024-09-21T10:09:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011537522 2024-09-21T10:09:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.508543+0000", 2024-09-21T10:09:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048854679999999996 2024-09-21T10:09:55.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510260+0000", 2024-09-21T10:09:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001717491 2024-09-21T10:09:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:09:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510334+0000", 2024-09-21T10:09:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3891000000000001e-05 2024-09-21T10:09:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:09:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510386+0000", 2024-09-21T10:09:55.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1854999999999997e-05 2024-09-21T10:09:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269039 5.5 5:a598dfdb:::benchmark_data_smithi098_135187_object133401:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:09:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.485891+0000", 2024-09-21T10:09:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 111.396400425, 2024-09-21T10:09:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0140555389999999, 2024-09-21T10:09:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:09:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:09:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:09:55.986 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:09:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:09:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269039 2024-09-21T10:09:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:09:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:09:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.485891+0000", 2024-09-21T10:09:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:09:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.485891+0000", 2024-09-21T10:09:55.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:09:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.485893+0000", 2024-09-21T10:09:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.84e-06 2024-09-21T10:09:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:09:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.485898+0000", 2024-09-21T10:09:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0139999999999998e-06 2024-09-21T10:09:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:09:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.485912+0000", 2024-09-21T10:09:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4304000000000001e-05 2024-09-21T10:09:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:09:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.485918+0000", 2024-09-21T10:09:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.789e-06 2024-09-21T10:09:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.485987+0000", 2024-09-21T10:09:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8888999999999997e-05 2024-09-21T10:09:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.485998+0000", 2024-09-21T10:09:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0888e-05 2024-09-21T10:09:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.486700+0000", 2024-09-21T10:09:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.000702228 2024-09-21T10:09:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:09:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.486815+0000", 2024-09-21T10:09:55.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114911 2024-09-21T10:09:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:09:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.486934+0000", 2024-09-21T10:09:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000119429 2024-09-21T10:09:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.499514+0000", 2024-09-21T10:09:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012579966 2024-09-21T10:09:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.499722+0000", 2024-09-21T10:09:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020798799999999999 2024-09-21T10:09:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:09:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.499883+0000", 2024-09-21T10:09:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016047899999999999 2024-09-21T10:09:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:09:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.499918+0000", 2024-09-21T10:09:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4950000000000002e-05 2024-09-21T10:09:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:09:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.499947+0000", 2024-09-21T10:09:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8864e-05 2024-09-21T10:09:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269045 5.5 5:aa267963:::benchmark_data_smithi098_135187_object133407:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:09:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.492092+0000", 2024-09-21T10:09:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 111.390199901, 2024-09-21T10:09:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0122435919999999, 2024-09-21T10:09:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:09:55.993 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:09:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:09:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:09:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:09:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269045 2024-09-21T10:09:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:09:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:09:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.492092+0000", 2024-09-21T10:09:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:09:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.492092+0000", 2024-09-21T10:09:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:09:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.492093+0000", 2024-09-21T10:09:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5540000000000001e-06 2024-09-21T10:09:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:09:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.492098+0000", 2024-09-21T10:09:55.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6070000000000001e-06 2024-09-21T10:09:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:09:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.492117+0000", 2024-09-21T10:09:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8819999999999999e-05 2024-09-21T10:09:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:09:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.492123+0000", 2024-09-21T10:09:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3999999999999997e-06 2024-09-21T10:09:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.492168+0000", 2024-09-21T10:09:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5092000000000001e-05 2024-09-21T10:09:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.492175+0000", 2024-09-21T10:09:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0829999999999998e-06 2024-09-21T10:09:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.495327+0000", 2024-09-21T10:09:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0031521329999999 2024-09-21T10:09:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:09:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.495433+0000", 2024-09-21T10:09:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010600299999999999 2024-09-21T10:09:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:09:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.495543+0000", 2024-09-21T10:09:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000109823 2024-09-21T10:09:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:09:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.499954+0000", 2024-09-21T10:09:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044105890000000003 2024-09-21T10:09:55.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.499985+0000", 2024-09-21T10:09:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0877999999999998e-05 2024-09-21T10:09:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.504256+0000", 2024-09-21T10:09:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004271263 2024-09-21T10:09:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:09:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.504300+0000", 2024-09-21T10:09:55.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4239000000000001e-05 2024-09-21T10:09:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:09:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.504335+0000", 2024-09-21T10:09:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5108000000000003e-05 2024-09-21T10:09:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269051 5.e 5:767e8b84:::benchmark_data_smithi098_135187_object133413:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:09:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.496876+0000", 2024-09-21T10:09:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 111.38541506999999, 2024-09-21T10:09:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0142875090000001, 2024-09-21T10:09:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:09:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:09:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:09:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:09:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:09:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269051 2024-09-21T10:09:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:09:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:09:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496876+0000", 2024-09-21T10:09:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:09:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496876+0000", 2024-09-21T10:09:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:09:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496878+0000", 2024-09-21T10:09:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.203e-06 2024-09-21T10:09:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:09:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496883+0000", 2024-09-21T10:09:56.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7370000000000003e-06 2024-09-21T10:09:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:09:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496895+0000", 2024-09-21T10:09:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1681e-05 2024-09-21T10:09:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:09:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496902+0000", 2024-09-21T10:09:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9040000000000003e-06 2024-09-21T10:09:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:56.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496966+0000", 2024-09-21T10:09:56.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3737000000000003e-05 2024-09-21T10:09:56.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496976+0000", 2024-09-21T10:09:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0540000000000001e-05 2024-09-21T10:09:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.496097+0000", 2024-09-21T10:09:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.99912093400000002 2024-09-21T10:09:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:09:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.496202+0000", 2024-09-21T10:09:56.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105011 2024-09-21T10:09:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:09:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.496309+0000", 2024-09-21T10:09:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010658399999999999 2024-09-21T10:09:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:09:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.503732+0000", 2024-09-21T10:09:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0074232619999999999 2024-09-21T10:09:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510432+0000", 2024-09-21T10:09:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.006700068 2024-09-21T10:09:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511065+0000", 2024-09-21T10:09:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00063310199999999997 2024-09-21T10:09:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:09:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511125+0000", 2024-09-21T10:09:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9633000000000003e-05 2024-09-21T10:09:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:09:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511164+0000", 2024-09-21T10:09:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9113000000000003e-05 2024-09-21T10:09:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269055 5.e 5:7ffeae19:::benchmark_data_smithi098_135187_object133417:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:09:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.505001+0000", 2024-09-21T10:09:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 111.377290042, 2024-09-21T10:09:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.006540456, 2024-09-21T10:09:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:09:56.008 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:09:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:09:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:09:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:09:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269055 2024-09-21T10:09:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:09:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:09:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505001+0000", 2024-09-21T10:09:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:09:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505001+0000", 2024-09-21T10:09:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:09:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505004+0000", 2024-09-21T10:09:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0599999999999999e-06 2024-09-21T10:09:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:09:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505015+0000", 2024-09-21T10:09:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0679999999999999e-05 2024-09-21T10:09:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:09:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505043+0000", 2024-09-21T10:09:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7399e-05 2024-09-21T10:09:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:09:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505057+0000", 2024-09-21T10:09:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4007e-05 2024-09-21T10:09:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505206+0000", 2024-09-21T10:09:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014969899999999999 2024-09-21T10:09:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505214+0000", 2024-09-21T10:09:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.909e-06 2024-09-21T10:09:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.500029+0000", 2024-09-21T10:09:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.99481489899999997 2024-09-21T10:09:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:09:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.500142+0000", 2024-09-21T10:09:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011260999999999999 2024-09-21T10:09:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:09:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.500256+0000", 2024-09-21T10:09:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114169 2024-09-21T10:09:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:09:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.508556+0000", 2024-09-21T10:09:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0083003420000000005 2024-09-21T10:09:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510525+0000", 2024-09-21T10:09:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001969157 2024-09-21T10:09:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511359+0000", 2024-09-21T10:09:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00083408899999999999 2024-09-21T10:09:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:09:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511501+0000", 2024-09-21T10:09:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014195500000000001 2024-09-21T10:09:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:09:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511542+0000", 2024-09-21T10:09:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0481000000000001e-05 2024-09-21T10:09:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269058 5.e 5:7ea5ca31:::benchmark_data_smithi098_135187_object133420:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:09:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.505504+0000", 2024-09-21T10:09:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 111.376787851, 2024-09-21T10:09:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0061627070000001, 2024-09-21T10:09:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:09:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:09:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:09:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:09:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:09:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269058 2024-09-21T10:09:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:09:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:09:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505504+0000", 2024-09-21T10:09:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:09:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505504+0000", 2024-09-21T10:09:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:09:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505504+0000", 2024-09-21T10:09:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5300000000000004e-07 2024-09-21T10:09:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:09:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505507+0000", 2024-09-21T10:09:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9299999999999999e-06 2024-09-21T10:09:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:09:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505516+0000", 2024-09-21T10:09:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3599999999999996e-06 2024-09-21T10:09:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:09:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505520+0000", 2024-09-21T10:09:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3529999999999997e-06 2024-09-21T10:09:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505564+0000", 2024-09-21T10:09:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3803000000000001e-05 2024-09-21T10:09:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:09:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505571+0000", 2024-09-21T10:09:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3300000000000001e-06 2024-09-21T10:09:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.503760+0000", 2024-09-21T10:09:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.99818844699999998 2024-09-21T10:09:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:09:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.503863+0000", 2024-09-21T10:09:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000103799 2024-09-21T10:09:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:09:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.503972+0000", 2024-09-21T10:09:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010820200000000001 2024-09-21T10:09:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:09:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.509003+0000", 2024-09-21T10:09:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050318159999999997 2024-09-21T10:09:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510701+0000", 2024-09-21T10:09:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001697334 2024-09-21T10:09:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511585+0000", 2024-09-21T10:09:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088442700000000002 2024-09-21T10:09:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:09:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511632+0000", 2024-09-21T10:09:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6854000000000002e-05 2024-09-21T10:09:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:09:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511666+0000", 2024-09-21T10:09:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4399e-05 2024-09-21T10:09:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:69646 6.9 6:9a5721c2:::benchmark_data_smithi098_169033_object69645:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T10:09:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:09:26.369680+0000", 2024-09-21T10:09:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 29.512611659000001, 2024-09-21T10:09:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.14260959400000001, 2024-09-21T10:09:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:09:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:09:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:09:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:09:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:09:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 69646 2024-09-21T10:09:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:09:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:09:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.369680+0000", 2024-09-21T10:09:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:09:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.369680+0000", 2024-09-21T10:09:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:56.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:09:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.369681+0000", 2024-09-21T10:09:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3480000000000001e-06 2024-09-21T10:09:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:09:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.369704+0000", 2024-09-21T10:09:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3010000000000002e-05 2024-09-21T10:09:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:09:56.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.369716+0000", 2024-09-21T10:09:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1736000000000001e-05 2024-09-21T10:09:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:09:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.369723+0000", 2024-09-21T10:09:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8050000000000001e-06 2024-09-21T10:09:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.369782+0000", 2024-09-21T10:09:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9738000000000003e-05 2024-09-21T10:09:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:09:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.369915+0000", 2024-09-21T10:09:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013228900000000001 2024-09-21T10:09:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:09:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.370042+0000", 2024-09-21T10:09:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012720900000000001 2024-09-21T10:09:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:09:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.373142+0000", 2024-09-21T10:09:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031001729999999999 2024-09-21T10:09:56.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.373162+0000", 2024-09-21T10:09:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0152e-05 2024-09-21T10:09:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.512013+0000", 2024-09-21T10:09:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.13885022799999999 2024-09-21T10:09:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:09:56.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.512149+0000", 2024-09-21T10:09:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013600900000000001 2024-09-21T10:09:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:09:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.512289+0000", 2024-09-21T10:09:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014089699999999999 2024-09-21T10:09:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:69650 6.9 6:9ce61a45:::benchmark_data_smithi098_169033_object69649:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T10:09:56.030 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:09:26.371076+0000", 2024-09-21T10:09:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 29.511215619000001, 2024-09-21T10:09:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.14173421999999999, 2024-09-21T10:09:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:09:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:09:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:09:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:09:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:09:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 69650 2024-09-21T10:09:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:09:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:09:56.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.371076+0000", 2024-09-21T10:09:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:09:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.371076+0000", 2024-09-21T10:09:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:09:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:09:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.371077+0000", 2024-09-21T10:09:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9999999999999995e-07 2024-09-21T10:09:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:09:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.371096+0000", 2024-09-21T10:09:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9151e-05 2024-09-21T10:09:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:09:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.371110+0000", 2024-09-21T10:09:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4312e-05 2024-09-21T10:09:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:09:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.371118+0000", 2024-09-21T10:09:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8979999999999996e-06 2024-09-21T10:09:56.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:09:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.371884+0000", 2024-09-21T10:09:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00076557299999999997 2024-09-21T10:09:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:09:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.371967+0000", 2024-09-21T10:09:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3203999999999998e-05 2024-09-21T10:09:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:09:56.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.372047+0000", 2024-09-21T10:09:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0348999999999999e-05 2024-09-21T10:09:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:09:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.373272+0000", 2024-09-21T10:09:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0012245170000000001 2024-09-21T10:09:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.373707+0000", 2024-09-21T10:09:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00043515300000000003 2024-09-21T10:09:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.512487+0000", 2024-09-21T10:09:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.13877957299999999 2024-09-21T10:09:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:09:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.512673+0000", 2024-09-21T10:09:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018620900000000001 2024-09-21T10:09:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:09:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:09:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:09:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.512810+0000", 2024-09-21T10:09:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000137281 2024-09-21T10:09:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:56.037 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:56.037 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:56.037 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:09:56.037 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:09:56.037 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:09:56.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:56.054 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:56.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:56.154 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:56.181 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:56.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:56.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:56.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:56.330 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:56.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:56.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:56.397 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 68 16 102536 102520 94.2044 58 0.0126771 0.0106102 2024-09-21T10:09:56.432 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:56.472 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:56.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:56.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:56.574 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:56.598 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:56.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:56.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:56.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:56.736 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:56.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:56.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:56.838 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:56.872 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:56.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:56.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:56.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:57.017 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:57.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:57.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:57.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:09:57.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:56 smithi098 ceph-mon[97580]: pgmap v493: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T10:09:57.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:56 smithi098 ceph-mon[100914]: pgmap v493: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T10:09:57.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:57.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:57.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:57.270 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:57.295 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:57.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:57.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:57.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:56 smithi196 ceph-mon[84784]: pgmap v493: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T10:09:57.397 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:57.398 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 69 16 103508 103492 93.7172 60.75 0.0189821 0.0106649 2024-09-21T10:09:57.437 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:57.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:57.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:57.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:57.566 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:57.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:57.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:57.668 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:57.693 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:57.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:57.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:57.795 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:57.821 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:57.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:57.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:57.923 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:57.952 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:57.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:57.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:58.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:58.096 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:58.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:58.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:58.197 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:58.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:09:58.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:58.252 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:58.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:58.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:58.398 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 70 16 104474 104458 93.2402 60.375 0.00973953 0.0107195 2024-09-21T10:09:58.418 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:58.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:58.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:58.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:58.542 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:58.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:58.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:58.644 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:58.673 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:58.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:58.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:58.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:58.796 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:58.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:58.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:58.897 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:58.925 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:58.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:58.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:59.027 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:59.052 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:59.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:59.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:59.154 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:59.191 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:59.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:59.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:59.293 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:59.319 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:59.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:59.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:59.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:09:58 smithi098 ceph-mon[97580]: pgmap v494: 89 pgs: 89 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T10:09:59.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:09:58 smithi098 ceph-mon[100914]: pgmap v494: 89 pgs: 89 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T10:09:59.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:09:58 smithi196 ceph-mon[84784]: pgmap v494: 89 pgs: 89 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T10:09:59.398 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 71 16 105307 105291 92.6603 52.0625 0.00499804 0.0107816 2024-09-21T10:09:59.421 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:59.442 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:59.448 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:59.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:59.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:59.550 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:59.575 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:59.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:59.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:59.677 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:59.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:59.723 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:09:59.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:59.824 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:59.850 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:59.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:59.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:59.854 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:59.952 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:59.976 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:09:59.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:59.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:00.078 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:00.118 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:00.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:00.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:00.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:00.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:00.265 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:00.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:00.366 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:00.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:00.392 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:00.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:00.398 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 72 16 106199 106183 92.1477 55.75 0.0118967 0.0108474 2024-09-21T10:10:00.493 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:00.525 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:00.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:00.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:00.626 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:00.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:10:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:10:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:10:00.642 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:00.661 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:10:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:10:00] "GET /metrics HTTP/1.1" 200 36449 "" "Prometheus/2.51.0" 2024-09-21T10:10:00.669 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:00.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:00.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:00.771 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:00.809 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:00.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:00.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:00.911 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:00.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:00.949 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:00.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:01.007 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T10:10:01.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:01.086 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:01.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:01.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:01.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:01.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:01.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:01.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:01 smithi098 ceph-mon[97580]: pgmap v495: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 62 MiB/s wr, 997 op/s 2024-09-21T10:10:01.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:01 smithi098 ceph-mon[97580]: overall HEALTH_OK 2024-09-21T10:10:01.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:01 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2193111034' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:10:01.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:01 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:10:01.259 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:01 smithi098 ceph-mon[100914]: pgmap v495: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 62 MiB/s wr, 997 op/s 2024-09-21T10:10:01.259 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:01 smithi098 ceph-mon[100914]: overall HEALTH_OK 2024-09-21T10:10:01.260 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:01 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2193111034' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:10:01.260 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:01 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:10:01.330 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:01.368 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:01.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:01.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:01.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:01 smithi196 ceph-mon[84784]: pgmap v495: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 62 MiB/s wr, 997 op/s 2024-09-21T10:10:01.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:01 smithi196 ceph-mon[84784]: overall HEALTH_OK 2024-09-21T10:10:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:01 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2193111034' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:10:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:01 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:10:01.399 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 73 16 107190 107174 91.7338 61.9375 0.0161072 0.0108961 2024-09-21T10:10:01.470 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:01.496 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:01.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:01.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:01.598 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:01.627 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:01.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:01.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:01.729 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:01.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:01.781 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:01.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:01.883 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:01.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:01.911 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:01.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:02.012 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:02.054 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:02.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:02.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:02.156 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:02.182 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:02.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:02.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:02.205 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T10:10:02.283 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:02.309 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:02.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:02.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:02.317 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:02 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:10:02.317 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:02 smithi098 ceph-mon[97580]: osdmap e174: 8 total, 8 up, 7 in 2024-09-21T10:10:02.317 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:02 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:10:02.317 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:02 smithi098 ceph-mon[100914]: osdmap e174: 8 total, 8 up, 7 in 2024-09-21T10:10:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:02 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:10:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:02 smithi196 ceph-mon[84784]: osdmap e174: 8 total, 8 up, 7 in 2024-09-21T10:10:02.399 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 74 15 108187 108172 91.3371 62.375 0.0188013 0.0109431 2024-09-21T10:10:02.411 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:02.439 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:02.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:02.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:02.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:02.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:02.578 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:02.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:02.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:02.729 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:02.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:02.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:02.836 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:02.865 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:02.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:02.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:02.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:03.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:03.005 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:03.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:03.106 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:03.134 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:03.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:03.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:03.235 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:03.272 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:03.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:03.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:03 smithi098 ceph-mon[97580]: pgmap v497: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 62 MiB/s wr, 990 op/s 2024-09-21T10:10:03.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:03 smithi098 ceph-mon[100914]: pgmap v497: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 62 MiB/s wr, 990 op/s 2024-09-21T10:10:03.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:03.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:03 smithi196 ceph-mon[84784]: pgmap v497: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 62 MiB/s wr, 990 op/s 2024-09-21T10:10:03.400 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 75 16 109162 109146 90.9303 60.875 0.0248176 0.0109923 2024-09-21T10:10:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:03.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:03.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:03.512 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:03.537 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:03.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:03.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:03.639 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:03.663 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:03.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:03.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:03.765 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:03.796 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:03.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:03.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:03.898 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:03.928 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:03.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:03.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:04.030 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:04.054 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:04.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:04.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:04.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:04.195 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:04.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:04.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:04.297 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:04.319 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:04.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:04.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:04.400 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 76 16 110141 110125 90.5389 61.1875 0.0159606 0.01104 2024-09-21T10:10:04.421 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:04.449 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:04.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:04.551 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:04.577 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:04.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:04.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:04.679 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:04.714 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:04.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:04.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:04.815 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:04.847 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:04.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:04.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:04.949 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:04.973 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:04.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:04.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:05.075 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:05.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:05.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:05.215 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:05.239 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:05.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:05.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:05.341 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:05.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:05 smithi098 ceph-mon[97580]: pgmap v498: 89 pgs: 89 active+clean; 6.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 60 MiB/s wr, 966 op/s 2024-09-21T10:10:05.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:05 smithi098 ceph-mon[100914]: pgmap v498: 89 pgs: 89 active+clean; 6.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 60 MiB/s wr, 966 op/s 2024-09-21T10:10:05.386 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:05.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:05.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:05.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:05 smithi196 ceph-mon[84784]: pgmap v498: 89 pgs: 89 active+clean; 6.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 60 MiB/s wr, 966 op/s 2024-09-21T10:10:05.400 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 77 16 111133 111117 90.1678 62 0.0212988 0.0110856 2024-09-21T10:10:05.488 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:05.515 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:05.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:05.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:05.617 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:05.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:05.655 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:05.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:05.756 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:05.780 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:05.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:05.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:05.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:05.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:05.906 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:05.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:05.915 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:06.008 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:06.068 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:06.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:06.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:06.170 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:06.217 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:10:06.218 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:10:06.229 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:06.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:06.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:06.265 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:10:06.265 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:10:06.301 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:10:06.301 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:10:06.332 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:06.340 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:10:06.340 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:10:06.367 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:06.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:06.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:06.373 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:10:06.374 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:10:06.400 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 78 16 111836 111820 89.5751 43.9375 0.0114865 0.011158 2024-09-21T10:10:06.409 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:10:06.409 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:10:06.444 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:10:06.444 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:10:06.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:06.474 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:10:06.474 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:10:06.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:06.495 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:06.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:06.596 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:06.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:06.633 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:06.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:06.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:06.777 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:06.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:06.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:06.879 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:06.904 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:06.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:06.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:07.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:07.030 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:07.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:07.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:07.056 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:10:07.133 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:07.173 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:07.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:07.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:07.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:07.301 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:07.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:07.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:07.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:07 smithi098 ceph-mon[97580]: pgmap v499: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 60 MiB/s wr, 959 op/s 2024-09-21T10:10:07.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:07 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4021278685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:10:07.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:07 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1489227226' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:10:07.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:07 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:10:07.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:07 smithi098 ceph-mon[100914]: pgmap v499: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 60 MiB/s wr, 959 op/s 2024-09-21T10:10:07.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:07 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4021278685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:10:07.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:07 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1489227226' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:10:07.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:07 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:10:07.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:07 smithi196 ceph-mon[84784]: pgmap v499: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 60 MiB/s wr, 959 op/s 2024-09-21T10:10:07.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:07 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4021278685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:10:07.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:07 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1489227226' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:10:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:07 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:10:07.400 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 79 16 112640 112624 89.0774 50.25 0.0134065 0.0112203 2024-09-21T10:10:07.403 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:07.453 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:07.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:07.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:07.555 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:07.591 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:07.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:07.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:07.693 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:07.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:07.724 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:07.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:07.826 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:07.857 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:07.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:07.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:07.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:08.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:08.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:08.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:08.141 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:08.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:08.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:08.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:08.284 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:08.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:08.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:08.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:08 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:10:08.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:08 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:10:08.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:08 smithi098 ceph-mon[97580]: osdmap e175: 8 total, 8 up, 7 in 2024-09-21T10:10:08.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:10:08.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:08.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:08 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:10:08.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:08 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:10:08.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:08 smithi098 ceph-mon[100914]: osdmap e175: 8 total, 8 up, 7 in 2024-09-21T10:10:08.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:10:08.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:08.386 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:08 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:10:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:08 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:10:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:08 smithi196 ceph-mon[84784]: osdmap e175: 8 total, 8 up, 7 in 2024-09-21T10:10:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:10:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:08.400 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:10:08.399367+0000 min lat: 0.00114388 max lat: 0.233826 avg lat: 0.0112636 2024-09-21T10:10:08.400 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:10:08.400 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 80 16 113632 113616 88.7389 62 0.00754889 0.0112636 2024-09-21T10:10:08.417 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:08.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:08.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:08.519 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:08.547 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:08.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:08.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:08.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:08.686 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:08.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:08.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:08.788 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:08.816 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:08.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:08.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:08.918 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:08.951 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:08.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:08.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:09.053 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:09.083 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:09.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:09.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:09.178 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:10:09.178 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:10:09.178 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:09 smithi098 ceph-mon[97580]: pgmap v501: 89 pgs: 89 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 60 MiB/s wr, 954 op/s 2024-09-21T10:10:09.179 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:09 smithi098 ceph-mon[100914]: pgmap v501: 89 pgs: 89 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 60 MiB/s wr, 954 op/s 2024-09-21T10:10:09.185 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:09.209 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:09.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:09.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:09.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:09.356 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:09.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:09.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:09.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:09 smithi196 ceph-mon[84784]: pgmap v501: 89 pgs: 89 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 60 MiB/s wr, 954 op/s 2024-09-21T10:10:09.400 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 81 16 114617 114601 88.4034 61.5625 0.00833356 0.0113071 2024-09-21T10:10:09.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:10:09.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:09.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:09.529 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:09.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:09.630 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:09.653 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:09.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:09.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:09.755 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:09.782 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:09.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:09.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:09.884 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:09.930 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:09.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:09.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:10.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:10.060 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:10.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:10.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:10.162 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:10.201 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:10.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:10.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:10.303 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:10.331 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:10.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:10.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:10.401 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 82 16 115541 115525 88.0296 57.75 0.0139521 0.0113548 2024-09-21T10:10:10.432 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:10.474 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:10.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:10.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:10.575 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:10.600 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:10.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:10.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:10.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:10:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:10:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:10:10.701 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:10.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:10.758 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:10.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:10.811 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:10:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:10:10] "GET /metrics HTTP/1.1" 200 36447 "" "Prometheus/2.51.0" 2024-09-21T10:10:10.859 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:10.898 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:10.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:10.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:10.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:10:11.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:11.024 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:11.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:11.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:11.064 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:11 smithi098 ceph-mon[97580]: pgmap v502: 89 pgs: 89 active+clean; 6.9 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 60 MiB/s wr, 958 op/s 2024-09-21T10:10:11.064 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:11 smithi098 ceph-mon[100914]: pgmap v502: 89 pgs: 89 active+clean; 6.9 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 60 MiB/s wr, 958 op/s 2024-09-21T10:10:11.066 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:10:11.125 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:11.150 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:11.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:11.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:11.252 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:11.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:11.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:11 smithi196 ceph-mon[84784]: pgmap v502: 89 pgs: 89 active+clean; 6.9 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 60 MiB/s wr, 958 op/s 2024-09-21T10:10:11.401 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 83 16 116389 116373 87.607 53 0.031118 0.0114076 2024-09-21T10:10:11.419 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:11.459 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:11.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:11.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:11.561 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:11.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:11.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:11.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:11.714 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:11.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:11.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:11.852 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:11.876 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:11.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:11.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:11.977 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:12.004 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:12.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:12.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T10:10:12.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:12.148 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:12.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:12.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:12.231 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:12 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2739996676' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:10:12.231 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:12 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:10:12.231 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:12 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2739996676' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:10:12.231 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:12 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:10:12.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:12.276 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:12.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:12.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:12.378 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:12.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:12 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2739996676' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:10:12.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:12 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:10:12.401 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 84 16 117225 117209 87.1861 52.25 0.0146415 0.0114646 2024-09-21T10:10:12.413 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:12.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:12.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:12.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:12.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:12.552 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:12.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:12.653 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:12.682 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:12.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:12.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:12.784 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:12.826 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:12.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:12.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:12.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:12.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:12.973 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:12.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:12.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T10:10:13.074 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:13.152 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:13.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:13.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:13.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:13.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:13.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:13 smithi098 ceph-mon[97580]: pgmap v503: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 57 MiB/s wr, 918 op/s 2024-09-21T10:10:13.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:13 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:10:13.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:13 smithi098 ceph-mon[97580]: osdmap e176: 8 total, 8 up, 7 in 2024-09-21T10:10:13.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:13 smithi098 ceph-mon[100914]: pgmap v503: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 57 MiB/s wr, 918 op/s 2024-09-21T10:10:13.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:13 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:10:13.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:13 smithi098 ceph-mon[100914]: osdmap e176: 8 total, 8 up, 7 in 2024-09-21T10:10:13.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:13 smithi196 ceph-mon[84784]: pgmap v503: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 57 MiB/s wr, 918 op/s 2024-09-21T10:10:13.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:13 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:10:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:13 smithi196 ceph-mon[84784]: osdmap e176: 8 total, 8 up, 7 in 2024-09-21T10:10:13.401 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 85 16 118162 118146 86.8493 58.5625 0.00955335 0.0115089 2024-09-21T10:10:13.410 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:13.436 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:13.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:13.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:13.538 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:13.567 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:13.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:13.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:13.669 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:13.696 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:13.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:13.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:13.798 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:13.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:13.828 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:13.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:13.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:13.953 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:13.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:13.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:14.055 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:14.085 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:14.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:14.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:14.186 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:14.207 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:14.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:14.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:14.309 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:14.343 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:14.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:14.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:14.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:14 smithi196 ceph-mon[84784]: pgmap v505: 89 pgs: 89 active+clean; 7.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 60 MiB/s wr, 952 op/s 2024-09-21T10:10:14.401 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 86 16 119017 119001 86.4608 53.4375 0.0290485 0.0115606 2024-09-21T10:10:14.446 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:14.473 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:14.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:14.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:14.574 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:14.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:14 smithi098 ceph-mon[97580]: pgmap v505: 89 pgs: 89 active+clean; 7.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 60 MiB/s wr, 952 op/s 2024-09-21T10:10:14.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:14 smithi098 ceph-mon[100914]: pgmap v505: 89 pgs: 89 active+clean; 7.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 60 MiB/s wr, 952 op/s 2024-09-21T10:10:14.618 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:14.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:14.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:14.719 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:14.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:14.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:14.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:14.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:14.890 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:14.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:14.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:14.992 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:15.016 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:15.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:15.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:15.118 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:15.160 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:15.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:15.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:15.262 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:15.294 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:15.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:15.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:15.395 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:15.401 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 87 16 120006 119990 86.1774 61.8125 0.0156733 0.0115988 2024-09-21T10:10:15.421 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:15.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:15.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:15.547 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:15.575 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:15.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:15.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:15.676 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:15.703 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:15.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:15.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:15.805 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:15.843 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:15.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:15.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:15.945 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:15.962 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:15.973 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:15.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:15.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:16.074 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:16.099 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:16.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:16.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:16.200 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:16.241 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:16.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:16.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:16.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:16.379 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:16.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:16.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:16.402 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 88 16 120937 120921 85.8594 58.1875 0.0365982 0.0116418 2024-09-21T10:10:16.481 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:16.515 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:16.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:16.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:16.576 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:16 smithi098 ceph-mon[97580]: pgmap v506: 89 pgs: 89 active+clean; 7.2 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 50 MiB/s wr, 796 op/s 2024-09-21T10:10:16.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:16 smithi098 ceph-mon[100914]: pgmap v506: 89 pgs: 89 active+clean; 7.2 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 50 MiB/s wr, 796 op/s 2024-09-21T10:10:16.617 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:16.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:16 smithi196 ceph-mon[84784]: pgmap v506: 89 pgs: 89 active+clean; 7.2 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 50 MiB/s wr, 796 op/s 2024-09-21T10:10:16.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:16.659 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:16.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:16.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:16.799 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:16.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:16.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:16.900 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:16.928 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:16.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:16.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:17.030 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:17.053 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:17.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:17.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:17.155 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:17.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:17.210 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:17.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:17.311 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:17.345 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:17.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:17.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:17.406 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 89 16 121866 121850 85.5448 58.0625 0.0175807 0.0116843 2024-09-21T10:10:17.447 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:17.470 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:17.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:17.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:17.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:17.604 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:17.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:17.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:17.706 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:17.754 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:17.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:17.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:17.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:17.893 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:17.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:17.895 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:17.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:17.961 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:10:17.961 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:10:17.996 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:18.007 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:10:18.008 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:10:18.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:18.034 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:18.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:18.047 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:10:18.047 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:10:18.103 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:10:18.103 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:10:18.135 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:18.138 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:10:18.139 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:10:18.158 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:18.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:18.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:18.180 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:10:18.181 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:10:18.217 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:10:18.218 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:10:18.255 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:10:18.255 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:10:18.260 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:18.283 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:18.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:18.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:18.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:18.404 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Total time run: 90.0083 2024-09-21T10:10:18.404 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Total writes made: 122656 2024-09-21T10:10:18.404 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Write size: 65536 2024-09-21T10:10:18.404 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Object size: 65536 2024-09-21T10:10:18.404 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Bandwidth (MB/sec): 85.1699 2024-09-21T10:10:18.405 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Stddev Bandwidth: 34.3764 2024-09-21T10:10:18.405 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Max bandwidth (MB/sec): 212.625 2024-09-21T10:10:18.405 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Min bandwidth (MB/sec): 43.9375 2024-09-21T10:10:18.405 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Average IOPS: 1362 2024-09-21T10:10:18.405 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Stddev IOPS: 550.023 2024-09-21T10:10:18.405 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Max IOPS: 3402 2024-09-21T10:10:18.405 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Min IOPS: 703 2024-09-21T10:10:18.405 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Average Latency(s): 0.0117369 2024-09-21T10:10:18.405 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Stddev Latency(s): 0.00913188 2024-09-21T10:10:18.405 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Max latency(s): 0.233826 2024-09-21T10:10:18.405 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Min latency(s): 0.00114388 2024-09-21T10:10:18.406 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:18.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:18.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:18.409 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:10:18.508 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:18.537 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:18.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:18.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:18.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:18 smithi098 ceph-mon[97580]: pgmap v507: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 57 MiB/s wr, 904 op/s 2024-09-21T10:10:18.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:18 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/482400794' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:10:18.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:18 smithi098 ceph-mon[100914]: pgmap v507: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 57 MiB/s wr, 904 op/s 2024-09-21T10:10:18.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:18 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/482400794' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:10:18.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:18 smithi196 ceph-mon[84784]: pgmap v507: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 57 MiB/s wr, 904 op/s 2024-09-21T10:10:18.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:18 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/482400794' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:10:18.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:18.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:18.675 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:18.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:18.776 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:18.802 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:18.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:18.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:18.904 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:18.930 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:18.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:18.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:19.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:19.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:19.063 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:19.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:19.164 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:19.202 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:19.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:19.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:19.233 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:10:19.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:19.338 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:19.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:19.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:19.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:19.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:19.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:19.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:19.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:19 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1966543539' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:10:19.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:19 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1966543539' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:10:19.616 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:19.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:19.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:19.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:19 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1966543539' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:10:19.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:19.751 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:19.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:19.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:19.853 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:19.877 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:19.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:19.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:19.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:20.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:20.016 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:20.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:20.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:20.152 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:20.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:20.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:20.254 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:20.279 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:20.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:20.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:20.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:20 smithi098 ceph-mon[97580]: pgmap v508: 89 pgs: 89 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 57 MiB/s wr, 906 op/s 2024-09-21T10:10:20.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:20 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:10:20.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:20 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1966543539' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:10:20.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:20 smithi098 ceph-mon[97580]: osdmap e177: 8 total, 8 up, 7 in 2024-09-21T10:10:20.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:20 smithi098 ceph-mon[100914]: pgmap v508: 89 pgs: 89 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 57 MiB/s wr, 906 op/s 2024-09-21T10:10:20.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:20 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:10:20.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:20 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1966543539' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:10:20.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:20 smithi098 ceph-mon[100914]: osdmap e177: 8 total, 8 up, 7 in 2024-09-21T10:10:20.381 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:20.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:20.417 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:20.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:20.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:20.562 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:20.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:20.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:20.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:10:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:10:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:10:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:20 smithi196 ceph-mon[84784]: pgmap v508: 89 pgs: 89 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 57 MiB/s wr, 906 op/s 2024-09-21T10:10:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:20 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:10:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:20 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1966543539' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:10:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:20 smithi196 ceph-mon[84784]: osdmap e177: 8 total, 8 up, 7 in 2024-09-21T10:10:20.665 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:20.689 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:20.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:20.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:20.791 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:20.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:20.818 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:20.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:20.919 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:20.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:20.964 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:20.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:21.066 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:21.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:21.088 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:21.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:21.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:10:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:10:20] "GET /metrics HTTP/1.1" 200 36447 "" "Prometheus/2.51.0" 2024-09-21T10:10:21.188 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:21.210 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:21.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:21.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:21.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:21.345 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:21.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:21.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:21.447 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:21.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:21.479 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:21.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:21.580 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:21.607 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:21.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:21.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:21.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:21.747 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:21.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:21.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:21.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:21.884 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:21.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:21.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:21.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:22.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:22.024 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:22.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:22.125 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:22.151 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:22.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:22.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:22.252 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:22.281 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:22.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:22.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:22.383 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:22.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:22.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:22.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:22.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:22.549 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:22.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:22.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:22.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:22 smithi098 ceph-mon[97580]: pgmap v510: 89 pgs: 89 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 112 B/s rd, 49 MiB/s wr, 850 op/s 2024-09-21T10:10:22.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:22.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:22 smithi098 ceph-mon[100914]: pgmap v510: 89 pgs: 89 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 112 B/s rd, 49 MiB/s wr, 850 op/s 2024-09-21T10:10:22.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:22.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:22 smithi196 ceph-mon[84784]: pgmap v510: 89 pgs: 89 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 112 B/s rd, 49 MiB/s wr, 850 op/s 2024-09-21T10:10:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:22.651 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:22.676 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:22.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:22.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:22.777 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:22.823 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:22.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:22.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:22.926 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:22.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:22.957 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:22.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:23.058 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:23.087 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:23.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:23.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:23.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:23.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:23.238 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:23.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:23.339 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:23.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:23.381 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:23.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:23.482 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:23.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:23.511 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:23.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:23.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:23 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:10:23.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:23 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:10:23.612 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:23 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:10:23.639 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:23.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:23.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:23.741 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:23.766 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:23.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:23.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:23.868 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:23.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:23.917 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:23.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:24.017 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:24.048 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:24.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:24.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:24.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:24.187 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:24.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:24.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:24.289 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:24.304 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:24 smithi098 ceph-mon[97580]: pgmap v511: 89 pgs: 89 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 43 MiB/s wr, 915 op/s 2024-09-21T10:10:24.304 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:24 smithi098 ceph-mon[100914]: pgmap v511: 89 pgs: 89 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 43 MiB/s wr, 915 op/s 2024-09-21T10:10:24.314 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:24.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:24.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:24.416 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:24.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:24.445 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:24.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:24.546 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:24.571 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:24.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:24.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:24 smithi196 ceph-mon[84784]: pgmap v511: 89 pgs: 89 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 43 MiB/s wr, 915 op/s 2024-09-21T10:10:24.673 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:24.698 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:24.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:24.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:24.791 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:10:24.791 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:10:24.791 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-21T10:10:24.791 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-21T10:10:24.791 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:10:24.801 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:24.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:24.847 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:24.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:24.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:10:24.948 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:25.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:25.000 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:25.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:25.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:25.122 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:25.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:25.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:25.224 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:25.261 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:25.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:25.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:25.282 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T10:10:25.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:10:25.363 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:25.394 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:25.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:25.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:25.497 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:25.522 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:25.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:25.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:25.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:25 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3177267859' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:10:25.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:25 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:10:25.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:25 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3177267859' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:10:25.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:25 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:10:25.626 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:25 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3177267859' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:10:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:25 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:10:25.653 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:25.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:25.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:25.755 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:25.783 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:25.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:25.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:25.885 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:25.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:25.919 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:25.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:26.020 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:26.052 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:26.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:26.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:26.154 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:26.189 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:26.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:26.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:26.291 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:26.317 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:26.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:26.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:26.419 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:26.448 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:26.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:26.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:26.549 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:26.587 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:26.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:26.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:26.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:26 smithi098 ceph-mon[97580]: pgmap v512: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 34 MiB/s wr, 918 op/s 2024-09-21T10:10:26.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:26 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:10:26.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:26 smithi098 ceph-mon[97580]: osdmap e178: 8 total, 8 up, 7 in 2024-09-21T10:10:26.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:26 smithi098 ceph-mon[100914]: pgmap v512: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 34 MiB/s wr, 918 op/s 2024-09-21T10:10:26.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:26 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:10:26.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:26 smithi098 ceph-mon[100914]: osdmap e178: 8 total, 8 up, 7 in 2024-09-21T10:10:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:26 smithi196 ceph-mon[84784]: pgmap v512: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 34 MiB/s wr, 918 op/s 2024-09-21T10:10:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:26 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:10:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:26 smithi196 ceph-mon[84784]: osdmap e178: 8 total, 8 up, 7 in 2024-09-21T10:10:26.688 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:26.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:26.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:26.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:26.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:26.853 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:26.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:26.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:26.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T10:10:26.956 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:26.973 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:27.035 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:27.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:27.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:27.137 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:27.174 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:27.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:27.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:27.276 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:27.303 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:27.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:27.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:27.405 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:27.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:27.432 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:27.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:27.532 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:27.557 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:27.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:27.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:27.569 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:10:27.659 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:27.684 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:27.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:27.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:27.786 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:27.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:27.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:27.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:27.962 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:27.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:27.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:28.063 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:28.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:28.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:28.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:28.202 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:28.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:28.241 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:28.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:28.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:28.371 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:28.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:28.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:28.472 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:28.508 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:28.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:28.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:28.610 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:28.636 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:28.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:28.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:28.738 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:28.781 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:28.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:28.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:28.883 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:28.910 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:28.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:28.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:29.012 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:29.058 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:29.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:29.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:29.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:28 smithi098 ceph-mon[97580]: pgmap v514: 89 pgs: 89 active+clean; 7.1 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 127 B/s rd, 9.8 MiB/s wr, 936 op/s 2024-09-21T10:10:29.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:28 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:10:29.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:28 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:10:29.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:28 smithi098 ceph-mon[100914]: pgmap v514: 89 pgs: 89 active+clean; 7.1 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 127 B/s rd, 9.8 MiB/s wr, 936 op/s 2024-09-21T10:10:29.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:28 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:10:29.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:28 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:10:29.159 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:29.184 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:29.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:29.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:29.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:29.321 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:29.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:29.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:29.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:28 smithi196 ceph-mon[84784]: pgmap v514: 89 pgs: 89 active+clean; 7.1 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 127 B/s rd, 9.8 MiB/s wr, 936 op/s 2024-09-21T10:10:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:28 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:10:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:28 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:10:29.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:29.456 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:29.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:29.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:29.557 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:29.585 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:29.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:29.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:29.687 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:29.751 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:29.764 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:29.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:29.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:29.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:29.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:29.920 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:29.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:30.021 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:30.082 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:30.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:30.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:30.184 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:30.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:30.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:30.350 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:30.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:30.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:30.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:30.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:30.537 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:30.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:30.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:30.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:10:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:10:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:10:30.664 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:30.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:30.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:30.765 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:30.786 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:10:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:10:30] "GET /metrics HTTP/1.1" 200 36455 "" "Prometheus/2.51.0" 2024-09-21T10:10:30.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:30.795 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:30.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:30.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:30.933 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:30.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:30.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:30.953 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:10:31.036 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:31.065 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:31.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:31.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:31.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:30 smithi098 ceph-mon[100914]: pgmap v515: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 103 B/s rd, 7.9 MiB/s wr, 922 op/s 2024-09-21T10:10:31.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:30 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2613239749' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:10:31.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:30 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:10:31.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:30 smithi098 ceph-mon[97580]: pgmap v515: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 103 B/s rd, 7.9 MiB/s wr, 922 op/s 2024-09-21T10:10:31.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:30 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2613239749' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:10:31.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:30 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:10:31.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:31.224 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:31.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:31.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:31.325 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:31.351 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:31.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:31.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:31.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:30 smithi196 ceph-mon[84784]: pgmap v515: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 103 B/s rd, 7.9 MiB/s wr, 922 op/s 2024-09-21T10:10:31.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:30 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2613239749' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:10:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:30 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:10:31.453 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:31.476 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:31.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:31.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:31.578 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:31.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:31.603 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:31.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:31.704 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:31.731 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:31.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:31.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:31.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:31.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:31.870 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:31.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:31.934 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:31.934 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:31.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:31.995 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:31.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:32.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:32.152 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:32.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:32.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:32.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:32.299 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:32.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:32.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:32.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:31 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:10:32.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:31 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:10:32.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:31 smithi098 ceph-mon[97580]: osdmap e179: 8 total, 8 up, 7 in 2024-09-21T10:10:32.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:31 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:10:32.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:31 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:10:32.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:31 smithi098 ceph-mon[100914]: osdmap e179: 8 total, 8 up, 7 in 2024-09-21T10:10:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:31 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:10:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:31 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:10:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:31 smithi196 ceph-mon[84784]: osdmap e179: 8 total, 8 up, 7 in 2024-09-21T10:10:32.401 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:32.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:32.439 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:32.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:32.540 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:32.568 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:32.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:32.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:32.670 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:32.697 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:32.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:32.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:32.799 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:32.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:32.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:32.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:32.991 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:32.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:32.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:33.066 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:32 smithi098 ceph-mon[97580]: pgmap v517: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 328 KiB/s wr, 893 op/s 2024-09-21T10:10:33.066 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:32 smithi098 ceph-mon[100914]: pgmap v517: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 328 KiB/s wr, 893 op/s 2024-09-21T10:10:33.093 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:33.132 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:33.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:33.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:33.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:33.276 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:33.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:33.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:33.378 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:33.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:32 smithi196 ceph-mon[84784]: pgmap v517: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 328 KiB/s wr, 893 op/s 2024-09-21T10:10:33.422 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:33.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:33.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:33.524 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:33.551 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:33.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:33.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:33.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:33.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:33.694 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:33.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:33.795 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:33.837 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:33.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:33.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:33.927 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:10:33.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:33.964 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:33.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:33.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:34.066 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:34.090 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:34.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:34.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:34.192 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:34.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:34.259 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:34.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:34.360 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:34.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:34.388 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:34.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:34.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:34.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:34.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:34.633 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:34.678 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:34.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:34.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:34.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:34.804 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:34.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:34.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:34.905 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:34.932 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:34.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:34.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:35.016 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:34 smithi098 ceph-mon[97580]: pgmap v518: 89 pgs: 89 active+clean; 6.8 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 968 op/s 2024-09-21T10:10:35.016 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:34 smithi098 ceph-mon[100914]: pgmap v518: 89 pgs: 89 active+clean; 6.8 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 968 op/s 2024-09-21T10:10:35.033 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:35.056 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:35.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:35.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:35.157 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:35.181 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:35.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:35.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:35.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:35.332 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:35.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:35.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:35.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:34 smithi196 ceph-mon[84784]: pgmap v518: 89 pgs: 89 active+clean; 6.8 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 968 op/s 2024-09-21T10:10:35.434 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:35.456 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:35.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:35.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:35.464 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:10:35.464 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:10:35.464 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:10:35.464 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:10:35.465 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:10:35.465 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:10:35.465 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:10:35.465 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:10:35.499 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:10:35.499 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:10:35.546 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:10:35.546 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:10:35.558 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:35.594 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:10:35.595 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:10:35.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:35.605 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:35.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:35.646 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:10:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:10:35.706 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:35.732 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:35.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:35.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:35.833 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:35.856 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:35.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:35.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:35.958 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:35.984 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:35.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:35.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:36.021 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T10:10:36.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:36.128 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:36.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:36.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:36.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:36.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:36.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:36.342 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:36 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2904068880' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:10:36.342 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:36 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2450386018' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:10:36.342 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:36 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:10:36.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:36 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2904068880' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:10:36.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:36 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2450386018' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:10:36.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:36 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:10:36.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:36 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2904068880' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:10:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:36 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2450386018' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:10:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:36 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:10:36.410 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:36.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:36.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:36.511 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:36.538 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:36.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:36.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:36.640 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:36.665 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:36.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:36.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:36.766 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:36.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:36.803 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:36.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:36.904 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:36.952 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:36.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:37.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:37.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:37.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:37.195 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:37.251 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:37.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:37 smithi098 ceph-mon[97580]: pgmap v519: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 931 op/s 2024-09-21T10:10:37.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:37 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:10:37.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:37 smithi098 ceph-mon[97580]: osdmap e180: 8 total, 8 up, 7 in 2024-09-21T10:10:37.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:37.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:37.253 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:37 smithi098 ceph-mon[100914]: pgmap v519: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 931 op/s 2024-09-21T10:10:37.253 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:37 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:10:37.253 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:37 smithi098 ceph-mon[100914]: osdmap e180: 8 total, 8 up, 7 in 2024-09-21T10:10:37.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:37 smithi196 ceph-mon[84784]: pgmap v519: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 931 op/s 2024-09-21T10:10:37.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:37 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:10:37.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:37 smithi196 ceph-mon[84784]: osdmap e180: 8 total, 8 up, 7 in 2024-09-21T10:10:37.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:37.387 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:37.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:37.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:37.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:37.522 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:37.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:37.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:37.623 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:37.656 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:37.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:37.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:37.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T10:10:37.758 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:37.787 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:37.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:37.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:37.888 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:37.922 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:37.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:37.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:38.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:38.050 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:38.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:38.152 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:38.194 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:38.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:38.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:38.234 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:38.295 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:38.332 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:38.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:38.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:38.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:38.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:38.471 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:38.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:38.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:38.573 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:38.601 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:38.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:38.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:38.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:10:38.703 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:38.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:38.733 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:38.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:38.834 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:38.860 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:38.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:38.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:38.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:38.986 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:38.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:38.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:39.023 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:39 smithi098 ceph-mon[97580]: pgmap v521: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 935 op/s 2024-09-21T10:10:39.023 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:39 smithi098 ceph-mon[100914]: pgmap v521: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 935 op/s 2024-09-21T10:10:39.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:39.133 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:39.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:39.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:39.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:39.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:39.286 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:39.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:39.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:39.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:39 smithi196 ceph-mon[84784]: pgmap v521: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 935 op/s 2024-09-21T10:10:39.421 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:39.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:39.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:39.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:39.549 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:39.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:39.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:39.651 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:39.675 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:39.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:39.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:39.777 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:39.819 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:39.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:39.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:39.847 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:10:39.847 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:10:39.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:39.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:39.965 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:39.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:40.066 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:40.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:40.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:40.208 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:40.234 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:40.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:40.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:40.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:40.370 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:40.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:40.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:40.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:40.506 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:40.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:40.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:40.607 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:10:40.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:40.639 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 10:10:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=sqlstore.transactions t=2024-09-21T10:10:40.459040093Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T10:10:40.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:10:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:10:40.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:10:40.708 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:40.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:40.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:40.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:40.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:10:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:10:40] "GET /metrics HTTP/1.1" 200 36447 "" "Prometheus/2.51.0" 2024-09-21T10:10:40.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:40.878 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:40.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:40.979 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:41.008 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:41.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:41.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:41.112 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:41.139 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:41.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:41.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:41.240 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:41.268 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:41.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:41.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:41.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:41 smithi098 ceph-mon[97580]: pgmap v522: 89 pgs: 89 active+clean; 6.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 935 op/s 2024-09-21T10:10:41.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:41 smithi098 ceph-mon[100914]: pgmap v522: 89 pgs: 89 active+clean; 6.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 935 op/s 2024-09-21T10:10:41.370 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:41 smithi196 ceph-mon[84784]: pgmap v522: 89 pgs: 89 active+clean; 6.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 935 op/s 2024-09-21T10:10:41.398 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:41.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:41.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:41.500 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:41.527 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:41.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:41.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:41.629 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:41.655 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:41.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:41.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:41.710 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:41.721 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:41.724 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:41.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:41.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:41.816 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:41.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:41.917 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:41.941 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:41.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:41.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:42.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:42.083 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:42.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:42.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:42.184 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:42.219 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:42.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:42.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:42.321 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:42.360 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:42.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:42.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:42.462 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:42.497 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:42.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:42.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:42.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:42.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:42.640 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:42.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:42.741 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:42.767 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:42.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:42.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:42.868 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:42.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:42.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:43.014 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:43.053 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:43.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:43.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:43.154 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:43.200 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:43.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:43.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:43.301 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:43.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:43 smithi098 ceph-mon[97580]: pgmap v523: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 905 op/s 2024-09-21T10:10:43.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:43 smithi098 ceph-mon[100914]: pgmap v523: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 905 op/s 2024-09-21T10:10:43.340 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:43.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:43.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:43.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:43 smithi196 ceph-mon[84784]: pgmap v523: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 905 op/s 2024-09-21T10:10:43.442 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:43.479 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:43.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:43.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:43.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:43.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:43.623 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:43.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:43.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:43.762 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:43.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:43.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:43.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:43.922 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:43.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:43.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:44.026 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:44.054 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:44.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:44.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:44.156 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:44.180 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:44.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:44.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:44.282 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:44.307 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:44.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:44.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:44.409 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:44.436 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:44.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:44.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:44.538 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:44.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:44.587 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:44.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:44.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:44.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:44.735 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:44.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:44.836 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:44.861 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:44.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:44.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:44.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:44.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:44.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:45.100 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:45.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:45.129 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:45.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:45.230 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:45.254 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:45.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:45.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:45.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:45 smithi098 ceph-mon[97580]: pgmap v524: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 896 op/s 2024-09-21T10:10:45.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:45 smithi098 ceph-mon[100914]: pgmap v524: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 896 op/s 2024-09-21T10:10:45.356 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:45.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:45 smithi196 ceph-mon[84784]: pgmap v524: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 896 op/s 2024-09-21T10:10:45.396 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:45.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:45.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:45.497 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:45.526 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:45.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:45.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:45.628 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:45.662 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:45.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:45.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:45.764 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:45.790 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:45.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:45.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:45.892 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:45.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:45.963 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:45.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:46.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:46.121 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:46.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:46.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:46.217 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:10:46.218 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:10:46.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:46.256 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:10:46.257 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:10:46.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:46.273 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:46.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:46.294 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:10:46.295 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:10:46.351 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:10:46.351 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:10:46.374 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:46.380 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:10:46.380 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:10:46.398 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:46.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:46.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:46.417 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:10:46.417 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:10:46.450 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:10:46.450 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:10:46.481 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:10:46.482 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:10:46.500 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:46.537 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:46.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:46.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:46.638 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:46.677 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:46.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:46.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:46.778 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:46.809 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:46.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:46.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:46.910 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:46.936 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:46.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:46.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:47.038 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:47.063 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:47.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:47.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:47.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:46 smithi098 ceph-mon[97580]: pgmap v525: 89 pgs: 89 active+clean; 6.2 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 893 op/s 2024-09-21T10:10:47.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:10:47.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:10:47.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:46 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4173316702' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:10:47.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:10:47.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:10:47.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:10:47.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:46 smithi098 ceph-mon[100914]: pgmap v525: 89 pgs: 89 active+clean; 6.2 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 893 op/s 2024-09-21T10:10:47.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:10:47.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:10:47.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:46 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4173316702' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:10:47.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:10:47.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:10:47.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:10:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:46 smithi196 ceph-mon[84784]: pgmap v525: 89 pgs: 89 active+clean; 6.2 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 893 op/s 2024-09-21T10:10:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:10:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:10:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:46 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4173316702' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:10:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:10:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:10:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:10:47.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:47.188 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:47.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:47.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:47.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:47.322 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:47.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:47.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:47.424 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:47.451 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:47.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:47.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:47.552 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:47.576 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:47.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:47.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:47.678 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:47.704 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:47.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:47.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:47.806 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:47.833 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:47.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:47.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:47.935 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:47.962 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:47.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:47.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:48.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:48.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:48.096 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:48.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:48.197 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:48.229 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:48.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:48.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:48.331 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:48.356 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:48.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:48.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:48.457 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:48.480 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:48.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:48.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:48.582 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:48.611 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:48.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:48.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:48.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:48.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:48.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:48.835 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:48.861 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:48.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:48.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:48 smithi196 ceph-mon[84784]: pgmap v526: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 938 op/s 2024-09-21T10:10:48.963 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:48.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:48.991 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:48.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:49.091 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:49.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:48 smithi098 ceph-mon[97580]: pgmap v526: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 938 op/s 2024-09-21T10:10:49.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:48 smithi098 ceph-mon[100914]: pgmap v526: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 938 op/s 2024-09-21T10:10:49.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:49.124 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:49.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:49.225 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:49.269 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:49.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:49.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:49.371 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:49.392 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:49.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:49.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:49.494 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:49.521 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:49.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:49.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:49.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:49.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:49.662 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:49.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:49.763 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:49.793 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:49.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:49.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:49.894 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:49.919 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:49.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:49.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:50.021 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:50.050 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:50.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:50.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:50.152 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:50.173 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:50.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:50.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:50.275 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:50.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:50.310 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:50.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:50.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:50.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:50.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:50.534 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:50.563 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:50.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:50.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:50.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:10:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:10:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:10:50.665 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:50.688 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:50.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:50.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:50.790 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:50.815 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:50.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:50.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:50.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:50 smithi098 ceph-mon[97580]: pgmap v527: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 902 op/s 2024-09-21T10:10:50.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:10:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:10:50] "GET /metrics HTTP/1.1" 200 36447 "" "Prometheus/2.51.0" 2024-09-21T10:10:50.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:50 smithi098 ceph-mon[100914]: pgmap v527: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 902 op/s 2024-09-21T10:10:50.917 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:50.940 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:50.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:50.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:51.042 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:51.070 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:51.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:51.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:50 smithi196 ceph-mon[84784]: pgmap v527: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 902 op/s 2024-09-21T10:10:51.172 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:51.207 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:51.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:51.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:51.308 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:51.348 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:51.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:51.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:51.450 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:51.475 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:51.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:51.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:51.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:51.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:51.611 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:51.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:51.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:51.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:51.746 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:51.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:51.846 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:51.875 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:51.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:51.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:51.976 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:52.003 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:52.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:52.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:52.105 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:52.129 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:52.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:52.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:52.231 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:52.256 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:52.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:52.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:52.358 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:52.402 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:52.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:52.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:52.504 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:52.530 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:52.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:52.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:52.631 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:52.658 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:52.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:52.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:52.759 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:52.781 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:52.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:52.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:52.883 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:52.907 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:52.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:52.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:53.009 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:53.046 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:53.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:53.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:53.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:52 smithi098 ceph-mon[100914]: pgmap v528: 89 pgs: 89 active+clean; 5.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 875 op/s 2024-09-21T10:10:53.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:52 smithi098 ceph-mon[97580]: pgmap v528: 89 pgs: 89 active+clean; 5.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 875 op/s 2024-09-21T10:10:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:53.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:52 smithi196 ceph-mon[84784]: pgmap v528: 89 pgs: 89 active+clean; 5.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 875 op/s 2024-09-21T10:10:53.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:53.148 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:53.174 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:53.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:53.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:53.276 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:53.310 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:53.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:53.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:53.412 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:53.444 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:53.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:53.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:53.494 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:10:53.546 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:53.572 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:53.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:53.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:53.674 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:53.712 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:53.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:53.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:53.753 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T10:10:53.811 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:53 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/301277262' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:10:53.811 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:53 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:10:53.811 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:53 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/301277262' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:10:53.811 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:53 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:10:53.814 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:53.837 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:53.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:53.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:53.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:53.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:53.968 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:53.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:54.069 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:54.098 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:54.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:54.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:54.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:53 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/301277262' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:10:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:53 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:10:54.200 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:54.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:54.238 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:54.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:54.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:54.367 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:54.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:54.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:54.469 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:54.507 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:54.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:54.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:54.609 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:54.617 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T10:10:54.634 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:54.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:54.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:54.737 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:54.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:54.780 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:54.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:54.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:54 smithi098 ceph-mon[97580]: pgmap v529: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 919 op/s 2024-09-21T10:10:54.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:54 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:10:54.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:54 smithi098 ceph-mon[97580]: osdmap e181: 8 total, 8 up, 7 in 2024-09-21T10:10:54.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:54 smithi098 ceph-mon[100914]: pgmap v529: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 919 op/s 2024-09-21T10:10:54.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:54 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:10:54.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:54 smithi098 ceph-mon[100914]: osdmap e181: 8 total, 8 up, 7 in 2024-09-21T10:10:54.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:54.933 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:54.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:54.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:55.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:55.087 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:55.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:55.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:55.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:54 smithi196 ceph-mon[84784]: pgmap v529: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 919 op/s 2024-09-21T10:10:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:54 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:10:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:54 smithi196 ceph-mon[84784]: osdmap e181: 8 total, 8 up, 7 in 2024-09-21T10:10:55.189 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:55.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:55.237 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:55.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:55.338 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:55.364 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:55.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:55.466 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:55.492 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:55.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:55.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:55.595 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:55.629 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:55.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:55.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:55.654 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:55.662 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:10:55.662 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:10:55.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.663 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:156399 6.f 6:f1c5d84a:::benchmark_data_smithi098_169033_object33740:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-21T10:10:55.663 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:10:55.642622+0000", 2024-09-21T10:10:55.663 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.015444773, 2024-09-21T10:10:55.663 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015477018, 2024-09-21T10:10:55.663 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:10:55.663 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:10:55.663 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:10:55.663 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:10:55.663 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:10:55.663 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:10:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 156399 2024-09-21T10:10:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:10:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:10:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.642622+0000", 2024-09-21T10:10:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:10:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:10:55.664 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.642622+0000", 2024-09-21T10:10:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:10:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:10:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.642624+0000", 2024-09-21T10:10:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.672e-06 2024-09-21T10:10:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:10:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.642628+0000", 2024-09-21T10:10:55.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7519999999999998e-06 2024-09-21T10:10:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:10:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.642642+0000", 2024-09-21T10:10:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4783999999999999e-05 2024-09-21T10:10:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:10:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.642651+0000", 2024-09-21T10:10:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.259e-06 2024-09-21T10:10:55.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:10:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.644705+0000", 2024-09-21T10:10:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0020547529999999999 2024-09-21T10:10:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:10:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.644816+0000", 2024-09-21T10:10:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000110676 2024-09-21T10:10:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:10:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.644915+0000", 2024-09-21T10:10:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8958999999999999e-05 2024-09-21T10:10:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:10:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:10:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:10:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:156400 6.f 6.b71d11ff (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-21T10:10:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:10:55.644598+0000", 2024-09-21T10:10:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.013468582999999999, 2024-09-21T10:10:55.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013583107000000001, 2024-09-21T10:10:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:10:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:10:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-21T10:10:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:10:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:10:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:10:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 156400 2024-09-21T10:10:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:10:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.669 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:10:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.644598+0000", 2024-09-21T10:10:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:10:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:10:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.644598+0000", 2024-09-21T10:10:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:10:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:10:55.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.644600+0000", 2024-09-21T10:10:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4360000000000001e-06 2024-09-21T10:10:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:10:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.644605+0000", 2024-09-21T10:10:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9799999999999998e-06 2024-09-21T10:10:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:10:55.671 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.644617+0000", 2024-09-21T10:10:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2014000000000001e-05 2024-09-21T10:10:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:10:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.644624+0000", 2024-09-21T10:10:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0879999999999997e-06 2024-09-21T10:10:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:10:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:10:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:10:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:156404 6.f 6.7f86a55f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-21T10:10:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:10:55.647872+0000", 2024-09-21T10:10:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.010195196, 2024-09-21T10:10:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010360090000000001, 2024-09-21T10:10:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:10:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:10:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-21T10:10:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:10:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:10:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:10:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 156404 2024-09-21T10:10:55.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:10:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:10:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.647872+0000", 2024-09-21T10:10:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:10:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:10:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.647872+0000", 2024-09-21T10:10:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:10:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:10:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.647873+0000", 2024-09-21T10:10:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.621e-06 2024-09-21T10:10:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:10:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.647878+0000", 2024-09-21T10:10:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6800000000000001e-06 2024-09-21T10:10:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.675 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:10:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.647892+0000", 2024-09-21T10:10:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3567000000000001e-05 2024-09-21T10:10:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:10:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.647900+0000", 2024-09-21T10:10:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1009999999999992e-06 2024-09-21T10:10:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:10:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:10:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:10:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.45263.0:156402 6.9 e181/165 6:96aeaa4c:::benchmark_data_smithi098_169033_object33743:head v 181'9915, mlcod=181'9911)", 2024-09-21T10:10:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:10:55.650035+0000", 2024-09-21T10:10:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0080315730000000002, 2024-09-21T10:10:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0082500290000000007, 2024-09-21T10:10:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:10:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:10:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:10:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:10:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:10:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.650035+0000", 2024-09-21T10:10:55.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:10:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:10:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.650035+0000", 2024-09-21T10:10:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:10:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:10:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.650036+0000", 2024-09-21T10:10:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6199999999999997e-07 2024-09-21T10:10:55.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:10:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.650041+0000", 2024-09-21T10:10:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5399999999999997e-06 2024-09-21T10:10:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:10:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.650054+0000", 2024-09-21T10:10:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3740999999999999e-05 2024-09-21T10:10:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:10:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.650061+0000", 2024-09-21T10:10:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5690000000000001e-06 2024-09-21T10:10:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:10:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.650134+0000", 2024-09-21T10:10:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3059999999999995e-05 2024-09-21T10:10:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:10:55.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.650156+0000", 2024-09-21T10:10:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1832e-05 2024-09-21T10:10:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:10:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:10:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:10:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.45263.0:156399 6.f e181/165)", 2024-09-21T10:10:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:10:55.654045+0000", 2024-09-21T10:10:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0040219909999999999, 2024-09-21T10:10:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0042909569999999998, 2024-09-21T10:10:55.681 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:10:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:10:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-21T10:10:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:10:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:10:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.654045+0000", 2024-09-21T10:10:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:10:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:10:55.682 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.654045+0000", 2024-09-21T10:10:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:10:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:10:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.654046+0000", 2024-09-21T10:10:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1829999999999999e-06 2024-09-21T10:10:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:10:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.654053+0000", 2024-09-21T10:10:55.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4999999999999996e-06 2024-09-21T10:10:55.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:10:55.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.654068+0000", 2024-09-21T10:10:55.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5452999999999999e-05 2024-09-21T10:10:55.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:10:55.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.654077+0000", 2024-09-21T10:10:55.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7409999999999998e-06 2024-09-21T10:10:55.684 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:10:55.685 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:10:55.685 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:10:55.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.685 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.45263.0:156403 6.1 e181/165 6:8baa02a6:::benchmark_data_smithi098_169033_object33744:head v 181'9930, mlcod=181'9928)", 2024-09-21T10:10:55.685 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:10:55.654648+0000", 2024-09-21T10:10:55.685 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0034187380000000002, 2024-09-21T10:10:55.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037305849999999998, 2024-09-21T10:10:55.685 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:10:55.686 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:10:55.686 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:10:55.686 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:10:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:10:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.654648+0000", 2024-09-21T10:10:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:10:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:10:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.654648+0000", 2024-09-21T10:10:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:10:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:10:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.654649+0000", 2024-09-21T10:10:55.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1800000000000005e-07 2024-09-21T10:10:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:10:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.654661+0000", 2024-09-21T10:10:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2271e-05 2024-09-21T10:10:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:10:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.654675+0000", 2024-09-21T10:10:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3501e-05 2024-09-21T10:10:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:10:55.689 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.654681+0000", 2024-09-21T10:10:55.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4169999999999997e-06 2024-09-21T10:10:55.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:10:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.654849+0000", 2024-09-21T10:10:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016768299999999999 2024-09-21T10:10:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:10:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:10:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:10:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:55.654897+0000", 2024-09-21T10:10:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8225e-05 2024-09-21T10:10:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:10:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:10:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:10:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:10:55.690 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-21T10:10:55.691 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 6 2024-09-21T10:10:55.691 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:10:55.731 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:55.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:55.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:55.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:55.773 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:10:55.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:55.885 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:55.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:55.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:55.987 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:56.014 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:56.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:56.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:56.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:55 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1928380641' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:10:56.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:55 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:10:56.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:55 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1928380641' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:10:56.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:55 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:10:56.115 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:55 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1928380641' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:10:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:55 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:10:56.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:56.168 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:56.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:56.257 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:56.269 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:56.293 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:56.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:56.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:56.395 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:56.424 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:56.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:56.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:56.526 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:56.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:56.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:56.662 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:56.699 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:56.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:56.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:56.801 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:56.842 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:56.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:56.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:56.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:56.991 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:56.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:56.992 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:10:56.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:57.071 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:56 smithi098 ceph-mon[97580]: pgmap v531: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 898 op/s 2024-09-21T10:10:57.071 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:56 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:10:57.072 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:56 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:10:57.072 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:56 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:10:57.072 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:56 smithi098 ceph-mon[97580]: osdmap e182: 8 total, 8 up, 7 in 2024-09-21T10:10:57.072 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:56 smithi098 ceph-mon[100914]: pgmap v531: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 898 op/s 2024-09-21T10:10:57.072 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:56 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:10:57.072 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:56 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:10:57.072 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:56 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:10:57.072 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:56 smithi098 ceph-mon[100914]: osdmap e182: 8 total, 8 up, 7 in 2024-09-21T10:10:57.094 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:57.117 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:57.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:57.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:57.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:56 smithi196 ceph-mon[84784]: pgmap v531: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 898 op/s 2024-09-21T10:10:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:56 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:10:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:56 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:10:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:56 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:10:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:56 smithi196 ceph-mon[84784]: osdmap e182: 8 total, 8 up, 7 in 2024-09-21T10:10:57.219 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:57.267 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:57.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:57.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:57.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:57.392 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:57.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:57.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:57.494 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:57.524 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:57.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:57.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:57.625 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:57.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:57.654 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:57.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:57.755 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:57.778 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:57.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:57.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:57.880 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:57.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:57.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:58.020 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:58.044 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:58.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:58.146 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:58.174 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:58.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:58.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:58.276 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:58.300 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:58.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:58.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:58.402 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:58.434 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:58.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:58.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:58.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:58.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:58.571 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:58.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:58.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:58.704 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:58.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:58.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:58.806 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:58.839 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:58.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:58.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:58.941 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:58.964 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:58.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:58.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:59.005 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:10:58 smithi098 ceph-mon[100914]: pgmap v533: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 718 op/s 2024-09-21T10:10:59.006 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:10:58 smithi098 ceph-mon[97580]: pgmap v533: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 718 op/s 2024-09-21T10:10:59.066 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:59.096 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:59.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:59.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:59.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:10:58 smithi196 ceph-mon[84784]: pgmap v533: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 718 op/s 2024-09-21T10:10:59.198 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:59.233 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:59.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:59.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:59.335 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:59.364 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:59.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:59.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:59.466 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:59.497 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:59.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:59.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:59.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:59.623 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:59.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:59.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:59.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:59.761 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:10:59.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:59.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:59.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:10:59.863 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:59.888 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:10:59.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:59.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:59.990 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:00.014 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:00.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:00.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:00.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:00.146 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:00.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:00.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:00.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:00.272 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:00.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:00.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:00.374 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:00.401 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:00.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:00.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:00.503 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:00.528 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:00.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:00.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:00.630 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:00.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:11:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:11:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:11:00.654 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:00.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:00.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:00.756 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:00.789 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:00.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:00.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:00.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:00 smithi098 ceph-mon[97580]: pgmap v534: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 938 op/s 2024-09-21T10:11:00.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:11:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:11:00] "GET /metrics HTTP/1.1" 200 36451 "" "Prometheus/2.51.0" 2024-09-21T10:11:00.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:00 smithi098 ceph-mon[100914]: pgmap v534: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 938 op/s 2024-09-21T10:11:00.891 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:00.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:00.914 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:00.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:01.015 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:01.039 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:01.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:01.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:01.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:00 smithi196 ceph-mon[84784]: pgmap v534: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 938 op/s 2024-09-21T10:11:01.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:01.178 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:01.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:01.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:01.280 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:01.309 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:01.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:01.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:01.410 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:01.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:01.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:01.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:01.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:01.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:01.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:01.679 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:01.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:01.714 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:01.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:01.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:01.860 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:01.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:01.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:01.962 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:01.988 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:01.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:01.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:02.090 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:02.134 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:02.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:02.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:02.169 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:02.222 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:11:02.223 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:11:02.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:02.274 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:11:02.274 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:11:02.291 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:02.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:02.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:02.307 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:11:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:11:02.344 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:11:02.344 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:11:02.379 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:11:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:11:02.393 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:02.420 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:11:02.421 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:11:02.431 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:02.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:02.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:02.456 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:11:02.457 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:11:02.488 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:11:02.488 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:11:02.532 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:02.575 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:02.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:02.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:02.677 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:02.723 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:02.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:02.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:02.778 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T10:11:02.826 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:02.852 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:02.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:02.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:02.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:02.984 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:02.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:02.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:03.086 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:03.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:02 smithi098 ceph-mon[97580]: pgmap v535: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 879 op/s 2024-09-21T10:11:03.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:02 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/337009784' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:03.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:02 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3453678057' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:11:03.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:02 smithi098 ceph-mon[100914]: pgmap v535: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 879 op/s 2024-09-21T10:11:03.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:02 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/337009784' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:03.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:02 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3453678057' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:11:03.114 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:03.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:03.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:03.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:02 smithi196 ceph-mon[84784]: pgmap v535: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 879 op/s 2024-09-21T10:11:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:02 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/337009784' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:02 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3453678057' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:11:03.216 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:03.249 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:03.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:03.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:03.351 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:03.376 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:03.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:03.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:03.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:03.517 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:03.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:03.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:03.619 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:03.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:03.647 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:03.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:03.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:03.779 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:03.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:03.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:03.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:03.921 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:03.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:03.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:03.949 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:03 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3453678057' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:11:03.949 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:03 smithi098 ceph-mon[100914]: osdmap e183: 8 total, 8 up, 7 in 2024-09-21T10:11:03.949 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:03 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3453678057' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:11:03.949 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:03 smithi098 ceph-mon[97580]: osdmap e183: 8 total, 8 up, 7 in 2024-09-21T10:11:04.023 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:04.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:04.054 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:04.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:04.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:03 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3453678057' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:11:04.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:03 smithi196 ceph-mon[84784]: osdmap e183: 8 total, 8 up, 7 in 2024-09-21T10:11:04.155 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:04.190 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:04.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:04.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:04.292 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:04.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:04.321 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:04.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:04.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:04.433 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T10:11:04.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:04.472 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:04.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:04.573 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:04.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:04.624 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:04.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:04.725 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:04.748 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:04.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:04.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:04.850 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:04.875 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:04.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:04.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:04.977 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:05.004 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:05.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:05.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:05.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:04 smithi098 ceph-mon[97580]: pgmap v537: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 956 op/s 2024-09-21T10:11:05.051 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:04 smithi098 ceph-mon[100914]: pgmap v537: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 956 op/s 2024-09-21T10:11:05.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:05.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:04 smithi196 ceph-mon[84784]: pgmap v537: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 956 op/s 2024-09-21T10:11:05.158 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:05.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:05.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:05.260 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:05.283 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:05.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:05.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:05.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:05.422 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:05.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:05.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:05.524 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:05.559 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:05.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:05.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:05.661 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:05.699 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:05.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:05.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:05.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:05.828 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:05.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:05.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:05.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:05.967 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:05.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:05.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:06.071 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:06.118 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:06.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:06.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:06.220 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:06.241 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:06.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:06.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:06.343 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:06.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:06.375 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:06.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:06.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:06.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:06.514 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:06.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:06.614 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:06.639 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:06.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:06.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:06.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:06.782 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:06.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:06.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:06.884 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:06.908 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:06.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:07.010 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:07.034 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:07.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:07.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:07.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:06 smithi098 ceph-mon[97580]: pgmap v538: 89 pgs: 89 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 968 op/s 2024-09-21T10:11:07.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:06 smithi098 ceph-mon[100914]: pgmap v538: 89 pgs: 89 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 968 op/s 2024-09-21T10:11:07.090 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:11:07.136 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:06 smithi196 ceph-mon[84784]: pgmap v538: 89 pgs: 89 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 968 op/s 2024-09-21T10:11:07.161 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:07.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:07.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:07.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:07.308 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:07.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:07.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:07.410 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:07.462 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:07.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:07.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:07.564 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:07.588 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:07.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:07.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:07.690 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:07.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:07.737 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:07.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:07.838 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:07.901 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:07.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:07.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:08.003 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:08.028 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:08.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:08.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:08.130 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:08.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:08.187 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:08.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:08.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:08 smithi098 ceph-mon[97580]: pgmap v539: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 909 op/s 2024-09-21T10:11:08.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:11:08.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:08.255 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:08 smithi098 ceph-mon[100914]: pgmap v539: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 909 op/s 2024-09-21T10:11:08.255 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:11:08.255 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:08.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:08.319 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:08.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:08.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:08.414 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:11:08.414 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:11:08.421 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:08.444 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:08.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:08.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:08.546 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:08.595 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:08.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:08.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:08 smithi196 ceph-mon[84784]: pgmap v539: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 909 op/s 2024-09-21T10:11:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:11:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:08.697 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:08.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:08.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:08.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:08.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:08.825 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:08.847 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:08.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:08.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:08.949 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:08.974 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:08.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:08.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:09.078 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:09.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:09.115 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:09.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:09.216 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:09.242 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:09.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:09.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:09.256 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:11:09.343 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:09.381 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:09.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:09.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:09.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:11:09.483 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:09.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:09 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3166928444' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:11:09.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:09 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:11:09.501 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:09 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3166928444' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:11:09.501 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:09 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:11:09.514 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:09.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:09.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:09.616 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:09.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:09 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3166928444' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:11:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:09 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:11:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:09.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:09.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:09.753 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:09.779 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:09.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:09.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:09.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:09.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:09.905 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:09.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:10.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:10.030 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:10.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:10.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:10.131 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:10.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:10.154 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:10.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:10.254 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:10.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:10.279 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:10.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:10.380 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:10.412 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:10.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:10.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:10.513 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:10.547 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:10.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:10.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:10.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:10 smithi098 ceph-mon[97580]: pgmap v540: 89 pgs: 89 active+clean; 4.8 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 897 op/s 2024-09-21T10:11:10.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:10 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:11:10.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:10 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:11:10.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:10 smithi098 ceph-mon[97580]: osdmap e184: 8 total, 8 up, 7 in 2024-09-21T10:11:10.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:10 smithi098 ceph-mon[100914]: pgmap v540: 89 pgs: 89 active+clean; 4.8 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 897 op/s 2024-09-21T10:11:10.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:10 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:11:10.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:10 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:11:10.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:10 smithi098 ceph-mon[100914]: osdmap e184: 8 total, 8 up, 7 in 2024-09-21T10:11:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:10 smithi196 ceph-mon[84784]: pgmap v540: 89 pgs: 89 active+clean; 4.8 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 897 op/s 2024-09-21T10:11:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:10 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:11:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:10 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:11:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:10 smithi196 ceph-mon[84784]: osdmap e184: 8 total, 8 up, 7 in 2024-09-21T10:11:10.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:11:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:11:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:11:10.649 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:10.675 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:10.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:10.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:10.777 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:10.804 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:10.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:10.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:10.906 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:10.941 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:10.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:10.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:10.960 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:11:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:11:10] "GET /metrics HTTP/1.1" 200 36453 "" "Prometheus/2.51.0" 2024-09-21T10:11:11.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:11.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:11.078 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:11.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:11.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:11.213 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:11.214 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:11.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:11.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:11.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:11.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:11.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:11.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:11.495 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:11.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:11.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:11.597 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:11.622 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:11.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:11.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:11.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:11.791 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:11.792 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:11.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:11.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:11.894 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:11.921 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:11.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:11.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:12.023 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:12.061 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:12.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:12.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:12.162 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:12.207 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:12.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:12.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:12.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:12.333 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:12.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:12.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:12.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:12 smithi098 ceph-mon[97580]: pgmap v542: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 816 op/s 2024-09-21T10:11:12.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:12 smithi098 ceph-mon[100914]: pgmap v542: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 816 op/s 2024-09-21T10:11:12.435 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:12.474 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:12.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:12.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:12.576 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:12.602 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:12.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:12.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:12.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:12 smithi196 ceph-mon[84784]: pgmap v542: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 816 op/s 2024-09-21T10:11:12.704 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:12.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:11:12.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:12.740 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:12.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:12.840 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:12.864 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:12.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:12.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:12.966 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:13.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:13.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:13.148 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:13.174 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:13.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:13.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:13.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:13.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:13.304 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:13.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:13.405 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:13.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:13.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:13.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:13.536 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:13.563 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:13.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:13.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:13.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:13.707 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:13.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:13.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:13.809 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:13.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:13.837 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:13.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:13.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:13.977 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:13.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:13.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:14.078 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:14.119 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:14.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:14.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:14.221 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:14.245 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:14.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:14.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:14.347 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:14.380 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:14.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:14.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:14.479 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:14 smithi098 ceph-mon[97580]: pgmap v543: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 881 op/s 2024-09-21T10:11:14.479 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:14 smithi098 ceph-mon[100914]: pgmap v543: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 881 op/s 2024-09-21T10:11:14.483 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:14.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:14.510 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:14.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:14.611 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:14.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:14 smithi196 ceph-mon[84784]: pgmap v543: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 881 op/s 2024-09-21T10:11:14.660 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:14.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:14.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:14.662 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:11:14.662 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:11:14.662 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:11:14.662 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:11:14.663 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:11:14.663 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:11:14.663 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:11:14.663 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:11:14.663 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:11:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:11:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:11:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:11:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:11:14.664 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:11:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:11:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:11:14.762 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:14.802 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:14.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:14.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:14.904 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:14.941 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:14.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:14.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:15.042 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:15.068 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:15.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:15.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:15.170 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:15.214 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:15.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:15.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:15.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:15.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:15 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3777432419' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:15.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:15 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3777432419' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:15.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:15.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:15.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:15.490 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:15.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:15.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:15.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:15.635 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:15.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:15.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:15.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:15 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3777432419' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:15.737 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:15.763 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:15.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:15.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:15.866 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:15.894 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:15.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:15.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:15.996 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:16.023 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:16.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:16.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:16.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:16.167 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:16.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:16.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:16.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:16.304 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:16.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:16.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:16.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:16.443 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:16.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:16.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:16.492 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:16.545 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:16.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:16.572 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:16.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:16.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:16 smithi098 ceph-mon[97580]: pgmap v544: 89 pgs: 89 active+clean; 4.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 874 op/s 2024-09-21T10:11:16.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:16 smithi098 ceph-mon[100914]: pgmap v544: 89 pgs: 89 active+clean; 4.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 874 op/s 2024-09-21T10:11:16.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:16 smithi196 ceph-mon[84784]: pgmap v544: 89 pgs: 89 active+clean; 4.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 874 op/s 2024-09-21T10:11:16.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:16.708 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:16.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:16.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:16.810 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:16.837 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:16.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:16.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:16.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:16.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:16.970 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:16.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:17.071 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:17.097 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:17.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:17.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:17.198 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:17.238 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:17.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:17.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:17.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:17.372 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:17.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:17.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:17.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:17.501 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:17.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:17.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:17.552 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:11:17.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:17.665 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:17.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:17.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:17.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:17.815 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:17.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:17.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:17.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:17.964 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:17.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:17.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:18.066 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:18.107 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:18.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:18.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:18.209 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:18.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:18.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:18.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:18.332 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:18.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:18 smithi098 ceph-mon[97580]: pgmap v545: 89 pgs: 89 active+clean; 4.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 870 op/s 2024-09-21T10:11:18.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:18 smithi098 ceph-mon[100914]: pgmap v545: 89 pgs: 89 active+clean; 4.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 870 op/s 2024-09-21T10:11:18.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:18.377 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:18.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:18.478 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:18.502 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:18.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:18.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:18.605 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:18 smithi196 ceph-mon[84784]: pgmap v545: 89 pgs: 89 active+clean; 4.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 870 op/s 2024-09-21T10:11:18.639 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:18.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:18.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:18.741 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:18.765 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:18.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:18.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:18.867 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:18.891 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:18.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:18.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:18.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:19.021 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:19.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:19.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:19.123 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:19.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:19.162 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:19.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:19.263 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:19.286 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:19.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:19.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T10:11:19.388 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:19.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:19.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:19.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:19.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:19.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:19.555 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:19.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:19.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:19 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3939679780' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:11:19.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:19 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:11:19.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:19 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3939679780' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:11:19.600 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:19 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:11:19.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:19 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3939679780' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:11:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:19 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:11:19.656 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:19.688 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:19.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:19.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:19.790 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:19.817 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:19.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:19.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:19.918 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:19.956 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:19.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:19.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:20.058 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:20.099 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:20.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:20.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:20.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:20.225 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T10:11:20.226 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:11:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:20.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:20.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:20.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:20.378 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:20.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:20.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:20.479 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:20.504 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:20.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:20.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:20.548 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:20 smithi098 ceph-mon[97580]: pgmap v546: 89 pgs: 89 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 900 op/s 2024-09-21T10:11:20.548 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:20 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:11:20.548 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:20 smithi098 ceph-mon[97580]: osdmap e185: 8 total, 8 up, 7 in 2024-09-21T10:11:20.548 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:20 smithi098 ceph-mon[100914]: pgmap v546: 89 pgs: 89 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 900 op/s 2024-09-21T10:11:20.548 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:20 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:11:20.548 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:20 smithi098 ceph-mon[100914]: osdmap e185: 8 total, 8 up, 7 in 2024-09-21T10:11:20.606 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:20.631 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:20.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:20.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:20.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:20 smithi196 ceph-mon[84784]: pgmap v546: 89 pgs: 89 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 900 op/s 2024-09-21T10:11:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:20 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:11:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:20 smithi196 ceph-mon[84784]: osdmap e185: 8 total, 8 up, 7 in 2024-09-21T10:11:20.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:11:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:11:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:11:20.732 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:20.756 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:20.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:20.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:20.798 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:11:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:11:20] "GET /metrics HTTP/1.1" 200 36453 "" "Prometheus/2.51.0" 2024-09-21T10:11:20.857 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:20.895 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:20.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:20.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:20.917 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:11:20.917 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-21T10:11:20.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-21T10:11:20.918 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:11:20.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.918 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:35209 5.b 5:d050c117:::benchmark_data_smithi098_135187_object35208:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:11:20.918 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.630227+0000", 2024-09-21T10:11:20.918 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 439.07122334299999, 2024-09-21T10:11:20.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.12529087799999999, 2024-09-21T10:11:20.918 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:20.918 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:20.918 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:20.918 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:20.918 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:11:20.918 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:11:20.919 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35209 2024-09-21T10:11:20.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.919 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:20.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:20.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.630227+0000", 2024-09-21T10:11:20.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:20.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.630227+0000", 2024-09-21T10:11:20.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:20.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.630227+0000", 2024-09-21T10:11:20.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5599999999999995e-07 2024-09-21T10:11:20.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:20.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.630247+0000", 2024-09-21T10:11:20.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.946e-05 2024-09-21T10:11:20.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:20.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.630254+0000", 2024-09-21T10:11:20.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4900000000000003e-06 2024-09-21T10:11:20.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.921 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:20.921 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.630259+0000", 2024-09-21T10:11:20.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1320000000000002e-06 2024-09-21T10:11:20.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.921 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:20.921 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.631477+0000", 2024-09-21T10:11:20.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001217641 2024-09-21T10:11:20.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.921 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:20.921 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.631596+0000", 2024-09-21T10:11:20.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000118804 2024-09-21T10:11:20.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:11:20.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.631725+0000", 2024-09-21T10:11:20.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012932699999999999 2024-09-21T10:11:20.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635243+0000", 2024-09-21T10:11:20.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035178549999999999 2024-09-21T10:11:20.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:20.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635565+0000", 2024-09-21T10:11:20.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000322412 2024-09-21T10:11:20.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.755430+0000", 2024-09-21T10:11:20.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.119864811 2024-09-21T10:11:20.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:20.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.755480+0000", 2024-09-21T10:11:20.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9493000000000002e-05 2024-09-21T10:11:20.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:20.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.755518+0000", 2024-09-21T10:11:20.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7897e-05 2024-09-21T10:11:20.924 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.924 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:20.924 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.924 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:35224 5.a 5:51fa5009:::benchmark_data_smithi098_135187_object35223:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:11:20.924 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.635579+0000", 2024-09-21T10:11:20.924 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 439.06587073200001, 2024-09-21T10:11:20.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11871541400000001, 2024-09-21T10:11:20.924 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:20.924 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:20.924 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:20.925 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:20.925 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:11:20.925 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:11:20.925 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35224 2024-09-21T10:11:20.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.925 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:20.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:20.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635579+0000", 2024-09-21T10:11:20.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:20.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635579+0000", 2024-09-21T10:11:20.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:20.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635581+0000", 2024-09-21T10:11:20.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3230000000000001e-06 2024-09-21T10:11:20.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:20.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635602+0000", 2024-09-21T10:11:20.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1001000000000001e-05 2024-09-21T10:11:20.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:20.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635610+0000", 2024-09-21T10:11:20.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1410000000000005e-06 2024-09-21T10:11:20.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:20.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635616+0000", 2024-09-21T10:11:20.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2570000000000004e-06 2024-09-21T10:11:20.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:20.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635659+0000", 2024-09-21T10:11:20.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2892999999999997e-05 2024-09-21T10:11:20.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:20.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635779+0000", 2024-09-21T10:11:20.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000120025 2024-09-21T10:11:20.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:11:20.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635905+0000", 2024-09-21T10:11:20.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000126382 2024-09-21T10:11:20.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:20.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754075+0000", 2024-09-21T10:11:20.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.118169815 2024-09-21T10:11:20.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754127+0000", 2024-09-21T10:11:20.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2080999999999998e-05 2024-09-21T10:11:20.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754200+0000", 2024-09-21T10:11:20.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3053999999999995e-05 2024-09-21T10:11:20.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754253+0000", 2024-09-21T10:11:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2377999999999999e-05 2024-09-21T10:11:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754295+0000", 2024-09-21T10:11:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2064000000000002e-05 2024-09-21T10:11:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:35226 5.a 5:577c758a:::benchmark_data_smithi098_135187_object35225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:11:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.635769+0000", 2024-09-21T10:11:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 439.06568084000003, 2024-09-21T10:11:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11862623799999999, 2024-09-21T10:11:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:11:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:11:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35226 2024-09-21T10:11:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635769+0000", 2024-09-21T10:11:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635769+0000", 2024-09-21T10:11:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635770+0000", 2024-09-21T10:11:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8199999999999999e-07 2024-09-21T10:11:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635788+0000", 2024-09-21T10:11:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8505999999999999e-05 2024-09-21T10:11:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635799+0000", 2024-09-21T10:11:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0173000000000001e-05 2024-09-21T10:11:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.635805+0000", 2024-09-21T10:11:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2299999999999996e-06 2024-09-21T10:11:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637399+0000", 2024-09-21T10:11:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0015946249999999999 2024-09-21T10:11:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637531+0000", 2024-09-21T10:11:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013183800000000001 2024-09-21T10:11:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:11:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637674+0000", 2024-09-21T10:11:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014236000000000001 2024-09-21T10:11:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754081+0000", 2024-09-21T10:11:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.116406883 2024-09-21T10:11:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754168+0000", 2024-09-21T10:11:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7264999999999999e-05 2024-09-21T10:11:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754327+0000", 2024-09-21T10:11:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015965599999999999 2024-09-21T10:11:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754367+0000", 2024-09-21T10:11:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9221999999999998e-05 2024-09-21T10:11:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754395+0000", 2024-09-21T10:11:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8697999999999999e-05 2024-09-21T10:11:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.44894.0:35224 5.a e110/105 ondisk, result = 0)", 2024-09-21T10:11:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.637962+0000", 2024-09-21T10:11:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 439.06348830600001, 2024-09-21T10:11:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.116172572, 2024-09-21T10:11:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:11:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637962+0000", 2024-09-21T10:11:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637962+0000", 2024-09-21T10:11:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637963+0000", 2024-09-21T10:11:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1179999999999999e-06 2024-09-21T10:11:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637968+0000", 2024-09-21T10:11:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8910000000000003e-06 2024-09-21T10:11:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637976+0000", 2024-09-21T10:11:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6289999999999999e-06 2024-09-21T10:11:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.637982+0000", 2024-09-21T10:11:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2299999999999999e-06 2024-09-21T10:11:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754114+0000", 2024-09-21T10:11:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.116132108 2024-09-21T10:11:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754122+0000", 2024-09-21T10:11:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1750000000000005e-06 2024-09-21T10:11:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754134+0000", 2024-09-21T10:11:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2421e-05 2024-09-21T10:11:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:35229 5.b 5:d08d135d:::benchmark_data_smithi098_135187_object35228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:11:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.638050+0000", 2024-09-21T10:11:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 439.06340034700003, 2024-09-21T10:11:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.117567405, 2024-09-21T10:11:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:11:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:11:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35229 2024-09-21T10:11:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638050+0000", 2024-09-21T10:11:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638050+0000", 2024-09-21T10:11:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638051+0000", 2024-09-21T10:11:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.502e-06 2024-09-21T10:11:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638072+0000", 2024-09-21T10:11:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0976e-05 2024-09-21T10:11:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638085+0000", 2024-09-21T10:11:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2988e-05 2024-09-21T10:11:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638092+0000", 2024-09-21T10:11:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1709999999999996e-06 2024-09-21T10:11:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638154+0000", 2024-09-21T10:11:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1567000000000002e-05 2024-09-21T10:11:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638258+0000", 2024-09-21T10:11:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000103959 2024-09-21T10:11:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:11:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638372+0000", 2024-09-21T10:11:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000113888 2024-09-21T10:11:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754684+0000", 2024-09-21T10:11:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11631245 2024-09-21T10:11:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754695+0000", 2024-09-21T10:11:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0969e-05 2024-09-21T10:11:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.755551+0000", 2024-09-21T10:11:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00085551900000000001 2024-09-21T10:11:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.755589+0000", 2024-09-21T10:11:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8213000000000002e-05 2024-09-21T10:11:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.755617+0000", 2024-09-21T10:11:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8203000000000001e-05 2024-09-21T10:11:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:35231 5.d 5:bb22eb7b:::benchmark_data_smithi098_135187_object35230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:11:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:04:01.638970+0000", 2024-09-21T10:11:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 439.06247996600001, 2024-09-21T10:11:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.12243952, 2024-09-21T10:11:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:11:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:11:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35231 2024-09-21T10:11:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638970+0000", 2024-09-21T10:11:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638970+0000", 2024-09-21T10:11:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.638971+0000", 2024-09-21T10:11:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1030000000000001e-06 2024-09-21T10:11:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.639022+0000", 2024-09-21T10:11:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0949999999999998e-05 2024-09-21T10:11:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754209+0000", 2024-09-21T10:11:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11518701200000001 2024-09-21T10:11:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754219+0000", 2024-09-21T10:11:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8430000000000005e-06 2024-09-21T10:11:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754669+0000", 2024-09-21T10:11:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044998299999999999 2024-09-21T10:11:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754792+0000", 2024-09-21T10:11:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012265999999999999 2024-09-21T10:11:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,7", 2024-09-21T10:11:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.754929+0000", 2024-09-21T10:11:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013770199999999999 2024-09-21T10:11:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.761225+0000", 2024-09-21T10:11:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.006295429 2024-09-21T10:11:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.761275+0000", 2024-09-21T10:11:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9883999999999999e-05 2024-09-21T10:11:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.761309+0000", 2024-09-21T10:11:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4767000000000001e-05 2024-09-21T10:11:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.761355+0000", 2024-09-21T10:11:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5918000000000003e-05 2024-09-21T10:11:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:04:01.761410+0000", 2024-09-21T10:11:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4268999999999998e-05 2024-09-21T10:11:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:159909 5.a 5:589715df:::benchmark_data_smithi098_135187_object24271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-21T10:11:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:05:46.477125+0000", 2024-09-21T10:11:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 334.224325158, 2024-09-21T10:11:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.119837549, 2024-09-21T10:11:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:20.954 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:20.954 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:20.954 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:11:20.954 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:11:20.954 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 159909 2024-09-21T10:11:20.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.954 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:20.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:20.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.477125+0000", 2024-09-21T10:11:20.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:20.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.477125+0000", 2024-09-21T10:11:20.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:20.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.477127+0000", 2024-09-21T10:11:20.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.826e-06 2024-09-21T10:11:20.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:20.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.477132+0000", 2024-09-21T10:11:20.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4709999999999998e-06 2024-09-21T10:11:20.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:20.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.477148+0000", 2024-09-21T10:11:20.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5449000000000001e-05 2024-09-21T10:11:20.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:20.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.477155+0000", 2024-09-21T10:11:20.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2869999999999999e-06 2024-09-21T10:11:20.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:20.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.478078+0000", 2024-09-21T10:11:20.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00092277099999999996 2024-09-21T10:11:20.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:20.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.478316+0000", 2024-09-21T10:11:20.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023818600000000001 2024-09-21T10:11:20.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:11:20.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.478412+0000", 2024-09-21T10:11:20.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6436999999999998e-05 2024-09-21T10:11:20.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:20.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.491697+0000", 2024-09-21T10:11:20.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013284374 2024-09-21T10:11:20.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.496201+0000", 2024-09-21T10:11:20.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0045046119999999999 2024-09-21T10:11:20.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.596874+0000", 2024-09-21T10:11:20.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.100672711 2024-09-21T10:11:20.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:20.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.596922+0000", 2024-09-21T10:11:20.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.774e-05 2024-09-21T10:11:20.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:20.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.596962+0000", 2024-09-21T10:11:20.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0685000000000001e-05 2024-09-21T10:11:20.959 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.959 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:20.959 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.960 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:159911 5.a 5:51b41833:::benchmark_data_smithi098_135187_object24273:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-21T10:11:20.960 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:05:46.481149+0000", 2024-09-21T10:11:20.960 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 334.22030123299999, 2024-09-21T10:11:20.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.115993818, 2024-09-21T10:11:20.960 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:20.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:20.960 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:20.960 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:20.960 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:11:20.960 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:11:20.960 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 159911 2024-09-21T10:11:20.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.960 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:20.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:20.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.481149+0000", 2024-09-21T10:11:20.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:20.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.481149+0000", 2024-09-21T10:11:20.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:20.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.481151+0000", 2024-09-21T10:11:20.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.739e-06 2024-09-21T10:11:20.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:20.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.481157+0000", 2024-09-21T10:11:20.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2720000000000003e-06 2024-09-21T10:11:20.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:20.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.481170+0000", 2024-09-21T10:11:20.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3508e-05 2024-09-21T10:11:20.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:20.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.481178+0000", 2024-09-21T10:11:20.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7929999999999998e-06 2024-09-21T10:11:20.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:20.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.485802+0000", 2024-09-21T10:11:20.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046242030000000003 2024-09-21T10:11:20.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:20.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.485909+0000", 2024-09-21T10:11:20.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106229 2024-09-21T10:11:20.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:11:20.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.486553+0000", 2024-09-21T10:11:20.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00064429199999999998 2024-09-21T10:11:20.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:20.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.496219+0000", 2024-09-21T10:11:20.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0096666579999999998 2024-09-21T10:11:20.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.496256+0000", 2024-09-21T10:11:20.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6879999999999999e-05 2024-09-21T10:11:20.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.597046+0000", 2024-09-21T10:11:20.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.100789183 2024-09-21T10:11:20.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:20.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.597089+0000", 2024-09-21T10:11:20.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3092e-05 2024-09-21T10:11:20.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:20.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:05:46.597143+0000", 2024-09-21T10:11:20.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3968999999999998e-05 2024-09-21T10:11:20.965 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.965 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:20.965 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.966 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:267699 5.a 5:51b91a91:::benchmark_data_smithi098_135187_object132061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:11:20.966 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:59.409137+0000", 2024-09-21T10:11:20.966 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 201.29231345100001, 2024-09-21T10:11:20.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3530456719999999, 2024-09-21T10:11:20.966 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:20.966 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:20.966 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:20.966 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:20.966 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:11:20.966 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:11:20.966 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 267699 2024-09-21T10:11:20.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.966 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:20.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:20.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409137+0000", 2024-09-21T10:11:20.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:20.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409137+0000", 2024-09-21T10:11:20.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:20.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409138+0000", 2024-09-21T10:11:20.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.331e-06 2024-09-21T10:11:20.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:20.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409143+0000", 2024-09-21T10:11:20.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0309999999999998e-06 2024-09-21T10:11:20.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:20.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409160+0000", 2024-09-21T10:11:20.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6694e-05 2024-09-21T10:11:20.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:20.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409168+0000", 2024-09-21T10:11:20.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2230000000000007e-06 2024-09-21T10:11:20.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:20.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409241+0000", 2024-09-21T10:11:20.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2818999999999995e-05 2024-09-21T10:11:20.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:11:20.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409253+0000", 2024-09-21T10:11:20.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1914e-05 2024-09-21T10:11:20.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:20.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.370041+0000", 2024-09-21T10:11:20.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.96078831799999997 2024-09-21T10:11:20.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:11:20.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.370053+0000", 2024-09-21T10:11:20.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2306999999999999e-05 2024-09-21T10:11:20.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:20.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.724460+0000", 2024-09-21T10:11:20.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3544063770000001 2024-09-21T10:11:20.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:20.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.724591+0000", 2024-09-21T10:11:20.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013132199999999999 2024-09-21T10:11:20.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3,4", 2024-09-21T10:11:20.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.724747+0000", 2024-09-21T10:11:20.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000156553 2024-09-21T10:11:20.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.742698+0000", 2024-09-21T10:11:20.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017950560000000001 2024-09-21T10:11:20.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:20.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.747832+0000", 2024-09-21T10:11:20.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051342310000000004 2024-09-21T10:11:20.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.755583+0000", 2024-09-21T10:11:20.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0077510879999999997 2024-09-21T10:11:20.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.761982+0000", 2024-09-21T10:11:20.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0063988789999999997 2024-09-21T10:11:20.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:20.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.762068+0000", 2024-09-21T10:11:20.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5698000000000005e-05 2024-09-21T10:11:20.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:20.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.762182+0000", 2024-09-21T10:11:20.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011432700000000001 2024-09-21T10:11:20.973 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.973 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:20.974 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.974 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269037 5.a 5:5ae457a8:::benchmark_data_smithi098_135187_object133399:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:11:20.974 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.480193+0000", 2024-09-21T10:11:20.974 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 196.221257006, 2024-09-21T10:11:20.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0309086750000001, 2024-09-21T10:11:20.974 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:20.974 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:20.974 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:20.974 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:20.974 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:11:20.974 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:11:20.975 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269037 2024-09-21T10:11:20.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.975 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:20.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:20.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480193+0000", 2024-09-21T10:11:20.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:20.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480193+0000", 2024-09-21T10:11:20.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480195+0000", 2024-09-21T10:11:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6479999999999999e-06 2024-09-21T10:11:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480200+0000", 2024-09-21T10:11:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7420000000000003e-06 2024-09-21T10:11:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480215+0000", 2024-09-21T10:11:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4293e-05 2024-09-21T10:11:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:20.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480222+0000", 2024-09-21T10:11:20.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3640000000000001e-06 2024-09-21T10:11:20.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:20.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480271+0000", 2024-09-21T10:11:20.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8466e-05 2024-09-21T10:11:20.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:11:20.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480276+0000", 2024-09-21T10:11:20.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9240000000000004e-06 2024-09-21T10:11:20.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:20.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.490543+0000", 2024-09-21T10:11:20.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0102668859999999 2024-09-21T10:11:20.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:20.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.490646+0000", 2024-09-21T10:11:20.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010246099999999999 2024-09-21T10:11:20.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:11:20.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.490749+0000", 2024-09-21T10:11:20.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000102839 2024-09-21T10:11:20.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510822+0000", 2024-09-21T10:11:20.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020072995999999999 2024-09-21T10:11:20.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:20.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510836+0000", 2024-09-21T10:11:20.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4445e-05 2024-09-21T10:11:20.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510993+0000", 2024-09-21T10:11:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000157207 2024-09-21T10:11:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511062+0000", 2024-09-21T10:11:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8923000000000004e-05 2024-09-21T10:11:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511102+0000", 2024-09-21T10:11:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9480999999999997e-05 2024-09-21T10:11:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269043 5.a 5:5d7883f5:::benchmark_data_smithi098_135187_object133405:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:11:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.487457+0000", 2024-09-21T10:11:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 196.21399253000001, 2024-09-21T10:11:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.040401159, 2024-09-21T10:11:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:11:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:11:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269043 2024-09-21T10:11:20.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.982 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:20.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:20.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487457+0000", 2024-09-21T10:11:20.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:20.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487457+0000", 2024-09-21T10:11:20.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:20.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487458+0000", 2024-09-21T10:11:20.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8599999999999996e-07 2024-09-21T10:11:20.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:20.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487463+0000", 2024-09-21T10:11:20.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2830000000000003e-06 2024-09-21T10:11:20.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:20.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487476+0000", 2024-09-21T10:11:20.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2819e-05 2024-09-21T10:11:20.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:20.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487481+0000", 2024-09-21T10:11:20.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2519999999999999e-06 2024-09-21T10:11:20.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:20.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487524+0000", 2024-09-21T10:11:20.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2701000000000003e-05 2024-09-21T10:11:20.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:11:20.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487529+0000", 2024-09-21T10:11:20.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0690000000000001e-06 2024-09-21T10:11:20.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:20.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.500694+0000", 2024-09-21T10:11:20.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0131655589999999 2024-09-21T10:11:20.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:20.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.500787+0000", 2024-09-21T10:11:20.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2785e-05 2024-09-21T10:11:20.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:11:20.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.500887+0000", 2024-09-21T10:11:20.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9599000000000006e-05 2024-09-21T10:11:20.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:20.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510920+0000", 2024-09-21T10:11:20.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010033686999999999 2024-09-21T10:11:20.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.517569+0000", 2024-09-21T10:11:20.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0066490730000000001 2024-09-21T10:11:20.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.527765+0000", 2024-09-21T10:11:20.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010196197000000001 2024-09-21T10:11:20.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:20.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.527818+0000", 2024-09-21T10:11:20.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2401000000000002e-05 2024-09-21T10:11:20.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:20.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.527859+0000", 2024-09-21T10:11:20.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0747999999999997e-05 2024-09-21T10:11:20.987 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.987 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:20.987 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.988 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269059 5.a 5:530cb704:::benchmark_data_smithi098_135187_object133421:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:11:20.988 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.508524+0000", 2024-09-21T10:11:20.988 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 196.19292553400001, 2024-09-21T10:11:20.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0242946340000001, 2024-09-21T10:11:20.988 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:20.988 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:20.988 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:20.988 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:20.988 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:11:20.988 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:11:20.988 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269059 2024-09-21T10:11:20.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.988 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:20.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:20.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508524+0000", 2024-09-21T10:11:20.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:20.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508524+0000", 2024-09-21T10:11:20.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:20.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508526+0000", 2024-09-21T10:11:20.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.387e-06 2024-09-21T10:11:20.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:20.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508530+0000", 2024-09-21T10:11:20.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.437e-06 2024-09-21T10:11:20.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:20.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508542+0000", 2024-09-21T10:11:20.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1249e-05 2024-09-21T10:11:20.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:20.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508547+0000", 2024-09-21T10:11:20.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5809999999999996e-06 2024-09-21T10:11:20.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:20.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508596+0000", 2024-09-21T10:11:20.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8641999999999999e-05 2024-09-21T10:11:20.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:11:20.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508602+0000", 2024-09-21T10:11:20.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8100000000000003e-06 2024-09-21T10:11:20.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:20.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.505838+0000", 2024-09-21T10:11:20.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.99723621699999998 2024-09-21T10:11:20.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:20.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.505943+0000", 2024-09-21T10:11:20.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010536500000000001 2024-09-21T10:11:20.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:11:20.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.506048+0000", 2024-09-21T10:11:20.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105221 2024-09-21T10:11:20.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:20.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.522571+0000", 2024-09-21T10:11:20.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016522236999999999 2024-09-21T10:11:20.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.523904+0000", 2024-09-21T10:11:20.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001332986 2024-09-21T10:11:20.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.532731+0000", 2024-09-21T10:11:20.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0088274440000000003 2024-09-21T10:11:20.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:20.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.532778+0000", 2024-09-21T10:11:20.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7413000000000002e-05 2024-09-21T10:11:20.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:20.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.532819+0000", 2024-09-21T10:11:20.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0645000000000001e-05 2024-09-21T10:11:20.994 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.994 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:20.994 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:20.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.994 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:68384 6.f 6:f1664d9c:::benchmark_data_smithi098_169033_object68383:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T10:11:20.994 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:09:25.279542+0000", 2024-09-21T10:11:20.994 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 115.421907788, 2024-09-21T10:11:20.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.168796999, 2024-09-21T10:11:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:11:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:11:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 68384 2024-09-21T10:11:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.279542+0000", 2024-09-21T10:11:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.279542+0000", 2024-09-21T10:11:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.279543+0000", 2024-09-21T10:11:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.265e-06 2024-09-21T10:11:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.279569+0000", 2024-09-21T10:11:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5505000000000001e-05 2024-09-21T10:11:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.279583+0000", 2024-09-21T10:11:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.367e-05 2024-09-21T10:11:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.279590+0000", 2024-09-21T10:11:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5020000000000003e-06 2024-09-21T10:11:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.282608+0000", 2024-09-21T10:11:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030175459999999999 2024-09-21T10:11:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.282726+0000", 2024-09-21T10:11:20.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000118219 2024-09-21T10:11:20.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:11:20.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.282854+0000", 2024-09-21T10:11:20.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000128109 2024-09-21T10:11:20.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.286365+0000", 2024-09-21T10:11:20.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035108069999999999 2024-09-21T10:11:20.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:20.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.286375+0000", 2024-09-21T10:11:20.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0458e-05 2024-09-21T10:11:20.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:20.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.448235+0000", 2024-09-21T10:11:20.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.16185946700000001 2024-09-21T10:11:20.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:20.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:20.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.448298+0000", 2024-09-21T10:11:20.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3656000000000002e-05 2024-09-21T10:11:20.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:20.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:21.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.448339+0000", 2024-09-21T10:11:21.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0794999999999998e-05 2024-09-21T10:11:21.000 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:21.000 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:21.000 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:21.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.000 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:68395 6.f 6:f5675bf7:::benchmark_data_smithi098_169033_object68394:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T10:11:21.000 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:09:25.285421+0000", 2024-09-21T10:11:21.000 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 115.41602938600001, 2024-09-21T10:11:21.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.16302048899999999, 2024-09-21T10:11:21.000 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:21.000 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:21.000 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:21.001 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:21.001 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:11:21.001 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:11:21.001 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 68395 2024-09-21T10:11:21.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.001 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:21.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:21.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.285421+0000", 2024-09-21T10:11:21.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:21.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:21.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.285421+0000", 2024-09-21T10:11:21.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:21.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:21.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.285422+0000", 2024-09-21T10:11:21.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.266e-06 2024-09-21T10:11:21.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:21.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.285441+0000", 2024-09-21T10:11:21.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9028000000000001e-05 2024-09-21T10:11:21.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:21.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.285452+0000", 2024-09-21T10:11:21.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1191e-05 2024-09-21T10:11:21.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:21.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.285460+0000", 2024-09-21T10:11:21.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1920000000000005e-06 2024-09-21T10:11:21.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:21.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.286399+0000", 2024-09-21T10:11:21.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00093865400000000003 2024-09-21T10:11:21.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:21.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.286525+0000", 2024-09-21T10:11:21.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012596300000000001 2024-09-21T10:11:21.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:11:21.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.286656+0000", 2024-09-21T10:11:21.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013059399999999999 2024-09-21T10:11:21.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:21.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.289472+0000", 2024-09-21T10:11:21.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028162 2024-09-21T10:11:21.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:21.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290034+0000", 2024-09-21T10:11:21.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00056179600000000004 2024-09-21T10:11:21.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:21.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.448374+0000", 2024-09-21T10:11:21.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.158340174 2024-09-21T10:11:21.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:21.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.448413+0000", 2024-09-21T10:11:21.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9050000000000001e-05 2024-09-21T10:11:21.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:21.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.448441+0000", 2024-09-21T10:11:21.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8381000000000002e-05 2024-09-21T10:11:21.006 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:21.006 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:21.006 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:21.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.006 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.45263.0:69631 6.8 e171/165 6:13176f2b:::benchmark_data_smithi098_169033_object69630:head v 171'4365, mlcod=171'4361)", 2024-09-21T10:11:21.006 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:09:26.362371+0000", 2024-09-21T10:11:21.006 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 114.339079486, 2024-09-21T10:11:21.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.14411320699999999, 2024-09-21T10:11:21.006 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:21.006 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:21.006 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:21.007 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:21.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:21.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.362371+0000", 2024-09-21T10:11:21.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:21.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:21.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.362371+0000", 2024-09-21T10:11:21.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:21.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:21.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.362371+0000", 2024-09-21T10:11:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1999999999999999e-07 2024-09-21T10:11:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.362456+0000", 2024-09-21T10:11:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4284999999999997e-05 2024-09-21T10:11:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.362467+0000", 2024-09-21T10:11:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1926999999999999e-05 2024-09-21T10:11:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.362473+0000", 2024-09-21T10:11:21.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0270000000000001e-06 2024-09-21T10:11:21.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:21.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.367318+0000", 2024-09-21T10:11:21.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048445160000000001 2024-09-21T10:11:21.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:21.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.367338+0000", 2024-09-21T10:11:21.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9813000000000002e-05 2024-09-21T10:11:21.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:21.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.506429+0000", 2024-09-21T10:11:21.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.13909102000000001 2024-09-21T10:11:21.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:21.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.506484+0000", 2024-09-21T10:11:21.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4898999999999997e-05 2024-09-21T10:11:21.010 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:21.010 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:21.010 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:21.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.010 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:69639 6.f 6:fce6225b:::benchmark_data_smithi098_169033_object69638:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T10:11:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:09:26.363945+0000", 2024-09-21T10:11:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 114.337505002, 2024-09-21T10:11:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.14220749799999999, 2024-09-21T10:11:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:11:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:11:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 69639 2024-09-21T10:11:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:21.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.363945+0000", 2024-09-21T10:11:21.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:21.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:21.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.363945+0000", 2024-09-21T10:11:21.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:21.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:21.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.363946+0000", 2024-09-21T10:11:21.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.48e-06 2024-09-21T10:11:21.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:21.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.363969+0000", 2024-09-21T10:11:21.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2288000000000001e-05 2024-09-21T10:11:21.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:21.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.363980+0000", 2024-09-21T10:11:21.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1596e-05 2024-09-21T10:11:21.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:21.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.363988+0000", 2024-09-21T10:11:21.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9689999999999994e-06 2024-09-21T10:11:21.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:21.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.364046+0000", 2024-09-21T10:11:21.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7757999999999998e-05 2024-09-21T10:11:21.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:21.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.364177+0000", 2024-09-21T10:11:21.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013133599999999999 2024-09-21T10:11:21.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:11:21.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.364308+0000", 2024-09-21T10:11:21.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013105899999999999 2024-09-21T10:11:21.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:21.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.369220+0000", 2024-09-21T10:11:21.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0049110780000000001 2024-09-21T10:11:21.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:21.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.369255+0000", 2024-09-21T10:11:21.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5398000000000002e-05 2024-09-21T10:11:21.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:21.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.506040+0000", 2024-09-21T10:11:21.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.13678535999999999 2024-09-21T10:11:21.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:21.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.506100+0000", 2024-09-21T10:11:21.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9681e-05 2024-09-21T10:11:21.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.506153+0000", 2024-09-21T10:11:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2494999999999998e-05 2024-09-21T10:11:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.45263.0:69634 6.8 e171/165 6:1ffca493:::benchmark_data_smithi098_169033_object69633:head v 171'4366, mlcod=171'4362)", 2024-09-21T10:11:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:09:26.368700+0000", 2024-09-21T10:11:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 114.332750084, 2024-09-21T10:11:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.13781283899999999, 2024-09-21T10:11:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:21.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:21.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.368700+0000", 2024-09-21T10:11:21.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:21.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:21.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.368700+0000", 2024-09-21T10:11:21.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:21.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:21.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.368701+0000", 2024-09-21T10:11:21.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9800000000000003e-07 2024-09-21T10:11:21.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:21.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.368735+0000", 2024-09-21T10:11:21.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3973000000000002e-05 2024-09-21T10:11:21.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:21.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.368744+0000", 2024-09-21T10:11:21.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3349999999999996e-06 2024-09-21T10:11:21.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:21.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.368750+0000", 2024-09-21T10:11:21.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1550000000000004e-06 2024-09-21T10:11:21.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:21.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.369847+0000", 2024-09-21T10:11:21.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010965079999999999 2024-09-21T10:11:21.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:21.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.369862+0000", 2024-09-21T10:11:21.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5563999999999999e-05 2024-09-21T10:11:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.506487+0000", 2024-09-21T10:11:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.136624736 2024-09-21T10:11:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.506513+0000", 2024-09-21T10:11:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5769999999999999e-05 2024-09-21T10:11:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.022 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.45263.0:69646 6.9 e171/165 6:9a5721c2:::benchmark_data_smithi098_169033_object69645:head v 171'4428, mlcod=171'4426)", 2024-09-21T10:11:21.022 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:09:26.370123+0000", 2024-09-21T10:11:21.022 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 114.331327449, 2024-09-21T10:11:21.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.14170012900000001, 2024-09-21T10:11:21.022 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:21.022 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:21.022 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:21.022 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:21.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:21.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.370123+0000", 2024-09-21T10:11:21.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:21.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:21.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.370123+0000", 2024-09-21T10:11:21.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:21.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:21.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.370123+0000", 2024-09-21T10:11:21.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9899999999999999e-07 2024-09-21T10:11:21.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:21.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.370144+0000", 2024-09-21T10:11:21.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1080000000000001e-05 2024-09-21T10:11:21.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.503755+0000", 2024-09-21T10:11:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.133610427 2024-09-21T10:11:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.503768+0000", 2024-09-21T10:11:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2850000000000001e-05 2024-09-21T10:11:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.503971+0000", 2024-09-21T10:11:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020311099999999999 2024-09-21T10:11:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:21.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.503992+0000", 2024-09-21T10:11:21.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0920999999999998e-05 2024-09-21T10:11:21.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:21.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.511572+0000", 2024-09-21T10:11:21.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0075799040000000002 2024-09-21T10:11:21.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:21.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.511823+0000", 2024-09-21T10:11:21.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00025103700000000002 2024-09-21T10:11:21.025 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:21.025 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:21.026 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:21.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.026 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.45263.0:69647 6.8 e171/165 6:12a2e610:::benchmark_data_smithi098_169033_object69646:head v 171'4367, mlcod=171'4363)", 2024-09-21T10:11:21.026 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:09:26.370393+0000", 2024-09-21T10:11:21.026 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 114.331056696, 2024-09-21T10:11:21.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.141974826, 2024-09-21T10:11:21.026 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:21.026 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:21.026 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:21.026 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:21.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:21.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.370393+0000", 2024-09-21T10:11:21.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:21.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:21.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.370393+0000", 2024-09-21T10:11:21.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:21.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:21.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.370394+0000", 2024-09-21T10:11:21.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1999999999999999e-07 2024-09-21T10:11:21.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:21.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.370456+0000", 2024-09-21T10:11:21.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2269999999999998e-05 2024-09-21T10:11:21.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:21.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.503755+0000", 2024-09-21T10:11:21.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.13329833499999999 2024-09-21T10:11:21.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:21.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.503767+0000", 2024-09-21T10:11:21.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.27e-05 2024-09-21T10:11:21.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:21.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.503829+0000", 2024-09-21T10:11:21.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1908000000000004e-05 2024-09-21T10:11:21.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:21.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.503851+0000", 2024-09-21T10:11:21.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2045e-05 2024-09-21T10:11:21.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:21.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.512320+0000", 2024-09-21T10:11:21.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0084691579999999992 2024-09-21T10:11:21.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:21.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:26.512368+0000", 2024-09-21T10:11:21.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7689999999999999e-05 2024-09-21T10:11:21.030 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:21.030 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:21.030 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:21.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.030 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:150298 6.f 6:f55fb0e4:::benchmark_data_smithi098_169033_object27639:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-21T10:11:21.030 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:10:48.675152+0000", 2024-09-21T10:11:21.030 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.026297823, 2024-09-21T10:11:21.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.12173680100000001, 2024-09-21T10:11:21.031 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:21.031 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:21.031 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:21.031 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:21.031 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:11:21.031 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:11:21.031 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 150298 2024-09-21T10:11:21.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.031 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:21.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:21.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:48.675152+0000", 2024-09-21T10:11:21.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:21.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:21.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:48.675152+0000", 2024-09-21T10:11:21.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:21.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:21.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:48.675154+0000", 2024-09-21T10:11:21.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.638e-06 2024-09-21T10:11:21.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:21.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:48.675158+0000", 2024-09-21T10:11:21.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4159999999999997e-06 2024-09-21T10:11:21.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:21.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:48.675173+0000", 2024-09-21T10:11:21.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4275e-05 2024-09-21T10:11:21.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:21.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:48.675180+0000", 2024-09-21T10:11:21.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0589999999999997e-06 2024-09-21T10:11:21.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:21.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:48.675239+0000", 2024-09-21T10:11:21.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9916e-05 2024-09-21T10:11:21.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:21.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:48.675364+0000", 2024-09-21T10:11:21.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012405200000000001 2024-09-21T10:11:21.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:11:21.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:48.675479+0000", 2024-09-21T10:11:21.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000115048 2024-09-21T10:11:21.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:21.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:48.682382+0000", 2024-09-21T10:11:21.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0069030389999999997 2024-09-21T10:11:21.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:21.035 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:48.682396+0000", 2024-09-21T10:11:21.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4474000000000001e-05 2024-09-21T10:11:21.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:21.035 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:48.796521+0000", 2024-09-21T10:11:21.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.114124455 2024-09-21T10:11:21.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:21.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:48.796839+0000", 2024-09-21T10:11:21.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031798699999999999 2024-09-21T10:11:21.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:21.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:21.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:21.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:10:48.796889+0000", 2024-09-21T10:11:21.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0442000000000003e-05 2024-09-21T10:11:21.036 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:21.036 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:21.036 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:21.036 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:21.036 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:21.036 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:11:21.037 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:21.071 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:21.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:21.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:21.173 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:21.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:21.200 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:21.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:21.301 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:11:21.330 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:21.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:21.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:21.432 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:21.461 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:21.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:21.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:21.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:21.596 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:21.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:21.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:21.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:21 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3792073021' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:11:21.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:21 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:11:21.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:21 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3792073021' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:11:21.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:21 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:11:21.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:21 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3792073021' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:11:21.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:21 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:11:21.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:21.729 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:21.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:21.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:21.830 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:21.855 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:21.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:21.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:21.931 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:11:21.957 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:22.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:22.036 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:22.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:22.137 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:22.208 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:22.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:22.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:22.304 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:22 smithi098 ceph-mon[97580]: pgmap v548: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 903 op/s 2024-09-21T10:11:22.304 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:22 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:11:22.304 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:22 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:11:22.304 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:22 smithi098 ceph-mon[97580]: osdmap e186: 8 total, 8 up, 7 in 2024-09-21T10:11:22.304 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:11:22.305 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:22.310 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:22.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:22.349 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:22.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:22.450 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:22.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:22.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:22.594 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:22.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:22 smithi098 ceph-mon[100914]: pgmap v548: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 903 op/s 2024-09-21T10:11:22.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:22 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:11:22.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:22 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:11:22.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:22 smithi098 ceph-mon[100914]: osdmap e186: 8 total, 8 up, 7 in 2024-09-21T10:11:22.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:11:22.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:22.621 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:22.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:22.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:22.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:22 smithi196 ceph-mon[84784]: pgmap v548: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 903 op/s 2024-09-21T10:11:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:22 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:11:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:22 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:11:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:22 smithi196 ceph-mon[84784]: osdmap e186: 8 total, 8 up, 7 in 2024-09-21T10:11:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:11:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:22.723 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:22.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:22.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:22.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:22.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:22.893 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:22.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:22.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:22.995 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:23.019 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:23.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:23.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:23.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:23.159 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:23.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:23.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:23.261 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:23.282 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:23.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:23.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:23.384 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:23.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:23.411 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:23.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:23.512 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:23.537 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:23.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:23.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:23.639 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:23.665 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:23.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:23.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:23.767 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:23.794 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:23.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:23.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:23.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:23.930 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:23.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:23.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:24.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:24.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:24.057 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:24.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:24.158 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:24.196 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:24.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:24.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:24.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:24.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:24.352 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:24.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:24.453 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:24.492 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:24.495 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:24.523 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:24.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:24.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:24.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:24 smithi098 ceph-mon[97580]: pgmap v550: 89 pgs: 89 active+clean; 4.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 935 op/s 2024-09-21T10:11:24.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:24 smithi098 ceph-mon[100914]: pgmap v550: 89 pgs: 89 active+clean; 4.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 935 op/s 2024-09-21T10:11:24.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:24.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:24 smithi196 ceph-mon[84784]: pgmap v550: 89 pgs: 89 active+clean; 4.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 935 op/s 2024-09-21T10:11:24.669 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:24.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:24.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:24.771 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:24.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:24.797 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:24.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:24.898 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:24.923 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:24.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:24.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:25.025 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:25.048 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:25.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:25.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:25.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:25.194 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:25.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:25.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:25.296 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:25.325 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:25.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:25.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:25.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:25.472 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:25.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:25.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:25.573 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:25.599 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:25.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:25.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:25.701 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:25.741 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:25.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:25.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:25.843 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:25.878 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:25.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:25.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:25.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:26.010 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:26.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:26.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:26.111 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:26.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:26.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:26.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:26.288 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:26.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:26.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:26.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:26 smithi098 ceph-mon[97580]: pgmap v551: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 956 op/s 2024-09-21T10:11:26.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:26 smithi098 ceph-mon[100914]: pgmap v551: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 956 op/s 2024-09-21T10:11:26.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:26.420 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:26.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:26.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:26.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:11:26.522 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:26.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:26.563 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:26.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:26.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:26 smithi196 ceph-mon[84784]: pgmap v551: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 956 op/s 2024-09-21T10:11:26.664 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:26.716 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:26.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:26.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:26.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:26.865 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:26.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:26.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:26.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:27.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:27.022 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:27.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:27.123 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:27.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:27.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:27.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:27.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:27.329 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:27.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:27.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:27.474 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:27.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:27.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:27.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:27.620 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:27.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:27.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:27.722 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:27.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:27.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:27.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:27.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:27.882 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:27.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:27.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:27.984 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:28.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:28.019 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:28.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:28.120 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:28.148 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:28.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:28.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:28.250 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:28.281 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:28.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:28.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:28.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:28.416 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:28.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:28.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:28.518 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:28.543 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:28.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:28.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:28.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:28 smithi098 ceph-mon[97580]: pgmap v552: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 699 op/s 2024-09-21T10:11:28.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:28 smithi098 ceph-mon[100914]: pgmap v552: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 699 op/s 2024-09-21T10:11:28.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:28 smithi196 ceph-mon[84784]: pgmap v552: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 699 op/s 2024-09-21T10:11:28.645 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:28.688 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:28.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:28.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:28.790 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:28.819 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:28.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:28.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:28.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:28.956 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:28.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:28.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:29.057 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:29.100 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:29.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:29.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:29.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:29.241 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:29.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:29.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:29.281 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:29.343 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:29.371 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:29.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:29.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:29.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:29.499 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:29.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:29.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:29.602 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:29.627 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:29.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:29.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:29.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:29.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:29.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:29.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:29.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:29.883 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:11:29.883 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:11:29.884 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:29.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:29.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:29.921 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:11:29.921 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:11:29.963 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:11:29.963 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:11:29.986 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:30.005 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:11:30.005 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:11:30.010 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:30.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:30.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:30.033 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:11:30.033 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:11:30.064 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:11:30.064 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:11:30.093 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:11:30.094 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:11:30.112 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:30.127 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:11:30.127 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:11:30.160 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:30.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:30.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:30.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:30.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:30.295 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:30.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:30.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:30.427 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:30.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:30.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:30.529 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:30.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:30 smithi098 ceph-mon[97580]: pgmap v553: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 881 op/s 2024-09-21T10:11:30.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:30 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/749473216' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:30.539 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:30 smithi098 ceph-mon[100914]: pgmap v553: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 881 op/s 2024-09-21T10:11:30.539 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:30 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/749473216' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:30.553 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:30.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:30.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:30 smithi196 ceph-mon[84784]: pgmap v553: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 881 op/s 2024-09-21T10:11:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:30 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/749473216' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:30.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:11:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:11:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:11:30.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:30.693 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:30.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:30.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:30.795 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:30.832 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:30.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:30.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:30.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:11:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:11:30] "GET /metrics HTTP/1.1" 200 36453 "" "Prometheus/2.51.0" 2024-09-21T10:11:30.934 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:30.968 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:30.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:30.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:31.070 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:31.112 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:31.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:31.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:31.214 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:31.239 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:31.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:31.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:31.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:31.367 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:31.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:31.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:31.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:31.493 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:31.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:31.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:31.595 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:31.620 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:31.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:31.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:31.721 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:31.755 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:11:31.755 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:31.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:31.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:31.759 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:11:31.858 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:31.885 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:31.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:31.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:31.986 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:32.013 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:32.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:32.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:32.115 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:32.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:32.157 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:32.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:32.258 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:32.301 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:32.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:32.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:32.403 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:32.438 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:32 smithi098 ceph-mon[97580]: pgmap v554: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 874 op/s 2024-09-21T10:11:32.439 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:32 smithi098 ceph-mon[97580]: from='client.54904 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:11:32.439 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:32 smithi098 ceph-mon[100914]: pgmap v554: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 874 op/s 2024-09-21T10:11:32.439 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:32 smithi098 ceph-mon[100914]: from='client.54904 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:11:32.444 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:32.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:32.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:32.545 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:32.582 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:32.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:32.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:32.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:32 smithi196 ceph-mon[84784]: pgmap v554: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 874 op/s 2024-09-21T10:11:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:32 smithi196 ceph-mon[84784]: from='client.54904 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:11:32.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:32.715 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:32.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:32.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:32.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:32.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:32.841 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:32.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:32.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:32.974 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:32.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:32.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:33.076 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:33.103 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:33.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:33.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:33.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:11:33.205 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:33.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:33.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:33.345 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:33.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:33.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:33.485 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:33.509 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:33.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:33.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:33.607 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:11:33.607 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:11:33.611 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:33.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:33.656 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:33.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:33.757 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:33.782 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:33.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:33.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:33.883 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:33.912 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:33.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:33.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:34.014 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:34.055 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:34.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:34.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:34.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:34.179 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:34.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:34.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:34.280 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:34.323 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:34.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:34.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:34.425 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:34.456 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:34.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:34.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:34.557 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:34.583 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:34.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:34.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:34.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:34 smithi098 ceph-mon[100914]: pgmap v555: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 909 op/s 2024-09-21T10:11:34.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:34 smithi098 ceph-mon[97580]: pgmap v555: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 909 op/s 2024-09-21T10:11:34.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:34 smithi196 ceph-mon[84784]: pgmap v555: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 909 op/s 2024-09-21T10:11:34.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:34.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:34.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:34.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:34.843 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:34.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:34.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:34.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:34.971 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:34.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:34.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:35.073 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:35.105 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:35.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:35.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:35.207 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:35.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:35.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:35.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:35.253 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":554,"stamp":"2024-09-21T10:11:31.191166+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3907786725,"num_objects":59584,"num_object_clones":0,"num_object_copies":178752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59584,"num_whiteouts":0,"num_read":6289,"num_read_kb":13765,"num_write":186814,"num_write_kb":7864236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":600294,"num_keys_recovered":30,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119090,"ondisk_log_size":119090,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13613748,"kb_used_data":11533308,"kb_used_omap":33,"kb_used_meta":2080286,"kb_avail":642602316,"statfs":{"total":671965249536,"available":658024771584,"internally_reserved":0,"allocated":11810107392,"data_stored":11803722661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34649,"internal_metadata":2130213031},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,11,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-573177856,"num_objects":-8746,"num_object_clones":0,"num_object_copies":-26238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8746,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002428"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":240,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.319872+0000","last_change":"2024-09-21T10:02:14.097991+0000","last_active":"2024-09-21T10:11:21.319872+0000","last_peered":"2024-09-21T10:11:21.319872+0000","last_clean":"2024-09-21T10:11:21.319872+0000","last_became_active":"2024-09-21T10:02:14.097407+0000","last_became_peered":"2024-09-21T10:02:14.097407+0000","last_unstale":"2024-09-21T10:11:21.319872+0000","last_undegraded":"2024-09-21T10:11:21.319872+0000","last_fullsized":"2024-09-21T10:11:21.319872+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:18:42.098886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":201,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.295470+0000","last_change":"2024-09-21T10:08:00.406360+0000","last_active":"2024-09-21T10:11:19.295470+0000","last_peered":"2024-09-21T10:11:19.295470+0000","last_clean":"2024-09-21T10:11:19.295470+0000","last_became_active":"2024-09-21T10:08:00.380042+0000","last_became_peered":"2024-09-21T10:08:00.380042+0000","last_unstale":"2024-09-21T10:11:19.295470+0000","last_undegraded":"2024-09-21T10:11:19.295470+0000","last_fullsized":"2024-09-21T10:11:19.295470+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:49:46.544051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":107,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.294918+0000","last_change":"2024-09-21T10:07:43.789751+0000","last_active":"2024-09-21T10:11:19.294918+0000","last_peered":"2024-09-21T10:11:19.294918+0000","last_clean":"2024-09-21T10:11:19.294918+0000","last_became_active":"2024-09-21T10:07:43.789448+0000","last_became_peered":"2024-09-21T10:07:43.789448+0000","last_unstale":"2024-09-21T10:11:19.294918+0000","last_undegraded":"2024-09-21T10:11:19.294918+0000","last_fullsized":"2024-09-21T10:11:19.294918+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:36.586942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":236,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.313061+0000","last_change":"2024-09-21T10:02:14.098949+0000","last_active":"2024-09-21T10:11:21.313061+0000","last_peered":"2024-09-21T10:11:21.313061+0000","last_clean":"2024-09-21T10:11:21.313061+0000","last_became_active":"2024-09-21T10:02:14.098646+0000","last_became_peered":"2024-09-21T10:02:14.098646+0000","last_unstale":"2024-09-21T10:11:21.313061+0000","last_undegraded":"2024-09-21T10:11:21.313061+0000","last_fullsized":"2024-09-21T10:11:21.313061+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:50:56.862529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":236,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.316558+0000","last_change":"2024-09-21T10:02:14.098205+0000","last_active":"2024-09-21T10:11:21.316558+0000","last_peered":"2024-09-21T10:11:21.316558+0000","last_clean":"2024-09-21T10:11:21.316558+0000","last_became_active":"2024-09-21T10:02:14.097911+0000","last_became_peered":"2024-09-21T10:02:14.097911+0000","last_unstale":"2024-09-21T10:11:21.316558+0000","last_undegraded":"2024-09-21T10:11:21.316558+0000","last_fullsized":"2024-09-21T10:11:21.316558+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:55.971191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":66,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.296784+0000","last_change":"2024-09-21T10:08:00.385671+0000","last_active":"2024-09-21T10:11:19.296784+0000","last_peered":"2024-09-21T10:11:19.296784+0000","last_clean":"2024-09-21T10:11:19.296784+0000","last_became_active":"2024-09-21T10:08:00.385368+0000","last_became_peered":"2024-09-21T10:08:00.385368+0000","last_unstale":"2024-09-21T10:11:19.296784+0000","last_undegraded":"2024-09-21T10:11:19.296784+0000","last_fullsized":"2024-09-21T10:11:19.296784+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:21.799149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":240,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.314234+0000","last_change":"2024-09-21T10:02:14.097817+0000","last_active":"2024-09-21T10:11:21.314234+0000","last_peered":"2024-09-21T10:11:21.314234+0000","last_clean":"2024-09-21T10:11:21.314234+0000","last_became_active":"2024-09-21T10:02:14.097466+0000","last_became_peered":"2024-09-21T10:02:14.097466+0000","last_unstale":"2024-09-21T10:11:21.314234+0000","last_undegraded":"2024-09-21T10:11:21.314234+0000","last_fullsized":"2024-09-21T10:11:21.314234+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:57.883118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":158,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.297305+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:11:19.297305+0000","last_peered":"2024-09-21T10:11:19.297305+0000","last_clean":"2024-09-21T10:11:19.297305+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:11:19.297305+0000","last_undegraded":"2024-09-21T10:11:19.297305+0000","last_fullsized":"2024-09-21T10:11:19.297305+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"186'11570","reported_seq":11599,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:28.138590+0000","last_change":"2024-09-21T10:08:40.915510+0000","last_active":"2024-09-21T10:11:28.138590+0000","last_peered":"2024-09-21T10:11:28.138590+0000","last_clean":"2024-09-21T10:11:28.138590+0000","last_became_active":"2024-09-21T10:08:40.914989+0000","last_became_peered":"2024-09-21T10:08:40.914989+0000","last_unstale":"2024-09-21T10:11:28.138590+0000","last_undegraded":"2024-09-21T10:11:28.138590+0000","last_fullsized":"2024-09-21T10:11:28.138590+0000","mapping_epoch":165,"log_start":"171'4200","ondisk_log_start":"171'4200","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":7370,"log_dups_size":0,"ondisk_log_size":7370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:36:20.075471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246284288,"num_objects":3758,"num_object_clones":0,"num_object_copies":11274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11570,"num_write_kb":490496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":194,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.292600+0000","last_change":"2024-09-21T10:08:00.392499+0000","last_active":"2024-09-21T10:11:19.292600+0000","last_peered":"2024-09-21T10:11:19.292600+0000","last_clean":"2024-09-21T10:11:19.292600+0000","last_became_active":"2024-09-21T10:08:00.379165+0000","last_became_peered":"2024-09-21T10:08:00.379165+0000","last_unstale":"2024-09-21T10:11:19.292600+0000","last_undegraded":"2024-09-21T10:11:19.292600+0000","last_fullsized":"2024-09-21T10:11:19.292600+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:18.933144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":50,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.291836+0000","last_change":"2024-09-21T10:07:43.794677+0000","last_active":"2024-09-21T10:11:19.291836+0000","last_peered":"2024-09-21T10:11:19.291836+0000","last_clean":"2024-09-21T10:11:19.291836+0000","last_became_active":"2024-09-21T10:07:43.794348+0000","last_became_peered":"2024-09-21T10:07:43.794348+0000","last_unstale":"2024-09-21T10:11:19.291836+0000","last_undegraded":"2024-09-21T10:11:19.291836+0000","last_fullsized":"2024-09-21T10:11:19.291836+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:11.159014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"186'11553","reported_seq":11605,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:27.395774+0000","last_change":"2024-09-21T10:08:40.600297+0000","last_active":"2024-09-21T10:11:27.395774+0000","last_peered":"2024-09-21T10:11:27.395774+0000","last_clean":"2024-09-21T10:11:27.395774+0000","last_became_active":"2024-09-21T10:08:40.600022+0000","last_became_peered":"2024-09-21T10:08:40.600022+0000","last_unstale":"2024-09-21T10:11:27.395774+0000","last_undegraded":"2024-09-21T10:11:27.395774+0000","last_fullsized":"2024-09-21T10:11:27.395774+0000","mapping_epoch":165,"log_start":"171'4700","ondisk_log_start":"171'4700","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":6853,"log_dups_size":0,"ondisk_log_size":6853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:39:48.049448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242679808,"num_objects":3703,"num_object_clones":0,"num_object_copies":11109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11553,"num_write_kb":488192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":195,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.292509+0000","last_change":"2024-09-21T10:08:00.412449+0000","last_active":"2024-09-21T10:11:19.292509+0000","last_peered":"2024-09-21T10:11:19.292509+0000","last_clean":"2024-09-21T10:11:19.292509+0000","last_became_active":"2024-09-21T10:08:00.386530+0000","last_became_peered":"2024-09-21T10:08:00.386530+0000","last_unstale":"2024-09-21T10:11:19.292509+0000","last_undegraded":"2024-09-21T10:11:19.292509+0000","last_fullsized":"2024-09-21T10:11:19.292509+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:12:50.581176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":162,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.310591+0000","last_change":"2024-09-21T10:08:18.475007+0000","last_active":"2024-09-21T10:11:21.310591+0000","last_peered":"2024-09-21T10:11:21.310591+0000","last_clean":"2024-09-21T10:11:21.310591+0000","last_became_active":"2024-09-21T10:08:18.474730+0000","last_became_peered":"2024-09-21T10:08:18.474730+0000","last_unstale":"2024-09-21T10:11:21.310591+0000","last_undegraded":"2024-09-21T10:11:21.310591+0000","last_fullsized":"2024-09-21T10:11:21.310591+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:35.455790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"186'11630","reported_seq":11659,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:28.239780+0000","last_change":"2024-09-21T10:08:40.598569+0000","last_active":"2024-09-21T10:11:28.239780+0000","last_peered":"2024-09-21T10:11:28.239780+0000","last_clean":"2024-09-21T10:11:28.239780+0000","last_became_active":"2024-09-21T10:08:40.598307+0000","last_became_peered":"2024-09-21T10:08:40.598307+0000","last_unstale":"2024-09-21T10:11:28.239780+0000","last_undegraded":"2024-09-21T10:11:28.239780+0000","last_fullsized":"2024-09-21T10:11:28.239780+0000","mapping_epoch":165,"log_start":"170'3700","ondisk_log_start":"170'3700","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":7930,"log_dups_size":0,"ondisk_log_size":7930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:22:30.794908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248512512,"num_objects":3792,"num_object_clones":0,"num_object_copies":11376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11630,"num_write_kb":493504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":135,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.295110+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:11:19.295110+0000","last_peered":"2024-09-21T10:11:19.295110+0000","last_clean":"2024-09-21T10:11:19.295110+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:11:19.295110+0000","last_undegraded":"2024-09-21T10:11:19.295110+0000","last_fullsized":"2024-09-21T10:11:19.295110+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":149,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.296861+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:11:19.296861+0000","last_peered":"2024-09-21T10:11:19.296861+0000","last_clean":"2024-09-21T10:11:19.296861+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:11:19.296861+0000","last_undegraded":"2024-09-21T10:11:19.296861+0000","last_fullsized":"2024-09-21T10:11:19.296861+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"186'11722","reported_seq":11776,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:27.539128+0000","last_change":"2024-09-21T10:08:40.601059+0000","last_active":"2024-09-21T10:11:27.539128+0000","last_peered":"2024-09-21T10:11:27.539128+0000","last_clean":"2024-09-21T10:11:27.539128+0000","last_became_active":"2024-09-21T10:08:40.600760+0000","last_became_peered":"2024-09-21T10:08:40.600760+0000","last_unstale":"2024-09-21T10:11:27.539128+0000","last_undegraded":"2024-09-21T10:11:27.539128+0000","last_fullsized":"2024-09-21T10:11:27.539128+0000","mapping_epoch":165,"log_start":"170'3600","ondisk_log_start":"170'3600","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":8122,"log_dups_size":0,"ondisk_log_size":8122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:52:59.796928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251396096,"num_objects":3836,"num_object_clones":0,"num_object_copies":11508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11722,"num_write_kb":497856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":146,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.320061+0000","last_change":"2024-09-21T10:07:43.793008+0000","last_active":"2024-09-21T10:11:21.320061+0000","last_peered":"2024-09-21T10:11:21.320061+0000","last_clean":"2024-09-21T10:11:21.320061+0000","last_became_active":"2024-09-21T10:07:43.792683+0000","last_became_peered":"2024-09-21T10:07:43.792683+0000","last_unstale":"2024-09-21T10:11:21.320061+0000","last_undegraded":"2024-09-21T10:11:21.320061+0000","last_fullsized":"2024-09-21T10:11:21.320061+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:46:13.137478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":258,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.314839+0000","last_change":"2024-09-21T10:08:00.389645+0000","last_active":"2024-09-21T10:11:21.314839+0000","last_peered":"2024-09-21T10:11:21.314839+0000","last_clean":"2024-09-21T10:11:21.314839+0000","last_became_active":"2024-09-21T10:08:00.386076+0000","last_became_peered":"2024-09-21T10:08:00.386076+0000","last_unstale":"2024-09-21T10:11:21.314839+0000","last_undegraded":"2024-09-21T10:11:21.314839+0000","last_fullsized":"2024-09-21T10:11:21.314839+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:58:10.646600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"186'11702","reported_seq":11755,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:30.990966+0000","last_change":"2024-09-21T10:08:40.915043+0000","last_active":"2024-09-21T10:11:30.990966+0000","last_peered":"2024-09-21T10:11:30.990966+0000","last_clean":"2024-09-21T10:11:30.990966+0000","last_became_active":"2024-09-21T10:08:40.914751+0000","last_became_peered":"2024-09-21T10:08:40.914751+0000","last_unstale":"2024-09-21T10:11:30.990966+0000","last_undegraded":"2024-09-21T10:11:30.990966+0000","last_fullsized":"2024-09-21T10:11:30.990966+0000","mapping_epoch":165,"log_start":"171'5100","ondisk_log_start":"171'5100","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":6602,"log_dups_size":0,"ondisk_log_size":6602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:19:35.778463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238813184,"num_objects":3644,"num_object_clones":0,"num_object_copies":10932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11702,"num_write_kb":491072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":272,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.318461+0000","last_change":"2024-09-21T10:08:00.395898+0000","last_active":"2024-09-21T10:11:21.318461+0000","last_peered":"2024-09-21T10:11:21.318461+0000","last_clean":"2024-09-21T10:11:21.318461+0000","last_became_active":"2024-09-21T10:08:00.390462+0000","last_became_peered":"2024-09-21T10:08:00.390462+0000","last_unstale":"2024-09-21T10:11:21.318461+0000","last_undegraded":"2024-09-21T10:11:21.318461+0000","last_fullsized":"2024-09-21T10:11:21.318461+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:29:01.877089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":159,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.320191+0000","last_change":"2024-09-21T10:08:00.383016+0000","last_active":"2024-09-21T10:11:21.320191+0000","last_peered":"2024-09-21T10:11:21.320191+0000","last_clean":"2024-09-21T10:11:21.320191+0000","last_became_active":"2024-09-21T10:08:00.382708+0000","last_became_peered":"2024-09-21T10:08:00.382708+0000","last_unstale":"2024-09-21T10:11:21.320191+0000","last_undegraded":"2024-09-21T10:11:21.320191+0000","last_fullsized":"2024-09-21T10:11:21.320191+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:30:52.797259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"186'11518","reported_seq":11572,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:29.909047+0000","last_change":"2024-09-21T10:08:40.601007+0000","last_active":"2024-09-21T10:11:29.909047+0000","last_peered":"2024-09-21T10:11:29.909047+0000","last_clean":"2024-09-21T10:11:29.909047+0000","last_became_active":"2024-09-21T10:08:40.600733+0000","last_became_peered":"2024-09-21T10:08:40.600733+0000","last_unstale":"2024-09-21T10:11:29.909047+0000","last_undegraded":"2024-09-21T10:11:29.909047+0000","last_fullsized":"2024-09-21T10:11:29.909047+0000","mapping_epoch":165,"log_start":"169'1800","ondisk_log_start":"169'1800","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":9718,"log_dups_size":0,"ondisk_log_size":9718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:31:47.984695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239861760,"num_objects":3660,"num_object_clones":0,"num_object_copies":10980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11518,"num_write_kb":485696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":180,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.296963+0000","last_change":"2024-09-21T10:08:00.379604+0000","last_active":"2024-09-21T10:11:19.296963+0000","last_peered":"2024-09-21T10:11:19.296963+0000","last_clean":"2024-09-21T10:11:19.296963+0000","last_became_active":"2024-09-21T10:08:00.379301+0000","last_became_peered":"2024-09-21T10:08:00.379301+0000","last_unstale":"2024-09-21T10:11:19.296963+0000","last_undegraded":"2024-09-21T10:11:19.296963+0000","last_fullsized":"2024-09-21T10:11:19.296963+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:32:41.963929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":170,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.310319+0000","last_change":"2024-09-21T10:08:00.387420+0000","last_active":"2024-09-21T10:11:21.310319+0000","last_peered":"2024-09-21T10:11:21.310319+0000","last_clean":"2024-09-21T10:11:21.310319+0000","last_became_active":"2024-09-21T10:08:00.386843+0000","last_became_peered":"2024-09-21T10:08:00.386843+0000","last_unstale":"2024-09-21T10:11:21.310319+0000","last_undegraded":"2024-09-21T10:11:21.310319+0000","last_fullsized":"2024-09-21T10:11:21.310319+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:51:00.654836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"186'11715","reported_seq":11769,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:27.375495+0000","last_change":"2024-09-21T10:08:40.915784+0000","last_active":"2024-09-21T10:11:27.375495+0000","last_peered":"2024-09-21T10:11:27.375495+0000","last_clean":"2024-09-21T10:11:27.375495+0000","last_became_active":"2024-09-21T10:08:40.915456+0000","last_became_peered":"2024-09-21T10:08:40.915456+0000","last_unstale":"2024-09-21T10:11:27.375495+0000","last_undegraded":"2024-09-21T10:11:27.375495+0000","last_fullsized":"2024-09-21T10:11:27.375495+0000","mapping_epoch":165,"log_start":"171'4800","ondisk_log_start":"171'4800","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":6915,"log_dups_size":0,"ondisk_log_size":6915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:24:25.107555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249102336,"num_objects":3801,"num_object_clones":0,"num_object_copies":11403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11715,"num_write_kb":496512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":154,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.311093+0000","last_change":"2024-09-21T10:08:00.382496+0000","last_active":"2024-09-21T10:11:21.311093+0000","last_peered":"2024-09-21T10:11:21.311093+0000","last_clean":"2024-09-21T10:11:21.311093+0000","last_became_active":"2024-09-21T10:08:00.382180+0000","last_became_peered":"2024-09-21T10:08:00.382180+0000","last_unstale":"2024-09-21T10:11:21.311093+0000","last_undegraded":"2024-09-21T10:11:21.311093+0000","last_fullsized":"2024-09-21T10:11:21.311093+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:16.074736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":119,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.297157+0000","last_change":"2024-09-21T10:08:38.906801+0000","last_active":"2024-09-21T10:11:19.297157+0000","last_peered":"2024-09-21T10:11:19.297157+0000","last_clean":"2024-09-21T10:11:19.297157+0000","last_became_active":"2024-09-21T10:08:38.906487+0000","last_became_peered":"2024-09-21T10:08:38.906487+0000","last_unstale":"2024-09-21T10:11:19.297157+0000","last_undegraded":"2024-09-21T10:11:19.297157+0000","last_fullsized":"2024-09-21T10:11:19.297157+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:31.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"186'11743","reported_seq":11798,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:30.947743+0000","last_change":"2024-09-21T10:08:40.915278+0000","last_active":"2024-09-21T10:11:30.947743+0000","last_peered":"2024-09-21T10:11:30.947743+0000","last_clean":"2024-09-21T10:11:30.947743+0000","last_became_active":"2024-09-21T10:08:40.914886+0000","last_became_peered":"2024-09-21T10:08:40.914886+0000","last_unstale":"2024-09-21T10:11:30.947743+0000","last_undegraded":"2024-09-21T10:11:30.947743+0000","last_fullsized":"2024-09-21T10:11:30.947743+0000","mapping_epoch":165,"log_start":"171'5200","ondisk_log_start":"171'5200","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":6543,"log_dups_size":0,"ondisk_log_size":6543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:47.589854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236060696,"num_objects":3603,"num_object_clones":0,"num_object_copies":10809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3603,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11743,"num_write_kb":491009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":242,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.317264+0000","last_change":"2024-09-21T10:02:14.097462+0000","last_active":"2024-09-21T10:11:21.317264+0000","last_peered":"2024-09-21T10:11:21.317264+0000","last_clean":"2024-09-21T10:11:21.317264+0000","last_became_active":"2024-09-21T10:02:14.097167+0000","last_became_peered":"2024-09-21T10:02:14.097167+0000","last_unstale":"2024-09-21T10:11:21.317264+0000","last_undegraded":"2024-09-21T10:11:21.317264+0000","last_fullsized":"2024-09-21T10:11:21.317264+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:54.922731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":246,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.310923+0000","last_change":"2024-09-21T10:08:00.378569+0000","last_active":"2024-09-21T10:11:21.310923+0000","last_peered":"2024-09-21T10:11:21.310923+0000","last_clean":"2024-09-21T10:11:21.310923+0000","last_became_active":"2024-09-21T10:08:00.378015+0000","last_became_peered":"2024-09-21T10:08:00.378015+0000","last_unstale":"2024-09-21T10:11:21.310923+0000","last_undegraded":"2024-09-21T10:11:21.310923+0000","last_fullsized":"2024-09-21T10:11:21.310923+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:58.121005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"186'11425","reported_seq":11479,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:27.351722+0000","last_change":"2024-09-21T10:08:40.915416+0000","last_active":"2024-09-21T10:11:27.351722+0000","last_peered":"2024-09-21T10:11:27.351722+0000","last_clean":"2024-09-21T10:11:27.351722+0000","last_became_active":"2024-09-21T10:08:40.914705+0000","last_became_peered":"2024-09-21T10:08:40.914705+0000","last_unstale":"2024-09-21T10:11:27.351722+0000","last_undegraded":"2024-09-21T10:11:27.351722+0000","last_fullsized":"2024-09-21T10:11:27.351722+0000","mapping_epoch":165,"log_start":"171'4600","ondisk_log_start":"171'4600","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":6825,"log_dups_size":0,"ondisk_log_size":6825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:34:22.667201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249102336,"num_objects":3801,"num_object_clones":0,"num_object_copies":11403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11425,"num_write_kb":487232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":268,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.292078+0000","last_change":"2024-09-21T09:59:54.265556+0000","last_active":"2024-09-21T10:11:19.292078+0000","last_peered":"2024-09-21T10:11:19.292078+0000","last_clean":"2024-09-21T10:11:19.292078+0000","last_became_active":"2024-09-21T09:59:54.265033+0000","last_became_peered":"2024-09-21T09:59:54.265033+0000","last_unstale":"2024-09-21T10:11:19.292078+0000","last_undegraded":"2024-09-21T10:11:19.292078+0000","last_fullsized":"2024-09-21T10:11:19.292078+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T09:48:34.850872+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:40.449756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":533,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.310148+0000","last_change":"2024-09-21T10:07:43.790490+0000","last_active":"2024-09-21T10:11:21.310148+0000","last_peered":"2024-09-21T10:11:21.310148+0000","last_clean":"2024-09-21T10:11:21.310148+0000","last_became_active":"2024-09-21T10:07:43.789986+0000","last_became_peered":"2024-09-21T10:07:43.789986+0000","last_unstale":"2024-09-21T10:11:21.310148+0000","last_undegraded":"2024-09-21T10:11:21.310148+0000","last_fullsized":"2024-09-21T10:11:21.310148+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:08.740356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":481,"num_read_kb":1580,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":195,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.314295+0000","last_change":"2024-09-21T10:02:14.099961+0000","last_active":"2024-09-21T10:11:21.314295+0000","last_peered":"2024-09-21T10:11:21.314295+0000","last_clean":"2024-09-21T10:11:21.314295+0000","last_became_active":"2024-09-21T10:02:14.099067+0000","last_became_peered":"2024-09-21T10:02:14.099067+0000","last_unstale":"2024-09-21T10:11:21.314295+0000","last_undegraded":"2024-09-21T10:11:21.314295+0000","last_fullsized":"2024-09-21T10:11:21.314295+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:41.037532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":162,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.297202+0000","last_change":"2024-09-21T10:08:00.386135+0000","last_active":"2024-09-21T10:11:19.297202+0000","last_peered":"2024-09-21T10:11:19.297202+0000","last_clean":"2024-09-21T10:11:19.297202+0000","last_became_active":"2024-09-21T10:08:00.385796+0000","last_became_peered":"2024-09-21T10:08:00.385796+0000","last_unstale":"2024-09-21T10:11:19.297202+0000","last_undegraded":"2024-09-21T10:11:19.297202+0000","last_fullsized":"2024-09-21T10:11:19.297202+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:27.054699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"186'11698","reported_seq":11751,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:30.988675+0000","last_change":"2024-09-21T10:08:40.916810+0000","last_active":"2024-09-21T10:11:30.988675+0000","last_peered":"2024-09-21T10:11:30.988675+0000","last_clean":"2024-09-21T10:11:30.988675+0000","last_became_active":"2024-09-21T10:08:40.916090+0000","last_became_peered":"2024-09-21T10:08:40.916090+0000","last_unstale":"2024-09-21T10:11:30.988675+0000","last_undegraded":"2024-09-21T10:11:30.988675+0000","last_fullsized":"2024-09-21T10:11:30.988675+0000","mapping_epoch":165,"log_start":"171'5100","ondisk_log_start":"171'5100","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":6598,"log_dups_size":0,"ondisk_log_size":6598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:06:45.072223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235012096,"num_objects":3586,"num_object_clones":0,"num_object_copies":10758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11698,"num_write_kb":489088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":531,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.291923+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:11:19.291923+0000","last_peered":"2024-09-21T10:11:19.291923+0000","last_clean":"2024-09-21T10:11:19.291923+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:11:19.291923+0000","last_undegraded":"2024-09-21T10:11:19.291923+0000","last_fullsized":"2024-09-21T10:11:19.291923+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":258,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.313714+0000","last_change":"2024-09-21T10:08:00.383928+0000","last_active":"2024-09-21T10:11:21.313714+0000","last_peered":"2024-09-21T10:11:21.313714+0000","last_clean":"2024-09-21T10:11:21.313714+0000","last_became_active":"2024-09-21T10:08:00.383625+0000","last_became_peered":"2024-09-21T10:08:00.383625+0000","last_unstale":"2024-09-21T10:11:21.313714+0000","last_undegraded":"2024-09-21T10:11:21.313714+0000","last_fullsized":"2024-09-21T10:11:21.313714+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:39.207283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":248,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.316754+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:11:21.316754+0000","last_peered":"2024-09-21T10:11:21.316754+0000","last_clean":"2024-09-21T10:11:21.316754+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:11:21.316754+0000","last_undegraded":"2024-09-21T10:11:21.316754+0000","last_fullsized":"2024-09-21T10:11:21.316754+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"186'11466","reported_seq":11494,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:28.239746+0000","last_change":"2024-09-21T10:08:40.597717+0000","last_active":"2024-09-21T10:11:28.239746+0000","last_peered":"2024-09-21T10:11:28.239746+0000","last_clean":"2024-09-21T10:11:28.239746+0000","last_became_active":"2024-09-21T10:08:40.597197+0000","last_became_peered":"2024-09-21T10:08:40.597197+0000","last_unstale":"2024-09-21T10:11:28.239746+0000","last_undegraded":"2024-09-21T10:11:28.239746+0000","last_fullsized":"2024-09-21T10:11:28.239746+0000","mapping_epoch":165,"log_start":"170'3500","ondisk_log_start":"170'3500","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":7966,"log_dups_size":0,"ondisk_log_size":7966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:25:03.341682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240779264,"num_objects":3674,"num_object_clones":0,"num_object_copies":11022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11466,"num_write_kb":484480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":985,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.311244+0000","last_change":"2024-09-21T10:02:14.100143+0000","last_active":"2024-09-21T10:11:21.311244+0000","last_peered":"2024-09-21T10:11:21.311244+0000","last_clean":"2024-09-21T10:11:21.311244+0000","last_became_active":"2024-09-21T10:02:14.099586+0000","last_became_peered":"2024-09-21T10:02:14.099586+0000","last_unstale":"2024-09-21T10:11:21.311244+0000","last_undegraded":"2024-09-21T10:11:21.311244+0000","last_fullsized":"2024-09-21T10:11:21.311244+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:34.064652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":120,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.309529+0000","last_change":"2024-09-21T10:08:18.470748+0000","last_active":"2024-09-21T10:11:21.309529+0000","last_peered":"2024-09-21T10:11:21.309529+0000","last_clean":"2024-09-21T10:11:21.309529+0000","last_became_active":"2024-09-21T10:08:18.470227+0000","last_became_peered":"2024-09-21T10:08:18.470227+0000","last_unstale":"2024-09-21T10:11:21.309529+0000","last_undegraded":"2024-09-21T10:11:21.309529+0000","last_fullsized":"2024-09-21T10:11:21.309529+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:16.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":242,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.311061+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:11:21.311061+0000","last_peered":"2024-09-21T10:11:21.311061+0000","last_clean":"2024-09-21T10:11:21.311061+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:11:21.311061+0000","last_undegraded":"2024-09-21T10:11:21.311061+0000","last_fullsized":"2024-09-21T10:11:21.311061+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"186'11399","reported_seq":11453,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:27.396371+0000","last_change":"2024-09-21T10:08:40.601145+0000","last_active":"2024-09-21T10:11:27.396371+0000","last_peered":"2024-09-21T10:11:27.396371+0000","last_clean":"2024-09-21T10:11:27.396371+0000","last_became_active":"2024-09-21T10:08:40.600872+0000","last_became_peered":"2024-09-21T10:08:40.600872+0000","last_unstale":"2024-09-21T10:11:27.396371+0000","last_undegraded":"2024-09-21T10:11:27.396371+0000","last_fullsized":"2024-09-21T10:11:27.396371+0000","mapping_epoch":165,"log_start":"171'4500","ondisk_log_start":"171'4500","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":6899,"log_dups_size":0,"ondisk_log_size":6899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:55:44.108331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249888768,"num_objects":3813,"num_object_clones":0,"num_object_copies":11439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11399,"num_write_kb":486784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":718,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.309357+0000","last_change":"2024-09-21T10:07:43.794767+0000","last_active":"2024-09-21T10:11:21.309357+0000","last_peered":"2024-09-21T10:11:21.309357+0000","last_clean":"2024-09-21T10:11:21.309357+0000","last_became_active":"2024-09-21T10:07:43.794062+0000","last_became_peered":"2024-09-21T10:07:43.794062+0000","last_unstale":"2024-09-21T10:11:21.309357+0000","last_undegraded":"2024-09-21T10:11:21.309357+0000","last_fullsized":"2024-09-21T10:11:21.309357+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:51.977955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":145,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.312932+0000","last_change":"2024-09-21T10:08:00.389176+0000","last_active":"2024-09-21T10:11:21.312932+0000","last_peered":"2024-09-21T10:11:21.312932+0000","last_clean":"2024-09-21T10:11:21.312932+0000","last_became_active":"2024-09-21T10:08:00.383854+0000","last_became_peered":"2024-09-21T10:08:00.383854+0000","last_unstale":"2024-09-21T10:11:21.312932+0000","last_undegraded":"2024-09-21T10:11:21.312932+0000","last_fullsized":"2024-09-21T10:11:21.312932+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:46.952216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":158,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.310348+0000","last_change":"2024-09-21T10:07:43.792153+0000","last_active":"2024-09-21T10:11:21.310348+0000","last_peered":"2024-09-21T10:11:21.310348+0000","last_clean":"2024-09-21T10:11:21.310348+0000","last_became_active":"2024-09-21T10:07:43.791770+0000","last_became_peered":"2024-09-21T10:07:43.791770+0000","last_unstale":"2024-09-21T10:11:21.310348+0000","last_undegraded":"2024-09-21T10:11:21.310348+0000","last_fullsized":"2024-09-21T10:11:21.310348+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:52.506941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"186'11490","reported_seq":11544,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:27.537051+0000","last_change":"2024-09-21T10:08:40.603257+0000","last_active":"2024-09-21T10:11:27.537051+0000","last_peered":"2024-09-21T10:11:27.537051+0000","last_clean":"2024-09-21T10:11:27.537051+0000","last_became_active":"2024-09-21T10:08:40.602870+0000","last_became_peered":"2024-09-21T10:08:40.602870+0000","last_unstale":"2024-09-21T10:11:27.537051+0000","last_undegraded":"2024-09-21T10:11:27.537051+0000","last_fullsized":"2024-09-21T10:11:27.537051+0000","mapping_epoch":165,"log_start":"170'3300","ondisk_log_start":"170'3300","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":8190,"log_dups_size":0,"ondisk_log_size":8190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:00:53.616267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245891072,"num_objects":3752,"num_object_clones":0,"num_object_copies":11256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11490,"num_write_kb":487744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":163,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.318258+0000","last_change":"2024-09-21T10:07:43.791405+0000","last_active":"2024-09-21T10:11:21.318258+0000","last_peered":"2024-09-21T10:11:21.318258+0000","last_clean":"2024-09-21T10:11:21.318258+0000","last_became_active":"2024-09-21T10:07:43.791078+0000","last_became_peered":"2024-09-21T10:07:43.791078+0000","last_unstale":"2024-09-21T10:11:21.318258+0000","last_undegraded":"2024-09-21T10:11:21.318258+0000","last_fullsized":"2024-09-21T10:11:21.318258+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:17:38.072612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":825,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.311024+0000","last_change":"2024-09-21T10:08:18.468995+0000","last_active":"2024-09-21T10:11:21.311024+0000","last_peered":"2024-09-21T10:11:21.311024+0000","last_clean":"2024-09-21T10:11:21.311024+0000","last_became_active":"2024-09-21T10:08:18.468243+0000","last_became_peered":"2024-09-21T10:08:18.468243+0000","last_unstale":"2024-09-21T10:11:21.311024+0000","last_undegraded":"2024-09-21T10:11:21.311024+0000","last_fullsized":"2024-09-21T10:11:21.311024+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:45.520968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":144,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.309850+0000","last_change":"2024-09-21T10:08:18.468595+0000","last_active":"2024-09-21T10:11:21.309850+0000","last_peered":"2024-09-21T10:11:21.309850+0000","last_clean":"2024-09-21T10:11:21.309850+0000","last_became_active":"2024-09-21T10:08:18.468296+0000","last_became_peered":"2024-09-21T10:08:18.468296+0000","last_unstale":"2024-09-21T10:11:21.309850+0000","last_undegraded":"2024-09-21T10:11:21.309850+0000","last_fullsized":"2024-09-21T10:11:21.309850+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:42.748130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"186'11888","reported_seq":11942,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:30.976134+0000","last_change":"2024-09-21T10:08:40.604276+0000","last_active":"2024-09-21T10:11:30.976134+0000","last_peered":"2024-09-21T10:11:30.976134+0000","last_clean":"2024-09-21T10:11:30.976134+0000","last_became_active":"2024-09-21T10:08:40.603906+0000","last_became_peered":"2024-09-21T10:08:40.603906+0000","last_unstale":"2024-09-21T10:11:30.976134+0000","last_undegraded":"2024-09-21T10:11:30.976134+0000","last_fullsized":"2024-09-21T10:11:30.976134+0000","mapping_epoch":165,"log_start":"171'5300","ondisk_log_start":"171'5300","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":6588,"log_dups_size":0,"ondisk_log_size":6588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:59:29.555786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239206400,"num_objects":3650,"num_object_clones":0,"num_object_copies":10950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11888,"num_write_kb":497216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":120,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.310933+0000","last_change":"2024-09-21T10:08:18.469907+0000","last_active":"2024-09-21T10:11:21.310933+0000","last_peered":"2024-09-21T10:11:21.310933+0000","last_clean":"2024-09-21T10:11:21.310933+0000","last_became_active":"2024-09-21T10:08:18.469443+0000","last_became_peered":"2024-09-21T10:08:18.469443+0000","last_unstale":"2024-09-21T10:11:21.310933+0000","last_undegraded":"2024-09-21T10:11:21.310933+0000","last_fullsized":"2024-09-21T10:11:21.310933+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:15.645803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":581,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.295089+0000","last_change":"2024-09-21T10:08:00.419641+0000","last_active":"2024-09-21T10:11:19.295089+0000","last_peered":"2024-09-21T10:11:19.295089+0000","last_clean":"2024-09-21T10:11:19.295089+0000","last_became_active":"2024-09-21T10:08:00.387050+0000","last_became_peered":"2024-09-21T10:08:00.387050+0000","last_unstale":"2024-09-21T10:11:19.295089+0000","last_undegraded":"2024-09-21T10:11:19.295089+0000","last_fullsized":"2024-09-21T10:11:19.295089+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:00:32.881619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":226,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.314789+0000","last_change":"2024-09-21T09:59:54.267798+0000","last_active":"2024-09-21T10:11:21.314789+0000","last_peered":"2024-09-21T10:11:21.314789+0000","last_clean":"2024-09-21T10:11:21.314789+0000","last_became_active":"2024-09-21T09:59:54.265615+0000","last_became_peered":"2024-09-21T09:59:54.265615+0000","last_unstale":"2024-09-21T10:11:21.314789+0000","last_undegraded":"2024-09-21T10:11:21.314789+0000","last_fullsized":"2024-09-21T10:11:21.314789+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:30.363872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"186'11575","reported_seq":11629,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:27.409376+0000","last_change":"2024-09-21T10:08:40.601929+0000","last_active":"2024-09-21T10:11:27.409376+0000","last_peered":"2024-09-21T10:11:27.409376+0000","last_clean":"2024-09-21T10:11:27.409376+0000","last_became_active":"2024-09-21T10:08:40.601656+0000","last_became_peered":"2024-09-21T10:08:40.601656+0000","last_unstale":"2024-09-21T10:11:27.409376+0000","last_undegraded":"2024-09-21T10:11:27.409376+0000","last_fullsized":"2024-09-21T10:11:27.409376+0000","mapping_epoch":165,"log_start":"171'4700","ondisk_log_start":"171'4700","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":6875,"log_dups_size":0,"ondisk_log_size":6875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:22:31.376121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247529472,"num_objects":3777,"num_object_clones":0,"num_object_copies":11331,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11575,"num_write_kb":491264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":239,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.318126+0000","last_change":"2024-09-21T10:02:14.101943+0000","last_active":"2024-09-21T10:11:21.318126+0000","last_peered":"2024-09-21T10:11:21.318126+0000","last_clean":"2024-09-21T10:11:21.318126+0000","last_became_active":"2024-09-21T10:02:14.101488+0000","last_became_peered":"2024-09-21T10:02:14.101488+0000","last_unstale":"2024-09-21T10:11:21.318126+0000","last_undegraded":"2024-09-21T10:11:21.318126+0000","last_fullsized":"2024-09-21T10:11:21.318126+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:52:27.866414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":489,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.310528+0000","last_change":"2024-09-21T10:08:18.472876+0000","last_active":"2024-09-21T10:11:21.310528+0000","last_peered":"2024-09-21T10:11:21.310528+0000","last_clean":"2024-09-21T10:11:21.310528+0000","last_became_active":"2024-09-21T10:08:18.472385+0000","last_became_peered":"2024-09-21T10:08:18.472385+0000","last_unstale":"2024-09-21T10:11:21.310528+0000","last_undegraded":"2024-09-21T10:11:21.310528+0000","last_fullsized":"2024-09-21T10:11:21.310528+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:33.109956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":57,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.310637+0000","last_change":"2024-09-21T10:08:38.797352+0000","last_active":"2024-09-21T10:11:21.310637+0000","last_peered":"2024-09-21T10:11:21.310637+0000","last_clean":"2024-09-21T10:11:21.310637+0000","last_became_active":"2024-09-21T10:08:38.796950+0000","last_became_peered":"2024-09-21T10:08:38.796950+0000","last_unstale":"2024-09-21T10:11:21.310637+0000","last_undegraded":"2024-09-21T10:11:21.310637+0000","last_fullsized":"2024-09-21T10:11:21.310637+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:02.646321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3","version":"186'11664","reported_seq":11693,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:28.194889+0000","last_change":"2024-09-21T10:08:40.598677+0000","last_active":"2024-09-21T10:11:28.194889+0000","last_peered":"2024-09-21T10:11:28.194889+0000","last_clean":"2024-09-21T10:11:28.194889+0000","last_became_active":"2024-09-21T10:08:40.598383+0000","last_became_peered":"2024-09-21T10:08:40.598383+0000","last_unstale":"2024-09-21T10:11:28.194889+0000","last_undegraded":"2024-09-21T10:11:28.194889+0000","last_fullsized":"2024-09-21T10:11:28.194889+0000","mapping_epoch":165,"log_start":"170'3700","ondisk_log_start":"170'3700","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":7964,"log_dups_size":0,"ondisk_log_size":7964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:03.675317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242876416,"num_objects":3706,"num_object_clones":0,"num_object_copies":11118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11664,"num_write_kb":491840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":274,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.316455+0000","last_change":"2024-09-21T10:08:00.390436+0000","last_active":"2024-09-21T10:11:21.316455+0000","last_peered":"2024-09-21T10:11:21.316455+0000","last_clean":"2024-09-21T10:11:21.316455+0000","last_became_active":"2024-09-21T10:08:00.381835+0000","last_became_peered":"2024-09-21T10:08:00.381835+0000","last_unstale":"2024-09-21T10:11:21.316455+0000","last_undegraded":"2024-09-21T10:11:21.316455+0000","last_fullsized":"2024-09-21T10:11:21.316455+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:01:35.697356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":412,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.297184+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:11:19.297184+0000","last_peered":"2024-09-21T10:11:19.297184+0000","last_clean":"2024-09-21T10:11:19.297184+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:11:19.297184+0000","last_undegraded":"2024-09-21T10:11:19.297184+0000","last_fullsized":"2024-09-21T10:11:19.297184+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":50,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.296750+0000","last_change":"2024-09-21T10:08:00.385774+0000","last_active":"2024-09-21T10:11:19.296750+0000","last_peered":"2024-09-21T10:11:19.296750+0000","last_clean":"2024-09-21T10:11:19.296750+0000","last_became_active":"2024-09-21T10:08:00.385445+0000","last_became_peered":"2024-09-21T10:08:00.385445+0000","last_unstale":"2024-09-21T10:11:19.296750+0000","last_undegraded":"2024-09-21T10:11:19.296750+0000","last_fullsized":"2024-09-21T10:11:19.296750+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:42.442190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":109,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.296911+0000","last_change":"2024-09-21T10:08:00.379661+0000","last_active":"2024-09-21T10:11:19.296911+0000","last_peered":"2024-09-21T10:11:19.296911+0000","last_clean":"2024-09-21T10:11:19.296911+0000","last_became_active":"2024-09-21T10:08:00.379301+0000","last_became_peered":"2024-09-21T10:08:00.379301+0000","last_unstale":"2024-09-21T10:11:19.296911+0000","last_undegraded":"2024-09-21T10:11:19.296911+0000","last_fullsized":"2024-09-21T10:11:19.296911+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:15.649523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":133,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.291994+0000","last_change":"2024-09-21T10:08:00.410731+0000","last_active":"2024-09-21T10:11:19.291994+0000","last_peered":"2024-09-21T10:11:19.291994+0000","last_clean":"2024-09-21T10:11:19.291994+0000","last_became_active":"2024-09-21T10:08:00.380324+0000","last_became_peered":"2024-09-21T10:08:00.380324+0000","last_unstale":"2024-09-21T10:11:19.291994+0000","last_undegraded":"2024-09-21T10:11:19.291994+0000","last_fullsized":"2024-09-21T10:11:19.291994+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:47:34.960221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":120,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.309700+0000","last_change":"2024-09-21T10:08:18.470378+0000","last_active":"2024-09-21T10:11:21.309700+0000","last_peered":"2024-09-21T10:11:21.309700+0000","last_clean":"2024-09-21T10:11:21.309700+0000","last_became_active":"2024-09-21T10:08:18.469482+0000","last_became_peered":"2024-09-21T10:08:18.469482+0000","last_unstale":"2024-09-21T10:11:21.309700+0000","last_undegraded":"2024-09-21T10:11:21.309700+0000","last_fullsized":"2024-09-21T10:11:21.309700+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:35.628724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":202,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.295005+0000","last_change":"2024-09-21T10:08:00.403606+0000","last_active":"2024-09-21T10:11:19.295005+0000","last_peered":"2024-09-21T10:11:19.295005+0000","last_clean":"2024-09-21T10:11:19.295005+0000","last_became_active":"2024-09-21T10:08:00.379330+0000","last_became_peered":"2024-09-21T10:08:00.379330+0000","last_unstale":"2024-09-21T10:11:19.295005+0000","last_undegraded":"2024-09-21T10:11:19.295005+0000","last_fullsized":"2024-09-21T10:11:19.295005+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:07.226027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":98,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.292400+0000","last_change":"2024-09-21T10:07:43.791830+0000","last_active":"2024-09-21T10:11:19.292400+0000","last_peered":"2024-09-21T10:11:19.292400+0000","last_clean":"2024-09-21T10:11:19.292400+0000","last_became_active":"2024-09-21T10:07:43.791314+0000","last_became_peered":"2024-09-21T10:07:43.791314+0000","last_unstale":"2024-09-21T10:11:19.292400+0000","last_undegraded":"2024-09-21T10:11:19.292400+0000","last_fullsized":"2024-09-21T10:11:19.292400+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:37.086629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":195,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.316362+0000","last_change":"2024-09-21T10:02:14.100451+0000","last_active":"2024-09-21T10:11:21.316362+0000","last_peered":"2024-09-21T10:11:21.316362+0000","last_clean":"2024-09-21T10:11:21.316362+0000","last_became_active":"2024-09-21T10:02:14.100160+0000","last_became_peered":"2024-09-21T10:02:14.100160+0000","last_unstale":"2024-09-21T10:11:21.316362+0000","last_undegraded":"2024-09-21T10:11:21.316362+0000","last_fullsized":"2024-09-21T10:11:21.316362+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:28.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":236,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.316917+0000","last_change":"2024-09-21T10:02:14.096098+0000","last_active":"2024-09-21T10:11:21.316917+0000","last_peered":"2024-09-21T10:11:21.316917+0000","last_clean":"2024-09-21T10:11:21.316917+0000","last_became_active":"2024-09-21T10:02:14.095811+0000","last_became_peered":"2024-09-21T10:02:14.095811+0000","last_unstale":"2024-09-21T10:11:21.316917+0000","last_undegraded":"2024-09-21T10:11:21.316917+0000","last_fullsized":"2024-09-21T10:11:21.316917+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:54.619333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":240,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.314043+0000","last_change":"2024-09-21T10:02:14.100712+0000","last_active":"2024-09-21T10:11:21.314043+0000","last_peered":"2024-09-21T10:11:21.314043+0000","last_clean":"2024-09-21T10:11:21.314043+0000","last_became_active":"2024-09-21T10:02:14.100427+0000","last_became_peered":"2024-09-21T10:02:14.100427+0000","last_unstale":"2024-09-21T10:11:21.314043+0000","last_undegraded":"2024-09-21T10:11:21.314043+0000","last_fullsized":"2024-09-21T10:11:21.314043+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:16:42.350642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":149,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.297129+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:11:19.297129+0000","last_peered":"2024-09-21T10:11:19.297129+0000","last_clean":"2024-09-21T10:11:19.297129+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:11:19.297129+0000","last_undegraded":"2024-09-21T10:11:19.297129+0000","last_fullsized":"2024-09-21T10:11:19.297129+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":258,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.314099+0000","last_change":"2024-09-21T10:02:14.097369+0000","last_active":"2024-09-21T10:11:21.314099+0000","last_peered":"2024-09-21T10:11:21.314099+0000","last_clean":"2024-09-21T10:11:21.314099+0000","last_became_active":"2024-09-21T10:02:14.097067+0000","last_became_peered":"2024-09-21T10:02:14.097067+0000","last_unstale":"2024-09-21T10:11:21.314099+0000","last_undegraded":"2024-09-21T10:11:21.314099+0000","last_fullsized":"2024-09-21T10:11:21.314099+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:50:48.123813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":237,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.316684+0000","last_change":"2024-09-21T10:02:14.098970+0000","last_active":"2024-09-21T10:11:21.316684+0000","last_peered":"2024-09-21T10:11:21.316684+0000","last_clean":"2024-09-21T10:11:21.316684+0000","last_became_active":"2024-09-21T10:02:14.098641+0000","last_became_peered":"2024-09-21T10:02:14.098641+0000","last_unstale":"2024-09-21T10:11:21.316684+0000","last_undegraded":"2024-09-21T10:11:21.316684+0000","last_fullsized":"2024-09-21T10:11:21.316684+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:25.215602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":246,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.310765+0000","last_change":"2024-09-21T10:08:00.381058+0000","last_active":"2024-09-21T10:11:21.310765+0000","last_peered":"2024-09-21T10:11:21.310765+0000","last_clean":"2024-09-21T10:11:21.310765+0000","last_became_active":"2024-09-21T10:08:00.380573+0000","last_became_peered":"2024-09-21T10:08:00.380573+0000","last_unstale":"2024-09-21T10:11:21.310765+0000","last_undegraded":"2024-09-21T10:11:21.310765+0000","last_fullsized":"2024-09-21T10:11:21.310765+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:01:04.137425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":261,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.317843+0000","last_change":"2024-09-21T10:08:00.387321+0000","last_active":"2024-09-21T10:11:21.317843+0000","last_peered":"2024-09-21T10:11:21.317843+0000","last_clean":"2024-09-21T10:11:21.317843+0000","last_became_active":"2024-09-21T10:08:00.386935+0000","last_became_peered":"2024-09-21T10:08:00.386935+0000","last_unstale":"2024-09-21T10:11:21.317843+0000","last_undegraded":"2024-09-21T10:11:21.317843+0000","last_fullsized":"2024-09-21T10:11:21.317843+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:57.706313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":66,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.309935+0000","last_change":"2024-09-21T10:08:18.471282+0000","last_active":"2024-09-21T10:11:21.309935+0000","last_peered":"2024-09-21T10:11:21.309935+0000","last_clean":"2024-09-21T10:11:21.309935+0000","last_became_active":"2024-09-21T10:08:18.470952+0000","last_became_peered":"2024-09-21T10:08:18.470952+0000","last_unstale":"2024-09-21T10:11:21.309935+0000","last_undegraded":"2024-09-21T10:11:21.309935+0000","last_fullsized":"2024-09-21T10:11:21.309935+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:25.597081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":259,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.310266+0000","last_change":"2024-09-21T10:02:14.102826+0000","last_active":"2024-09-21T10:11:21.310266+0000","last_peered":"2024-09-21T10:11:21.310266+0000","last_clean":"2024-09-21T10:11:21.310266+0000","last_became_active":"2024-09-21T10:02:14.102499+0000","last_became_peered":"2024-09-21T10:02:14.102499+0000","last_unstale":"2024-09-21T10:11:21.310266+0000","last_undegraded":"2024-09-21T10:11:21.310266+0000","last_fullsized":"2024-09-21T10:11:21.310266+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:30.924687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":250,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.314638+0000","last_change":"2024-09-21T10:08:00.386104+0000","last_active":"2024-09-21T10:11:21.314638+0000","last_peered":"2024-09-21T10:11:21.314638+0000","last_clean":"2024-09-21T10:11:21.314638+0000","last_became_active":"2024-09-21T10:08:00.381484+0000","last_became_peered":"2024-09-21T10:08:00.381484+0000","last_unstale":"2024-09-21T10:11:21.314638+0000","last_undegraded":"2024-09-21T10:11:21.314638+0000","last_fullsized":"2024-09-21T10:11:21.314638+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:39.190191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":175,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.314916+0000","last_change":"2024-09-21T10:08:00.381149+0000","last_active":"2024-09-21T10:11:21.314916+0000","last_peered":"2024-09-21T10:11:21.314916+0000","last_clean":"2024-09-21T10:11:21.314916+0000","last_became_active":"2024-09-21T10:08:00.380795+0000","last_became_peered":"2024-09-21T10:08:00.380795+0000","last_unstale":"2024-09-21T10:11:21.314916+0000","last_undegraded":"2024-09-21T10:11:21.314916+0000","last_fullsized":"2024-09-21T10:11:21.314916+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:50:46.793182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":77,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.311973+0000","last_change":"2024-09-21T10:08:18.466882+0000","last_active":"2024-09-21T10:11:21.311973+0000","last_peered":"2024-09-21T10:11:21.311973+0000","last_clean":"2024-09-21T10:11:21.311973+0000","last_became_active":"2024-09-21T10:08:18.466470+0000","last_became_peered":"2024-09-21T10:08:18.466470+0000","last_unstale":"2024-09-21T10:11:21.311973+0000","last_undegraded":"2024-09-21T10:11:21.311973+0000","last_fullsized":"2024-09-21T10:11:21.311973+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:51:31.020051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":175,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.318131+0000","last_change":"2024-09-21T10:08:00.389061+0000","last_active":"2024-09-21T10:11:21.318131+0000","last_peered":"2024-09-21T10:11:21.318131+0000","last_clean":"2024-09-21T10:11:21.318131+0000","last_became_active":"2024-09-21T10:08:00.388657+0000","last_became_peered":"2024-09-21T10:08:00.388657+0000","last_unstale":"2024-09-21T10:11:21.318131+0000","last_undegraded":"2024-09-21T10:11:21.318131+0000","last_fullsized":"2024-09-21T10:11:21.318131+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:58:17.239589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":61,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.297124+0000","last_change":"2024-09-21T10:08:00.394350+0000","last_active":"2024-09-21T10:11:19.297124+0000","last_peered":"2024-09-21T10:11:19.297124+0000","last_clean":"2024-09-21T10:11:19.297124+0000","last_became_active":"2024-09-21T10:08:00.387232+0000","last_became_peered":"2024-09-21T10:08:00.387232+0000","last_unstale":"2024-09-21T10:11:19.297124+0000","last_undegraded":"2024-09-21T10:11:19.297124+0000","last_fullsized":"2024-09-21T10:11:19.297124+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:56.160018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":261,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.316630+0000","last_change":"2024-09-21T10:02:14.099799+0000","last_active":"2024-09-21T10:11:21.316630+0000","last_peered":"2024-09-21T10:11:21.316630+0000","last_clean":"2024-09-21T10:11:21.316630+0000","last_became_active":"2024-09-21T10:02:14.099375+0000","last_became_peered":"2024-09-21T10:02:14.099375+0000","last_unstale":"2024-09-21T10:11:21.316630+0000","last_undegraded":"2024-09-21T10:11:21.316630+0000","last_fullsized":"2024-09-21T10:11:21.316630+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:13.407638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":247,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.317576+0000","last_change":"2024-09-21T10:02:14.099439+0000","last_active":"2024-09-21T10:11:21.317576+0000","last_peered":"2024-09-21T10:11:21.317576+0000","last_clean":"2024-09-21T10:11:21.317576+0000","last_became_active":"2024-09-21T10:02:14.099117+0000","last_became_peered":"2024-09-21T10:02:14.099117+0000","last_unstale":"2024-09-21T10:11:21.317576+0000","last_undegraded":"2024-09-21T10:11:21.317576+0000","last_fullsized":"2024-09-21T10:11:21.317576+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:39.681817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":124,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.296549+0000","last_change":"2024-09-21T10:08:00.381529+0000","last_active":"2024-09-21T10:11:19.296549+0000","last_peered":"2024-09-21T10:11:19.296549+0000","last_clean":"2024-09-21T10:11:19.296549+0000","last_became_active":"2024-09-21T10:08:00.381061+0000","last_became_peered":"2024-09-21T10:08:00.381061+0000","last_unstale":"2024-09-21T10:11:19.296549+0000","last_undegraded":"2024-09-21T10:11:19.296549+0000","last_fullsized":"2024-09-21T10:11:19.296549+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:56:58.775586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":241,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:21.316836+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:11:21.316836+0000","last_peered":"2024-09-21T10:11:21.316836+0000","last_clean":"2024-09-21T10:11:21.316836+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:11:21.316836+0000","last_undegraded":"2024-09-21T10:11:21.316836+0000","last_fullsized":"2024-09-21T10:11:21.316836+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":3902996504,"num_objects":59556,"num_object_clones":0,"num_object_copies":178668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59556,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185758,"num_write_kb":7849985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11705004032,"data_stored":11704991816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117958,"ondisk_log_size":117958,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":9926,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":21480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5982,"num_read_kb":13005,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397760,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1768324,"kb_used_data":1458492,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91976828,"statfs":{"total":95995035648,"available":94184271872,"internally_reserved":0,"allocated":1493495808,"data_stored":1492571497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593988,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2223808,"kb_used_data":1918136,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91521344,"statfs":{"total":95995035648,"available":93717856256,"internally_reserved":0,"allocated":1964171264,"data_stored":1963250956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790218,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1761712,"kb_used_data":1451880,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91983440,"statfs":{"total":95995035648,"available":94191042560,"internally_reserved":0,"allocated":1486725120,"data_stored":1485816153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986447,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":101,"seq":433791697009,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2700328,"kb_used_data":2443808,"kb_used_omap":5,"kb_used_meta":256506,"kb_avail":91044824,"statfs":{"total":95995035648,"available":93229899776,"internally_reserved":0,"allocated":2502459392,"data_stored":2501558351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":262662928},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860536,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2199096,"kb_used_data":1888048,"kb_used_omap":8,"kb_used_meta":311031,"kb_avail":91546056,"statfs":{"total":95995035648,"available":93743161344,"internally_reserved":0,"allocated":1933361152,"data_stored":1932448993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":318496620},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024063,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1657768,"kb_used_data":1387296,"kb_used_omap":7,"kb_used_meta":270456,"kb_avail":92087384,"statfs":{"total":95995035648,"available":94297481216,"internally_reserved":0,"allocated":1420591104,"data_stored":1419689416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":276947506},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187597,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1302712,"kb_used_data":985648,"kb_used_omap":8,"kb_used_meta":316983,"kb_avail":92442440,"statfs":{"total":95995035648,"available":94661058560,"internally_reserved":0,"allocated":1009303552,"data_stored":1008387295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":324591601},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":995364864,"data_stored":995360792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1412960256,"data_stored":1412956184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1921777664,"data_stored":1921777664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2475294720,"data_stored":2475294720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1472598016,"data_stored":1472593944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1960116224,"data_stored":1960116224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1466892288,"data_stored":1466892288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:11:35.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:11:35.333 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:35.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:35.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:35.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:35.499 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:35.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:35.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:35.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:35.641 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:35.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:35.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:35.743 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:35.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:35.788 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:35.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:35.889 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:35.912 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:35.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:35.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:36.014 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:36.038 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:36.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:36.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:36.140 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:36.162 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:36.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:36.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:36.263 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:36.288 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:36.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:36.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:36.390 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:36.435 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:36.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:36.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:36.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:36.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:36 smithi098 ceph-mon[97580]: pgmap v556: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 862 op/s 2024-09-21T10:11:36.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:36 smithi098 ceph-mon[100914]: pgmap v556: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 862 op/s 2024-09-21T10:11:36.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:11:36.598 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:36.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:36.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:36.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:36 smithi196 ceph-mon[84784]: pgmap v556: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 862 op/s 2024-09-21T10:11:36.646 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:36.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:36.750 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:36.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:36.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:36.852 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:36.878 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:36.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:36.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:36.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:37.003 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:37.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:37.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:37.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:37.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:37.150 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:37.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:37.202 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:37.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:37.276 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:37.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:37.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:37.377 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:37.403 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:37.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:37.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:37.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:37.550 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:37.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:37.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:37.580 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:37.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:37.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:37.652 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:37.678 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:37.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:37.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:37.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:37.804 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:37.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:37.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:37.906 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:37.932 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:37.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:37.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:38.034 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:38.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:38.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:38.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:38.204 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:38.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:38.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:38.305 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:38.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:38.333 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:38.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:38.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:38.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:38.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:38.569 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:38.595 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:38.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:38.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:38.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:38 smithi098 ceph-mon[97580]: pgmap v557: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 848 op/s 2024-09-21T10:11:38.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:38 smithi098 ceph-mon[100914]: pgmap v557: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 848 op/s 2024-09-21T10:11:38.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:38 smithi196 ceph-mon[84784]: pgmap v557: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 848 op/s 2024-09-21T10:11:38.698 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:38.723 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:38.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:38.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:38.825 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:38.854 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:38.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:38.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:38.955 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:38.994 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:38.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:38.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:39.096 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:39.122 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:39.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:39.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:39.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:39.263 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:39.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:39.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:39.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:39.383 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:11:39.419 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:39.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:39.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:39.521 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:39.545 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:39.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:39.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:39.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:39 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/925619843' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:11:39.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:39 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/925619843' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:11:39.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:39 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/925619843' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:11:39.648 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:39.650 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:39.673 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:39.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:39.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:39.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:39.813 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:39.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:39.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:39.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:39.938 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:39.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:39.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:40.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:40.082 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:40.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:40.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:40.183 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:40.209 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:40.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:40.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:40.311 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:40.351 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:40.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:40.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:40.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:40.488 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:40.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:40.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:40.590 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:40 smithi098 ceph-mon[97580]: pgmap v558: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 989 op/s 2024-09-21T10:11:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:40 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:11:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:40 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/925619843' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:11:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:40 smithi098 ceph-mon[97580]: osdmap e187: 8 total, 8 up, 7 in 2024-09-21T10:11:40.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:40 smithi098 ceph-mon[100914]: pgmap v558: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 989 op/s 2024-09-21T10:11:40.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:40 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:11:40.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:40 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/925619843' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:11:40.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:40 smithi098 ceph-mon[100914]: osdmap e187: 8 total, 8 up, 7 in 2024-09-21T10:11:40.625 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:40.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:40.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:40.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:40 smithi196 ceph-mon[84784]: pgmap v558: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 989 op/s 2024-09-21T10:11:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:40 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:11:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:40 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/925619843' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:11:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:40 smithi196 ceph-mon[84784]: osdmap e187: 8 total, 8 up, 7 in 2024-09-21T10:11:40.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:11:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:11:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:11:40.727 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:40.773 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:40.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:40.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:40.875 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:40.897 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:40.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:40.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:40.999 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:41.028 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:41.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:41.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:41.034 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:11:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:11:40] "GET /metrics HTTP/1.1" 200 36452 "" "Prometheus/2.51.0" 2024-09-21T10:11:41.130 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:41.156 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:41.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:41.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:41.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:41.295 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:41.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:41.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:41.397 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:41.428 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:41.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:41.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:41.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:41.564 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:41.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:41.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:41.667 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:41.698 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:41.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:41.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:41.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:41.835 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:41.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:41.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:41.937 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:41.962 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:41.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:41.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:42.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:42.104 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:42.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:42.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:42.206 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:42.232 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:42.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:42.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:42.334 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:42.376 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:42.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:42.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:42.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:42.527 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:42.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:42.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:42.629 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:42.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:42 smithi196 ceph-mon[84784]: pgmap v560: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 875 op/s 2024-09-21T10:11:42.667 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:42.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:42.769 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:42.775 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:42 smithi098 ceph-mon[97580]: pgmap v560: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 875 op/s 2024-09-21T10:11:42.775 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:42 smithi098 ceph-mon[100914]: pgmap v560: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 875 op/s 2024-09-21T10:11:42.792 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:42.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:42.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:42.893 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:42.922 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:42.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:42.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:43.024 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:43.050 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:43.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:43.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:43.152 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:43.175 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:43.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:43.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:43.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:43.324 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:43.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:43.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:43.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:43.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:43.469 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:43.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:43.570 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:11:43.571 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:11:43.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:43.615 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:43.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:43.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:43.717 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:43.742 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:43.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:43.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:43.844 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:43.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:43.881 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:43.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:43.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:44.019 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:44.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:44.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:44.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:44.162 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:44.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:44.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:44.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:44.305 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:44.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:44.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:44.407 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:44.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:44.446 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:44.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:44.465 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":561,"stamp":"2024-09-21T10:11:43.203383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3199932389,"num_objects":48783,"num_object_clones":0,"num_object_copies":146349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48783,"num_whiteouts":0,"num_read":6289,"num_read_kb":13765,"num_write":197615,"num_write_kb":7864236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":600294,"num_keys_recovered":30,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118991,"ondisk_log_size":118991,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":11703532,"kb_used_data":9607148,"kb_used_omap":33,"kb_used_meta":2096286,"kb_avail":644512532,"statfs":{"total":671965249536,"available":659980832768,"internally_reserved":0,"allocated":9837719552,"data_stored":9831319237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34649,"internal_metadata":2146597031},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-572260352,"num_objects":-8732,"num_object_clones":0,"num_object_copies":-26196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8732,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.010113"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":243,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.405479+0000","last_change":"2024-09-21T10:02:14.097991+0000","last_active":"2024-09-21T10:11:39.405479+0000","last_peered":"2024-09-21T10:11:39.405479+0000","last_clean":"2024-09-21T10:11:39.405479+0000","last_became_active":"2024-09-21T10:02:14.097407+0000","last_became_peered":"2024-09-21T10:02:14.097407+0000","last_unstale":"2024-09-21T10:11:39.405479+0000","last_undegraded":"2024-09-21T10:11:39.405479+0000","last_fullsized":"2024-09-21T10:11:39.405479+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:18:42.098886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":201,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.295470+0000","last_change":"2024-09-21T10:08:00.406360+0000","last_active":"2024-09-21T10:11:19.295470+0000","last_peered":"2024-09-21T10:11:19.295470+0000","last_clean":"2024-09-21T10:11:19.295470+0000","last_became_active":"2024-09-21T10:08:00.380042+0000","last_became_peered":"2024-09-21T10:08:00.380042+0000","last_unstale":"2024-09-21T10:11:19.295470+0000","last_undegraded":"2024-09-21T10:11:19.295470+0000","last_fullsized":"2024-09-21T10:11:19.295470+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:49:46.544051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":107,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.294918+0000","last_change":"2024-09-21T10:07:43.789751+0000","last_active":"2024-09-21T10:11:19.294918+0000","last_peered":"2024-09-21T10:11:19.294918+0000","last_clean":"2024-09-21T10:11:19.294918+0000","last_became_active":"2024-09-21T10:07:43.789448+0000","last_became_peered":"2024-09-21T10:07:43.789448+0000","last_unstale":"2024-09-21T10:11:19.294918+0000","last_undegraded":"2024-09-21T10:11:19.294918+0000","last_fullsized":"2024-09-21T10:11:19.294918+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:36.586942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":239,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.407669+0000","last_change":"2024-09-21T10:02:14.098949+0000","last_active":"2024-09-21T10:11:39.407669+0000","last_peered":"2024-09-21T10:11:39.407669+0000","last_clean":"2024-09-21T10:11:39.407669+0000","last_became_active":"2024-09-21T10:02:14.098646+0000","last_became_peered":"2024-09-21T10:02:14.098646+0000","last_unstale":"2024-09-21T10:11:39.407669+0000","last_undegraded":"2024-09-21T10:11:39.407669+0000","last_fullsized":"2024-09-21T10:11:39.407669+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:50:56.862529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":238,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.408277+0000","last_change":"2024-09-21T10:02:14.098205+0000","last_active":"2024-09-21T10:11:39.408277+0000","last_peered":"2024-09-21T10:11:39.408277+0000","last_clean":"2024-09-21T10:11:39.408277+0000","last_became_active":"2024-09-21T10:02:14.097911+0000","last_became_peered":"2024-09-21T10:02:14.097911+0000","last_unstale":"2024-09-21T10:11:39.408277+0000","last_undegraded":"2024-09-21T10:11:39.408277+0000","last_fullsized":"2024-09-21T10:11:39.408277+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:55.971191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":66,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.296784+0000","last_change":"2024-09-21T10:08:00.385671+0000","last_active":"2024-09-21T10:11:19.296784+0000","last_peered":"2024-09-21T10:11:19.296784+0000","last_clean":"2024-09-21T10:11:19.296784+0000","last_became_active":"2024-09-21T10:08:00.385368+0000","last_became_peered":"2024-09-21T10:08:00.385368+0000","last_unstale":"2024-09-21T10:11:19.296784+0000","last_undegraded":"2024-09-21T10:11:19.296784+0000","last_fullsized":"2024-09-21T10:11:19.296784+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:21.799149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":242,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.403691+0000","last_change":"2024-09-21T10:02:14.097817+0000","last_active":"2024-09-21T10:11:39.403691+0000","last_peered":"2024-09-21T10:11:39.403691+0000","last_clean":"2024-09-21T10:11:39.403691+0000","last_became_active":"2024-09-21T10:02:14.097466+0000","last_became_peered":"2024-09-21T10:02:14.097466+0000","last_unstale":"2024-09-21T10:11:39.403691+0000","last_undegraded":"2024-09-21T10:11:39.403691+0000","last_fullsized":"2024-09-21T10:11:39.403691+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:57.883118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":158,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.297305+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:11:19.297305+0000","last_peered":"2024-09-21T10:11:19.297305+0000","last_clean":"2024-09-21T10:11:19.297305+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:11:19.297305+0000","last_undegraded":"2024-09-21T10:11:19.297305+0000","last_fullsized":"2024-09-21T10:11:19.297305+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"186'12103","reported_seq":12132,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:38.153740+0000","last_change":"2024-09-21T10:08:40.915510+0000","last_active":"2024-09-21T10:11:38.153740+0000","last_peered":"2024-09-21T10:11:38.153740+0000","last_clean":"2024-09-21T10:11:38.153740+0000","last_became_active":"2024-09-21T10:08:40.914989+0000","last_became_peered":"2024-09-21T10:08:40.914989+0000","last_unstale":"2024-09-21T10:11:38.153740+0000","last_undegraded":"2024-09-21T10:11:38.153740+0000","last_fullsized":"2024-09-21T10:11:38.153740+0000","mapping_epoch":165,"log_start":"171'4700","ondisk_log_start":"171'4700","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":7403,"log_dups_size":0,"ondisk_log_size":7403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:36:20.075471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211353600,"num_objects":3225,"num_object_clones":0,"num_object_copies":9675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12103,"num_write_kb":490496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":194,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.292600+0000","last_change":"2024-09-21T10:08:00.392499+0000","last_active":"2024-09-21T10:11:19.292600+0000","last_peered":"2024-09-21T10:11:19.292600+0000","last_clean":"2024-09-21T10:11:19.292600+0000","last_became_active":"2024-09-21T10:08:00.379165+0000","last_became_peered":"2024-09-21T10:08:00.379165+0000","last_unstale":"2024-09-21T10:11:19.292600+0000","last_undegraded":"2024-09-21T10:11:19.292600+0000","last_fullsized":"2024-09-21T10:11:19.292600+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:18.933144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":50,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.291836+0000","last_change":"2024-09-21T10:07:43.794677+0000","last_active":"2024-09-21T10:11:19.291836+0000","last_peered":"2024-09-21T10:11:19.291836+0000","last_clean":"2024-09-21T10:11:19.291836+0000","last_became_active":"2024-09-21T10:07:43.794348+0000","last_became_peered":"2024-09-21T10:07:43.794348+0000","last_unstale":"2024-09-21T10:11:19.291836+0000","last_undegraded":"2024-09-21T10:11:19.291836+0000","last_fullsized":"2024-09-21T10:11:19.291836+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:11.159014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"187'12406","reported_seq":12462,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:42.413651+0000","last_change":"2024-09-21T10:08:40.600297+0000","last_active":"2024-09-21T10:11:42.413651+0000","last_peered":"2024-09-21T10:11:42.413651+0000","last_clean":"2024-09-21T10:11:42.413651+0000","last_became_active":"2024-09-21T10:08:40.600022+0000","last_became_peered":"2024-09-21T10:08:40.600022+0000","last_unstale":"2024-09-21T10:11:42.413651+0000","last_undegraded":"2024-09-21T10:11:42.413651+0000","last_fullsized":"2024-09-21T10:11:42.413651+0000","mapping_epoch":165,"log_start":"172'5500","ondisk_log_start":"172'5500","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":6906,"log_dups_size":0,"ondisk_log_size":6906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:39:48.049448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186777600,"num_objects":2850,"num_object_clones":0,"num_object_copies":8550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12406,"num_write_kb":488192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":195,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.292509+0000","last_change":"2024-09-21T10:08:00.412449+0000","last_active":"2024-09-21T10:11:19.292509+0000","last_peered":"2024-09-21T10:11:19.292509+0000","last_clean":"2024-09-21T10:11:19.292509+0000","last_became_active":"2024-09-21T10:08:00.386530+0000","last_became_peered":"2024-09-21T10:08:00.386530+0000","last_unstale":"2024-09-21T10:11:19.292509+0000","last_undegraded":"2024-09-21T10:11:19.292509+0000","last_fullsized":"2024-09-21T10:11:19.292509+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:12:50.581176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":164,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.404543+0000","last_change":"2024-09-21T10:08:18.475007+0000","last_active":"2024-09-21T10:11:39.404543+0000","last_peered":"2024-09-21T10:11:39.404543+0000","last_clean":"2024-09-21T10:11:39.404543+0000","last_became_active":"2024-09-21T10:08:18.474730+0000","last_became_peered":"2024-09-21T10:08:18.474730+0000","last_unstale":"2024-09-21T10:11:39.404543+0000","last_undegraded":"2024-09-21T10:11:39.404543+0000","last_fullsized":"2024-09-21T10:11:39.404543+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:35.455790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"186'12180","reported_seq":12209,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:38.167008+0000","last_change":"2024-09-21T10:08:40.598569+0000","last_active":"2024-09-21T10:11:38.167008+0000","last_peered":"2024-09-21T10:11:38.167008+0000","last_clean":"2024-09-21T10:11:38.167008+0000","last_became_active":"2024-09-21T10:08:40.598307+0000","last_became_peered":"2024-09-21T10:08:40.598307+0000","last_unstale":"2024-09-21T10:11:38.167008+0000","last_undegraded":"2024-09-21T10:11:38.167008+0000","last_fullsized":"2024-09-21T10:11:38.167008+0000","mapping_epoch":165,"log_start":"171'4200","ondisk_log_start":"171'4200","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":7980,"log_dups_size":0,"ondisk_log_size":7980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:22:30.794908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212467712,"num_objects":3242,"num_object_clones":0,"num_object_copies":9726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12180,"num_write_kb":493504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":135,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.295110+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:11:19.295110+0000","last_peered":"2024-09-21T10:11:19.295110+0000","last_clean":"2024-09-21T10:11:19.295110+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:11:19.295110+0000","last_undegraded":"2024-09-21T10:11:19.295110+0000","last_fullsized":"2024-09-21T10:11:19.295110+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":149,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.296861+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:11:19.296861+0000","last_peered":"2024-09-21T10:11:19.296861+0000","last_clean":"2024-09-21T10:11:19.296861+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:11:19.296861+0000","last_undegraded":"2024-09-21T10:11:19.296861+0000","last_fullsized":"2024-09-21T10:11:19.296861+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"187'12606","reported_seq":12662,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:42.518661+0000","last_change":"2024-09-21T10:08:40.601059+0000","last_active":"2024-09-21T10:11:42.518661+0000","last_peered":"2024-09-21T10:11:42.518661+0000","last_clean":"2024-09-21T10:11:42.518661+0000","last_became_active":"2024-09-21T10:08:40.600760+0000","last_became_peered":"2024-09-21T10:08:40.600760+0000","last_unstale":"2024-09-21T10:11:42.518661+0000","last_undegraded":"2024-09-21T10:11:42.518661+0000","last_fullsized":"2024-09-21T10:11:42.518661+0000","mapping_epoch":165,"log_start":"171'4400","ondisk_log_start":"171'4400","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":8206,"log_dups_size":0,"ondisk_log_size":8206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:52:59.796928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193462272,"num_objects":2952,"num_object_clones":0,"num_object_copies":8856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12606,"num_write_kb":497856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":148,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.405680+0000","last_change":"2024-09-21T10:07:43.793008+0000","last_active":"2024-09-21T10:11:39.405680+0000","last_peered":"2024-09-21T10:11:39.405680+0000","last_clean":"2024-09-21T10:11:39.405680+0000","last_became_active":"2024-09-21T10:07:43.792683+0000","last_became_peered":"2024-09-21T10:07:43.792683+0000","last_unstale":"2024-09-21T10:11:39.405680+0000","last_undegraded":"2024-09-21T10:11:39.405680+0000","last_fullsized":"2024-09-21T10:11:39.405680+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:46:13.137478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":260,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.408013+0000","last_change":"2024-09-21T10:08:00.389645+0000","last_active":"2024-09-21T10:11:39.408013+0000","last_peered":"2024-09-21T10:11:39.408013+0000","last_clean":"2024-09-21T10:11:39.408013+0000","last_became_active":"2024-09-21T10:08:00.386076+0000","last_became_peered":"2024-09-21T10:08:00.386076+0000","last_unstale":"2024-09-21T10:11:39.408013+0000","last_undegraded":"2024-09-21T10:11:39.408013+0000","last_fullsized":"2024-09-21T10:11:39.408013+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:58:10.646600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"187'12243","reported_seq":12299,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:40.984066+0000","last_change":"2024-09-21T10:08:40.915043+0000","last_active":"2024-09-21T10:11:40.984066+0000","last_peered":"2024-09-21T10:11:40.984066+0000","last_clean":"2024-09-21T10:11:40.984066+0000","last_became_active":"2024-09-21T10:08:40.914751+0000","last_became_peered":"2024-09-21T10:08:40.914751+0000","last_unstale":"2024-09-21T10:11:40.984066+0000","last_undegraded":"2024-09-21T10:11:40.984066+0000","last_fullsized":"2024-09-21T10:11:40.984066+0000","mapping_epoch":165,"log_start":"172'5700","ondisk_log_start":"172'5700","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":6543,"log_dups_size":0,"ondisk_log_size":6543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:19:35.778463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203358208,"num_objects":3103,"num_object_clones":0,"num_object_copies":9309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12243,"num_write_kb":491072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":274,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.402993+0000","last_change":"2024-09-21T10:08:00.395898+0000","last_active":"2024-09-21T10:11:39.402993+0000","last_peered":"2024-09-21T10:11:39.402993+0000","last_clean":"2024-09-21T10:11:39.402993+0000","last_became_active":"2024-09-21T10:08:00.390462+0000","last_became_peered":"2024-09-21T10:08:00.390462+0000","last_unstale":"2024-09-21T10:11:39.402993+0000","last_undegraded":"2024-09-21T10:11:39.402993+0000","last_fullsized":"2024-09-21T10:11:39.402993+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:29:01.877089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":161,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.405777+0000","last_change":"2024-09-21T10:08:00.383016+0000","last_active":"2024-09-21T10:11:39.405777+0000","last_peered":"2024-09-21T10:11:39.405777+0000","last_clean":"2024-09-21T10:11:39.405777+0000","last_became_active":"2024-09-21T10:08:00.382708+0000","last_became_peered":"2024-09-21T10:08:00.382708+0000","last_unstale":"2024-09-21T10:11:39.405777+0000","last_undegraded":"2024-09-21T10:11:39.405777+0000","last_fullsized":"2024-09-21T10:11:39.405777+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:30:52.797259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"187'12083","reported_seq":12139,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.893004+0000","last_change":"2024-09-21T10:08:40.601007+0000","last_active":"2024-09-21T10:11:39.893004+0000","last_peered":"2024-09-21T10:11:39.893004+0000","last_clean":"2024-09-21T10:11:39.893004+0000","last_became_active":"2024-09-21T10:08:40.600733+0000","last_became_peered":"2024-09-21T10:08:40.600733+0000","last_unstale":"2024-09-21T10:11:39.893004+0000","last_undegraded":"2024-09-21T10:11:39.893004+0000","last_fullsized":"2024-09-21T10:11:39.893004+0000","mapping_epoch":165,"log_start":"169'2400","ondisk_log_start":"169'2400","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":9683,"log_dups_size":0,"ondisk_log_size":9683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:31:47.984695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202833920,"num_objects":3095,"num_object_clones":0,"num_object_copies":9285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12083,"num_write_kb":485696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":180,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.296963+0000","last_change":"2024-09-21T10:08:00.379604+0000","last_active":"2024-09-21T10:11:19.296963+0000","last_peered":"2024-09-21T10:11:19.296963+0000","last_clean":"2024-09-21T10:11:19.296963+0000","last_became_active":"2024-09-21T10:08:00.379301+0000","last_became_peered":"2024-09-21T10:08:00.379301+0000","last_unstale":"2024-09-21T10:11:19.296963+0000","last_undegraded":"2024-09-21T10:11:19.296963+0000","last_fullsized":"2024-09-21T10:11:19.296963+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:32:41.963929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":172,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.412267+0000","last_change":"2024-09-21T10:08:00.387420+0000","last_active":"2024-09-21T10:11:39.412267+0000","last_peered":"2024-09-21T10:11:39.412267+0000","last_clean":"2024-09-21T10:11:39.412267+0000","last_became_active":"2024-09-21T10:08:00.386843+0000","last_became_peered":"2024-09-21T10:08:00.386843+0000","last_unstale":"2024-09-21T10:11:39.412267+0000","last_undegraded":"2024-09-21T10:11:39.412267+0000","last_fullsized":"2024-09-21T10:11:39.412267+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:51:00.654836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"187'12550","reported_seq":12606,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:42.444686+0000","last_change":"2024-09-21T10:08:40.915784+0000","last_active":"2024-09-21T10:11:42.444686+0000","last_peered":"2024-09-21T10:11:42.444686+0000","last_clean":"2024-09-21T10:11:42.444686+0000","last_became_active":"2024-09-21T10:08:40.915456+0000","last_became_peered":"2024-09-21T10:08:40.915456+0000","last_unstale":"2024-09-21T10:11:42.444686+0000","last_undegraded":"2024-09-21T10:11:42.444686+0000","last_fullsized":"2024-09-21T10:11:42.444686+0000","mapping_epoch":165,"log_start":"172'5700","ondisk_log_start":"172'5700","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":6850,"log_dups_size":0,"ondisk_log_size":6850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:24:25.107555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194379776,"num_objects":2966,"num_object_clones":0,"num_object_copies":8898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12550,"num_write_kb":496512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":156,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.412739+0000","last_change":"2024-09-21T10:08:00.382496+0000","last_active":"2024-09-21T10:11:39.412739+0000","last_peered":"2024-09-21T10:11:39.412739+0000","last_clean":"2024-09-21T10:11:39.412739+0000","last_became_active":"2024-09-21T10:08:00.382180+0000","last_became_peered":"2024-09-21T10:08:00.382180+0000","last_unstale":"2024-09-21T10:11:39.412739+0000","last_undegraded":"2024-09-21T10:11:39.412739+0000","last_fullsized":"2024-09-21T10:11:39.412739+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:16.074736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":119,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.297157+0000","last_change":"2024-09-21T10:08:38.906801+0000","last_active":"2024-09-21T10:11:19.297157+0000","last_peered":"2024-09-21T10:11:19.297157+0000","last_clean":"2024-09-21T10:11:19.297157+0000","last_became_active":"2024-09-21T10:08:38.906487+0000","last_became_peered":"2024-09-21T10:08:38.906487+0000","last_unstale":"2024-09-21T10:11:19.297157+0000","last_undegraded":"2024-09-21T10:11:19.297157+0000","last_fullsized":"2024-09-21T10:11:19.297157+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:31.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"187'12298","reported_seq":12355,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:40.983764+0000","last_change":"2024-09-21T10:08:40.915278+0000","last_active":"2024-09-21T10:11:40.983764+0000","last_peered":"2024-09-21T10:11:40.983764+0000","last_clean":"2024-09-21T10:11:40.983764+0000","last_became_active":"2024-09-21T10:08:40.914886+0000","last_became_peered":"2024-09-21T10:08:40.914886+0000","last_unstale":"2024-09-21T10:11:40.983764+0000","last_undegraded":"2024-09-21T10:11:40.983764+0000","last_fullsized":"2024-09-21T10:11:40.983764+0000","mapping_epoch":165,"log_start":"172'5700","ondisk_log_start":"172'5700","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":6598,"log_dups_size":0,"ondisk_log_size":6598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:47.589854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199688216,"num_objects":3048,"num_object_clones":0,"num_object_copies":9144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3048,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12298,"num_write_kb":491009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":244,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.403328+0000","last_change":"2024-09-21T10:02:14.097462+0000","last_active":"2024-09-21T10:11:39.403328+0000","last_peered":"2024-09-21T10:11:39.403328+0000","last_clean":"2024-09-21T10:11:39.403328+0000","last_became_active":"2024-09-21T10:02:14.097167+0000","last_became_peered":"2024-09-21T10:02:14.097167+0000","last_unstale":"2024-09-21T10:11:39.403328+0000","last_undegraded":"2024-09-21T10:11:39.403328+0000","last_fullsized":"2024-09-21T10:11:39.403328+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:54.922731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":248,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.405371+0000","last_change":"2024-09-21T10:08:00.378569+0000","last_active":"2024-09-21T10:11:39.405371+0000","last_peered":"2024-09-21T10:11:39.405371+0000","last_clean":"2024-09-21T10:11:39.405371+0000","last_became_active":"2024-09-21T10:08:00.378015+0000","last_became_peered":"2024-09-21T10:08:00.378015+0000","last_unstale":"2024-09-21T10:11:39.405371+0000","last_undegraded":"2024-09-21T10:11:39.405371+0000","last_fullsized":"2024-09-21T10:11:39.405371+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:58.121005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"187'12280","reported_seq":12336,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:42.469081+0000","last_change":"2024-09-21T10:08:40.915416+0000","last_active":"2024-09-21T10:11:42.469081+0000","last_peered":"2024-09-21T10:11:42.469081+0000","last_clean":"2024-09-21T10:11:42.469081+0000","last_became_active":"2024-09-21T10:08:40.914705+0000","last_became_peered":"2024-09-21T10:08:40.914705+0000","last_unstale":"2024-09-21T10:11:42.469081+0000","last_undegraded":"2024-09-21T10:11:42.469081+0000","last_fullsized":"2024-09-21T10:11:42.469081+0000","mapping_epoch":165,"log_start":"172'5400","ondisk_log_start":"172'5400","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":6880,"log_dups_size":0,"ondisk_log_size":6880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:34:22.667201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193069056,"num_objects":2946,"num_object_clones":0,"num_object_copies":8838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12280,"num_write_kb":487232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":268,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.292078+0000","last_change":"2024-09-21T09:59:54.265556+0000","last_active":"2024-09-21T10:11:19.292078+0000","last_peered":"2024-09-21T10:11:19.292078+0000","last_clean":"2024-09-21T10:11:19.292078+0000","last_became_active":"2024-09-21T09:59:54.265033+0000","last_became_peered":"2024-09-21T09:59:54.265033+0000","last_unstale":"2024-09-21T10:11:19.292078+0000","last_undegraded":"2024-09-21T10:11:19.292078+0000","last_fullsized":"2024-09-21T10:11:19.292078+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T09:48:34.850872+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:40.449756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":535,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.412906+0000","last_change":"2024-09-21T10:07:43.790490+0000","last_active":"2024-09-21T10:11:39.412906+0000","last_peered":"2024-09-21T10:11:39.412906+0000","last_clean":"2024-09-21T10:11:39.412906+0000","last_became_active":"2024-09-21T10:07:43.789986+0000","last_became_peered":"2024-09-21T10:07:43.789986+0000","last_unstale":"2024-09-21T10:11:39.412906+0000","last_undegraded":"2024-09-21T10:11:39.412906+0000","last_fullsized":"2024-09-21T10:11:39.412906+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:08.740356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":481,"num_read_kb":1580,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":197,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.399945+0000","last_change":"2024-09-21T10:02:14.099961+0000","last_active":"2024-09-21T10:11:39.399945+0000","last_peered":"2024-09-21T10:11:39.399945+0000","last_clean":"2024-09-21T10:11:39.399945+0000","last_became_active":"2024-09-21T10:02:14.099067+0000","last_became_peered":"2024-09-21T10:02:14.099067+0000","last_unstale":"2024-09-21T10:11:39.399945+0000","last_undegraded":"2024-09-21T10:11:39.399945+0000","last_fullsized":"2024-09-21T10:11:39.399945+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:41.037532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":162,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.297202+0000","last_change":"2024-09-21T10:08:00.386135+0000","last_active":"2024-09-21T10:11:19.297202+0000","last_peered":"2024-09-21T10:11:19.297202+0000","last_clean":"2024-09-21T10:11:19.297202+0000","last_became_active":"2024-09-21T10:08:00.385796+0000","last_became_peered":"2024-09-21T10:08:00.385796+0000","last_unstale":"2024-09-21T10:11:19.297202+0000","last_undegraded":"2024-09-21T10:11:19.297202+0000","last_fullsized":"2024-09-21T10:11:19.297202+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:27.054699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"187'12236","reported_seq":12291,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:40.991393+0000","last_change":"2024-09-21T10:08:40.916810+0000","last_active":"2024-09-21T10:11:40.991393+0000","last_peered":"2024-09-21T10:11:40.991393+0000","last_clean":"2024-09-21T10:11:40.991393+0000","last_became_active":"2024-09-21T10:08:40.916090+0000","last_became_peered":"2024-09-21T10:08:40.916090+0000","last_unstale":"2024-09-21T10:11:40.991393+0000","last_undegraded":"2024-09-21T10:11:40.991393+0000","last_fullsized":"2024-09-21T10:11:40.991393+0000","mapping_epoch":165,"log_start":"172'5700","ondisk_log_start":"172'5700","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":6536,"log_dups_size":0,"ondisk_log_size":6536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:06:45.072223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199753728,"num_objects":3048,"num_object_clones":0,"num_object_copies":9144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12236,"num_write_kb":489088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":531,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.291923+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:11:19.291923+0000","last_peered":"2024-09-21T10:11:19.291923+0000","last_clean":"2024-09-21T10:11:19.291923+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:11:19.291923+0000","last_undegraded":"2024-09-21T10:11:19.291923+0000","last_fullsized":"2024-09-21T10:11:19.291923+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":260,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.408622+0000","last_change":"2024-09-21T10:08:00.383928+0000","last_active":"2024-09-21T10:11:39.408622+0000","last_peered":"2024-09-21T10:11:39.408622+0000","last_clean":"2024-09-21T10:11:39.408622+0000","last_became_active":"2024-09-21T10:08:00.383625+0000","last_became_peered":"2024-09-21T10:08:00.383625+0000","last_unstale":"2024-09-21T10:11:39.408622+0000","last_undegraded":"2024-09-21T10:11:39.408622+0000","last_fullsized":"2024-09-21T10:11:39.408622+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:39.207283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":250,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.408517+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:11:39.408517+0000","last_peered":"2024-09-21T10:11:39.408517+0000","last_clean":"2024-09-21T10:11:39.408517+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:11:39.408517+0000","last_undegraded":"2024-09-21T10:11:39.408517+0000","last_fullsized":"2024-09-21T10:11:39.408517+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"186'11939","reported_seq":11968,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:38.211904+0000","last_change":"2024-09-21T10:08:40.597717+0000","last_active":"2024-09-21T10:11:38.211904+0000","last_peered":"2024-09-21T10:11:38.211904+0000","last_clean":"2024-09-21T10:11:38.211904+0000","last_became_active":"2024-09-21T10:08:40.597197+0000","last_became_peered":"2024-09-21T10:08:40.597197+0000","last_unstale":"2024-09-21T10:11:38.211904+0000","last_undegraded":"2024-09-21T10:11:38.211904+0000","last_fullsized":"2024-09-21T10:11:38.211904+0000","mapping_epoch":165,"log_start":"170'4000","ondisk_log_start":"170'4000","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":7939,"log_dups_size":0,"ondisk_log_size":7939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:25:03.341682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209780736,"num_objects":3201,"num_object_clones":0,"num_object_copies":9603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11939,"num_write_kb":484480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":987,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.412888+0000","last_change":"2024-09-21T10:02:14.100143+0000","last_active":"2024-09-21T10:11:39.412888+0000","last_peered":"2024-09-21T10:11:39.412888+0000","last_clean":"2024-09-21T10:11:39.412888+0000","last_became_active":"2024-09-21T10:02:14.099586+0000","last_became_peered":"2024-09-21T10:02:14.099586+0000","last_unstale":"2024-09-21T10:11:39.412888+0000","last_undegraded":"2024-09-21T10:11:39.412888+0000","last_fullsized":"2024-09-21T10:11:39.412888+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:34.064652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":122,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.404647+0000","last_change":"2024-09-21T10:08:18.470748+0000","last_active":"2024-09-21T10:11:39.404647+0000","last_peered":"2024-09-21T10:11:39.404647+0000","last_clean":"2024-09-21T10:11:39.404647+0000","last_became_active":"2024-09-21T10:08:18.470227+0000","last_became_peered":"2024-09-21T10:08:18.470227+0000","last_unstale":"2024-09-21T10:11:39.404647+0000","last_undegraded":"2024-09-21T10:11:39.404647+0000","last_fullsized":"2024-09-21T10:11:39.404647+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:16.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":244,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.412751+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:11:39.412751+0000","last_peered":"2024-09-21T10:11:39.412751+0000","last_clean":"2024-09-21T10:11:39.412751+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:11:39.412751+0000","last_undegraded":"2024-09-21T10:11:39.412751+0000","last_fullsized":"2024-09-21T10:11:39.412751+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"187'12247","reported_seq":12303,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:42.464830+0000","last_change":"2024-09-21T10:08:40.601145+0000","last_active":"2024-09-21T10:11:42.464830+0000","last_peered":"2024-09-21T10:11:42.464830+0000","last_clean":"2024-09-21T10:11:42.464830+0000","last_became_active":"2024-09-21T10:08:40.600872+0000","last_became_peered":"2024-09-21T10:08:40.600872+0000","last_unstale":"2024-09-21T10:11:42.464830+0000","last_undegraded":"2024-09-21T10:11:42.464830+0000","last_fullsized":"2024-09-21T10:11:42.464830+0000","mapping_epoch":165,"log_start":"172'5400","ondisk_log_start":"172'5400","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":6847,"log_dups_size":0,"ondisk_log_size":6847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:55:44.108331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194314240,"num_objects":2965,"num_object_clones":0,"num_object_copies":8895,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12247,"num_write_kb":486784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":720,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.406657+0000","last_change":"2024-09-21T10:07:43.794767+0000","last_active":"2024-09-21T10:11:39.406657+0000","last_peered":"2024-09-21T10:11:39.406657+0000","last_clean":"2024-09-21T10:11:39.406657+0000","last_became_active":"2024-09-21T10:07:43.794062+0000","last_became_peered":"2024-09-21T10:07:43.794062+0000","last_unstale":"2024-09-21T10:11:39.406657+0000","last_undegraded":"2024-09-21T10:11:39.406657+0000","last_fullsized":"2024-09-21T10:11:39.406657+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:51.977955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":147,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.407660+0000","last_change":"2024-09-21T10:08:00.389176+0000","last_active":"2024-09-21T10:11:39.407660+0000","last_peered":"2024-09-21T10:11:39.407660+0000","last_clean":"2024-09-21T10:11:39.407660+0000","last_became_active":"2024-09-21T10:08:00.383854+0000","last_became_peered":"2024-09-21T10:08:00.383854+0000","last_unstale":"2024-09-21T10:11:39.407660+0000","last_undegraded":"2024-09-21T10:11:39.407660+0000","last_fullsized":"2024-09-21T10:11:39.407660+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:46.952216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":160,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.412450+0000","last_change":"2024-09-21T10:07:43.792153+0000","last_active":"2024-09-21T10:11:39.412450+0000","last_peered":"2024-09-21T10:11:39.412450+0000","last_clean":"2024-09-21T10:11:39.412450+0000","last_became_active":"2024-09-21T10:07:43.791770+0000","last_became_peered":"2024-09-21T10:07:43.791770+0000","last_unstale":"2024-09-21T10:11:39.412450+0000","last_undegraded":"2024-09-21T10:11:39.412450+0000","last_fullsized":"2024-09-21T10:11:39.412450+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:52.506941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"187'12332","reported_seq":12388,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:42.565706+0000","last_change":"2024-09-21T10:08:40.603257+0000","last_active":"2024-09-21T10:11:42.565706+0000","last_peered":"2024-09-21T10:11:42.565706+0000","last_clean":"2024-09-21T10:11:42.565706+0000","last_became_active":"2024-09-21T10:08:40.602870+0000","last_became_peered":"2024-09-21T10:08:40.602870+0000","last_unstale":"2024-09-21T10:11:42.565706+0000","last_undegraded":"2024-09-21T10:11:42.565706+0000","last_fullsized":"2024-09-21T10:11:42.565706+0000","mapping_epoch":165,"log_start":"171'4200","ondisk_log_start":"171'4200","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":8132,"log_dups_size":0,"ondisk_log_size":8132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:00:53.616267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190709760,"num_objects":2910,"num_object_clones":0,"num_object_copies":8730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12332,"num_write_kb":487744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":165,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.400475+0000","last_change":"2024-09-21T10:07:43.791405+0000","last_active":"2024-09-21T10:11:39.400475+0000","last_peered":"2024-09-21T10:11:39.400475+0000","last_clean":"2024-09-21T10:11:39.400475+0000","last_became_active":"2024-09-21T10:07:43.791078+0000","last_became_peered":"2024-09-21T10:07:43.791078+0000","last_unstale":"2024-09-21T10:11:39.400475+0000","last_undegraded":"2024-09-21T10:11:39.400475+0000","last_fullsized":"2024-09-21T10:11:39.400475+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:17:38.072612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":827,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.405365+0000","last_change":"2024-09-21T10:08:18.468995+0000","last_active":"2024-09-21T10:11:39.405365+0000","last_peered":"2024-09-21T10:11:39.405365+0000","last_clean":"2024-09-21T10:11:39.405365+0000","last_became_active":"2024-09-21T10:08:18.468243+0000","last_became_peered":"2024-09-21T10:08:18.468243+0000","last_unstale":"2024-09-21T10:11:39.405365+0000","last_undegraded":"2024-09-21T10:11:39.405365+0000","last_fullsized":"2024-09-21T10:11:39.405365+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:45.520968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":146,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.406894+0000","last_change":"2024-09-21T10:08:18.468595+0000","last_active":"2024-09-21T10:11:39.406894+0000","last_peered":"2024-09-21T10:11:39.406894+0000","last_clean":"2024-09-21T10:11:39.406894+0000","last_became_active":"2024-09-21T10:08:18.468296+0000","last_became_peered":"2024-09-21T10:08:18.468296+0000","last_unstale":"2024-09-21T10:11:39.406894+0000","last_undegraded":"2024-09-21T10:11:39.406894+0000","last_fullsized":"2024-09-21T10:11:39.406894+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:42.748130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"187'12446","reported_seq":12502,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:40.987719+0000","last_change":"2024-09-21T10:08:40.604276+0000","last_active":"2024-09-21T10:11:40.987719+0000","last_peered":"2024-09-21T10:11:40.987719+0000","last_clean":"2024-09-21T10:11:40.987719+0000","last_became_active":"2024-09-21T10:08:40.603906+0000","last_became_peered":"2024-09-21T10:08:40.603906+0000","last_unstale":"2024-09-21T10:11:40.987719+0000","last_undegraded":"2024-09-21T10:11:40.987719+0000","last_fullsized":"2024-09-21T10:11:40.987719+0000","mapping_epoch":165,"log_start":"172'5900","ondisk_log_start":"172'5900","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":6546,"log_dups_size":0,"ondisk_log_size":6546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:59:29.555786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202637312,"num_objects":3092,"num_object_clones":0,"num_object_copies":9276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12446,"num_write_kb":497216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":122,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.405255+0000","last_change":"2024-09-21T10:08:18.469907+0000","last_active":"2024-09-21T10:11:39.405255+0000","last_peered":"2024-09-21T10:11:39.405255+0000","last_clean":"2024-09-21T10:11:39.405255+0000","last_became_active":"2024-09-21T10:08:18.469443+0000","last_became_peered":"2024-09-21T10:08:18.469443+0000","last_unstale":"2024-09-21T10:11:39.405255+0000","last_undegraded":"2024-09-21T10:11:39.405255+0000","last_fullsized":"2024-09-21T10:11:39.405255+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:15.645803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":581,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.295089+0000","last_change":"2024-09-21T10:08:00.419641+0000","last_active":"2024-09-21T10:11:19.295089+0000","last_peered":"2024-09-21T10:11:19.295089+0000","last_clean":"2024-09-21T10:11:19.295089+0000","last_became_active":"2024-09-21T10:08:00.387050+0000","last_became_peered":"2024-09-21T10:08:00.387050+0000","last_unstale":"2024-09-21T10:11:19.295089+0000","last_undegraded":"2024-09-21T10:11:19.295089+0000","last_fullsized":"2024-09-21T10:11:19.295089+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:00:32.881619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":228,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.407933+0000","last_change":"2024-09-21T09:59:54.267798+0000","last_active":"2024-09-21T10:11:39.407933+0000","last_peered":"2024-09-21T10:11:39.407933+0000","last_clean":"2024-09-21T10:11:39.407933+0000","last_became_active":"2024-09-21T09:59:54.265615+0000","last_became_peered":"2024-09-21T09:59:54.265615+0000","last_unstale":"2024-09-21T10:11:39.407933+0000","last_undegraded":"2024-09-21T10:11:39.407933+0000","last_fullsized":"2024-09-21T10:11:39.407933+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:30.363872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"187'12406","reported_seq":12462,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:42.401295+0000","last_change":"2024-09-21T10:08:40.601929+0000","last_active":"2024-09-21T10:11:42.401295+0000","last_peered":"2024-09-21T10:11:42.401295+0000","last_clean":"2024-09-21T10:11:42.401295+0000","last_became_active":"2024-09-21T10:08:40.601656+0000","last_became_peered":"2024-09-21T10:08:40.601656+0000","last_unstale":"2024-09-21T10:11:42.401295+0000","last_undegraded":"2024-09-21T10:11:42.401295+0000","last_fullsized":"2024-09-21T10:11:42.401295+0000","mapping_epoch":165,"log_start":"172'5500","ondisk_log_start":"172'5500","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":6906,"log_dups_size":0,"ondisk_log_size":6906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:22:31.376121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193069056,"num_objects":2946,"num_object_clones":0,"num_object_copies":8838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12406,"num_write_kb":491264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":241,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.403898+0000","last_change":"2024-09-21T10:02:14.101943+0000","last_active":"2024-09-21T10:11:39.403898+0000","last_peered":"2024-09-21T10:11:39.403898+0000","last_clean":"2024-09-21T10:11:39.403898+0000","last_became_active":"2024-09-21T10:02:14.101488+0000","last_became_peered":"2024-09-21T10:02:14.101488+0000","last_unstale":"2024-09-21T10:11:39.403898+0000","last_undegraded":"2024-09-21T10:11:39.403898+0000","last_fullsized":"2024-09-21T10:11:39.403898+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:52:27.866414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":491,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.404883+0000","last_change":"2024-09-21T10:08:18.472876+0000","last_active":"2024-09-21T10:11:39.404883+0000","last_peered":"2024-09-21T10:11:39.404883+0000","last_clean":"2024-09-21T10:11:39.404883+0000","last_became_active":"2024-09-21T10:08:18.472385+0000","last_became_peered":"2024-09-21T10:08:18.472385+0000","last_unstale":"2024-09-21T10:11:39.404883+0000","last_undegraded":"2024-09-21T10:11:39.404883+0000","last_fullsized":"2024-09-21T10:11:39.404883+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:33.109956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":59,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.413399+0000","last_change":"2024-09-21T10:08:38.797352+0000","last_active":"2024-09-21T10:11:39.413399+0000","last_peered":"2024-09-21T10:11:39.413399+0000","last_clean":"2024-09-21T10:11:39.413399+0000","last_became_active":"2024-09-21T10:08:38.796950+0000","last_became_peered":"2024-09-21T10:08:38.796950+0000","last_unstale":"2024-09-21T10:11:39.413399+0000","last_undegraded":"2024-09-21T10:11:39.413399+0000","last_fullsized":"2024-09-21T10:11:39.413399+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:02.646321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3","version":"186'12204","reported_seq":12233,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T10:11:38.217868+0000","last_change":"2024-09-21T10:08:40.598677+0000","last_active":"2024-09-21T10:11:38.217868+0000","last_peered":"2024-09-21T10:11:38.217868+0000","last_clean":"2024-09-21T10:11:38.217868+0000","last_became_active":"2024-09-21T10:08:40.598383+0000","last_became_peered":"2024-09-21T10:08:40.598383+0000","last_unstale":"2024-09-21T10:11:38.217868+0000","last_undegraded":"2024-09-21T10:11:38.217868+0000","last_fullsized":"2024-09-21T10:11:38.217868+0000","mapping_epoch":165,"log_start":"171'4300","ondisk_log_start":"171'4300","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:08:39.578252+0000","last_clean_scrub_stamp":"2024-09-21T10:08:39.578252+0000","objects_scrubbed":0,"log_size":7904,"log_dups_size":0,"ondisk_log_size":7904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:03.675317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207486976,"num_objects":3166,"num_object_clones":0,"num_object_copies":9498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12204,"num_write_kb":491840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":276,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.408184+0000","last_change":"2024-09-21T10:08:00.390436+0000","last_active":"2024-09-21T10:11:39.408184+0000","last_peered":"2024-09-21T10:11:39.408184+0000","last_clean":"2024-09-21T10:11:39.408184+0000","last_became_active":"2024-09-21T10:08:00.381835+0000","last_became_peered":"2024-09-21T10:08:00.381835+0000","last_unstale":"2024-09-21T10:11:39.408184+0000","last_undegraded":"2024-09-21T10:11:39.408184+0000","last_fullsized":"2024-09-21T10:11:39.408184+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:01:35.697356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":412,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.297184+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:11:19.297184+0000","last_peered":"2024-09-21T10:11:19.297184+0000","last_clean":"2024-09-21T10:11:19.297184+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:11:19.297184+0000","last_undegraded":"2024-09-21T10:11:19.297184+0000","last_fullsized":"2024-09-21T10:11:19.297184+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":50,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.296750+0000","last_change":"2024-09-21T10:08:00.385774+0000","last_active":"2024-09-21T10:11:19.296750+0000","last_peered":"2024-09-21T10:11:19.296750+0000","last_clean":"2024-09-21T10:11:19.296750+0000","last_became_active":"2024-09-21T10:08:00.385445+0000","last_became_peered":"2024-09-21T10:08:00.385445+0000","last_unstale":"2024-09-21T10:11:19.296750+0000","last_undegraded":"2024-09-21T10:11:19.296750+0000","last_fullsized":"2024-09-21T10:11:19.296750+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:42.442190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":109,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.296911+0000","last_change":"2024-09-21T10:08:00.379661+0000","last_active":"2024-09-21T10:11:19.296911+0000","last_peered":"2024-09-21T10:11:19.296911+0000","last_clean":"2024-09-21T10:11:19.296911+0000","last_became_active":"2024-09-21T10:08:00.379301+0000","last_became_peered":"2024-09-21T10:08:00.379301+0000","last_unstale":"2024-09-21T10:11:19.296911+0000","last_undegraded":"2024-09-21T10:11:19.296911+0000","last_fullsized":"2024-09-21T10:11:19.296911+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:15.649523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":133,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.291994+0000","last_change":"2024-09-21T10:08:00.410731+0000","last_active":"2024-09-21T10:11:19.291994+0000","last_peered":"2024-09-21T10:11:19.291994+0000","last_clean":"2024-09-21T10:11:19.291994+0000","last_became_active":"2024-09-21T10:08:00.380324+0000","last_became_peered":"2024-09-21T10:08:00.380324+0000","last_unstale":"2024-09-21T10:11:19.291994+0000","last_undegraded":"2024-09-21T10:11:19.291994+0000","last_fullsized":"2024-09-21T10:11:19.291994+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:47:34.960221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":122,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.406752+0000","last_change":"2024-09-21T10:08:18.470378+0000","last_active":"2024-09-21T10:11:39.406752+0000","last_peered":"2024-09-21T10:11:39.406752+0000","last_clean":"2024-09-21T10:11:39.406752+0000","last_became_active":"2024-09-21T10:08:18.469482+0000","last_became_peered":"2024-09-21T10:08:18.469482+0000","last_unstale":"2024-09-21T10:11:39.406752+0000","last_undegraded":"2024-09-21T10:11:39.406752+0000","last_fullsized":"2024-09-21T10:11:39.406752+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:35.628724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":202,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.295005+0000","last_change":"2024-09-21T10:08:00.403606+0000","last_active":"2024-09-21T10:11:19.295005+0000","last_peered":"2024-09-21T10:11:19.295005+0000","last_clean":"2024-09-21T10:11:19.295005+0000","last_became_active":"2024-09-21T10:08:00.379330+0000","last_became_peered":"2024-09-21T10:08:00.379330+0000","last_unstale":"2024-09-21T10:11:19.295005+0000","last_undegraded":"2024-09-21T10:11:19.295005+0000","last_fullsized":"2024-09-21T10:11:19.295005+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:07.226027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":98,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.292400+0000","last_change":"2024-09-21T10:07:43.791830+0000","last_active":"2024-09-21T10:11:19.292400+0000","last_peered":"2024-09-21T10:11:19.292400+0000","last_clean":"2024-09-21T10:11:19.292400+0000","last_became_active":"2024-09-21T10:07:43.791314+0000","last_became_peered":"2024-09-21T10:07:43.791314+0000","last_unstale":"2024-09-21T10:11:19.292400+0000","last_undegraded":"2024-09-21T10:11:19.292400+0000","last_fullsized":"2024-09-21T10:11:19.292400+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:37.086629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":198,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.399516+0000","last_change":"2024-09-21T10:02:14.100451+0000","last_active":"2024-09-21T10:11:39.399516+0000","last_peered":"2024-09-21T10:11:39.399516+0000","last_clean":"2024-09-21T10:11:39.399516+0000","last_became_active":"2024-09-21T10:02:14.100160+0000","last_became_peered":"2024-09-21T10:02:14.100160+0000","last_unstale":"2024-09-21T10:11:39.399516+0000","last_undegraded":"2024-09-21T10:11:39.399516+0000","last_fullsized":"2024-09-21T10:11:39.399516+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:28.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":238,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.408711+0000","last_change":"2024-09-21T10:02:14.096098+0000","last_active":"2024-09-21T10:11:39.408711+0000","last_peered":"2024-09-21T10:11:39.408711+0000","last_clean":"2024-09-21T10:11:39.408711+0000","last_became_active":"2024-09-21T10:02:14.095811+0000","last_became_peered":"2024-09-21T10:02:14.095811+0000","last_unstale":"2024-09-21T10:11:39.408711+0000","last_undegraded":"2024-09-21T10:11:39.408711+0000","last_fullsized":"2024-09-21T10:11:39.408711+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:54.619333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":243,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.403052+0000","last_change":"2024-09-21T10:02:14.100712+0000","last_active":"2024-09-21T10:11:39.403052+0000","last_peered":"2024-09-21T10:11:39.403052+0000","last_clean":"2024-09-21T10:11:39.403052+0000","last_became_active":"2024-09-21T10:02:14.100427+0000","last_became_peered":"2024-09-21T10:02:14.100427+0000","last_unstale":"2024-09-21T10:11:39.403052+0000","last_undegraded":"2024-09-21T10:11:39.403052+0000","last_fullsized":"2024-09-21T10:11:39.403052+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:16:42.350642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":149,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.297129+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:11:19.297129+0000","last_peered":"2024-09-21T10:11:19.297129+0000","last_clean":"2024-09-21T10:11:19.297129+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:11:19.297129+0000","last_undegraded":"2024-09-21T10:11:19.297129+0000","last_fullsized":"2024-09-21T10:11:19.297129+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":260,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.403226+0000","last_change":"2024-09-21T10:02:14.097369+0000","last_active":"2024-09-21T10:11:39.403226+0000","last_peered":"2024-09-21T10:11:39.403226+0000","last_clean":"2024-09-21T10:11:39.403226+0000","last_became_active":"2024-09-21T10:02:14.097067+0000","last_became_peered":"2024-09-21T10:02:14.097067+0000","last_unstale":"2024-09-21T10:11:39.403226+0000","last_undegraded":"2024-09-21T10:11:39.403226+0000","last_fullsized":"2024-09-21T10:11:39.403226+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:50:48.123813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":239,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.408428+0000","last_change":"2024-09-21T10:02:14.098970+0000","last_active":"2024-09-21T10:11:39.408428+0000","last_peered":"2024-09-21T10:11:39.408428+0000","last_clean":"2024-09-21T10:11:39.408428+0000","last_became_active":"2024-09-21T10:02:14.098641+0000","last_became_peered":"2024-09-21T10:02:14.098641+0000","last_unstale":"2024-09-21T10:11:39.408428+0000","last_undegraded":"2024-09-21T10:11:39.408428+0000","last_fullsized":"2024-09-21T10:11:39.408428+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:25.215602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":248,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.405233+0000","last_change":"2024-09-21T10:08:00.381058+0000","last_active":"2024-09-21T10:11:39.405233+0000","last_peered":"2024-09-21T10:11:39.405233+0000","last_clean":"2024-09-21T10:11:39.405233+0000","last_became_active":"2024-09-21T10:08:00.380573+0000","last_became_peered":"2024-09-21T10:08:00.380573+0000","last_unstale":"2024-09-21T10:11:39.405233+0000","last_undegraded":"2024-09-21T10:11:39.405233+0000","last_fullsized":"2024-09-21T10:11:39.405233+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:01:04.137425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":263,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.400313+0000","last_change":"2024-09-21T10:08:00.387321+0000","last_active":"2024-09-21T10:11:39.400313+0000","last_peered":"2024-09-21T10:11:39.400313+0000","last_clean":"2024-09-21T10:11:39.400313+0000","last_became_active":"2024-09-21T10:08:00.386935+0000","last_became_peered":"2024-09-21T10:08:00.386935+0000","last_unstale":"2024-09-21T10:11:39.400313+0000","last_undegraded":"2024-09-21T10:11:39.400313+0000","last_fullsized":"2024-09-21T10:11:39.400313+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:57.706313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":68,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.406975+0000","last_change":"2024-09-21T10:08:18.471282+0000","last_active":"2024-09-21T10:11:39.406975+0000","last_peered":"2024-09-21T10:11:39.406975+0000","last_clean":"2024-09-21T10:11:39.406975+0000","last_became_active":"2024-09-21T10:08:18.470952+0000","last_became_peered":"2024-09-21T10:08:18.470952+0000","last_unstale":"2024-09-21T10:11:39.406975+0000","last_undegraded":"2024-09-21T10:11:39.406975+0000","last_fullsized":"2024-09-21T10:11:39.406975+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:25.597081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":262,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.412406+0000","last_change":"2024-09-21T10:02:14.102826+0000","last_active":"2024-09-21T10:11:39.412406+0000","last_peered":"2024-09-21T10:11:39.412406+0000","last_clean":"2024-09-21T10:11:39.412406+0000","last_became_active":"2024-09-21T10:02:14.102499+0000","last_became_peered":"2024-09-21T10:02:14.102499+0000","last_unstale":"2024-09-21T10:11:39.412406+0000","last_undegraded":"2024-09-21T10:11:39.412406+0000","last_fullsized":"2024-09-21T10:11:39.412406+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:30.924687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":252,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.407832+0000","last_change":"2024-09-21T10:08:00.386104+0000","last_active":"2024-09-21T10:11:39.407832+0000","last_peered":"2024-09-21T10:11:39.407832+0000","last_clean":"2024-09-21T10:11:39.407832+0000","last_became_active":"2024-09-21T10:08:00.381484+0000","last_became_peered":"2024-09-21T10:08:00.381484+0000","last_unstale":"2024-09-21T10:11:39.407832+0000","last_undegraded":"2024-09-21T10:11:39.407832+0000","last_fullsized":"2024-09-21T10:11:39.407832+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:39.190191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":177,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.402896+0000","last_change":"2024-09-21T10:08:00.381149+0000","last_active":"2024-09-21T10:11:39.402896+0000","last_peered":"2024-09-21T10:11:39.402896+0000","last_clean":"2024-09-21T10:11:39.402896+0000","last_became_active":"2024-09-21T10:08:00.380795+0000","last_became_peered":"2024-09-21T10:08:00.380795+0000","last_unstale":"2024-09-21T10:11:39.402896+0000","last_undegraded":"2024-09-21T10:11:39.402896+0000","last_fullsized":"2024-09-21T10:11:39.402896+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:50:46.793182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":79,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.405553+0000","last_change":"2024-09-21T10:08:18.466882+0000","last_active":"2024-09-21T10:11:39.405553+0000","last_peered":"2024-09-21T10:11:39.405553+0000","last_clean":"2024-09-21T10:11:39.405553+0000","last_became_active":"2024-09-21T10:08:18.466470+0000","last_became_peered":"2024-09-21T10:08:18.466470+0000","last_unstale":"2024-09-21T10:11:39.405553+0000","last_undegraded":"2024-09-21T10:11:39.405553+0000","last_fullsized":"2024-09-21T10:11:39.405553+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:51:31.020051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":177,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.400177+0000","last_change":"2024-09-21T10:08:00.389061+0000","last_active":"2024-09-21T10:11:39.400177+0000","last_peered":"2024-09-21T10:11:39.400177+0000","last_clean":"2024-09-21T10:11:39.400177+0000","last_became_active":"2024-09-21T10:08:00.388657+0000","last_became_peered":"2024-09-21T10:08:00.388657+0000","last_unstale":"2024-09-21T10:11:39.400177+0000","last_undegraded":"2024-09-21T10:11:39.400177+0000","last_fullsized":"2024-09-21T10:11:39.400177+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:58:17.239589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":61,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.297124+0000","last_change":"2024-09-21T10:08:00.394350+0000","last_active":"2024-09-21T10:11:19.297124+0000","last_peered":"2024-09-21T10:11:19.297124+0000","last_clean":"2024-09-21T10:11:19.297124+0000","last_became_active":"2024-09-21T10:08:00.387232+0000","last_became_peered":"2024-09-21T10:08:00.387232+0000","last_unstale":"2024-09-21T10:11:19.297124+0000","last_undegraded":"2024-09-21T10:11:19.297124+0000","last_fullsized":"2024-09-21T10:11:19.297124+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:56.160018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":263,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.399832+0000","last_change":"2024-09-21T10:02:14.099799+0000","last_active":"2024-09-21T10:11:39.399832+0000","last_peered":"2024-09-21T10:11:39.399832+0000","last_clean":"2024-09-21T10:11:39.399832+0000","last_became_active":"2024-09-21T10:02:14.099375+0000","last_became_peered":"2024-09-21T10:02:14.099375+0000","last_unstale":"2024-09-21T10:11:39.399832+0000","last_undegraded":"2024-09-21T10:11:39.399832+0000","last_fullsized":"2024-09-21T10:11:39.399832+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:13.407638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":249,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.403417+0000","last_change":"2024-09-21T10:02:14.099439+0000","last_active":"2024-09-21T10:11:39.403417+0000","last_peered":"2024-09-21T10:11:39.403417+0000","last_clean":"2024-09-21T10:11:39.403417+0000","last_became_active":"2024-09-21T10:02:14.099117+0000","last_became_peered":"2024-09-21T10:02:14.099117+0000","last_unstale":"2024-09-21T10:11:39.403417+0000","last_undegraded":"2024-09-21T10:11:39.403417+0000","last_fullsized":"2024-09-21T10:11:39.403417+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:39.681817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":124,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.296549+0000","last_change":"2024-09-21T10:08:00.381529+0000","last_active":"2024-09-21T10:11:19.296549+0000","last_peered":"2024-09-21T10:11:19.296549+0000","last_clean":"2024-09-21T10:11:19.296549+0000","last_became_active":"2024-09-21T10:08:00.381061+0000","last_became_peered":"2024-09-21T10:08:00.381061+0000","last_unstale":"2024-09-21T10:11:19.296549+0000","last_undegraded":"2024-09-21T10:11:19.296549+0000","last_fullsized":"2024-09-21T10:11:19.296549+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:56:58.775586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":243,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T10:11:39.408620+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:11:39.408620+0000","last_peered":"2024-09-21T10:11:39.408620+0000","last_clean":"2024-09-21T10:11:39.408620+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:11:39.408620+0000","last_undegraded":"2024-09-21T10:11:39.408620+0000","last_fullsized":"2024-09-21T10:11:39.408620+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":3195142168,"num_objects":48755,"num_object_clones":0,"num_object_copies":146265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48755,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196559,"num_write_kb":7849985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9726275584,"data_stored":9726263368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117859,"ondisk_log_size":117859,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":9926,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":21480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5982,"num_read_kb":13005,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397762,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1538756,"kb_used_data":1233084,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92206396,"statfs":{"total":95995035648,"available":94419349504,"internally_reserved":0,"allocated":1262678016,"data_stored":1261753705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593990,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1975808,"kb_used_data":1670136,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91769344,"statfs":{"total":95995035648,"available":93971808256,"internally_reserved":0,"allocated":1710219264,"data_stored":1709298956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790220,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1548272,"kb_used_data":1242600,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92196880,"statfs":{"total":95995035648,"available":94409605120,"internally_reserved":0,"allocated":1272422400,"data_stored":1271513433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986449,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":101,"seq":433791697012,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2186932,"kb_used_data":1910252,"kb_used_omap":5,"kb_used_meta":276666,"kb_avail":91558220,"statfs":{"total":95995035648,"available":93755617280,"internally_reserved":0,"allocated":1956098048,"data_stored":1955193111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":283306768},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860538,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1914056,"kb_used_data":1607164,"kb_used_omap":8,"kb_used_meta":306871,"kb_avail":91831096,"statfs":{"total":95995035648,"available":94035042304,"internally_reserved":0,"allocated":1645735936,"data_stored":1644819881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":314236780},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024065,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1458296,"kb_used_data":1175340,"kb_used_omap":7,"kb_used_meta":282936,"kb_avail":92286856,"statfs":{"total":95995035648,"available":94501740544,"internally_reserved":0,"allocated":1203548160,"data_stored":1202642576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":289727026},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187600,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1081412,"kb_used_data":768572,"kb_used_omap":8,"kb_used_meta":312823,"kb_avail":92663740,"statfs":{"total":95995035648,"available":94887669760,"internally_reserved":0,"allocated":787017728,"data_stored":786097575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":320331761},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":773132288,"data_stored":773128216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1196429312,"data_stored":1196425240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1631518720,"data_stored":1631518720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1929183232,"data_stored":1929183232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1260326912,"data_stored":1260322840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1680736256,"data_stored":1680736256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1254948864,"data_stored":1254948864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:11:44.469 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T10:11:44.469 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:11:44.470 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-21T10:11:44.470 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 110 0.05 2024-09-21T10:11:44.521 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:11:44.522 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:11:44.522 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:11:44.522 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:11:44.522 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:11:44.522 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:11:44.522 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:11:44.522 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:11:44.523 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:11:44.523 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:11:44.523 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:11:44.523 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:11:44.523 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:11:44.523 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:11:44.523 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:11:44.523 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:11:44.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:44.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:44.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:44.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:44 smithi196 ceph-mon[84784]: pgmap v561: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 872 op/s 2024-09-21T10:11:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:44 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1450712062' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:44 smithi196 ceph-mon[84784]: from='client.35559 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:11:44.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:44.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:44.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:44.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:11:44.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:44 smithi098 ceph-mon[97580]: pgmap v561: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 872 op/s 2024-09-21T10:11:44.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:44 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1450712062' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:44.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:44 smithi098 ceph-mon[97580]: from='client.35559 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:11:44.851 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:44 smithi098 ceph-mon[100914]: pgmap v561: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 872 op/s 2024-09-21T10:11:44.851 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:44 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1450712062' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:44.851 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:44 smithi098 ceph-mon[100914]: from='client.35559 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:11:44.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:44.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:44.911 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:44.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:45.012 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:45.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:45.046 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:45.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:45.147 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:45.170 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:45.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:45.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:45.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:45.285 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:11:45.285 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:11:45.285 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-21T10:11:45.285 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-21T10:11:45.286 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:11:45.338 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:45.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:45.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:45.440 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:45.467 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:45.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:45.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:45.568 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:45.610 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:45.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:45.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:45.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:45.747 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:45.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:45.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:45.848 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:45.876 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:45.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:45.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:45.978 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:46.006 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:46.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:46.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:46.108 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:46.136 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:46.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:46.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:46.238 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:46.273 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:46.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:46.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:46.374 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:46.398 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:46.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:46.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:46.500 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:46.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:46.541 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:46.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:46.576 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:11:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:46.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:46 smithi196 ceph-mon[84784]: pgmap v562: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 873 op/s 2024-09-21T10:11:46.643 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:46.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:46 smithi098 ceph-mon[97580]: pgmap v562: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 873 op/s 2024-09-21T10:11:46.662 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:46 smithi098 ceph-mon[100914]: pgmap v562: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 873 op/s 2024-09-21T10:11:46.665 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:46.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:46.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:46.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:46.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:46.851 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:46.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:46.952 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:46.991 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:46.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:46.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:47.092 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:47.120 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:47.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:47.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:47.221 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:47.256 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:47.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:47.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:47.358 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:47.402 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:47.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:47.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:47.504 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:47.534 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:47.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:47.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:47.636 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:47.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:47 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:11:47.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:47.674 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:47.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:47.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:47.823 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:47.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:47.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:47.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:47 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:11:47.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:47 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:11:47.925 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:47.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:47.982 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:47.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:48.083 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:48.113 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:48.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:48.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:48.215 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:48.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:48.264 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:48.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:48.366 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:48.408 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:48.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:48.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:48.510 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:48.539 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:48.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:48.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:48.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:48 smithi098 ceph-mon[97580]: pgmap v563: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 873 op/s 2024-09-21T10:11:48.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:48 smithi098 ceph-mon[100914]: pgmap v563: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 873 op/s 2024-09-21T10:11:48.642 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:48.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:48.682 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:48.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:48.730 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:48.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:48.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:48.837 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:48.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:48.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:48 smithi196 ceph-mon[84784]: pgmap v563: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 873 op/s 2024-09-21T10:11:48.938 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:48.966 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:48.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:48.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:49.067 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:49.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:49.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:49.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:49.203 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:49.228 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:49.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:49.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:49.330 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:49.363 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:49.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:49.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:49.465 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:49.491 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:49.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:49.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:49.592 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:49.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:49.626 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:49.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:49.726 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:49.751 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:49.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:49.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:49.852 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:49.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:49.891 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:49.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:49.937 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:11:49.992 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:50.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:50.041 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:50.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:50.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:50.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:50.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:50.306 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:50.351 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:50.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:50.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:50.453 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:50.490 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:50.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:50.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:50.591 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:50.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:50 smithi098 ceph-mon[97580]: pgmap v564: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 880 op/s 2024-09-21T10:11:50.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:50 smithi098 ceph-mon[100914]: pgmap v564: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 880 op/s 2024-09-21T10:11:50.632 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:50.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:50.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:50 smithi196 ceph-mon[84784]: pgmap v564: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 880 op/s 2024-09-21T10:11:50.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:11:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:11:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:11:50.733 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:50.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:50.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:50.873 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:50.920 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:50.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:50.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:51.023 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:51.044 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:51.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:51.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:51.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:11:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:11:50] "GET /metrics HTTP/1.1" 200 36452 "" "Prometheus/2.51.0" 2024-09-21T10:11:51.145 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:51.172 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:51.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:51.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:51.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:51.299 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:51.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:51.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:51.402 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:51.424 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:51.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:51.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:51.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:51.553 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:51.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:51.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:51.654 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:51.686 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:51.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:51.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:51.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:51.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:51.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:51.914 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:51.956 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:51.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:51.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:52.057 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:52.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:52.102 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:52.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:52.203 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:52.234 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:52.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:52.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:52.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:52.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:52.385 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:52.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:52.485 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:52.508 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:52.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:52.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:52.610 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:52.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:52.651 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:52.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:52.752 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:52.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:52.925 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:52.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:53.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:53.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:53.080 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:53.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:53.181 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:53.205 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:53.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:53.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:53.307 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:53.329 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:53.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:53.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:53.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:53 smithi098 ceph-mon[97580]: pgmap v565: 89 pgs: 89 active+clean; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 866 op/s 2024-09-21T10:11:53.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:11:53.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:11:53.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:53.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'2':'62260','3':'62260','6':'62260'}"} : dispatch 2024-09-21T10:11:53.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:53 smithi098 ceph-mon[100914]: pgmap v565: 89 pgs: 89 active+clean; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 866 op/s 2024-09-21T10:11:53.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:11:53.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:11:53.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:53.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'2':'62260','3':'62260','6':'62260'}"} : dispatch 2024-09-21T10:11:53.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:53 smithi196 ceph-mon[84784]: pgmap v565: 89 pgs: 89 active+clean; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 866 op/s 2024-09-21T10:11:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:11:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:11:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'2':'62260','3':'62260','6':'62260'}"} : dispatch 2024-09-21T10:11:53.431 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:53.452 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:53.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:53.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:53.554 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:53.575 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:53.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:53.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:53.624 INFO:teuthology.orchestra.run.smithi098.stdout:moved 2 / 267 (0.749064%) 2024-09-21T10:11:53.624 INFO:teuthology.orchestra.run.smithi098.stdout:avg 38.1429 2024-09-21T10:11:53.624 INFO:teuthology.orchestra.run.smithi098.stdout:stddev 5.64241 -> 5.56593 (expected baseline 5.71786) 2024-09-21T10:11:53.624 INFO:teuthology.orchestra.run.smithi098.stdout:min osd.5 with 30 -> 31 pgs (0.786517 -> 0.812734 * mean) 2024-09-21T10:11:53.624 INFO:teuthology.orchestra.run.smithi098.stdout:max osd.1 with 46 -> 46 pgs (1.20599 -> 1.20599 * mean) 2024-09-21T10:11:53.624 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:11:53.624 INFO:teuthology.orchestra.run.smithi098.stdout:oload 110 2024-09-21T10:11:53.624 INFO:teuthology.orchestra.run.smithi098.stdout:max_change 0.05 2024-09-21T10:11:53.624 INFO:teuthology.orchestra.run.smithi098.stdout:max_change_osds 4 2024-09-21T10:11:53.624 INFO:teuthology.orchestra.run.smithi098.stdout:average_utilization 0.0157 2024-09-21T10:11:53.625 INFO:teuthology.orchestra.run.smithi098.stdout:overload_utilization 0.0172 2024-09-21T10:11:53.625 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 weight 1.0000 -> 0.9500 2024-09-21T10:11:53.625 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 weight 1.0000 -> 0.9500 2024-09-21T10:11:53.625 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 weight 1.0000 -> 0.9500 2024-09-21T10:11:53.677 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:53.698 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:53.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:53.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:53.800 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:53.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:53.821 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:53.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:53.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:53.944 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:53.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:53.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:54.046 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:54.069 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:54.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:54.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:54.170 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:54.192 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:54.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:54.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:54.293 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:54.314 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:54.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:54.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:54.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:54 smithi098 ceph-mon[100914]: from='client.45437 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:11:54.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:54 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'62260','3':'62260','6':'62260'}"}]': finished 2024-09-21T10:11:54.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:54 smithi098 ceph-mon[100914]: osdmap e188: 8 total, 8 up, 7 in 2024-09-21T10:11:54.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:54 smithi098 ceph-mon[97580]: from='client.45437 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:11:54.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:54 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'62260','3':'62260','6':'62260'}"}]': finished 2024-09-21T10:11:54.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:54 smithi098 ceph-mon[97580]: osdmap e188: 8 total, 8 up, 7 in 2024-09-21T10:11:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:54 smithi196 ceph-mon[84784]: from='client.45437 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:11:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:54 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'62260','3':'62260','6':'62260'}"}]': finished 2024-09-21T10:11:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:54 smithi196 ceph-mon[84784]: osdmap e188: 8 total, 8 up, 7 in 2024-09-21T10:11:54.416 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:54.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:54.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:54.538 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:54.559 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:54.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:54.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:54.660 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:54.682 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:54.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:54.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:54.784 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:54.806 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:54.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:54.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:54.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:54.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:54.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:55.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:55.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:55.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:55.155 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:55.177 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:55.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:55.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:55.253 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:11:55.254 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-21T10:11:55.254 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-21T10:11:55.254 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:11:55.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.254 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:267693 5.0 5:02a5fe86:::benchmark_data_smithi098_135187_object132055:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:11:55.254 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:59.398968+0000", 2024-09-21T10:11:55.254 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 234.11303114399999, 2024-09-21T10:11:55.254 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.363874005, 2024-09-21T10:11:55.254 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:55.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:55.254 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:55.254 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:55.254 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:11:55.255 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:11:55.255 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 267693 2024-09-21T10:11:55.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.255 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:55.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:55.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.398968+0000", 2024-09-21T10:11:55.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:55.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.398968+0000", 2024-09-21T10:11:55.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.256 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:55.256 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.398970+0000", 2024-09-21T10:11:55.256 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0310000000000001e-06 2024-09-21T10:11:55.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.256 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:55.256 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.398977+0000", 2024-09-21T10:11:55.256 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1559999999999998e-06 2024-09-21T10:11:55.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.256 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:55.256 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.398999+0000", 2024-09-21T10:11:55.256 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1103000000000001e-05 2024-09-21T10:11:55.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.257 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:55.257 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399008+0000", 2024-09-21T10:11:55.257 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6350000000000001e-06 2024-09-21T10:11:55.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.257 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.257 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399115+0000", 2024-09-21T10:11:55.257 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106959 2024-09-21T10:11:55.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.257 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:11:55.257 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399130+0000", 2024-09-21T10:11:55.258 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4918e-05 2024-09-21T10:11:55.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.258 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.258 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.372566+0000", 2024-09-21T10:11:55.258 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.97343550599999995 2024-09-21T10:11:55.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.258 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:11:55.258 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.372578+0000", 2024-09-21T10:11:55.258 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2269000000000001e-05 2024-09-21T10:11:55.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.258 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.705405+0000", 2024-09-21T10:11:55.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3328273150000001 2024-09-21T10:11:55.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:55.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.705537+0000", 2024-09-21T10:11:55.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000131891 2024-09-21T10:11:55.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,1,4", 2024-09-21T10:11:55.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.706214+0000", 2024-09-21T10:11:55.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00067717799999999996 2024-09-21T10:11:55.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.260 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.260 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.740214+0000", 2024-09-21T10:11:55.260 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034000056000000001 2024-09-21T10:11:55.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.260 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.260 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.740297+0000", 2024-09-21T10:11:55.260 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2787999999999994e-05 2024-09-21T10:11:55.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.260 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:55.260 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.741535+0000", 2024-09-21T10:11:55.260 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001237602 2024-09-21T10:11:55.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.261 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.261 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.762486+0000", 2024-09-21T10:11:55.261 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020951805 2024-09-21T10:11:55.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.261 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:55.261 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.762689+0000", 2024-09-21T10:11:55.261 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020263099999999999 2024-09-21T10:11:55.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.261 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:55.261 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.762842+0000", 2024-09-21T10:11:55.261 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015316200000000001 2024-09-21T10:11:55.262 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.262 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:55.262 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.262 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:267694 5.1 5:88acc045:::benchmark_data_smithi098_135187_object132056:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:11:55.262 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:59.399033+0000", 2024-09-21T10:11:55.262 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 234.11296645100001, 2024-09-21T10:11:55.262 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2681572970000001, 2024-09-21T10:11:55.262 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:55.262 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:55.262 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:55.262 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:55.263 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:11:55.263 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:11:55.263 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 267694 2024-09-21T10:11:55.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.263 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:55.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.263 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:55.263 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399033+0000", 2024-09-21T10:11:55.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.263 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:55.263 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399033+0000", 2024-09-21T10:11:55.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:55.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399034+0000", 2024-09-21T10:11:55.264 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7800000000000001e-07 2024-09-21T10:11:55.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:55.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399036+0000", 2024-09-21T10:11:55.264 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8590000000000001e-06 2024-09-21T10:11:55.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:55.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399045+0000", 2024-09-21T10:11:55.264 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8950000000000004e-06 2024-09-21T10:11:55.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:55.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399051+0000", 2024-09-21T10:11:55.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6540000000000004e-06 2024-09-21T10:11:55.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399172+0000", 2024-09-21T10:11:55.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000121465 2024-09-21T10:11:55.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:11:55.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399182+0000", 2024-09-21T10:11:55.266 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9199999999999999e-06 2024-09-21T10:11:55.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.266 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.266 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.377429+0000", 2024-09-21T10:11:55.266 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.97824708500000002 2024-09-21T10:11:55.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.266 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:11:55.266 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.377440+0000", 2024-09-21T10:11:55.266 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0424999999999999e-05 2024-09-21T10:11:55.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.266 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.267 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.646211+0000", 2024-09-21T10:11:55.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2687708520000001 2024-09-21T10:11:55.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.267 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:55.267 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.646342+0000", 2024-09-21T10:11:55.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013128700000000001 2024-09-21T10:11:55.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.267 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4,7", 2024-09-21T10:11:55.267 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.646466+0000", 2024-09-21T10:11:55.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012438799999999999 2024-09-21T10:11:55.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.268 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.652333+0000", 2024-09-21T10:11:55.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058668069999999999 2024-09-21T10:11:55.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:55.268 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.661482+0000", 2024-09-21T10:11:55.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0091486180000000007 2024-09-21T10:11:55.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.268 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.662947+0000", 2024-09-21T10:11:55.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001465638 2024-09-21T10:11:55.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.666986+0000", 2024-09-21T10:11:55.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040387859999999999 2024-09-21T10:11:55.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:55.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.667053+0000", 2024-09-21T10:11:55.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6304999999999996e-05 2024-09-21T10:11:55.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:55.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.667190+0000", 2024-09-21T10:11:55.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000137735 2024-09-21T10:11:55.270 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.270 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:55.270 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.270 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:267704 5.1 5:8e7993d3:::benchmark_data_smithi098_135187_object132066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:11:55.270 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:59.421065+0000", 2024-09-21T10:11:55.270 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 234.09093486399999, 2024-09-21T10:11:55.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.250816414, 2024-09-21T10:11:55.270 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:55.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:55.270 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:55.270 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:55.271 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:11:55.271 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:11:55.271 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 267704 2024-09-21T10:11:55.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.271 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:55.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:55.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421065+0000", 2024-09-21T10:11:55.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:55.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421065+0000", 2024-09-21T10:11:55.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:55.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421066+0000", 2024-09-21T10:11:55.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.871e-06 2024-09-21T10:11:55.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:55.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421073+0000", 2024-09-21T10:11:55.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9059999999999996e-06 2024-09-21T10:11:55.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:55.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421091+0000", 2024-09-21T10:11:55.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7703e-05 2024-09-21T10:11:55.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:55.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421099+0000", 2024-09-21T10:11:55.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.464e-06 2024-09-21T10:11:55.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421173+0000", 2024-09-21T10:11:55.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4025999999999999e-05 2024-09-21T10:11:55.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:11:55.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421180+0000", 2024-09-21T10:11:55.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9240000000000007e-06 2024-09-21T10:11:55.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.380372+0000", 2024-09-21T10:11:55.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.95919200400000004 2024-09-21T10:11:55.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:11:55.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.380384+0000", 2024-09-21T10:11:55.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1955e-05 2024-09-21T10:11:55.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.652368+0000", 2024-09-21T10:11:55.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2719833110000001 2024-09-21T10:11:55.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:55.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.652497+0000", 2024-09-21T10:11:55.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012972 2024-09-21T10:11:55.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4,7", 2024-09-21T10:11:55.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.652632+0000", 2024-09-21T10:11:55.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013412799999999999 2024-09-21T10:11:55.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:55.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.661488+0000", 2024-09-21T10:11:55.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0088560349999999999 2024-09-21T10:11:55.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.661596+0000", 2024-09-21T10:11:55.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108306 2024-09-21T10:11:55.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.667234+0000", 2024-09-21T10:11:55.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056377440000000001 2024-09-21T10:11:55.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.277 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.671567+0000", 2024-09-21T10:11:55.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043329320000000003 2024-09-21T10:11:55.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:55.277 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.671746+0000", 2024-09-21T10:11:55.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017953199999999999 2024-09-21T10:11:55.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:55.278 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.671881+0000", 2024-09-21T10:11:55.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013485299999999999 2024-09-21T10:11:55.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.278 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:55.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.278 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269031 5.0 5:098330c7:::benchmark_data_smithi098_135187_object133393:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:11:55.278 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.462832+0000", 2024-09-21T10:11:55.278 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 229.049167098, 2024-09-21T10:11:55.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.042491235, 2024-09-21T10:11:55.278 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:55.279 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:55.279 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:55.279 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:55.279 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:11:55.279 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:11:55.279 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269031 2024-09-21T10:11:55.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.279 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:55.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.279 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:55.279 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462832+0000", 2024-09-21T10:11:55.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:55.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462832+0000", 2024-09-21T10:11:55.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:55.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462834+0000", 2024-09-21T10:11:55.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.68e-06 2024-09-21T10:11:55.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:55.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462840+0000", 2024-09-21T10:11:55.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6670000000000001e-06 2024-09-21T10:11:55.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:55.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462857+0000", 2024-09-21T10:11:55.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7028999999999999e-05 2024-09-21T10:11:55.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:55.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462866+0000", 2024-09-21T10:11:55.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9080000000000001e-06 2024-09-21T10:11:55.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462933+0000", 2024-09-21T10:11:55.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7100999999999993e-05 2024-09-21T10:11:55.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.282 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:11:55.282 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462943+0000", 2024-09-21T10:11:55.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0098999999999999e-05 2024-09-21T10:11:55.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.282 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.282 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.489116+0000", 2024-09-21T10:11:55.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0261731869999999 2024-09-21T10:11:55.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.283 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:55.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.489223+0000", 2024-09-21T10:11:55.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106709 2024-09-21T10:11:55.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.283 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:11:55.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.489331+0000", 2024-09-21T10:11:55.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010831399999999999 2024-09-21T10:11:55.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.283 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:55.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.498893+0000", 2024-09-21T10:11:55.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0095622350000000005 2024-09-21T10:11:55.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.500295+0000", 2024-09-21T10:11:55.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0014013249999999999 2024-09-21T10:11:55.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.505231+0000", 2024-09-21T10:11:55.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0049369000000000001 2024-09-21T10:11:55.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:55.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.505284+0000", 2024-09-21T10:11:55.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2510999999999999e-05 2024-09-21T10:11:55.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:55.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.505324+0000", 2024-09-21T10:11:55.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9570000000000002e-05 2024-09-21T10:11:55.285 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.285 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:55.285 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.286 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269041 5.1 5:89c4bb2f:::benchmark_data_smithi098_135187_object133403:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:11:55.286 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.486118+0000", 2024-09-21T10:11:55.286 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 229.02588117400001, 2024-09-21T10:11:55.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0245291729999999, 2024-09-21T10:11:55.286 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:55.286 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:55.286 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:55.286 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:55.286 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:11:55.286 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:11:55.286 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269041 2024-09-21T10:11:55.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.287 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:55.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:55.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.486118+0000", 2024-09-21T10:11:55.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:55.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.486118+0000", 2024-09-21T10:11:55.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:55.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.486121+0000", 2024-09-21T10:11:55.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2539999999999999e-06 2024-09-21T10:11:55.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:55.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.486128+0000", 2024-09-21T10:11:55.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4880000000000001e-06 2024-09-21T10:11:55.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:55.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.486145+0000", 2024-09-21T10:11:55.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7200000000000001e-05 2024-09-21T10:11:55.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:55.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.486154+0000", 2024-09-21T10:11:55.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1379999999999996e-06 2024-09-21T10:11:55.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.486211+0000", 2024-09-21T10:11:55.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6759000000000003e-05 2024-09-21T10:11:55.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:11:55.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.486217+0000", 2024-09-21T10:11:55.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3520000000000003e-06 2024-09-21T10:11:55.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.488241+0000", 2024-09-21T10:11:55.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.002023962 2024-09-21T10:11:55.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:55.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.488411+0000", 2024-09-21T10:11:55.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000169951 2024-09-21T10:11:55.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:11:55.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.488526+0000", 2024-09-21T10:11:55.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114225 2024-09-21T10:11:55.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.504331+0000", 2024-09-21T10:11:55.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015805092999999999 2024-09-21T10:11:55.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510365+0000", 2024-09-21T10:11:55.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0060348060000000002 2024-09-21T10:11:55.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:55.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510540+0000", 2024-09-21T10:11:55.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000174258 2024-09-21T10:11:55.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:55.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510601+0000", 2024-09-21T10:11:55.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1678000000000001e-05 2024-09-21T10:11:55.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:55.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510647+0000", 2024-09-21T10:11:55.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6008999999999999e-05 2024-09-21T10:11:55.292 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.292 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:55.292 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.293 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269050 5.1 5:811e02ab:::benchmark_data_smithi098_135187_object133412:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:11:55.293 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.496839+0000", 2024-09-21T10:11:55.293 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 229.01516059599999, 2024-09-21T10:11:55.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0140269550000001, 2024-09-21T10:11:55.293 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:55.293 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:55.293 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:55.293 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:55.293 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:11:55.293 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:11:55.293 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269050 2024-09-21T10:11:55.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.294 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:55.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:55.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496839+0000", 2024-09-21T10:11:55.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:55.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496839+0000", 2024-09-21T10:11:55.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:55.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496840+0000", 2024-09-21T10:11:55.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4899999999999999e-06 2024-09-21T10:11:55.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:55.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496848+0000", 2024-09-21T10:11:55.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2490000000000004e-06 2024-09-21T10:11:55.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:55.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496867+0000", 2024-09-21T10:11:55.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8949000000000001e-05 2024-09-21T10:11:55.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:55.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496877+0000", 2024-09-21T10:11:55.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0536e-05 2024-09-21T10:11:55.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496948+0000", 2024-09-21T10:11:55.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1159000000000001e-05 2024-09-21T10:11:55.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:11:55.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.496958+0000", 2024-09-21T10:11:55.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0169e-05 2024-09-21T10:11:55.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.493320+0000", 2024-09-21T10:11:55.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.99636132300000002 2024-09-21T10:11:55.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:55.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.493421+0000", 2024-09-21T10:11:55.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010134899999999999 2024-09-21T10:11:55.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:11:55.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.493525+0000", 2024-09-21T10:11:55.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000103624 2024-09-21T10:11:55.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.504366+0000", 2024-09-21T10:11:55.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010841151 2024-09-21T10:11:55.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510409+0000", 2024-09-21T10:11:55.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0060434240000000004 2024-09-21T10:11:55.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:55.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510741+0000", 2024-09-21T10:11:55.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00033127500000000002 2024-09-21T10:11:55.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:55.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510811+0000", 2024-09-21T10:11:55.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0066000000000003e-05 2024-09-21T10:11:55.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:55.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510866+0000", 2024-09-21T10:11:55.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5191000000000001e-05 2024-09-21T10:11:55.299 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.300 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:55.300 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.300 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269053 5.1 5:8f082889:::benchmark_data_smithi098_135187_object133415:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:11:55.300 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.502117+0000", 2024-09-21T10:11:55.300 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 229.009882202, 2024-09-21T10:11:55.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0088990069999999, 2024-09-21T10:11:55.300 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:55.300 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:55.300 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:55.300 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:55.300 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:11:55.300 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:11:55.301 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269053 2024-09-21T10:11:55.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.301 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:55.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.301 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:55.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502117+0000", 2024-09-21T10:11:55.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.301 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:55.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502117+0000", 2024-09-21T10:11:55.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:55.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502119+0000", 2024-09-21T10:11:55.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.561e-06 2024-09-21T10:11:55.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:55.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502125+0000", 2024-09-21T10:11:55.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8619999999999999e-06 2024-09-21T10:11:55.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:55.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502138+0000", 2024-09-21T10:11:55.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3349e-05 2024-09-21T10:11:55.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:55.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502145+0000", 2024-09-21T10:11:55.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6309999999999997e-06 2024-09-21T10:11:55.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502203+0000", 2024-09-21T10:11:55.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8208999999999998e-05 2024-09-21T10:11:55.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:11:55.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502214+0000", 2024-09-21T10:11:55.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1096e-05 2024-09-21T10:11:55.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.497567+0000", 2024-09-21T10:11:55.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.99535269800000004 2024-09-21T10:11:55.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:55.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.497669+0000", 2024-09-21T10:11:55.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000102094 2024-09-21T10:11:55.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:11:55.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.497772+0000", 2024-09-21T10:11:55.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000103498 2024-09-21T10:11:55.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.504400+0000", 2024-09-21T10:11:55.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0066274569999999998 2024-09-21T10:11:55.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510446+0000", 2024-09-21T10:11:55.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0060458229999999996 2024-09-21T10:11:55.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:55.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510873+0000", 2024-09-21T10:11:55.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042730600000000002 2024-09-21T10:11:55.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:55.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510977+0000", 2024-09-21T10:11:55.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010447599999999999 2024-09-21T10:11:55.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:55.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511016+0000", 2024-09-21T10:11:55.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8946999999999998e-05 2024-09-21T10:11:55.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.307 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:55.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.307 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269054 5.1 5:8b4ad385:::benchmark_data_smithi098_135187_object133416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:11:55.307 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.502494+0000", 2024-09-21T10:11:55.307 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 229.009505244, 2024-09-21T10:11:55.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0086457820000001, 2024-09-21T10:11:55.307 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:55.307 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:55.307 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:55.307 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:55.307 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:11:55.308 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:11:55.308 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269054 2024-09-21T10:11:55.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.308 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:55.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:55.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502494+0000", 2024-09-21T10:11:55.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:55.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502494+0000", 2024-09-21T10:11:55.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:55.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502495+0000", 2024-09-21T10:11:55.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7899999999999998e-07 2024-09-21T10:11:55.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:55.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502499+0000", 2024-09-21T10:11:55.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7220000000000001e-06 2024-09-21T10:11:55.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:55.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502507+0000", 2024-09-21T10:11:55.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8659999999999995e-06 2024-09-21T10:11:55.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:55.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502512+0000", 2024-09-21T10:11:55.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5820000000000003e-06 2024-09-21T10:11:55.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502561+0000", 2024-09-21T10:11:55.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8516000000000001e-05 2024-09-21T10:11:55.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:11:55.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.502569+0000", 2024-09-21T10:11:55.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6300000000000004e-06 2024-09-21T10:11:55.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.504428+0000", 2024-09-21T10:11:55.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.001859013 2024-09-21T10:11:55.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:55.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.504525+0000", 2024-09-21T10:11:55.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6582999999999993e-05 2024-09-21T10:11:55.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:11:55.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.504620+0000", 2024-09-21T10:11:55.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5313e-05 2024-09-21T10:11:55.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510486+0000", 2024-09-21T10:11:55.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058654859999999996 2024-09-21T10:11:55.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510519+0000", 2024-09-21T10:11:55.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3717999999999998e-05 2024-09-21T10:11:55.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:55.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511023+0000", 2024-09-21T10:11:55.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00050323200000000005 2024-09-21T10:11:55.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:55.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511094+0000", 2024-09-21T10:11:55.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1662000000000006e-05 2024-09-21T10:11:55.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:55.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511140+0000", 2024-09-21T10:11:55.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5779999999999999e-05 2024-09-21T10:11:55.313 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.313 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:55.313 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.314 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269056 5.0 5:097ecfd1:::benchmark_data_smithi098_135187_object133418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:11:55.314 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.505026+0000", 2024-09-21T10:11:55.314 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 229.00697381500001, 2024-09-21T10:11:55.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0033554609999999, 2024-09-21T10:11:55.314 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:55.314 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:55.314 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:55.314 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:55.314 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:11:55.314 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:11:55.314 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269056 2024-09-21T10:11:55.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.315 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:55.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:55.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505026+0000", 2024-09-21T10:11:55.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:55.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505026+0000", 2024-09-21T10:11:55.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:55.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505027+0000", 2024-09-21T10:11:55.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6300000000000001e-06 2024-09-21T10:11:55.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:55.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505035+0000", 2024-09-21T10:11:55.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5700000000000004e-06 2024-09-21T10:11:55.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:55.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505062+0000", 2024-09-21T10:11:55.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7098999999999999e-05 2024-09-21T10:11:55.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:55.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505080+0000", 2024-09-21T10:11:55.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7603000000000001e-05 2024-09-21T10:11:55.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505219+0000", 2024-09-21T10:11:55.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000139732 2024-09-21T10:11:55.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:11:55.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.505229+0000", 2024-09-21T10:11:55.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7310000000000005e-06 2024-09-21T10:11:55.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.493545+0000", 2024-09-21T10:11:55.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.98831621599999997 2024-09-21T10:11:55.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:55.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.493636+0000", 2024-09-21T10:11:55.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0378999999999996e-05 2024-09-21T10:11:55.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:11:55.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.493732+0000", 2024-09-21T10:11:55.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6831999999999997e-05 2024-09-21T10:11:55.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:55.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.504194+0000", 2024-09-21T10:11:55.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010461981 2024-09-21T10:11:55.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.506161+0000", 2024-09-21T10:11:55.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0019666000000000002 2024-09-21T10:11:55.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.508250+0000", 2024-09-21T10:11:55.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002089025 2024-09-21T10:11:55.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:55.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.508337+0000", 2024-09-21T10:11:55.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7369999999999999e-05 2024-09-21T10:11:55.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:55.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.508381+0000", 2024-09-21T10:11:55.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3693000000000003e-05 2024-09-21T10:11:55.320 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.320 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:55.320 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.320 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:68389 6.6 6:6cc4d7d2:::benchmark_data_smithi098_169033_object68388:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T10:11:55.320 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:09:25.280343+0000", 2024-09-21T10:11:55.321 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 148.23165693000001, 2024-09-21T10:11:55.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.15904647599999999, 2024-09-21T10:11:55.321 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:55.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:55.321 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:55.321 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:55.321 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:11:55.321 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:11:55.321 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 68389 2024-09-21T10:11:55.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.321 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:55.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:55.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.280343+0000", 2024-09-21T10:11:55.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:55.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.280343+0000", 2024-09-21T10:11:55.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:55.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.280343+0000", 2024-09-21T10:11:55.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.75e-07 2024-09-21T10:11:55.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:55.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.280362+0000", 2024-09-21T10:11:55.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9304e-05 2024-09-21T10:11:55.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:55.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.280371+0000", 2024-09-21T10:11:55.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5949999999999999e-06 2024-09-21T10:11:55.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:55.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.280377+0000", 2024-09-21T10:11:55.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1539999999999999e-06 2024-09-21T10:11:55.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.283368+0000", 2024-09-21T10:11:55.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029912250000000001 2024-09-21T10:11:55.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:55.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.283526+0000", 2024-09-21T10:11:55.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015740400000000001 2024-09-21T10:11:55.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:11:55.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.283656+0000", 2024-09-21T10:11:55.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013055099999999999 2024-09-21T10:11:55.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290009+0000", 2024-09-21T10:11:55.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0063528109999999999 2024-09-21T10:11:55.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290075+0000", 2024-09-21T10:11:55.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5866999999999998e-05 2024-09-21T10:11:55.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:55.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.439286+0000", 2024-09-21T10:11:55.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.14921104199999999 2024-09-21T10:11:55.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:55.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.439336+0000", 2024-09-21T10:11:55.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0167999999999998e-05 2024-09-21T10:11:55.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:55.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.439389+0000", 2024-09-21T10:11:55.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2779999999999999e-05 2024-09-21T10:11:55.327 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.327 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:55.327 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.327 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.45263.0:68384 6.f e171/165 6:f1664d9c:::benchmark_data_smithi098_169033_object68383:head v 171'4173, mlcod=171'4170)", 2024-09-21T10:11:55.327 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:09:25.283223+0000", 2024-09-21T10:11:55.327 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 148.22877646699999, 2024-09-21T10:11:55.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.164139595, 2024-09-21T10:11:55.327 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:55.327 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:55.327 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:55.327 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:55.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:55.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.283223+0000", 2024-09-21T10:11:55.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:55.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.283223+0000", 2024-09-21T10:11:55.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:55.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.283224+0000", 2024-09-21T10:11:55.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9299999999999997e-07 2024-09-21T10:11:55.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:55.329 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.283244+0000", 2024-09-21T10:11:55.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.033e-05 2024-09-21T10:11:55.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:55.329 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.283253+0000", 2024-09-21T10:11:55.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7960000000000001e-06 2024-09-21T10:11:55.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:55.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.283259+0000", 2024-09-21T10:11:55.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2979999999999997e-06 2024-09-21T10:11:55.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.286608+0000", 2024-09-21T10:11:55.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033488739999999999 2024-09-21T10:11:55.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:55.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.286627+0000", 2024-09-21T10:11:55.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9006999999999999e-05 2024-09-21T10:11:55.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:55.331 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.444514+0000", 2024-09-21T10:11:55.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.15788702900000001 2024-09-21T10:11:55.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:55.331 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.447363+0000", 2024-09-21T10:11:55.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028485680000000001 2024-09-21T10:11:55.331 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.331 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:55.331 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.331 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:68394 6.6 6:66d47173:::benchmark_data_smithi098_169033_object68393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T10:11:55.332 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:09:25.285778+0000", 2024-09-21T10:11:55.332 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 148.22622148799999, 2024-09-21T10:11:55.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.156564019, 2024-09-21T10:11:55.332 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:55.332 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:55.332 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:55.332 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:55.332 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:11:55.332 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:11:55.332 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 68394 2024-09-21T10:11:55.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.332 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:55.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.332 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:55.333 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.285778+0000", 2024-09-21T10:11:55.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.333 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:55.333 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.285778+0000", 2024-09-21T10:11:55.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.333 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:55.333 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.285779+0000", 2024-09-21T10:11:55.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.313e-06 2024-09-21T10:11:55.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.333 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:55.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.285802+0000", 2024-09-21T10:11:55.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2483000000000001e-05 2024-09-21T10:11:55.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:55.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.285811+0000", 2024-09-21T10:11:55.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9099999999999994e-06 2024-09-21T10:11:55.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:55.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.285817+0000", 2024-09-21T10:11:55.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3770000000000001e-06 2024-09-21T10:11:55.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.286709+0000", 2024-09-21T10:11:55.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00089149900000000002 2024-09-21T10:11:55.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:55.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.286817+0000", 2024-09-21T10:11:55.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010882699999999999 2024-09-21T10:11:55.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:11:55.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.287057+0000", 2024-09-21T10:11:55.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000239955 2024-09-21T10:11:55.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.336 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.336 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290041+0000", 2024-09-21T10:11:55.336 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029835410000000001 2024-09-21T10:11:55.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.336 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:55.336 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.439399+0000", 2024-09-21T10:11:55.336 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.14935817800000001 2024-09-21T10:11:55.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.336 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.336 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.442248+0000", 2024-09-21T10:11:55.336 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028491010000000002 2024-09-21T10:11:55.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.337 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:55.337 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.442297+0000", 2024-09-21T10:11:55.337 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8804000000000002e-05 2024-09-21T10:11:55.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.337 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:55.337 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.442342+0000", 2024-09-21T10:11:55.337 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5031000000000003e-05 2024-09-21T10:11:55.337 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.337 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:55.337 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.338 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.45263.0:68395 6.f e171/165 6:f5675bf7:::benchmark_data_smithi098_169033_object68394:head v 171'4174, mlcod=171'4170)", 2024-09-21T10:11:55.338 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:09:25.286791+0000", 2024-09-21T10:11:55.338 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 148.225208364, 2024-09-21T10:11:55.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.160608741, 2024-09-21T10:11:55.338 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:55.338 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:55.338 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:55.338 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:55.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.338 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:55.338 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.286791+0000", 2024-09-21T10:11:55.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.286791+0000", 2024-09-21T10:11:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.286792+0000", 2024-09-21T10:11:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7400000000000003e-07 2024-09-21T10:11:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.286820+0000", 2024-09-21T10:11:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8670000000000002e-05 2024-09-21T10:11:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.286830+0000", 2024-09-21T10:11:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9639999999999998e-06 2024-09-21T10:11:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.286836+0000", 2024-09-21T10:11:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0410000000000003e-06 2024-09-21T10:11:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.341 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.288829+0000", 2024-09-21T10:11:55.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0019928849999999998 2024-09-21T10:11:55.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:55.341 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.288879+0000", 2024-09-21T10:11:55.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9551e-05 2024-09-21T10:11:55.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:55.341 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.447368+0000", 2024-09-21T10:11:55.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.15848926099999999 2024-09-21T10:11:55.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.342 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:55.342 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.447400+0000", 2024-09-21T10:11:55.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1695000000000001e-05 2024-09-21T10:11:55.342 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.342 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:55.342 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.342 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:68399 6.6 6:62a5fa8a:::benchmark_data_smithi098_169033_object68398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T10:11:55.342 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:09:25.289291+0000", 2024-09-21T10:11:55.342 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 148.222708643, 2024-09-21T10:11:55.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.15372851700000001, 2024-09-21T10:11:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:11:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:11:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 68399 2024-09-21T10:11:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.289291+0000", 2024-09-21T10:11:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:55.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.289291+0000", 2024-09-21T10:11:55.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:55.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.289292+0000", 2024-09-21T10:11:55.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.654e-06 2024-09-21T10:11:55.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:55.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.289323+0000", 2024-09-21T10:11:55.345 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0620000000000002e-05 2024-09-21T10:11:55.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.345 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:55.345 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.289332+0000", 2024-09-21T10:11:55.345 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4329999999999993e-06 2024-09-21T10:11:55.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.345 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:55.345 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.289338+0000", 2024-09-21T10:11:55.345 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6710000000000004e-06 2024-09-21T10:11:55.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.345 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.345 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290101+0000", 2024-09-21T10:11:55.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000762428 2024-09-21T10:11:55.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:55.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290211+0000", 2024-09-21T10:11:55.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000110363 2024-09-21T10:11:55.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:11:55.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290330+0000", 2024-09-21T10:11:55.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000119505 2024-09-21T10:11:55.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:55.347 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.439404+0000", 2024-09-21T10:11:55.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.14907330199999999 2024-09-21T10:11:55.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.347 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.347 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.442379+0000", 2024-09-21T10:11:55.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029754059999999999 2024-09-21T10:11:55.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.347 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.347 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.442946+0000", 2024-09-21T10:11:55.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00056695400000000005 2024-09-21T10:11:55.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:55.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.442986+0000", 2024-09-21T10:11:55.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0099999999999999e-05 2024-09-21T10:11:55.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:55.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.443019+0000", 2024-09-21T10:11:55.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3080999999999997e-05 2024-09-21T10:11:55.348 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.348 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:55.348 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.348 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:68400 6.2 6:44e8f439:::benchmark_data_smithi098_169033_object68399:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T10:11:55.348 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:09:25.289351+0000", 2024-09-21T10:11:55.348 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 148.22264863699999, 2024-09-21T10:11:55.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.15354641599999999, 2024-09-21T10:11:55.349 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:55.349 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:55.349 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:55.349 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:55.349 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:11:55.349 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:11:55.349 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 68400 2024-09-21T10:11:55.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.349 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:55.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:55.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.289351+0000", 2024-09-21T10:11:55.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:55.350 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.289351+0000", 2024-09-21T10:11:55.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:55.350 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.289351+0000", 2024-09-21T10:11:55.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1899999999999998e-07 2024-09-21T10:11:55.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:55.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.289372+0000", 2024-09-21T10:11:55.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0715000000000001e-05 2024-09-21T10:11:55.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:55.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.289378+0000", 2024-09-21T10:11:55.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.224e-06 2024-09-21T10:11:55.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:55.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.289383+0000", 2024-09-21T10:11:55.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7940000000000002e-06 2024-09-21T10:11:55.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.289507+0000", 2024-09-21T10:11:55.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000124306 2024-09-21T10:11:55.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:55.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.289640+0000", 2024-09-21T10:11:55.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000132552 2024-09-21T10:11:55.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:11:55.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.289763+0000", 2024-09-21T10:11:55.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123139 2024-09-21T10:11:55.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:55.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.439505+0000", 2024-09-21T10:11:55.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.14974164300000001 2024-09-21T10:11:55.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.442670+0000", 2024-09-21T10:11:55.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003165242 2024-09-21T10:11:55.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.442816+0000", 2024-09-21T10:11:55.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014615900000000001 2024-09-21T10:11:55.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:55.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.442862+0000", 2024-09-21T10:11:55.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6167e-05 2024-09-21T10:11:55.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:55.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.442897+0000", 2024-09-21T10:11:55.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5055999999999998e-05 2024-09-21T10:11:55.354 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.354 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:55.354 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:68401 6.6 6:6574ed13:::benchmark_data_smithi098_169033_object68400:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T10:11:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:09:25.290645+0000", 2024-09-21T10:11:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 148.221354865, 2024-09-21T10:11:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.153102707, 2024-09-21T10:11:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:11:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:11:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 68401 2024-09-21T10:11:55.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290645+0000", 2024-09-21T10:11:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290645+0000", 2024-09-21T10:11:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:55.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290646+0000", 2024-09-21T10:11:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2440000000000001e-06 2024-09-21T10:11:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290666+0000", 2024-09-21T10:11:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9967e-05 2024-09-21T10:11:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290676+0000", 2024-09-21T10:11:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0665999999999999e-05 2024-09-21T10:11:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290683+0000", 2024-09-21T10:11:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0029999999999997e-06 2024-09-21T10:11:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.439553+0000", 2024-09-21T10:11:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.148869217 2024-09-21T10:11:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.439697+0000", 2024-09-21T10:11:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000144432 2024-09-21T10:11:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.358 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:11:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.439830+0000", 2024-09-21T10:11:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000133394 2024-09-21T10:11:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.443034+0000", 2024-09-21T10:11:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003203624 2024-09-21T10:11:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.443617+0000", 2024-09-21T10:11:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00058324500000000003 2024-09-21T10:11:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.443657+0000", 2024-09-21T10:11:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9545999999999997e-05 2024-09-21T10:11:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.443709+0000", 2024-09-21T10:11:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2543000000000002e-05 2024-09-21T10:11:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.443747+0000", 2024-09-21T10:11:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7826e-05 2024-09-21T10:11:55.360 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:68402 6.c 6:360b64bf:::benchmark_data_smithi098_169033_object68401:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T10:11:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:09:25.290700+0000", 2024-09-21T10:11:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 148.221299141, 2024-09-21T10:11:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.16906326599999999, 2024-09-21T10:11:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:55.361 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:11:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:11:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 68402 2024-09-21T10:11:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290700+0000", 2024-09-21T10:11:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290700+0000", 2024-09-21T10:11:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290701+0000", 2024-09-21T10:11:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1699999999999999e-07 2024-09-21T10:11:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290720+0000", 2024-09-21T10:11:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9349999999999999e-05 2024-09-21T10:11:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:55.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290727+0000", 2024-09-21T10:11:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8689999999999998e-06 2024-09-21T10:11:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290732+0000", 2024-09-21T10:11:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2059999999999998e-06 2024-09-21T10:11:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290791+0000", 2024-09-21T10:11:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9249000000000001e-05 2024-09-21T10:11:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290916+0000", 2024-09-21T10:11:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012441500000000001 2024-09-21T10:11:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:11:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.291031+0000", 2024-09-21T10:11:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114796 2024-09-21T10:11:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.445693+0000", 2024-09-21T10:11:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.154662091 2024-09-21T10:11:55.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.447605+0000", 2024-09-21T10:11:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001911949 2024-09-21T10:11:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.459642+0000", 2024-09-21T10:11:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012037612 2024-09-21T10:11:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:55.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.459721+0000", 2024-09-21T10:11:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8348000000000003e-05 2024-09-21T10:11:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.459764+0000", 2024-09-21T10:11:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2963999999999997e-05 2024-09-21T10:11:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:68403 6.7 6:ec90c830:::benchmark_data_smithi098_169033_object68402:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T10:11:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:09:25.290802+0000", 2024-09-21T10:11:55.367 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 148.22119769099999, 2024-09-21T10:11:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.158664888, 2024-09-21T10:11:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:11:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:11:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 68403 2024-09-21T10:11:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290802+0000", 2024-09-21T10:11:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290802+0000", 2024-09-21T10:11:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290802+0000", 2024-09-21T10:11:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7700000000000004e-07 2024-09-21T10:11:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290834+0000", 2024-09-21T10:11:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1822999999999997e-05 2024-09-21T10:11:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290846+0000", 2024-09-21T10:11:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.201e-05 2024-09-21T10:11:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.290852+0000", 2024-09-21T10:11:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6239999999999999e-06 2024-09-21T10:11:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.291065+0000", 2024-09-21T10:11:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000213229 2024-09-21T10:11:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.439552+0000", 2024-09-21T10:11:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.14848686999999999 2024-09-21T10:11:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:11:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.439701+0000", 2024-09-21T10:11:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000148739 2024-09-21T10:11:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.447222+0000", 2024-09-21T10:11:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0075210820000000001 2024-09-21T10:11:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.447258+0000", 2024-09-21T10:11:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6220999999999998e-05 2024-09-21T10:11:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.449383+0000", 2024-09-21T10:11:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002124674 2024-09-21T10:11:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.449428+0000", 2024-09-21T10:11:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5720000000000003e-05 2024-09-21T10:11:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.449467+0000", 2024-09-21T10:11:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8219000000000001e-05 2024-09-21T10:11:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.45263.0:68405 6.4 e171/165 6:2046ec29:::benchmark_data_smithi098_169033_object68404:head v 171'4260, mlcod=171'4258)", 2024-09-21T10:11:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:09:25.291935+0000", 2024-09-21T10:11:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 148.22006451999999, 2024-09-21T10:11:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.159303319, 2024-09-21T10:11:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.291935+0000", 2024-09-21T10:11:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.291935+0000", 2024-09-21T10:11:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.291936+0000", 2024-09-21T10:11:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.057e-06 2024-09-21T10:11:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.291964+0000", 2024-09-21T10:11:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7674e-05 2024-09-21T10:11:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.439197+0000", 2024-09-21T10:11:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.147233429 2024-09-21T10:11:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.439220+0000", 2024-09-21T10:11:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3010000000000002e-05 2024-09-21T10:11:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.440495+0000", 2024-09-21T10:11:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0012752320000000001 2024-09-21T10:11:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.440516+0000", 2024-09-21T10:11:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0877999999999999e-05 2024-09-21T10:11:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.451201+0000", 2024-09-21T10:11:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010684391999999999 2024-09-21T10:11:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.451238+0000", 2024-09-21T10:11:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7647000000000001e-05 2024-09-21T10:11:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:68406 6.2 6:462e9e4c:::benchmark_data_smithi098_169033_object68405:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T10:11:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:09:25.292008+0000", 2024-09-21T10:11:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 148.21999188999999, 2024-09-21T10:11:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.15706932400000001, 2024-09-21T10:11:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:11:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:11:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:11:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:11:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:11:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 68406 2024-09-21T10:11:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:11:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:11:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.292008+0000", 2024-09-21T10:11:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:11:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.292008+0000", 2024-09-21T10:11:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:11:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:11:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.292009+0000", 2024-09-21T10:11:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1009999999999999e-06 2024-09-21T10:11:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:11:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.292028+0000", 2024-09-21T10:11:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9312000000000001e-05 2024-09-21T10:11:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:11:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.439561+0000", 2024-09-21T10:11:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.14753329900000001 2024-09-21T10:11:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:11:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.439573+0000", 2024-09-21T10:11:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.216e-05 2024-09-21T10:11:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:11:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.439982+0000", 2024-09-21T10:11:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00040811100000000001 2024-09-21T10:11:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:11:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.440116+0000", 2024-09-21T10:11:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000134031 2024-09-21T10:11:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:11:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.440252+0000", 2024-09-21T10:11:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013666500000000001 2024-09-21T10:11:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:11:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.442911+0000", 2024-09-21T10:11:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026583650000000002 2024-09-21T10:11:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.444419+0000", 2024-09-21T10:11:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0015085879999999999 2024-09-21T10:11:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.448971+0000", 2024-09-21T10:11:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0045517439999999999 2024-09-21T10:11:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:11:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.449036+0000", 2024-09-21T10:11:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5395000000000007e-05 2024-09-21T10:11:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:11:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:11:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:11:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:09:25.449077+0000", 2024-09-21T10:11:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0553000000000003e-05 2024-09-21T10:11:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:11:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:11:55.384 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:11:55.384 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:55.385 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:55 smithi098 ceph-mon[97580]: pgmap v567: 89 pgs: 89 active+clean; 2.4 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 878 op/s 2024-09-21T10:11:55.385 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:55 smithi098 ceph-mon[97580]: osdmap e189: 8 total, 8 up, 7 in 2024-09-21T10:11:55.385 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:55 smithi098 ceph-mon[100914]: pgmap v567: 89 pgs: 89 active+clean; 2.4 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 878 op/s 2024-09-21T10:11:55.386 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:55 smithi098 ceph-mon[100914]: osdmap e189: 8 total, 8 up, 7 in 2024-09-21T10:11:55.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:55 smithi196 ceph-mon[84784]: pgmap v567: 89 pgs: 89 active+clean; 2.4 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 878 op/s 2024-09-21T10:11:55.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:55 smithi196 ceph-mon[84784]: osdmap e189: 8 total, 8 up, 7 in 2024-09-21T10:11:55.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:55.429 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:55.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:55.530 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:55.559 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:55.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:55.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:55.661 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:55.703 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:55.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:55.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:55.805 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:55.836 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:55.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:55.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:55.938 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:55.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:55.967 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:55.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:56.066 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:11:56.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:56.107 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:56.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:56.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:56.208 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:56.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:56.243 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:56.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:56.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:56.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:56 smithi098 ceph-mon[97580]: osdmap e190: 8 total, 8 up, 7 in 2024-09-21T10:11:56.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:56 smithi098 ceph-mon[97580]: 6.6 starting backfill to osd.5 from (0'0,0'0] MAX to 187'12863 2024-09-21T10:11:56.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:56 smithi098 ceph-mon[100914]: osdmap e190: 8 total, 8 up, 7 in 2024-09-21T10:11:56.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:56 smithi098 ceph-mon[100914]: 6.6 starting backfill to osd.5 from (0'0,0'0] MAX to 187'12863 2024-09-21T10:11:56.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:56.379 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:56.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:56.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:56 smithi196 ceph-mon[84784]: osdmap e190: 8 total, 8 up, 7 in 2024-09-21T10:11:56.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:56 smithi196 ceph-mon[84784]: 6.6 starting backfill to osd.5 from (0'0,0'0] MAX to 187'12863 2024-09-21T10:11:56.437 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:11:56.480 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:56.510 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:56.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:56.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:56.612 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:56.653 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:56.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:56.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:56.755 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:56.791 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:56.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:56.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:56.893 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:56.920 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:56.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:56.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:56.977 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-21T10:11:56.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 1000 3.0 2024-09-21T10:11:57.021 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:57.028 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:57.043 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:57.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:57.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:57.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:57 smithi196 ceph-mon[84784]: pgmap v570: 89 pgs: 89 active+clean; 2.4 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 777 op/s 2024-09-21T10:11:57.145 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:57.170 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:57.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:57.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:57.188 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:57 smithi098 ceph-mon[97580]: pgmap v570: 89 pgs: 89 active+clean; 2.4 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 777 op/s 2024-09-21T10:11:57.188 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:57 smithi098 ceph-mon[100914]: pgmap v570: 89 pgs: 89 active+clean; 2.4 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 777 op/s 2024-09-21T10:11:57.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:57.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:57.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:57.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:57.485 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:57.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:57.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:57.587 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:57.635 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:57.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:57.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:57.737 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:57.779 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:57.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:57.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:57.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:57.906 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:57.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:57.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:58.009 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:58.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:58.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:58.144 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:58.168 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:58.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:58.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:58.270 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:58.299 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:58.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:58.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:58.401 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:58.440 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:58.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:58.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:58.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:58.580 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:58.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:58.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:58.612 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:11:58.612 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:11:58.612 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-21T10:11:58.612 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:11:58.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:58.721 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:58.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:58.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:58.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:58.851 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:58.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:58.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:58.953 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:58.980 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:58.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:58.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:59.071 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:11:59.082 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:59.136 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:59.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:59.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:59.149 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:59 smithi098 ceph-mon[97580]: pgmap v571: 89 pgs: 89 active+clean; 2.3 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 640 op/s 2024-09-21T10:11:59.149 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:11:59 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3113376621' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:11:59.149 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:59 smithi098 ceph-mon[100914]: pgmap v571: 89 pgs: 89 active+clean; 2.3 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 640 op/s 2024-09-21T10:11:59.149 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:11:59 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3113376621' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:11:59.224 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:59 smithi196 ceph-mon[84784]: pgmap v571: 89 pgs: 89 active+clean; 2.3 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 640 op/s 2024-09-21T10:11:59.224 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:11:59 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3113376621' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:11:59.238 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:59.258 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:59.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:59.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:59.360 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:59.396 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:59.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:59.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:59.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:59.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:59.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:59.640 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:59.665 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:11:59.676 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:59.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:59.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:59.777 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:59.818 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:11:59.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:59.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:59.920 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:59.962 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:11:59.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:59.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:00.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:00.083 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:12:00.100 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:00.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:00.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:00.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:00.242 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:00.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:00.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:00.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:00 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:12:00.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:00 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:12:00.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:00 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3113376621' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:12:00.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:00 smithi098 ceph-mon[97580]: osdmap e191: 8 total, 8 up, 7 in 2024-09-21T10:12:00.292 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:00 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:12:00.292 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:00 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:12:00.293 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:00 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3113376621' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:12:00.293 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:00 smithi098 ceph-mon[100914]: osdmap e191: 8 total, 8 up, 7 in 2024-09-21T10:12:00.293 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:12:00.293 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:12:00.343 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:12:00.343 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:12:00.344 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:00.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:00 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:12:00.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:00 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:12:00.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:00 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3113376621' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:12:00.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:00 smithi196 ceph-mon[84784]: osdmap e191: 8 total, 8 up, 7 in 2024-09-21T10:12:00.351 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:12:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:12:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:12:00.361 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:00.386 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:12:00.386 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:12:00.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:00.401 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:00.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:00.436 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:12:00.436 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:12:00.479 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:12:00.480 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:12:00.502 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:00.514 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:12:00.515 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:12:00.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:00.532 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:00.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:00.542 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:12:00.542 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:12:00.574 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:12:00.574 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:12:00.632 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:00.661 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:00.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:00.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:00.763 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:00.788 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:00.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:00.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:00.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:12:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:12:00] "GET /metrics HTTP/1.1" 200 36449 "" "Prometheus/2.51.0" 2024-09-21T10:12:00.890 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:00.916 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:00.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:00.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:01.018 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:01.045 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:01.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:01.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:01.146 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:01.174 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:01.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:01.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:01.275 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:01.311 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:01.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:01.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:01.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:01 smithi196 ceph-mon[84784]: pgmap v573: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 2.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 618 op/s; 1974/108840 objects misplaced (1.814%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:12:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:01 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2918541146' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:01.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:01 smithi098 ceph-mon[97580]: pgmap v573: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 2.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 618 op/s; 1974/108840 objects misplaced (1.814%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:12:01.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:01 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2918541146' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:01.410 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:01 smithi098 ceph-mon[100914]: pgmap v573: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 2.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 618 op/s; 1974/108840 objects misplaced (1.814%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:12:01.410 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:01 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2918541146' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:01.413 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:01.437 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:01.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:01.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:01.538 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:01.571 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:01.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:01.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:01.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:01.707 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:01.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:01.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:01.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:01.834 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:12:01.834 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:12:01.842 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:01.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:01.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:01.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:01.974 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:01.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:01.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:01.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:12:02.076 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:02.141 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:02.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:02.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:02.243 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:02.274 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:02.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:02.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:02.375 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:02.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:02.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:02.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:02.510 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:02.557 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:12:02.560 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:02.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:02.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:02.601 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:02.664 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:02.692 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:02.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:02.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:02.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:02.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:02.834 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:02.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:02.910 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:12:02.935 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:02.976 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:02.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:02.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:03.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:03.111 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:03.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:03.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:03.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:03.267 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:03.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:03.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:03.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:03 smithi196 ceph-mon[84784]: pgmap v574: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 2.1 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 553 op/s; 1974/104715 objects misplaced (1.885%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:12:03.368 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:03.402 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:03.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:03.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:03.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:03 smithi098 ceph-mon[100914]: pgmap v574: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 2.1 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 553 op/s; 1974/104715 objects misplaced (1.885%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:12:03.423 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:03 smithi098 ceph-mon[97580]: pgmap v574: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 2.1 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 553 op/s; 1974/104715 objects misplaced (1.885%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:12:03.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:12:03.504 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:03.526 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:03.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:03.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:03.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:03.700 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:03.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:03.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:03.803 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:03.828 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:03.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:03.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:03.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:03.970 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:03.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:03.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:04.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:04.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:04.112 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:04.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:04.213 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:04.252 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:04.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:04.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:04.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:04.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:04.398 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:04.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:04.499 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:04.526 INFO:teuthology.orchestra.run.smithi098.stdout:moved 0 / 267 (0%) 2024-09-21T10:12:04.526 INFO:teuthology.orchestra.run.smithi098.stdout:avg 38.1429 2024-09-21T10:12:04.526 INFO:teuthology.orchestra.run.smithi098.stdout:stddev 5.56593 -> 5.56593 (expected baseline 5.71786) 2024-09-21T10:12:04.526 INFO:teuthology.orchestra.run.smithi098.stdout:min osd.2 with 30 -> 30 pgs (0.786517 -> 0.786517 * mean) 2024-09-21T10:12:04.526 INFO:teuthology.orchestra.run.smithi098.stdout:max osd.1 with 46 -> 46 pgs (1.20599 -> 1.20599 * mean) 2024-09-21T10:12:04.526 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:12:04.526 INFO:teuthology.orchestra.run.smithi098.stdout:oload 1000 2024-09-21T10:12:04.526 INFO:teuthology.orchestra.run.smithi098.stdout:max_change 3 2024-09-21T10:12:04.526 INFO:teuthology.orchestra.run.smithi098.stdout:max_change_osds 4 2024-09-21T10:12:04.526 INFO:teuthology.orchestra.run.smithi098.stdout:average_utilization 0.0132 2024-09-21T10:12:04.527 INFO:teuthology.orchestra.run.smithi098.stdout:overload_utilization 0.1325 2024-09-21T10:12:04.527 INFO:teuthology.orchestra.run.smithi098.stderr:no change 2024-09-21T10:12:04.530 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:04.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:04.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:04.613 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:12:04.613 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:12:04.632 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:04.664 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:04.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:04.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:04.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:04.790 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:04.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:04.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:04.892 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:04.922 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:04.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:04.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:05.023 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:05.056 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:05.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:05.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:05.158 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:05.206 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:05.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:05.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:05.308 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:05.336 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:05.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:05.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:05.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:05 smithi196 ceph-mon[84784]: pgmap v575: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 2.0 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 694 op/s; 1196/99087 objects misplaced (1.207%); 6.4 MiB/s, 102 objects/s recovering 2024-09-21T10:12:05.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:05.468 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:05.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:05.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:05.553 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:12:05.570 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:05.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:05 smithi098 ceph-mon[100914]: pgmap v575: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 2.0 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 694 op/s; 1196/99087 objects misplaced (1.207%); 6.4 MiB/s, 102 objects/s recovering 2024-09-21T10:12:05.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:05 smithi098 ceph-mon[97580]: pgmap v575: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 2.0 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 694 op/s; 1196/99087 objects misplaced (1.207%); 6.4 MiB/s, 102 objects/s recovering 2024-09-21T10:12:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:05.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:05.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:05.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:05.727 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:05.769 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:05.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:05.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:05.871 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:05.887 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:12:05.920 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:05.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:05.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:06.022 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:06.050 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:06.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:06.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:06.151 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:06.207 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:06.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:06.208 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-21T10:12:06.208 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-09-21T10:12:06.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:06.310 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:06.325 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:06 smithi196 ceph-mon[84784]: from='client.54925 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:12:06.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:06.366 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:06.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:06.441 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:06 smithi098 ceph-mon[97580]: from='client.54925 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:12:06.441 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:06 smithi098 ceph-mon[100914]: from='client.54925 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:12:06.467 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:06.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:06.506 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:06.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:06.607 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:06.633 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:06.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:06.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:06.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:06.798 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:06.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:06.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:06.900 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:06.924 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:06.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:06.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:07.025 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:07.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:07.050 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:07.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:07.150 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:07.194 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:07.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:07.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:07.295 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:07.326 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:07.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:07.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:07 smithi196 ceph-mon[84784]: pgmap v576: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 1.9 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 706 op/s; 1196/94926 objects misplaced (1.260%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:12:07.428 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:07.483 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:07.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:07.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:07.553 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:07 smithi098 ceph-mon[97580]: pgmap v576: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 1.9 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 706 op/s; 1196/94926 objects misplaced (1.260%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:12:07.554 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:07 smithi098 ceph-mon[100914]: pgmap v576: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 1.9 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 706 op/s; 1196/94926 objects misplaced (1.260%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:12:07.585 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:07.612 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:07.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:07.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:07.687 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:12:07.687 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:12:07.714 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:07.739 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:07.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:07.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:07.840 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:07.894 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:07.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:07.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:07.996 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:08.022 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:08.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:08.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:08.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:08.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:08.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:08.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:08.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:08.298 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:08.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:08.341 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:08 smithi196 ceph-mon[84784]: pgmap v577: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 1.9 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 757 op/s; 1196/91548 objects misplaced (1.306%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:12:08.341 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:08.341 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:08.379 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:12:08.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:08.433 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:08.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:08.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:08.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:08.569 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:08.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:08.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:08 smithi098 ceph-mon[97580]: pgmap v577: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 1.9 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 757 op/s; 1196/91548 objects misplaced (1.306%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:12:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:08.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:08 smithi098 ceph-mon[100914]: pgmap v577: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 1.9 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 757 op/s; 1196/91548 objects misplaced (1.306%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:12:08.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:08.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:08.671 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:08.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:08.711 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:08.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:08.724 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:12:08.811 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:08.847 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:08.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:08.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:08.949 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:08.976 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:08.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:08.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:09.078 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:09.119 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:09.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:09.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:09.221 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:09.250 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:09.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:09.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:09.352 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:09.376 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:09.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:09.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:09.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:09.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:09.518 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:09.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:09.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:09.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:09.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:09.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:09.802 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:09.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:09.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:09.904 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:09.932 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:09.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:09.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:10.034 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:10.058 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:10.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:10.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:10.160 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:10.199 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:10.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:10.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:10.300 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:10.326 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:10.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:10.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:10.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:10 smithi098 ceph-mon[97580]: pgmap v578: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 1.7 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 853 op/s; 526/82893 objects misplaced (0.635%); 7.1 MiB/s, 113 objects/s recovering 2024-09-21T10:12:10.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:10 smithi098 ceph-mon[100914]: pgmap v578: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 1.7 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 853 op/s; 526/82893 objects misplaced (0.635%); 7.1 MiB/s, 113 objects/s recovering 2024-09-21T10:12:10.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:10.449 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:10.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:10.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:10.517 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:10 smithi196 ceph-mon[84784]: pgmap v578: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 1.7 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 853 op/s; 526/82893 objects misplaced (0.635%); 7.1 MiB/s, 113 objects/s recovering 2024-09-21T10:12:10.517 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:12:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:12:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:12:10.551 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:10.583 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:10.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:10.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:10.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:10.707 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:10.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:10.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:10.791 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:12:10.791 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:12:10.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:10.847 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:10.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:10.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:10.948 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:10.981 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:10.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:10.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:10.988 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:12:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:12:10] "GET /metrics HTTP/1.1" 200 36505 "" "Prometheus/2.51.0" 2024-09-21T10:12:11.083 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:11.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:11.112 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:11.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:11.213 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:11.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:11.241 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:11.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:11.246 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:12:11.246 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:11 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1830381817' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:12:11.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:11 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1830381817' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:12:11.341 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:11.377 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:11.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:11.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:11.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:11 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1830381817' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:12:11.479 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:11.512 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:11.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:11.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:11.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:11.656 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:11.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:11.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:11.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:11.771 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:12:11.798 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:11.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:11.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:11.900 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:11.929 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:11.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:11.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:12.031 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:12.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:12.061 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:12.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:12.118 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:12:12.163 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:12.190 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:12.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:12.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:12.291 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:12.293 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:12.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:12.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:12.460 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:12.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:12.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:12.524 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:12 smithi196 ceph-mon[84784]: pgmap v579: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 1.6 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 820 op/s; 526/79296 objects misplaced (0.663%); 6.0 MiB/s, 95 objects/s recovering 2024-09-21T10:12:12.524 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:12 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:12:12.524 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:12 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1830381817' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:12:12.525 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:12 smithi196 ceph-mon[84784]: osdmap e192: 8 total, 8 up, 7 in 2024-09-21T10:12:12.525 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:12 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:12.525 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:12 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:12.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:12 smithi098 ceph-mon[97580]: pgmap v579: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 1.6 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 820 op/s; 526/79296 objects misplaced (0.663%); 6.0 MiB/s, 95 objects/s recovering 2024-09-21T10:12:12.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:12 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:12:12.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:12 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1830381817' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:12:12.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:12 smithi098 ceph-mon[97580]: osdmap e192: 8 total, 8 up, 7 in 2024-09-21T10:12:12.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:12 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:12.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:12 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:12.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:12 smithi098 ceph-mon[100914]: pgmap v579: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 1.6 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 820 op/s; 526/79296 objects misplaced (0.663%); 6.0 MiB/s, 95 objects/s recovering 2024-09-21T10:12:12.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:12 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:12:12.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:12 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1830381817' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:12:12.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:12 smithi098 ceph-mon[100914]: osdmap e192: 8 total, 8 up, 7 in 2024-09-21T10:12:12.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:12 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:12.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:12 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:12.615 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:12.640 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:12.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:12.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:12.742 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:12.773 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:12.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:12.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:12.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:12.901 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:12.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:12.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:13.003 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:13.027 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:13.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:13.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:13.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:13.158 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:13.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:13.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:13.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:13.286 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:13.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:13.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:13.388 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:13.409 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:13.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:13.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:13.510 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:13.532 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:13.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:13.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:13.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:13 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:12:13.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:13 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:12:13.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:13 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:13.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:13 smithi098 ceph-mon[97580]: osdmap e193: 8 total, 8 up, 7 in 2024-09-21T10:12:13.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:13 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:12:13.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:13 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:12:13.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:13 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:13.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:13 smithi098 ceph-mon[100914]: osdmap e193: 8 total, 8 up, 7 in 2024-09-21T10:12:13.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:13 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:12:13.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:13 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:12:13.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:13 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:13.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:13 smithi196 ceph-mon[84784]: osdmap e193: 8 total, 8 up, 7 in 2024-09-21T10:12:13.634 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:13.656 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:13.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:13.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:13.757 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:13.779 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:13.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:13.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:13.793 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:12:13.793 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:12:13.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:13.902 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:13.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:13.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:14.003 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:14.056 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:14.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:14.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:14.158 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:14.183 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:14.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:14.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:14.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:14.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:14.319 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:14.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:14.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:14.460 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:14.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:14.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:14.562 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:14.569 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:12:14.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:14.583 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:14.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:14.685 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:14.716 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:14.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:14.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:14.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:14.844 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:14.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:14.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:14.917 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:12:14.945 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:14.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:14.980 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:14.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:15.081 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:15.115 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:15.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:15.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:15.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:14 smithi196 ceph-mon[84784]: pgmap v582: 89 pgs: 1 active+clean+remapped, 88 active+clean; 1.5 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 883 op/s; 8.2 MiB/s, 131 objects/s recovering 2024-09-21T10:12:15.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:14 smithi196 ceph-mon[84784]: osdmap e194: 8 total, 8 up, 7 in 2024-09-21T10:12:15.177 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:14 smithi098 ceph-mon[97580]: pgmap v582: 89 pgs: 1 active+clean+remapped, 88 active+clean; 1.5 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 883 op/s; 8.2 MiB/s, 131 objects/s recovering 2024-09-21T10:12:15.177 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:14 smithi098 ceph-mon[97580]: osdmap e194: 8 total, 8 up, 7 in 2024-09-21T10:12:15.178 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:14 smithi098 ceph-mon[100914]: pgmap v582: 89 pgs: 1 active+clean+remapped, 88 active+clean; 1.5 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 883 op/s; 8.2 MiB/s, 131 objects/s recovering 2024-09-21T10:12:15.178 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:14 smithi098 ceph-mon[100914]: osdmap e194: 8 total, 8 up, 7 in 2024-09-21T10:12:15.216 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:15.245 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:15.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:15.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:15.347 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:15.395 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:15.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:15.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:15.496 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:15.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:15.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:15.634 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:15.659 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:15.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:15.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:15.761 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:15.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:12:15.815 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:15.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:15.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:15.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:15.959 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:12:15.959 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:12:15.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:15.964 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:15.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:16.000 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:12:16.001 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:12:16.036 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:12:16.036 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:12:16.065 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:16.084 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:12:16.084 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:12:16.117 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:12:16.117 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:12:16.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:16.118 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:16.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:16.153 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:12:16.154 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:12:16.183 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:12:16.183 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:12:16.219 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:16.221 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:12:16.221 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:12:16.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:16.268 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:16.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:16.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:16.398 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:16.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:16.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:16.500 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:16.537 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:16.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:16.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:16.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:16.680 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:16.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:16.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:16.782 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:16.800 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:12:16.800 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:12:16.800 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:16.800 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.45263.0:225047 6.a 6:5b71cc71:::benchmark_data_smithi098_169033_object102388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-21T10:12:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:12:16.793292+0000", 2024-09-21T10:12:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0041297080000000002, 2024-09-21T10:12:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041601629999999997, 2024-09-21T10:12:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:12:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:12:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:12:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.45263", 2024-09-21T10:12:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:12:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 225047 2024-09-21T10:12:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:12:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:12:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:12:16.793292+0000", 2024-09-21T10:12:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:12:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:12:16.793292+0000", 2024-09-21T10:12:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:12:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:12:16.793290+0000", 2024-09-21T10:12:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:12:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:12:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:12:16.793298+0000", 2024-09-21T10:12:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2810000000000008e-06 2024-09-21T10:12:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:12:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:12:16.793312+0000", 2024-09-21T10:12:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3971999999999999e-05 2024-09-21T10:12:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:12:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:12:16.793318+0000", 2024-09-21T10:12:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9900000000000002e-06 2024-09-21T10:12:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:12:16.793367+0000", 2024-09-21T10:12:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9070000000000003e-05 2024-09-21T10:12:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:12:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:12:16.793578+0000", 2024-09-21T10:12:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00021119800000000001 2024-09-21T10:12:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:12:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:12:16.793703+0000", 2024-09-21T10:12:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012440699999999999 2024-09-21T10:12:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.45263.0:225050 6.a 6:55f1e194:::benchmark_data_smithi098_169033_object102391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-21T10:12:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:12:16.793383+0000", 2024-09-21T10:12:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.004038565, 2024-09-21T10:12:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041513690000000002, 2024-09-21T10:12:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:12:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:12:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:12:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.45263", 2024-09-21T10:12:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:12:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 225050 2024-09-21T10:12:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:12:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:12:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:12:16.793383+0000", 2024-09-21T10:12:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:12:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:12:16.793383+0000", 2024-09-21T10:12:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:12:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:12:16.793382+0000", 2024-09-21T10:12:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:12:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:12:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:12:16.793386+0000", 2024-09-21T10:12:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9330000000000003e-06 2024-09-21T10:12:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:12:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:12:16.793393+0000", 2024-09-21T10:12:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2620000000000001e-06 2024-09-21T10:12:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:12:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:12:16.793398+0000", 2024-09-21T10:12:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.994e-06 2024-09-21T10:12:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:12:16.797047+0000", 2024-09-21T10:12:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036491990000000001 2024-09-21T10:12:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:12:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:12:16.797148+0000", 2024-09-21T10:12:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100516 2024-09-21T10:12:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:12:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:12:16.797257+0000", 2024-09-21T10:12:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010889499999999999 2024-09-21T10:12:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:16.810 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:16.810 INFO:teuthology.orchestra.run.smithi196.stdout: ], 2024-09-21T10:12:16.810 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 2 2024-09-21T10:12:16.810 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:12:16.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:16.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:16.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:16.836 INFO:teuthology.orchestra.run.smithi098.stdout:moved 0 / 267 (0%) 2024-09-21T10:12:16.836 INFO:teuthology.orchestra.run.smithi098.stdout:avg 38.1429 2024-09-21T10:12:16.836 INFO:teuthology.orchestra.run.smithi098.stdout:stddev 5.56593 -> 5.56593 (expected baseline 5.71786) 2024-09-21T10:12:16.836 INFO:teuthology.orchestra.run.smithi098.stdout:min osd.2 with 30 -> 30 pgs (0.786517 -> 0.786517 * mean) 2024-09-21T10:12:16.836 INFO:teuthology.orchestra.run.smithi098.stdout:max osd.1 with 46 -> 46 pgs (1.20599 -> 1.20599 * mean) 2024-09-21T10:12:16.837 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:12:16.837 INFO:teuthology.orchestra.run.smithi098.stdout:oload 1000 2024-09-21T10:12:16.837 INFO:teuthology.orchestra.run.smithi098.stdout:max_change 1 2024-09-21T10:12:16.837 INFO:teuthology.orchestra.run.smithi098.stdout:max_change_osds 4 2024-09-21T10:12:16.837 INFO:teuthology.orchestra.run.smithi098.stdout:average_utilization 0.0102 2024-09-21T10:12:16.837 INFO:teuthology.orchestra.run.smithi098.stdout:overload_utilization 0.1025 2024-09-21T10:12:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:no change 2024-09-21T10:12:16.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:16.945 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:16.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:16.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:17.047 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:17.073 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:17.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:17.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:17.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:16 smithi098 ceph-mon[97580]: pgmap v584: 89 pgs: 1 active+clean+remapped, 88 active+clean; 1.4 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 710 op/s; 5.1 MiB/s, 82 objects/s recovering 2024-09-21T10:12:17.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:16 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4112446022' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:17.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:16 smithi098 ceph-mon[100914]: pgmap v584: 89 pgs: 1 active+clean+remapped, 88 active+clean; 1.4 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 710 op/s; 5.1 MiB/s, 82 objects/s recovering 2024-09-21T10:12:17.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:16 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4112446022' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:17.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:16 smithi196 ceph-mon[84784]: pgmap v584: 89 pgs: 1 active+clean+remapped, 88 active+clean; 1.4 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 710 op/s; 5.1 MiB/s, 82 objects/s recovering 2024-09-21T10:12:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:16 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4112446022' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:17.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:17.209 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:17.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:17.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:17.311 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:17.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:17.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:17.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:17.500 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:17.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:17.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:17.602 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:17.629 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:17.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:17.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:17.730 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:17.765 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:17.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:17.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:17.867 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:17.902 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:17.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:17.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:17.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:17 smithi098 ceph-mon[97580]: from='client.54934 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:12:17.977 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:17 smithi098 ceph-mon[100914]: from='client.54934 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:12:17.989 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:18.004 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:18.009 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:17 smithi196 ceph-mon[84784]: from='client.54934 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:12:18.022 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-21T10:12:18.023 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 110 0.05 2024-09-21T10:12:18.031 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:18.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:18.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:18.044 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:12:18.133 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:18.157 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:18.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:18.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:18.259 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:18.287 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:18.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:18.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:18.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:18.413 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:12:18.432 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:18.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:18.534 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:18.577 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:18.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:18.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:18.678 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:18.706 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:18.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:18.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:18.808 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:18.835 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:18.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:18.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:18.937 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:18.961 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:18.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:18.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:19.063 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:19.092 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:19.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:19.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:19.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:18 smithi196 ceph-mon[84784]: pgmap v585: 89 pgs: 1 active+clean+remapped, 88 active+clean; 1.4 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 663 op/s; 5.1 MiB/s, 82 objects/s recovering 2024-09-21T10:12:19.194 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:19.222 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:19.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:19.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:19.324 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:19.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:18 smithi098 ceph-mon[97580]: pgmap v585: 89 pgs: 1 active+clean+remapped, 88 active+clean; 1.4 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 663 op/s; 5.1 MiB/s, 82 objects/s recovering 2024-09-21T10:12:19.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:18 smithi098 ceph-mon[100914]: pgmap v585: 89 pgs: 1 active+clean+remapped, 88 active+clean; 1.4 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 663 op/s; 5.1 MiB/s, 82 objects/s recovering 2024-09-21T10:12:19.347 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:19.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:19.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:19.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:19.486 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:19.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:19.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:19.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:12:19.588 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:19.610 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:19.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:19.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:19.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:19.747 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:19.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:19.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:19.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:19.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:19.897 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:19.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:19.998 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:20.029 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:20.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:20.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:20.131 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:20.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:20.276 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:20.307 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:20.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:20.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:20.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:20.452 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:20.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:20.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:20.534 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:12:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:12:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:12:20.553 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:20.579 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:20.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:20.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:20.611 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:20.624 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:12:20.624 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:12:20.624 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-21T10:12:20.624 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-21T10:12:20.624 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:12:20.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:20.725 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:20.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:20.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:20.749 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:12:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:12:20] "GET /metrics HTTP/1.1" 200 36505 "" "Prometheus/2.51.0" 2024-09-21T10:12:20.827 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:20.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:20.855 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:20.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:20.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:20.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:20.990 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:20.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:21.011 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:20 smithi098 ceph-mon[97580]: pgmap v586: 89 pgs: 89 active+clean; 1.2 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 822 op/s; 3.9 MiB/s, 61 objects/s recovering 2024-09-21T10:12:21.012 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:20 smithi098 ceph-mon[100914]: pgmap v586: 89 pgs: 89 active+clean; 1.2 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 822 op/s; 3.9 MiB/s, 61 objects/s recovering 2024-09-21T10:12:21.091 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:21.126 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:21.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:21.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:21.261 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:21.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:21.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:21.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:20 smithi196 ceph-mon[84784]: pgmap v586: 89 pgs: 89 active+clean; 1.2 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 822 op/s; 3.9 MiB/s, 61 objects/s recovering 2024-09-21T10:12:21.338 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:12:21.363 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:21.406 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:21.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:21.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:21.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:21.548 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:21.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:21.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:21.650 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:21.680 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:21.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:21.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:21.685 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:12:21.782 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:21.806 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:21.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:21.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:21.908 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:21.948 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:21.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:21.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:22.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:22.067 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:21 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2624303368' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:12:22.067 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:21 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:12:22.087 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:22.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:22.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:22.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:21 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2624303368' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:12:22.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:21 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:12:22.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:21 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2624303368' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:12:22.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:21 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:12:22.188 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:22.215 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:22.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:22.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:22.316 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:22.342 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:22.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:22.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:22.444 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:22.473 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:22.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:22.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:22.576 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:22.599 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:22.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:22.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:22.701 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:22.747 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:22.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:22.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:22.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:22.885 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:22.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:22.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:22.986 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:23.017 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:23.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:23.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:23.118 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:23.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:22 smithi196 ceph-mon[84784]: pgmap v587: 89 pgs: 89 active+clean; 1.1 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 734 op/s 2024-09-21T10:12:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:22 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:12:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:22 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:12:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:22 smithi196 ceph-mon[84784]: osdmap e195: 8 total, 8 up, 7 in 2024-09-21T10:12:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:23.166 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:23.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:23.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:23.267 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:23.301 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:23.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:23.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:23.318 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:22 smithi098 ceph-mon[97580]: pgmap v587: 89 pgs: 89 active+clean; 1.1 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 734 op/s 2024-09-21T10:12:23.318 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:22 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:12:23.318 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:22 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:12:23.318 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:22 smithi098 ceph-mon[97580]: osdmap e195: 8 total, 8 up, 7 in 2024-09-21T10:12:23.318 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:23.318 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:23.318 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:23.319 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:22 smithi098 ceph-mon[100914]: pgmap v587: 89 pgs: 89 active+clean; 1.1 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 734 op/s 2024-09-21T10:12:23.319 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:22 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:12:23.319 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:22 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:12:23.319 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:22 smithi098 ceph-mon[100914]: osdmap e195: 8 total, 8 up, 7 in 2024-09-21T10:12:23.319 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:23.319 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:23.319 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:23.403 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:12:23.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:23.588 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:23.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:23.589 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:12:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-21T10:12:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-21T10:12:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:12:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:35225 5.2 5:444626bb:::benchmark_data_smithi098_135187_object35224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:12:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:01.635693+0000", 2024-09-21T10:12:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 501.869907904, 2024-09-21T10:12:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.10436506800000001, 2024-09-21T10:12:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:12:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:12:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:12:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:12:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:12:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35225 2024-09-21T10:12:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:12:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:12:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.635693+0000", 2024-09-21T10:12:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:12:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.635693+0000", 2024-09-21T10:12:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:12:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.635691+0000", 2024-09-21T10:12:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:12:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:12:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.635732+0000", 2024-09-21T10:12:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1350000000000002e-05 2024-09-21T10:12:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:12:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.635744+0000", 2024-09-21T10:12:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1780000000000001e-05 2024-09-21T10:12:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:12:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.635751+0000", 2024-09-21T10:12:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1509999999999998e-06 2024-09-21T10:12:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.635791+0000", 2024-09-21T10:12:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9376999999999999e-05 2024-09-21T10:12:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:12:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.635898+0000", 2024-09-21T10:12:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010759 2024-09-21T10:12:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:12:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.636020+0000", 2024-09-21T10:12:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000121445 2024-09-21T10:12:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:12:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.637439+0000", 2024-09-21T10:12:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014191309999999999 2024-09-21T10:12:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.638516+0000", 2024-09-21T10:12:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001077644 2024-09-21T10:12:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.739996+0000", 2024-09-21T10:12:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.101479032 2024-09-21T10:12:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:12:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.740033+0000", 2024-09-21T10:12:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7373999999999998e-05 2024-09-21T10:12:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:12:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.740058+0000", 2024-09-21T10:12:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4746e-05 2024-09-21T10:12:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:35230 5.2 5:495d761a:::benchmark_data_smithi098_135187_object35229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-21T10:12:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:01.638968+0000", 2024-09-21T10:12:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 501.866632477, 2024-09-21T10:12:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.101622658, 2024-09-21T10:12:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:12:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:12:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:12:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:12:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:12:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35230 2024-09-21T10:12:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:12:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:12:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.638968+0000", 2024-09-21T10:12:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:12:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.638968+0000", 2024-09-21T10:12:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:12:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.638966+0000", 2024-09-21T10:12:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:12:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:12:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.639039+0000", 2024-09-21T10:12:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2706000000000005e-05 2024-09-21T10:12:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:12:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.639054+0000", 2024-09-21T10:12:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4598e-05 2024-09-21T10:12:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:12:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.639060+0000", 2024-09-21T10:12:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0299999999999999e-06 2024-09-21T10:12:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.639109+0000", 2024-09-21T10:12:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9636999999999999e-05 2024-09-21T10:12:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:12:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.639196+0000", 2024-09-21T10:12:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6482000000000003e-05 2024-09-21T10:12:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:12:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.639317+0000", 2024-09-21T10:12:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000121459 2024-09-21T10:12:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:12:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.640629+0000", 2024-09-21T10:12:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001312159 2024-09-21T10:12:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.640820+0000", 2024-09-21T10:12:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00019025800000000001 2024-09-21T10:12:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.740510+0000", 2024-09-21T10:12:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.099690117999999994 2024-09-21T10:12:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:12:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.740557+0000", 2024-09-21T10:12:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7058000000000002e-05 2024-09-21T10:12:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:12:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:01.740591+0000", 2024-09-21T10:12:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3918000000000003e-05 2024-09-21T10:12:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:72288 5.2 5:4f79b5b0:::benchmark_data_smithi098_135187_object72287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T10:12:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:22.602287+0000", 2024-09-21T10:12:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 480.90331321500003, 2024-09-21T10:12:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.073969280999999998, 2024-09-21T10:12:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:12:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:12:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:12:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:12:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:12:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 72288 2024-09-21T10:12:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:12:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:12:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602287+0000", 2024-09-21T10:12:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:12:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602287+0000", 2024-09-21T10:12:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:12:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602285+0000", 2024-09-21T10:12:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999976 2024-09-21T10:12:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:12:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602326+0000", 2024-09-21T10:12:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1257999999999997e-05 2024-09-21T10:12:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:12:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602343+0000", 2024-09-21T10:12:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.632e-05 2024-09-21T10:12:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:12:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602350+0000", 2024-09-21T10:12:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8339999999999999e-06 2024-09-21T10:12:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602445+0000", 2024-09-21T10:12:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5016e-05 2024-09-21T10:12:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:12:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602554+0000", 2024-09-21T10:12:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108857 2024-09-21T10:12:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:12:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602686+0000", 2024-09-21T10:12:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000131376 2024-09-21T10:12:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:12:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.607760+0000", 2024-09-21T10:12:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0050746979999999999 2024-09-21T10:12:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.610206+0000", 2024-09-21T10:12:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002446045 2024-09-21T10:12:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.676164+0000", 2024-09-21T10:12:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.065957872000000001 2024-09-21T10:12:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:12:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.676219+0000", 2024-09-21T10:12:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4969999999999997e-05 2024-09-21T10:12:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:12:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.676257+0000", 2024-09-21T10:12:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7335000000000001e-05 2024-09-21T10:12:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:72289 5.2 5:46d682ec:::benchmark_data_smithi098_135187_object72288:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T10:12:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:04:22.602365+0000", 2024-09-21T10:12:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 480.903235596, 2024-09-21T10:12:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.074037214000000004, 2024-09-21T10:12:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:12:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:12:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:12:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:12:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:12:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 72289 2024-09-21T10:12:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:12:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:12:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602365+0000", 2024-09-21T10:12:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:12:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602365+0000", 2024-09-21T10:12:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:12:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602364+0000", 2024-09-21T10:12:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:12:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:12:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602397+0000", 2024-09-21T10:12:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2181000000000002e-05 2024-09-21T10:12:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:12:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602404+0000", 2024-09-21T10:12:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0829999999999998e-06 2024-09-21T10:12:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:12:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.602407+0000", 2024-09-21T10:12:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8800000000000001e-06 2024-09-21T10:12:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605159+0000", 2024-09-21T10:12:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027516979999999999 2024-09-21T10:12:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:12:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605271+0000", 2024-09-21T10:12:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011178100000000001 2024-09-21T10:12:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:12:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.605402+0000", 2024-09-21T10:12:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013075700000000001 2024-09-21T10:12:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:12:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.607870+0000", 2024-09-21T10:12:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002467959 2024-09-21T10:12:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.610271+0000", 2024-09-21T10:12:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024008850000000002 2024-09-21T10:12:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.676294+0000", 2024-09-21T10:12:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.066023803000000006 2024-09-21T10:12:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:12:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.676369+0000", 2024-09-21T10:12:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4213000000000002e-05 2024-09-21T10:12:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:12:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:04:22.676402+0000", 2024-09-21T10:12:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3491000000000002e-05 2024-09-21T10:12:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:212086 5.2 5:49ac909f:::benchmark_data_smithi098_135187_object76448:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-21T10:12:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:06:49.073940+0000", 2024-09-21T10:12:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 334.43166043899998, 2024-09-21T10:12:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.066318825999999997, 2024-09-21T10:12:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:12:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:12:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:12:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:12:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:12:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212086 2024-09-21T10:12:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:12:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:12:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:49.073940+0000", 2024-09-21T10:12:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:12:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:49.073940+0000", 2024-09-21T10:12:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:12:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:49.073939+0000", 2024-09-21T10:12:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:12:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:12:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:49.073947+0000", 2024-09-21T10:12:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5420000000000005e-06 2024-09-21T10:12:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:12:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:49.073961+0000", 2024-09-21T10:12:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3669000000000001e-05 2024-09-21T10:12:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:12:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:49.073967+0000", 2024-09-21T10:12:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6289999999999998e-06 2024-09-21T10:12:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:49.078306+0000", 2024-09-21T10:12:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043389880000000002 2024-09-21T10:12:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:12:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:49.078612+0000", 2024-09-21T10:12:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00030687299999999999 2024-09-21T10:12:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:12:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:49.078771+0000", 2024-09-21T10:12:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015892099999999999 2024-09-21T10:12:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:12:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:49.084397+0000", 2024-09-21T10:12:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0056254369999999996 2024-09-21T10:12:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:49.084647+0000", 2024-09-21T10:12:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000250478 2024-09-21T10:12:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:49.140173+0000", 2024-09-21T10:12:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.055526088000000001 2024-09-21T10:12:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:12:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:49.140219+0000", 2024-09-21T10:12:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5580000000000001e-05 2024-09-21T10:12:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:12:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:06:49.140259+0000", 2024-09-21T10:12:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9870000000000003e-05 2024-09-21T10:12:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:267684 5.f 5:f923f360:::benchmark_data_smithi098_135187_object132046:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:12:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:07:59.386796+0000", 2024-09-21T10:12:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 264.118804221, 2024-09-21T10:12:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2285444640000001, 2024-09-21T10:12:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:12:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:12:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:12:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:12:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:12:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 267684 2024-09-21T10:12:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:12:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:12:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.386796+0000", 2024-09-21T10:12:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:12:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.386796+0000", 2024-09-21T10:12:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:12:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.386795+0000", 2024-09-21T10:12:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:12:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:12:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.386799+0000", 2024-09-21T10:12:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2699999999999998e-06 2024-09-21T10:12:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:12:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.386807+0000", 2024-09-21T10:12:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5639999999999999e-06 2024-09-21T10:12:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:12:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.386813+0000", 2024-09-21T10:12:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7450000000000001e-06 2024-09-21T10:12:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.389452+0000", 2024-09-21T10:12:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026394589999999998 2024-09-21T10:12:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:12:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.389460+0000", 2024-09-21T10:12:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1289999999999996e-06 2024-09-21T10:12:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:00.373649+0000", 2024-09-21T10:12:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.98418915200000001 2024-09-21T10:12:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:12:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:00.373662+0000", 2024-09-21T10:12:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.236e-05 2024-09-21T10:12:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.603922+0000", 2024-09-21T10:12:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2302606309999999 2024-09-21T10:12:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:12:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.604044+0000", 2024-09-21T10:12:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000121914 2024-09-21T10:12:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,4,5", 2024-09-21T10:12:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.604139+0000", 2024-09-21T10:12:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5185000000000004e-05 2024-09-21T10:12:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:12:23.628 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.614058+0000", 2024-09-21T10:12:23.628 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0099189120000000002 2024-09-21T10:12:23.628 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.628 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.628 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.628 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.614287+0000", 2024-09-21T10:12:23.628 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00022896200000000001 2024-09-21T10:12:23.628 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.628 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.628 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.628 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.614341+0000", 2024-09-21T10:12:23.628 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3458999999999998e-05 2024-09-21T10:12:23.628 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.628 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.615174+0000", 2024-09-21T10:12:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00083303000000000001 2024-09-21T10:12:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:12:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.615228+0000", 2024-09-21T10:12:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4710999999999998e-05 2024-09-21T10:12:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:12:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.615341+0000", 2024-09-21T10:12:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000112281 2024-09-21T10:12:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:267701 5.f 5:f834efde:::benchmark_data_smithi098_135187_object132063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:12:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:07:59.412832+0000", 2024-09-21T10:12:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 264.09276830200002, 2024-09-21T10:12:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.20275146, 2024-09-21T10:12:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:12:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:12:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:12:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:12:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:12:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 267701 2024-09-21T10:12:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:12:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:12:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.412832+0000", 2024-09-21T10:12:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:12:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.412832+0000", 2024-09-21T10:12:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:12:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.412831+0000", 2024-09-21T10:12:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:12:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:12:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.412837+0000", 2024-09-21T10:12:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6069999999999998e-06 2024-09-21T10:12:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:12:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.412848+0000", 2024-09-21T10:12:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0991e-05 2024-09-21T10:12:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:12:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.412853+0000", 2024-09-21T10:12:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0000000000000004e-06 2024-09-21T10:12:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.412913+0000", 2024-09-21T10:12:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0652999999999997e-05 2024-09-21T10:12:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:12:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.412922+0000", 2024-09-21T10:12:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2959999999999999e-06 2024-09-21T10:12:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:00.374014+0000", 2024-09-21T10:12:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.96109220699999998 2024-09-21T10:12:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:12:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:00.374023+0000", 2024-09-21T10:12:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2909999999999996e-06 2024-09-21T10:12:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.607188+0000", 2024-09-21T10:12:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.233165227 2024-09-21T10:12:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:12:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.607268+0000", 2024-09-21T10:12:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9999000000000004e-05 2024-09-21T10:12:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,4,5", 2024-09-21T10:12:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.607345+0000", 2024-09-21T10:12:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6730000000000006e-05 2024-09-21T10:12:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:12:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.614298+0000", 2024-09-21T10:12:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0069524610000000001 2024-09-21T10:12:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.614380+0000", 2024-09-21T10:12:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2667e-05 2024-09-21T10:12:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.615136+0000", 2024-09-21T10:12:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00075546800000000002 2024-09-21T10:12:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.615447+0000", 2024-09-21T10:12:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00031086200000000002 2024-09-21T10:12:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:12:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.615478+0000", 2024-09-21T10:12:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1129e-05 2024-09-21T10:12:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:12:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.615584+0000", 2024-09-21T10:12:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105895 2024-09-21T10:12:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:267706 5.f 5:fcecc627:::benchmark_data_smithi098_135187_object132068:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:12:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:07:59.426683+0000", 2024-09-21T10:12:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 264.07891716900002, 2024-09-21T10:12:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1988511480000001, 2024-09-21T10:12:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:12:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:12:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:12:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:12:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:12:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 267706 2024-09-21T10:12:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:12:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:12:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.426683+0000", 2024-09-21T10:12:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:12:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.426683+0000", 2024-09-21T10:12:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:12:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.426682+0000", 2024-09-21T10:12:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:12:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:12:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.426689+0000", 2024-09-21T10:12:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6709999999999999e-06 2024-09-21T10:12:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:12:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.426703+0000", 2024-09-21T10:12:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3968000000000001e-05 2024-09-21T10:12:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:12:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.426709+0000", 2024-09-21T10:12:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3909999999999997e-06 2024-09-21T10:12:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.426759+0000", 2024-09-21T10:12:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0587000000000002e-05 2024-09-21T10:12:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:12:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.426768+0000", 2024-09-21T10:12:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3680000000000002e-06 2024-09-21T10:12:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:00.374044+0000", 2024-09-21T10:12:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.947276377 2024-09-21T10:12:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:12:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:00.374051+0000", 2024-09-21T10:12:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8349999999999998e-06 2024-09-21T10:12:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.610059+0000", 2024-09-21T10:12:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2360081759999999 2024-09-21T10:12:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:12:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.610184+0000", 2024-09-21T10:12:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012485799999999999 2024-09-21T10:12:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,4,5", 2024-09-21T10:12:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.610310+0000", 2024-09-21T10:12:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012570699999999999 2024-09-21T10:12:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:12:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.614563+0000", 2024-09-21T10:12:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042535170000000001 2024-09-21T10:12:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.615411+0000", 2024-09-21T10:12:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00084803600000000004 2024-09-21T10:12:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.615693+0000", 2024-09-21T10:12:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00028183199999999998 2024-09-21T10:12:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.625367+0000", 2024-09-21T10:12:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0096744350000000003 2024-09-21T10:12:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:12:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.625422+0000", 2024-09-21T10:12:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4279e-05 2024-09-21T10:12:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:12:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.625534+0000", 2024-09-21T10:12:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000112825 2024-09-21T10:12:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:268913 5.f 5:fe84a087:::benchmark_data_smithi098_135187_object133275:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-21T10:12:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:08:03.451359+0000", 2024-09-21T10:12:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 260.05424144800003, 2024-09-21T10:12:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0111894130000001, 2024-09-21T10:12:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:12:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:12:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:12:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:12:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:12:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 268913 2024-09-21T10:12:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:12:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:12:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451359+0000", 2024-09-21T10:12:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:12:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451359+0000", 2024-09-21T10:12:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:12:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451357+0000", 2024-09-21T10:12:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:12:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:12:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451365+0000", 2024-09-21T10:12:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5190000000000003e-06 2024-09-21T10:12:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:12:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451383+0000", 2024-09-21T10:12:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8697000000000001e-05 2024-09-21T10:12:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:12:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451391+0000", 2024-09-21T10:12:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5619999999999998e-06 2024-09-21T10:12:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451437+0000", 2024-09-21T10:12:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6418999999999997e-05 2024-09-21T10:12:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:12:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451443+0000", 2024-09-21T10:12:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8320000000000002e-06 2024-09-21T10:12:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.453088+0000", 2024-09-21T10:12:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0016444179999999 2024-09-21T10:12:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:12:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.453176+0000", 2024-09-21T10:12:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8582e-05 2024-09-21T10:12:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:12:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.453250+0000", 2024-09-21T10:12:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4245999999999994e-05 2024-09-21T10:12:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:12:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462271+0000", 2024-09-21T10:12:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0090206030000000003 2024-09-21T10:12:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462311+0000", 2024-09-21T10:12:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9798000000000001e-05 2024-09-21T10:12:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462477+0000", 2024-09-21T10:12:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000165845 2024-09-21T10:12:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:12:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462526+0000", 2024-09-21T10:12:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9472999999999999e-05 2024-09-21T10:12:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:12:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462548+0000", 2024-09-21T10:12:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2235e-05 2024-09-21T10:12:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:268916 5.f 5:f65a8683:::benchmark_data_smithi098_135187_object133278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-21T10:12:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:08:03.451594+0000", 2024-09-21T10:12:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 260.05400682499999, 2024-09-21T10:12:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.011084791, 2024-09-21T10:12:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:12:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:12:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:12:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:12:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:12:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 268916 2024-09-21T10:12:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:12:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:12:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451594+0000", 2024-09-21T10:12:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:12:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451594+0000", 2024-09-21T10:12:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:12:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451593+0000", 2024-09-21T10:12:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:12:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:12:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451602+0000", 2024-09-21T10:12:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9290000000000004e-06 2024-09-21T10:12:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:12:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451613+0000", 2024-09-21T10:12:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1637999999999999e-05 2024-09-21T10:12:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:12:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451619+0000", 2024-09-21T10:12:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4729999999999999e-06 2024-09-21T10:12:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451664+0000", 2024-09-21T10:12:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5396999999999999e-05 2024-09-21T10:12:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:12:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451670+0000", 2024-09-21T10:12:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0390000000000001e-06 2024-09-21T10:12:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.456144+0000", 2024-09-21T10:12:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.004474139 2024-09-21T10:12:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:12:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.456256+0000", 2024-09-21T10:12:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000111902 2024-09-21T10:12:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:12:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.456363+0000", 2024-09-21T10:12:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107057 2024-09-21T10:12:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462350+0000", 2024-09-21T10:12:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059866920000000001 2024-09-21T10:12:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:12:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462434+0000", 2024-09-21T10:12:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3663000000000002e-05 2024-09-21T10:12:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462635+0000", 2024-09-21T10:12:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000201184 2024-09-21T10:12:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:12:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462660+0000", 2024-09-21T10:12:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4604999999999999e-05 2024-09-21T10:12:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:12:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462678+0000", 2024-09-21T10:12:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8858000000000001e-05 2024-09-21T10:12:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:268925 5.f 5:f31b12d3:::benchmark_data_smithi098_135187_object133287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-21T10:12:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:08:03.462291+0000", 2024-09-21T10:12:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 260.043309787, 2024-09-21T10:12:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0005015799999999, 2024-09-21T10:12:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:12:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:12:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:12:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:12:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:12:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 268925 2024-09-21T10:12:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:12:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:12:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.462291+0000", 2024-09-21T10:12:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:12:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.462291+0000", 2024-09-21T10:12:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:12:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.462289+0000", 2024-09-21T10:12:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:12:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:12:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.462296+0000", 2024-09-21T10:12:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7329999999999997e-06 2024-09-21T10:12:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:12:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.462308+0000", 2024-09-21T10:12:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1691e-05 2024-09-21T10:12:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:12:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.462314+0000", 2024-09-21T10:12:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8130000000000001e-06 2024-09-21T10:12:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.462361+0000", 2024-09-21T10:12:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7240000000000002e-05 2024-09-21T10:12:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:12:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.462367+0000", 2024-09-21T10:12:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7509999999999997e-06 2024-09-21T10:12:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.458802+0000", 2024-09-21T10:12:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99643555699999997 2024-09-21T10:12:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:12:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.458919+0000", 2024-09-21T10:12:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000117297 2024-09-21T10:12:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:12:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.459025+0000", 2024-09-21T10:12:23.667 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105159 2024-09-21T10:12:23.667 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.667 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.667 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.667 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462592+0000", 2024-09-21T10:12:23.667 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00356719 2024-09-21T10:12:23.667 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.667 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.667 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:12:23.667 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462602+0000", 2024-09-21T10:12:23.667 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9890000000000004e-06 2024-09-21T10:12:23.667 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.667 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.667 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.667 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462718+0000", 2024-09-21T10:12:23.668 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000116588 2024-09-21T10:12:23.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.668 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.668 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:12:23.668 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462757+0000", 2024-09-21T10:12:23.668 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8271e-05 2024-09-21T10:12:23.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.668 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.668 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:12:23.668 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462792+0000", 2024-09-21T10:12:23.668 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.561e-05 2024-09-21T10:12:23.668 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.668 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:23.668 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:268939 5.f 5:f8ea3d45:::benchmark_data_smithi098_135187_object133301:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-21T10:12:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:08:03.483485+0000", 2024-09-21T10:12:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 260.02211505299999, 2024-09-21T10:12:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.98675756599999997, 2024-09-21T10:12:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:12:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:12:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:12:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:12:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:12:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 268939 2024-09-21T10:12:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:12:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:12:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.483485+0000", 2024-09-21T10:12:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:12:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.483485+0000", 2024-09-21T10:12:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:12:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.483483+0000", 2024-09-21T10:12:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:12:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:12:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.483492+0000", 2024-09-21T10:12:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1310000000000006e-06 2024-09-21T10:12:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:12:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.483510+0000", 2024-09-21T10:12:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8431e-05 2024-09-21T10:12:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:12:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.483516+0000", 2024-09-21T10:12:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0299999999999999e-06 2024-09-21T10:12:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.483561+0000", 2024-09-21T10:12:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4793999999999998e-05 2024-09-21T10:12:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:12:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.483571+0000", 2024-09-21T10:12:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0135e-05 2024-09-21T10:12:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462824+0000", 2024-09-21T10:12:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97925344000000003 2024-09-21T10:12:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:12:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462901+0000", 2024-09-21T10:12:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6489999999999994e-05 2024-09-21T10:12:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:12:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462960+0000", 2024-09-21T10:12:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9107999999999997e-05 2024-09-21T10:12:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.470058+0000", 2024-09-21T10:12:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0070984150000000003 2024-09-21T10:12:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.675 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.470100+0000", 2024-09-21T10:12:23.675 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1520000000000002e-05 2024-09-21T10:12:23.675 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.675 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.675 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:12:23.675 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.470109+0000", 2024-09-21T10:12:23.675 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5559999999999996e-06 2024-09-21T10:12:23.675 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.675 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.675 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:12:23.675 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.470202+0000", 2024-09-21T10:12:23.675 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2547999999999996e-05 2024-09-21T10:12:23.675 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.676 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.676 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:12:23.676 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.470243+0000", 2024-09-21T10:12:23.676 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0985999999999997e-05 2024-09-21T10:12:23.676 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.676 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:23.676 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.676 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.676 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.676 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:268948 5.f 5:f12efb92:::benchmark_data_smithi098_135187_object133310:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-21T10:12:23.676 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:08:03.495650+0000", 2024-09-21T10:12:23.676 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 260.00995091800002, 2024-09-21T10:12:23.677 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99013958999999996, 2024-09-21T10:12:23.677 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:12:23.677 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:12:23.677 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:12:23.677 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:12:23.677 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:12:23.677 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 268948 2024-09-21T10:12:23.677 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.677 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:12:23.677 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.677 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:12:23.677 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.495650+0000", 2024-09-21T10:12:23.677 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.678 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.678 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:12:23.678 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.495650+0000", 2024-09-21T10:12:23.678 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.678 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.678 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:12:23.678 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.495648+0000", 2024-09-21T10:12:23.678 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:12:23.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.678 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.678 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:12:23.679 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.495656+0000", 2024-09-21T10:12:23.679 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6070000000000002e-06 2024-09-21T10:12:23.679 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.679 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.679 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:12:23.679 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.495670+0000", 2024-09-21T10:12:23.679 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4619e-05 2024-09-21T10:12:23.679 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.679 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.679 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:12:23.679 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.495676+0000", 2024-09-21T10:12:23.679 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9719999999999997e-06 2024-09-21T10:12:23.680 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.680 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.680 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.680 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.495726+0000", 2024-09-21T10:12:23.680 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9317999999999998e-05 2024-09-21T10:12:23.680 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.680 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.680 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:12:23.680 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.495736+0000", 2024-09-21T10:12:23.680 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0316000000000001e-05 2024-09-21T10:12:23.680 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.680 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.680 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.681 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.466400+0000", 2024-09-21T10:12:23.681 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97066425899999997 2024-09-21T10:12:23.681 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.681 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.681 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:12:23.681 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.466517+0000", 2024-09-21T10:12:23.681 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011711500000000001 2024-09-21T10:12:23.681 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.681 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.681 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:12:23.681 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.466628+0000", 2024-09-21T10:12:23.681 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000110221 2024-09-21T10:12:23.681 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.682 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.682 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.682 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.470299+0000", 2024-09-21T10:12:23.682 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003671278 2024-09-21T10:12:23.682 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.682 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.682 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.682 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.470333+0000", 2024-09-21T10:12:23.682 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3581999999999998e-05 2024-09-21T10:12:23.682 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.682 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.682 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:12:23.682 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.485700+0000", 2024-09-21T10:12:23.683 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015367159 2024-09-21T10:12:23.683 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.683 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.683 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:12:23.683 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.485747+0000", 2024-09-21T10:12:23.683 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6897000000000002e-05 2024-09-21T10:12:23.683 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.683 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.683 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:12:23.683 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.485789+0000", 2024-09-21T10:12:23.683 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2540999999999998e-05 2024-09-21T10:12:23.683 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.683 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:23.684 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.684 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.684 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.684 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:268977 5.f 5:f9611bc4:::benchmark_data_smithi098_135187_object133339:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-21T10:12:23.684 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:08:03.534671+0000", 2024-09-21T10:12:23.684 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 259.97092944600001, 2024-09-21T10:12:23.684 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.95125447399999996, 2024-09-21T10:12:23.684 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:12:23.684 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:12:23.684 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:12:23.684 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:12:23.684 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:12:23.684 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 268977 2024-09-21T10:12:23.684 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.684 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:12:23.685 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.685 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:12:23.685 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.534671+0000", 2024-09-21T10:12:23.685 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.685 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.685 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.685 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:12:23.685 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.534671+0000", 2024-09-21T10:12:23.685 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.685 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.685 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.685 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:12:23.685 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.534670+0000", 2024-09-21T10:12:23.686 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:12:23.686 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.686 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.686 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:12:23.686 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.534677+0000", 2024-09-21T10:12:23.686 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1019999999999999e-06 2024-09-21T10:12:23.686 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.686 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.686 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:12:23.686 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.534690+0000", 2024-09-21T10:12:23.686 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3586e-05 2024-09-21T10:12:23.686 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.686 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.687 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:12:23.687 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.534696+0000", 2024-09-21T10:12:23.687 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7559999999999996e-06 2024-09-21T10:12:23.687 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.687 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.687 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.687 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.534769+0000", 2024-09-21T10:12:23.687 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3295999999999998e-05 2024-09-21T10:12:23.687 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.687 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.687 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:12:23.687 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.534779+0000", 2024-09-21T10:12:23.687 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8330000000000006e-06 2024-09-21T10:12:23.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.688 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.688 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.688 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.470351+0000", 2024-09-21T10:12:23.688 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.93557197700000005 2024-09-21T10:12:23.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.688 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.688 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:12:23.688 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.470478+0000", 2024-09-21T10:12:23.688 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012648999999999999 2024-09-21T10:12:23.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.688 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.688 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:12:23.688 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.470578+0000", 2024-09-21T10:12:23.689 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010086699999999999 2024-09-21T10:12:23.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.689 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.689 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.689 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.478079+0000", 2024-09-21T10:12:23.689 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0075000980000000002 2024-09-21T10:12:23.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.689 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.689 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.689 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.478104+0000", 2024-09-21T10:12:23.689 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5689000000000001e-05 2024-09-21T10:12:23.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.690 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.690 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:12:23.690 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.485838+0000", 2024-09-21T10:12:23.690 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077339920000000003 2024-09-21T10:12:23.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.690 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.690 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:12:23.690 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.485881+0000", 2024-09-21T10:12:23.690 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3013999999999998e-05 2024-09-21T10:12:23.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.690 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.690 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:12:23.691 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.485925+0000", 2024-09-21T10:12:23.691 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4223e-05 2024-09-21T10:12:23.691 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.691 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:23.691 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.691 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.691 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.691 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:269009 5.f 5:f68d3502:::benchmark_data_smithi098_135187_object133371:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-21T10:12:23.691 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:08:03.580762+0000", 2024-09-21T10:12:23.691 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 259.92483850000002, 2024-09-21T10:12:23.691 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.90523814300000005, 2024-09-21T10:12:23.691 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:12:23.691 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:12:23.691 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:12:23.692 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:12:23.692 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:12:23.692 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 269009 2024-09-21T10:12:23.692 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.692 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:12:23.692 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.692 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:12:23.692 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.580762+0000", 2024-09-21T10:12:23.692 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.692 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.692 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.692 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:12:23.692 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.580762+0000", 2024-09-21T10:12:23.692 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.693 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.693 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:12:23.693 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.580761+0000", 2024-09-21T10:12:23.693 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:12:23.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.693 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.693 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:12:23.693 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.580768+0000", 2024-09-21T10:12:23.693 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1810000000000004e-06 2024-09-21T10:12:23.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.693 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.693 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:12:23.694 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.580781+0000", 2024-09-21T10:12:23.694 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3421000000000001e-05 2024-09-21T10:12:23.694 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.694 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.694 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:12:23.694 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.580787+0000", 2024-09-21T10:12:23.694 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6710000000000004e-06 2024-09-21T10:12:23.694 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.694 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.694 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.694 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.580824+0000", 2024-09-21T10:12:23.695 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7515999999999998e-05 2024-09-21T10:12:23.695 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.695 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.695 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:12:23.695 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.580832+0000", 2024-09-21T10:12:23.695 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2300000000000002e-06 2024-09-21T10:12:23.695 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.695 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.695 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.695 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.474329+0000", 2024-09-21T10:12:23.695 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.89349757200000002 2024-09-21T10:12:23.695 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.695 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.696 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:12:23.696 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.474434+0000", 2024-09-21T10:12:23.696 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104351 2024-09-21T10:12:23.696 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.696 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.696 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:12:23.696 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.474536+0000", 2024-09-21T10:12:23.696 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010213 2024-09-21T10:12:23.696 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.696 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.696 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.696 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.478132+0000", 2024-09-21T10:12:23.696 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0035959859999999998 2024-09-21T10:12:23.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.697 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.697 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.485833+0000", 2024-09-21T10:12:23.697 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077007810000000003 2024-09-21T10:12:23.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.697 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.697 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:12:23.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.485931+0000", 2024-09-21T10:12:23.697 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8894000000000006e-05 2024-09-21T10:12:23.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.697 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:12:23.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.485967+0000", 2024-09-21T10:12:23.698 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5966000000000001e-05 2024-09-21T10:12:23.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:12:23.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.486000+0000", 2024-09-21T10:12:23.698 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2727e-05 2024-09-21T10:12:23.698 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.698 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:23.698 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.699 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:269025 5.f 5:f5c512e1:::benchmark_data_smithi098_135187_object133387:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-21T10:12:23.699 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:08:03.627608+0000", 2024-09-21T10:12:23.699 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 259.87799235199998, 2024-09-21T10:12:23.699 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.85976989199999998, 2024-09-21T10:12:23.699 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:12:23.699 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:12:23.699 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:12:23.699 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:12:23.699 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:12:23.699 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 269025 2024-09-21T10:12:23.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.699 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:12:23.699 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.699 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:12:23.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.627608+0000", 2024-09-21T10:12:23.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.700 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.700 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:12:23.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.627608+0000", 2024-09-21T10:12:23.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.700 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.700 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:12:23.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.627607+0000", 2024-09-21T10:12:23.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:12:23.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.700 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.701 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:12:23.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.627615+0000", 2024-09-21T10:12:23.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8240000000000006e-06 2024-09-21T10:12:23.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.701 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.701 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:12:23.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.627627+0000", 2024-09-21T10:12:23.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1963999999999999e-05 2024-09-21T10:12:23.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.701 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.701 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:12:23.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.627634+0000", 2024-09-21T10:12:23.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.516e-06 2024-09-21T10:12:23.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.702 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.702 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.702 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.627672+0000", 2024-09-21T10:12:23.702 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8012000000000001e-05 2024-09-21T10:12:23.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.702 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.702 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:12:23.702 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.627679+0000", 2024-09-21T10:12:23.702 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3950000000000003e-06 2024-09-21T10:12:23.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.702 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.702 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.478150+0000", 2024-09-21T10:12:23.703 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.85047037999999997 2024-09-21T10:12:23.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.703 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.703 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:12:23.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.478251+0000", 2024-09-21T10:12:23.703 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000101469 2024-09-21T10:12:23.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.703 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.703 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:12:23.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.478357+0000", 2024-09-21T10:12:23.703 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105981 2024-09-21T10:12:23.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.704 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.704 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.704 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.485693+0000", 2024-09-21T10:12:23.704 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0073359719999999996 2024-09-21T10:12:23.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.704 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.704 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:12:23.704 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.486005+0000", 2024-09-21T10:12:23.704 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00031183900000000001 2024-09-21T10:12:23.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.704 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.704 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.704 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.487305+0000", 2024-09-21T10:12:23.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001300163 2024-09-21T10:12:23.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.705 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:12:23.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.487345+0000", 2024-09-21T10:12:23.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0055000000000002e-05 2024-09-21T10:12:23.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.705 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:12:23.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.487378+0000", 2024-09-21T10:12:23.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2793000000000003e-05 2024-09-21T10:12:23.705 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.705 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:23.706 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.706 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.706 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:269029 5.f 5:f51cff0c:::benchmark_data_smithi098_135187_object133391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-21T10:12:23.706 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:08:03.645273+0000", 2024-09-21T10:12:23.706 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 259.86032758599998, 2024-09-21T10:12:23.706 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.84219354899999999, 2024-09-21T10:12:23.706 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:12:23.706 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:12:23.706 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:12:23.706 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:12:23.706 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:12:23.707 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 269029 2024-09-21T10:12:23.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.707 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:12:23.707 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.707 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:12:23.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.645273+0000", 2024-09-21T10:12:23.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.707 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.707 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:12:23.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.645273+0000", 2024-09-21T10:12:23.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.708 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.708 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.708 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:12:23.708 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.645266+0000", 2024-09-21T10:12:23.708 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999933 2024-09-21T10:12:23.708 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.708 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.708 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:12:23.708 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.645284+0000", 2024-09-21T10:12:23.708 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7552999999999999e-05 2024-09-21T10:12:23.708 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.708 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.708 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:12:23.708 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.645317+0000", 2024-09-21T10:12:23.709 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3918999999999998e-05 2024-09-21T10:12:23.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.709 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.709 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:12:23.709 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.645335+0000", 2024-09-21T10:12:23.709 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7193e-05 2024-09-21T10:12:23.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.709 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.709 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.709 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.645454+0000", 2024-09-21T10:12:23.709 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119295 2024-09-21T10:12:23.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.709 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.710 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:12:23.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.645467+0000", 2024-09-21T10:12:23.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2969000000000001e-05 2024-09-21T10:12:23.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.710 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.710 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.481903+0000", 2024-09-21T10:12:23.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83643561499999997 2024-09-21T10:12:23.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.710 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.710 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:12:23.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.482031+0000", 2024-09-21T10:12:23.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012832499999999999 2024-09-21T10:12:23.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.711 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:12:23.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.482139+0000", 2024-09-21T10:12:23.711 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108121 2024-09-21T10:12:23.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.711 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.486032+0000", 2024-09-21T10:12:23.711 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0038929809999999998 2024-09-21T10:12:23.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.712 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:12:23.712 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.486037+0000", 2024-09-21T10:12:23.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3029999999999999e-06 2024-09-21T10:12:23.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.712 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.712 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.712 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.487407+0000", 2024-09-21T10:12:23.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001369459 2024-09-21T10:12:23.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.712 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.712 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:12:23.712 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.487440+0000", 2024-09-21T10:12:23.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2870000000000002e-05 2024-09-21T10:12:23.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:12:23.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.487466+0000", 2024-09-21T10:12:23.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6798999999999999e-05 2024-09-21T10:12:23.713 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.713 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:23.713 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.713 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.45263.0:6377 6.a 6:56128e94:::benchmark_data_smithi098_169033_object6376:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-09-21T10:12:23.713 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:08:50.303143+0000", 2024-09-21T10:12:23.713 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 213.20245757500001, 2024-09-21T10:12:23.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.085429045999999995, 2024-09-21T10:12:23.714 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:12:23.714 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:12:23.714 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:12:23.714 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.45263", 2024-09-21T10:12:23.714 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:12:23.714 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 6377 2024-09-21T10:12:23.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.714 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:12:23.714 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.714 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:12:23.714 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:50.303143+0000", 2024-09-21T10:12:23.714 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.714 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.715 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:12:23.715 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:50.303143+0000", 2024-09-21T10:12:23.715 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.715 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.715 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:12:23.715 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:50.303141+0000", 2024-09-21T10:12:23.715 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:12:23.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.715 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.715 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:12:23.715 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:50.303186+0000", 2024-09-21T10:12:23.715 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4320000000000003e-05 2024-09-21T10:12:23.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.716 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:12:23.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:50.303198+0000", 2024-09-21T10:12:23.716 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2274e-05 2024-09-21T10:12:23.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.716 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:12:23.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:50.303202+0000", 2024-09-21T10:12:23.716 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7079999999999996e-06 2024-09-21T10:12:23.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.716 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:50.303245+0000", 2024-09-21T10:12:23.717 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2502000000000001e-05 2024-09-21T10:12:23.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.717 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.717 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:12:23.717 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:50.303335+0000", 2024-09-21T10:12:23.717 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0012000000000005e-05 2024-09-21T10:12:23.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.717 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.717 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:12:23.717 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:50.303418+0000", 2024-09-21T10:12:23.717 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2731000000000005e-05 2024-09-21T10:12:23.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.717 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.717 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:12:23.718 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:50.303898+0000", 2024-09-21T10:12:23.718 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00048033800000000002 2024-09-21T10:12:23.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.718 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.718 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.718 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:50.304126+0000", 2024-09-21T10:12:23.718 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000227746 2024-09-21T10:12:23.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.718 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.718 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.718 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:50.388519+0000", 2024-09-21T10:12:23.718 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.084393212999999995 2024-09-21T10:12:23.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.719 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.719 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:12:23.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:50.388550+0000", 2024-09-21T10:12:23.719 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1282999999999998e-05 2024-09-21T10:12:23.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.719 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.719 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:12:23.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:50.388572+0000", 2024-09-21T10:12:23.719 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1625e-05 2024-09-21T10:12:23.719 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.719 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:23.719 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.719 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.720 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.45263.0:150820 6.a 6:515de673:::benchmark_data_smithi098_169033_object28161:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-21T10:12:23.720 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:10:49.356542+0000", 2024-09-21T10:12:23.720 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 94.149058065000006, 2024-09-21T10:12:23.720 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.10854290599999999, 2024-09-21T10:12:23.720 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:12:23.720 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:12:23.720 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:12:23.720 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.45263", 2024-09-21T10:12:23.720 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:12:23.720 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 150820 2024-09-21T10:12:23.720 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.720 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:12:23.720 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.720 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:12:23.721 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.356542+0000", 2024-09-21T10:12:23.721 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.721 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.721 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:12:23.721 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.356542+0000", 2024-09-21T10:12:23.721 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.721 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.721 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:12:23.721 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.356541+0000", 2024-09-21T10:12:23.721 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:12:23.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.722 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.722 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:12:23.722 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.356548+0000", 2024-09-21T10:12:23.722 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3810000000000001e-06 2024-09-21T10:12:23.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.722 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.722 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:12:23.722 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.356562+0000", 2024-09-21T10:12:23.722 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.364e-05 2024-09-21T10:12:23.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.722 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.722 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:12:23.723 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.356567+0000", 2024-09-21T10:12:23.723 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3990000000000003e-06 2024-09-21T10:12:23.723 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.723 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.723 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.723 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.356615+0000", 2024-09-21T10:12:23.723 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7565000000000003e-05 2024-09-21T10:12:23.723 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.723 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.723 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:12:23.723 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.356747+0000", 2024-09-21T10:12:23.723 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000132741 2024-09-21T10:12:23.724 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.724 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.725 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:12:23.725 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.356897+0000", 2024-09-21T10:12:23.725 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014957300000000001 2024-09-21T10:12:23.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.726 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.726 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:12:23.726 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.359950+0000", 2024-09-21T10:12:23.726 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003053226 2024-09-21T10:12:23.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.726 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.726 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.726 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.360497+0000", 2024-09-21T10:12:23.726 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00054701999999999995 2024-09-21T10:12:23.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.726 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.726 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.726 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.464992+0000", 2024-09-21T10:12:23.727 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.104494792 2024-09-21T10:12:23.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.727 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.727 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:12:23.727 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.465040+0000", 2024-09-21T10:12:23.727 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7901000000000001e-05 2024-09-21T10:12:23.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.727 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.727 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:12:23.727 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.465085+0000", 2024-09-21T10:12:23.727 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5371999999999999e-05 2024-09-21T10:12:23.727 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.727 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:23.727 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.728 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.728 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.45263.0:150837 6.a 6:5f002dcc:::benchmark_data_smithi098_169033_object28178:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-21T10:12:23.728 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:10:49.391308+0000", 2024-09-21T10:12:23.728 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 94.114292732999999, 2024-09-21T10:12:23.728 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.073853779999999994, 2024-09-21T10:12:23.728 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:12:23.728 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:12:23.728 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:12:23.728 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.45263", 2024-09-21T10:12:23.728 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:12:23.728 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 150837 2024-09-21T10:12:23.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.728 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:12:23.728 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.729 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:12:23.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.391308+0000", 2024-09-21T10:12:23.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.729 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.729 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:12:23.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.391308+0000", 2024-09-21T10:12:23.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:12:23.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.729 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.729 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:12:23.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.391306+0000", 2024-09-21T10:12:23.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:12:23.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.730 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.730 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:12:23.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.391314+0000", 2024-09-21T10:12:23.730 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3869999999999997e-06 2024-09-21T10:12:23.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.730 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.730 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:12:23.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.391326+0000", 2024-09-21T10:12:23.730 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2779000000000001e-05 2024-09-21T10:12:23.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.730 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.730 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:12:23.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.391333+0000", 2024-09-21T10:12:23.730 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5390000000000004e-06 2024-09-21T10:12:23.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.731 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.731 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:12:23.731 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.391374+0000", 2024-09-21T10:12:23.731 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.104e-05 2024-09-21T10:12:23.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.731 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.731 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:12:23.731 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.391466+0000", 2024-09-21T10:12:23.731 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2320999999999999e-05 2024-09-21T10:12:23.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.731 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.731 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:12:23.731 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.391549+0000", 2024-09-21T10:12:23.731 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3009000000000004e-05 2024-09-21T10:12:23.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.732 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.732 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:12:23.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.394616+0000", 2024-09-21T10:12:23.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003066839 2024-09-21T10:12:23.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.732 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.732 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.394664+0000", 2024-09-21T10:12:23.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8238000000000001e-05 2024-09-21T10:12:23.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.732 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.732 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:12:23.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.465111+0000", 2024-09-21T10:12:23.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.070446920999999996 2024-09-21T10:12:23.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.733 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.733 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:12:23.733 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.465140+0000", 2024-09-21T10:12:23.733 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8581999999999998e-05 2024-09-21T10:12:23.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:12:23.733 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:12:23.733 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:12:23.733 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:10:49.465162+0000", 2024-09-21T10:12:23.733 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1534000000000001e-05 2024-09-21T10:12:23.733 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.733 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:23.733 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.733 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:12:23.733 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:12:23.734 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:12:23.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:23.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:23.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:23.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:23.899 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:23.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:23.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:24.001 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:24.044 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:24.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:24.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:24.146 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:24.172 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:24.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:24.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:24.274 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:24.302 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:24.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:24.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:24.403 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:24.427 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:24.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:24.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:24.529 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:24.569 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:24.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:24.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:24.670 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:24.694 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:24.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:24.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:24.796 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:24.820 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:24.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:24.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:24.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:24.960 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:24.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:24.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:25.062 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:25.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:24 smithi098 ceph-mon[97580]: pgmap v589: 89 pgs: 89 active+clean; 1.0 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 760 op/s 2024-09-21T10:12:25.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:24 smithi098 ceph-mon[100914]: pgmap v589: 89 pgs: 89 active+clean; 1.0 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 760 op/s 2024-09-21T10:12:25.111 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:25.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:25.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:25.118 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:12:25.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:25.250 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:25.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:25.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:25.352 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:25.375 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:25.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:25.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:25.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:24 smithi196 ceph-mon[84784]: pgmap v589: 89 pgs: 89 active+clean; 1.0 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 760 op/s 2024-09-21T10:12:25.419 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:12:25.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:25.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:25.515 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:25.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:25.616 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:25.639 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:25.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:25.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:25.741 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:25.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:25.785 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:25.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:25.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:25.925 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:25.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:25.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:26.027 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:26.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:26.073 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:26.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:26.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:26.220 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:26.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:26.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:26.306 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:12:26.307 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:12:26.322 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:26.346 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:12:26.346 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:12:26.347 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:26.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:26.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:26.386 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:12:26.386 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:12:26.446 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:12:26.446 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:12:26.449 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:26.472 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:26.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:26.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:26.475 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:12:26.475 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:12:26.483 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:12:26.505 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:12:26.505 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:12:26.544 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:12:26.544 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:12:26.574 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:26.581 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:12:26.581 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:12:26.598 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:26.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:26.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:26.700 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:26.724 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:26.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:26.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:26.826 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:26.849 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:26.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:26.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:26.951 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:26.952 INFO:teuthology.orchestra.run.smithi098.stdout:moved 1 / 267 (0.374532%) 2024-09-21T10:12:26.952 INFO:teuthology.orchestra.run.smithi098.stdout:avg 38.1429 2024-09-21T10:12:26.952 INFO:teuthology.orchestra.run.smithi098.stdout:stddev 5.56593 -> 5.64241 (expected baseline 5.71786) 2024-09-21T10:12:26.952 INFO:teuthology.orchestra.run.smithi098.stdout:min osd.2 with 30 -> 30 pgs (0.786517 -> 0.786517 * mean) 2024-09-21T10:12:26.952 INFO:teuthology.orchestra.run.smithi098.stdout:max osd.1 with 46 -> 47 pgs (1.20599 -> 1.23221 * mean) 2024-09-21T10:12:26.952 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:12:26.952 INFO:teuthology.orchestra.run.smithi098.stdout:oload 110 2024-09-21T10:12:26.952 INFO:teuthology.orchestra.run.smithi098.stdout:max_change 0.05 2024-09-21T10:12:26.952 INFO:teuthology.orchestra.run.smithi098.stdout:max_change_osds 4 2024-09-21T10:12:26.952 INFO:teuthology.orchestra.run.smithi098.stdout:average_utilization 0.0078 2024-09-21T10:12:26.952 INFO:teuthology.orchestra.run.smithi098.stdout:overload_utilization 0.0086 2024-09-21T10:12:26.953 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 weight 0.9500 -> 0.9000 2024-09-21T10:12:26.953 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 weight 0.9500 -> 0.9000 2024-09-21T10:12:26.975 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:26.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:26.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:27.077 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:27.106 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:27.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:27.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:27.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:26 smithi196 ceph-mon[84784]: pgmap v590: 89 pgs: 89 active+clean; 945 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 828 op/s 2024-09-21T10:12:27.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:26 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2272312373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:26 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'3':'58984','6':'58984'}"} : dispatch 2024-09-21T10:12:27.208 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:27.257 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:27.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:27.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:27.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:26 smithi098 ceph-mon[97580]: pgmap v590: 89 pgs: 89 active+clean; 945 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 828 op/s 2024-09-21T10:12:27.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:26 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2272312373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:27.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:26 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'3':'58984','6':'58984'}"} : dispatch 2024-09-21T10:12:27.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:26 smithi098 ceph-mon[100914]: pgmap v590: 89 pgs: 89 active+clean; 945 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 828 op/s 2024-09-21T10:12:27.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:26 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2272312373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:27.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:26 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'3':'58984','6':'58984'}"} : dispatch 2024-09-21T10:12:27.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:27.405 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:27.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:27.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:27.414 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:12:27.414 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:12:27.506 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:27.535 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:27.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:27.637 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:27.667 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:27.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:27.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:27.768 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:27.795 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:27.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:27.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:27.897 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:27.922 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:27.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:27.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:28.023 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:28.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:28.049 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:28.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:28.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:27 smithi196 ceph-mon[84784]: from='client.35601 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:12:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:27 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'58984','6':'58984'}"}]': finished 2024-09-21T10:12:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:27 smithi196 ceph-mon[84784]: osdmap e196: 8 total, 8 up, 7 in 2024-09-21T10:12:28.141 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-21T10:12:28.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-09-21T10:12:28.149 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:28.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:28.180 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:28.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:28.197 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:12:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:28.281 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:28.323 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:28.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:28.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:28.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:27 smithi098 ceph-mon[97580]: from='client.35601 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:12:28.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:27 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'58984','6':'58984'}"}]': finished 2024-09-21T10:12:28.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:27 smithi098 ceph-mon[97580]: osdmap e196: 8 total, 8 up, 7 in 2024-09-21T10:12:28.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:27 smithi098 ceph-mon[100914]: from='client.35601 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:12:28.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:27 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'58984','6':'58984'}"}]': finished 2024-09-21T10:12:28.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:27 smithi098 ceph-mon[100914]: osdmap e196: 8 total, 8 up, 7 in 2024-09-21T10:12:28.424 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:28.466 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:28.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:28.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:28.557 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:12:28.568 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:28.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:28.619 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:28.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:28.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:28.756 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:28.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:28.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:28.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:28.902 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:28.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:28.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:29.003 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:29.041 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:29.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:29.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:29.143 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:29.172 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:29.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:29.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:29.274 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:29.317 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:29.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:29.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:29.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:28 smithi098 ceph-mon[97580]: pgmap v592: 89 pgs: 89 active+clean; 878 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 734 op/s 2024-09-21T10:12:29.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:28 smithi098 ceph-mon[97580]: osdmap e197: 8 total, 8 up, 7 in 2024-09-21T10:12:29.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:28 smithi098 ceph-mon[100914]: pgmap v592: 89 pgs: 89 active+clean; 878 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 734 op/s 2024-09-21T10:12:29.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:28 smithi098 ceph-mon[100914]: osdmap e197: 8 total, 8 up, 7 in 2024-09-21T10:12:29.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:28 smithi196 ceph-mon[84784]: pgmap v592: 89 pgs: 89 active+clean; 878 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 734 op/s 2024-09-21T10:12:29.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:28 smithi196 ceph-mon[84784]: osdmap e197: 8 total, 8 up, 7 in 2024-09-21T10:12:29.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:12:29.419 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:29.446 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:29.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:29.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:29.547 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:29.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:29.574 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:29.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:29.675 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:29.701 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:29.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:29.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:29.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:29.835 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:29.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:29.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:29.937 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:29.970 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:29.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:29.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:30.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:30.083 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:30.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:30.127 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:30.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:30.227 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:30.258 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:30.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:30.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:30.339 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:12:30.339 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:12:30.339 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:12:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:12:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:12:30.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:30.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:30.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:30.496 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:30.534 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:30.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:30.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:30.636 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:30.666 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:30.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:30.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:30.768 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:30.795 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:30.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:30.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:30.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:12:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:12:30] "GET /metrics HTTP/1.1" 200 36485 "" "Prometheus/2.51.0" 2024-09-21T10:12:30.897 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:30.936 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:30.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:31.038 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:31.063 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:31.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:31.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:31.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:31.177 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:30 smithi098 ceph-mon[97580]: pgmap v594: 89 pgs: 1 peering, 88 active+clean; 699 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-21T10:12:31.177 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:30 smithi098 ceph-mon[100914]: pgmap v594: 89 pgs: 1 peering, 88 active+clean; 699 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-21T10:12:31.193 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:31.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:31.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:31.232 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:30 smithi196 ceph-mon[84784]: pgmap v594: 89 pgs: 1 peering, 88 active+clean; 699 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-21T10:12:31.233 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:12:31.295 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:31.329 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:31.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:31.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:31.431 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:31.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:31.466 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:31.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:31.475 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:31.567 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:31.612 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:31.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:31.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:31.659 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:12:31.714 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:31.756 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:31.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:31.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:31.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:31.891 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:31.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:31.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:31.987 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:12:31.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:32.036 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:32.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:32.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:32.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:31 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3162488726' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:12:32.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:31 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:12:32.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:32.163 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:32.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:32.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:32.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:32.311 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:32.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:32.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:32.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:31 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3162488726' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:12:32.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:31 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:12:32.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:31 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3162488726' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:12:32.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:31 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:12:32.413 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:32.452 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:32.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:32.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:32.553 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:32.581 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:32.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:32.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:32.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:32.718 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:32.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:32.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:32.819 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:32.841 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:32.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:32.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:32.943 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:32.975 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:32.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:32.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:33.076 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:33.097 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:32 smithi196 ceph-mon[84784]: pgmap v595: 89 pgs: 1 peering, 88 active+clean; 612 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 844 op/s 2024-09-21T10:12:33.097 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:32 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:12:33.097 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:32 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:12:33.097 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:32 smithi196 ceph-mon[84784]: osdmap e198: 8 total, 8 up, 7 in 2024-09-21T10:12:33.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:33.107 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:33.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:33.208 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:33.233 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:33.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:33.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:33.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:33.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:32 smithi098 ceph-mon[97580]: pgmap v595: 89 pgs: 1 peering, 88 active+clean; 612 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 844 op/s 2024-09-21T10:12:33.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:32 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:12:33.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:32 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:12:33.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:32 smithi098 ceph-mon[97580]: osdmap e198: 8 total, 8 up, 7 in 2024-09-21T10:12:33.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:32 smithi098 ceph-mon[100914]: pgmap v595: 89 pgs: 1 peering, 88 active+clean; 612 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 844 op/s 2024-09-21T10:12:33.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:32 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:12:33.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:32 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:12:33.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:32 smithi098 ceph-mon[100914]: osdmap e198: 8 total, 8 up, 7 in 2024-09-21T10:12:33.365 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:33.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:33.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:33.467 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:33.495 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:33.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:33.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:33.530 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:12:33.531 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:12:33.597 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:33.630 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:33.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:33.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:33.732 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:33.762 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:33.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:33.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:33.863 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:33.901 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:33.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:33.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:34.002 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:34.030 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:34.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:34.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:34.131 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:34.186 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:34.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:34.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:34.227 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:12:34.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:34.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:34.328 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:34.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:34.429 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:34.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:34.454 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:34.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:34.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:34.578 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:34.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:34.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:34.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:34.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:34.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:34.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:34.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:34.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:34.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:34.983 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:34.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:34.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:35.085 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:35.130 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:35.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:35.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:35.232 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:35.261 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:35.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:35.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:35.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:34 smithi098 ceph-mon[97580]: pgmap v597: 89 pgs: 89 active+clean; 490 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 989 op/s 2024-09-21T10:12:35.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:34 smithi098 ceph-mon[100914]: pgmap v597: 89 pgs: 89 active+clean; 490 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 989 op/s 2024-09-21T10:12:35.363 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:35.387 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:35.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:35.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:35.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:34 smithi196 ceph-mon[84784]: pgmap v597: 89 pgs: 89 active+clean; 490 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 989 op/s 2024-09-21T10:12:35.489 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:35.512 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:35.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:35.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:35.596 INFO:teuthology.orchestra.run.smithi098.stdout:moved 0 / 267 (0%) 2024-09-21T10:12:35.596 INFO:teuthology.orchestra.run.smithi098.stdout:avg 38.1429 2024-09-21T10:12:35.596 INFO:teuthology.orchestra.run.smithi098.stdout:stddev 5.64241 -> 5.64241 (expected baseline 5.71786) 2024-09-21T10:12:35.596 INFO:teuthology.orchestra.run.smithi098.stdout:min osd.2 with 30 -> 30 pgs (0.786517 -> 0.786517 * mean) 2024-09-21T10:12:35.597 INFO:teuthology.orchestra.run.smithi098.stdout:max osd.1 with 47 -> 47 pgs (1.23221 -> 1.23221 * mean) 2024-09-21T10:12:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:12:35.597 INFO:teuthology.orchestra.run.smithi098.stdout:oload 1000 2024-09-21T10:12:35.597 INFO:teuthology.orchestra.run.smithi098.stdout:max_change 3 2024-09-21T10:12:35.597 INFO:teuthology.orchestra.run.smithi098.stdout:max_change_osds 4 2024-09-21T10:12:35.597 INFO:teuthology.orchestra.run.smithi098.stdout:average_utilization 0.0053 2024-09-21T10:12:35.597 INFO:teuthology.orchestra.run.smithi098.stdout:overload_utilization 0.0526 2024-09-21T10:12:35.597 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 weight 0.9500 -> 0.9855 2024-09-21T10:12:35.597 INFO:teuthology.orchestra.run.smithi098.stderr:no change 2024-09-21T10:12:35.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:35.621 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:12:35 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T10:12:35.591+0000 7f6c37c82640 -1 mgr.server reply reply (1) Operation not permitted no change 2024-09-21T10:12:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:35.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:35.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:35.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:35.794 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:35.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:35.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:35.896 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:35.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:35.929 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:35.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:36.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:36.071 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:36.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:36.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:36.140 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:12:36.172 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:36.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:36.198 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:36.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:36.299 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:36.340 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:36.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:36.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:36.442 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:36.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:36.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:36.597 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:36.627 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:36.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:36.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:36.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:36.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:36.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:36.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:36.859 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:36.900 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:36.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:36.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:37.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:37.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:37.042 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:37.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:37.142 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:37.167 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:37.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:37.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:37.198 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:36 smithi098 ceph-mon[97580]: pgmap v598: 89 pgs: 89 active+clean; 402 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 951 op/s 2024-09-21T10:12:37.198 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:36 smithi098 ceph-mon[97580]: from='client.45488 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:12:37.198 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:36 smithi098 ceph-mon[100914]: pgmap v598: 89 pgs: 89 active+clean; 402 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 951 op/s 2024-09-21T10:12:37.198 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:36 smithi098 ceph-mon[100914]: from='client.45488 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:12:37.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:37.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:37.320 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:37.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:37.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:36 smithi196 ceph-mon[84784]: pgmap v598: 89 pgs: 89 active+clean; 402 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 951 op/s 2024-09-21T10:12:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:36 smithi196 ceph-mon[84784]: from='client.45488 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:12:37.421 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:37.447 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:37.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:37.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:37.549 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:37.579 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:37.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:37.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:37.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:37.716 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:37.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:37.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:37.771 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:37.817 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:37.840 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:37.844 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:12:37.845 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:12:37.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:37.874 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:37.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:37.890 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:12:37.890 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:12:37.926 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:12:37.926 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:12:37.961 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:12:37.961 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:12:37.976 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:37.997 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:12:37.998 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:12:38.000 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:38.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:38.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:38.026 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:12:38.027 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:12:38.027 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:38.027 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:37 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4138493266' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:38.027 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:38.027 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:37 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4138493266' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:38.059 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:12:38.060 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:12:38.095 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:12:38.095 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:12:38.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:38.147 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:38.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:38.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:38.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:38.289 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:38.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:38.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:37 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4138493266' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:38.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:38.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:38.391 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:38.436 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:38.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:38.537 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:38.565 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:38.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:38.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:38.667 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:38.693 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:38.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:38.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:38.794 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:38.866 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:38.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:38.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:38.968 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:38.992 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:38.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:38.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:39.094 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:39.130 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:39.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:39.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:39.232 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:39.260 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:39.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:39.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:39.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:39 smithi098 ceph-mon[97580]: pgmap v599: 89 pgs: 89 active+clean; 333 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 743 op/s 2024-09-21T10:12:39.284 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:39 smithi098 ceph-mon[100914]: pgmap v599: 89 pgs: 89 active+clean; 333 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 743 op/s 2024-09-21T10:12:39.362 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:39.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:39 smithi196 ceph-mon[84784]: pgmap v599: 89 pgs: 89 active+clean; 333 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 743 op/s 2024-09-21T10:12:39.392 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:39.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:39.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:39.494 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:39.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:39.524 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:39.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:39.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:39.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:39.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:39.761 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:39.787 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:39.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:39.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:39.818 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Removed 122656 objects 2024-09-21T10:12:39.818 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Clean up completed and total clean up time :141.408 2024-09-21T10:12:39.889 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:39.910 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:39.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:39.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:40.012 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:40.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:40.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:40.134 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:40.155 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:40.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:40.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:40.254 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-21T10:12:40.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-21T10:12:40.256 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:40.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:40.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:40.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:40.399 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:40.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:40.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:40.501 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:40.522 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:40.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:40.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:40.624 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:40.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:12:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:12:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:12:40.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:40.645 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:40.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:40.746 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:40.771 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:40.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:40.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:40.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:40.894 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:40.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:40.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:40.996 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:41.005 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:12:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:12:40] "GET /metrics HTTP/1.1" 200 36480 "" "Prometheus/2.51.0" 2024-09-21T10:12:41.022 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:41.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:41.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:41.124 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:41.149 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:41.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:41.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:41.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:12:41.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:41.276 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:41.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:41.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:41.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:41 smithi098 ceph-mon[97580]: pgmap v600: 89 pgs: 89 active+clean; 149 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 879 op/s 2024-09-21T10:12:41.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:41 smithi098 ceph-mon[100914]: pgmap v600: 89 pgs: 89 active+clean; 149 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 879 op/s 2024-09-21T10:12:41.378 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:41.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:41 smithi196 ceph-mon[84784]: pgmap v600: 89 pgs: 89 active+clean; 149 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 879 op/s 2024-09-21T10:12:41.404 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:41.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:41.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:41.506 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:41.527 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:41.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:41.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:41.629 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:41.649 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:12:41.649 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:12:41.649 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-21T10:12:41.649 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:12:41.655 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:41.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:41.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:41.756 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:41.782 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:41.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:41.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:41.884 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:41.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:41.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:42.008 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:42.033 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:42.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:42.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:42.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:42.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:42.159 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:42.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:42.239 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:12:42.240 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:12:42.260 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:42.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:42.286 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:42.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:42.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:12:42.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:42.424 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:42.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:42.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:42.440 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:42.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:42.551 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:42.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:42.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:42.653 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:42.674 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:42.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:42.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:42.776 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:42.802 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:42.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:42.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:42.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:42.925 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:42.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:43.026 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:43.052 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:43.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:43.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:43.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:43 smithi098 ceph-mon[97580]: pgmap v601: 89 pgs: 89 active+clean; 79 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 852 op/s 2024-09-21T10:12:43.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:43 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:43.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:43 smithi098 ceph-mon[100914]: pgmap v601: 89 pgs: 89 active+clean; 79 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 852 op/s 2024-09-21T10:12:43.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:43 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:43.154 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:43.179 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:43.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:43.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:43.281 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:43.306 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:43.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:43.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:43 smithi196 ceph-mon[84784]: pgmap v601: 89 pgs: 89 active+clean; 79 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 852 op/s 2024-09-21T10:12:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:43 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:43.408 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:43.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:43.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:43.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:43.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:43.555 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:43.560 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:43.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:43.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:43.662 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:43.687 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:43.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:43.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:43.789 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:43.813 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:43.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:43.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:43.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:43.940 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:43.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:43.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:44.020 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:12:44.042 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:44.063 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:44.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:44.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:44.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:44.190 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:44.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:44.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:44.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:44.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:44.314 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:44.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:44.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:44 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/19757030' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:12:44.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:44 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:12:44.328 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:44 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/19757030' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:12:44.328 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:44 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:12:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:44 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/19757030' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:12:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:44 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:12:44.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:44.437 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:44.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:44.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:44.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:44.565 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:44.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:44.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:44.666 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:44.691 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:44.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:44.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:44.793 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:44.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:44.816 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:44.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:44.917 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:44.942 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:44.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:44.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:45.044 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:45.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:45.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:45.168 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:45.194 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:45.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:45.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:45.298 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:45.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:45.324 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:45.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:45.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:45 smithi098 ceph-mon[97580]: pgmap v602: 89 pgs: 89 active+clean; 27 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 831 op/s 2024-09-21T10:12:45.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:45 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:12:45.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:45 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:12:45.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:45 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:12:45.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:45 smithi098 ceph-mon[97580]: osdmap e199: 8 total, 8 up, 7 in 2024-09-21T10:12:45.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:45 smithi098 ceph-mon[100914]: pgmap v602: 89 pgs: 89 active+clean; 27 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 831 op/s 2024-09-21T10:12:45.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:45 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:12:45.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:45 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:12:45.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:45 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:12:45.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:45 smithi098 ceph-mon[100914]: osdmap e199: 8 total, 8 up, 7 in 2024-09-21T10:12:45.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:45 smithi196 ceph-mon[84784]: pgmap v602: 89 pgs: 89 active+clean; 27 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 831 op/s 2024-09-21T10:12:45.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:45 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:12:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:45 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:12:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:45 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:12:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:45 smithi196 ceph-mon[84784]: osdmap e199: 8 total, 8 up, 7 in 2024-09-21T10:12:45.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:45.447 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:45.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:45.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:45.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:45.570 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:45.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:45.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:45.671 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:45.697 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:45.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:45.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:45.799 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:45.820 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:45.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:45.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:45.922 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:45.948 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:45.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:45.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:46.049 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:46.075 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:46.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:46.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:46.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:46.199 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:46.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:46.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:46.300 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:46.325 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:46.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:46.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:46.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:46.450 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:46.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:46.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:46.552 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:46.577 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:46.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:46.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:46.679 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:46.704 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:46.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:46.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:46.805 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:46.831 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:46.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:46.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:46.933 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:46.958 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:46.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:46.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:47.060 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:47.085 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:47.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:47.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:47.187 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:47.212 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:47.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:47.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:47.263 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:47.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:47.337 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:47.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:47.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:47.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:47 smithi098 ceph-mon[97580]: pgmap v604: 89 pgs: 89 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 635 op/s 2024-09-21T10:12:47.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:47 smithi098 ceph-mon[100914]: pgmap v604: 89 pgs: 89 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 635 op/s 2024-09-21T10:12:47.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:47 smithi196 ceph-mon[84784]: pgmap v604: 89 pgs: 89 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 635 op/s 2024-09-21T10:12:47.439 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:47.465 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:47.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:47.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:47.558 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:12:47.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:47.622 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:47.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:47.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:47.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:47.747 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:47.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:47.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:47.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:47.872 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:47.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:47.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:47.974 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:48.000 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:48.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:48.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:48.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:48.123 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:48.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:48.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:48.225 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:48.251 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:48.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:48.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:48.352 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:48.373 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:48.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:48.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:48.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:48.500 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:48.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:48.602 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:48.625 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:48.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:48.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:48.727 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:48.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:48.749 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:48.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:48.751 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:12:48.752 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:12:48.768 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:12:48.768 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:12:48.784 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:12:48.785 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:12:48.800 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:12:48.800 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:12:48.816 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:12:48.816 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:12:48.832 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:12:48.832 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:12:48.848 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:12:48.848 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:12:48.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:48.863 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:12:48.863 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:12:48.871 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:48.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:48.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:48.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:48.994 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:48.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:48.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:49.029 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'unique_pool_1' removed 2024-09-21T10:12:49.096 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:49.117 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:49.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:49.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:49.219 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:49.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:49.241 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:49.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:49.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:49.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:49 smithi098 ceph-mon[97580]: pgmap v605: 89 pgs: 89 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 525 op/s 2024-09-21T10:12:49.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:49 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3995306651' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:12:49.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:49 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:12:49.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:49 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/568368773' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:49.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:49 smithi098 ceph-mon[100914]: pgmap v605: 89 pgs: 89 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 525 op/s 2024-09-21T10:12:49.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:49 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3995306651' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:12:49.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:49 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:12:49.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:49 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/568368773' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:49.363 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:49.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:49.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:49.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:49 smithi196 ceph-mon[84784]: pgmap v605: 89 pgs: 89 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 525 op/s 2024-09-21T10:12:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:49 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3995306651' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:12:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:49 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:12:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:49 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/568368773' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:49.465 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:49.486 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:49.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:49.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:49.588 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:49.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:49.611 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:49.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:49.712 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:49.718 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:12:49.719 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:12:49.719 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-21T10:12:49.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_2 16 2024-09-21T10:12:49.733 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:49.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:49.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:49.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:49.856 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:49.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:49.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:49.957 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:49.980 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:49.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:49.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:50.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:50.103 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:50.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:50.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:50.191 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:50.205 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:50.227 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:50.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:50.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:50.301 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:50 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:12:50.301 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:50 smithi098 ceph-mon[97580]: osdmap e200: 8 total, 8 up, 7 in 2024-09-21T10:12:50.302 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:50 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:12:50.302 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:50 smithi098 ceph-mon[100914]: osdmap e200: 8 total, 8 up, 7 in 2024-09-21T10:12:50.328 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:50.350 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:50.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:50.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:50 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:12:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:50 smithi196 ceph-mon[84784]: osdmap e200: 8 total, 8 up, 7 in 2024-09-21T10:12:50.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:12:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:12:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:12:50.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:50.473 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:50.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:50.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:50.574 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:50.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:50.596 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:50.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:50.697 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:50.723 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:50.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:50.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:50.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:50.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:50.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:50.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:50.980 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:12:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:12:50] "GET /metrics HTTP/1.1" 200 36480 "" "Prometheus/2.51.0" 2024-09-21T10:12:50.991 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:50.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:50.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:51.093 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:51.115 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:51.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:51.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:51.217 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:51.240 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:51.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:51.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:51.341 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:51.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:51 smithi098 ceph-mon[97580]: pgmap v607: 73 pgs: 73 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:12:51.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:51 smithi098 ceph-mon[100914]: pgmap v607: 73 pgs: 73 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:12:51.364 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:51.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:51.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:51 smithi196 ceph-mon[84784]: pgmap v607: 73 pgs: 73 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:12:51.466 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:51.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:51.490 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:51.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:51.591 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:51.613 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:51.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:51.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:51.714 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:51.737 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:51.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:51.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:51.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:51.861 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:51.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:51.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:51.963 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:51.985 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:51.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:51.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:52.087 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:52.110 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:52.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:52.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:52.163 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:12:52.212 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:52.235 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:52.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:52.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:52.336 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:52.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:52.359 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:52.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:52.460 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:52.483 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:52.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:52.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:52.521 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:12:52.521 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:12:52.585 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:52.607 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:52.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:52.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:52.709 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:52.731 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:52.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:52.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:52.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:52.855 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:52.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:52.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:52.957 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:52.981 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:52.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:52.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:53.083 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:53.106 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:53.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:53.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:53.208 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:53.231 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:53 smithi098 ceph-mon[97580]: pgmap v608: 73 pgs: 73 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:12:53.231 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:53.231 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:53.231 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:53 smithi098 ceph-mon[100914]: pgmap v608: 73 pgs: 73 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:12:53.231 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:53.231 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:53.232 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:53.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:53.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:53.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:12:53.333 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:53.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:53.359 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:53.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:53 smithi196 ceph-mon[84784]: pgmap v608: 73 pgs: 73 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:12:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:12:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:53.460 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:53.485 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:53.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:53.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:53.586 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:53.609 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:53.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:53.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:53.644 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:53.647 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:53.683 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:53.710 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:53.735 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:53.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:53.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:53.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:53.857 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:53.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:53.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:53.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:53.980 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:53.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:53.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:54.043 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:12:54.055 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:54 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2868518299' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:12:54.055 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:54 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:12:54.055 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:54 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2868518299' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:12:54.055 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:54 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:12:54.082 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:54.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:54.105 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:54.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:54.206 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:54.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:54.229 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:54.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:54.330 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:54.351 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:54.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:54.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:54 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2868518299' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:12:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:54 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:12:54.453 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:54.480 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:54.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:54.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:54.581 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:54.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:54.607 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:54.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:54.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:54.731 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:54.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:54.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:54.833 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:54.858 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:54.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:54.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:54.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:54.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:54.984 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:54.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:55.085 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:55.111 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:55.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:55.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:55.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:55.237 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:55.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:55.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:55.303 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:55 smithi098 ceph-mon[97580]: pgmap v609: 73 pgs: 73 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:12:55.304 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:55 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:12:55.304 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:55 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:12:55.304 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:55 smithi098 ceph-mon[97580]: osdmap e201: 8 total, 8 up, 7 in 2024-09-21T10:12:55.304 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:55 smithi098 ceph-mon[100914]: pgmap v609: 73 pgs: 73 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:12:55.304 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:55 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:12:55.304 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:55 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:12:55.304 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:55 smithi098 ceph-mon[100914]: osdmap e201: 8 total, 8 up, 7 in 2024-09-21T10:12:55.339 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:55.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:55 smithi196 ceph-mon[84784]: pgmap v609: 73 pgs: 73 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:12:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:55 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:12:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:55 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:12:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:55 smithi196 ceph-mon[84784]: osdmap e201: 8 total, 8 up, 7 in 2024-09-21T10:12:55.466 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:55.492 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:55.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:55.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:55.594 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:55.617 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:55.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:55.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:55.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:55.743 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:55.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:55.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:55.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:55.868 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:55.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:55.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:55.970 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:55.994 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:55.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:55.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:56.096 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:56.122 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:56.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:56.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:56.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:56.248 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:56.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:56.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:56.350 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:56.375 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:56.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:56.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:56.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:56.499 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:56.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:56.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:56.601 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:56.626 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:56.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:56.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:56.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:56.753 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:56.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:56.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:56.854 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:56.879 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:56.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:56.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:56.981 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:57.007 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:57.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:57.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:57.109 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:57.134 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:57.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:57.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:57.235 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:57.257 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:57.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:57.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:57.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:57 smithi098 ceph-mon[97580]: pgmap v611: 73 pgs: 73 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:12:57.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:57 smithi098 ceph-mon[100914]: pgmap v611: 73 pgs: 73 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:12:57.359 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:57.385 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:57.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:57.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:57.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:57 smithi196 ceph-mon[84784]: pgmap v611: 73 pgs: 73 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:12:57.488 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:57.513 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:57.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:57.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:57.615 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:57.640 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:57.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:57.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:57.742 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:57.768 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:57.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:57.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:57.869 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:57.892 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:57.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:57.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:57.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:58.014 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:58.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:58.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:58.116 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:58.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:58.138 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:58.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:58.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:58.265 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:58.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:58.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:58.366 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:58.392 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:58.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:58.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:58.493 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:58.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:58.516 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:58.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:58.616 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:58.642 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:58.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:58.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:58.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:58.765 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:58.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:58.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:58.779 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:12:58.780 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:12:58.801 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:12:58.867 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:58.889 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:58.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:58.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:58.990 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:59.003 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:12:59.003 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:12:59.013 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:59.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:59.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:59.022 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:12:59.022 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:12:59.039 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:12:59.039 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:12:59.049 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'unique_pool_2' created 2024-09-21T10:12:59.065 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:12:59.065 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:12:59.082 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:12:59.083 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:12:59.098 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:12:59.099 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:12:59.114 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:12:59.114 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:12:59.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:59.131 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:12:59.131 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:12:59.139 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:59.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:59.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:59.241 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:59.265 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:59.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:59.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:59.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:59 smithi098 ceph-mon[97580]: pgmap v612: 73 pgs: 73 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:12:59.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:59 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2416443784' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T10:12:59.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:12:59 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1066297929' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:59.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:59 smithi098 ceph-mon[100914]: pgmap v612: 73 pgs: 73 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:12:59.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:59 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2416443784' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T10:12:59.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:12:59 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1066297929' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:59.367 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:59.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:59 smithi196 ceph-mon[84784]: pgmap v612: 73 pgs: 73 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:12:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:59 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2416443784' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T10:12:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:12:59 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1066297929' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:59.389 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:59.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:59.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:59.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:59.512 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:59.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:59.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:59.614 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:59.640 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:12:59.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:59.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:59.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:59.764 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:59.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:59.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:59.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:59.888 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:12:59.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:59.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:59.990 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:00.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:00.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:00.114 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:00.135 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:00.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:00.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:00.237 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:00.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:00 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2416443784' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-21T10:13:00.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:00 smithi098 ceph-mon[97580]: osdmap e202: 8 total, 8 up, 7 in 2024-09-21T10:13:00.259 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:00 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2416443784' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-21T10:13:00.259 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:00 smithi098 ceph-mon[100914]: osdmap e202: 8 total, 8 up, 7 in 2024-09-21T10:13:00.262 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:00.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:00.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:00.364 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:00.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:00 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2416443784' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-21T10:13:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:00 smithi196 ceph-mon[84784]: osdmap e202: 8 total, 8 up, 7 in 2024-09-21T10:13:00.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:13:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:13:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:13:00.389 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:00.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:00.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:00.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:00.513 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:00.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:00.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:00.615 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:00.640 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:00.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:00.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:00.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:13:00.741 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:00.767 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:00.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:00.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:00.868 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:00.893 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:00.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:00.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:00.996 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:00.997 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:13:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:13:00] "GET /metrics HTTP/1.1" 200 34132 "" "Prometheus/2.51.0" 2024-09-21T10:13:01.017 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:01.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:01.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:01.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:01.140 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:01.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:01.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:01.242 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:01.268 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:01.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:01.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:01.298 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:01 smithi098 ceph-mon[97580]: pgmap v614: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:01.299 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:01 smithi098 ceph-mon[97580]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:13:01.299 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:01 smithi098 ceph-mon[97580]: osdmap e203: 8 total, 8 up, 7 in 2024-09-21T10:13:01.299 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:01 smithi098 ceph-mon[100914]: pgmap v614: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:01.299 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:01 smithi098 ceph-mon[100914]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:13:01.299 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:01 smithi098 ceph-mon[100914]: osdmap e203: 8 total, 8 up, 7 in 2024-09-21T10:13:01.321 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-21T10:13:01.362 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:01.370 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:01.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:01 smithi196 ceph-mon[84784]: pgmap v614: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:01 smithi196 ceph-mon[84784]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:13:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:01 smithi196 ceph-mon[84784]: osdmap e203: 8 total, 8 up, 7 in 2024-09-21T10:13:01.394 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:01.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:01.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:01.399 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:01.496 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:01.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:01.518 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:01.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:01.619 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:01.645 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:01.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:01.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:01.747 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:01.772 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:01.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:01.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:01.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:01.899 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:01.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:01.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:01.923 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:13:02.001 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:02.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:02.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:02.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:02.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:02.145 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:02.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:02.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:02.269 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:02.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:02.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:02.270 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:02 smithi098 ceph-mon[97580]: osdmap e204: 8 total, 8 up, 7 in 2024-09-21T10:13:02.271 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:02 smithi098 ceph-mon[100914]: osdmap e204: 8 total, 8 up, 7 in 2024-09-21T10:13:02.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:02 smithi196 ceph-mon[84784]: osdmap e204: 8 total, 8 up, 7 in 2024-09-21T10:13:02.393 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:02.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:02.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:02.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:02.517 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:02.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:02.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:02.618 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:02.643 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:02.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:02.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:02.745 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:02.771 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:02.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:02.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:02.872 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:02.898 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:02.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:02.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:03.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:03.025 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:03.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:03.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:03.127 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:03.153 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:03.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:03.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:03.254 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:03.278 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:03.280 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:03.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:03.281 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:03.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:03.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:03 smithi098 ceph-mon[100914]: pgmap v617: 89 pgs: 2 creating+peering, 11 unknown, 76 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:03 smithi098 ceph-mon[97580]: pgmap v617: 89 pgs: 2 creating+peering, 11 unknown, 76 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:03.382 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:03.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:03 smithi196 ceph-mon[84784]: pgmap v617: 89 pgs: 2 creating+peering, 11 unknown, 76 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:03.407 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:03.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:03.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:03.509 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:03.530 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:03.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:03.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:03.632 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:03.657 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:03.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:03.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:03.759 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:03.784 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:03.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:03.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:03.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:03.907 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:03.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:03.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:04.008 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:04.035 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:04.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:04.136 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:04.162 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:04.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:04.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:04.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:04.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:04.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:04.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:04.407 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:04.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:04.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:04.509 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:04.534 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:04.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:04.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:04.636 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:04.658 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:04.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:04.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:04.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:04.782 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:04.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:04.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:04.811 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:13:04.811 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:13:04.812 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-21T10:13:04.812 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:13:04.883 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:04.905 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:04.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:04.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:05.007 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:05.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:05.034 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:05.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:05.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:05 smithi098 ceph-mon[97580]: pgmap v618: 89 pgs: 2 creating+peering, 4 unknown, 83 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:05.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:05.159 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:05.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:05.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:05.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:05.285 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:05.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:05.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:05.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:05 smithi098 ceph-mon[100914]: pgmap v618: 89 pgs: 2 creating+peering, 4 unknown, 83 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:05.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:05.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:05 smithi196 ceph-mon[84784]: pgmap v618: 89 pgs: 2 creating+peering, 4 unknown, 83 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:05.408 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:05.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:05.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:05.509 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:05.536 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:05.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:05.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:05.637 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:05.659 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:05.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:05.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:05.761 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:05.786 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:05.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:05.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:05.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:13:05.887 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:05.913 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:05.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:05.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:06.014 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:06.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:06.037 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:06.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:06.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:06.158 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:06.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:06.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:06.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:06.284 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:06.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:06.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:06.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:06.407 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:06.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:06.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:06.509 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:06.531 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:06.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:06.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:06.632 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:06.659 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:06.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:06.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:06.760 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:06.785 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:06.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:06.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:06.887 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:06.909 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:06.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:06.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:07.011 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:07.033 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:07.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:07.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:07.083 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:13:07.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:07 smithi098 ceph-mon[97580]: pgmap v619: 89 pgs: 89 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:07.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:07 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/116450264' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:13:07.095 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:07 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:13:07.095 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:07 smithi098 ceph-mon[100914]: pgmap v619: 89 pgs: 89 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:07.095 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:07 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/116450264' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:13:07.095 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:07 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:13:07.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:07.156 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:07.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:07.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:07.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:07.278 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:07.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:07.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:07.380 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:07.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:07 smithi196 ceph-mon[84784]: pgmap v619: 89 pgs: 89 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:07.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:07 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/116450264' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:13:07.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:07 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:13:07.406 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:07.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:07.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:07.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:07.528 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:07.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:07.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:07.629 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:07.656 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:07.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:07.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:07.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:07.779 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:07.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:07.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:07.882 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:07.904 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:07.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:07.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:08.005 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:08.030 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:08.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:08.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:08.132 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:08.156 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:08.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:08.240 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:08 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:13:08.240 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:08 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:13:08.240 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:08 smithi098 ceph-mon[97580]: osdmap e205: 8 total, 8 up, 7 in 2024-09-21T10:13:08.240 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:08.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:08.241 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:08 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:13:08.241 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:08 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:13:08.241 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:08 smithi098 ceph-mon[100914]: osdmap e205: 8 total, 8 up, 7 in 2024-09-21T10:13:08.241 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:08.241 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:08.257 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:08.281 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:08.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:08.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:08.383 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:08.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:08 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:13:08.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:08 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:13:08.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:08 smithi196 ceph-mon[84784]: osdmap e205: 8 total, 8 up, 7 in 2024-09-21T10:13:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:08.406 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:08.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:08.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:08.508 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:08.532 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:08.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:08.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:08.597 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:08.633 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:08.656 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:08.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:08.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:08.757 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:08.780 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:08.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:08.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:08.882 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:08.905 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:08.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:08.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:09.007 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:09.030 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:09.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:09.132 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:09.155 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:09 smithi098 ceph-mon[97580]: pgmap v621: 89 pgs: 89 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:09.155 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:09 smithi098 ceph-mon[100914]: pgmap v621: 89 pgs: 89 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:09.155 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:09.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:09.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:09.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:09.279 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:09.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:09.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:09.381 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:09.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:09 smithi196 ceph-mon[84784]: pgmap v621: 89 pgs: 89 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:09.406 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:09.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:09.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:09.508 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:09.533 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:09.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:09.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:09.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:09.656 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:09.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:09.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:09.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:09.780 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:09.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:09.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:09.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:09.907 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:09.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:09.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:10.009 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:10.032 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:10.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:10.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:10.134 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:10.160 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:10.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:10.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:10.262 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:10.285 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:10.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:10.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:10.388 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:10.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:10.410 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:10.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:10.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:10.534 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:10.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:10.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:10.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:10.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:13:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:13:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:13:10.657 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:10.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:10.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:10.759 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:10.781 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:10.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:10.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:10.882 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:10.909 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:10.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:10.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:11.010 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:11.035 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:11.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:11.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:11.081 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:13:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:13:10] "GET /metrics HTTP/1.1" 200 36495 "" "Prometheus/2.51.0" 2024-09-21T10:13:11.082 INFO:teuthology.orchestra.run.smithi098.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-21T10:13:11.137 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:11.161 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:11.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:11.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:11.263 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:11.287 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:11.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:11.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:11.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:11 smithi098 ceph-mon[97580]: pgmap v622: 89 pgs: 89 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:11.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:11 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3482713221' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:11.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:11 smithi098 ceph-mon[100914]: pgmap v622: 89 pgs: 89 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:11.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:11 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3482713221' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:11 smithi196 ceph-mon[84784]: pgmap v622: 89 pgs: 89 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:11 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3482713221' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:11.389 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:11.413 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:11.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:11.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:11.422 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:13:11.422 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:13:11.422 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:13:11.422 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:13:11.422 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:13:11.422 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:13:11.425 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:13:11.425 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:13:11.441 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:13:11.441 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:13:11.457 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:13:11.457 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:13:11.473 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:13:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:13:11.488 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:13:11.488 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:13:11.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:11.536 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:11.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:11.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:11.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:11.662 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:11.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:11.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:11.763 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:11.786 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:11.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:11.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:11.888 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:11.910 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:11.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:11.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:12.011 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:12.034 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:12.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:12.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:12.135 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:12.158 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:12.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:12.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:12.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:12.281 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:12.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:12.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:12.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:12 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3482713221' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:13:12.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:12 smithi098 ceph-mon[97580]: osdmap e206: 8 total, 8 up, 7 in 2024-09-21T10:13:12.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:12 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1042563632' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:13:12.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:12 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3482713221' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:13:12.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:12 smithi098 ceph-mon[100914]: osdmap e206: 8 total, 8 up, 7 in 2024-09-21T10:13:12.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:12 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1042563632' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:13:12.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:12.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:12 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3482713221' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:13:12.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:12 smithi196 ceph-mon[84784]: osdmap e206: 8 total, 8 up, 7 in 2024-09-21T10:13:12.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:12 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1042563632' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:13:12.395 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:13:12.395 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:13:12.405 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:12.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:12.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:12.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:12.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:12.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:12.629 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:12.653 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:12.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:12.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:12.755 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:12.776 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:12.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:12.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:12.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:12.900 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:12.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:12.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:13.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:13.024 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:13.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:13.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:13.126 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:13.148 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:13.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:13.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:13.187 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:13:13.188 DEBUG:teuthology.orchestra.run.smithi098:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-21T10:13:13.188 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_2 2024-09-21T10:13:13.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:13:13.192 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:13:13.192 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:13:13.238 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:hints = 1 2024-09-21T10:13:13.238 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:13:13.238 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Object prefix: benchmark_data_smithi098_197834 2024-09-21T10:13:13.238 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:13:13.239 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:13:13.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:13.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:13:13.318 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:13.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:13.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:13.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:13 smithi098 ceph-mon[97580]: pgmap v624: 89 pgs: 89 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:13.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:13 smithi098 ceph-mon[97580]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:13:13.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:13 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:13.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:13 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:13:13.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:13 smithi098 ceph-mon[100914]: pgmap v624: 89 pgs: 89 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:13.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:13 smithi098 ceph-mon[100914]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:13:13.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:13 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:13.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:13 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:13:13.358 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:13 smithi196 ceph-mon[84784]: pgmap v624: 89 pgs: 89 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:13.358 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:13 smithi196 ceph-mon[84784]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:13:13.358 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:13 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:13.358 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:13 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:13:13.419 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:13.445 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:13.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:13.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:13.547 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:13.571 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:13.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:13.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:13.673 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:13.702 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:13.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:13.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:13.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:13.843 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:13.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:13.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:13.945 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:13.975 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:13.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:13.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:14.077 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:14.103 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:14.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:14.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:14.149 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:14.169 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:14.204 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:14.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:14.230 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:14.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:14.238 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 1 16 2919 2903 181.4 181.438 0.00579621 0.00550373 2024-09-21T10:13:14.246 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:14.331 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:14.375 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:14.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:14.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:14.477 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:14.504 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:14.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:14.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:14.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:14.649 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:14.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:14.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:14.871 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:14.899 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:14.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:14.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:15.001 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:15.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:15.037 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:15.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:15.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:15.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:15.168 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:15.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:15.239 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 2 16 6174 6158 192.409 203.438 0.0084243 0.00518706 2024-09-21T10:13:15.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:15.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:15.306 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:15.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:15.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:15 smithi196 ceph-mon[84784]: pgmap v625: 89 pgs: 89 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:15.386 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:15 smithi098 ceph-mon[97580]: pgmap v625: 89 pgs: 89 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:15.386 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:15 smithi098 ceph-mon[100914]: pgmap v625: 89 pgs: 89 active+clean; 4.6 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:13:15.407 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:15.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:15.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:15.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:15.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:15.570 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:15.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:15.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:15.671 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:15.709 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:15.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:15.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:15.811 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:15.851 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:15.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:15.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:15.953 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:15.990 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:15.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:15.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:16.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:13:16.092 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:16.139 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:16.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:16.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:16.239 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 3 16 9167 9151 190.621 187.062 0.00308434 0.00523831 2024-09-21T10:13:16.241 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:16.268 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:16.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:16.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:16.370 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:16.395 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:16.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:16.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:16.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:16.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:16.527 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:16.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:16.627 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:16.654 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:16.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:16.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:16.756 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:16.783 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:16.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:16.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:16.885 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:16.911 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:16.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:16.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:17.013 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:17.056 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:17.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:17.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:17.158 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:17.188 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:17.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:17.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:17.239 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 4 16 11633 11617 181.494 154.125 0.00529729 0.00550288 2024-09-21T10:13:17.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:17.329 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:17.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:17.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:17.380 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:17 smithi098 ceph-mon[97580]: pgmap v626: 89 pgs: 89 active+clean; 50 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 5.5 MiB/s wr, 88 op/s 2024-09-21T10:13:17.380 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:17 smithi098 ceph-mon[100914]: pgmap v626: 89 pgs: 89 active+clean; 50 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 5.5 MiB/s wr, 88 op/s 2024-09-21T10:13:17.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:17 smithi196 ceph-mon[84784]: pgmap v626: 89 pgs: 89 active+clean; 50 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 5.5 MiB/s wr, 88 op/s 2024-09-21T10:13:17.432 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:17.461 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:17.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:17.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:17.563 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:17.591 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:17.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:17.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:17.693 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:17.718 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:17.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:17.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:17.813 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:13:17.814 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:13:17.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:17.844 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:17.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:17.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:17.946 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:17.975 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:17.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:18.076 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:18.111 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:18.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:18.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:18.213 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:18.239 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 5 16 14153 14137 176.693 157.5 0.00663259 0.00565298 2024-09-21T10:13:18.240 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:18.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:18.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:18.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:18.377 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:18.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:18.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:18.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:18.512 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:18.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:18.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:18.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:18.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:18 smithi196 ceph-mon[84784]: pgmap v627: 89 pgs: 89 active+clean; 159 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 15 MiB/s wr, 246 op/s 2024-09-21T10:13:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:18 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:18 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:18 smithi196 ceph-mon[84784]: from='client.55006 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:13:18.652 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:18.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:18.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:18.753 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:18.790 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:18.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:18.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:18 smithi098 ceph-mon[97580]: pgmap v627: 89 pgs: 89 active+clean; 159 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 15 MiB/s wr, 246 op/s 2024-09-21T10:13:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:18 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:18 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:18 smithi098 ceph-mon[97580]: from='client.55006 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:13:18.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:18 smithi098 ceph-mon[100914]: pgmap v627: 89 pgs: 89 active+clean; 159 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 15 MiB/s wr, 246 op/s 2024-09-21T10:13:18.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:18 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:18.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:18 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:18.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:18 smithi098 ceph-mon[100914]: from='client.55006 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:13:18.891 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:18.927 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:18.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:18.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:19.029 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:19.072 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:19.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:19.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:19.176 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:19.202 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:19.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:19.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:19.239 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 6 16 16564 16548 172.356 150.688 0.00259027 0.00576452 2024-09-21T10:13:19.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:19.340 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:19.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:19.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:19.442 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:19.477 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:19.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:19.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:19.578 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:19.605 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:19.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:19.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:19.707 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:19.733 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:19.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:19.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:19.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:19.876 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:19.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:19.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:19.978 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:20.018 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:20.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:20.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:20.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:20.157 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:20.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:20.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:20.239 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 7 16 18987 18971 169.366 151.438 0.0056627 0.00589551 2024-09-21T10:13:20.259 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:20.284 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:20.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:20.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:20.386 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:20.415 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:20.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:20.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:20.517 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:20.544 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:20.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:20.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:20.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:20 smithi098 ceph-mon[97580]: pgmap v628: 89 pgs: 89 active+clean; 721 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-21T10:13:20.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:20 smithi098 ceph-mon[100914]: pgmap v628: 89 pgs: 89 active+clean; 721 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-21T10:13:20.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:20 smithi196 ceph-mon[84784]: pgmap v628: 89 pgs: 89 active+clean; 721 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-21T10:13:20.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:13:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:13:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:13:20.646 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:20.683 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:20.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:20.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:20.785 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:20.822 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:20.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:20.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:20.924 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:20.951 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:20.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:20.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:21.053 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:21.088 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:21.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:21.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:21.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:13:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:13:20] "GET /metrics HTTP/1.1" 200 36495 "" "Prometheus/2.51.0" 2024-09-21T10:13:21.179 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":627,"stamp":"2024-09-21T10:13:17.229581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":166401997,"num_objects":2494,"num_object_clones":0,"num_object_copies":7482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2494,"num_whiteouts":0,"num_read":6288,"num_read_kb":13764,"num_write":3522,"num_write_kb":172075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":600294,"num_keys_recovered":30,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3598,"ondisk_log_size":3598,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2480348,"kb_used_data":314512,"kb_used_omap":33,"kb_used_meta":2165726,"kb_avail":653735716,"statfs":{"total":671965249536,"available":669425373184,"internally_reserved":0,"allocated":322060288,"data_stored":315124322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34649,"internal_metadata":2217703591},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":161611776,"num_objects":2466,"num_object_clones":0,"num_object_copies":7398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2466,"num_write_kb":157824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002552"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":281,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.327674+0000","last_change":"2024-09-21T10:02:14.097991+0000","last_active":"2024-09-21T10:13:11.327674+0000","last_peered":"2024-09-21T10:13:11.327674+0000","last_clean":"2024-09-21T10:13:11.327674+0000","last_became_active":"2024-09-21T10:02:14.097407+0000","last_became_peered":"2024-09-21T10:02:14.097407+0000","last_unstale":"2024-09-21T10:13:11.327674+0000","last_undegraded":"2024-09-21T10:13:11.327674+0000","last_fullsized":"2024-09-21T10:13:11.327674+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:18:42.098886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":221,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.082063+0000","last_change":"2024-09-21T10:08:00.406360+0000","last_active":"2024-09-21T10:13:07.082063+0000","last_peered":"2024-09-21T10:13:07.082063+0000","last_clean":"2024-09-21T10:13:07.082063+0000","last_became_active":"2024-09-21T10:08:00.380042+0000","last_became_peered":"2024-09-21T10:08:00.380042+0000","last_unstale":"2024-09-21T10:13:07.082063+0000","last_undegraded":"2024-09-21T10:13:07.082063+0000","last_fullsized":"2024-09-21T10:13:07.082063+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:49:46.544051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":127,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.082351+0000","last_change":"2024-09-21T10:07:43.789751+0000","last_active":"2024-09-21T10:13:07.082351+0000","last_peered":"2024-09-21T10:13:07.082351+0000","last_clean":"2024-09-21T10:13:07.082351+0000","last_became_active":"2024-09-21T10:07:43.789448+0000","last_became_peered":"2024-09-21T10:07:43.789448+0000","last_unstale":"2024-09-21T10:13:07.082351+0000","last_undegraded":"2024-09-21T10:13:07.082351+0000","last_fullsized":"2024-09-21T10:13:07.082351+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:36.586942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":277,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.918084+0000","last_change":"2024-09-21T10:02:14.098949+0000","last_active":"2024-09-21T10:13:11.918084+0000","last_peered":"2024-09-21T10:13:11.918084+0000","last_clean":"2024-09-21T10:13:11.918084+0000","last_became_active":"2024-09-21T10:02:14.098646+0000","last_became_peered":"2024-09-21T10:02:14.098646+0000","last_unstale":"2024-09-21T10:13:11.918084+0000","last_undegraded":"2024-09-21T10:13:11.918084+0000","last_fullsized":"2024-09-21T10:13:11.918084+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:50:56.862529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":277,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.919019+0000","last_change":"2024-09-21T10:02:14.098205+0000","last_active":"2024-09-21T10:13:11.919019+0000","last_peered":"2024-09-21T10:13:11.919019+0000","last_clean":"2024-09-21T10:13:11.919019+0000","last_became_active":"2024-09-21T10:02:14.097911+0000","last_became_peered":"2024-09-21T10:02:14.097911+0000","last_unstale":"2024-09-21T10:13:11.919019+0000","last_undegraded":"2024-09-21T10:13:11.919019+0000","last_fullsized":"2024-09-21T10:13:11.919019+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:55.971191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":86,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.700577+0000","last_change":"2024-09-21T10:08:00.385671+0000","last_active":"2024-09-21T10:13:07.700577+0000","last_peered":"2024-09-21T10:13:07.700577+0000","last_clean":"2024-09-21T10:13:07.700577+0000","last_became_active":"2024-09-21T10:08:00.385368+0000","last_became_peered":"2024-09-21T10:08:00.385368+0000","last_unstale":"2024-09-21T10:13:07.700577+0000","last_undegraded":"2024-09-21T10:13:07.700577+0000","last_fullsized":"2024-09-21T10:13:07.700577+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:21.799149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":281,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.327980+0000","last_change":"2024-09-21T10:02:14.097817+0000","last_active":"2024-09-21T10:13:11.327980+0000","last_peered":"2024-09-21T10:13:11.327980+0000","last_clean":"2024-09-21T10:13:11.327980+0000","last_became_active":"2024-09-21T10:02:14.097466+0000","last_became_peered":"2024-09-21T10:02:14.097466+0000","last_unstale":"2024-09-21T10:13:11.327980+0000","last_undegraded":"2024-09-21T10:13:11.327980+0000","last_fullsized":"2024-09-21T10:13:11.327980+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:57.883118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":178,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.700456+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:13:07.700456+0000","last_peered":"2024-09-21T10:13:07.700456+0000","last_clean":"2024-09-21T10:13:07.700456+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:13:07.700456+0000","last_undegraded":"2024-09-21T10:13:07.700456+0000","last_fullsized":"2024-09-21T10:13:07.700456+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"206'598","reported_seq":617,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:16.126095+0000","last_change":"2024-09-21T10:13:00.065993+0000","last_active":"2024-09-21T10:13:16.126095+0000","last_peered":"2024-09-21T10:13:16.126095+0000","last_clean":"2024-09-21T10:13:16.126095+0000","last_became_active":"2024-09-21T10:13:00.065574+0000","last_became_peered":"2024-09-21T10:13:00.065574+0000","last_unstale":"2024-09-21T10:13:16.126095+0000","last_undegraded":"2024-09-21T10:13:16.126095+0000","last_fullsized":"2024-09-21T10:13:16.126095+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":598,"log_dups_size":0,"ondisk_log_size":598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:58.605844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39190528,"num_objects":598,"num_object_clones":0,"num_object_copies":1794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":598,"num_write_kb":38272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":214,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.082537+0000","last_change":"2024-09-21T10:08:00.392499+0000","last_active":"2024-09-21T10:13:07.082537+0000","last_peered":"2024-09-21T10:13:07.082537+0000","last_clean":"2024-09-21T10:13:07.082537+0000","last_became_active":"2024-09-21T10:08:00.379165+0000","last_became_peered":"2024-09-21T10:08:00.379165+0000","last_unstale":"2024-09-21T10:13:07.082537+0000","last_undegraded":"2024-09-21T10:13:07.082537+0000","last_fullsized":"2024-09-21T10:13:07.082537+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:18.933144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":70,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.081940+0000","last_change":"2024-09-21T10:07:43.794677+0000","last_active":"2024-09-21T10:13:07.081940+0000","last_peered":"2024-09-21T10:13:07.081940+0000","last_clean":"2024-09-21T10:13:07.081940+0000","last_became_active":"2024-09-21T10:07:43.794348+0000","last_became_peered":"2024-09-21T10:07:43.794348+0000","last_unstale":"2024-09-21T10:13:07.081940+0000","last_undegraded":"2024-09-21T10:13:07.081940+0000","last_fullsized":"2024-09-21T10:13:07.081940+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:11.159014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":19,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.327389+0000","last_change":"2024-09-21T10:13:00.064683+0000","last_active":"2024-09-21T10:13:11.327389+0000","last_peered":"2024-09-21T10:13:11.327389+0000","last_clean":"2024-09-21T10:13:11.327389+0000","last_became_active":"2024-09-21T10:13:00.064382+0000","last_became_peered":"2024-09-21T10:13:00.064382+0000","last_unstale":"2024-09-21T10:13:11.327389+0000","last_undegraded":"2024-09-21T10:13:11.327389+0000","last_fullsized":"2024-09-21T10:13:11.327389+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:46:48.424008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":215,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.082459+0000","last_change":"2024-09-21T10:08:00.412449+0000","last_active":"2024-09-21T10:13:07.082459+0000","last_peered":"2024-09-21T10:13:07.082459+0000","last_clean":"2024-09-21T10:13:07.082459+0000","last_became_active":"2024-09-21T10:08:00.386530+0000","last_became_peered":"2024-09-21T10:08:00.386530+0000","last_unstale":"2024-09-21T10:13:07.082459+0000","last_undegraded":"2024-09-21T10:13:07.082459+0000","last_fullsized":"2024-09-21T10:13:07.082459+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:12:50.581176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":203,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.918361+0000","last_change":"2024-09-21T10:08:18.475007+0000","last_active":"2024-09-21T10:13:11.918361+0000","last_peered":"2024-09-21T10:13:11.918361+0000","last_clean":"2024-09-21T10:13:11.918361+0000","last_became_active":"2024-09-21T10:08:18.474730+0000","last_became_peered":"2024-09-21T10:08:18.474730+0000","last_unstale":"2024-09-21T10:13:11.918361+0000","last_undegraded":"2024-09-21T10:13:11.918361+0000","last_fullsized":"2024-09-21T10:13:11.918361+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:35.455790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":19,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.917700+0000","last_change":"2024-09-21T10:13:00.064686+0000","last_active":"2024-09-21T10:13:11.917700+0000","last_peered":"2024-09-21T10:13:11.917700+0000","last_clean":"2024-09-21T10:13:11.917700+0000","last_became_active":"2024-09-21T10:13:00.064433+0000","last_became_peered":"2024-09-21T10:13:00.064433+0000","last_unstale":"2024-09-21T10:13:11.917700+0000","last_undegraded":"2024-09-21T10:13:11.917700+0000","last_fullsized":"2024-09-21T10:13:11.917700+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:08:28.139003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":155,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.082480+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:13:07.082480+0000","last_peered":"2024-09-21T10:13:07.082480+0000","last_clean":"2024-09-21T10:13:07.082480+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:13:07.082480+0000","last_undegraded":"2024-09-21T10:13:07.082480+0000","last_fullsized":"2024-09-21T10:13:07.082480+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":169,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.700303+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:13:07.700303+0000","last_peered":"2024-09-21T10:13:07.700303+0000","last_clean":"2024-09-21T10:13:07.700303+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:13:07.700303+0000","last_undegraded":"2024-09-21T10:13:07.700303+0000","last_fullsized":"2024-09-21T10:13:07.700303+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"206'353","reported_seq":372,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:15.002653+0000","last_change":"2024-09-21T10:13:00.064699+0000","last_active":"2024-09-21T10:13:15.002653+0000","last_peered":"2024-09-21T10:13:15.002653+0000","last_clean":"2024-09-21T10:13:15.002653+0000","last_became_active":"2024-09-21T10:13:00.064399+0000","last_became_peered":"2024-09-21T10:13:00.064399+0000","last_unstale":"2024-09-21T10:13:15.002653+0000","last_undegraded":"2024-09-21T10:13:15.002653+0000","last_fullsized":"2024-09-21T10:13:15.002653+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:17:32.049189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23134208,"num_objects":353,"num_object_clones":0,"num_object_copies":1059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":353,"num_write_kb":22592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":187,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.327774+0000","last_change":"2024-09-21T10:07:43.793008+0000","last_active":"2024-09-21T10:13:11.327774+0000","last_peered":"2024-09-21T10:13:11.327774+0000","last_clean":"2024-09-21T10:13:11.327774+0000","last_became_active":"2024-09-21T10:07:43.792683+0000","last_became_peered":"2024-09-21T10:07:43.792683+0000","last_unstale":"2024-09-21T10:13:11.327774+0000","last_undegraded":"2024-09-21T10:13:11.327774+0000","last_fullsized":"2024-09-21T10:13:11.327774+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:46:13.137478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":298,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.918506+0000","last_change":"2024-09-21T10:08:00.389645+0000","last_active":"2024-09-21T10:13:11.918506+0000","last_peered":"2024-09-21T10:13:11.918506+0000","last_clean":"2024-09-21T10:13:11.918506+0000","last_became_active":"2024-09-21T10:08:00.386076+0000","last_became_peered":"2024-09-21T10:08:00.386076+0000","last_unstale":"2024-09-21T10:13:11.918506+0000","last_undegraded":"2024-09-21T10:13:11.918506+0000","last_fullsized":"2024-09-21T10:13:11.918506+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:58:10.646600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":19,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.919249+0000","last_change":"2024-09-21T10:13:00.064670+0000","last_active":"2024-09-21T10:13:11.919249+0000","last_peered":"2024-09-21T10:13:11.919249+0000","last_clean":"2024-09-21T10:13:11.919249+0000","last_became_active":"2024-09-21T10:13:00.063176+0000","last_became_peered":"2024-09-21T10:13:00.063176+0000","last_unstale":"2024-09-21T10:13:11.919249+0000","last_undegraded":"2024-09-21T10:13:11.919249+0000","last_fullsized":"2024-09-21T10:13:11.919249+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:45:41.060053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":312,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.327607+0000","last_change":"2024-09-21T10:08:00.395898+0000","last_active":"2024-09-21T10:13:11.327607+0000","last_peered":"2024-09-21T10:13:11.327607+0000","last_clean":"2024-09-21T10:13:11.327607+0000","last_became_active":"2024-09-21T10:08:00.390462+0000","last_became_peered":"2024-09-21T10:08:00.390462+0000","last_unstale":"2024-09-21T10:13:11.327607+0000","last_undegraded":"2024-09-21T10:13:11.327607+0000","last_fullsized":"2024-09-21T10:13:11.327607+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:29:01.877089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":199,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.327814+0000","last_change":"2024-09-21T10:08:00.383016+0000","last_active":"2024-09-21T10:13:11.327814+0000","last_peered":"2024-09-21T10:13:11.327814+0000","last_clean":"2024-09-21T10:13:11.327814+0000","last_became_active":"2024-09-21T10:08:00.382708+0000","last_became_peered":"2024-09-21T10:08:00.382708+0000","last_unstale":"2024-09-21T10:13:11.327814+0000","last_undegraded":"2024-09-21T10:13:11.327814+0000","last_fullsized":"2024-09-21T10:13:11.327814+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:30:52.797259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"206'336","reported_seq":355,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:14.994297+0000","last_change":"2024-09-21T10:13:00.067640+0000","last_active":"2024-09-21T10:13:14.994297+0000","last_peered":"2024-09-21T10:13:14.994297+0000","last_clean":"2024-09-21T10:13:14.994297+0000","last_became_active":"2024-09-21T10:13:00.067380+0000","last_became_peered":"2024-09-21T10:13:00.067380+0000","last_unstale":"2024-09-21T10:13:14.994297+0000","last_undegraded":"2024-09-21T10:13:14.994297+0000","last_fullsized":"2024-09-21T10:13:14.994297+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:32:08.426925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22020096,"num_objects":336,"num_object_clones":0,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":336,"num_write_kb":21504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":200,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.700576+0000","last_change":"2024-09-21T10:08:00.379604+0000","last_active":"2024-09-21T10:13:07.700576+0000","last_peered":"2024-09-21T10:13:07.700576+0000","last_clean":"2024-09-21T10:13:07.700576+0000","last_became_active":"2024-09-21T10:08:00.379301+0000","last_became_peered":"2024-09-21T10:08:00.379301+0000","last_unstale":"2024-09-21T10:13:07.700576+0000","last_undegraded":"2024-09-21T10:13:07.700576+0000","last_fullsized":"2024-09-21T10:13:07.700576+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:32:41.963929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":210,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.918334+0000","last_change":"2024-09-21T10:08:00.387420+0000","last_active":"2024-09-21T10:13:11.918334+0000","last_peered":"2024-09-21T10:13:11.918334+0000","last_clean":"2024-09-21T10:13:11.918334+0000","last_became_active":"2024-09-21T10:08:00.386843+0000","last_became_peered":"2024-09-21T10:08:00.386843+0000","last_unstale":"2024-09-21T10:13:11.918334+0000","last_undegraded":"2024-09-21T10:13:11.918334+0000","last_fullsized":"2024-09-21T10:13:11.918334+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:51:00.654836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"206'6","reported_seq":17,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:13.274986+0000","last_change":"2024-09-21T10:13:00.067618+0000","last_active":"2024-09-21T10:13:13.274986+0000","last_peered":"2024-09-21T10:13:13.274986+0000","last_clean":"2024-09-21T10:13:13.274986+0000","last_became_active":"2024-09-21T10:13:00.067306+0000","last_became_peered":"2024-09-21T10:13:00.067306+0000","last_unstale":"2024-09-21T10:13:13.274986+0000","last_undegraded":"2024-09-21T10:13:13.274986+0000","last_fullsized":"2024-09-21T10:13:13.274986+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:45:24.816636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":194,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.917641+0000","last_change":"2024-09-21T10:08:00.382496+0000","last_active":"2024-09-21T10:13:11.917641+0000","last_peered":"2024-09-21T10:13:11.917641+0000","last_clean":"2024-09-21T10:13:11.917641+0000","last_became_active":"2024-09-21T10:08:00.382180+0000","last_became_peered":"2024-09-21T10:08:00.382180+0000","last_unstale":"2024-09-21T10:13:11.917641+0000","last_undegraded":"2024-09-21T10:13:11.917641+0000","last_fullsized":"2024-09-21T10:13:11.917641+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:16.074736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":139,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.700828+0000","last_change":"2024-09-21T10:08:38.906801+0000","last_active":"2024-09-21T10:13:07.700828+0000","last_peered":"2024-09-21T10:13:07.700828+0000","last_clean":"2024-09-21T10:13:07.700828+0000","last_became_active":"2024-09-21T10:08:38.906487+0000","last_became_peered":"2024-09-21T10:08:38.906487+0000","last_unstale":"2024-09-21T10:13:07.700828+0000","last_undegraded":"2024-09-21T10:13:07.700828+0000","last_fullsized":"2024-09-21T10:13:07.700828+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:31.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"206'4","reported_seq":16,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:13.264916+0000","last_change":"2024-09-21T10:13:00.063591+0000","last_active":"2024-09-21T10:13:13.264916+0000","last_peered":"2024-09-21T10:13:13.264916+0000","last_clean":"2024-09-21T10:13:13.264916+0000","last_became_active":"2024-09-21T10:13:00.063262+0000","last_became_peered":"2024-09-21T10:13:00.063262+0000","last_unstale":"2024-09-21T10:13:13.264916+0000","last_undegraded":"2024-09-21T10:13:13.264916+0000","last_fullsized":"2024-09-21T10:13:13.264916+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:52:08.153986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262144,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":283,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.327453+0000","last_change":"2024-09-21T10:02:14.097462+0000","last_active":"2024-09-21T10:13:11.327453+0000","last_peered":"2024-09-21T10:13:11.327453+0000","last_clean":"2024-09-21T10:13:11.327453+0000","last_became_active":"2024-09-21T10:02:14.097167+0000","last_became_peered":"2024-09-21T10:02:14.097167+0000","last_unstale":"2024-09-21T10:13:11.327453+0000","last_undegraded":"2024-09-21T10:13:11.327453+0000","last_fullsized":"2024-09-21T10:13:11.327453+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:54.922731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":287,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.919665+0000","last_change":"2024-09-21T10:08:00.378569+0000","last_active":"2024-09-21T10:13:11.919665+0000","last_peered":"2024-09-21T10:13:11.919665+0000","last_clean":"2024-09-21T10:13:11.919665+0000","last_became_active":"2024-09-21T10:08:00.378015+0000","last_became_peered":"2024-09-21T10:08:00.378015+0000","last_unstale":"2024-09-21T10:13:11.919665+0000","last_undegraded":"2024-09-21T10:13:11.919665+0000","last_fullsized":"2024-09-21T10:13:11.919665+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:58.121005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":19,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.918863+0000","last_change":"2024-09-21T10:13:00.066787+0000","last_active":"2024-09-21T10:13:11.918863+0000","last_peered":"2024-09-21T10:13:11.918863+0000","last_clean":"2024-09-21T10:13:11.918863+0000","last_became_active":"2024-09-21T10:13:00.066494+0000","last_became_peered":"2024-09-21T10:13:00.066494+0000","last_unstale":"2024-09-21T10:13:11.918863+0000","last_undegraded":"2024-09-21T10:13:11.918863+0000","last_fullsized":"2024-09-21T10:13:11.918863+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:40:20.387852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":288,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.082168+0000","last_change":"2024-09-21T09:59:54.265556+0000","last_active":"2024-09-21T10:13:07.082168+0000","last_peered":"2024-09-21T10:13:07.082168+0000","last_clean":"2024-09-21T10:13:07.082168+0000","last_became_active":"2024-09-21T09:59:54.265033+0000","last_became_peered":"2024-09-21T09:59:54.265033+0000","last_unstale":"2024-09-21T10:13:07.082168+0000","last_undegraded":"2024-09-21T10:13:07.082168+0000","last_fullsized":"2024-09-21T10:13:07.082168+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T09:48:34.850872+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:40.449756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":574,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.917541+0000","last_change":"2024-09-21T10:07:43.790490+0000","last_active":"2024-09-21T10:13:11.917541+0000","last_peered":"2024-09-21T10:13:11.917541+0000","last_clean":"2024-09-21T10:13:11.917541+0000","last_became_active":"2024-09-21T10:07:43.789986+0000","last_became_peered":"2024-09-21T10:07:43.789986+0000","last_unstale":"2024-09-21T10:13:11.917541+0000","last_undegraded":"2024-09-21T10:13:11.917541+0000","last_fullsized":"2024-09-21T10:13:11.917541+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:08.740356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":481,"num_read_kb":1580,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":236,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.327727+0000","last_change":"2024-09-21T10:02:14.099961+0000","last_active":"2024-09-21T10:13:11.327727+0000","last_peered":"2024-09-21T10:13:11.327727+0000","last_clean":"2024-09-21T10:13:11.327727+0000","last_became_active":"2024-09-21T10:02:14.099067+0000","last_became_peered":"2024-09-21T10:02:14.099067+0000","last_unstale":"2024-09-21T10:13:11.327727+0000","last_undegraded":"2024-09-21T10:13:11.327727+0000","last_fullsized":"2024-09-21T10:13:11.327727+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:41.037532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":182,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.700992+0000","last_change":"2024-09-21T10:08:00.386135+0000","last_active":"2024-09-21T10:13:07.700992+0000","last_peered":"2024-09-21T10:13:07.700992+0000","last_clean":"2024-09-21T10:13:07.700992+0000","last_became_active":"2024-09-21T10:08:00.385796+0000","last_became_peered":"2024-09-21T10:08:00.385796+0000","last_unstale":"2024-09-21T10:13:07.700992+0000","last_undegraded":"2024-09-21T10:13:07.700992+0000","last_fullsized":"2024-09-21T10:13:07.700992+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:27.054699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"206'11","reported_seq":23,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:13.293997+0000","last_change":"2024-09-21T10:13:00.065594+0000","last_active":"2024-09-21T10:13:13.293997+0000","last_peered":"2024-09-21T10:13:13.293997+0000","last_clean":"2024-09-21T10:13:13.293997+0000","last_became_active":"2024-09-21T10:13:00.065331+0000","last_became_peered":"2024-09-21T10:13:00.065331+0000","last_unstale":"2024-09-21T10:13:13.293997+0000","last_undegraded":"2024-09-21T10:13:13.293997+0000","last_fullsized":"2024-09-21T10:13:13.293997+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:58:00.082638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":720896,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":551,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.082077+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:13:07.082077+0000","last_peered":"2024-09-21T10:13:07.082077+0000","last_clean":"2024-09-21T10:13:07.082077+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:13:07.082077+0000","last_undegraded":"2024-09-21T10:13:07.082077+0000","last_fullsized":"2024-09-21T10:13:07.082077+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":299,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.917824+0000","last_change":"2024-09-21T10:08:00.383928+0000","last_active":"2024-09-21T10:13:11.917824+0000","last_peered":"2024-09-21T10:13:11.917824+0000","last_clean":"2024-09-21T10:13:11.917824+0000","last_became_active":"2024-09-21T10:08:00.383625+0000","last_became_peered":"2024-09-21T10:08:00.383625+0000","last_unstale":"2024-09-21T10:13:11.917824+0000","last_undegraded":"2024-09-21T10:13:11.917824+0000","last_fullsized":"2024-09-21T10:13:11.917824+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:39.207283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":289,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.919151+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:13:11.919151+0000","last_peered":"2024-09-21T10:13:11.919151+0000","last_clean":"2024-09-21T10:13:11.919151+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:13:11.919151+0000","last_undegraded":"2024-09-21T10:13:11.919151+0000","last_fullsized":"2024-09-21T10:13:11.919151+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":19,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.918611+0000","last_change":"2024-09-21T10:13:00.067116+0000","last_active":"2024-09-21T10:13:11.918611+0000","last_peered":"2024-09-21T10:13:11.918611+0000","last_clean":"2024-09-21T10:13:11.918611+0000","last_became_active":"2024-09-21T10:13:00.066827+0000","last_became_peered":"2024-09-21T10:13:00.066827+0000","last_unstale":"2024-09-21T10:13:11.918611+0000","last_undegraded":"2024-09-21T10:13:11.918611+0000","last_fullsized":"2024-09-21T10:13:11.918611+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:33:31.809910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":1026,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.917804+0000","last_change":"2024-09-21T10:02:14.100143+0000","last_active":"2024-09-21T10:13:11.917804+0000","last_peered":"2024-09-21T10:13:11.917804+0000","last_clean":"2024-09-21T10:13:11.917804+0000","last_became_active":"2024-09-21T10:02:14.099586+0000","last_became_peered":"2024-09-21T10:02:14.099586+0000","last_unstale":"2024-09-21T10:13:11.917804+0000","last_undegraded":"2024-09-21T10:13:11.917804+0000","last_fullsized":"2024-09-21T10:13:11.917804+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:34.064652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":161,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.919393+0000","last_change":"2024-09-21T10:08:18.470748+0000","last_active":"2024-09-21T10:13:11.919393+0000","last_peered":"2024-09-21T10:13:11.919393+0000","last_clean":"2024-09-21T10:13:11.919393+0000","last_became_active":"2024-09-21T10:08:18.470227+0000","last_became_peered":"2024-09-21T10:08:18.470227+0000","last_unstale":"2024-09-21T10:13:11.919393+0000","last_undegraded":"2024-09-21T10:13:11.919393+0000","last_fullsized":"2024-09-21T10:13:11.919393+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:16.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":283,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.918536+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:13:11.918536+0000","last_peered":"2024-09-21T10:13:11.918536+0000","last_clean":"2024-09-21T10:13:11.918536+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:13:11.918536+0000","last_undegraded":"2024-09-21T10:13:11.918536+0000","last_fullsized":"2024-09-21T10:13:11.918536+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"206'10","reported_seq":22,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:13.274988+0000","last_change":"2024-09-21T10:13:00.067624+0000","last_active":"2024-09-21T10:13:13.274988+0000","last_peered":"2024-09-21T10:13:13.274988+0000","last_clean":"2024-09-21T10:13:13.274988+0000","last_became_active":"2024-09-21T10:13:00.067336+0000","last_became_peered":"2024-09-21T10:13:00.067336+0000","last_unstale":"2024-09-21T10:13:13.274988+0000","last_undegraded":"2024-09-21T10:13:13.274988+0000","last_fullsized":"2024-09-21T10:13:13.274988+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:12.763796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":655360,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":759,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.918675+0000","last_change":"2024-09-21T10:07:43.794767+0000","last_active":"2024-09-21T10:13:11.918675+0000","last_peered":"2024-09-21T10:13:11.918675+0000","last_clean":"2024-09-21T10:13:11.918675+0000","last_became_active":"2024-09-21T10:07:43.794062+0000","last_became_peered":"2024-09-21T10:07:43.794062+0000","last_unstale":"2024-09-21T10:13:11.918675+0000","last_undegraded":"2024-09-21T10:13:11.918675+0000","last_fullsized":"2024-09-21T10:13:11.918675+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:51.977955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":186,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.918286+0000","last_change":"2024-09-21T10:08:00.389176+0000","last_active":"2024-09-21T10:13:11.918286+0000","last_peered":"2024-09-21T10:13:11.918286+0000","last_clean":"2024-09-21T10:13:11.918286+0000","last_became_active":"2024-09-21T10:08:00.383854+0000","last_became_peered":"2024-09-21T10:08:00.383854+0000","last_unstale":"2024-09-21T10:13:11.918286+0000","last_undegraded":"2024-09-21T10:13:11.918286+0000","last_fullsized":"2024-09-21T10:13:11.918286+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:46.952216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":199,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.917556+0000","last_change":"2024-09-21T10:07:43.792153+0000","last_active":"2024-09-21T10:13:11.917556+0000","last_peered":"2024-09-21T10:13:11.917556+0000","last_clean":"2024-09-21T10:13:11.917556+0000","last_became_active":"2024-09-21T10:07:43.791770+0000","last_became_peered":"2024-09-21T10:07:43.791770+0000","last_unstale":"2024-09-21T10:13:11.917556+0000","last_undegraded":"2024-09-21T10:13:11.917556+0000","last_fullsized":"2024-09-21T10:13:11.917556+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:52.506941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"206'571","reported_seq":590,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:16.115551+0000","last_change":"2024-09-21T10:13:00.064709+0000","last_active":"2024-09-21T10:13:16.115551+0000","last_peered":"2024-09-21T10:13:16.115551+0000","last_clean":"2024-09-21T10:13:16.115551+0000","last_became_active":"2024-09-21T10:13:00.064453+0000","last_became_peered":"2024-09-21T10:13:00.064453+0000","last_unstale":"2024-09-21T10:13:16.115551+0000","last_undegraded":"2024-09-21T10:13:16.115551+0000","last_fullsized":"2024-09-21T10:13:16.115551+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":571,"log_dups_size":0,"ondisk_log_size":571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:17:27.616341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37421056,"num_objects":571,"num_object_clones":0,"num_object_copies":1713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":571,"num_write_kb":36544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":204,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.327945+0000","last_change":"2024-09-21T10:07:43.791405+0000","last_active":"2024-09-21T10:13:11.327945+0000","last_peered":"2024-09-21T10:13:11.327945+0000","last_clean":"2024-09-21T10:13:11.327945+0000","last_became_active":"2024-09-21T10:07:43.791078+0000","last_became_peered":"2024-09-21T10:07:43.791078+0000","last_unstale":"2024-09-21T10:13:11.327945+0000","last_undegraded":"2024-09-21T10:13:11.327945+0000","last_fullsized":"2024-09-21T10:13:11.327945+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:17:38.072612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":866,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.919037+0000","last_change":"2024-09-21T10:08:18.468995+0000","last_active":"2024-09-21T10:13:11.919037+0000","last_peered":"2024-09-21T10:13:11.919037+0000","last_clean":"2024-09-21T10:13:11.919037+0000","last_became_active":"2024-09-21T10:08:18.468243+0000","last_became_peered":"2024-09-21T10:08:18.468243+0000","last_unstale":"2024-09-21T10:13:11.919037+0000","last_undegraded":"2024-09-21T10:13:11.919037+0000","last_fullsized":"2024-09-21T10:13:11.919037+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:45.520968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":185,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.918905+0000","last_change":"2024-09-21T10:08:18.468595+0000","last_active":"2024-09-21T10:13:11.918905+0000","last_peered":"2024-09-21T10:13:11.918905+0000","last_clean":"2024-09-21T10:13:11.918905+0000","last_became_active":"2024-09-21T10:08:18.468296+0000","last_became_peered":"2024-09-21T10:08:18.468296+0000","last_unstale":"2024-09-21T10:13:11.918905+0000","last_undegraded":"2024-09-21T10:13:11.918905+0000","last_fullsized":"2024-09-21T10:13:11.918905+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:42.748130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"206'577","reported_seq":596,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:16.110608+0000","last_change":"2024-09-21T10:13:00.063878+0000","last_active":"2024-09-21T10:13:16.110608+0000","last_peered":"2024-09-21T10:13:16.110608+0000","last_clean":"2024-09-21T10:13:16.110608+0000","last_became_active":"2024-09-21T10:13:00.063612+0000","last_became_peered":"2024-09-21T10:13:00.063612+0000","last_unstale":"2024-09-21T10:13:16.110608+0000","last_undegraded":"2024-09-21T10:13:16.110608+0000","last_fullsized":"2024-09-21T10:13:16.110608+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":577,"log_dups_size":0,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:42.374449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37814272,"num_objects":577,"num_object_clones":0,"num_object_copies":1731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":577,"num_write_kb":36928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":161,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.919222+0000","last_change":"2024-09-21T10:08:18.469907+0000","last_active":"2024-09-21T10:13:11.919222+0000","last_peered":"2024-09-21T10:13:11.919222+0000","last_clean":"2024-09-21T10:13:11.919222+0000","last_became_active":"2024-09-21T10:08:18.469443+0000","last_became_peered":"2024-09-21T10:08:18.469443+0000","last_unstale":"2024-09-21T10:13:11.919222+0000","last_undegraded":"2024-09-21T10:13:11.919222+0000","last_fullsized":"2024-09-21T10:13:11.919222+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:15.645803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":601,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.082531+0000","last_change":"2024-09-21T10:08:00.419641+0000","last_active":"2024-09-21T10:13:07.082531+0000","last_peered":"2024-09-21T10:13:07.082531+0000","last_clean":"2024-09-21T10:13:07.082531+0000","last_became_active":"2024-09-21T10:08:00.387050+0000","last_became_peered":"2024-09-21T10:08:00.387050+0000","last_unstale":"2024-09-21T10:13:07.082531+0000","last_undegraded":"2024-09-21T10:13:07.082531+0000","last_fullsized":"2024-09-21T10:13:07.082531+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:00:32.881619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":267,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.918479+0000","last_change":"2024-09-21T09:59:54.267798+0000","last_active":"2024-09-21T10:13:11.918479+0000","last_peered":"2024-09-21T10:13:11.918479+0000","last_clean":"2024-09-21T10:13:11.918479+0000","last_became_active":"2024-09-21T09:59:54.265615+0000","last_became_peered":"2024-09-21T09:59:54.265615+0000","last_unstale":"2024-09-21T10:13:11.918479+0000","last_undegraded":"2024-09-21T10:13:11.918479+0000","last_fullsized":"2024-09-21T10:13:11.918479+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:30.363872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":19,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.327468+0000","last_change":"2024-09-21T10:13:00.064672+0000","last_active":"2024-09-21T10:13:11.327468+0000","last_peered":"2024-09-21T10:13:11.327468+0000","last_clean":"2024-09-21T10:13:11.327468+0000","last_became_active":"2024-09-21T10:13:00.064381+0000","last_became_peered":"2024-09-21T10:13:00.064381+0000","last_unstale":"2024-09-21T10:13:11.327468+0000","last_undegraded":"2024-09-21T10:13:11.327468+0000","last_fullsized":"2024-09-21T10:13:11.327468+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:40.727640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":280,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.328068+0000","last_change":"2024-09-21T10:02:14.101943+0000","last_active":"2024-09-21T10:13:11.328068+0000","last_peered":"2024-09-21T10:13:11.328068+0000","last_clean":"2024-09-21T10:13:11.328068+0000","last_became_active":"2024-09-21T10:02:14.101488+0000","last_became_peered":"2024-09-21T10:02:14.101488+0000","last_unstale":"2024-09-21T10:13:11.328068+0000","last_undegraded":"2024-09-21T10:13:11.328068+0000","last_fullsized":"2024-09-21T10:13:11.328068+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:52:27.866414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":530,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.919587+0000","last_change":"2024-09-21T10:08:18.472876+0000","last_active":"2024-09-21T10:13:11.919587+0000","last_peered":"2024-09-21T10:13:11.919587+0000","last_clean":"2024-09-21T10:13:11.919587+0000","last_became_active":"2024-09-21T10:08:18.472385+0000","last_became_peered":"2024-09-21T10:08:18.472385+0000","last_unstale":"2024-09-21T10:13:11.919587+0000","last_undegraded":"2024-09-21T10:13:11.919587+0000","last_fullsized":"2024-09-21T10:13:11.919587+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:33.109956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":98,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.917977+0000","last_change":"2024-09-21T10:08:38.797352+0000","last_active":"2024-09-21T10:13:11.917977+0000","last_peered":"2024-09-21T10:13:11.917977+0000","last_clean":"2024-09-21T10:13:11.917977+0000","last_became_active":"2024-09-21T10:08:38.796950+0000","last_became_peered":"2024-09-21T10:08:38.796950+0000","last_unstale":"2024-09-21T10:13:11.917977+0000","last_undegraded":"2024-09-21T10:13:11.917977+0000","last_fullsized":"2024-09-21T10:13:11.917977+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:02.646321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":19,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.327585+0000","last_change":"2024-09-21T10:13:00.064993+0000","last_active":"2024-09-21T10:13:11.327585+0000","last_peered":"2024-09-21T10:13:11.327585+0000","last_clean":"2024-09-21T10:13:11.327585+0000","last_became_active":"2024-09-21T10:13:00.063504+0000","last_became_peered":"2024-09-21T10:13:00.063504+0000","last_unstale":"2024-09-21T10:13:11.327585+0000","last_undegraded":"2024-09-21T10:13:11.327585+0000","last_fullsized":"2024-09-21T10:13:11.327585+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:29:00.080618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":315,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.918993+0000","last_change":"2024-09-21T10:08:00.390436+0000","last_active":"2024-09-21T10:13:11.918993+0000","last_peered":"2024-09-21T10:13:11.918993+0000","last_clean":"2024-09-21T10:13:11.918993+0000","last_became_active":"2024-09-21T10:08:00.381835+0000","last_became_peered":"2024-09-21T10:08:00.381835+0000","last_unstale":"2024-09-21T10:13:11.918993+0000","last_undegraded":"2024-09-21T10:13:11.918993+0000","last_fullsized":"2024-09-21T10:13:11.918993+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:01:35.697356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":432,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.700969+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:13:07.700969+0000","last_peered":"2024-09-21T10:13:07.700969+0000","last_clean":"2024-09-21T10:13:07.700969+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:13:07.700969+0000","last_undegraded":"2024-09-21T10:13:07.700969+0000","last_fullsized":"2024-09-21T10:13:07.700969+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":70,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.700436+0000","last_change":"2024-09-21T10:08:00.385774+0000","last_active":"2024-09-21T10:13:07.700436+0000","last_peered":"2024-09-21T10:13:07.700436+0000","last_clean":"2024-09-21T10:13:07.700436+0000","last_became_active":"2024-09-21T10:08:00.385445+0000","last_became_peered":"2024-09-21T10:08:00.385445+0000","last_unstale":"2024-09-21T10:13:07.700436+0000","last_undegraded":"2024-09-21T10:13:07.700436+0000","last_fullsized":"2024-09-21T10:13:07.700436+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:42.442190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":129,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.700578+0000","last_change":"2024-09-21T10:08:00.379661+0000","last_active":"2024-09-21T10:13:07.700578+0000","last_peered":"2024-09-21T10:13:07.700578+0000","last_clean":"2024-09-21T10:13:07.700578+0000","last_became_active":"2024-09-21T10:08:00.379301+0000","last_became_peered":"2024-09-21T10:08:00.379301+0000","last_unstale":"2024-09-21T10:13:07.700578+0000","last_undegraded":"2024-09-21T10:13:07.700578+0000","last_fullsized":"2024-09-21T10:13:07.700578+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:15.649523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":153,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.082231+0000","last_change":"2024-09-21T10:08:00.410731+0000","last_active":"2024-09-21T10:13:07.082231+0000","last_peered":"2024-09-21T10:13:07.082231+0000","last_clean":"2024-09-21T10:13:07.082231+0000","last_became_active":"2024-09-21T10:08:00.380324+0000","last_became_peered":"2024-09-21T10:08:00.380324+0000","last_unstale":"2024-09-21T10:13:07.082231+0000","last_undegraded":"2024-09-21T10:13:07.082231+0000","last_fullsized":"2024-09-21T10:13:07.082231+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:47:34.960221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":161,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.918767+0000","last_change":"2024-09-21T10:08:18.470378+0000","last_active":"2024-09-21T10:13:11.918767+0000","last_peered":"2024-09-21T10:13:11.918767+0000","last_clean":"2024-09-21T10:13:11.918767+0000","last_became_active":"2024-09-21T10:08:18.469482+0000","last_became_peered":"2024-09-21T10:08:18.469482+0000","last_unstale":"2024-09-21T10:13:11.918767+0000","last_undegraded":"2024-09-21T10:13:11.918767+0000","last_fullsized":"2024-09-21T10:13:11.918767+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:35.628724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":222,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.082382+0000","last_change":"2024-09-21T10:08:00.403606+0000","last_active":"2024-09-21T10:13:07.082382+0000","last_peered":"2024-09-21T10:13:07.082382+0000","last_clean":"2024-09-21T10:13:07.082382+0000","last_became_active":"2024-09-21T10:08:00.379330+0000","last_became_peered":"2024-09-21T10:08:00.379330+0000","last_unstale":"2024-09-21T10:13:07.082382+0000","last_undegraded":"2024-09-21T10:13:07.082382+0000","last_fullsized":"2024-09-21T10:13:07.082382+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:07.226027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":118,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.082355+0000","last_change":"2024-09-21T10:07:43.791830+0000","last_active":"2024-09-21T10:13:07.082355+0000","last_peered":"2024-09-21T10:13:07.082355+0000","last_clean":"2024-09-21T10:13:07.082355+0000","last_became_active":"2024-09-21T10:07:43.791314+0000","last_became_peered":"2024-09-21T10:07:43.791314+0000","last_unstale":"2024-09-21T10:13:07.082355+0000","last_undegraded":"2024-09-21T10:13:07.082355+0000","last_fullsized":"2024-09-21T10:13:07.082355+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:37.086629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":236,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.327302+0000","last_change":"2024-09-21T10:02:14.100451+0000","last_active":"2024-09-21T10:13:11.327302+0000","last_peered":"2024-09-21T10:13:11.327302+0000","last_clean":"2024-09-21T10:13:11.327302+0000","last_became_active":"2024-09-21T10:02:14.100160+0000","last_became_peered":"2024-09-21T10:02:14.100160+0000","last_unstale":"2024-09-21T10:13:11.327302+0000","last_undegraded":"2024-09-21T10:13:11.327302+0000","last_fullsized":"2024-09-21T10:13:11.327302+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:28.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":277,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.919391+0000","last_change":"2024-09-21T10:02:14.096098+0000","last_active":"2024-09-21T10:13:11.919391+0000","last_peered":"2024-09-21T10:13:11.919391+0000","last_clean":"2024-09-21T10:13:11.919391+0000","last_became_active":"2024-09-21T10:02:14.095811+0000","last_became_peered":"2024-09-21T10:02:14.095811+0000","last_unstale":"2024-09-21T10:13:11.919391+0000","last_undegraded":"2024-09-21T10:13:11.919391+0000","last_fullsized":"2024-09-21T10:13:11.919391+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:54.619333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":281,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.327326+0000","last_change":"2024-09-21T10:02:14.100712+0000","last_active":"2024-09-21T10:13:11.327326+0000","last_peered":"2024-09-21T10:13:11.327326+0000","last_clean":"2024-09-21T10:13:11.327326+0000","last_became_active":"2024-09-21T10:02:14.100427+0000","last_became_peered":"2024-09-21T10:02:14.100427+0000","last_unstale":"2024-09-21T10:13:11.327326+0000","last_undegraded":"2024-09-21T10:13:11.327326+0000","last_fullsized":"2024-09-21T10:13:11.327326+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:16:42.350642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":169,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.700919+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:13:07.700919+0000","last_peered":"2024-09-21T10:13:07.700919+0000","last_clean":"2024-09-21T10:13:07.700919+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:13:07.700919+0000","last_undegraded":"2024-09-21T10:13:07.700919+0000","last_fullsized":"2024-09-21T10:13:07.700919+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":299,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.327472+0000","last_change":"2024-09-21T10:02:14.097369+0000","last_active":"2024-09-21T10:13:11.327472+0000","last_peered":"2024-09-21T10:13:11.327472+0000","last_clean":"2024-09-21T10:13:11.327472+0000","last_became_active":"2024-09-21T10:02:14.097067+0000","last_became_peered":"2024-09-21T10:02:14.097067+0000","last_unstale":"2024-09-21T10:13:11.327472+0000","last_undegraded":"2024-09-21T10:13:11.327472+0000","last_fullsized":"2024-09-21T10:13:11.327472+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:50:48.123813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":285,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.919094+0000","last_change":"2024-09-21T10:12:27.977282+0000","last_active":"2024-09-21T10:13:11.919094+0000","last_peered":"2024-09-21T10:13:11.919094+0000","last_clean":"2024-09-21T10:13:11.919094+0000","last_became_active":"2024-09-21T10:12:27.976924+0000","last_became_peered":"2024-09-21T10:12:27.976924+0000","last_unstale":"2024-09-21T10:13:11.919094+0000","last_undegraded":"2024-09-21T10:13:11.919094+0000","last_fullsized":"2024-09-21T10:13:11.919094+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:57:19.477172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":287,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.918445+0000","last_change":"2024-09-21T10:08:00.381058+0000","last_active":"2024-09-21T10:13:11.918445+0000","last_peered":"2024-09-21T10:13:11.918445+0000","last_clean":"2024-09-21T10:13:11.918445+0000","last_became_active":"2024-09-21T10:08:00.380573+0000","last_became_peered":"2024-09-21T10:08:00.380573+0000","last_unstale":"2024-09-21T10:13:11.918445+0000","last_undegraded":"2024-09-21T10:13:11.918445+0000","last_fullsized":"2024-09-21T10:13:11.918445+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:01:04.137425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":302,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.327752+0000","last_change":"2024-09-21T10:08:00.387321+0000","last_active":"2024-09-21T10:13:11.327752+0000","last_peered":"2024-09-21T10:13:11.327752+0000","last_clean":"2024-09-21T10:13:11.327752+0000","last_became_active":"2024-09-21T10:08:00.386935+0000","last_became_peered":"2024-09-21T10:08:00.386935+0000","last_unstale":"2024-09-21T10:13:11.327752+0000","last_undegraded":"2024-09-21T10:13:11.327752+0000","last_fullsized":"2024-09-21T10:13:11.327752+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:57.706313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":106,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.918922+0000","last_change":"2024-09-21T10:08:18.471282+0000","last_active":"2024-09-21T10:13:11.918922+0000","last_peered":"2024-09-21T10:13:11.918922+0000","last_clean":"2024-09-21T10:13:11.918922+0000","last_became_active":"2024-09-21T10:08:18.470952+0000","last_became_peered":"2024-09-21T10:08:18.470952+0000","last_unstale":"2024-09-21T10:13:11.918922+0000","last_undegraded":"2024-09-21T10:13:11.918922+0000","last_fullsized":"2024-09-21T10:13:11.918922+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:25.597081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":127,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.919253+0000","last_change":"2024-09-21T10:11:54.672536+0000","last_active":"2024-09-21T10:13:11.919253+0000","last_peered":"2024-09-21T10:13:11.919253+0000","last_clean":"2024-09-21T10:13:11.919253+0000","last_became_active":"2024-09-21T10:11:54.672206+0000","last_became_peered":"2024-09-21T10:11:54.672206+0000","last_unstale":"2024-09-21T10:13:11.919253+0000","last_undegraded":"2024-09-21T10:13:11.919253+0000","last_fullsized":"2024-09-21T10:13:11.919253+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:45:20.652276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":291,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.918388+0000","last_change":"2024-09-21T10:08:00.386104+0000","last_active":"2024-09-21T10:13:11.918388+0000","last_peered":"2024-09-21T10:13:11.918388+0000","last_clean":"2024-09-21T10:13:11.918388+0000","last_became_active":"2024-09-21T10:08:00.381484+0000","last_became_peered":"2024-09-21T10:08:00.381484+0000","last_unstale":"2024-09-21T10:13:11.918388+0000","last_undegraded":"2024-09-21T10:13:11.918388+0000","last_fullsized":"2024-09-21T10:13:11.918388+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:39.190191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":216,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.327571+0000","last_change":"2024-09-21T10:08:00.381149+0000","last_active":"2024-09-21T10:13:11.327571+0000","last_peered":"2024-09-21T10:13:11.327571+0000","last_clean":"2024-09-21T10:13:11.327571+0000","last_became_active":"2024-09-21T10:08:00.380795+0000","last_became_peered":"2024-09-21T10:08:00.380795+0000","last_unstale":"2024-09-21T10:13:11.327571+0000","last_undegraded":"2024-09-21T10:13:11.327571+0000","last_fullsized":"2024-09-21T10:13:11.327571+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:50:46.793182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":117,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.919318+0000","last_change":"2024-09-21T10:08:18.466882+0000","last_active":"2024-09-21T10:13:11.919318+0000","last_peered":"2024-09-21T10:13:11.919318+0000","last_clean":"2024-09-21T10:13:11.919318+0000","last_became_active":"2024-09-21T10:08:18.466470+0000","last_became_peered":"2024-09-21T10:08:18.466470+0000","last_unstale":"2024-09-21T10:13:11.919318+0000","last_undegraded":"2024-09-21T10:13:11.919318+0000","last_fullsized":"2024-09-21T10:13:11.919318+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:51:31.020051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":216,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.327819+0000","last_change":"2024-09-21T10:08:00.389061+0000","last_active":"2024-09-21T10:13:11.327819+0000","last_peered":"2024-09-21T10:13:11.327819+0000","last_clean":"2024-09-21T10:13:11.327819+0000","last_became_active":"2024-09-21T10:08:00.388657+0000","last_became_peered":"2024-09-21T10:08:00.388657+0000","last_unstale":"2024-09-21T10:13:11.327819+0000","last_undegraded":"2024-09-21T10:13:11.327819+0000","last_fullsized":"2024-09-21T10:13:11.327819+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:58:17.239589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":81,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.700985+0000","last_change":"2024-09-21T10:08:00.394350+0000","last_active":"2024-09-21T10:13:07.700985+0000","last_peered":"2024-09-21T10:13:07.700985+0000","last_clean":"2024-09-21T10:13:07.700985+0000","last_became_active":"2024-09-21T10:08:00.387232+0000","last_became_peered":"2024-09-21T10:08:00.387232+0000","last_unstale":"2024-09-21T10:13:07.700985+0000","last_undegraded":"2024-09-21T10:13:07.700985+0000","last_fullsized":"2024-09-21T10:13:07.700985+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:56.160018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":302,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.327472+0000","last_change":"2024-09-21T10:02:14.099799+0000","last_active":"2024-09-21T10:13:11.327472+0000","last_peered":"2024-09-21T10:13:11.327472+0000","last_clean":"2024-09-21T10:13:11.327472+0000","last_became_active":"2024-09-21T10:02:14.099375+0000","last_became_peered":"2024-09-21T10:02:14.099375+0000","last_unstale":"2024-09-21T10:13:11.327472+0000","last_undegraded":"2024-09-21T10:13:11.327472+0000","last_fullsized":"2024-09-21T10:13:11.327472+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:13.407638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":288,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.327922+0000","last_change":"2024-09-21T10:02:14.099439+0000","last_active":"2024-09-21T10:13:11.327922+0000","last_peered":"2024-09-21T10:13:11.327922+0000","last_clean":"2024-09-21T10:13:11.327922+0000","last_became_active":"2024-09-21T10:02:14.099117+0000","last_became_peered":"2024-09-21T10:02:14.099117+0000","last_unstale":"2024-09-21T10:13:11.327922+0000","last_undegraded":"2024-09-21T10:13:11.327922+0000","last_fullsized":"2024-09-21T10:13:11.327922+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:39.681817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":144,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.700303+0000","last_change":"2024-09-21T10:08:00.381529+0000","last_active":"2024-09-21T10:13:07.700303+0000","last_peered":"2024-09-21T10:13:07.700303+0000","last_clean":"2024-09-21T10:13:07.700303+0000","last_became_active":"2024-09-21T10:08:00.381061+0000","last_became_peered":"2024-09-21T10:08:00.381061+0000","last_unstale":"2024-09-21T10:13:07.700303+0000","last_undegraded":"2024-09-21T10:13:07.700303+0000","last_fullsized":"2024-09-21T10:13:07.700303+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:56:58.775586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":282,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-21T10:13:11.919283+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:13:11.919283+0000","last_peered":"2024-09-21T10:13:11.919283+0000","last_clean":"2024-09-21T10:13:11.919283+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:13:11.919283+0000","last_undegraded":"2024-09-21T10:13:11.919283+0000","last_fullsized":"2024-09-21T10:13:11.919283+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":161611776,"num_objects":2466,"num_object_clones":0,"num_object_copies":7398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2466,"num_write_kb":157824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":407961600,"data_stored":407961600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2466,"ondisk_log_size":2466,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":9926,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":21480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5982,"num_read_kb":13005,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397781,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308604,"kb_used_data":7088,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93436548,"statfs":{"total":95995035648,"available":95679025152,"internally_reserved":0,"allocated":7258112,"data_stored":6252800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594009,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308664,"kb_used_data":2988,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93436488,"statfs":{"total":95995035648,"available":95678963712,"internally_reserved":0,"allocated":3059712,"data_stored":2058403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790239,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312164,"kb_used_data":6488,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93432988,"statfs":{"total":95995035648,"available":95675379712,"internally_reserved":0,"allocated":6643712,"data_stored":5657816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986468,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":101,"seq":433791697030,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297120,"kb_used_data":2388,"kb_used_omap":5,"kb_used_meta":294714,"kb_avail":93448032,"statfs":{"total":95995035648,"available":95690784768,"internally_reserved":0,"allocated":2445312,"data_stored":1463270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":301787920},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860557,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":450988,"kb_used_data":112164,"kb_used_omap":8,"kb_used_meta":338807,"kb_avail":93294164,"statfs":{"total":95995035648,"available":95533223936,"internally_reserved":0,"allocated":114855936,"data_stored":113862776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":346939244},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024084,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":473112,"kb_used_data":180432,"kb_used_omap":7,"kb_used_meta":292664,"kb_avail":93272040,"statfs":{"total":95995035648,"available":95510568960,"internally_reserved":0,"allocated":184762368,"data_stored":183783751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":299688498},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187618,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":329696,"kb_used_data":2964,"kb_used_omap":8,"kb_used_meta":326711,"kb_avail":93415456,"statfs":{"total":95995035648,"available":95657426944,"internally_reserved":0,"allocated":3035136,"data_stored":2045506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":334553073},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":255197184,"data_stored":255197184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":132251648,"data_stored":132251648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":13303808,"data_stored":13303808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2949120,"data_stored":2949120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4259840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:13:21.183 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T10:13:21.185 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_2 pg_num 26 2024-09-21T10:13:21.190 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:21.217 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:21.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:21.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:21.241 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 8 16 21388 21372 166.914 150.062 0.00327682 0.00598458 2024-09-21T10:13:21.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:21.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:21.361 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:21.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:21.462 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:21.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:21.501 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:21.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:21.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:21.644 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:21.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:21.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:21.745 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:21.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:21.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:21.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:21.907 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:21.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:21.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:22.009 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:22.035 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:22.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:22.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:22.137 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:22.157 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:22.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:22.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:22.241 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 9 16 23359 23343 162.056 123.188 0.00348251 0.0061629 2024-09-21T10:13:22.259 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:22.298 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:22.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:22.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:22.400 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:22.427 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:22.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:22.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:22.530 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:22.552 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:22.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:22.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:22.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:22 smithi098 ceph-mon[100914]: pgmap v629: 89 pgs: 89 active+clean; 966 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 95 MiB/s wr, 1.51k op/s 2024-09-21T10:13:22.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:22.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:22.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:22 smithi098 ceph-mon[97580]: pgmap v629: 89 pgs: 89 active+clean; 966 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 95 MiB/s wr, 1.51k op/s 2024-09-21T10:13:22.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:22.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:22.653 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:22.680 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:22.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:22.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:22.782 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:22.808 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:22.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:22.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:22 smithi196 ceph-mon[84784]: pgmap v629: 89 pgs: 89 active+clean; 966 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 95 MiB/s wr, 1.51k op/s 2024-09-21T10:13:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:22.910 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:22.937 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:22.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:22.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:23.038 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:23.064 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:23.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:23.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:23.155 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:23.166 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:23.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:23.191 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:23.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:23.242 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 10 16 25228 25212 157.523 116.812 0.00728183 0.00634198 2024-09-21T10:13:23.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:23.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:23.337 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:23.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:23.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:23.461 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:23.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:23.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:23.563 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:23.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:23.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:23.705 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:23.744 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:23.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:23.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:23.846 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:23.867 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:23.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:23.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:23.969 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:23.996 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:23.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:23.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:24.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:24.124 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:24.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:24.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:24.226 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:24.242 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 11 16 26915 26899 152.788 105.438 0.0091399 0.00653943 2024-09-21T10:13:24.264 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:24.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:24.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:24.366 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:24.393 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:24.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:24.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:24.409 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:13:24.495 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:24.521 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:24.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:24.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:24.623 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:24.645 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:24.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:24.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:24.747 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:24.774 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:24.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:24.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:24 smithi098 ceph-mon[97580]: pgmap v630: 89 pgs: 89 active+clean; 1.2 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-21T10:13:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:24 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1562499671' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:13:24.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:24 smithi098 ceph-mon[100914]: pgmap v630: 89 pgs: 89 active+clean; 1.2 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-21T10:13:24.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:24 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1562499671' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:13:24.876 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:24 smithi196 ceph-mon[84784]: pgmap v630: 89 pgs: 89 active+clean; 1.2 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-21T10:13:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:24 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1562499671' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:13:24.900 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:24.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:24.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:25.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:25.041 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:25.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:25.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:25.143 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:25.184 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:25.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:25.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:25.242 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 12 16 28725 28709 149.483 113.125 0.00534364 0.00668319 2024-09-21T10:13:25.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:25.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:25.325 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:25.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:25.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:25.462 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:25.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:25.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:25.564 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:25.586 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:25.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:25.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:25.649 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:25 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:13:25.649 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:25 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1562499671' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:13:25.650 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:25 smithi098 ceph-mon[100914]: osdmap e207: 8 total, 8 up, 7 in 2024-09-21T10:13:25.650 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:25 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:13:25.650 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:25 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1562499671' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:13:25.650 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:25 smithi098 ceph-mon[97580]: osdmap e207: 8 total, 8 up, 7 in 2024-09-21T10:13:25.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:25.726 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:25.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:25.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:25.818 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:13:25.818 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:13:25.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:25.852 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:25.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:25.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:25 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:13:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:25 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1562499671' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:13:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:25 smithi196 ceph-mon[84784]: osdmap e207: 8 total, 8 up, 7 in 2024-09-21T10:13:25.955 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:25.980 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:25.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:25.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:26.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:26.110 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:26.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:26.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:26.212 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:26.240 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:26.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:26.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:26.242 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 13 16 30614 30598 147.065 118.062 0.00930608 0.00679413 2024-09-21T10:13:26.341 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:26.379 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:26.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:26.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:26.481 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:26.509 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:26.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:26.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:26.611 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:26.638 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:26.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:26.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:26.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:26.777 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:26.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:26.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:26.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:26 smithi098 ceph-mon[97580]: pgmap v632: 89 pgs: 89 active+clean; 1.5 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 148 MiB/s wr, 2.37k op/s 2024-09-21T10:13:26.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:26 smithi098 ceph-mon[100914]: pgmap v632: 89 pgs: 89 active+clean; 1.5 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 148 MiB/s wr, 2.37k op/s 2024-09-21T10:13:26.879 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:26 smithi196 ceph-mon[84784]: pgmap v632: 89 pgs: 89 active+clean; 1.5 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 148 MiB/s wr, 2.37k op/s 2024-09-21T10:13:26.905 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:26.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:26.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:26.988 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:27.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:27.052 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:27.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:27.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:27.154 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:27.192 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:27.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:27.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:27.242 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 14 16 32361 32345 144.36 109.188 0.00349453 0.00692246 2024-09-21T10:13:27.294 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:27.315 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:27.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:27.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:27.417 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:27.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:27.454 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:27.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:27.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:27.583 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:27.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:27.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:27.685 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:27.712 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:27.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:27.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:27.814 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:27.838 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:27.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:27.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:27.940 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:27.979 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:27.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:27.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:28.080 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:28.108 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:28.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:28.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:28.209 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:28.242 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 15 16 34220 34204 142.481 116.188 0.00917217 0.00701383 2024-09-21T10:13:28.244 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:28.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:28.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:28.346 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:28.350 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:13:28.385 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:28.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:28.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:28.487 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:28.533 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:28.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:28.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:28.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:28 smithi098 ceph-mon[97580]: pgmap v633: 89 pgs: 89 active+clean; 1.6 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 148 MiB/s wr, 2.36k op/s 2024-09-21T10:13:28.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:28 smithi098 ceph-mon[100914]: pgmap v633: 89 pgs: 89 active+clean; 1.6 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 148 MiB/s wr, 2.36k op/s 2024-09-21T10:13:28.635 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:28.659 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:28.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:28.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:28.761 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:28.806 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:28.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:28.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:28 smithi196 ceph-mon[84784]: pgmap v633: 89 pgs: 89 active+clean; 1.6 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 148 MiB/s wr, 2.36k op/s 2024-09-21T10:13:28.908 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:28.935 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:28.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:28.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:29.036 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:29.063 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:29.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:29.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:29.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:29.192 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:29.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:29.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:29.242 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 16 16 36040 36024 140.685 113.75 0.00848308 0.00710256 2024-09-21T10:13:29.294 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:29.341 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:29.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:29.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:29.443 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:29.495 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:29.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:29.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:29.597 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:29.623 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:29.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:29.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:29.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:29.765 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:29.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:29.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:29.866 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:29.894 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:29.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:29.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:29.996 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:30.091 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:30.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:30.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:30.193 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:30.219 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:30.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:30.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:30.242 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 17 16 37846 37830 139.049 112.875 0.0122264 0.00718653 2024-09-21T10:13:30.321 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:30.347 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:30.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:30.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:30.449 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:30.477 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:30.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:30.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:30.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:30.614 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:30.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:30.631 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:30.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:30.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:30 smithi196 ceph-mon[84784]: pgmap v634: 89 pgs: 89 active+clean; 2.0 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 130 MiB/s wr, 2.09k op/s 2024-09-21T10:13:30.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:13:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:13:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:13:30.720 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:30 smithi098 ceph-mon[97580]: pgmap v634: 89 pgs: 89 active+clean; 2.0 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 130 MiB/s wr, 2.09k op/s 2024-09-21T10:13:30.721 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:13:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:13:30] "GET /metrics HTTP/1.1" 200 36555 "" "Prometheus/2.51.0" 2024-09-21T10:13:30.721 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:30 smithi098 ceph-mon[100914]: pgmap v634: 89 pgs: 89 active+clean; 2.0 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 130 MiB/s wr, 2.09k op/s 2024-09-21T10:13:30.731 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:30.770 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:30.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:30.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:30.827 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:13:30.872 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:30.914 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:30.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:30.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:31.015 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:31.041 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:31.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:31.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:31.142 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:31.170 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:31.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:31.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:31.242 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 18 16 39594 39578 137.393 109.25 0.00943051 0.00727335 2024-09-21T10:13:31.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:31.309 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:31.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:31.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:31.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:31.458 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:31.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:31.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:31.561 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:31.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:31.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:31.702 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:31.729 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:31.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:31.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:31.831 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:31.857 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:31.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:31.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:31.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:32.005 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:32.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:32.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:32.107 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:32.146 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:32.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:32.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:32.242 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 19 16 41334 41318 135.885 108.75 0.00638544 0.00735366 2024-09-21T10:13:32.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:32.274 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:32.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:32.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:32.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:32.426 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:32.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:32.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:32.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:32.578 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:32.579 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:32 smithi098 ceph-mon[97580]: pgmap v635: 89 pgs: 89 active+clean; 2.2 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-09-21T10:13:32.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:32 smithi098 ceph-mon[100914]: pgmap v635: 89 pgs: 89 active+clean; 2.2 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-09-21T10:13:32.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:32.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:32.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:32.721 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:32.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:32.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:32.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:32.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:32.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:32.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:32 smithi196 ceph-mon[84784]: pgmap v635: 89 pgs: 89 active+clean; 2.2 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-09-21T10:13:32.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:33.018 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:33.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:33.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:33.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:33.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:33.242 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:13:33.241576+0000 min lat: 0.00126097 max lat: 0.108229 avg lat: 0.00745781 2024-09-21T10:13:33.243 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:13:33.243 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 20 16 42885 42869 133.937 96.9375 0.0124734 0.00745781 2024-09-21T10:13:33.260 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:33.285 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:33.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:33.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:33.387 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:33.412 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:33.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:33.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:33.514 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:33.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:33.556 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:33.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:33.656 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:33.692 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:33.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:33.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:33.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:33.835 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:33.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:33.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:33.937 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:33.961 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:33.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:33.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:34.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:34.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:34.103 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:34.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:34.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:34.244 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 21 16 44235 44219 131.566 84.375 0.00633959 0.00759385 2024-09-21T10:13:34.248 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:34.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:34.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:34.350 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:34.399 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:34.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:34.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:34.475 INFO:teuthology.orchestra.run.smithi098.stderr:set pool 7 pg_num to 26 2024-09-21T10:13:34.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:34.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:34.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:34.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:34 smithi098 ceph-mon[100914]: pgmap v636: 89 pgs: 89 active+clean; 2.4 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 117 MiB/s wr, 1.88k op/s 2024-09-21T10:13:34.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:34 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3286045884' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:13:34.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:34 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:13:34.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:34 smithi098 ceph-mon[97580]: pgmap v636: 89 pgs: 89 active+clean; 2.4 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 117 MiB/s wr, 1.88k op/s 2024-09-21T10:13:34.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:34 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3286045884' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:13:34.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:34 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:13:34.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:34.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:34.682 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:34.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:34.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:34.823 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:34.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:34.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:34 smithi196 ceph-mon[84784]: pgmap v636: 89 pgs: 89 active+clean; 2.4 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 117 MiB/s wr, 1.88k op/s 2024-09-21T10:13:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:34 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3286045884' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:13:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:34 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:13:34.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:34.950 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:34.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:34.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:35.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:35.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:35.086 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:35.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:35.187 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:35.231 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:35.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:35.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:35.244 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 22 16 45628 45612 129.543 87.0625 0.00314567 0.00771383 2024-09-21T10:13:35.333 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:35.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:35.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:35.473 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:35.499 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:35.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:35.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:35.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:35.646 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:35.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:35.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:35.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:35.769 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:35.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:35.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:35.773 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:35 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-09-21T10:13:35.773 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:35 smithi098 ceph-mon[97580]: osdmap e208: 8 total, 8 up, 7 in 2024-09-21T10:13:35.773 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:35 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T10:13:35.773 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:35 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-09-21T10:13:35.773 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:35 smithi098 ceph-mon[100914]: osdmap e208: 8 total, 8 up, 7 in 2024-09-21T10:13:35.773 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:35 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T10:13:35.871 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:35 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-09-21T10:13:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:35 smithi196 ceph-mon[84784]: osdmap e208: 8 total, 8 up, 7 in 2024-09-21T10:13:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:35 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T10:13:35.892 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:35.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:35.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:35.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:36.014 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:36.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:36.116 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:36.137 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:36.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:36.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:36.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:36.244 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 23 16 46070 46054 125.112 27.625 0.0362777 0.00774975 2024-09-21T10:13:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:36.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:36.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:36.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:36.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:36.384 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:36.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:36.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:36.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:36.516 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:36.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:36.617 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:36.652 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:36.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:36.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:36.754 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:36.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:36.790 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:36.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:36.799 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:36 smithi098 ceph-mon[97580]: pgmap v638: 89 pgs: 89 active+clean; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-21T10:13:36.799 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:36 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T10:13:36.799 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:36 smithi098 ceph-mon[97580]: osdmap e209: 8 total, 8 up, 7 in 2024-09-21T10:13:36.800 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:36 smithi098 ceph-mon[100914]: pgmap v638: 89 pgs: 89 active+clean; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-21T10:13:36.800 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:36 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T10:13:36.800 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:36 smithi098 ceph-mon[100914]: osdmap e209: 8 total, 8 up, 7 in 2024-09-21T10:13:36.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:36 smithi196 ceph-mon[84784]: pgmap v638: 89 pgs: 89 active+clean; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-21T10:13:36.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:36 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T10:13:36.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:36 smithi196 ceph-mon[84784]: osdmap e209: 8 total, 8 up, 7 in 2024-09-21T10:13:36.891 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:36.929 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:36.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:36.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:36.986 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:13:36.987 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:13:37.025 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:13:37.025 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:13:37.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:37.071 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:13:37.071 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:13:37.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:37.084 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:37.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:37.116 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:13:37.116 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:13:37.155 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:13:37.155 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:13:37.184 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:37.194 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:37.207 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:37.208 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:13:37.208 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:13:37.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:37.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:37.213 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:13:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:13:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:47236 7.e 7:7ba0d425:::benchmark_data_smithi098_197834_object47235:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-21T10:13:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:37.198318+0000", 2024-09-21T10:13:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0091147369999999995, 2024-09-21T10:13:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0091505019999999996, 2024-09-21T10:13:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:13:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:13:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:13:37.213 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:13:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:13:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:13:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 47236 2024-09-21T10:13:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:13:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:13:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:37.198318+0000", 2024-09-21T10:13:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:13:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.214 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:13:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:37.198318+0000", 2024-09-21T10:13:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:13:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:13:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:37.198320+0000", 2024-09-21T10:13:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9630000000000001e-06 2024-09-21T10:13:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:13:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:37.198346+0000", 2024-09-21T10:13:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5632000000000001e-05 2024-09-21T10:13:37.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:13:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:37.198365+0000", 2024-09-21T10:13:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9083999999999999e-05 2024-09-21T10:13:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:13:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:37.198374+0000", 2024-09-21T10:13:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3689999999999996e-06 2024-09-21T10:13:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:13:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:37.198472+0000", 2024-09-21T10:13:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7349000000000006e-05 2024-09-21T10:13:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:13:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:37.198626+0000", 2024-09-21T10:13:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015436700000000001 2024-09-21T10:13:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:13:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:37.198766+0000", 2024-09-21T10:13:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013955400000000001 2024-09-21T10:13:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:37.217 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:37.202652+0000", 2024-09-21T10:13:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003886483 2024-09-21T10:13:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:13:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:37.205769+0000", 2024-09-21T10:13:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031170429999999999 2024-09-21T10:13:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:13:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:13:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:13:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:47238 7.e 7:791e8305:::benchmark_data_smithi098_197834_object47237:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-21T10:13:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:37.202221+0000", 2024-09-21T10:13:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0052125699999999997, 2024-09-21T10:13:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053240199999999996, 2024-09-21T10:13:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:13:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:13:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:13:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:13:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:13:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:13:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 47238 2024-09-21T10:13:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:13:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:13:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:37.202221+0000", 2024-09-21T10:13:37.219 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:13:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:13:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:37.202221+0000", 2024-09-21T10:13:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:13:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:13:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:37.202222+0000", 2024-09-21T10:13:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.658e-06 2024-09-21T10:13:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:13:37.220 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:37.202249+0000", 2024-09-21T10:13:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6789000000000001e-05 2024-09-21T10:13:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:13:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:37.202262+0000", 2024-09-21T10:13:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2799000000000001e-05 2024-09-21T10:13:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:13:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:37.202269+0000", 2024-09-21T10:13:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6799999999999993e-06 2024-09-21T10:13:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.221 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:13:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:37.202745+0000", 2024-09-21T10:13:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00047547900000000001 2024-09-21T10:13:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:13:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:37.202901+0000", 2024-09-21T10:13:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000156019 2024-09-21T10:13:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:13:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:37.203068+0000", 2024-09-21T10:13:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016659699999999999 2024-09-21T10:13:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:37.205757+0000", 2024-09-21T10:13:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026893870000000001 2024-09-21T10:13:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:13:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:13:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:13:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:37.206750+0000", 2024-09-21T10:13:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00099324099999999996 2024-09-21T10:13:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:13:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:13:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:13:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:13:37.223 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-21T10:13:37.224 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 2 2024-09-21T10:13:37.224 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:13:37.245 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 24 16 47272 47256 123.029 75.125 0.0134852 0.00811972 2024-09-21T10:13:37.249 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:13:37.249 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:13:37.286 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:13:37.286 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:13:37.310 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:37.335 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:37.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:37.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:37.437 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:37.465 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:37.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:37.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:37.567 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:37.593 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:37.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:37.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:37 smithi098 ceph-mon[97580]: osdmap e210: 8 total, 8 up, 7 in 2024-09-21T10:13:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:37 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4035158737' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:13:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:37.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:37 smithi098 ceph-mon[100914]: osdmap e210: 8 total, 8 up, 7 in 2024-09-21T10:13:37.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:37 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4035158737' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:13:37.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:37.695 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:37.735 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:37.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:37.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:37.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:37.861 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:37.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:37.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:37.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:37 smithi196 ceph-mon[84784]: osdmap e210: 8 total, 8 up, 7 in 2024-09-21T10:13:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:37 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4035158737' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:13:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:37.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:38.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:38.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:38.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:38.138 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:38.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:38.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:38.240 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:38.245 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 25 16 49207 49191 122.945 120.938 0.00933194 0.00812896 2024-09-21T10:13:38.264 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:38.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:38.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:38.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:38.403 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:38.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:38.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:38.504 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:38.531 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:38.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:38.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:38.632 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:38.674 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:38.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:38.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:38.775 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:38.789 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:38 smithi196 ceph-mon[84784]: pgmap v641: 99 pgs: 2 peering, 10 unknown, 87 active+clean; 2.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-09-21T10:13:38.803 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:38.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:38.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:38.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:38 smithi098 ceph-mon[97580]: pgmap v641: 99 pgs: 2 peering, 10 unknown, 87 active+clean; 2.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-09-21T10:13:38.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:38 smithi098 ceph-mon[100914]: pgmap v641: 99 pgs: 2 peering, 10 unknown, 87 active+clean; 2.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-09-21T10:13:38.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:38.941 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:38.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:38.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:39.042 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:39.070 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:39.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:39.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:39.171 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:39.213 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:39.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:39.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:39.245 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 26 16 50743 50727 121.909 96 0.012767 0.00819808 2024-09-21T10:13:39.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:39.345 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:39.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:39.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:39.447 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:39.479 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:39.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:39.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:39.581 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:39.612 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:39.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:39.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:39.714 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:39.756 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:39.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:39.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:39.775 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:39 smithi098 ceph-mon[97580]: 7.2 scrub starts 2024-09-21T10:13:39.775 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:39 smithi098 ceph-mon[97580]: 7.2 scrub ok 2024-09-21T10:13:39.775 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:39 smithi098 ceph-mon[100914]: 7.2 scrub starts 2024-09-21T10:13:39.775 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:39 smithi098 ceph-mon[100914]: 7.2 scrub ok 2024-09-21T10:13:39.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:39 smithi196 ceph-mon[84784]: 7.2 scrub starts 2024-09-21T10:13:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:39 smithi196 ceph-mon[84784]: 7.2 scrub ok 2024-09-21T10:13:39.895 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:39.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:39.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:39.996 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:40.036 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:40.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:40.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:40.139 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:40.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:40.185 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:40.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:40.245 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 27 16 52301 52285 121 97.375 0.00501063 0.00825747 2024-09-21T10:13:40.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:40.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:40.324 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:40.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:40.426 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:40.454 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:40.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:40.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:40.526 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:13:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:13:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:13:40.556 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:40.583 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:40.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:40.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:40.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:40 smithi098 ceph-mon[97580]: pgmap v642: 99 pgs: 2 peering, 3 unknown, 94 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-09-21T10:13:40.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:40 smithi098 ceph-mon[97580]: 7.7 scrub starts 2024-09-21T10:13:40.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:40 smithi098 ceph-mon[97580]: 7.0 scrub starts 2024-09-21T10:13:40.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:40 smithi098 ceph-mon[97580]: 7.8 scrub starts 2024-09-21T10:13:40.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:40 smithi098 ceph-mon[100914]: pgmap v642: 99 pgs: 2 peering, 3 unknown, 94 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-09-21T10:13:40.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:40 smithi098 ceph-mon[100914]: 7.7 scrub starts 2024-09-21T10:13:40.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:40 smithi098 ceph-mon[100914]: 7.0 scrub starts 2024-09-21T10:13:40.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:40 smithi098 ceph-mon[100914]: 7.8 scrub starts 2024-09-21T10:13:40.684 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:40.713 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:40.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:40.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:40.815 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:40.842 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:40.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:40.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:40.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:13:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:13:40] "GET /metrics HTTP/1.1" 200 36556 "" "Prometheus/2.51.0" 2024-09-21T10:13:40.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:40 smithi196 ceph-mon[84784]: pgmap v642: 99 pgs: 2 peering, 3 unknown, 94 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-09-21T10:13:40.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:40 smithi196 ceph-mon[84784]: 7.7 scrub starts 2024-09-21T10:13:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:40 smithi196 ceph-mon[84784]: 7.0 scrub starts 2024-09-21T10:13:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:40 smithi196 ceph-mon[84784]: 7.8 scrub starts 2024-09-21T10:13:40.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:40.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:40.988 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:40.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:41.088 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:41.113 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:41.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:41.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:41.215 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:41.243 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:41.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:41.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:41.245 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 28 16 54144 54128 120.792 115.188 0.0134832 0.00827302 2024-09-21T10:13:41.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:41.371 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:41.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:41.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:41.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:41.498 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:41.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:41.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:41.600 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:41.626 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:41.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:41.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:41.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:41.754 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:41.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:41.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:41.856 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:41.879 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:41.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:41.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:41.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:41 smithi098 ceph-mon[97580]: 7.3 scrub starts 2024-09-21T10:13:41.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:41 smithi098 ceph-mon[97580]: 7.7 scrub ok 2024-09-21T10:13:41.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:41 smithi098 ceph-mon[97580]: 7.0 scrub ok 2024-09-21T10:13:41.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:41 smithi098 ceph-mon[97580]: 7.8 scrub ok 2024-09-21T10:13:41.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:41 smithi098 ceph-mon[97580]: 7.3 scrub ok 2024-09-21T10:13:41.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:41 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-21T10:13:41.972 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:41 smithi098 ceph-mon[100914]: 7.3 scrub starts 2024-09-21T10:13:41.972 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:41 smithi098 ceph-mon[100914]: 7.7 scrub ok 2024-09-21T10:13:41.972 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:41 smithi098 ceph-mon[100914]: 7.0 scrub ok 2024-09-21T10:13:41.972 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:41 smithi098 ceph-mon[100914]: 7.8 scrub ok 2024-09-21T10:13:41.972 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:41 smithi098 ceph-mon[100914]: 7.3 scrub ok 2024-09-21T10:13:41.972 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:41 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-21T10:13:41.982 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:42.004 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:42.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:42.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:42.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:42.130 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:42.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:42.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:42.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:41 smithi196 ceph-mon[84784]: 7.3 scrub starts 2024-09-21T10:13:42.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:41 smithi196 ceph-mon[84784]: 7.7 scrub ok 2024-09-21T10:13:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:41 smithi196 ceph-mon[84784]: 7.0 scrub ok 2024-09-21T10:13:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:41 smithi196 ceph-mon[84784]: 7.8 scrub ok 2024-09-21T10:13:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:41 smithi196 ceph-mon[84784]: 7.3 scrub ok 2024-09-21T10:13:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:41 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-21T10:13:42.231 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:42.245 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 29 16 55400 55384 119.334 78.5 0.00174816 0.00827215 2024-09-21T10:13:42.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:42.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:42.355 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:42.377 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:42.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:42.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:42.381 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:13:42.394 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:13:42.478 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:42.500 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:42.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:42.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:42.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:42.624 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:42.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:42.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:42.726 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:42.749 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:42.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:42.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:42.851 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:42.854 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:13:42.854 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:13:42.854 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_2 2024-09-21T10:13:42.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:13:42.873 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:42.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:42.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:42.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:43.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:43.000 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:43.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:43.050 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:42 smithi098 ceph-mon[97580]: 7.6 scrub starts 2024-09-21T10:13:43.050 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:42 smithi098 ceph-mon[97580]: pgmap v643: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:13:43.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:42 smithi098 ceph-mon[97580]: 7.6 scrub ok 2024-09-21T10:13:43.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:42 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-21T10:13:43.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:42 smithi098 ceph-mon[97580]: osdmap e211: 8 total, 8 up, 7 in 2024-09-21T10:13:43.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:42 smithi098 ceph-mon[97580]: 7.11 scrub starts 2024-09-21T10:13:43.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:42 smithi098 ceph-mon[97580]: 7.18 scrub starts 2024-09-21T10:13:43.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:42 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:43.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:42 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:43.051 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:42 smithi098 ceph-mon[100914]: 7.6 scrub starts 2024-09-21T10:13:43.051 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:42 smithi098 ceph-mon[100914]: pgmap v643: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:13:43.051 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:42 smithi098 ceph-mon[100914]: 7.6 scrub ok 2024-09-21T10:13:43.051 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:42 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-21T10:13:43.052 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:42 smithi098 ceph-mon[100914]: osdmap e211: 8 total, 8 up, 7 in 2024-09-21T10:13:43.052 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:42 smithi098 ceph-mon[100914]: 7.11 scrub starts 2024-09-21T10:13:43.052 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:42 smithi098 ceph-mon[100914]: 7.18 scrub starts 2024-09-21T10:13:43.052 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:42 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:43.052 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:42 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:43.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:43.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:43.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:42 smithi196 ceph-mon[84784]: 7.6 scrub starts 2024-09-21T10:13:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:42 smithi196 ceph-mon[84784]: pgmap v643: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:13:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:42 smithi196 ceph-mon[84784]: 7.6 scrub ok 2024-09-21T10:13:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:42 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-21T10:13:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:42 smithi196 ceph-mon[84784]: osdmap e211: 8 total, 8 up, 7 in 2024-09-21T10:13:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:42 smithi196 ceph-mon[84784]: 7.11 scrub starts 2024-09-21T10:13:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:42 smithi196 ceph-mon[84784]: 7.18 scrub starts 2024-09-21T10:13:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:42 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:42 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:43.227 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:43.245 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 30 16 55400 55384 115.356 0 - 0.00827215 2024-09-21T10:13:43.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:43.249 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:43.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:43.350 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:43.372 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:43.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:43.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:43.473 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:43.497 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:43.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:43.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:43.520 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:43.524 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:43.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:43.619 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:43.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:43.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:43.721 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:43.742 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:43.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:43.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:43.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:43.869 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:43.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:43.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:43.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[97580]: 7.11 scrub ok 2024-09-21T10:13:43.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[97580]: 7.18 scrub ok 2024-09-21T10:13:43.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[97580]: osdmap e212: 8 total, 8 up, 7 in 2024-09-21T10:13:43.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[97580]: 7.1 scrub starts 2024-09-21T10:13:43.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[97580]: 7.1 scrub ok 2024-09-21T10:13:43.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:13:43.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:13:43.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:43.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-21T10:13:43.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-21T10:13:43.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[97580]: osdmap e213: 8 total, 8 up, 7 in 2024-09-21T10:13:43.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2208018262' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:13:43.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:13:43.934 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[100914]: 7.11 scrub ok 2024-09-21T10:13:43.934 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[100914]: 7.18 scrub ok 2024-09-21T10:13:43.934 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[100914]: osdmap e212: 8 total, 8 up, 7 in 2024-09-21T10:13:43.934 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[100914]: 7.1 scrub starts 2024-09-21T10:13:43.934 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[100914]: 7.1 scrub ok 2024-09-21T10:13:43.934 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:13:43.934 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:13:43.934 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:43.934 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-21T10:13:43.934 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-21T10:13:43.935 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[100914]: osdmap e213: 8 total, 8 up, 7 in 2024-09-21T10:13:43.935 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2208018262' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:13:43.935 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:43 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:13:43.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:43.993 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:43.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:43.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:44.095 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:44.123 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:44.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:44.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:43 smithi196 ceph-mon[84784]: 7.11 scrub ok 2024-09-21T10:13:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:43 smithi196 ceph-mon[84784]: 7.18 scrub ok 2024-09-21T10:13:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:43 smithi196 ceph-mon[84784]: osdmap e212: 8 total, 8 up, 7 in 2024-09-21T10:13:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:43 smithi196 ceph-mon[84784]: 7.1 scrub starts 2024-09-21T10:13:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:43 smithi196 ceph-mon[84784]: 7.1 scrub ok 2024-09-21T10:13:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:43 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:13:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:43 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:13:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:43 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:43 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-21T10:13:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:43 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-21T10:13:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:43 smithi196 ceph-mon[84784]: osdmap e213: 8 total, 8 up, 7 in 2024-09-21T10:13:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:43 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2208018262' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:13:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:43 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:13:44.224 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:44.245 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 31 16 55839 55823 112.521 13.7188 0.00196067 0.0087866 2024-09-21T10:13:44.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:44.246 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:44.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:44.347 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:44.368 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:44.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:44.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:44.470 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:44.492 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:44.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:44.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:44.593 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:44.614 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:44.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:44.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:44.682 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:13:44.716 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:44.737 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:44.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:44.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:44.838 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:44.859 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:44.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:44.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:44.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:44.982 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:44.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:44.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:45.083 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:45.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:44 smithi098 ceph-mon[100914]: 7.12 scrub starts 2024-09-21T10:13:45.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:44 smithi098 ceph-mon[100914]: 7.12 scrub ok 2024-09-21T10:13:45.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:44 smithi098 ceph-mon[100914]: pgmap v646: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 29 MiB/s wr, 471 op/s 2024-09-21T10:13:45.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:44 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:13:45.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:44 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:13:45.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:44 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:13:45.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:44 smithi098 ceph-mon[100914]: osdmap e214: 8 total, 8 up, 7 in 2024-09-21T10:13:45.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:44 smithi098 ceph-mon[97580]: 7.12 scrub starts 2024-09-21T10:13:45.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:44 smithi098 ceph-mon[97580]: 7.12 scrub ok 2024-09-21T10:13:45.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:44 smithi098 ceph-mon[97580]: pgmap v646: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 29 MiB/s wr, 471 op/s 2024-09-21T10:13:45.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:44 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:13:45.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:44 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:13:45.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:44 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:13:45.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:44 smithi098 ceph-mon[97580]: osdmap e214: 8 total, 8 up, 7 in 2024-09-21T10:13:45.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:45.106 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:45.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:45.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:44 smithi196 ceph-mon[84784]: 7.12 scrub starts 2024-09-21T10:13:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:44 smithi196 ceph-mon[84784]: 7.12 scrub ok 2024-09-21T10:13:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:44 smithi196 ceph-mon[84784]: pgmap v646: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 29 MiB/s wr, 471 op/s 2024-09-21T10:13:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:44 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:13:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:44 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:13:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:44 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:13:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:44 smithi196 ceph-mon[84784]: osdmap e214: 8 total, 8 up, 7 in 2024-09-21T10:13:45.208 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:45.229 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:45.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:45.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:45.245 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 32 16 55852 55836 109.03 0.8125 0.00252828 0.00878514 2024-09-21T10:13:45.331 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:45.352 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:45.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:45.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:45.453 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:45.474 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:45.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:45.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:45.576 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:45.597 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:45.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:45.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:45.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:45.721 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:45.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:45.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:45.822 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:45.843 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:45.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:45.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:45.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:45.966 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:45.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:45.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:46.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:46.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:45 smithi098 ceph-mon[97580]: 7.4 scrub starts 2024-09-21T10:13:46.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:45 smithi098 ceph-mon[97580]: 7.4 scrub ok 2024-09-21T10:13:46.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:45 smithi098 ceph-mon[97580]: 7.17 scrub starts 2024-09-21T10:13:46.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:45 smithi098 ceph-mon[97580]: 7.17 scrub ok 2024-09-21T10:13:46.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:45 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-21T10:13:46.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:45 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-21T10:13:46.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:45 smithi098 ceph-mon[97580]: osdmap e215: 8 total, 8 up, 7 in 2024-09-21T10:13:46.089 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:45 smithi098 ceph-mon[100914]: 7.4 scrub starts 2024-09-21T10:13:46.089 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:45 smithi098 ceph-mon[100914]: 7.4 scrub ok 2024-09-21T10:13:46.089 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:45 smithi098 ceph-mon[100914]: 7.17 scrub starts 2024-09-21T10:13:46.089 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:45 smithi098 ceph-mon[100914]: 7.17 scrub ok 2024-09-21T10:13:46.089 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:45 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-21T10:13:46.089 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:45 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-21T10:13:46.089 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:45 smithi098 ceph-mon[100914]: osdmap e215: 8 total, 8 up, 7 in 2024-09-21T10:13:46.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:46.104 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:46.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:45 smithi196 ceph-mon[84784]: 7.4 scrub starts 2024-09-21T10:13:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:45 smithi196 ceph-mon[84784]: 7.4 scrub ok 2024-09-21T10:13:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:45 smithi196 ceph-mon[84784]: 7.17 scrub starts 2024-09-21T10:13:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:45 smithi196 ceph-mon[84784]: 7.17 scrub ok 2024-09-21T10:13:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:45 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-21T10:13:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:45 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-21T10:13:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:45 smithi196 ceph-mon[84784]: osdmap e215: 8 total, 8 up, 7 in 2024-09-21T10:13:46.205 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:46.226 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:46.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:46.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:46.245 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 33 16 56066 56050 106.132 13.375 0.00157599 0.00938692 2024-09-21T10:13:46.328 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:46.354 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:46.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:46.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:46.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:46.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:46.482 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:46.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:46.583 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:46.606 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:46.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:46.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:46.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:46.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:46.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:46.831 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:46.853 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:46.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:46.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:46.954 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:46.977 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:46.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:46.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:47.079 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:47.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:46 smithi098 ceph-mon[97580]: pgmap v649: 99 pgs: 99 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail 2024-09-21T10:13:47.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:46 smithi098 ceph-mon[97580]: osdmap e216: 8 total, 8 up, 7 in 2024-09-21T10:13:47.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:46 smithi098 ceph-mon[100914]: pgmap v649: 99 pgs: 99 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail 2024-09-21T10:13:47.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:46 smithi098 ceph-mon[100914]: osdmap e216: 8 total, 8 up, 7 in 2024-09-21T10:13:47.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:47.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:47.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:47.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:46 smithi196 ceph-mon[84784]: pgmap v649: 99 pgs: 99 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail 2024-09-21T10:13:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:46 smithi196 ceph-mon[84784]: osdmap e216: 8 total, 8 up, 7 in 2024-09-21T10:13:47.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:47.225 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:47.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:47.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:47.245 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 34 16 56332 56316 103.5 16.625 0.00262249 0.00937081 2024-09-21T10:13:47.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:47.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:47.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:47.451 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:47.473 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:47.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:47.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:47.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:47.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:47.597 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:47.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:47.698 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:47.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:47.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:47.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:47.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:47.856 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:47.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:47.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:47.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:47.989 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:47.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:47.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:48.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:48.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:47 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-21T10:13:48.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:47 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:48.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:47 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-21T10:13:48.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:47 smithi098 ceph-mon[97580]: osdmap e217: 8 total, 8 up, 7 in 2024-09-21T10:13:48.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:47 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-21T10:13:48.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:47 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:48.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:47 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-21T10:13:48.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:47 smithi098 ceph-mon[100914]: osdmap e217: 8 total, 8 up, 7 in 2024-09-21T10:13:48.112 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:48.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:48.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:48.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:47 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-21T10:13:48.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:47 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:48.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:47 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-21T10:13:48.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:47 smithi196 ceph-mon[84784]: osdmap e217: 8 total, 8 up, 7 in 2024-09-21T10:13:48.214 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:48.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:48.237 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:48.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:48.246 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 35 16 56580 56564 100.986 15.5 0.00469504 0.00980592 2024-09-21T10:13:48.338 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:48.361 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:48.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:48.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:48.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:48.485 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:48.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:48.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:48.586 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:48.608 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:48.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:48.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:48.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:48.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:48.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:48.834 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:48.856 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:48.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:48.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:48.957 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:48.979 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:48.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:48.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:49.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:49.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:48 smithi098 ceph-mon[97580]: pgmap v652: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 2664/166977 objects misplaced (1.595%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:13:49.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:48 smithi098 ceph-mon[97580]: osdmap e218: 8 total, 8 up, 7 in 2024-09-21T10:13:49.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:48 smithi098 ceph-mon[100914]: pgmap v652: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 2664/166977 objects misplaced (1.595%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:13:49.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:48 smithi098 ceph-mon[100914]: osdmap e218: 8 total, 8 up, 7 in 2024-09-21T10:13:49.105 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:49.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:49.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:49.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:48 smithi196 ceph-mon[84784]: pgmap v652: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 2664/166977 objects misplaced (1.595%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:13:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:48 smithi196 ceph-mon[84784]: osdmap e218: 8 total, 8 up, 7 in 2024-09-21T10:13:49.206 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:49.228 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:49.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:49.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:49.246 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 36 16 56580 56564 98.1809 0 - 0.00980592 2024-09-21T10:13:49.330 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:49.352 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:49.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:49.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:49.454 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:49.476 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:49.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:49.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:49.577 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:49.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:49.601 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:49.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:49.702 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:49.725 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:49.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:49.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:49.827 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:49.849 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:49.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:49.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:49.951 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:49.982 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:49.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:49.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:50.083 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:50.115 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:50.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:50.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:50.216 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:50.246 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 37 16 57076 57060 96.3654 15.5 0.0136179 0.0103721 2024-09-21T10:13:50.246 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:50.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:50.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:50.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:50.390 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:50.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:50.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:50.492 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:50.533 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:50.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:50.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:50.540 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:13:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:13:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:13:50.635 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:50.664 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:50.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:50.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:50.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:50.794 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:50.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:50.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:50 smithi196 ceph-mon[84784]: pgmap v655: 99 pgs: 1 unknown, 2 active+recovering+undersized+remapped, 96 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 5568/163962 objects misplaced (3.396%); 10 MiB/s, 165 objects/s recovering 2024-09-21T10:13:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:50 smithi196 ceph-mon[84784]: osdmap e219: 8 total, 8 up, 7 in 2024-09-21T10:13:50.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:50.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:50.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:51.036 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:51.038 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:50 smithi098 ceph-mon[97580]: pgmap v655: 99 pgs: 1 unknown, 2 active+recovering+undersized+remapped, 96 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 5568/163962 objects misplaced (3.396%); 10 MiB/s, 165 objects/s recovering 2024-09-21T10:13:51.038 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:50 smithi098 ceph-mon[97580]: osdmap e219: 8 total, 8 up, 7 in 2024-09-21T10:13:51.038 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:13:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:13:50] "GET /metrics HTTP/1.1" 200 36556 "" "Prometheus/2.51.0" 2024-09-21T10:13:51.038 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:50 smithi098 ceph-mon[100914]: pgmap v655: 99 pgs: 1 unknown, 2 active+recovering+undersized+remapped, 96 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 5568/163962 objects misplaced (3.396%); 10 MiB/s, 165 objects/s recovering 2024-09-21T10:13:51.038 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:50 smithi098 ceph-mon[100914]: osdmap e219: 8 total, 8 up, 7 in 2024-09-21T10:13:51.065 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:51.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:51.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:51.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:51.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:51.217 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:51.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:51.246 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 38 16 58656 58640 96.4279 98.75 0.00995002 0.0103652 2024-09-21T10:13:51.317 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:51.355 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:51.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:51.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:51.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:51.484 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:51.509 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:51.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:51.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:51.610 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:51.639 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:51.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:51.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:51.741 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:51.769 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:51.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:51.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:51.871 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:51.902 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:51.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:51.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:52.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:52.047 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:52.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:52.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:52.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:51 smithi098 ceph-mon[100914]: Health check failed: Degraded data redundancy: 5151/170943 objects degraded (3.013%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:13:52.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:51 smithi098 ceph-mon[97580]: Health check failed: Degraded data redundancy: 5151/170943 objects degraded (3.013%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:13:52.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:51 smithi196 ceph-mon[84784]: Health check failed: Degraded data redundancy: 5151/170943 objects degraded (3.013%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:13:52.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:52.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:52.190 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:52.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:52.246 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 39 16 60138 60122 96.3302 92.625 0.011731 0.0103762 2024-09-21T10:13:52.291 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:52.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:13:52.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:52.329 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:52.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:52.418 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:13:52.418 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:13:52.430 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:52.456 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:13:52.456 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:13:52.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:52.457 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:52.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:52.486 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:13:52.487 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:13:52.526 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:13:52.526 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:13:52.555 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:13:52.555 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:13:52.558 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:52.584 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:52.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:52.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:52.598 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:13:52.598 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:13:52.629 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:13:52.630 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:13:52.664 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:13:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:13:52.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:52.739 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:52.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:52.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:52.841 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:52.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:52.873 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:52.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:52.974 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:53.001 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:53.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:53.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:52 smithi098 ceph-mon[97580]: pgmap v657: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 5151/170943 objects degraded (3.013%); 5745/170943 objects misplaced (3.361%); 15 MiB/s, 240 objects/s recovering 2024-09-21T10:13:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:53.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:53.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:52 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4235070062' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:13:53.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:52 smithi098 ceph-mon[100914]: pgmap v657: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 5151/170943 objects degraded (3.013%); 5745/170943 objects misplaced (3.361%); 15 MiB/s, 240 objects/s recovering 2024-09-21T10:13:53.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:53.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:53.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:52 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4235070062' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:13:53.103 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:53.132 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:53.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:53.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:53.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:52 smithi196 ceph-mon[84784]: pgmap v657: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 5151/170943 objects degraded (3.013%); 5745/170943 objects misplaced (3.361%); 15 MiB/s, 240 objects/s recovering 2024-09-21T10:13:53.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:13:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:52 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4235070062' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:13:53.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:53.247 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:13:53.245210+0000 min lat: 0.00126097 max lat: 2.24757 avg lat: 0.0103963 2024-09-21T10:13:53.247 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:13:53.247 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 40 16 61558 61542 96.1405 88.75 0.0160036 0.0103963 2024-09-21T10:13:53.272 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:53.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:53.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:53.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:53.414 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:53.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:53.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:53.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:53.550 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:53.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:53.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:53.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:53.688 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:53.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:53.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:53.790 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:53.820 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:53.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:53.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:53.921 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:53.951 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:53.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:53.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:54.053 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:54.088 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:54.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:54.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:54.190 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:54.218 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:54.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:54.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:54.246 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 41 16 63081 63065 96.1171 95.1875 0.00289998 0.0103979 2024-09-21T10:13:54.320 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:54.343 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:54.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:54.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:54.445 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:54.475 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:54.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:54.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:54.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:54.617 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:54.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:54.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:54.719 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:54.751 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:54.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:54.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:54.853 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:54.882 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:54.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:54.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:54.984 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:55.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:55.022 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:55.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:55.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:54 smithi098 ceph-mon[97580]: pgmap v658: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 5151/176013 objects degraded (2.926%); 5745/176013 objects misplaced (3.264%); 11 MiB/s, 182 objects/s recovering 2024-09-21T10:13:55.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:54 smithi098 ceph-mon[100914]: pgmap v658: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 5151/176013 objects degraded (2.926%); 5745/176013 objects misplaced (3.264%); 11 MiB/s, 182 objects/s recovering 2024-09-21T10:13:55.123 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:55.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:54 smithi196 ceph-mon[84784]: pgmap v658: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 5151/176013 objects degraded (2.926%); 5745/176013 objects misplaced (3.264%); 11 MiB/s, 182 objects/s recovering 2024-09-21T10:13:55.153 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:55.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:55.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:55.246 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 42 16 64434 64418 95.8418 84.5625 0.092424 0.0104279 2024-09-21T10:13:55.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:55.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:55.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:55.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:55.437 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:55.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:55.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:55.539 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:55.574 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:55.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:55.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:55.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:55.718 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:55.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:55.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:55.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:55.848 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:55.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:55.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:55.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:55.979 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:55.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:55.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:56.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:56.111 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:56.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:56.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:56.213 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:56.247 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 43 16 65915 65899 95.7654 92.5625 0.0111944 0.010437 2024-09-21T10:13:56.250 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:56.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:56.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:56.352 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:56.372 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:56.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:56.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:56.473 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:56.503 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:56.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:56.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:56.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:56.641 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:56.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:56.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:56.743 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:56.779 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:56.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:56.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:56.880 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:56.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:56.913 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:56.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:57.014 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:57.051 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:57.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:57.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:57.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:56 smithi098 ceph-mon[97580]: pgmap v659: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 5151/183792 objects degraded (2.803%); 4465/183792 objects misplaced (2.429%); 14 MiB/s, 229 objects/s recovering 2024-09-21T10:13:57.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:56 smithi098 ceph-mon[100914]: pgmap v659: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 5151/183792 objects degraded (2.803%); 4465/183792 objects misplaced (2.429%); 14 MiB/s, 229 objects/s recovering 2024-09-21T10:13:57.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:56 smithi196 ceph-mon[84784]: pgmap v659: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 5151/183792 objects degraded (2.803%); 4465/183792 objects misplaced (2.429%); 14 MiB/s, 229 objects/s recovering 2024-09-21T10:13:57.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:57.192 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:57.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:57.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:57.246 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 44 16 67314 67298 95.5759 87.4375 0.00407843 0.0104494 2024-09-21T10:13:57.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:57.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:57.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:57.441 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:57.464 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:57.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:57.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:57.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:57.602 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:57.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:57.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:57.705 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:57.733 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:57.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:57.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:57.835 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:57.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:57.867 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:57.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:57.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:58.004 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:58.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:58.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:58.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:57 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 5151/187830 objects degraded (2.742%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:13:58.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:57 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 5151/187830 objects degraded (2.742%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:13:58.106 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:58.134 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:58.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:58.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:58.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:57 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 5151/187830 objects degraded (2.742%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:13:58.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:58.246 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 45 16 68741 68725 95.4338 89.1875 0.0227495 0.010473 2024-09-21T10:13:58.265 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:58.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:58.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:58.367 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:58.394 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:58.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:58.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:58.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:58.545 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:58.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:58.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:58.647 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:58.676 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:58.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:58.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:58.777 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:58.816 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:58.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:58.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:58.918 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:58.944 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:58.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:58.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:59.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:59.083 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:59.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:59.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:59.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:58 smithi098 ceph-mon[97580]: pgmap v660: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 46 MiB/s wr, 729 op/s; 5151/187830 objects degraded (2.742%); 3697/187830 objects misplaced (1.968%); 13 MiB/s, 209 objects/s recovering 2024-09-21T10:13:59.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:58 smithi098 ceph-mon[100914]: pgmap v660: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 46 MiB/s wr, 729 op/s; 5151/187830 objects degraded (2.742%); 3697/187830 objects misplaced (1.968%); 13 MiB/s, 209 objects/s recovering 2024-09-21T10:13:59.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:58 smithi196 ceph-mon[84784]: pgmap v660: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 46 MiB/s wr, 729 op/s; 5151/187830 objects degraded (2.742%); 3697/187830 objects misplaced (1.968%); 13 MiB/s, 209 objects/s recovering 2024-09-21T10:13:59.186 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:59.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:59.216 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:59.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:59.247 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 46 16 70010 69994 95.0832 79.3125 0.00382483 0.0104654 2024-09-21T10:13:59.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:13:59.330 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:13:59.341 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:59.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:59.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:59.443 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:59.465 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:59.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:59.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:59.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:59.591 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:59.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:59.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:59.692 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:59.713 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:59.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:59.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:59.815 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:59.820 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:13:59.820 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:13:59.837 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:13:59.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:59.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:59.939 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:13:59.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:59.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:00.079 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:00.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:13:59 smithi098 ceph-mon[97580]: osdmap e220: 8 total, 8 up, 7 in 2024-09-21T10:14:00.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:13:59 smithi098 ceph-mon[100914]: osdmap e220: 8 total, 8 up, 7 in 2024-09-21T10:14:00.106 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:00.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:00.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:00.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:13:59 smithi196 ceph-mon[84784]: osdmap e220: 8 total, 8 up, 7 in 2024-09-21T10:14:00.207 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:00.231 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:00.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:00.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:00.247 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 47 16 70543 70527 93.769 33.3125 0.00335623 0.010624 2024-09-21T10:14:00.333 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:00.354 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:00.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:00.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:00.456 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:00.477 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:00.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:00.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:00.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:00.595 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:14:00.601 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:00.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:00.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:00.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:14:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:14:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:14:00.703 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:00.744 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:00.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:00.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:00.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:14:00.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:14:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:14:00] "GET /metrics HTTP/1.1" 200 36586 "" "Prometheus/2.51.0" 2024-09-21T10:14:00.846 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:00.901 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:00.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:00.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:01.003 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:01.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:01.045 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:01.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:01.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:00 smithi098 ceph-mon[100914]: pgmap v662: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s; 5151/201945 objects degraded (2.551%); 2555/201945 objects misplaced (1.265%); 15 MiB/s, 247 objects/s recovering 2024-09-21T10:14:01.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:00 smithi098 ceph-mon[100914]: from='client.55033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:14:01.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:00 smithi098 ceph-mon[100914]: osdmap e221: 8 total, 8 up, 7 in 2024-09-21T10:14:01.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:00 smithi098 ceph-mon[97580]: pgmap v662: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s; 5151/201945 objects degraded (2.551%); 2555/201945 objects misplaced (1.265%); 15 MiB/s, 247 objects/s recovering 2024-09-21T10:14:01.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:00 smithi098 ceph-mon[97580]: from='client.55033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:14:01.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:00 smithi098 ceph-mon[97580]: osdmap e221: 8 total, 8 up, 7 in 2024-09-21T10:14:01.104 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:01.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:00 smithi196 ceph-mon[84784]: pgmap v662: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s; 5151/201945 objects degraded (2.551%); 2555/201945 objects misplaced (1.265%); 15 MiB/s, 247 objects/s recovering 2024-09-21T10:14:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:00 smithi196 ceph-mon[84784]: from='client.55033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:14:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:00 smithi196 ceph-mon[84784]: osdmap e221: 8 total, 8 up, 7 in 2024-09-21T10:14:01.146 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:01.152 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":662,"stamp":"2024-09-21T10:13:59.242136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4414511053,"num_objects":67315,"num_object_clones":0,"num_object_copies":201945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5151,"num_objects_misplaced":2555,"num_objects_unfound":0,"num_objects_dirty":67315,"num_whiteouts":0,"num_read":6288,"num_read_kb":13764,"num_write":48091,"num_write_kb":3024491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3085,"num_bytes_recovered":198912230,"num_keys_recovered":30,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97218,"ondisk_log_size":97218,"up":297,"acting":293,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":305,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":15689152,"kb_used_data":13298440,"kb_used_omap":33,"kb_used_meta":2390622,"kb_avail":640526912,"statfs":{"total":671965249536,"available":655899557888,"internally_reserved":0,"allocated":13617602560,"data_stored":13610156462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34649,"internal_metadata":2447997095},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":829751296,"num_objects":12661,"num_object_clones":0,"num_object_copies":37983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5151,"num_objects_misplaced":-3013,"num_objects_unfound":0,"num_objects_dirty":12661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10957,"num_write_kb":701248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2364,"num_bytes_recovered":154927104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.564644"},"pg_stats":[{"pgid":"7.17","version":"219'3646","reported_seq":3761,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:58.267372+0000","last_change":"2024-09-21T10:13:44.684354+0000","last_active":"2024-09-21T10:13:58.267372+0000","last_peered":"2024-09-21T10:13:58.267372+0000","last_clean":"2024-09-21T10:13:58.267372+0000","last_became_active":"2024-09-21T10:13:36.527577+0000","last_became_peered":"2024-09-21T10:13:36.527577+0000","last_unstale":"2024-09-21T10:13:58.267372+0000","last_undegraded":"2024-09-21T10:13:58.267372+0000","last_fullsized":"2024-09-21T10:13:58.267372+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":5,"last_scrub":"213'3271","last_scrub_stamp":"2024-09-21T10:13:44.684263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:44.684263+0000","objects_scrubbed":1783,"log_size":3646,"log_dups_size":0,"ondisk_log_size":3646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:11:57.613857+0000","scrub_duration":0.165485878,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141426688,"num_objects":2158,"num_object_clones":0,"num_object_copies":6474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":375,"num_write_kb":24000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":308,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.690864+0000","last_change":"2024-09-21T10:02:14.097991+0000","last_active":"2024-09-21T10:13:49.690864+0000","last_peered":"2024-09-21T10:13:49.690864+0000","last_clean":"2024-09-21T10:13:49.690864+0000","last_became_active":"2024-09-21T10:02:14.097407+0000","last_became_peered":"2024-09-21T10:02:14.097407+0000","last_unstale":"2024-09-21T10:13:49.690864+0000","last_undegraded":"2024-09-21T10:13:49.690864+0000","last_fullsized":"2024-09-21T10:13:49.690864+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:18:42.098886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":234,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.677744+0000","last_change":"2024-09-21T10:08:00.406360+0000","last_active":"2024-09-21T10:13:48.677744+0000","last_peered":"2024-09-21T10:13:48.677744+0000","last_clean":"2024-09-21T10:13:48.677744+0000","last_became_active":"2024-09-21T10:08:00.380042+0000","last_became_peered":"2024-09-21T10:08:00.380042+0000","last_unstale":"2024-09-21T10:13:48.677744+0000","last_undegraded":"2024-09-21T10:13:48.677744+0000","last_fullsized":"2024-09-21T10:13:48.677744+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:49:46.544051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.16","version":"219'3560","reported_seq":3618,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:57.702890+0000","last_change":"2024-09-21T10:13:36.537825+0000","last_active":"2024-09-21T10:13:57.702890+0000","last_peered":"2024-09-21T10:13:57.702890+0000","last_clean":"2024-09-21T10:13:57.702890+0000","last_became_active":"2024-09-21T10:13:36.536028+0000","last_became_peered":"2024-09-21T10:13:36.536028+0000","last_unstale":"2024-09-21T10:13:57.702890+0000","last_undegraded":"2024-09-21T10:13:57.702890+0000","last_fullsized":"2024-09-21T10:13:57.702890+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":3560,"log_dups_size":0,"ondisk_log_size":3560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138870784,"num_objects":2119,"num_object_clones":0,"num_object_copies":6357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2119,"num_write_kb":135616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":140,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.678025+0000","last_change":"2024-09-21T10:07:43.789751+0000","last_active":"2024-09-21T10:13:48.678025+0000","last_peered":"2024-09-21T10:13:48.678025+0000","last_clean":"2024-09-21T10:13:48.678025+0000","last_became_active":"2024-09-21T10:07:43.789448+0000","last_became_peered":"2024-09-21T10:07:43.789448+0000","last_unstale":"2024-09-21T10:13:48.678025+0000","last_undegraded":"2024-09-21T10:13:48.678025+0000","last_fullsized":"2024-09-21T10:13:48.678025+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:36.586942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":304,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.896218+0000","last_change":"2024-09-21T10:02:14.098949+0000","last_active":"2024-09-21T10:13:49.896218+0000","last_peered":"2024-09-21T10:13:49.896218+0000","last_clean":"2024-09-21T10:13:49.896218+0000","last_became_active":"2024-09-21T10:02:14.098646+0000","last_became_peered":"2024-09-21T10:02:14.098646+0000","last_unstale":"2024-09-21T10:13:49.896218+0000","last_undegraded":"2024-09-21T10:13:49.896218+0000","last_fullsized":"2024-09-21T10:13:49.896218+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:50:56.862529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"219'3618","reported_seq":3653,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:58.247689+0000","last_change":"2024-09-21T10:13:36.538796+0000","last_active":"2024-09-21T10:13:58.247689+0000","last_peered":"2024-09-21T10:13:58.247689+0000","last_clean":"2024-09-21T10:13:58.247689+0000","last_became_active":"2024-09-21T10:13:36.537425+0000","last_became_peered":"2024-09-21T10:13:36.537425+0000","last_unstale":"2024-09-21T10:13:58.247689+0000","last_undegraded":"2024-09-21T10:13:58.247689+0000","last_fullsized":"2024-09-21T10:13:58.247689+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":3618,"log_dups_size":0,"ondisk_log_size":3618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141131776,"num_objects":2154,"num_object_clones":0,"num_object_copies":6462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2154,"num_write_kb":137824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":303,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.896281+0000","last_change":"2024-09-21T10:02:14.098205+0000","last_active":"2024-09-21T10:13:49.896281+0000","last_peered":"2024-09-21T10:13:49.896281+0000","last_clean":"2024-09-21T10:13:49.896281+0000","last_became_active":"2024-09-21T10:02:14.097911+0000","last_became_peered":"2024-09-21T10:02:14.097911+0000","last_unstale":"2024-09-21T10:13:49.896281+0000","last_undegraded":"2024-09-21T10:13:49.896281+0000","last_fullsized":"2024-09-21T10:13:49.896281+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:55.971191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":99,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.684696+0000","last_change":"2024-09-21T10:08:00.385671+0000","last_active":"2024-09-21T10:13:48.684696+0000","last_peered":"2024-09-21T10:13:48.684696+0000","last_clean":"2024-09-21T10:13:48.684696+0000","last_became_active":"2024-09-21T10:08:00.385368+0000","last_became_peered":"2024-09-21T10:08:00.385368+0000","last_unstale":"2024-09-21T10:13:48.684696+0000","last_undegraded":"2024-09-21T10:13:48.684696+0000","last_fullsized":"2024-09-21T10:13:48.684696+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:21.799149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.14","version":"219'3553","reported_seq":3611,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:57.723530+0000","last_change":"2024-09-21T10:13:49.896198+0000","last_active":"2024-09-21T10:13:57.723530+0000","last_peered":"2024-09-21T10:13:57.723530+0000","last_clean":"2024-09-21T10:13:57.723530+0000","last_became_active":"2024-09-21T10:13:36.532292+0000","last_became_peered":"2024-09-21T10:13:36.532292+0000","last_unstale":"2024-09-21T10:13:57.723530+0000","last_undegraded":"2024-09-21T10:13:57.723530+0000","last_fullsized":"2024-09-21T10:13:57.723530+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":3553,"log_dups_size":0,"ondisk_log_size":3553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T00:06:34.492508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139001856,"num_objects":2121,"num_object_clones":0,"num_object_copies":6363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2121,"num_write_kb":135744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":307,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.691190+0000","last_change":"2024-09-21T10:02:14.097817+0000","last_active":"2024-09-21T10:13:49.691190+0000","last_peered":"2024-09-21T10:13:49.691190+0000","last_clean":"2024-09-21T10:13:49.691190+0000","last_became_active":"2024-09-21T10:02:14.097466+0000","last_became_peered":"2024-09-21T10:02:14.097466+0000","last_unstale":"2024-09-21T10:13:49.691190+0000","last_undegraded":"2024-09-21T10:13:49.691190+0000","last_fullsized":"2024-09-21T10:13:49.691190+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:57.883118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":191,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.684157+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:13:48.684157+0000","last_peered":"2024-09-21T10:13:48.684157+0000","last_clean":"2024-09-21T10:13:48.684157+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:13:48.684157+0000","last_undegraded":"2024-09-21T10:13:48.684157+0000","last_fullsized":"2024-09-21T10:13:48.684157+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"219'4142","reported_seq":4193,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:56.151364+0000","last_change":"2024-09-21T10:13:00.065993+0000","last_active":"2024-09-21T10:13:56.151364+0000","last_peered":"2024-09-21T10:13:56.151364+0000","last_clean":"2024-09-21T10:13:56.151364+0000","last_became_active":"2024-09-21T10:13:00.065574+0000","last_became_peered":"2024-09-21T10:13:00.065574+0000","last_unstale":"2024-09-21T10:13:56.151364+0000","last_undegraded":"2024-09-21T10:13:56.151364+0000","last_fullsized":"2024-09-21T10:13:56.151364+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":4142,"log_dups_size":0,"ondisk_log_size":4142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:54.036188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271450112,"num_objects":4142,"num_object_clones":0,"num_object_copies":12426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4142,"num_write_kb":265088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":227,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.678501+0000","last_change":"2024-09-21T10:08:00.392499+0000","last_active":"2024-09-21T10:13:48.678501+0000","last_peered":"2024-09-21T10:13:48.678501+0000","last_clean":"2024-09-21T10:13:48.678501+0000","last_became_active":"2024-09-21T10:08:00.379165+0000","last_became_peered":"2024-09-21T10:08:00.379165+0000","last_unstale":"2024-09-21T10:13:48.678501+0000","last_undegraded":"2024-09-21T10:13:48.678501+0000","last_fullsized":"2024-09-21T10:13:48.678501+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:18.933144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":83,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.677559+0000","last_change":"2024-09-21T10:07:43.794677+0000","last_active":"2024-09-21T10:13:48.677559+0000","last_peered":"2024-09-21T10:13:48.677559+0000","last_clean":"2024-09-21T10:13:48.677559+0000","last_became_active":"2024-09-21T10:07:43.794348+0000","last_became_peered":"2024-09-21T10:07:43.794348+0000","last_unstale":"2024-09-21T10:13:48.677559+0000","last_undegraded":"2024-09-21T10:13:48.677559+0000","last_fullsized":"2024-09-21T10:13:48.677559+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:11.159014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"219'4292","reported_seq":4343,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:57.841131+0000","last_change":"2024-09-21T10:13:00.064683+0000","last_active":"2024-09-21T10:13:57.841131+0000","last_peered":"2024-09-21T10:13:57.841131+0000","last_clean":"2024-09-21T10:13:57.841131+0000","last_became_active":"2024-09-21T10:13:00.064382+0000","last_became_peered":"2024-09-21T10:13:00.064382+0000","last_unstale":"2024-09-21T10:13:57.841131+0000","last_undegraded":"2024-09-21T10:13:57.841131+0000","last_fullsized":"2024-09-21T10:13:57.841131+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":4292,"log_dups_size":0,"ondisk_log_size":4292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:11:53.539027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281280512,"num_objects":4292,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4292,"num_write_kb":274688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":228,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.678435+0000","last_change":"2024-09-21T10:08:00.412449+0000","last_active":"2024-09-21T10:13:48.678435+0000","last_peered":"2024-09-21T10:13:48.678435+0000","last_clean":"2024-09-21T10:13:48.678435+0000","last_became_active":"2024-09-21T10:08:00.386530+0000","last_became_peered":"2024-09-21T10:08:00.386530+0000","last_unstale":"2024-09-21T10:13:48.678435+0000","last_undegraded":"2024-09-21T10:13:48.678435+0000","last_fullsized":"2024-09-21T10:13:48.678435+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:12:50.581176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":229,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.682340+0000","last_change":"2024-09-21T10:08:18.475007+0000","last_active":"2024-09-21T10:13:49.682340+0000","last_peered":"2024-09-21T10:13:49.682340+0000","last_clean":"2024-09-21T10:13:49.682340+0000","last_became_active":"2024-09-21T10:08:18.474730+0000","last_became_peered":"2024-09-21T10:08:18.474730+0000","last_unstale":"2024-09-21T10:13:49.682340+0000","last_undegraded":"2024-09-21T10:13:49.682340+0000","last_fullsized":"2024-09-21T10:13:49.682340+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:35.455790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.9","version":"219'3546","reported_seq":3605,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:57.722024+0000","last_change":"2024-09-21T10:13:49.895960+0000","last_active":"2024-09-21T10:13:57.722024+0000","last_peered":"2024-09-21T10:13:57.722024+0000","last_clean":"2024-09-21T10:13:57.722024+0000","last_became_active":"2024-09-21T10:13:36.526784+0000","last_became_peered":"2024-09-21T10:13:36.526784+0000","last_unstale":"2024-09-21T10:13:57.722024+0000","last_undegraded":"2024-09-21T10:13:57.722024+0000","last_fullsized":"2024-09-21T10:13:57.722024+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":3546,"log_dups_size":0,"ondisk_log_size":3546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-29T09:25:46.218423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139132928,"num_objects":2123,"num_object_clones":0,"num_object_copies":6369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2123,"num_write_kb":135872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":168,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.678148+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:13:48.678148+0000","last_peered":"2024-09-21T10:13:48.678148+0000","last_clean":"2024-09-21T10:13:48.678148+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:13:48.678148+0000","last_undegraded":"2024-09-21T10:13:48.678148+0000","last_fullsized":"2024-09-21T10:13:48.678148+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":182,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.684209+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:13:48.684209+0000","last_peered":"2024-09-21T10:13:48.684209+0000","last_clean":"2024-09-21T10:13:48.684209+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:13:48.684209+0000","last_undegraded":"2024-09-21T10:13:48.684209+0000","last_fullsized":"2024-09-21T10:13:48.684209+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"219'3442","reported_seq":3525,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:55.034480+0000","last_change":"2024-09-21T10:13:40.147612+0000","last_active":"2024-09-21T10:13:55.034480+0000","last_peered":"2024-09-21T10:13:55.034480+0000","last_clean":"2024-09-21T10:13:55.034480+0000","last_became_active":"2024-09-21T10:13:36.522799+0000","last_became_peered":"2024-09-21T10:13:36.522799+0000","last_unstale":"2024-09-21T10:13:55.034480+0000","last_undegraded":"2024-09-21T10:13:55.034480+0000","last_fullsized":"2024-09-21T10:13:55.034480+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'3074","last_scrub_stamp":"2024-09-21T10:13:40.147275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:40.147275+0000","objects_scrubbed":1635,"log_size":3442,"log_dups_size":0,"ondisk_log_size":3442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:33:43.657300+0000","scrub_duration":458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132120576,"num_objects":2016,"num_object_clones":0,"num_object_copies":6048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":381,"num_write_kb":24384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":213,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.690916+0000","last_change":"2024-09-21T10:07:43.793008+0000","last_active":"2024-09-21T10:13:49.690916+0000","last_peered":"2024-09-21T10:13:49.690916+0000","last_clean":"2024-09-21T10:13:49.690916+0000","last_became_active":"2024-09-21T10:07:43.792683+0000","last_became_peered":"2024-09-21T10:07:43.792683+0000","last_unstale":"2024-09-21T10:13:49.690916+0000","last_undegraded":"2024-09-21T10:13:49.690916+0000","last_fullsized":"2024-09-21T10:13:49.690916+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:46:13.137478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":325,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.931925+0000","last_change":"2024-09-21T10:08:00.389645+0000","last_active":"2024-09-21T10:13:49.931925+0000","last_peered":"2024-09-21T10:13:49.931925+0000","last_clean":"2024-09-21T10:13:49.931925+0000","last_became_active":"2024-09-21T10:08:00.386076+0000","last_became_peered":"2024-09-21T10:08:00.386076+0000","last_unstale":"2024-09-21T10:13:49.931925+0000","last_undegraded":"2024-09-21T10:13:49.931925+0000","last_fullsized":"2024-09-21T10:13:49.931925+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:58:10.646600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"219'4270","reported_seq":4321,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:57.733771+0000","last_change":"2024-09-21T10:13:00.064670+0000","last_active":"2024-09-21T10:13:57.733771+0000","last_peered":"2024-09-21T10:13:57.733771+0000","last_clean":"2024-09-21T10:13:57.733771+0000","last_became_active":"2024-09-21T10:13:00.063176+0000","last_became_peered":"2024-09-21T10:13:00.063176+0000","last_unstale":"2024-09-21T10:13:57.733771+0000","last_undegraded":"2024-09-21T10:13:57.733771+0000","last_fullsized":"2024-09-21T10:13:57.733771+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:16:34.728475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279838720,"num_objects":4270,"num_object_clones":0,"num_object_copies":12810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4270,"num_write_kb":273280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":339,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.690777+0000","last_change":"2024-09-21T10:08:00.395898+0000","last_active":"2024-09-21T10:13:49.690777+0000","last_peered":"2024-09-21T10:13:49.690777+0000","last_clean":"2024-09-21T10:13:49.690777+0000","last_became_active":"2024-09-21T10:08:00.390462+0000","last_became_peered":"2024-09-21T10:08:00.390462+0000","last_unstale":"2024-09-21T10:13:49.690777+0000","last_undegraded":"2024-09-21T10:13:49.690777+0000","last_fullsized":"2024-09-21T10:13:49.690777+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:29:01.877089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":226,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.690957+0000","last_change":"2024-09-21T10:08:00.383016+0000","last_active":"2024-09-21T10:13:49.690957+0000","last_peered":"2024-09-21T10:13:49.690957+0000","last_clean":"2024-09-21T10:13:49.690957+0000","last_became_active":"2024-09-21T10:08:00.382708+0000","last_became_peered":"2024-09-21T10:08:00.382708+0000","last_unstale":"2024-09-21T10:13:49.690957+0000","last_undegraded":"2024-09-21T10:13:49.690957+0000","last_fullsized":"2024-09-21T10:13:49.690957+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:30:52.797259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"219'3980","reported_seq":4031,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:55.041662+0000","last_change":"2024-09-21T10:13:00.067640+0000","last_active":"2024-09-21T10:13:55.041662+0000","last_peered":"2024-09-21T10:13:55.041662+0000","last_clean":"2024-09-21T10:13:55.041662+0000","last_became_active":"2024-09-21T10:13:00.067380+0000","last_became_peered":"2024-09-21T10:13:00.067380+0000","last_unstale":"2024-09-21T10:13:55.041662+0000","last_undegraded":"2024-09-21T10:13:55.041662+0000","last_fullsized":"2024-09-21T10:13:55.041662+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":3980,"log_dups_size":0,"ondisk_log_size":3980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:14.874175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260833280,"num_objects":3980,"num_object_clones":0,"num_object_copies":11940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3980,"num_write_kb":254720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":213,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.684342+0000","last_change":"2024-09-21T10:08:00.379604+0000","last_active":"2024-09-21T10:13:48.684342+0000","last_peered":"2024-09-21T10:13:48.684342+0000","last_clean":"2024-09-21T10:13:48.684342+0000","last_became_active":"2024-09-21T10:08:00.379301+0000","last_became_peered":"2024-09-21T10:08:00.379301+0000","last_unstale":"2024-09-21T10:13:48.684342+0000","last_undegraded":"2024-09-21T10:13:48.684342+0000","last_fullsized":"2024-09-21T10:13:48.684342+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:32:41.963929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":237,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.679581+0000","last_change":"2024-09-21T10:08:00.387420+0000","last_active":"2024-09-21T10:13:49.679581+0000","last_peered":"2024-09-21T10:13:49.679581+0000","last_clean":"2024-09-21T10:13:49.679581+0000","last_became_active":"2024-09-21T10:08:00.386843+0000","last_became_peered":"2024-09-21T10:08:00.386843+0000","last_unstale":"2024-09-21T10:13:49.679581+0000","last_undegraded":"2024-09-21T10:13:49.679581+0000","last_fullsized":"2024-09-21T10:13:49.679581+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:51:00.654836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"219'4291","reported_seq":4315,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:58.290527+0000","last_change":"2024-09-21T10:13:00.067618+0000","last_active":"2024-09-21T10:13:58.290527+0000","last_peered":"2024-09-21T10:13:58.290527+0000","last_clean":"2024-09-21T10:13:58.290527+0000","last_became_active":"2024-09-21T10:13:00.067306+0000","last_became_peered":"2024-09-21T10:13:00.067306+0000","last_unstale":"2024-09-21T10:13:58.290527+0000","last_undegraded":"2024-09-21T10:13:58.290527+0000","last_fullsized":"2024-09-21T10:13:58.290527+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":4291,"log_dups_size":0,"ondisk_log_size":4291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:45:24.816636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281214976,"num_objects":4291,"num_object_clones":0,"num_object_copies":12873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4291,"num_write_kb":274624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":221,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.678779+0000","last_change":"2024-09-21T10:08:00.382496+0000","last_active":"2024-09-21T10:13:49.678779+0000","last_peered":"2024-09-21T10:13:49.678779+0000","last_clean":"2024-09-21T10:13:49.678779+0000","last_became_active":"2024-09-21T10:08:00.382180+0000","last_became_peered":"2024-09-21T10:08:00.382180+0000","last_unstale":"2024-09-21T10:13:49.678779+0000","last_undegraded":"2024-09-21T10:13:49.678779+0000","last_fullsized":"2024-09-21T10:13:49.678779+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:16.074736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":152,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.684584+0000","last_change":"2024-09-21T10:08:38.906801+0000","last_active":"2024-09-21T10:13:48.684584+0000","last_peered":"2024-09-21T10:13:48.684584+0000","last_clean":"2024-09-21T10:13:48.684584+0000","last_became_active":"2024-09-21T10:08:38.906487+0000","last_became_peered":"2024-09-21T10:08:38.906487+0000","last_unstale":"2024-09-21T10:13:48.684584+0000","last_undegraded":"2024-09-21T10:13:48.684584+0000","last_fullsized":"2024-09-21T10:13:48.684584+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:31.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"219'4274","reported_seq":4299,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:58.276183+0000","last_change":"2024-09-21T10:13:00.063591+0000","last_active":"2024-09-21T10:13:58.276183+0000","last_peered":"2024-09-21T10:13:58.276183+0000","last_clean":"2024-09-21T10:13:58.276183+0000","last_became_active":"2024-09-21T10:13:00.063262+0000","last_became_peered":"2024-09-21T10:13:00.063262+0000","last_unstale":"2024-09-21T10:13:58.276183+0000","last_undegraded":"2024-09-21T10:13:58.276183+0000","last_fullsized":"2024-09-21T10:13:58.276183+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":4274,"log_dups_size":0,"ondisk_log_size":4274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:52:08.153986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280100864,"num_objects":4274,"num_object_clones":0,"num_object_copies":12822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4274,"num_write_kb":273536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":309,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.690371+0000","last_change":"2024-09-21T10:02:14.097462+0000","last_active":"2024-09-21T10:13:49.690371+0000","last_peered":"2024-09-21T10:13:49.690371+0000","last_clean":"2024-09-21T10:13:49.690371+0000","last_became_active":"2024-09-21T10:02:14.097167+0000","last_became_peered":"2024-09-21T10:02:14.097167+0000","last_unstale":"2024-09-21T10:13:49.690371+0000","last_undegraded":"2024-09-21T10:13:49.690371+0000","last_fullsized":"2024-09-21T10:13:49.690371+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:54.922731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":313,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.683177+0000","last_change":"2024-09-21T10:08:00.378569+0000","last_active":"2024-09-21T10:13:49.683177+0000","last_peered":"2024-09-21T10:13:49.683177+0000","last_clean":"2024-09-21T10:13:49.683177+0000","last_became_active":"2024-09-21T10:08:00.378015+0000","last_became_peered":"2024-09-21T10:08:00.378015+0000","last_unstale":"2024-09-21T10:13:49.683177+0000","last_undegraded":"2024-09-21T10:13:49.683177+0000","last_fullsized":"2024-09-21T10:13:49.683177+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:58.121005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"219'3508","reported_seq":3592,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:57.721713+0000","last_change":"2024-09-21T10:13:41.424804+0000","last_active":"2024-09-21T10:13:57.721713+0000","last_peered":"2024-09-21T10:13:57.721713+0000","last_clean":"2024-09-21T10:13:57.721713+0000","last_became_active":"2024-09-21T10:13:36.511697+0000","last_became_peered":"2024-09-21T10:13:36.511697+0000","last_unstale":"2024-09-21T10:13:57.721713+0000","last_undegraded":"2024-09-21T10:13:57.721713+0000","last_fullsized":"2024-09-21T10:13:57.721713+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'3132","last_scrub_stamp":"2024-09-21T10:13:41.424720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:41.424720+0000","objects_scrubbed":1704,"log_size":3508,"log_dups_size":0,"ondisk_log_size":3508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:53:46.613625+0000","scrub_duration":522,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136904704,"num_objects":2089,"num_object_clones":0,"num_object_copies":6267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":385,"num_write_kb":24640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":301,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.677720+0000","last_change":"2024-09-21T09:59:54.265556+0000","last_active":"2024-09-21T10:13:48.677720+0000","last_peered":"2024-09-21T10:13:48.677720+0000","last_clean":"2024-09-21T10:13:48.677720+0000","last_became_active":"2024-09-21T09:59:54.265033+0000","last_became_peered":"2024-09-21T09:59:54.265033+0000","last_unstale":"2024-09-21T10:13:48.677720+0000","last_undegraded":"2024-09-21T10:13:48.677720+0000","last_fullsized":"2024-09-21T10:13:48.677720+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T09:48:34.850872+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:40.449756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":600,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.679289+0000","last_change":"2024-09-21T10:07:43.790490+0000","last_active":"2024-09-21T10:13:49.679289+0000","last_peered":"2024-09-21T10:13:49.679289+0000","last_clean":"2024-09-21T10:13:49.679289+0000","last_became_active":"2024-09-21T10:07:43.789986+0000","last_became_peered":"2024-09-21T10:07:43.789986+0000","last_unstale":"2024-09-21T10:13:49.679289+0000","last_undegraded":"2024-09-21T10:13:49.679289+0000","last_fullsized":"2024-09-21T10:13:49.679289+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:08.740356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":481,"num_read_kb":1580,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":262,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.690590+0000","last_change":"2024-09-21T10:02:14.099961+0000","last_active":"2024-09-21T10:13:49.690590+0000","last_peered":"2024-09-21T10:13:49.690590+0000","last_clean":"2024-09-21T10:13:49.690590+0000","last_became_active":"2024-09-21T10:02:14.099067+0000","last_became_peered":"2024-09-21T10:02:14.099067+0000","last_unstale":"2024-09-21T10:13:49.690590+0000","last_undegraded":"2024-09-21T10:13:49.690590+0000","last_fullsized":"2024-09-21T10:13:49.690590+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:41.037532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":195,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.684814+0000","last_change":"2024-09-21T10:08:00.386135+0000","last_active":"2024-09-21T10:13:48.684814+0000","last_peered":"2024-09-21T10:13:48.684814+0000","last_clean":"2024-09-21T10:13:48.684814+0000","last_became_active":"2024-09-21T10:08:00.385796+0000","last_became_peered":"2024-09-21T10:08:00.385796+0000","last_unstale":"2024-09-21T10:13:48.684814+0000","last_undegraded":"2024-09-21T10:13:48.684814+0000","last_fullsized":"2024-09-21T10:13:48.684814+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:27.054699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"219'3601","reported_seq":3637,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:58.273071+0000","last_change":"2024-09-21T10:13:36.530842+0000","last_active":"2024-09-21T10:13:58.273071+0000","last_peered":"2024-09-21T10:13:58.273071+0000","last_clean":"2024-09-21T10:13:58.273071+0000","last_became_active":"2024-09-21T10:13:36.529528+0000","last_became_peered":"2024-09-21T10:13:36.529528+0000","last_unstale":"2024-09-21T10:13:58.273071+0000","last_undegraded":"2024-09-21T10:13:58.273071+0000","last_fullsized":"2024-09-21T10:13:58.273071+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":3601,"log_dups_size":0,"ondisk_log_size":3601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140017664,"num_objects":2136,"num_object_clones":0,"num_object_copies":6408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2136,"num_write_kb":136736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":564,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.677652+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:13:48.677652+0000","last_peered":"2024-09-21T10:13:48.677652+0000","last_clean":"2024-09-21T10:13:48.677652+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:13:48.677652+0000","last_undegraded":"2024-09-21T10:13:48.677652+0000","last_fullsized":"2024-09-21T10:13:48.677652+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":325,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.896184+0000","last_change":"2024-09-21T10:08:00.383928+0000","last_active":"2024-09-21T10:13:49.896184+0000","last_peered":"2024-09-21T10:13:49.896184+0000","last_clean":"2024-09-21T10:13:49.896184+0000","last_became_active":"2024-09-21T10:08:00.383625+0000","last_became_peered":"2024-09-21T10:08:00.383625+0000","last_unstale":"2024-09-21T10:13:49.896184+0000","last_undegraded":"2024-09-21T10:13:49.896184+0000","last_fullsized":"2024-09-21T10:13:49.896184+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:39.207283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":315,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.896583+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:13:49.896583+0000","last_peered":"2024-09-21T10:13:49.896583+0000","last_clean":"2024-09-21T10:13:49.896583+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:13:49.896583+0000","last_undegraded":"2024-09-21T10:13:49.896583+0000","last_fullsized":"2024-09-21T10:13:49.896583+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"219'3541","reported_seq":3623,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:57.702068+0000","last_change":"2024-09-21T10:13:44.137191+0000","last_active":"2024-09-21T10:13:57.702068+0000","last_peered":"2024-09-21T10:13:57.702068+0000","last_clean":"2024-09-21T10:13:57.702068+0000","last_became_active":"2024-09-21T10:13:36.520394+0000","last_became_peered":"2024-09-21T10:13:36.520394+0000","last_unstale":"2024-09-21T10:13:57.702068+0000","last_undegraded":"2024-09-21T10:13:57.702068+0000","last_fullsized":"2024-09-21T10:13:57.702068+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'3166","last_scrub_stamp":"2024-09-21T10:13:44.137119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:44.137119+0000","objects_scrubbed":1736,"log_size":3541,"log_dups_size":0,"ondisk_log_size":3541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:11:36.314773+0000","scrub_duration":264,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138608640,"num_objects":2115,"num_object_clones":0,"num_object_copies":6345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":379,"num_write_kb":24256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":1052,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.679414+0000","last_change":"2024-09-21T10:02:14.100143+0000","last_active":"2024-09-21T10:13:49.679414+0000","last_peered":"2024-09-21T10:13:49.679414+0000","last_clean":"2024-09-21T10:13:49.679414+0000","last_became_active":"2024-09-21T10:02:14.099586+0000","last_became_peered":"2024-09-21T10:02:14.099586+0000","last_unstale":"2024-09-21T10:13:49.679414+0000","last_undegraded":"2024-09-21T10:13:49.679414+0000","last_fullsized":"2024-09-21T10:13:49.679414+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:34.064652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":187,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.682482+0000","last_change":"2024-09-21T10:08:18.470748+0000","last_active":"2024-09-21T10:13:49.682482+0000","last_peered":"2024-09-21T10:13:49.682482+0000","last_clean":"2024-09-21T10:13:49.682482+0000","last_became_active":"2024-09-21T10:08:18.470227+0000","last_became_peered":"2024-09-21T10:08:18.470227+0000","last_unstale":"2024-09-21T10:13:49.682482+0000","last_undegraded":"2024-09-21T10:13:49.682482+0000","last_fullsized":"2024-09-21T10:13:49.682482+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:16.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":309,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.679841+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:13:49.679841+0000","last_peered":"2024-09-21T10:13:49.679841+0000","last_clean":"2024-09-21T10:13:49.679841+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:13:49.679841+0000","last_undegraded":"2024-09-21T10:13:49.679841+0000","last_fullsized":"2024-09-21T10:13:49.679841+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"219'3681","reported_seq":3792,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:58.170620+0000","last_change":"2024-09-21T10:13:40.065182+0000","last_active":"2024-09-21T10:13:58.170620+0000","last_peered":"2024-09-21T10:13:58.170620+0000","last_clean":"2024-09-21T10:13:58.170620+0000","last_became_active":"2024-09-21T10:13:36.518471+0000","last_became_peered":"2024-09-21T10:13:36.518471+0000","last_unstale":"2024-09-21T10:13:58.170620+0000","last_undegraded":"2024-09-21T10:13:58.170620+0000","last_fullsized":"2024-09-21T10:13:58.170620+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'3140","last_scrub_stamp":"2024-09-21T10:13:40.065082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:40.065082+0000","objects_scrubbed":1666,"log_size":3681,"log_dups_size":0,"ondisk_log_size":3681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:34.697455+0000","scrub_duration":0.54074375200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145424384,"num_objects":2219,"num_object_clones":0,"num_object_copies":6657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":553,"num_write_kb":35392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":785,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.682393+0000","last_change":"2024-09-21T10:07:43.794767+0000","last_active":"2024-09-21T10:13:49.682393+0000","last_peered":"2024-09-21T10:13:49.682393+0000","last_clean":"2024-09-21T10:13:49.682393+0000","last_became_active":"2024-09-21T10:07:43.794062+0000","last_became_peered":"2024-09-21T10:07:43.794062+0000","last_unstale":"2024-09-21T10:13:49.682393+0000","last_undegraded":"2024-09-21T10:13:49.682393+0000","last_fullsized":"2024-09-21T10:13:49.682393+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:51.977955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":212,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.895942+0000","last_change":"2024-09-21T10:08:00.389176+0000","last_active":"2024-09-21T10:13:49.895942+0000","last_peered":"2024-09-21T10:13:49.895942+0000","last_clean":"2024-09-21T10:13:49.895942+0000","last_became_active":"2024-09-21T10:08:00.383854+0000","last_became_peered":"2024-09-21T10:08:00.383854+0000","last_unstale":"2024-09-21T10:13:49.895942+0000","last_undegraded":"2024-09-21T10:13:49.895942+0000","last_fullsized":"2024-09-21T10:13:49.895942+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:46.952216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":225,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.679401+0000","last_change":"2024-09-21T10:07:43.792153+0000","last_active":"2024-09-21T10:13:49.679401+0000","last_peered":"2024-09-21T10:13:49.679401+0000","last_clean":"2024-09-21T10:13:49.679401+0000","last_became_active":"2024-09-21T10:07:43.791770+0000","last_became_peered":"2024-09-21T10:07:43.791770+0000","last_unstale":"2024-09-21T10:13:49.679401+0000","last_undegraded":"2024-09-21T10:13:49.679401+0000","last_fullsized":"2024-09-21T10:13:49.679401+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:52.506941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"219'3530","reported_seq":3612,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:56.137668+0000","last_change":"2024-09-21T10:13:42.905750+0000","last_active":"2024-09-21T10:13:56.137668+0000","last_peered":"2024-09-21T10:13:56.137668+0000","last_clean":"2024-09-21T10:13:56.137668+0000","last_became_active":"2024-09-21T10:13:36.529866+0000","last_became_peered":"2024-09-21T10:13:36.529866+0000","last_unstale":"2024-09-21T10:13:56.137668+0000","last_undegraded":"2024-09-21T10:13:56.137668+0000","last_fullsized":"2024-09-21T10:13:56.137668+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'3209","last_scrub_stamp":"2024-09-21T10:13:42.905547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:42.905547+0000","objects_scrubbed":1742,"log_size":3530,"log_dups_size":0,"ondisk_log_size":3530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:14:37.177835+0000","scrub_duration":186,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135200768,"num_objects":2063,"num_object_clones":0,"num_object_copies":6189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":321,"num_write_kb":20544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":230,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.691000+0000","last_change":"2024-09-21T10:07:43.791405+0000","last_active":"2024-09-21T10:13:49.691000+0000","last_peered":"2024-09-21T10:13:49.691000+0000","last_clean":"2024-09-21T10:13:49.691000+0000","last_became_active":"2024-09-21T10:07:43.791078+0000","last_became_peered":"2024-09-21T10:07:43.791078+0000","last_unstale":"2024-09-21T10:13:49.691000+0000","last_undegraded":"2024-09-21T10:13:49.691000+0000","last_fullsized":"2024-09-21T10:13:49.691000+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:17:38.072612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":892,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.682538+0000","last_change":"2024-09-21T10:08:18.468995+0000","last_active":"2024-09-21T10:13:49.682538+0000","last_peered":"2024-09-21T10:13:49.682538+0000","last_clean":"2024-09-21T10:13:49.682538+0000","last_became_active":"2024-09-21T10:08:18.468243+0000","last_became_peered":"2024-09-21T10:08:18.468243+0000","last_unstale":"2024-09-21T10:13:49.682538+0000","last_undegraded":"2024-09-21T10:13:49.682538+0000","last_fullsized":"2024-09-21T10:13:49.682538+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:45.520968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":211,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.682618+0000","last_change":"2024-09-21T10:08:18.468595+0000","last_active":"2024-09-21T10:13:49.682618+0000","last_peered":"2024-09-21T10:13:49.682618+0000","last_clean":"2024-09-21T10:13:49.682618+0000","last_became_active":"2024-09-21T10:08:18.468296+0000","last_became_peered":"2024-09-21T10:08:18.468296+0000","last_unstale":"2024-09-21T10:13:49.682618+0000","last_undegraded":"2024-09-21T10:13:49.682618+0000","last_fullsized":"2024-09-21T10:13:49.682618+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:42.748130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"219'3438","reported_seq":3519,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:56.143545+0000","last_change":"2024-09-21T10:13:40.091216+0000","last_active":"2024-09-21T10:13:56.143545+0000","last_peered":"2024-09-21T10:13:56.143545+0000","last_clean":"2024-09-21T10:13:56.143545+0000","last_became_active":"2024-09-21T10:13:36.519364+0000","last_became_peered":"2024-09-21T10:13:36.519364+0000","last_unstale":"2024-09-21T10:13:56.143545+0000","last_undegraded":"2024-09-21T10:13:56.143545+0000","last_fullsized":"2024-09-21T10:13:56.143545+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'3004","last_scrub_stamp":"2024-09-21T10:13:40.090769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:40.090769+0000","objects_scrubbed":1593,"log_size":3438,"log_dups_size":0,"ondisk_log_size":3438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:28.924724+0000","scrub_duration":445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133300224,"num_objects":2034,"num_object_clones":0,"num_object_copies":6102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":441,"num_write_kb":28224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":187,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.682342+0000","last_change":"2024-09-21T10:08:18.469907+0000","last_active":"2024-09-21T10:13:49.682342+0000","last_peered":"2024-09-21T10:13:49.682342+0000","last_clean":"2024-09-21T10:13:49.682342+0000","last_became_active":"2024-09-21T10:08:18.469443+0000","last_became_peered":"2024-09-21T10:08:18.469443+0000","last_unstale":"2024-09-21T10:13:49.682342+0000","last_undegraded":"2024-09-21T10:13:49.682342+0000","last_fullsized":"2024-09-21T10:13:49.682342+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:15.645803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":614,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.678159+0000","last_change":"2024-09-21T10:08:00.419641+0000","last_active":"2024-09-21T10:13:48.678159+0000","last_peered":"2024-09-21T10:13:48.678159+0000","last_clean":"2024-09-21T10:13:48.678159+0000","last_became_active":"2024-09-21T10:08:00.387050+0000","last_became_peered":"2024-09-21T10:08:00.387050+0000","last_unstale":"2024-09-21T10:13:48.678159+0000","last_undegraded":"2024-09-21T10:13:48.678159+0000","last_fullsized":"2024-09-21T10:13:48.678159+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:00:32.881619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":293,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.931925+0000","last_change":"2024-09-21T09:59:54.267798+0000","last_active":"2024-09-21T10:13:49.931925+0000","last_peered":"2024-09-21T10:13:49.931925+0000","last_clean":"2024-09-21T10:13:49.931925+0000","last_became_active":"2024-09-21T09:59:54.265615+0000","last_became_peered":"2024-09-21T09:59:54.265615+0000","last_unstale":"2024-09-21T10:13:49.931925+0000","last_undegraded":"2024-09-21T10:13:49.931925+0000","last_fullsized":"2024-09-21T10:13:49.931925+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:30.363872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"219'3546","reported_seq":3628,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:57.806978+0000","last_change":"2024-09-21T10:13:40.433495+0000","last_active":"2024-09-21T10:13:57.806978+0000","last_peered":"2024-09-21T10:13:57.806978+0000","last_clean":"2024-09-21T10:13:57.806978+0000","last_became_active":"2024-09-21T10:13:36.518368+0000","last_became_peered":"2024-09-21T10:13:36.518368+0000","last_unstale":"2024-09-21T10:13:57.806978+0000","last_undegraded":"2024-09-21T10:13:57.806978+0000","last_fullsized":"2024-09-21T10:13:57.806978+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'3047","last_scrub_stamp":"2024-09-21T10:13:40.433382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:40.433382+0000","objects_scrubbed":1631,"log_size":3546,"log_dups_size":0,"ondisk_log_size":3546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:58:53.250061+0000","scrub_duration":476,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140771328,"num_objects":2148,"num_object_clones":0,"num_object_copies":6444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":517,"num_write_kb":33088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":306,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.691277+0000","last_change":"2024-09-21T10:02:14.101943+0000","last_active":"2024-09-21T10:13:49.691277+0000","last_peered":"2024-09-21T10:13:49.691277+0000","last_clean":"2024-09-21T10:13:49.691277+0000","last_became_active":"2024-09-21T10:02:14.101488+0000","last_became_peered":"2024-09-21T10:02:14.101488+0000","last_unstale":"2024-09-21T10:13:49.691277+0000","last_undegraded":"2024-09-21T10:13:49.691277+0000","last_fullsized":"2024-09-21T10:13:49.691277+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:52:27.866414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":556,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.683055+0000","last_change":"2024-09-21T10:08:18.472876+0000","last_active":"2024-09-21T10:13:49.683055+0000","last_peered":"2024-09-21T10:13:49.683055+0000","last_clean":"2024-09-21T10:13:49.683055+0000","last_became_active":"2024-09-21T10:08:18.472385+0000","last_became_peered":"2024-09-21T10:08:18.472385+0000","last_unstale":"2024-09-21T10:13:49.683055+0000","last_undegraded":"2024-09-21T10:13:49.683055+0000","last_fullsized":"2024-09-21T10:13:49.683055+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:33.109956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":124,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.678701+0000","last_change":"2024-09-21T10:08:38.797352+0000","last_active":"2024-09-21T10:13:49.678701+0000","last_peered":"2024-09-21T10:13:49.678701+0000","last_clean":"2024-09-21T10:13:49.678701+0000","last_became_active":"2024-09-21T10:08:38.796950+0000","last_became_peered":"2024-09-21T10:08:38.796950+0000","last_unstale":"2024-09-21T10:13:49.678701+0000","last_undegraded":"2024-09-21T10:13:49.678701+0000","last_fullsized":"2024-09-21T10:13:49.678701+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:02.646321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2","version":"219'3527","reported_seq":3606,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:57.847394+0000","last_change":"2024-09-21T10:13:38.781080+0000","last_active":"2024-09-21T10:13:57.847394+0000","last_peered":"2024-09-21T10:13:57.847394+0000","last_clean":"2024-09-21T10:13:57.847394+0000","last_became_active":"2024-09-21T10:13:36.518804+0000","last_became_peered":"2024-09-21T10:13:36.518804+0000","last_unstale":"2024-09-21T10:13:57.847394+0000","last_undegraded":"2024-09-21T10:13:57.847394+0000","last_fullsized":"2024-09-21T10:13:57.847394+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'2979","last_scrub_stamp":"2024-09-21T10:13:38.780919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:38.780919+0000","objects_scrubbed":1566,"log_size":3527,"log_dups_size":0,"ondisk_log_size":3527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:13:53.612648+0000","scrub_duration":769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139329536,"num_objects":2126,"num_object_clones":0,"num_object_copies":6378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":560,"num_write_kb":35840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":341,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.896640+0000","last_change":"2024-09-21T10:08:00.390436+0000","last_active":"2024-09-21T10:13:49.896640+0000","last_peered":"2024-09-21T10:13:49.896640+0000","last_clean":"2024-09-21T10:13:49.896640+0000","last_became_active":"2024-09-21T10:08:00.381835+0000","last_became_peered":"2024-09-21T10:08:00.381835+0000","last_unstale":"2024-09-21T10:13:49.896640+0000","last_undegraded":"2024-09-21T10:13:49.896640+0000","last_fullsized":"2024-09-21T10:13:49.896640+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:01:35.697356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":445,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.684914+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:13:48.684914+0000","last_peered":"2024-09-21T10:13:48.684914+0000","last_clean":"2024-09-21T10:13:48.684914+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:13:48.684914+0000","last_undegraded":"2024-09-21T10:13:48.684914+0000","last_fullsized":"2024-09-21T10:13:48.684914+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":83,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.684578+0000","last_change":"2024-09-21T10:08:00.385774+0000","last_active":"2024-09-21T10:13:48.684578+0000","last_peered":"2024-09-21T10:13:48.684578+0000","last_clean":"2024-09-21T10:13:48.684578+0000","last_became_active":"2024-09-21T10:08:00.385445+0000","last_became_peered":"2024-09-21T10:08:00.385445+0000","last_unstale":"2024-09-21T10:13:48.684578+0000","last_undegraded":"2024-09-21T10:13:48.684578+0000","last_fullsized":"2024-09-21T10:13:48.684578+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:42.442190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.10","version":"219'3431","reported_seq":7950,"reported_epoch":219,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T10:13:56.150968+0000","last_change":"2024-09-21T10:13:43.758169+0000","last_active":"2024-09-21T10:13:56.150968+0000","last_peered":"2024-09-21T10:13:56.150968+0000","last_clean":"2024-09-21T10:13:41.660161+0000","last_became_active":"2024-09-21T10:13:43.743910+0000","last_became_peered":"2024-09-21T10:13:43.743910+0000","last_unstale":"2024-09-21T10:13:56.150968+0000","last_undegraded":"2024-09-21T10:13:56.150968+0000","last_fullsized":"2024-09-21T10:13:43.672803+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":3431,"log_dups_size":0,"ondisk_log_size":3431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132251648,"num_objects":2018,"num_object_clones":0,"num_object_copies":6054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":358,"num_objects_unfound":0,"num_objects_dirty":2018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2018,"num_write_kb":129152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1487,"num_bytes_recovered":97452032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[1,2],"avail_no_missing":["1","3","5"],"object_location_counts":[{"shards":"1,2","objects":1839},{"shards":"1,3,5","objects":179}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":142,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.684790+0000","last_change":"2024-09-21T10:08:00.379661+0000","last_active":"2024-09-21T10:13:48.684790+0000","last_peered":"2024-09-21T10:13:48.684790+0000","last_clean":"2024-09-21T10:13:48.684790+0000","last_became_active":"2024-09-21T10:08:00.379301+0000","last_became_peered":"2024-09-21T10:08:00.379301+0000","last_unstale":"2024-09-21T10:13:48.684790+0000","last_undegraded":"2024-09-21T10:13:48.684790+0000","last_fullsized":"2024-09-21T10:13:48.684790+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:15.649523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":166,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.677827+0000","last_change":"2024-09-21T10:08:00.410731+0000","last_active":"2024-09-21T10:13:48.677827+0000","last_peered":"2024-09-21T10:13:48.677827+0000","last_clean":"2024-09-21T10:13:48.677827+0000","last_became_active":"2024-09-21T10:08:00.380324+0000","last_became_peered":"2024-09-21T10:08:00.380324+0000","last_unstale":"2024-09-21T10:13:48.677827+0000","last_undegraded":"2024-09-21T10:13:48.677827+0000","last_fullsized":"2024-09-21T10:13:48.677827+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:47:34.960221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.11","version":"219'3682","reported_seq":8380,"reported_epoch":219,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T10:13:58.303431+0000","last_change":"2024-09-21T10:13:46.019675+0000","last_active":"2024-09-21T10:13:58.303431+0000","last_peered":"2024-09-21T10:13:58.303431+0000","last_clean":"2024-09-21T10:13:42.083185+0000","last_became_active":"2024-09-21T10:13:45.994456+0000","last_became_peered":"2024-09-21T10:13:45.994456+0000","last_unstale":"2024-09-21T10:13:58.303431+0000","last_undegraded":"2024-09-21T10:13:58.303431+0000","last_fullsized":"2024-09-21T10:13:45.673989+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":5,"last_scrub":"211'3233","last_scrub_stamp":"2024-09-21T10:13:42.082966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:42.082966+0000","objects_scrubbed":1774,"log_size":3682,"log_dups_size":0,"ondisk_log_size":3682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:03.099207+0000","scrub_duration":372,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146014208,"num_objects":2228,"num_object_clones":0,"num_object_copies":6684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":482,"num_objects_unfound":0,"num_objects_dirty":2228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":454,"num_write_kb":29056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1539,"num_bytes_recovered":100859904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,3],"avail_no_missing":["7","0","1"],"object_location_counts":[{"shards":"0,1,7","objects":241},{"shards":"3,7","objects":1987}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":187,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.682485+0000","last_change":"2024-09-21T10:08:18.470378+0000","last_active":"2024-09-21T10:13:49.682485+0000","last_peered":"2024-09-21T10:13:49.682485+0000","last_clean":"2024-09-21T10:13:49.682485+0000","last_became_active":"2024-09-21T10:08:18.469482+0000","last_became_peered":"2024-09-21T10:08:18.469482+0000","last_unstale":"2024-09-21T10:13:49.682485+0000","last_undegraded":"2024-09-21T10:13:49.682485+0000","last_fullsized":"2024-09-21T10:13:49.682485+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:35.628724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":235,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.678073+0000","last_change":"2024-09-21T10:08:00.403606+0000","last_active":"2024-09-21T10:13:48.678073+0000","last_peered":"2024-09-21T10:13:48.678073+0000","last_clean":"2024-09-21T10:13:48.678073+0000","last_became_active":"2024-09-21T10:08:00.379330+0000","last_became_peered":"2024-09-21T10:08:00.379330+0000","last_unstale":"2024-09-21T10:13:48.678073+0000","last_undegraded":"2024-09-21T10:13:48.678073+0000","last_fullsized":"2024-09-21T10:13:48.678073+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:07.226027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"219'3515","reported_seq":3596,"reported_epoch":219,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:13:56.148822+0000","last_change":"2024-09-21T10:13:47.808806+0000","last_active":"2024-09-21T10:13:56.148822+0000","last_peered":"2024-09-21T10:13:56.148822+0000","last_clean":"2024-09-21T10:13:44.026224+0000","last_became_active":"2024-09-21T10:13:47.786447+0000","last_became_peered":"2024-09-21T10:13:47.786447+0000","last_unstale":"2024-09-21T10:13:56.148822+0000","last_undegraded":"2024-09-21T10:13:47.783646+0000","last_fullsized":"2024-09-21T10:13:47.681002+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":5,"last_scrub":"211'3189","last_scrub_stamp":"2024-09-21T10:13:43.205779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:43.205779+0000","objects_scrubbed":1704,"log_size":3515,"log_dups_size":0,"ondisk_log_size":3515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":220,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133038080,"num_objects":2030,"num_object_clones":0,"num_object_copies":6090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1715,"num_objects_misplaced":1715,"num_objects_unfound":0,"num_objects_dirty":2030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":326,"num_write_kb":20864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[1,2],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2","objects":315},{"shards":"1,2,3","objects":1715}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":131,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.678320+0000","last_change":"2024-09-21T10:07:43.791830+0000","last_active":"2024-09-21T10:13:48.678320+0000","last_peered":"2024-09-21T10:13:48.678320+0000","last_clean":"2024-09-21T10:13:48.678320+0000","last_became_active":"2024-09-21T10:07:43.791314+0000","last_became_peered":"2024-09-21T10:07:43.791314+0000","last_unstale":"2024-09-21T10:13:48.678320+0000","last_undegraded":"2024-09-21T10:13:48.678320+0000","last_fullsized":"2024-09-21T10:13:48.678320+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:37.086629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":263,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.690318+0000","last_change":"2024-09-21T10:02:14.100451+0000","last_active":"2024-09-21T10:13:49.690318+0000","last_peered":"2024-09-21T10:13:49.690318+0000","last_clean":"2024-09-21T10:13:49.690318+0000","last_became_active":"2024-09-21T10:02:14.100160+0000","last_became_peered":"2024-09-21T10:02:14.100160+0000","last_unstale":"2024-09-21T10:13:49.690318+0000","last_undegraded":"2024-09-21T10:13:49.690318+0000","last_fullsized":"2024-09-21T10:13:49.690318+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:28.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"219'3441","reported_seq":3500,"reported_epoch":219,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:13:55.012055+0000","last_change":"2024-09-21T10:13:49.936709+0000","last_active":"2024-09-21T10:13:55.012055+0000","last_peered":"2024-09-21T10:13:55.012055+0000","last_clean":"2024-09-21T10:13:46.684614+0000","last_became_active":"2024-09-21T10:13:49.933524+0000","last_became_peered":"2024-09-21T10:13:49.933524+0000","last_unstale":"2024-09-21T10:13:55.012055+0000","last_undegraded":"2024-09-21T10:13:49.716869+0000","last_fullsized":"2024-09-21T10:13:49.679048+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":3441,"log_dups_size":0,"ondisk_log_size":3441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131661824,"num_objects":2009,"num_object_clones":0,"num_object_copies":6027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2009,"num_write_kb":128576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[2,0],"avail_no_missing":["2","3","6"],"object_location_counts":[{"shards":"0,2","objects":291},{"shards":"2,3,6","objects":1718}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":303,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.896552+0000","last_change":"2024-09-21T10:02:14.096098+0000","last_active":"2024-09-21T10:13:49.896552+0000","last_peered":"2024-09-21T10:13:49.896552+0000","last_clean":"2024-09-21T10:13:49.896552+0000","last_became_active":"2024-09-21T10:02:14.095811+0000","last_became_peered":"2024-09-21T10:02:14.095811+0000","last_unstale":"2024-09-21T10:13:49.896552+0000","last_undegraded":"2024-09-21T10:13:49.896552+0000","last_fullsized":"2024-09-21T10:13:49.896552+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:54.619333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":308,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.690320+0000","last_change":"2024-09-21T10:02:14.100712+0000","last_active":"2024-09-21T10:13:49.690320+0000","last_peered":"2024-09-21T10:13:49.690320+0000","last_clean":"2024-09-21T10:13:49.690320+0000","last_became_active":"2024-09-21T10:02:14.100427+0000","last_became_peered":"2024-09-21T10:02:14.100427+0000","last_unstale":"2024-09-21T10:13:49.690320+0000","last_undegraded":"2024-09-21T10:13:49.690320+0000","last_fullsized":"2024-09-21T10:13:49.690320+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:16:42.350642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":182,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.684797+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:13:48.684797+0000","last_peered":"2024-09-21T10:13:48.684797+0000","last_clean":"2024-09-21T10:13:48.684797+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:13:48.684797+0000","last_undegraded":"2024-09-21T10:13:48.684797+0000","last_fullsized":"2024-09-21T10:13:48.684797+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":325,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.690326+0000","last_change":"2024-09-21T10:02:14.097369+0000","last_active":"2024-09-21T10:13:49.690326+0000","last_peered":"2024-09-21T10:13:49.690326+0000","last_clean":"2024-09-21T10:13:49.690326+0000","last_became_active":"2024-09-21T10:02:14.097067+0000","last_became_peered":"2024-09-21T10:02:14.097067+0000","last_unstale":"2024-09-21T10:13:49.690326+0000","last_undegraded":"2024-09-21T10:13:49.690326+0000","last_fullsized":"2024-09-21T10:13:49.690326+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:50:48.123813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":312,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.896519+0000","last_change":"2024-09-21T10:12:27.977282+0000","last_active":"2024-09-21T10:13:49.896519+0000","last_peered":"2024-09-21T10:13:49.896519+0000","last_clean":"2024-09-21T10:13:49.896519+0000","last_became_active":"2024-09-21T10:12:27.976924+0000","last_became_peered":"2024-09-21T10:12:27.976924+0000","last_unstale":"2024-09-21T10:13:49.896519+0000","last_undegraded":"2024-09-21T10:13:49.896519+0000","last_fullsized":"2024-09-21T10:13:49.896519+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:57:19.477172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":313,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.682684+0000","last_change":"2024-09-21T10:08:00.381058+0000","last_active":"2024-09-21T10:13:49.682684+0000","last_peered":"2024-09-21T10:13:49.682684+0000","last_clean":"2024-09-21T10:13:49.682684+0000","last_became_active":"2024-09-21T10:08:00.380573+0000","last_became_peered":"2024-09-21T10:08:00.380573+0000","last_unstale":"2024-09-21T10:13:49.682684+0000","last_undegraded":"2024-09-21T10:13:49.682684+0000","last_fullsized":"2024-09-21T10:13:49.682684+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:01:04.137425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":328,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.690883+0000","last_change":"2024-09-21T10:08:00.387321+0000","last_active":"2024-09-21T10:13:49.690883+0000","last_peered":"2024-09-21T10:13:49.690883+0000","last_clean":"2024-09-21T10:13:49.690883+0000","last_became_active":"2024-09-21T10:08:00.386935+0000","last_became_peered":"2024-09-21T10:08:00.386935+0000","last_unstale":"2024-09-21T10:13:49.690883+0000","last_undegraded":"2024-09-21T10:13:49.690883+0000","last_fullsized":"2024-09-21T10:13:49.690883+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:57.706313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":133,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.682699+0000","last_change":"2024-09-21T10:08:18.471282+0000","last_active":"2024-09-21T10:13:49.682699+0000","last_peered":"2024-09-21T10:13:49.682699+0000","last_clean":"2024-09-21T10:13:49.682699+0000","last_became_active":"2024-09-21T10:08:18.470952+0000","last_became_peered":"2024-09-21T10:08:18.470952+0000","last_unstale":"2024-09-21T10:13:49.682699+0000","last_undegraded":"2024-09-21T10:13:49.682699+0000","last_fullsized":"2024-09-21T10:13:49.682699+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:25.597081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":153,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.682785+0000","last_change":"2024-09-21T10:11:54.672536+0000","last_active":"2024-09-21T10:13:49.682785+0000","last_peered":"2024-09-21T10:13:49.682785+0000","last_clean":"2024-09-21T10:13:49.682785+0000","last_became_active":"2024-09-21T10:11:54.672206+0000","last_became_peered":"2024-09-21T10:11:54.672206+0000","last_unstale":"2024-09-21T10:13:49.682785+0000","last_undegraded":"2024-09-21T10:13:49.682785+0000","last_fullsized":"2024-09-21T10:13:49.682785+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:45:20.652276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":317,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.896485+0000","last_change":"2024-09-21T10:08:00.386104+0000","last_active":"2024-09-21T10:13:49.896485+0000","last_peered":"2024-09-21T10:13:49.896485+0000","last_clean":"2024-09-21T10:13:49.896485+0000","last_became_active":"2024-09-21T10:08:00.381484+0000","last_became_peered":"2024-09-21T10:08:00.381484+0000","last_unstale":"2024-09-21T10:13:49.896485+0000","last_undegraded":"2024-09-21T10:13:49.896485+0000","last_fullsized":"2024-09-21T10:13:49.896485+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:39.190191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"219'3476","reported_seq":3559,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:54.988044+0000","last_change":"2024-09-21T10:13:42.089302+0000","last_active":"2024-09-21T10:13:54.988044+0000","last_peered":"2024-09-21T10:13:54.988044+0000","last_clean":"2024-09-21T10:13:54.988044+0000","last_became_active":"2024-09-21T10:13:36.538007+0000","last_became_peered":"2024-09-21T10:13:36.538007+0000","last_unstale":"2024-09-21T10:13:54.988044+0000","last_undegraded":"2024-09-21T10:13:54.988044+0000","last_fullsized":"2024-09-21T10:13:54.988044+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":5,"last_scrub":"211'3195","last_scrub_stamp":"2024-09-21T10:13:42.089240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:42.089240+0000","objects_scrubbed":1718,"log_size":3476,"log_dups_size":0,"ondisk_log_size":3476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:48.412757+0000","scrub_duration":332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072000,"num_objects":2000,"num_object_clones":0,"num_object_copies":6000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":282,"num_write_kb":18048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":242,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.690771+0000","last_change":"2024-09-21T10:08:00.381149+0000","last_active":"2024-09-21T10:13:49.690771+0000","last_peered":"2024-09-21T10:13:49.690771+0000","last_clean":"2024-09-21T10:13:49.690771+0000","last_became_active":"2024-09-21T10:08:00.380795+0000","last_became_peered":"2024-09-21T10:08:00.380795+0000","last_unstale":"2024-09-21T10:13:49.690771+0000","last_undegraded":"2024-09-21T10:13:49.690771+0000","last_fullsized":"2024-09-21T10:13:49.690771+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:50:46.793182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":144,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.682787+0000","last_change":"2024-09-21T10:08:18.466882+0000","last_active":"2024-09-21T10:13:49.682787+0000","last_peered":"2024-09-21T10:13:49.682787+0000","last_clean":"2024-09-21T10:13:49.682787+0000","last_became_active":"2024-09-21T10:08:18.466470+0000","last_became_peered":"2024-09-21T10:08:18.466470+0000","last_unstale":"2024-09-21T10:13:49.682787+0000","last_undegraded":"2024-09-21T10:13:49.682787+0000","last_fullsized":"2024-09-21T10:13:49.682787+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:51:31.020051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.19","version":"219'3555","reported_seq":3614,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:57.717537+0000","last_change":"2024-09-21T10:13:47.865419+0000","last_active":"2024-09-21T10:13:57.717537+0000","last_peered":"2024-09-21T10:13:57.717537+0000","last_clean":"2024-09-21T10:13:57.717537+0000","last_became_active":"2024-09-21T10:13:36.535264+0000","last_became_peered":"2024-09-21T10:13:36.535264+0000","last_unstale":"2024-09-21T10:13:57.717537+0000","last_undegraded":"2024-09-21T10:13:57.717537+0000","last_fullsized":"2024-09-21T10:13:57.717537+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":3555,"log_dups_size":0,"ondisk_log_size":3555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T05:01:23.353763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139722752,"num_objects":2132,"num_object_clones":0,"num_object_copies":6396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2132,"num_write_kb":136448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":242,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.690678+0000","last_change":"2024-09-21T10:08:00.389061+0000","last_active":"2024-09-21T10:13:49.690678+0000","last_peered":"2024-09-21T10:13:49.690678+0000","last_clean":"2024-09-21T10:13:49.690678+0000","last_became_active":"2024-09-21T10:08:00.388657+0000","last_became_peered":"2024-09-21T10:08:00.388657+0000","last_unstale":"2024-09-21T10:13:49.690678+0000","last_undegraded":"2024-09-21T10:13:49.690678+0000","last_fullsized":"2024-09-21T10:13:49.690678+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:58:17.239589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":94,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.684623+0000","last_change":"2024-09-21T10:08:00.394350+0000","last_active":"2024-09-21T10:13:48.684623+0000","last_peered":"2024-09-21T10:13:48.684623+0000","last_clean":"2024-09-21T10:13:48.684623+0000","last_became_active":"2024-09-21T10:08:00.387232+0000","last_became_peered":"2024-09-21T10:08:00.387232+0000","last_unstale":"2024-09-21T10:13:48.684623+0000","last_undegraded":"2024-09-21T10:13:48.684623+0000","last_fullsized":"2024-09-21T10:13:48.684623+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:56.160018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":328,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.690590+0000","last_change":"2024-09-21T10:02:14.099799+0000","last_active":"2024-09-21T10:13:49.690590+0000","last_peered":"2024-09-21T10:13:49.690590+0000","last_clean":"2024-09-21T10:13:49.690590+0000","last_became_active":"2024-09-21T10:02:14.099375+0000","last_became_peered":"2024-09-21T10:02:14.099375+0000","last_unstale":"2024-09-21T10:13:49.690590+0000","last_undegraded":"2024-09-21T10:13:49.690590+0000","last_fullsized":"2024-09-21T10:13:49.690590+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:13.407638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":314,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.691036+0000","last_change":"2024-09-21T10:02:14.099439+0000","last_active":"2024-09-21T10:13:49.691036+0000","last_peered":"2024-09-21T10:13:49.691036+0000","last_clean":"2024-09-21T10:13:49.691036+0000","last_became_active":"2024-09-21T10:02:14.099117+0000","last_became_peered":"2024-09-21T10:02:14.099117+0000","last_unstale":"2024-09-21T10:13:49.691036+0000","last_undegraded":"2024-09-21T10:13:49.691036+0000","last_fullsized":"2024-09-21T10:13:49.691036+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:39.681817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":157,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-21T10:13:48.684043+0000","last_change":"2024-09-21T10:08:00.381529+0000","last_active":"2024-09-21T10:13:48.684043+0000","last_peered":"2024-09-21T10:13:48.684043+0000","last_clean":"2024-09-21T10:13:48.684043+0000","last_became_active":"2024-09-21T10:08:00.381061+0000","last_became_peered":"2024-09-21T10:08:00.381061+0000","last_unstale":"2024-09-21T10:13:48.684043+0000","last_undegraded":"2024-09-21T10:13:48.684043+0000","last_fullsized":"2024-09-21T10:13:48.684043+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:56:58.775586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":308,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-21T10:13:49.896536+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:13:49.896536+0000","last_peered":"2024-09-21T10:13:49.896536+0000","last_clean":"2024-09-21T10:13:49.896536+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:13:49.896536+0000","last_undegraded":"2024-09-21T10:13:49.896536+0000","last_fullsized":"2024-09-21T10:13:49.896536+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":4409720832,"num_objects":67287,"num_object_clones":0,"num_object_copies":201861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5151,"num_objects_misplaced":2555,"num_objects_unfound":0,"num_objects_dirty":67287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47035,"num_write_kb":3010240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3026,"num_bytes_recovered":198311936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13706264576,"data_stored":13706264576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96086,"ondisk_log_size":96086,"up":78,"acting":74,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":9926,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":21480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5982,"num_read_kb":13005,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397790,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2418540,"kb_used_data":2117028,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91326612,"statfs":{"total":95995035648,"available":93518450688,"internally_reserved":0,"allocated":2167836672,"data_stored":2166758992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594018,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2033640,"kb_used_data":1719648,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":91711512,"statfs":{"total":95995035648,"available":93912588288,"internally_reserved":0,"allocated":1760919552,"data_stored":1759845875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790248,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1648788,"kb_used_data":1343116,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92096364,"statfs":{"total":95995035648,"available":94306676736,"internally_reserved":0,"allocated":1375350784,"data_stored":1374292520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986477,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":101,"seq":433791697039,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2552592,"kb_used_data":2193992,"kb_used_omap":5,"kb_used_meta":358586,"kb_avail":91192560,"statfs":{"total":95995035648,"available":93381181440,"internally_reserved":0,"allocated":2246647808,"data_stored":2245593398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":367192848},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860565,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2016672,"kb_used_data":1641752,"kb_used_omap":8,"kb_used_meta":374903,"kb_avail":91728480,"statfs":{"total":95995035648,"available":93929963520,"internally_reserved":0,"allocated":1681154048,"data_stored":1680088520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":383901548},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024092,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2145100,"kb_used_data":1812164,"kb_used_omap":7,"kb_used_meta":332920,"kb_avail":91600052,"statfs":{"total":95995035648,"available":93798453248,"internally_reserved":0,"allocated":1855655936,"data_stored":1854604951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":340910642},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187627,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2873820,"kb_used_data":2470740,"kb_used_omap":8,"kb_used_meta":403063,"kb_avail":90871332,"statfs":{"total":95995035648,"available":93052243968,"internally_reserved":0,"allocated":2530037760,"data_stored":2528972206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":412737521},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2531524608,"data_stored":2531524608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1860632576,"data_stored":1860632576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1688403968,"data_stored":1688403968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2281308160,"data_stored":2281308160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1381105664,"data_stored":1381105664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1797521408,"data_stored":1797521408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2165768192,"data_stored":2165768192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:14:01.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_2 pgp_num 26 2024-09-21T10:14:01.169 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:01.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:01.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:01.247 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 48 16 71038 71022 92.4595 30.9375 0.0165415 0.0108098 2024-09-21T10:14:01.271 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:01.295 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:01.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:01.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:01.397 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:01.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:01.443 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:01.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:01.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:01.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:01.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:01.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:01.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:01.698 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:01.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:01.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:01.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:01.840 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:01.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:01.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:01.941 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:01.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:01.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:02.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:01 smithi098 ceph-mon[97580]: osdmap e222: 8 total, 8 up, 7 in 2024-09-21T10:14:02.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:01 smithi098 ceph-mon[100914]: osdmap e222: 8 total, 8 up, 7 in 2024-09-21T10:14:02.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:02.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:02.129 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:02.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:02.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:01 smithi196 ceph-mon[84784]: osdmap e222: 8 total, 8 up, 7 in 2024-09-21T10:14:02.174 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:02.229 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:02.247 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 49 16 72425 72409 92.3415 86.6875 0.00632612 0.0108238 2024-09-21T10:14:02.260 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:02.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:02.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:02.362 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:02.390 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:02.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:02.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:02.492 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:02.516 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:02.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:02.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:02.618 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:02.645 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:02.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:02.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:02.747 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:02.790 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:02.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:02.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:02.892 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:02.931 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:02.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:02.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:03.033 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:03.076 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:03.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:03.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:03.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:02 smithi098 ceph-mon[97580]: pgmap v665: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 83 MiB/s wr, 1.33k op/s; 1715/207759 objects degraded (0.825%); 5335/207759 objects misplaced (2.568%); 13 MiB/s, 213 objects/s recovering 2024-09-21T10:14:03.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:02 smithi098 ceph-mon[100914]: pgmap v665: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 83 MiB/s wr, 1.33k op/s; 1715/207759 objects degraded (0.825%); 5335/207759 objects misplaced (2.568%); 13 MiB/s, 213 objects/s recovering 2024-09-21T10:14:03.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:02 smithi196 ceph-mon[84784]: pgmap v665: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 83 MiB/s wr, 1.33k op/s; 1715/207759 objects degraded (0.825%); 5335/207759 objects misplaced (2.568%); 13 MiB/s, 213 objects/s recovering 2024-09-21T10:14:03.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:03.206 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:03.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:03.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:03.247 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 50 16 73917 73901 92.3595 93.25 0.0131927 0.0108226 2024-09-21T10:14:03.308 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:03.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:03.348 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:03.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:03.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:03.501 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:03.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:03.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:03.603 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:03.630 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:03.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:03.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:03.732 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:03.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:03.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:03.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:03.854 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:03.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:03.883 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:03.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:03.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:03.984 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:04.012 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:04.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:04.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:04.114 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:04.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:03 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 1715/213570 objects degraded (0.803%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:04.152 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:04.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:04.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:04.247 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 51 16 75356 75340 92.3118 89.9375 0.0103019 0.0108279 2024-09-21T10:14:04.254 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:04.279 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:04.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:04.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:04.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:03 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 1715/213570 objects degraded (0.803%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:04.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:03 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 1715/213570 objects degraded (0.803%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:04.381 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:04.407 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:04.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:04.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:04.509 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:04.558 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:04.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:04.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:04.660 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:04.681 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:04.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:04.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:04.783 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:04.808 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:04.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:04.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:04.896 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:04 smithi098 ceph-mon[97580]: pgmap v666: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 4.3 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 89 MiB/s wr, 1.43k op/s; 1715/213570 objects degraded (0.803%); 5335/213570 objects misplaced (2.498%); 9.4 MiB/s, 149 objects/s recovering 2024-09-21T10:14:04.897 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:04 smithi098 ceph-mon[100914]: pgmap v666: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 4.3 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 89 MiB/s wr, 1.43k op/s; 1715/213570 objects degraded (0.803%); 5335/213570 objects misplaced (2.498%); 9.4 MiB/s, 149 objects/s recovering 2024-09-21T10:14:04.910 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:04.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:04.948 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:04.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:05.049 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:05.073 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:05.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:05.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:05.175 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:05.219 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:05.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:05.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:05.247 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 52 16 76591 76575 92.0209 77.1875 0.0094692 0.0108604 2024-09-21T10:14:05.321 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:05.364 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:05.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:05.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:05.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:04 smithi196 ceph-mon[84784]: pgmap v666: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 4.3 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 89 MiB/s wr, 1.43k op/s; 1715/213570 objects degraded (0.803%); 5335/213570 objects misplaced (2.498%); 9.4 MiB/s, 149 objects/s recovering 2024-09-21T10:14:05.466 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:05.502 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:05.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:05.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:05.603 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:05.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:05.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:05.737 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:05.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:05.781 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:05.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:05.876 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:05.882 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:05.931 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:14:05.931 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:14:05.934 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:14:05.942 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:05.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:05.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:05.966 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:14:05.966 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:14:06.027 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:14:06.028 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:14:06.044 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:06.064 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:14:06.064 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:14:06.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:06.093 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:06.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:06.096 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:14:06.096 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:14:06.134 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:14:06.134 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:14:06.167 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:05 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/140314567' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:14:06.167 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:05 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:14:06.167 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:05 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/638936682' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:06.167 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:14:06.167 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:14:06.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:05 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/140314567' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:14:06.168 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:05 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:14:06.168 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:05 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/638936682' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:06.194 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:06.202 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:14:06.202 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:14:06.220 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:06.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:06.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:06.248 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 53 16 77641 77625 91.5228 65.625 0.00855332 0.0109208 2024-09-21T10:14:06.321 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:06.357 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:06.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:06.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:06.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:05 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/140314567' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:14:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:05 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:14:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:05 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/638936682' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:06.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:06.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:06.494 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:06.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:06.595 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:06.621 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:06.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:06.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:06.723 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:06.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:06.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:06.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:06.851 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:06.878 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:06.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:06.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:06.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:06 smithi098 ceph-mon[97580]: pgmap v667: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 62 MiB/s wr, 998 op/s; 1715/221214 objects degraded (0.775%); 4193/221214 objects misplaced (1.895%); 8.7 MiB/s, 139 objects/s recovering 2024-09-21T10:14:06.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:06 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:14:06.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:06 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:14:06.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:06 smithi098 ceph-mon[97580]: osdmap e223: 8 total, 8 up, 7 in 2024-09-21T10:14:06.976 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:06 smithi098 ceph-mon[100914]: pgmap v667: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 62 MiB/s wr, 998 op/s; 1715/221214 objects degraded (0.775%); 4193/221214 objects misplaced (1.895%); 8.7 MiB/s, 139 objects/s recovering 2024-09-21T10:14:06.977 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:06 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:14:06.977 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:06 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:14:06.977 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:06 smithi098 ceph-mon[100914]: osdmap e223: 8 total, 8 up, 7 in 2024-09-21T10:14:06.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:07.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:07.028 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:07.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:07.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:07.164 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:07.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:07.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:07.248 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 54 16 78884 78868 91.2665 77.6875 0.0130938 0.0109523 2024-09-21T10:14:07.266 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:07.300 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:07.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:07.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:07.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:06 smithi196 ceph-mon[84784]: pgmap v667: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 62 MiB/s wr, 998 op/s; 1715/221214 objects degraded (0.775%); 4193/221214 objects misplaced (1.895%); 8.7 MiB/s, 139 objects/s recovering 2024-09-21T10:14:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:06 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:14:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:06 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:14:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:06 smithi196 ceph-mon[84784]: osdmap e223: 8 total, 8 up, 7 in 2024-09-21T10:14:07.402 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:07.428 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:07.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:07.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:07.530 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:07.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:07.551 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:07.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:07.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:07.688 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:07.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:07.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:07.790 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:07.817 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:07.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:07.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:07.919 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:07.946 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:07.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:07.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:08.048 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:08.073 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:08.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:08.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:08.175 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:08.203 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:08.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:08.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:08.248 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 55 16 80196 80180 91.0979 82 0.0104847 0.0109723 2024-09-21T10:14:08.304 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:08.333 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:08.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:08.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:08.435 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:08.470 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:08.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:08.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:08.571 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:14:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:08 smithi098 ceph-mon[97580]: pgmap v669: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 63 MiB/s wr, 1.00k op/s; 1715/224316 objects degraded (0.765%); 4193/224316 objects misplaced (1.869%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T10:14:08.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:14:08.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:08.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:08 smithi098 ceph-mon[100914]: pgmap v669: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 63 MiB/s wr, 1.00k op/s; 1715/224316 objects degraded (0.765%); 4193/224316 objects misplaced (1.869%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T10:14:08.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:08.609 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:08.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:08.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:14:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:08 smithi196 ceph-mon[84784]: pgmap v669: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 63 MiB/s wr, 1.00k op/s; 1715/224316 objects degraded (0.765%); 4193/224316 objects misplaced (1.869%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T10:14:08.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:08.747 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:08.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:08.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:08.848 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:08.873 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:08.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:08.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:08.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:09.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:09.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:09.110 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:09.130 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:09.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:09.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:09.232 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:09.248 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 56 16 81408 81392 90.8237 75.75 0.00696593 0.0110051 2024-09-21T10:14:09.260 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:09.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:09.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:09.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:09.399 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:09.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:09.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:09.501 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:09.530 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:09.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:09.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:09.632 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:09.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:09 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 1715/236796 objects degraded (0.724%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:09.656 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:09.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:09.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:09.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:09.797 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:09.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:09.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:09.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:09 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 1715/236796 objects degraded (0.724%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:09.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:09 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 1715/236796 objects degraded (0.724%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:09.899 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:09.924 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:09.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:09.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:10.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:10.064 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:10.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:10.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:10.166 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:10.194 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:10.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:10.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:10.248 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 57 16 82640 82624 90.5811 77 0.00506196 0.0110344 2024-09-21T10:14:10.295 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:10.337 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:10.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:10.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:10.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:10.460 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:10.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:10.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:10.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:10.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:10.601 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:10.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:10.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:10 smithi196 ceph-mon[84784]: pgmap v670: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 78 MiB/s wr, 1.24k op/s; 1715/236796 objects degraded (0.724%); 4193/236796 objects misplaced (1.771%); 4.2 MiB/s, 67 objects/s recovering 2024-09-21T10:14:10.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:14:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:14:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:14:10.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:10 smithi098 ceph-mon[100914]: pgmap v670: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 78 MiB/s wr, 1.24k op/s; 1715/236796 objects degraded (0.724%); 4193/236796 objects misplaced (1.771%); 4.2 MiB/s, 67 objects/s recovering 2024-09-21T10:14:10.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:10 smithi098 ceph-mon[97580]: pgmap v670: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 78 MiB/s wr, 1.24k op/s; 1715/236796 objects degraded (0.724%); 4193/236796 objects misplaced (1.771%); 4.2 MiB/s, 67 objects/s recovering 2024-09-21T10:14:10.703 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:10.723 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:10.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:10.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:10.825 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:10.849 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:10.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:10.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:10.951 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:10.999 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:11.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:11.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:11.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:14:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:14:10] "GET /metrics HTTP/1.1" 200 36582 "" "Prometheus/2.51.0" 2024-09-21T10:14:11.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:11.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:11.145 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:11.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:11.246 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:11.248 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 58 16 83780 83764 90.2478 71.25 0.0130733 0.0110759 2024-09-21T10:14:11.273 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:11.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:11.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:11.375 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:11.400 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:11.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:11.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:11.502 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:11.527 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:11.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:11.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:11.629 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:11.654 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:11.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:11.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:11.755 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:11.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:11.788 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:11.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:11.888 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:11.914 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:11.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:11.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:12.015 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:12.040 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:12.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:12.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:12.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:12.184 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:12.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:12.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:12.248 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 59 16 85078 85062 90.093 81.125 0.0199836 0.0110945 2024-09-21T10:14:12.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:12.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:12.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:12.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:12.457 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:12.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:12.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:12.559 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:12.588 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:12.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:12.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:12.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:12 smithi196 ceph-mon[84784]: pgmap v671: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 1715/243057 objects degraded (0.706%); 3733/243057 objects misplaced (1.536%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:14:12.690 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:12.711 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:12.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:12.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:12.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:12.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:12 smithi098 ceph-mon[100914]: pgmap v671: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 1715/243057 objects degraded (0.706%); 3733/243057 objects misplaced (1.536%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:14:12.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:12 smithi098 ceph-mon[97580]: pgmap v671: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 1715/243057 objects degraded (0.706%); 3733/243057 objects misplaced (1.536%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:14:12.853 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:12.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:12.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:12.954 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:12.995 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:12.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:12.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:13.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:13.134 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:13.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:13.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:13.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:13.248 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:14:13.247296+0000 min lat: 0.00126097 max lat: 2.24757 avg lat: 0.0111401 2024-09-21T10:14:13.248 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:14:13.248 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 60 16 86153 86137 89.7112 67.1875 0.00451523 0.0111401 2024-09-21T10:14:13.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:13.261 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:13.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:13.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:13.401 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:13.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:13.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:13.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:13.542 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:13.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:13.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:13.644 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:13.673 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:13.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:13.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:13.774 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:13.814 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:13.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:13.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:13.916 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:13.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:13.946 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:13.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:14.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:14.087 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:14.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:14.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:14.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:14.225 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:14.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:14.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:14.248 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 61 16 87321 87305 89.4371 73 0.0064484 0.0111763 2024-09-21T10:14:14.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:14.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:14.363 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:14.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:14.464 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:14.504 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:14.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:14.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:14.605 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:14.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:14 smithi196 ceph-mon[84784]: pgmap v672: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 76 MiB/s wr, 1.22k op/s; 1715/250113 objects degraded (0.686%); 3733/250113 objects misplaced (1.493%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:14:14.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:14.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:14.745 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:14.790 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:14.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:14.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:14.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:14 smithi098 ceph-mon[100914]: pgmap v672: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 76 MiB/s wr, 1.22k op/s; 1715/250113 objects degraded (0.686%); 3733/250113 objects misplaced (1.493%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:14:14.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:14 smithi098 ceph-mon[97580]: pgmap v672: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 76 MiB/s wr, 1.22k op/s; 1715/250113 objects degraded (0.686%); 3733/250113 objects misplaced (1.493%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:14:14.892 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:14.956 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:14.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:14.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:15.058 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:15.082 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:15.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:15.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:15.184 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:15.221 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:15.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:15.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:15.248 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 62 16 88584 88568 89.2677 78.9375 0.00983599 0.0111973 2024-09-21T10:14:15.323 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:15.349 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:15.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:15.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:15.450 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:15.477 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:15.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:15.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:15.578 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:15.604 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:15.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:15.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:15.706 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:15.747 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:15.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:15.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:15.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:15 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 1715/258351 objects degraded (0.664%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:15.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:15 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 1715/258351 objects degraded (0.664%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:15.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:15.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:15 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 1715/258351 objects degraded (0.664%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:15.889 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:15.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:15.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:15.991 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:16.016 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:16.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:16.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:16.118 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:16.165 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:16.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:16.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:16.248 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 63 16 89873 89857 89.1294 80.5625 0.0134539 0.0112149 2024-09-21T10:14:16.267 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:16.309 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:16.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:16.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:16.411 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:16.435 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:16.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:16.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:16.536 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:16.561 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:16.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:16.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:16.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:16 smithi098 ceph-mon[100914]: pgmap v673: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 5.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 77 MiB/s wr, 1.24k op/s; 1715/258351 objects degraded (0.664%); 3199/258351 objects misplaced (1.238%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T10:14:16.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:16 smithi098 ceph-mon[97580]: pgmap v673: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 5.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 77 MiB/s wr, 1.24k op/s; 1715/258351 objects degraded (0.664%); 3199/258351 objects misplaced (1.238%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T10:14:16.663 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:16.706 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:16.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:16.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:16.807 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:16.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:16.854 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:16.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:16 smithi196 ceph-mon[84784]: pgmap v673: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 5.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 77 MiB/s wr, 1.24k op/s; 1715/258351 objects degraded (0.664%); 3199/258351 objects misplaced (1.238%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T10:14:16.948 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:14:16.956 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:17.032 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:17.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:17.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:17.134 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:17.157 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:17.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:17.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:17.249 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 64 16 91033 91017 88.8694 72.5 0.00742613 0.0112478 2024-09-21T10:14:17.258 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:17.282 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:17.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:17.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:17.384 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:17.408 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:17.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:17.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:17.502 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:14:17.510 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:17.533 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:17.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:17.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:17.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:17.651 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:14:17.651 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-21T10:14:17.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-21T10:14:17.651 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:14:17.652 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.652 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:267699 5.a 5:51b91a91:::benchmark_data_smithi098_135187_object132061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:14:17.652 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:59.409137+0000", 2024-09-21T10:14:17.652 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 378.23581214500001, 2024-09-21T10:14:17.652 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3530456719999999, 2024-09-21T10:14:17.652 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:14:17.652 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:14:17.652 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:17.652 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:14:17.652 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:14:17.652 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:14:17.652 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 267699 2024-09-21T10:14:17.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.653 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:14:17.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.653 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:14:17.653 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409137+0000", 2024-09-21T10:14:17.653 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.653 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:14:17.653 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409137+0000", 2024-09-21T10:14:17.653 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.653 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:14:17.654 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409138+0000", 2024-09-21T10:14:17.654 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.331e-06 2024-09-21T10:14:17.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.654 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:14:17.654 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409143+0000", 2024-09-21T10:14:17.654 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0309999999999998e-06 2024-09-21T10:14:17.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.654 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:14:17.654 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409160+0000", 2024-09-21T10:14:17.654 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6694e-05 2024-09-21T10:14:17.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.655 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:14:17.655 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409168+0000", 2024-09-21T10:14:17.655 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2230000000000007e-06 2024-09-21T10:14:17.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.655 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.655 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409241+0000", 2024-09-21T10:14:17.655 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2818999999999995e-05 2024-09-21T10:14:17.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.655 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:14:17.655 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.409253+0000", 2024-09-21T10:14:17.655 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1914e-05 2024-09-21T10:14:17.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.656 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.656 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.370041+0000", 2024-09-21T10:14:17.656 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.96078831799999997 2024-09-21T10:14:17.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.656 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:14:17.656 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.370053+0000", 2024-09-21T10:14:17.656 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2306999999999999e-05 2024-09-21T10:14:17.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.656 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.656 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.724460+0000", 2024-09-21T10:14:17.656 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3544063770000001 2024-09-21T10:14:17.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.657 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.657 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:14:17.657 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.724591+0000", 2024-09-21T10:14:17.657 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013132199999999999 2024-09-21T10:14:17.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.657 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.657 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3,4", 2024-09-21T10:14:17.657 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.724747+0000", 2024-09-21T10:14:17.657 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000156553 2024-09-21T10:14:17.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.657 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.657 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.657 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.742698+0000", 2024-09-21T10:14:17.657 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017950560000000001 2024-09-21T10:14:17.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.658 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:14:17.658 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.747832+0000", 2024-09-21T10:14:17.658 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051342310000000004 2024-09-21T10:14:17.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.658 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.658 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.755583+0000", 2024-09-21T10:14:17.658 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0077510879999999997 2024-09-21T10:14:17.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.658 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.658 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.761982+0000", 2024-09-21T10:14:17.658 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0063988789999999997 2024-09-21T10:14:17.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.659 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:14:17.659 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.762068+0000", 2024-09-21T10:14:17.659 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5698000000000005e-05 2024-09-21T10:14:17.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.659 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:14:17.659 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.762182+0000", 2024-09-21T10:14:17.659 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011432700000000001 2024-09-21T10:14:17.659 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.659 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:14:17.659 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.660 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269037 5.a 5:5ae457a8:::benchmark_data_smithi098_135187_object133399:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:14:17.660 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.480193+0000", 2024-09-21T10:14:17.660 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 373.1647557, 2024-09-21T10:14:17.660 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0309086750000001, 2024-09-21T10:14:17.660 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:14:17.660 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:14:17.660 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:17.660 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:14:17.660 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:14:17.660 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:14:17.660 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269037 2024-09-21T10:14:17.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.661 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:14:17.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.661 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:14:17.661 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480193+0000", 2024-09-21T10:14:17.661 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.661 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:14:17.661 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480193+0000", 2024-09-21T10:14:17.661 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.661 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:14:17.661 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480195+0000", 2024-09-21T10:14:17.661 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6479999999999999e-06 2024-09-21T10:14:17.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.662 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:14:17.662 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480200+0000", 2024-09-21T10:14:17.662 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7420000000000003e-06 2024-09-21T10:14:17.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.662 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:14:17.662 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480215+0000", 2024-09-21T10:14:17.662 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4293e-05 2024-09-21T10:14:17.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.662 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:14:17.662 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480222+0000", 2024-09-21T10:14:17.662 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3640000000000001e-06 2024-09-21T10:14:17.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.663 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.663 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480271+0000", 2024-09-21T10:14:17.663 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8466e-05 2024-09-21T10:14:17.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.663 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:14:17.663 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.480276+0000", 2024-09-21T10:14:17.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9240000000000004e-06 2024-09-21T10:14:17.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.490543+0000", 2024-09-21T10:14:17.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0102668859999999 2024-09-21T10:14:17.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.666 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:14:17.666 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.490646+0000", 2024-09-21T10:14:17.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010246099999999999 2024-09-21T10:14:17.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.666 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:14:17.666 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.490749+0000", 2024-09-21T10:14:17.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000102839 2024-09-21T10:14:17.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.666 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.667 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510822+0000", 2024-09-21T10:14:17.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020072995999999999 2024-09-21T10:14:17.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:14:17.667 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510836+0000", 2024-09-21T10:14:17.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4445e-05 2024-09-21T10:14:17.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.667 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510993+0000", 2024-09-21T10:14:17.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000157207 2024-09-21T10:14:17.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.668 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:14:17.668 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511062+0000", 2024-09-21T10:14:17.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8923000000000004e-05 2024-09-21T10:14:17.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.668 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:14:17.668 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.511102+0000", 2024-09-21T10:14:17.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9480999999999997e-05 2024-09-21T10:14:17.668 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.668 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:14:17.668 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.668 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269043 5.a 5:5d7883f5:::benchmark_data_smithi098_135187_object133405:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:14:17.669 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.487457+0000", 2024-09-21T10:14:17.669 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 373.15749122400001, 2024-09-21T10:14:17.669 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.040401159, 2024-09-21T10:14:17.669 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:14:17.669 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:14:17.669 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:17.669 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:14:17.669 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:14:17.669 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:14:17.669 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269043 2024-09-21T10:14:17.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.669 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:14:17.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:14:17.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487457+0000", 2024-09-21T10:14:17.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:14:17.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487457+0000", 2024-09-21T10:14:17.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:14:17.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487458+0000", 2024-09-21T10:14:17.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8599999999999996e-07 2024-09-21T10:14:17.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.671 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:14:17.671 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487463+0000", 2024-09-21T10:14:17.671 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2830000000000003e-06 2024-09-21T10:14:17.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.671 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:14:17.671 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487476+0000", 2024-09-21T10:14:17.671 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2819e-05 2024-09-21T10:14:17.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.671 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:14:17.671 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487481+0000", 2024-09-21T10:14:17.671 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2519999999999999e-06 2024-09-21T10:14:17.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.672 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487524+0000", 2024-09-21T10:14:17.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2701000000000003e-05 2024-09-21T10:14:17.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:14:17.672 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.487529+0000", 2024-09-21T10:14:17.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0690000000000001e-06 2024-09-21T10:14:17.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.672 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.500694+0000", 2024-09-21T10:14:17.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0131655589999999 2024-09-21T10:14:17.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.673 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:14:17.673 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.500787+0000", 2024-09-21T10:14:17.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2785e-05 2024-09-21T10:14:17.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.673 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:14:17.673 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.500887+0000", 2024-09-21T10:14:17.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9599000000000006e-05 2024-09-21T10:14:17.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.673 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:14:17.674 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.510920+0000", 2024-09-21T10:14:17.674 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010033686999999999 2024-09-21T10:14:17.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.674 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.674 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.517569+0000", 2024-09-21T10:14:17.674 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0066490730000000001 2024-09-21T10:14:17.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.674 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.674 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.527765+0000", 2024-09-21T10:14:17.674 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010196197000000001 2024-09-21T10:14:17.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.675 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:14:17.675 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.527818+0000", 2024-09-21T10:14:17.675 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2401000000000002e-05 2024-09-21T10:14:17.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.675 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:14:17.675 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.527859+0000", 2024-09-21T10:14:17.675 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0747999999999997e-05 2024-09-21T10:14:17.675 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.675 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:14:17.675 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.675 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269059 5.a 5:530cb704:::benchmark_data_smithi098_135187_object133421:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:14:17.676 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.508524+0000", 2024-09-21T10:14:17.676 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 373.13642422800001, 2024-09-21T10:14:17.676 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0242946340000001, 2024-09-21T10:14:17.676 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:14:17.676 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:14:17.676 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:17.676 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:14:17.676 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:14:17.676 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:14:17.676 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269059 2024-09-21T10:14:17.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.676 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:14:17.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.676 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:14:17.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508524+0000", 2024-09-21T10:14:17.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:14:17.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508524+0000", 2024-09-21T10:14:17.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:14:17.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508526+0000", 2024-09-21T10:14:17.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.387e-06 2024-09-21T10:14:17.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:14:17.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508530+0000", 2024-09-21T10:14:17.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.437e-06 2024-09-21T10:14:17.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:14:17.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508542+0000", 2024-09-21T10:14:17.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1249e-05 2024-09-21T10:14:17.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:14:17.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508547+0000", 2024-09-21T10:14:17.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5809999999999996e-06 2024-09-21T10:14:17.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508596+0000", 2024-09-21T10:14:17.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8641999999999999e-05 2024-09-21T10:14:17.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:14:17.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.508602+0000", 2024-09-21T10:14:17.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8100000000000003e-06 2024-09-21T10:14:17.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.505838+0000", 2024-09-21T10:14:17.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.99723621699999998 2024-09-21T10:14:17.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:14:17.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.505943+0000", 2024-09-21T10:14:17.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010536500000000001 2024-09-21T10:14:17.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.680 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:14:17.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.506048+0000", 2024-09-21T10:14:17.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105221 2024-09-21T10:14:17.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.680 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:14:17.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.522571+0000", 2024-09-21T10:14:17.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016522236999999999 2024-09-21T10:14:17.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.523904+0000", 2024-09-21T10:14:17.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001332986 2024-09-21T10:14:17.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.682 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.532731+0000", 2024-09-21T10:14:17.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0088274440000000003 2024-09-21T10:14:17.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.682 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:14:17.682 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.532778+0000", 2024-09-21T10:14:17.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7413000000000002e-05 2024-09-21T10:14:17.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.682 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:14:17.682 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.532819+0000", 2024-09-21T10:14:17.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0645000000000001e-05 2024-09-21T10:14:17.682 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.682 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:14:17.683 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.683 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56334 7.13 7:cac58bb8:::benchmark_data_smithi098_197834_object56333:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e218)", 2024-09-21T10:14:17.683 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:48.681554+0000", 2024-09-21T10:14:17.683 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.963394373, 2024-09-21T10:14:17.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2618041179999999, 2024-09-21T10:14:17.683 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:14:17.683 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:14:17.683 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:17.683 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:14:17.683 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:14:17.683 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:14:17.683 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56334 2024-09-21T10:14:17.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.684 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:14:17.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:14:17.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681554+0000", 2024-09-21T10:14:17.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:14:17.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681554+0000", 2024-09-21T10:14:17.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:14:17.685 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681556+0000", 2024-09-21T10:14:17.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7990000000000001e-06 2024-09-21T10:14:17.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:14:17.685 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681588+0000", 2024-09-21T10:14:17.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2166999999999999e-05 2024-09-21T10:14:17.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:14:17.685 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681605+0000", 2024-09-21T10:14:17.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6694e-05 2024-09-21T10:14:17.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:14:17.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681614+0000", 2024-09-21T10:14:17.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2879999999999998e-06 2024-09-21T10:14:17.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681675+0000", 2024-09-21T10:14:17.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1198999999999995e-05 2024-09-21T10:14:17.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-09-21T10:14:17.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681682+0000", 2024-09-21T10:14:17.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7859999999999999e-06 2024-09-21T10:14:17.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683072+0000", 2024-09-21T10:14:17.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001389567 2024-09-21T10:14:17.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:14:17.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683081+0000", 2024-09-21T10:14:17.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2930000000000006e-06 2024-09-21T10:14:17.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.935461+0000", 2024-09-21T10:14:17.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2523800389999999 2024-09-21T10:14:17.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:14:17.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.935598+0000", 2024-09-21T10:14:17.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000136778 2024-09-21T10:14:17.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:14:17.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.935714+0000", 2024-09-21T10:14:17.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000115678 2024-09-21T10:14:17.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.939763+0000", 2024-09-21T10:14:17.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004049024 2024-09-21T10:14:17.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:14:17.689 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.939785+0000", 2024-09-21T10:14:17.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2651999999999999e-05 2024-09-21T10:14:17.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.689 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.942239+0000", 2024-09-21T10:14:17.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002454164 2024-09-21T10:14:17.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:14:17.689 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.942298+0000", 2024-09-21T10:14:17.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8687999999999998e-05 2024-09-21T10:14:17.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:14:17.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.943358+0000", 2024-09-21T10:14:17.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010603019999999999 2024-09-21T10:14:17.690 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.690 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:14:17.690 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.690 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56347 7.13 7:ce372e59:::benchmark_data_smithi098_197834_object56346:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e218)", 2024-09-21T10:14:17.690 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:48.681718+0000", 2024-09-21T10:14:17.690 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.963230710000001, 2024-09-21T10:14:17.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.262767231, 2024-09-21T10:14:17.691 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:14:17.691 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:14:17.691 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:17.691 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:14:17.691 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:14:17.691 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:14:17.691 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56347 2024-09-21T10:14:17.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.691 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:14:17.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:14:17.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681718+0000", 2024-09-21T10:14:17.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:14:17.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681718+0000", 2024-09-21T10:14:17.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:14:17.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681719+0000", 2024-09-21T10:14:17.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7100000000000001e-07 2024-09-21T10:14:17.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:14:17.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681749+0000", 2024-09-21T10:14:17.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0580999999999997e-05 2024-09-21T10:14:17.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.693 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:14:17.693 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681761+0000", 2024-09-21T10:14:17.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1426999999999999e-05 2024-09-21T10:14:17.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.693 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:14:17.693 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681767+0000", 2024-09-21T10:14:17.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4450000000000001e-06 2024-09-21T10:14:17.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.693 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.693 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681824+0000", 2024-09-21T10:14:17.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6489000000000001e-05 2024-09-21T10:14:17.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.694 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting_for_map not empty", 2024-09-21T10:14:17.694 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681829+0000", 2024-09-21T10:14:17.694 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6679999999999998e-06 2024-09-21T10:14:17.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.694 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.694 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683116+0000", 2024-09-21T10:14:17.694 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00128629 2024-09-21T10:14:17.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.694 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:14:17.694 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683124+0000", 2024-09-21T10:14:17.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2039999999999997e-06 2024-09-21T10:14:17.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.695 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.695 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.937306+0000", 2024-09-21T10:14:17.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.254182251 2024-09-21T10:14:17.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.695 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:14:17.695 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.937418+0000", 2024-09-21T10:14:17.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011215899999999999 2024-09-21T10:14:17.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.695 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:14:17.695 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.937577+0000", 2024-09-21T10:14:17.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000158641 2024-09-21T10:14:17.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.696 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:14:17.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.941237+0000", 2024-09-21T10:14:17.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036600209999999998 2024-09-21T10:14:17.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.696 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.943433+0000", 2024-09-21T10:14:17.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0021958120000000001 2024-09-21T10:14:17.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.696 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.943588+0000", 2024-09-21T10:14:17.697 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000155463 2024-09-21T10:14:17.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.697 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:14:17.697 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.943645+0000", 2024-09-21T10:14:17.697 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6925999999999997e-05 2024-09-21T10:14:17.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.697 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:14:17.697 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.944485+0000", 2024-09-21T10:14:17.697 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00084008300000000004 2024-09-21T10:14:17.697 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.697 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:14:17.697 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.698 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56354 7.13 7:cc8e7d2b:::benchmark_data_smithi098_197834_object56353:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e218)", 2024-09-21T10:14:17.698 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:48.681842+0000", 2024-09-21T10:14:17.698 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.963106705000001, 2024-09-21T10:14:17.698 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.265416978, 2024-09-21T10:14:17.698 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:14:17.698 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:14:17.698 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:17.698 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:14:17.698 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:14:17.698 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:14:17.698 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56354 2024-09-21T10:14:17.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.699 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:14:17.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:14:17.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681842+0000", 2024-09-21T10:14:17.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:14:17.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681842+0000", 2024-09-21T10:14:17.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:14:17.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681843+0000", 2024-09-21T10:14:17.700 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9599999999999999e-07 2024-09-21T10:14:17.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.700 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:14:17.700 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681871+0000", 2024-09-21T10:14:17.700 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8017999999999998e-05 2024-09-21T10:14:17.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.700 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:14:17.700 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681882+0000", 2024-09-21T10:14:17.700 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1241e-05 2024-09-21T10:14:17.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.700 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:14:17.700 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681888+0000", 2024-09-21T10:14:17.701 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2179999999999996e-06 2024-09-21T10:14:17.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.701 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.701 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681940+0000", 2024-09-21T10:14:17.701 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1431000000000003e-05 2024-09-21T10:14:17.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.701 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting_for_map not empty", 2024-09-21T10:14:17.701 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681945+0000", 2024-09-21T10:14:17.701 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1930000000000001e-06 2024-09-21T10:14:17.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.702 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.702 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683155+0000", 2024-09-21T10:14:17.702 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001209966 2024-09-21T10:14:17.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.702 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:14:17.702 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683162+0000", 2024-09-21T10:14:17.702 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5020000000000003e-06 2024-09-21T10:14:17.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.702 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.702 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.938481+0000", 2024-09-21T10:14:17.702 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2553183290000001 2024-09-21T10:14:17.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:14:17.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.938616+0000", 2024-09-21T10:14:17.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013564800000000001 2024-09-21T10:14:17.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:14:17.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.938767+0000", 2024-09-21T10:14:17.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000150307 2024-09-21T10:14:17.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:14:17.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.943375+0000", 2024-09-21T10:14:17.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046085479999999996 2024-09-21T10:14:17.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.704 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.704 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.943494+0000", 2024-09-21T10:14:17.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000119397 2024-09-21T10:14:17.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.704 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.704 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.946147+0000", 2024-09-21T10:14:17.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026526950000000001 2024-09-21T10:14:17.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.704 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:14:17.704 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.946206+0000", 2024-09-21T10:14:17.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9021999999999998e-05 2024-09-21T10:14:17.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.705 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:14:17.705 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.947259+0000", 2024-09-21T10:14:17.705 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001052767 2024-09-21T10:14:17.705 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.705 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:14:17.705 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.705 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56355 7.13 7:cd0285fd:::benchmark_data_smithi098_197834_object56354:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e218)", 2024-09-21T10:14:17.705 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:48.681958+0000", 2024-09-21T10:14:17.705 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.962990816000001, 2024-09-21T10:14:17.705 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2663260160000001, 2024-09-21T10:14:17.705 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:14:17.706 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:14:17.706 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:17.706 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:14:17.706 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:14:17.706 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:14:17.706 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56355 2024-09-21T10:14:17.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.706 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:14:17.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.706 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:14:17.706 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681958+0000", 2024-09-21T10:14:17.706 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.707 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:14:17.707 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681958+0000", 2024-09-21T10:14:17.707 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.707 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:14:17.707 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681958+0000", 2024-09-21T10:14:17.707 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9599999999999999e-07 2024-09-21T10:14:17.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.707 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:14:17.707 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681988+0000", 2024-09-21T10:14:17.707 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9907999999999999e-05 2024-09-21T10:14:17.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.708 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:14:17.708 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.681997+0000", 2024-09-21T10:14:17.708 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8410000000000006e-06 2024-09-21T10:14:17.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.708 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:14:17.708 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682003+0000", 2024-09-21T10:14:17.708 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8810000000000001e-06 2024-09-21T10:14:17.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.708 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.708 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682054+0000", 2024-09-21T10:14:17.708 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0707000000000001e-05 2024-09-21T10:14:17.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.709 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting_for_map not empty", 2024-09-21T10:14:17.709 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682059+0000", 2024-09-21T10:14:17.709 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0200000000000002e-06 2024-09-21T10:14:17.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.709 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.709 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683192+0000", 2024-09-21T10:14:17.709 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001133408 2024-09-21T10:14:17.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.709 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:14:17.709 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683198+0000", 2024-09-21T10:14:17.710 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0499999999999997e-06 2024-09-21T10:14:17.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.710 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.710 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.939924+0000", 2024-09-21T10:14:17.710 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2567260769999999 2024-09-21T10:14:17.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.710 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:14:17.710 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.940082+0000", 2024-09-21T10:14:17.710 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000157295 2024-09-21T10:14:17.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.711 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:14:17.711 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.940267+0000", 2024-09-21T10:14:17.711 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018506999999999999 2024-09-21T10:14:17.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.711 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:14:17.711 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.943520+0000", 2024-09-21T10:14:17.711 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032536069999999999 2024-09-21T10:14:17.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.711 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.711 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.946055+0000", 2024-09-21T10:14:17.711 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002534204 2024-09-21T10:14:17.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.712 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.712 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.947333+0000", 2024-09-21T10:14:17.712 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0012787359999999999 2024-09-21T10:14:17.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.712 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:14:17.712 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.947396+0000", 2024-09-21T10:14:17.712 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3119000000000003e-05 2024-09-21T10:14:17.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.712 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:14:17.712 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.948284+0000", 2024-09-21T10:14:17.712 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000887497 2024-09-21T10:14:17.713 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.713 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:14:17.713 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.713 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56375 7.13 7:cb435e0c:::benchmark_data_smithi098_197834_object56374:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e218)", 2024-09-21T10:14:17.713 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:48.682071+0000", 2024-09-21T10:14:17.713 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.962877258999999, 2024-09-21T10:14:17.713 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.266988931, 2024-09-21T10:14:17.713 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:14:17.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:14:17.713 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:17.713 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:14:17.713 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:14:17.714 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:14:17.714 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56375 2024-09-21T10:14:17.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.714 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:14:17.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.714 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:14:17.714 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682071+0000", 2024-09-21T10:14:17.714 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.714 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:14:17.714 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682071+0000", 2024-09-21T10:14:17.714 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.715 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:14:17.715 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682072+0000", 2024-09-21T10:14:17.715 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2499999999999995e-07 2024-09-21T10:14:17.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.715 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:14:17.715 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682092+0000", 2024-09-21T10:14:17.715 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0145000000000001e-05 2024-09-21T10:14:17.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.715 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:14:17.715 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682100+0000", 2024-09-21T10:14:17.715 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4250000000000001e-06 2024-09-21T10:14:17.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.716 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:14:17.716 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682115+0000", 2024-09-21T10:14:17.716 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5089000000000001e-05 2024-09-21T10:14:17.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.716 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.716 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682164+0000", 2024-09-21T10:14:17.716 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9191999999999999e-05 2024-09-21T10:14:17.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.716 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting_for_map not empty", 2024-09-21T10:14:17.716 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682170+0000", 2024-09-21T10:14:17.716 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1580000000000002e-06 2024-09-21T10:14:17.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.717 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.717 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683225+0000", 2024-09-21T10:14:17.717 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010546990000000001 2024-09-21T10:14:17.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.717 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:14:17.717 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683232+0000", 2024-09-21T10:14:17.717 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1609999999999997e-06 2024-09-21T10:14:17.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.717 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.718 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.941279+0000", 2024-09-21T10:14:17.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.258047124 2024-09-21T10:14:17.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.718 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:14:17.718 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.941381+0000", 2024-09-21T10:14:17.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000102142 2024-09-21T10:14:17.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.718 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:14:17.718 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.941485+0000", 2024-09-21T10:14:17.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000104034 2024-09-21T10:14:17.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.718 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:14:17.719 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.943528+0000", 2024-09-21T10:14:17.719 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002042964 2024-09-21T10:14:17.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.719 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.719 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.946101+0000", 2024-09-21T10:14:17.719 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025732390000000002 2024-09-21T10:14:17.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.719 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.719 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.948334+0000", 2024-09-21T10:14:17.719 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002232894 2024-09-21T10:14:17.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.720 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:14:17.720 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.948383+0000", 2024-09-21T10:14:17.720 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8273999999999999e-05 2024-09-21T10:14:17.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.720 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:14:17.720 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.949060+0000", 2024-09-21T10:14:17.720 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00067786599999999995 2024-09-21T10:14:17.720 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.720 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:14:17.720 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.721 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56405 7.13 7:c9883c86:::benchmark_data_smithi098_197834_object56404:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e218)", 2024-09-21T10:14:17.721 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:48.682135+0000", 2024-09-21T10:14:17.721 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.962813601000001, 2024-09-21T10:14:17.721 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.268181011, 2024-09-21T10:14:17.722 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:14:17.722 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:14:17.722 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:17.722 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:14:17.722 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:14:17.722 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:14:17.722 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56405 2024-09-21T10:14:17.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.722 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:14:17.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.722 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:14:17.722 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682135+0000", 2024-09-21T10:14:17.722 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.723 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:14:17.723 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682135+0000", 2024-09-21T10:14:17.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.723 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:14:17.723 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682136+0000", 2024-09-21T10:14:17.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.89e-07 2024-09-21T10:14:17.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.723 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:14:17.723 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682155+0000", 2024-09-21T10:14:17.724 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9278000000000001e-05 2024-09-21T10:14:17.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.724 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:14:17.724 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682163+0000", 2024-09-21T10:14:17.724 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2339999999999994e-06 2024-09-21T10:14:17.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.724 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:14:17.724 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682169+0000", 2024-09-21T10:14:17.724 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6269999999999997e-06 2024-09-21T10:14:17.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.724 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.724 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682205+0000", 2024-09-21T10:14:17.725 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5988999999999997e-05 2024-09-21T10:14:17.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.725 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting_for_map not empty", 2024-09-21T10:14:17.725 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682210+0000", 2024-09-21T10:14:17.725 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2490000000000001e-06 2024-09-21T10:14:17.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.725 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.725 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683271+0000", 2024-09-21T10:14:17.725 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001060688 2024-09-21T10:14:17.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.725 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:14:17.726 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683280+0000", 2024-09-21T10:14:17.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0229999999999998e-06 2024-09-21T10:14:17.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.726 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.726 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.944543+0000", 2024-09-21T10:14:17.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.261263668 2024-09-21T10:14:17.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.726 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:14:17.726 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.944694+0000", 2024-09-21T10:14:17.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015107900000000001 2024-09-21T10:14:17.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:14:17.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.944977+0000", 2024-09-21T10:14:17.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000282105 2024-09-21T10:14:17.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:14:17.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.949073+0000", 2024-09-21T10:14:17.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040965130000000004 2024-09-21T10:14:17.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.949178+0000", 2024-09-21T10:14:17.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000104793 2024-09-21T10:14:17.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.949229+0000", 2024-09-21T10:14:17.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1238e-05 2024-09-21T10:14:17.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:14:17.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.949298+0000", 2024-09-21T10:14:17.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8454999999999994e-05 2024-09-21T10:14:17.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:14:17.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.950316+0000", 2024-09-21T10:14:17.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001018583 2024-09-21T10:14:17.729 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.729 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:14:17.729 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.729 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56409 7.13 7:cc89f9a0:::benchmark_data_smithi098_197834_object56408:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e218)", 2024-09-21T10:14:17.729 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:48.682182+0000", 2024-09-21T10:14:17.729 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.962767024000001, 2024-09-21T10:14:17.729 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.279406431, 2024-09-21T10:14:17.729 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:14:17.729 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:14:17.729 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:17.729 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:14:17.729 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:14:17.730 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:14:17.730 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56409 2024-09-21T10:14:17.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.730 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:14:17.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:14:17.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682182+0000", 2024-09-21T10:14:17.730 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:14:17.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682182+0000", 2024-09-21T10:14:17.730 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.731 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:14:17.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682182+0000", 2024-09-21T10:14:17.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3099999999999998e-07 2024-09-21T10:14:17.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.731 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:14:17.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682260+0000", 2024-09-21T10:14:17.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7389000000000001e-05 2024-09-21T10:14:17.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.731 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:14:17.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682267+0000", 2024-09-21T10:14:17.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8930000000000005e-06 2024-09-21T10:14:17.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:14:17.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682273+0000", 2024-09-21T10:14:17.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7370000000000003e-06 2024-09-21T10:14:17.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683359+0000", 2024-09-21T10:14:17.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010854860000000001 2024-09-21T10:14:17.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:14:17.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683365+0000", 2024-09-21T10:14:17.733 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.156e-06 2024-09-21T10:14:17.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.733 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.733 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.950447+0000", 2024-09-21T10:14:17.733 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.267082458 2024-09-21T10:14:17.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.733 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:14:17.733 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.950587+0000", 2024-09-21T10:14:17.733 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013990000000000001 2024-09-21T10:14:17.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.733 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:14:17.734 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.950733+0000", 2024-09-21T10:14:17.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000146005 2024-09-21T10:14:17.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.734 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.734 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.959652+0000", 2024-09-21T10:14:17.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0089191010000000005 2024-09-21T10:14:17.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.734 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:14:17.734 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.959667+0000", 2024-09-21T10:14:17.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4766000000000001e-05 2024-09-21T10:14:17.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.959869+0000", 2024-09-21T10:14:17.735 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000201511 2024-09-21T10:14:17.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:14:17.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.959929+0000", 2024-09-21T10:14:17.735 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0687999999999999e-05 2024-09-21T10:14:17.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:14:17.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.961588+0000", 2024-09-21T10:14:17.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00165891 2024-09-21T10:14:17.736 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.736 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:14:17.736 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.736 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56419 7.13 7:cabce2af:::benchmark_data_smithi098_197834_object56418:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e218)", 2024-09-21T10:14:17.736 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:48.682281+0000", 2024-09-21T10:14:17.736 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.962667843999998, 2024-09-21T10:14:17.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.292392166, 2024-09-21T10:14:17.736 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:14:17.736 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:14:17.736 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:17.736 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:14:17.737 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:14:17.737 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:14:17.737 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56419 2024-09-21T10:14:17.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.737 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:14:17.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:14:17.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682281+0000", 2024-09-21T10:14:17.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:14:17.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682281+0000", 2024-09-21T10:14:17.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.738 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:14:17.738 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682281+0000", 2024-09-21T10:14:17.738 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4099999999999999e-07 2024-09-21T10:14:17.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.738 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:14:17.738 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682330+0000", 2024-09-21T10:14:17.738 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9026000000000001e-05 2024-09-21T10:14:17.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.738 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:14:17.738 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682338+0000", 2024-09-21T10:14:17.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.272e-06 2024-09-21T10:14:17.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.739 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:14:17.739 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682343+0000", 2024-09-21T10:14:17.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7729999999999996e-06 2024-09-21T10:14:17.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.739 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.739 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683391+0000", 2024-09-21T10:14:17.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010477150000000001 2024-09-21T10:14:17.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:14:17.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683397+0000", 2024-09-21T10:14:17.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2040000000000002e-06 2024-09-21T10:14:17.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.951845+0000", 2024-09-21T10:14:17.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2684472339999999 2024-09-21T10:14:17.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:14:17.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.951968+0000", 2024-09-21T10:14:17.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012336000000000001 2024-09-21T10:14:17.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:14:17.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.952124+0000", 2024-09-21T10:14:17.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015602899999999999 2024-09-21T10:14:17.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:14:17.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.959761+0000", 2024-09-21T10:14:17.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0076368479999999999 2024-09-21T10:14:17.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.959821+0000", 2024-09-21T10:14:17.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0112999999999999e-05 2024-09-21T10:14:17.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.742 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.742 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.973225+0000", 2024-09-21T10:14:17.742 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013403651000000001 2024-09-21T10:14:17.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.742 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:14:17.742 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.973286+0000", 2024-09-21T10:14:17.742 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1798999999999996e-05 2024-09-21T10:14:17.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.742 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:14:17.743 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.974673+0000", 2024-09-21T10:14:17.743 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013867009999999999 2024-09-21T10:14:17.743 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.743 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:14:17.743 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.743 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56429 7.13 7:cf855ba8:::benchmark_data_smithi098_197834_object56428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e218)", 2024-09-21T10:14:17.743 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:48.682351+0000", 2024-09-21T10:14:17.743 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.962597563999999, 2024-09-21T10:14:17.743 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2945721479999999, 2024-09-21T10:14:17.743 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:14:17.743 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:14:17.743 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:17.744 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:14:17.744 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:14:17.744 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:14:17.744 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56429 2024-09-21T10:14:17.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.744 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:14:17.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:14:17.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682351+0000", 2024-09-21T10:14:17.744 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:14:17.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682351+0000", 2024-09-21T10:14:17.744 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:14:17.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682352+0000", 2024-09-21T10:14:17.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2e-07 2024-09-21T10:14:17.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:14:17.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682394+0000", 2024-09-21T10:14:17.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2252999999999997e-05 2024-09-21T10:14:17.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:14:17.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682401+0000", 2024-09-21T10:14:17.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0829999999999998e-06 2024-09-21T10:14:17.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:14:17.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682406+0000", 2024-09-21T10:14:17.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4809999999999997e-06 2024-09-21T10:14:17.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683423+0000", 2024-09-21T10:14:17.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010165490000000001 2024-09-21T10:14:17.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.747 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:14:17.747 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683429+0000", 2024-09-21T10:14:17.747 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7139999999999998e-06 2024-09-21T10:14:17.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.747 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.747 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.953313+0000", 2024-09-21T10:14:17.747 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2698842290000001 2024-09-21T10:14:17.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.747 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:14:17.747 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.953492+0000", 2024-09-21T10:14:17.747 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017870700000000001 2024-09-21T10:14:17.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.748 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:14:17.748 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.953655+0000", 2024-09-21T10:14:17.748 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016300200000000001 2024-09-21T10:14:17.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.748 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:14:17.748 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.959767+0000", 2024-09-21T10:14:17.748 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061125650000000004 2024-09-21T10:14:17.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.748 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.748 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.964631+0000", 2024-09-21T10:14:17.748 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048637769999999997 2024-09-21T10:14:17.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.749 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.749 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.975076+0000", 2024-09-21T10:14:17.749 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010445481 2024-09-21T10:14:17.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.749 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:14:17.749 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.975153+0000", 2024-09-21T10:14:17.749 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6557999999999994e-05 2024-09-21T10:14:17.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.749 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:14:17.749 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.976923+0000", 2024-09-21T10:14:17.749 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001770329 2024-09-21T10:14:17.750 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.750 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:14:17.750 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.750 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56459 7.13 7:cf31230a:::benchmark_data_smithi098_197834_object56458:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e218)", 2024-09-21T10:14:17.750 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:48.682414+0000", 2024-09-21T10:14:17.750 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.962534634000001, 2024-09-21T10:14:17.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2957976259999999, 2024-09-21T10:14:17.750 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:14:17.750 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:14:17.750 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:17.750 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:14:17.750 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:14:17.751 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:14:17.751 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56459 2024-09-21T10:14:17.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.751 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:14:17.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.751 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:14:17.751 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682414+0000", 2024-09-21T10:14:17.751 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.751 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:14:17.751 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682414+0000", 2024-09-21T10:14:17.751 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.752 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:14:17.752 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682414+0000", 2024-09-21T10:14:17.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2300000000000002e-07 2024-09-21T10:14:17.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.752 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:14:17.752 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682456+0000", 2024-09-21T10:14:17.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1467999999999997e-05 2024-09-21T10:14:17.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.752 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:14:17.752 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682463+0000", 2024-09-21T10:14:17.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9160000000000004e-06 2024-09-21T10:14:17.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:14:17.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682468+0000", 2024-09-21T10:14:17.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3149999999999999e-06 2024-09-21T10:14:17.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683455+0000", 2024-09-21T10:14:17.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00098726999999999994 2024-09-21T10:14:17.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:14:17.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683462+0000", 2024-09-21T10:14:17.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2140000000000001e-06 2024-09-21T10:14:17.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.955008+0000", 2024-09-21T10:14:17.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2715465180000001 2024-09-21T10:14:17.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:14:17.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.955202+0000", 2024-09-21T10:14:17.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000194253 2024-09-21T10:14:17.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.755 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:14:17.755 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.955468+0000", 2024-09-21T10:14:17.755 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00026577300000000002 2024-09-21T10:14:17.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.755 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:14:17.755 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.959773+0000", 2024-09-21T10:14:17.755 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043049899999999999 2024-09-21T10:14:17.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.755 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.755 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.964683+0000", 2024-09-21T10:14:17.755 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0049093870000000003 2024-09-21T10:14:17.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.756 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.976990+0000", 2024-09-21T10:14:17.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012307201 2024-09-21T10:14:17.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:14:17.756 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.977056+0000", 2024-09-21T10:14:17.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6557999999999995e-05 2024-09-21T10:14:17.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:14:17.756 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.978212+0000", 2024-09-21T10:14:17.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011553399999999999 2024-09-21T10:14:17.756 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.757 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:14:17.757 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.757 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56467 7.13 7:c9308525:::benchmark_data_smithi098_197834_object56466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e218)", 2024-09-21T10:14:17.757 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:48.682476+0000", 2024-09-21T10:14:17.757 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.962473154000001, 2024-09-21T10:14:17.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.296674023, 2024-09-21T10:14:17.757 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:14:17.757 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:14:17.757 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:17.757 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:14:17.757 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:14:17.758 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:14:17.758 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56467 2024-09-21T10:14:17.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.758 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:14:17.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:14:17.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682476+0000", 2024-09-21T10:14:17.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:14:17.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682476+0000", 2024-09-21T10:14:17.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:14:17.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682476+0000", 2024-09-21T10:14:17.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3700000000000001e-07 2024-09-21T10:14:17.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:14:17.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682519+0000", 2024-09-21T10:14:17.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3284000000000001e-05 2024-09-21T10:14:17.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:14:17.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682526+0000", 2024-09-21T10:14:17.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3090000000000001e-06 2024-09-21T10:14:17.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:14:17.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682530+0000", 2024-09-21T10:14:17.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7720000000000002e-06 2024-09-21T10:14:17.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683484+0000", 2024-09-21T10:14:17.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00095406899999999999 2024-09-21T10:14:17.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:14:17.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683490+0000", 2024-09-21T10:14:17.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9970000000000004e-06 2024-09-21T10:14:17.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.761 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.761 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.956657+0000", 2024-09-21T10:14:17.761 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.27316618 2024-09-21T10:14:17.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.761 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:14:17.761 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.956815+0000", 2024-09-21T10:14:17.761 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015856900000000001 2024-09-21T10:14:17.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.761 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:14:17.761 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.956982+0000", 2024-09-21T10:14:17.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:17.763 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:17.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:17.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016681700000000001 2024-09-21T10:14:17.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:14:17.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.959780+0000", 2024-09-21T10:14:17.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027979950000000002 2024-09-21T10:14:17.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.964729+0000", 2024-09-21T10:14:17.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0049487419999999999 2024-09-21T10:14:17.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.978270+0000", 2024-09-21T10:14:17.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013541661999999999 2024-09-21T10:14:17.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:14:17.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.978330+0000", 2024-09-21T10:14:17.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0087000000000003e-05 2024-09-21T10:14:17.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:14:17.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.979150+0000", 2024-09-21T10:14:17.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00081910299999999995 2024-09-21T10:14:17.765 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.765 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:14:17.765 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.766 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56496 7.13 7:cca029f7:::benchmark_data_smithi098_197834_object56495:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e218)", 2024-09-21T10:14:17.766 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:48.682538+0000", 2024-09-21T10:14:17.766 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.962410744, 2024-09-21T10:14:17.766 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2974343159999999, 2024-09-21T10:14:17.766 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:14:17.766 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:14:17.766 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:17.766 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:14:17.766 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:14:17.766 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:14:17.766 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56496 2024-09-21T10:14:17.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.767 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:14:17.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.767 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:14:17.767 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682538+0000", 2024-09-21T10:14:17.767 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.767 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:14:17.767 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682538+0000", 2024-09-21T10:14:17.767 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.767 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:14:17.767 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682538+0000", 2024-09-21T10:14:17.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8800000000000003e-07 2024-09-21T10:14:17.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:14:17.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682580+0000", 2024-09-21T10:14:17.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1805999999999999e-05 2024-09-21T10:14:17.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:14:17.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682654+0000", 2024-09-21T10:14:17.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3535999999999995e-05 2024-09-21T10:14:17.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:14:17.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682660+0000", 2024-09-21T10:14:17.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2090000000000002e-06 2024-09-21T10:14:17.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683551+0000", 2024-09-21T10:14:17.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00089108099999999995 2024-09-21T10:14:17.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:14:17.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683557+0000", 2024-09-21T10:14:17.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8050000000000003e-06 2024-09-21T10:14:17.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.958192+0000", 2024-09-21T10:14:17.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.274634625 2024-09-21T10:14:17.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:14:17.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.958334+0000", 2024-09-21T10:14:17.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014211599999999999 2024-09-21T10:14:17.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:14:17.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.958509+0000", 2024-09-21T10:14:17.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017526600000000001 2024-09-21T10:14:17.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.771 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:14:17.771 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.964588+0000", 2024-09-21T10:14:17.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0060786360000000001 2024-09-21T10:14:17.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.771 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.771 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.972333+0000", 2024-09-21T10:14:17.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00774532 2024-09-21T10:14:17.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.771 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.772 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.979201+0000", 2024-09-21T10:14:17.772 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0068677540000000002 2024-09-21T10:14:17.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.772 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:14:17.772 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.979252+0000", 2024-09-21T10:14:17.772 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1403000000000002e-05 2024-09-21T10:14:17.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.772 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:14:17.772 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.979972+0000", 2024-09-21T10:14:17.772 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00072027099999999998 2024-09-21T10:14:17.772 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.772 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:14:17.773 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.773 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56527 7.13 7:cd6ce9c8:::benchmark_data_smithi098_197834_object56526:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e218)", 2024-09-21T10:14:17.773 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:48.682671+0000", 2024-09-21T10:14:17.773 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.962277608000001, 2024-09-21T10:14:17.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.298020245, 2024-09-21T10:14:17.773 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:14:17.773 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:14:17.773 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:17.773 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:14:17.773 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:14:17.773 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:14:17.773 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56527 2024-09-21T10:14:17.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.774 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:14:17.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:14:17.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682671+0000", 2024-09-21T10:14:17.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:14:17.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682671+0000", 2024-09-21T10:14:17.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:14:17.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682672+0000", 2024-09-21T10:14:17.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9999999999999997e-07 2024-09-21T10:14:17.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:14:17.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682718+0000", 2024-09-21T10:14:17.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6508999999999998e-05 2024-09-21T10:14:17.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:14:17.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682726+0000", 2024-09-21T10:14:17.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5390000000000002e-06 2024-09-21T10:14:17.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:14:17.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682731+0000", 2024-09-21T10:14:17.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3419999999999998e-06 2024-09-21T10:14:17.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683579+0000", 2024-09-21T10:14:17.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00084829899999999995 2024-09-21T10:14:17.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:14:17.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683584+0000", 2024-09-21T10:14:17.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9699999999999998e-06 2024-09-21T10:14:17.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.961633+0000", 2024-09-21T10:14:17.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2780490900000001 2024-09-21T10:14:17.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:14:17.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.961778+0000", 2024-09-21T10:14:17.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014414500000000001 2024-09-21T10:14:17.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:14:17.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.961940+0000", 2024-09-21T10:14:17.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016275999999999999 2024-09-21T10:14:17.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.778 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:14:17.778 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.964644+0000", 2024-09-21T10:14:17.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027038359999999998 2024-09-21T10:14:17.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.778 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.778 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.972379+0000", 2024-09-21T10:14:17.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0077352289999999997 2024-09-21T10:14:17.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.778 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.778 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.980023+0000", 2024-09-21T10:14:17.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0076437939999999998 2024-09-21T10:14:17.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:14:17.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.980082+0000", 2024-09-21T10:14:17.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9188999999999999e-05 2024-09-21T10:14:17.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:14:17.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.980691+0000", 2024-09-21T10:14:17.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00060894300000000003 2024-09-21T10:14:17.779 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.779 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:14:17.779 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.780 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56528 7.13 7:cd96644e:::benchmark_data_smithi098_197834_object56527:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e218)", 2024-09-21T10:14:17.780 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:48.682739+0000", 2024-09-21T10:14:17.780 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.962209945000001, 2024-09-21T10:14:17.780 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.299880548, 2024-09-21T10:14:17.780 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:14:17.780 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:14:17.780 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:17.780 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:14:17.780 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:14:17.780 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:14:17.780 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56528 2024-09-21T10:14:17.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.781 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:14:17.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:14:17.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682739+0000", 2024-09-21T10:14:17.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:14:17.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682739+0000", 2024-09-21T10:14:17.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:14:17.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682739+0000", 2024-09-21T10:14:17.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1399999999999997e-07 2024-09-21T10:14:17.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:14:17.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682784+0000", 2024-09-21T10:14:17.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4354999999999998e-05 2024-09-21T10:14:17.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:14:17.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682791+0000", 2024-09-21T10:14:17.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.627e-06 2024-09-21T10:14:17.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:14:17.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682797+0000", 2024-09-21T10:14:17.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5809999999999996e-06 2024-09-21T10:14:17.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683607+0000", 2024-09-21T10:14:17.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00080977700000000002 2024-09-21T10:14:17.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:14:17.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683612+0000", 2024-09-21T10:14:17.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1900000000000003e-06 2024-09-21T10:14:17.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.963284+0000", 2024-09-21T10:14:17.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.279671894 2024-09-21T10:14:17.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:14:17.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.963425+0000", 2024-09-21T10:14:17.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014132399999999999 2024-09-21T10:14:17.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:14:17.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.963561+0000", 2024-09-21T10:14:17.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000135649 2024-09-21T10:14:17.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:14:17.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.967633+0000", 2024-09-21T10:14:17.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040719340000000001 2024-09-21T10:14:17.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.974730+0000", 2024-09-21T10:14:17.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070972400000000003 2024-09-21T10:14:17.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.981528+0000", 2024-09-21T10:14:17.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0067984769999999998 2024-09-21T10:14:17.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:14:17.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.981579+0000", 2024-09-21T10:14:17.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0389000000000002e-05 2024-09-21T10:14:17.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:14:17.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.982619+0000", 2024-09-21T10:14:17.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010406969999999999 2024-09-21T10:14:17.786 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.786 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:14:17.786 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.787 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56572 7.13 7:cfd9caef:::benchmark_data_smithi098_197834_object56571:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e218)", 2024-09-21T10:14:17.787 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:48.682805+0000", 2024-09-21T10:14:17.787 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.962143788999999, 2024-09-21T10:14:17.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3008730929999999, 2024-09-21T10:14:17.787 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:14:17.787 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:14:17.787 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:17.787 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:14:17.787 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:14:17.787 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:14:17.787 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56572 2024-09-21T10:14:17.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.787 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:14:17.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:14:17.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682805+0000", 2024-09-21T10:14:17.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:14:17.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682805+0000", 2024-09-21T10:14:17.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:14:17.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682805+0000", 2024-09-21T10:14:17.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8700000000000001e-07 2024-09-21T10:14:17.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:14:17.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682848+0000", 2024-09-21T10:14:17.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2301e-05 2024-09-21T10:14:17.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:14:17.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682855+0000", 2024-09-21T10:14:17.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9630000000000001e-06 2024-09-21T10:14:17.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:14:17.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682860+0000", 2024-09-21T10:14:17.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7710000000000003e-06 2024-09-21T10:14:17.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683635+0000", 2024-09-21T10:14:17.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000774368 2024-09-21T10:14:17.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:14:17.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683641+0000", 2024-09-21T10:14:17.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.105e-06 2024-09-21T10:14:17.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.964762+0000", 2024-09-21T10:14:17.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.281121116 2024-09-21T10:14:17.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:14:17.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.964903+0000", 2024-09-21T10:14:17.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014153200000000001 2024-09-21T10:14:17.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:14:17.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.965057+0000", 2024-09-21T10:14:17.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015336700000000001 2024-09-21T10:14:17.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:14:17.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.968424+0000", 2024-09-21T10:14:17.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033670520000000002 2024-09-21T10:14:17.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.974976+0000", 2024-09-21T10:14:17.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0065518709999999999 2024-09-21T10:14:17.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.982725+0000", 2024-09-21T10:14:17.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0077488890000000001 2024-09-21T10:14:17.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:14:17.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.982780+0000", 2024-09-21T10:14:17.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5291999999999998e-05 2024-09-21T10:14:17.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:14:17.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.983678+0000", 2024-09-21T10:14:17.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00089807899999999998 2024-09-21T10:14:17.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.793 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:14:17.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.793 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56580 7.13 7:cb4a131c:::benchmark_data_smithi098_197834_object56579:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e218)", 2024-09-21T10:14:17.794 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:48.682868+0000", 2024-09-21T10:14:17.794 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.962080885999999, 2024-09-21T10:14:17.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3018799350000001, 2024-09-21T10:14:17.794 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:14:17.794 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:14:17.794 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:17.794 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:14:17.794 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:14:17.794 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:14:17.794 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56580 2024-09-21T10:14:17.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.794 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:14:17.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:14:17.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682868+0000", 2024-09-21T10:14:17.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:14:17.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682868+0000", 2024-09-21T10:14:17.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:14:17.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:14:17.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682868+0000", 2024-09-21T10:14:17.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0600000000000001e-07 2024-09-21T10:14:17.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:14:17.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682911+0000", 2024-09-21T10:14:17.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3069999999999999e-05 2024-09-21T10:14:17.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:14:17.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682919+0000", 2024-09-21T10:14:17.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2899999999999997e-06 2024-09-21T10:14:17.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:14:17.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.682924+0000", 2024-09-21T10:14:17.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4620000000000004e-06 2024-09-21T10:14:17.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683688+0000", 2024-09-21T10:14:17.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00076389299999999997 2024-09-21T10:14:17.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:14:17.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:48.683694+0000", 2024-09-21T10:14:17.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4679999999999997e-06 2024-09-21T10:14:17.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:14:17.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.966212+0000", 2024-09-21T10:14:17.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.282517704 2024-09-21T10:14:17.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:14:17.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.966359+0000", 2024-09-21T10:14:17.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014715099999999999 2024-09-21T10:14:17.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:14:17.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.966534+0000", 2024-09-21T10:14:17.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017447900000000001 2024-09-21T10:14:17.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:14:17.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.968460+0000", 2024-09-21T10:14:17.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0019261549999999999 2024-09-21T10:14:17.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.975027+0000", 2024-09-21T10:14:17.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0065672430000000004 2024-09-21T10:14:17.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:17.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.983739+0000", 2024-09-21T10:14:17.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0087115249999999995 2024-09-21T10:14:17.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:14:17.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.983833+0000", 2024-09-21T10:14:17.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4349000000000001e-05 2024-09-21T10:14:17.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:14:17.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:14:17.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:14:17.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:49.984748+0000", 2024-09-21T10:14:17.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00091474000000000004 2024-09-21T10:14:17.800 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.800 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:14:17.800 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.800 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:14:17.800 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:14:17.800 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:14:17.864 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:17.890 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:17.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:17.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:17.992 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:18.026 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:18.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:18.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:18.093 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:18.128 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:18.183 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:18.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:18.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:18.251 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 65 16 92047 92031 88.4754 63.375 0.0106691 0.0112978 2024-09-21T10:14:18.285 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:18.311 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:18.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:18.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:18.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:18.435 INFO:teuthology.orchestra.run.smithi098.stderr:set pool 7 pgp_num to 26 2024-09-21T10:14:18.439 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:18.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:18.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:18.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:18.580 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:18.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:18.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:18.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:18 smithi098 ceph-mon[97580]: pgmap v674: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 1715/261327 objects degraded (0.656%); 3199/261327 objects misplaced (1.224%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:14:18.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:18 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1399280727' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"} : dispatch 2024-09-21T10:14:18.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:18 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"} : dispatch 2024-09-21T10:14:18.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:18 smithi098 ceph-mon[100914]: pgmap v674: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 1715/261327 objects degraded (0.656%); 3199/261327 objects misplaced (1.224%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:14:18.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:18 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1399280727' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"} : dispatch 2024-09-21T10:14:18.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:18 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"} : dispatch 2024-09-21T10:14:18.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:18.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:18.711 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:18.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:18.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:18.854 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:18.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:18.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:18.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:18 smithi196 ceph-mon[84784]: pgmap v674: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 1715/261327 objects degraded (0.656%); 3199/261327 objects misplaced (1.224%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:14:18.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:18 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1399280727' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"} : dispatch 2024-09-21T10:14:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:18 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"} : dispatch 2024-09-21T10:14:18.893 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:14:18.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:18.982 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:18.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:18.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:19.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:19.155 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:19.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:19.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:19.250 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 66 16 93277 93261 88.2993 76.875 0.0107386 0.0113193 2024-09-21T10:14:19.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:19.297 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:19.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:19.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:19.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:19.441 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:19.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:19.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:19.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:19.565 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:19.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:19.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:19.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:19 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"}]': finished 2024-09-21T10:14:19.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:19 smithi098 ceph-mon[97580]: osdmap e224: 8 total, 8 up, 7 in 2024-09-21T10:14:19.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:19 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"}]': finished 2024-09-21T10:14:19.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:19 smithi098 ceph-mon[100914]: osdmap e224: 8 total, 8 up, 7 in 2024-09-21T10:14:19.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:19.707 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:19.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:19.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:19.810 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:19.837 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:19.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:19.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:19.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:19 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"}]': finished 2024-09-21T10:14:19.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:19 smithi196 ceph-mon[84784]: osdmap e224: 8 total, 8 up, 7 in 2024-09-21T10:14:19.939 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:19.974 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:19.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:19.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:20.075 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:20.113 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:20.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:20.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:20.218 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:20.250 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 67 16 94414 94398 88.042 71.0625 0.00935089 0.011353 2024-09-21T10:14:20.255 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:20.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:20.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:20.357 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:20.381 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:20.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:20.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:20.483 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:20.506 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:20.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:20.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:20.608 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:20.633 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:20.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:20.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:20.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:20 smithi196 ceph-mon[84784]: pgmap v676: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s; 1715/272907 objects degraded (0.628%); 3199/272907 objects misplaced (1.172%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T10:14:20.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:14:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:14:20.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:14:20.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:20 smithi098 ceph-mon[97580]: pgmap v676: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s; 1715/272907 objects degraded (0.628%); 3199/272907 objects misplaced (1.172%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T10:14:20.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:20 smithi098 ceph-mon[100914]: pgmap v676: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s; 1715/272907 objects degraded (0.628%); 3199/272907 objects misplaced (1.172%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T10:14:20.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:20.775 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:20.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:20.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:20.877 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:20.899 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:20.904 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:20.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:20.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:21.006 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:14:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:14:20] "GET /metrics HTTP/1.1" 200 36582 "" "Prometheus/2.51.0" 2024-09-21T10:14:21.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:21.032 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:21.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:21.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:21.134 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:21.162 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:21.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:21.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:21.250 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 68 16 95610 95594 87.8464 74.75 0.00607757 0.0113788 2024-09-21T10:14:21.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:21.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:21.305 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:21.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:21.406 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:21.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:21.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:21.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:21.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:21.582 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:21.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:21.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:21.684 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:21.709 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:21.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:21.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:21.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:21 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 1715/272907 objects degraded (0.628%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:21.742 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:21 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 1715/272907 objects degraded (0.628%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:21.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:21.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:21.854 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:21.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:21.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:21 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 1715/272907 objects degraded (0.628%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:21.955 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:21.980 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:21.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:21.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:22.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:22.103 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:22.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:22.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:22.205 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:22.230 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:22.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:22.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:22.250 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 69 16 96795 96779 87.6466 74.0625 0.0089559 0.0114038 2024-09-21T10:14:22.332 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:22.370 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:22.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:22.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:22.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:22.496 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:22.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:22.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:22.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:22 smithi098 ceph-mon[97580]: pgmap v677: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s; 1715/278739 objects degraded (0.615%); 2563/278739 objects misplaced (0.919%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:14:22.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:22.597 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:22.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:22 smithi098 ceph-mon[100914]: pgmap v677: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s; 1715/278739 objects degraded (0.615%); 2563/278739 objects misplaced (0.919%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:14:22.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:22.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:22.634 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:22.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:22.735 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:22.746 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:22.760 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:22.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:22.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:22.862 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:22 smithi196 ceph-mon[84784]: pgmap v677: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s; 1715/278739 objects degraded (0.615%); 2563/278739 objects misplaced (0.919%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:14:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:22.906 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:22.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:22.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:23.008 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:23.048 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:23.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:23.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:23.190 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:23.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:23.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:23.251 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 70 16 97943 97927 87.4194 71.75 0.00808075 0.0114342 2024-09-21T10:14:23.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:23.335 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:23.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:23.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:23.437 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:23.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:23.472 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:23.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:23.486 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:14:23.573 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:23.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:23.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:23.711 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:23.737 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:23.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:23.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:23.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:23.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:23 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3852902275' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:14:23.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:23 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:14:23.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:23 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3852902275' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:14:23.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:23 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:14:23.870 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:23.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:23.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:23.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:23 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3852902275' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:14:23.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:23 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:14:23.972 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:24.009 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:24.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:24.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:24.111 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:24.138 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:24.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:24.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:24.240 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:24.251 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 71 16 99125 99109 87.2286 73.875 0.0121642 0.0114593 2024-09-21T10:14:24.267 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:24.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:24.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:24.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:24.394 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:24.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:24.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:24.496 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:24.535 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:24.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:24.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:24.631 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:24 smithi098 ceph-mon[100914]: pgmap v678: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s; 1715/285321 objects degraded (0.601%); 2563/285321 objects misplaced (0.898%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:14:24.631 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:24 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:14:24.631 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:24 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:14:24.631 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:24 smithi098 ceph-mon[100914]: osdmap e225: 8 total, 8 up, 7 in 2024-09-21T10:14:24.631 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:24 smithi098 ceph-mon[97580]: pgmap v678: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s; 1715/285321 objects degraded (0.601%); 2563/285321 objects misplaced (0.898%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:14:24.631 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:24 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:14:24.632 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:24 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:14:24.632 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:24 smithi098 ceph-mon[97580]: osdmap e225: 8 total, 8 up, 7 in 2024-09-21T10:14:24.637 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:24.664 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:24.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:24.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:24.765 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:24.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:24.794 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:24.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:24 smithi196 ceph-mon[84784]: pgmap v678: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s; 1715/285321 objects degraded (0.601%); 2563/285321 objects misplaced (0.898%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:14:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:24 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:14:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:24 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:14:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:24 smithi196 ceph-mon[84784]: osdmap e225: 8 total, 8 up, 7 in 2024-09-21T10:14:24.895 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:24.930 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:24.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:24.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:25.033 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:25.073 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:25.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:25.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:25.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:25.202 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:25.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:25.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:25.251 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 72 16 100403 100387 87.1264 79.875 0.00550176 0.0114724 2024-09-21T10:14:25.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:25.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:25.347 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:25.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:25.448 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:25.478 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:25.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:25.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:25.580 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:25.608 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:25.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:25.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:25.628 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:14:25.628 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:14:25.667 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:14:25.667 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:14:25.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:25.711 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:14:25.712 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:14:25.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:25.750 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:25.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:25.751 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:14:25.752 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:14:25.789 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:14:25.789 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:14:25.820 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:14:25.820 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:14:25.852 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:25.853 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:14:25.853 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:14:25.878 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:25.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:25.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:25.886 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:14:25.886 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:14:25.979 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:26.004 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:26.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:26.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:26.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:26.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:26.140 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:26.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:26.241 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:26.251 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 73 16 101467 101451 86.8437 66.5 0.00404953 0.0114824 2024-09-21T10:14:26.264 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:26.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:26.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:26.366 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:26.391 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:26.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:26.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:26.493 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:26.517 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:26.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:26.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:26.618 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:26.638 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:26 smithi098 ceph-mon[97580]: pgmap v680: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s; 1715/293613 objects degraded (0.584%); 1805/293613 objects misplaced (0.615%); 5.4 MiB/s, 87 objects/s recovering 2024-09-21T10:14:26.638 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:26 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/395784874' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:26.638 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:26 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 1715/293613 objects degraded (0.584%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:26.639 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:26 smithi098 ceph-mon[97580]: osdmap e226: 8 total, 8 up, 7 in 2024-09-21T10:14:26.639 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:26 smithi098 ceph-mon[100914]: pgmap v680: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s; 1715/293613 objects degraded (0.584%); 1805/293613 objects misplaced (0.615%); 5.4 MiB/s, 87 objects/s recovering 2024-09-21T10:14:26.639 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:26 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/395784874' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:26.639 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:26 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 1715/293613 objects degraded (0.584%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:26.639 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:26 smithi098 ceph-mon[100914]: osdmap e226: 8 total, 8 up, 7 in 2024-09-21T10:14:26.640 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:26.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:26.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:26.742 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:26.769 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:26.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:26.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:26.871 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:26 smithi196 ceph-mon[84784]: pgmap v680: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s; 1715/293613 objects degraded (0.584%); 1805/293613 objects misplaced (0.615%); 5.4 MiB/s, 87 objects/s recovering 2024-09-21T10:14:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:26 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/395784874' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:26 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 1715/293613 objects degraded (0.584%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:26 smithi196 ceph-mon[84784]: osdmap e226: 8 total, 8 up, 7 in 2024-09-21T10:14:26.911 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:26.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:26.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:27.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:14:27.013 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:27.039 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:27.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:27.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:27.141 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:27.164 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:27.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:27.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:27.254 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 74 16 102280 102264 86.3529 50.8125 0.0380387 0.0115751 2024-09-21T10:14:27.266 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:27.333 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:27.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:27.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:27.435 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:27.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:27.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:27.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:27.630 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:27.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:27.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:27.732 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:27.766 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:27.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:27.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:27.868 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:27.876 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:14:27.876 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:14:27.892 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:27.908 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:27.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:27.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:28.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:28.034 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:28.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:28.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:28.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:27 smithi098 ceph-mon[97580]: osdmap e227: 8 total, 8 up, 7 in 2024-09-21T10:14:28.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:27 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T10:14:28.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:27 smithi098 ceph-mon[100914]: osdmap e227: 8 total, 8 up, 7 in 2024-09-21T10:14:28.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:27 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T10:14:28.137 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:27 smithi196 ceph-mon[84784]: osdmap e227: 8 total, 8 up, 7 in 2024-09-21T10:14:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:27 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T10:14:28.159 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:28.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:28.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:28.237 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:14:28.237 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:14:28.238 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-21T10:14:28.238 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:14:28.238 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:14:28.238 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-21T10:14:28.254 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 75 16 103125 103109 85.9056 52.8125 0.00377708 0.0115893 2024-09-21T10:14:28.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:28.282 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:28.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:28.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:28.384 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:28.405 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:28.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:28.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:28.487 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:14:28.506 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:28.528 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:28.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:28.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:28.629 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:28.650 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:28.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:28.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:28.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:14:28.752 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:28.775 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:28.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:28.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:28.869 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:28 smithi196 ceph-mon[84784]: pgmap v683: 99 pgs: 2 active+recovering+undersized+remapped, 97 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 61 MiB/s wr, 981 op/s; 3454/296412 objects misplaced (1.165%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T10:14:28.869 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:28 smithi196 ceph-mon[84784]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1715/293613 objects degraded (0.584%), 1 pg degraded) 2024-09-21T10:14:28.869 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:28 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T10:14:28.869 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:28 smithi196 ceph-mon[84784]: osdmap e228: 8 total, 8 up, 7 in 2024-09-21T10:14:28.876 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:28.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:28.899 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:28.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:29.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:29.022 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:29.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:29.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:29.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:28 smithi098 ceph-mon[97580]: pgmap v683: 99 pgs: 2 active+recovering+undersized+remapped, 97 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 61 MiB/s wr, 981 op/s; 3454/296412 objects misplaced (1.165%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T10:14:29.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:28 smithi098 ceph-mon[97580]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1715/293613 objects degraded (0.584%), 1 pg degraded) 2024-09-21T10:14:29.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:28 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T10:14:29.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:28 smithi098 ceph-mon[97580]: osdmap e228: 8 total, 8 up, 7 in 2024-09-21T10:14:29.089 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:28 smithi098 ceph-mon[100914]: pgmap v683: 99 pgs: 2 active+recovering+undersized+remapped, 97 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 61 MiB/s wr, 981 op/s; 3454/296412 objects misplaced (1.165%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T10:14:29.089 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:28 smithi098 ceph-mon[100914]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1715/293613 objects degraded (0.584%), 1 pg degraded) 2024-09-21T10:14:29.089 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:28 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T10:14:29.089 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:28 smithi098 ceph-mon[100914]: osdmap e228: 8 total, 8 up, 7 in 2024-09-21T10:14:29.114 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:14:29.124 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:29.145 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:29.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:29.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:29.246 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:29.254 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 76 16 103125 103109 84.7755 0 - 0.0115893 2024-09-21T10:14:29.268 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:29.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:29.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:29.369 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:29.391 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:29.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:29.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:29.493 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:29.514 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:29.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:29.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:29.615 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:29.636 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:29.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:29.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:29.738 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:29.764 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:29.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:29.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:29.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:29.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:29 smithi196 ceph-mon[84784]: osdmap e229: 8 total, 8 up, 7 in 2024-09-21T10:14:29.889 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:29.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:29.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:29.990 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:30.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:30.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:30.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:29 smithi098 ceph-mon[97580]: osdmap e229: 8 total, 8 up, 7 in 2024-09-21T10:14:30.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:29 smithi098 ceph-mon[100914]: osdmap e229: 8 total, 8 up, 7 in 2024-09-21T10:14:30.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:30.137 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:30.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:30.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:30.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:30.255 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 77 16 103229 103213 83.7588 3.25 0.0169965 0.0119339 2024-09-21T10:14:30.289 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:30.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:30.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:30.333 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:14:30.333 INFO:teuthology.orchestra.run.smithi196.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:14:30.333 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:14:30.391 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:30.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:30.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:30.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:30.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:30.547 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:30.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:30.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:30.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:14:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:14:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:14:30.649 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:30.675 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:30.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:30.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:30.777 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:30.803 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:30.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:30.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:30.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:14:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:14:30] "GET /metrics HTTP/1.1" 200 36578 "" "Prometheus/2.51.0" 2024-09-21T10:14:30.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:30 smithi098 ceph-mon[97580]: pgmap v686: 99 pgs: 1 unknown, 1 active+recovering+undersized+remapped, 97 active+clean; 6.1 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 21 MiB/s wr, 332 op/s; 3364/297600 objects misplaced (1.130%); 1.2 MiB/s, 18 objects/s recovering 2024-09-21T10:14:30.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:30 smithi098 ceph-mon[97580]: osdmap e230: 8 total, 8 up, 7 in 2024-09-21T10:14:30.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:30 smithi098 ceph-mon[100914]: pgmap v686: 99 pgs: 1 unknown, 1 active+recovering+undersized+remapped, 97 active+clean; 6.1 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 21 MiB/s wr, 332 op/s; 3364/297600 objects misplaced (1.130%); 1.2 MiB/s, 18 objects/s recovering 2024-09-21T10:14:30.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:30 smithi098 ceph-mon[100914]: osdmap e230: 8 total, 8 up, 7 in 2024-09-21T10:14:30.905 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:30.930 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:30.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:30.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:31.025 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:30 smithi196 ceph-mon[84784]: pgmap v686: 99 pgs: 1 unknown, 1 active+recovering+undersized+remapped, 97 active+clean; 6.1 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 21 MiB/s wr, 332 op/s; 3364/297600 objects misplaced (1.130%); 1.2 MiB/s, 18 objects/s recovering 2024-09-21T10:14:31.025 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:30 smithi196 ceph-mon[84784]: osdmap e230: 8 total, 8 up, 7 in 2024-09-21T10:14:31.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:31.062 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:31.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:31.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:31.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:31.190 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:31.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:31.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:31.255 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 78 16 104422 104406 83.6408 74.5625 0.0100088 0.0119513 2024-09-21T10:14:31.291 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:31.321 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:31.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:31.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:31.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:31.463 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:31.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:31.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:31.565 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:31.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:31.592 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:31.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:31.693 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:31.719 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:31.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:31.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:31.821 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:31.847 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:31.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:31.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:31.949 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:31.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:31.992 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:31.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:32.093 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:32.114 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:32.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:32.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:32.216 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:32.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:32.245 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:32.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:32.255 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 79 16 105439 105423 83.3866 63.5625 0.0154681 0.0119872 2024-09-21T10:14:32.346 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:32.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:32.373 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:32.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:32.474 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:32.510 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:32.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:32.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:32.612 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:32.651 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:32.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:32.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:32.752 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:32.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:32.800 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:32.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:32.901 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:32.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:32.937 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:32.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:33.038 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:33.063 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:33.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:33.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:33.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:32 smithi098 ceph-mon[97580]: pgmap v688: 99 pgs: 1 active+recovering+undersized+remapped, 1 unknown, 97 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 18 MiB/s wr, 284 op/s; 2318/300339 objects misplaced (0.772%); 7.7 MiB/s, 123 objects/s recovering 2024-09-21T10:14:33.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:32 smithi098 ceph-mon[100914]: pgmap v688: 99 pgs: 1 active+recovering+undersized+remapped, 1 unknown, 97 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 18 MiB/s wr, 284 op/s; 2318/300339 objects misplaced (0.772%); 7.7 MiB/s, 123 objects/s recovering 2024-09-21T10:14:33.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:32 smithi196 ceph-mon[84784]: pgmap v688: 99 pgs: 1 active+recovering+undersized+remapped, 1 unknown, 97 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 18 MiB/s wr, 284 op/s; 2318/300339 objects misplaced (0.772%); 7.7 MiB/s, 123 objects/s recovering 2024-09-21T10:14:33.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:33.194 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:33.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:33.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:33.255 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:14:33.254216+0000 min lat: 0.00126097 max lat: 2.35514 avg lat: 0.0120119 2024-09-21T10:14:33.255 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:14:33.255 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 80 16 106555 106539 83.2161 69.75 0.0151391 0.0120119 2024-09-21T10:14:33.295 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:33.335 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:33.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:33.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:33.437 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:33.464 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:33.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:33.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:33.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:33.602 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:33.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:33.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:33.704 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:33.727 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:33.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:33.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:33.813 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:33.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:33.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:33.891 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:33.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:33.905 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:33 smithi098 ceph-mon[97580]: Health check failed: Degraded data redundancy: 6416/313857 objects degraded (2.044%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:33.905 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:33 smithi098 ceph-mon[100914]: Health check failed: Degraded data redundancy: 6416/313857 objects degraded (2.044%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:33.992 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:34.016 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:34.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:34.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:34.118 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:34.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:33 smithi196 ceph-mon[84784]: Health check failed: Degraded data redundancy: 6416/313857 objects degraded (2.044%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:34.143 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:34.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:34.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:34.244 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:34.255 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 81 16 107610 107594 83.0027 65.9375 0.0141616 0.0120422 2024-09-21T10:14:34.280 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:34.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:34.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:34.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:34.420 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:34.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:34.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:34.522 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:34.562 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:34.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:34.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:34.664 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:34.707 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:34.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:34.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:34.808 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:34.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:34.831 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:34.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:34.932 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:34.969 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:34.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:34.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:35.070 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:35.096 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:35.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:35.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:35.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:34 smithi098 ceph-mon[97580]: pgmap v689: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 61 MiB/s wr, 968 op/s; 6416/313857 objects degraded (2.044%); 5526/313857 objects misplaced (1.761%); 5.9 MiB/s, 94 objects/s recovering 2024-09-21T10:14:35.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:34 smithi098 ceph-mon[100914]: pgmap v689: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 61 MiB/s wr, 968 op/s; 6416/313857 objects degraded (2.044%); 5526/313857 objects misplaced (1.761%); 5.9 MiB/s, 94 objects/s recovering 2024-09-21T10:14:35.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:34 smithi196 ceph-mon[84784]: pgmap v689: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 61 MiB/s wr, 968 op/s; 6416/313857 objects degraded (2.044%); 5526/313857 objects misplaced (1.761%); 5.9 MiB/s, 94 objects/s recovering 2024-09-21T10:14:35.198 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:35.223 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:35.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:35.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:35.255 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 82 16 108600 108584 82.745 61.875 0.00982766 0.0120803 2024-09-21T10:14:35.325 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:35.352 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:35.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:35.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:35.454 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:35.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:35.497 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:35.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:35.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:35.649 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:35.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:35.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:35.751 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:35.776 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:35.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:35.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:35.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:35.899 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:35.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:35.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:36.001 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:36.027 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:36.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:36.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:36.065 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:14:36.065 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:14:36.065 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-21T10:14:36.065 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:14:36.065 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:14:36.065 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-21T10:14:36.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:36.189 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:36.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:36.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:36.255 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 83 16 109465 109449 82.3994 54.0625 0.0830218 0.0121268 2024-09-21T10:14:36.291 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:36.314 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:36.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:36.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:36.416 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:36.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:36.466 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:36.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:36.567 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:36.593 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:36.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:36.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:36.694 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:36.727 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:36.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:36.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:36.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:36.853 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:36.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:36.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:36.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:36.978 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:36.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:36.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:37.080 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:37.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:36 smithi098 ceph-mon[97580]: pgmap v690: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s; 6416/319380 objects degraded (2.009%); 5526/319380 objects misplaced (1.730%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T10:14:37.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:36 smithi098 ceph-mon[100914]: pgmap v690: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s; 6416/319380 objects degraded (2.009%); 5526/319380 objects misplaced (1.730%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T10:14:37.121 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:37.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:37.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:37.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:36 smithi196 ceph-mon[84784]: pgmap v690: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s; 6416/319380 objects degraded (2.009%); 5526/319380 objects misplaced (1.730%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T10:14:37.223 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:37.245 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:37.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:37.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:37.256 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 84 16 110333 110317 82.0643 54.25 0.0171474 0.0121796 2024-09-21T10:14:37.347 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:37.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:37.394 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:37.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:37.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:37.532 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:37.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:37.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:37.634 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:37.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:37.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:37.773 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:37.806 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:37.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:37.830 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:37.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:37.930 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:37.957 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:37.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:37.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:37.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:37.979 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:38.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:38.059 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:38.098 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:38.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:38.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:38.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:38.238 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:38.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:38.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:38.256 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 85 16 111307 111291 81.8149 60.875 0.0273807 0.0122173 2024-09-21T10:14:38.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:38.384 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:38.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:38.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:38.486 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:38.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:38.513 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:38.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:38.614 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:38.635 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:38.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:38.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:38.736 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:38.761 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:38.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:38.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:38.831 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:14:38.863 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:38.888 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:38.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:38.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:38.990 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:39.018 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:39.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:39.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:39.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:38 smithi098 ceph-mon[97580]: pgmap v691: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 6.6 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 59 MiB/s wr, 941 op/s; 6416/321834 objects degraded (1.994%); 5026/321834 objects misplaced (1.562%); 5.6 MiB/s, 90 objects/s recovering 2024-09-21T10:14:39.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:38 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1475741456' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:14:39.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:38 smithi098 ceph-mon[100914]: pgmap v691: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 6.6 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 59 MiB/s wr, 941 op/s; 6416/321834 objects degraded (1.994%); 5026/321834 objects misplaced (1.562%); 5.6 MiB/s, 90 objects/s recovering 2024-09-21T10:14:39.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:38 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1475741456' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:14:39.120 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:39.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:38 smithi196 ceph-mon[84784]: pgmap v691: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 6.6 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 59 MiB/s wr, 941 op/s; 6416/321834 objects degraded (1.994%); 5026/321834 objects misplaced (1.562%); 5.6 MiB/s, 90 objects/s recovering 2024-09-21T10:14:39.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:38 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1475741456' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:14:39.144 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:39.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:39.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:39.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:39.259 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 86 16 112341 112325 81.6133 64.625 0.0137172 0.0122473 2024-09-21T10:14:39.292 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:39.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:39.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:39.393 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:39.416 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:39.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:39.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:39.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:39.565 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:39.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:39.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:39.666 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:39.691 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:39.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:39.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:39.793 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:39.831 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:39.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:39.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:39.933 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:39.957 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:39.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:39.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:40.058 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:40.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:39 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:14:40.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:39 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1475741456' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:14:40.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:39 smithi098 ceph-mon[97580]: osdmap e231: 8 total, 8 up, 7 in 2024-09-21T10:14:40.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:39 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:14:40.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:39 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1475741456' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:14:40.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:39 smithi098 ceph-mon[100914]: osdmap e231: 8 total, 8 up, 7 in 2024-09-21T10:14:40.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:40.099 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:40.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:40.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:39 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:14:40.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:39 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1475741456' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:14:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:39 smithi196 ceph-mon[84784]: osdmap e231: 8 total, 8 up, 7 in 2024-09-21T10:14:40.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:40.227 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:40.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:40.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:40.257 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 87 16 113405 113389 81.4396 66.5 0.0106584 0.0122736 2024-09-21T10:14:40.329 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:40.353 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:40.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:40.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:40.454 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:40.488 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:40.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:40.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:40.589 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:40.614 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:40.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:40.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:40.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:14:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:14:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:14:40.716 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:40.740 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:40.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:40.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:40.842 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:40.876 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:40.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:40.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:40.978 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:41.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:41.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:41.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:40 smithi098 ceph-mon[97580]: pgmap v693: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 6416/331476 objects degraded (1.936%); 5026/331476 objects misplaced (1.516%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:14:41.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:40 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 6416/331476 objects degraded (1.936%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:41.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:14:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:14:40] "GET /metrics HTTP/1.1" 200 36578 "" "Prometheus/2.51.0" 2024-09-21T10:14:41.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:40 smithi098 ceph-mon[100914]: pgmap v693: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 6416/331476 objects degraded (1.936%); 5026/331476 objects misplaced (1.516%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:14:41.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:40 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 6416/331476 objects degraded (1.936%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:41.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:41.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:40 smithi196 ceph-mon[84784]: pgmap v693: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 6416/331476 objects degraded (1.936%); 5026/331476 objects misplaced (1.516%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:14:41.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:40 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 6416/331476 objects degraded (1.936%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:41.145 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:41.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:41.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:41.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:41.257 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 88 16 114453 114437 81.2584 65.5 0.00940381 0.0123018 2024-09-21T10:14:41.287 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:41.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:41.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:41.389 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:41.415 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:41.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:41.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:41.517 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:41.538 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:41.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:41.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:41.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:41.682 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:41.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:41.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:41.784 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:41.814 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:41.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:41.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:41.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:41.953 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:41.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:41.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:42.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:42.095 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:42.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:42.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:42.197 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:42.231 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:42.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:42.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:42.258 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 89 16 115489 115473 81.0728 64.75 0.0095539 0.0123295 2024-09-21T10:14:42.333 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:42.358 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:42.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:42.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:42.460 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:42.497 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:42.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:42.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:42.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:42.623 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:42.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:42.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:42.725 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:42.750 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:42.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:42.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:42.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:42.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:42.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:43.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:43.041 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:43.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:43.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:43.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:42 smithi098 ceph-mon[97580]: pgmap v694: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s; 6416/333843 objects degraded (1.922%); 5026/333843 objects misplaced (1.505%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T10:14:43.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:42 smithi098 ceph-mon[100914]: pgmap v694: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s; 6416/333843 objects degraded (1.922%); 5026/333843 objects misplaced (1.505%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T10:14:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:42 smithi196 ceph-mon[84784]: pgmap v694: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s; 6416/333843 objects degraded (1.922%); 5026/333843 objects misplaced (1.505%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T10:14:43.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:43.177 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:43.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:43.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:43.258 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Total time run: 90.0083 2024-09-21T10:14:43.258 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Total writes made: 116514 2024-09-21T10:14:43.258 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Write size: 65536 2024-09-21T10:14:43.258 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Object size: 65536 2024-09-21T10:14:43.258 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Bandwidth (MB/sec): 80.9051 2024-09-21T10:14:43.258 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Stddev Bandwidth: 40.9406 2024-09-21T10:14:43.258 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Max bandwidth (MB/sec): 203.438 2024-09-21T10:14:43.258 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Min bandwidth (MB/sec): 0 2024-09-21T10:14:43.258 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Average IOPS: 1294 2024-09-21T10:14:43.258 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Stddev IOPS: 655.059 2024-09-21T10:14:43.258 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Max IOPS: 3255 2024-09-21T10:14:43.259 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Min IOPS: 0 2024-09-21T10:14:43.259 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Average Latency(s): 0.0123559 2024-09-21T10:14:43.259 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Stddev Latency(s): 0.0558198 2024-09-21T10:14:43.259 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Max latency(s): 2.35514 2024-09-21T10:14:43.259 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Min latency(s): 0.00126097 2024-09-21T10:14:43.263 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:14:43.279 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:43.306 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:43.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:43.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:43.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:43.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:43.459 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:43.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:43.560 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:43.608 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:43.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:43.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:43.709 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:43.733 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:43.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:43.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:43.834 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:43.877 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:43.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:43.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:43.941 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:43 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:14:43.941 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:43 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:14:43.979 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:44.003 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:44.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:44.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:44.080 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:43 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:14:44.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:44.150 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:44.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:44.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:44.252 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:44.277 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:44.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:44.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:44.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:44.419 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:44.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:44.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:44.521 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:44.547 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:44.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:44.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:44.648 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:44.673 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:44.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:44.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:44.775 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:44.809 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:44.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:44.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:44.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:44 smithi098 ceph-mon[97580]: pgmap v695: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 60 MiB/s wr, 952 op/s; 6416/342456 objects degraded (1.874%); 4480/342456 objects misplaced (1.308%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:14:44.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:44 smithi098 ceph-mon[100914]: pgmap v695: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 60 MiB/s wr, 952 op/s; 6416/342456 objects degraded (1.874%); 4480/342456 objects misplaced (1.308%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:14:44.910 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:44.948 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:44.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:44.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:45.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:45.087 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:45.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:45.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:45.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:14:45.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:44 smithi196 ceph-mon[84784]: pgmap v695: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 60 MiB/s wr, 952 op/s; 6416/342456 objects degraded (1.874%); 4480/342456 objects misplaced (1.308%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:14:45.190 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:45.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:45.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:45.346 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:45.366 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:45.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:45.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:45.467 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:45.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:45.493 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:45.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:45.593 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:45.632 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:45.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:45.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:45.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:45.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:45.758 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:45.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:45.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:45.883 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:45.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:45.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:45.985 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:46.010 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:46.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:46.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:46.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:45 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 6416/342456 objects degraded (1.874%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:46.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:45 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 6416/342456 objects degraded (1.874%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:46.112 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:45 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 6416/342456 objects degraded (1.874%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:46.156 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:46.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:46.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:46.258 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:46.291 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:46.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:46.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:46.393 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:46.427 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:46.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:46.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:46.528 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:46.557 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:46.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:46.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:46.659 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:46.694 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:46.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:46.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:46.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:46.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:46.834 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:46.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:46.935 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:46.970 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:46.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:46.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:47.072 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:47.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:47.098 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:47.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:47.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:47.232 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:47.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:47.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:47.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:46 smithi196 ceph-mon[84784]: pgmap v696: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 60 MiB/s wr, 997 op/s; 6416/347487 objects degraded (1.846%); 4480/347487 objects misplaced (1.289%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:14:47.333 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:47.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:46 smithi098 ceph-mon[97580]: pgmap v696: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 60 MiB/s wr, 997 op/s; 6416/347487 objects degraded (1.846%); 4480/347487 objects misplaced (1.289%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:14:47.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:46 smithi098 ceph-mon[100914]: pgmap v696: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 60 MiB/s wr, 997 op/s; 6416/347487 objects degraded (1.846%); 4480/347487 objects misplaced (1.289%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:14:47.369 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:47.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:47.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:47.471 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:47.521 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:47.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:47.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:47.623 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:47.644 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:47.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:47.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:47.745 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:47.771 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:14:47.771 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:14:47.771 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:14:47.771 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:14:47.772 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:14:47.772 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:14:47.772 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:14:47.772 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:14:47.772 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:47.773 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:14:47.773 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:14:47.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:47.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:47.806 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:14:47.806 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:14:47.835 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:14:47.836 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:14:47.863 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:14:47.863 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:14:47.868 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:14:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:14:47.874 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:47.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:47.933 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:47.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:48.034 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:48.074 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:48.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:48.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:48.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:47 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/286112379' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:48.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:47 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:14:48.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:47 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:14:48.102 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:47 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/286112379' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:48.103 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:47 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:14:48.103 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:47 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:14:48.176 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:48.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:48.226 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:48.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:48.327 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:48.353 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:48.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:48.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:48.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:47 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/286112379' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:48.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:47 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:14:48.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:47 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:14:48.455 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:48.493 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:48.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:48.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:48.595 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:48.619 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:48.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:48.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:48.721 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:48.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:48.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:48.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:48.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:48.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:48.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:48.994 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:49.022 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:49.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:49.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:49.124 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:49.148 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:49.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:49.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:49.174 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:49.250 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:49.286 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:48 smithi098 ceph-mon[97580]: pgmap v697: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 57 MiB/s wr, 996 op/s; 6416/346998 objects degraded (1.849%); 3846/346998 objects misplaced (1.108%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:14:49.286 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:48 smithi098 ceph-mon[100914]: pgmap v697: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 57 MiB/s wr, 996 op/s; 6416/346998 objects degraded (1.849%); 3846/346998 objects misplaced (1.108%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:14:49.296 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:49.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:49.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:49.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:48 smithi196 ceph-mon[84784]: pgmap v697: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 57 MiB/s wr, 996 op/s; 6416/346998 objects degraded (1.849%); 3846/346998 objects misplaced (1.108%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:14:49.398 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:49.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:49.436 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:49.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:49.536 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:49.562 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:49.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:49.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:49.664 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:49.689 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:49.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:49.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:49.729 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:14:49.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:49.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:49.849 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:49.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:49.950 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:50.002 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:50.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:50.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:50.104 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:50.130 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:50.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:50.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:50.232 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:50.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:50.278 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:50.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:50.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:50.417 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:50.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:50.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:50.519 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:50.544 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:50.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:50.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:50.552 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:14:50.553 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:14:50.553 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:14:50.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:14:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:14:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:14:50.646 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:50.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:50.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:50.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:50.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:14:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:14:50] "GET /metrics HTTP/1.1" 200 36578 "" "Prometheus/2.51.0" 2024-09-21T10:14:50.829 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:50.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:50.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:50.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:14:50.931 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:50.955 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:50.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:50.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:51.056 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:51.079 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:51.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:51.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:51.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:50 smithi098 ceph-mon[97580]: pgmap v698: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 6.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 98 B/s rd, 37 MiB/s wr, 955 op/s; 6152/338328 objects degraded (1.818%); 3714/338328 objects misplaced (1.098%); 4.3 MiB/s, 69 objects/s recovering 2024-09-21T10:14:51.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:50 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 6152/338328 objects degraded (1.818%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:14:51.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:50 smithi098 ceph-mon[97580]: osdmap e232: 8 total, 8 up, 7 in 2024-09-21T10:14:51.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:50 smithi098 ceph-mon[100914]: pgmap v698: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 6.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 98 B/s rd, 37 MiB/s wr, 955 op/s; 6152/338328 objects degraded (1.818%); 3714/338328 objects misplaced (1.098%); 4.3 MiB/s, 69 objects/s recovering 2024-09-21T10:14:51.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:50 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 6152/338328 objects degraded (1.818%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:14:51.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:50 smithi098 ceph-mon[100914]: osdmap e232: 8 total, 8 up, 7 in 2024-09-21T10:14:51.181 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:51.203 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:51.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:51.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:51.305 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:51.329 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:51.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:51.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:51.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:50 smithi196 ceph-mon[84784]: pgmap v698: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 6.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 98 B/s rd, 37 MiB/s wr, 955 op/s; 6152/338328 objects degraded (1.818%); 3714/338328 objects misplaced (1.098%); 4.3 MiB/s, 69 objects/s recovering 2024-09-21T10:14:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:50 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 6152/338328 objects degraded (1.818%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:14:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:50 smithi196 ceph-mon[84784]: osdmap e232: 8 total, 8 up, 7 in 2024-09-21T10:14:51.432 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:51.455 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:51.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:51.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:51.557 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:51.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:51.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:51.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:51.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:51.713 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:51.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:51.814 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:51.839 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:51.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:51.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:51.941 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:51.965 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:51.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:51.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:52.067 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:52.092 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:52.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:52.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:52.194 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:52.217 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:52.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:52.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:52.319 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:52.346 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:52.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:52.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:52.448 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:52.471 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:52.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:52.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:52.573 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:52.599 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:52.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:52.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:52.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:52.741 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:52.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:52.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:52.842 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:52.869 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:52.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:52.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:52.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:52.996 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:52.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:52.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:52 smithi098 ceph-mon[97580]: pgmap v700: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 33 MiB/s wr, 994 op/s; 6152/336003 objects degraded (1.831%); 3714/336003 objects misplaced (1.105%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:14:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:52 smithi098 ceph-mon[97580]: osdmap e233: 8 total, 8 up, 7 in 2024-09-21T10:14:53.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:53.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:52 smithi098 ceph-mon[100914]: pgmap v700: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 33 MiB/s wr, 994 op/s; 6152/336003 objects degraded (1.831%); 3714/336003 objects misplaced (1.105%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:14:53.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:52 smithi098 ceph-mon[100914]: osdmap e233: 8 total, 8 up, 7 in 2024-09-21T10:14:53.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:53.098 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:53.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:52 smithi196 ceph-mon[84784]: pgmap v700: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 33 MiB/s wr, 994 op/s; 6152/336003 objects degraded (1.831%); 3714/336003 objects misplaced (1.105%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:14:53.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:52 smithi196 ceph-mon[84784]: osdmap e233: 8 total, 8 up, 7 in 2024-09-21T10:14:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:53.139 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:53.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:53.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:53.241 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:53.267 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:53.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:53.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:53.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:53.396 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:53.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:53.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:53.497 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:53.522 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:53.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:53.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:53.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:53.662 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:53.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:53.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:53.764 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:53.789 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:53.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:53.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:53.891 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:53.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:53.925 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:53.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:54.025 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:54.046 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:54.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:54.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:54.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:53 smithi098 ceph-mon[97580]: 7.5 scrub starts 2024-09-21T10:14:54.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-09-21T10:14:54.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:53 smithi098 ceph-mon[100914]: 7.5 scrub starts 2024-09-21T10:14:54.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-09-21T10:14:54.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:53 smithi196 ceph-mon[84784]: 7.5 scrub starts 2024-09-21T10:14:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-09-21T10:14:54.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:54.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:54.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:54.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:54.296 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:54.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:54.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:54.397 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:54.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:54.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:54.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:54.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:54.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:54.544 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:54.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:54.645 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:54.666 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:54.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:54.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:54.768 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:54.789 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:54.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:54.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:54.891 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:54.911 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:54.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:54.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:55.013 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:55.034 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:55.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:55.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:55.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:54 smithi098 ceph-mon[97580]: pgmap v702: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 3.7 MiB/s wr, 901 op/s; 8838/328719 objects misplaced (2.689%); 7.0 MiB/s, 112 objects/s recovering 2024-09-21T10:14:55.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:54 smithi098 ceph-mon[97580]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6152/336003 objects degraded (1.831%), 1 pg degraded, 1 pg undersized) 2024-09-21T10:14:55.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:54 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "22"}]': finished 2024-09-21T10:14:55.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:54 smithi098 ceph-mon[97580]: osdmap e234: 8 total, 8 up, 7 in 2024-09-21T10:14:55.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:54 smithi098 ceph-mon[97580]: 7.5 scrub ok 2024-09-21T10:14:55.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:54 smithi098 ceph-mon[100914]: pgmap v702: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 3.7 MiB/s wr, 901 op/s; 8838/328719 objects misplaced (2.689%); 7.0 MiB/s, 112 objects/s recovering 2024-09-21T10:14:55.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:54 smithi098 ceph-mon[100914]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6152/336003 objects degraded (1.831%), 1 pg degraded, 1 pg undersized) 2024-09-21T10:14:55.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:54 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "22"}]': finished 2024-09-21T10:14:55.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:54 smithi098 ceph-mon[100914]: osdmap e234: 8 total, 8 up, 7 in 2024-09-21T10:14:55.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:54 smithi098 ceph-mon[100914]: 7.5 scrub ok 2024-09-21T10:14:55.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:54 smithi196 ceph-mon[84784]: pgmap v702: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 3.7 MiB/s wr, 901 op/s; 8838/328719 objects misplaced (2.689%); 7.0 MiB/s, 112 objects/s recovering 2024-09-21T10:14:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:54 smithi196 ceph-mon[84784]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6152/336003 objects degraded (1.831%), 1 pg degraded, 1 pg undersized) 2024-09-21T10:14:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:54 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "22"}]': finished 2024-09-21T10:14:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:54 smithi196 ceph-mon[84784]: osdmap e234: 8 total, 8 up, 7 in 2024-09-21T10:14:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:54 smithi196 ceph-mon[84784]: 7.5 scrub ok 2024-09-21T10:14:55.156 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:55.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:55.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:55.258 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:55.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:55.280 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:55.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:55.381 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:55.402 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:55.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:55.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:55.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:55.525 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:55.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:55.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:55.626 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:55.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:55.648 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:55.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:55.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:55.771 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:55.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:55.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:55.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:55.897 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:55.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:55.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:55.999 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:56.021 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:56.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:56.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:56.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:55 smithi098 ceph-mon[97580]: osdmap e235: 8 total, 8 up, 7 in 2024-09-21T10:14:56.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:55 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-09-21T10:14:56.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:55 smithi098 ceph-mon[100914]: osdmap e235: 8 total, 8 up, 7 in 2024-09-21T10:14:56.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:55 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-09-21T10:14:56.123 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:55 smithi196 ceph-mon[84784]: osdmap e235: 8 total, 8 up, 7 in 2024-09-21T10:14:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:55 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-09-21T10:14:56.163 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:56.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:56.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:56.265 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:56.288 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:56.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:56.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:56.390 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:56.415 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:56.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:56.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:56.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:56.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:56.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:56.643 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:56.664 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:56.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:56.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:56.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:56.787 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:56.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:56.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:56.888 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:56.911 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:56.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:56.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:56.946 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:56 smithi098 ceph-mon[97580]: pgmap v705: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 97 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 925 op/s; 8838/323247 objects misplaced (2.734%); 6.1 MiB/s, 97 objects/s recovering 2024-09-21T10:14:56.947 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:56 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "23"}]': finished 2024-09-21T10:14:56.947 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:56 smithi098 ceph-mon[97580]: osdmap e236: 8 total, 8 up, 7 in 2024-09-21T10:14:56.947 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:56 smithi098 ceph-mon[100914]: pgmap v705: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 97 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 925 op/s; 8838/323247 objects misplaced (2.734%); 6.1 MiB/s, 97 objects/s recovering 2024-09-21T10:14:56.947 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:56 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "23"}]': finished 2024-09-21T10:14:56.947 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:56 smithi098 ceph-mon[100914]: osdmap e236: 8 total, 8 up, 7 in 2024-09-21T10:14:57.013 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:57.034 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:57.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:57.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:57.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:57.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:56 smithi196 ceph-mon[84784]: pgmap v705: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 97 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 925 op/s; 8838/323247 objects misplaced (2.734%); 6.1 MiB/s, 97 objects/s recovering 2024-09-21T10:14:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:56 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "23"}]': finished 2024-09-21T10:14:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:56 smithi196 ceph-mon[84784]: osdmap e236: 8 total, 8 up, 7 in 2024-09-21T10:14:57.156 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:57.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:57.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:57.258 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:57.279 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:57.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:57.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:57.381 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:57.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:57.405 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:57.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:57.506 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:57.528 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:57.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:57.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:57.629 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:57.651 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:57.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:57.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:57.752 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:57.773 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:57.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:57.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:57.874 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:57.895 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:57.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:57.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:57.997 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:58.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:58.024 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:58.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:58.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:57 smithi098 ceph-mon[97580]: osdmap e237: 8 total, 8 up, 7 in 2024-09-21T10:14:58.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:57 smithi098 ceph-mon[100914]: osdmap e237: 8 total, 8 up, 7 in 2024-09-21T10:14:58.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:58.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:57 smithi196 ceph-mon[84784]: osdmap e237: 8 total, 8 up, 7 in 2024-09-21T10:14:58.162 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:58.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:58.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:58.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:58.305 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:58.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:58.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:58.408 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:58.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:58.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:58.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:58.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:58.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:58.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:58.677 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:58.712 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:58.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:58.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:58.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:58.860 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:58.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:58.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:58.962 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:58.989 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:14:58.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:58.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:59.091 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:59.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:58 smithi098 ceph-mon[97580]: pgmap v708: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 96 active+clean; 6.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 739 op/s; 8205/319848 objects misplaced (2.565%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:14:59.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:58 smithi098 ceph-mon[97580]: osdmap e238: 8 total, 8 up, 7 in 2024-09-21T10:14:59.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:58 smithi098 ceph-mon[100914]: pgmap v708: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 96 active+clean; 6.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 739 op/s; 8205/319848 objects misplaced (2.565%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:14:59.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:58 smithi098 ceph-mon[100914]: osdmap e238: 8 total, 8 up, 7 in 2024-09-21T10:14:59.128 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:59.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:59.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:59.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:58 smithi196 ceph-mon[84784]: pgmap v708: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 96 active+clean; 6.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 739 op/s; 8205/319848 objects misplaced (2.565%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:14:59.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:58 smithi196 ceph-mon[84784]: osdmap e238: 8 total, 8 up, 7 in 2024-09-21T10:14:59.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:59.267 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:59.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:59.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:59.368 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:59.389 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:59.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:59.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:59.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:59.528 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:59.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:59.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:59.630 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:59.650 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:59.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:59.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:59.752 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:59.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:59.774 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:59.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:59.875 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:59.897 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:14:59.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:59.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:59.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:00.009 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:14:59 smithi098 ceph-mon[97580]: Health check failed: Degraded data redundancy: 3629/317952 objects degraded (1.141%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:15:00.009 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:14:59 smithi098 ceph-mon[100914]: Health check failed: Degraded data redundancy: 3629/317952 objects degraded (1.141%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:15:00.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:00.045 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:00.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:00.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:14:59 smithi196 ceph-mon[84784]: Health check failed: Degraded data redundancy: 3629/317952 objects degraded (1.141%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:15:00.146 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:00.171 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:00.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:00.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:00.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:00.297 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:00.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:00.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:00.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:00.423 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:00.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:00.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:00.464 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:00.467 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:00.525 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:00.550 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:00.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:00.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:00.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:15:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:15:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:15:00.653 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:00.677 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:00.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:00.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:00.768 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:15:00.779 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:00.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:00.834 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:00.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:00.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:15:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:15:00] "GET /metrics HTTP/1.1" 200 36587 "" "Prometheus/2.51.0" 2024-09-21T10:15:00.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:00 smithi098 ceph-mon[97580]: pgmap v710: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 1 active+recovering+undersized+remapped, 96 active+clean; 6.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 3629/317952 objects degraded (1.141%); 8205/317952 objects misplaced (2.581%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:15:00.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:00 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1939683467' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:15:00.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:00 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:15:00.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:00 smithi098 ceph-mon[100914]: pgmap v710: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 1 active+recovering+undersized+remapped, 96 active+clean; 6.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 3629/317952 objects degraded (1.141%); 8205/317952 objects misplaced (2.581%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:15:00.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:00 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1939683467' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:15:00.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:00 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:15:00.935 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:00.960 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:00.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:00.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:01.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:01.088 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:01.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:01.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:00 smithi196 ceph-mon[84784]: pgmap v710: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 1 active+recovering+undersized+remapped, 96 active+clean; 6.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 3629/317952 objects degraded (1.141%); 8205/317952 objects misplaced (2.581%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:15:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:00 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1939683467' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:15:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:00 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:15:01.190 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:01.214 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:01.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:01.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:01.316 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:01.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:01.343 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:01.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:01.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:01.484 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:01.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:01.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:01.586 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:01.633 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:01.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:01.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:01.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:01.781 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:01.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:01.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:01.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:01 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:15:01.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:01 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:15:01.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:01 smithi098 ceph-mon[97580]: osdmap e239: 8 total, 8 up, 7 in 2024-09-21T10:15:01.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:01 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:15:01.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:01 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:15:01.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:01 smithi098 ceph-mon[100914]: osdmap e239: 8 total, 8 up, 7 in 2024-09-21T10:15:01.882 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:01.923 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:01.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:01.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:01.949 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:15:01.949 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T10:15:02.025 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:02.051 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:02.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:02.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:02.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:01 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:15:02.184 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:01 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:15:02.184 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:01 smithi196 ceph-mon[84784]: osdmap e239: 8 total, 8 up, 7 in 2024-09-21T10:15:02.184 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:02.208 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:02.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:02.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:02.310 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:02.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:02.362 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:02.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:02.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:02.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:02.503 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:02.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:02.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:02.646 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:02.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:02.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:02.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:02.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:02.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:02.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:02 smithi098 ceph-mon[97580]: pgmap v712: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 1 active+recovering+undersized+remapped, 96 active+clean; 6.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 3629/316902 objects degraded (1.145%); 8205/316902 objects misplaced (2.589%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:15:02.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:02 smithi098 ceph-mon[100914]: pgmap v712: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 1 active+recovering+undersized+remapped, 96 active+clean; 6.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 3629/316902 objects degraded (1.145%); 8205/316902 objects misplaced (2.589%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:15:02.890 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:02.914 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:02.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:02.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:03.015 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:03.041 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:03.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:03.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:03.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:02 smithi196 ceph-mon[84784]: pgmap v712: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 1 active+recovering+undersized+remapped, 96 active+clean; 6.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 3629/316902 objects degraded (1.145%); 8205/316902 objects misplaced (2.589%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:15:03.143 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:03.179 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:03.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:03.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:03.281 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:03.324 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:03.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:03.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:03.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:03.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:03.472 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:03.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:03.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:15:03.573 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:03.596 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:03.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:03.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:03.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:03.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:03.752 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:03.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:03.853 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:03.876 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:03.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:03.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:03.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:04.028 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:04.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:04.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:04.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:04.162 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:04.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:04.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:04.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:04.307 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:04.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:04.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:04.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:04.493 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:04.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:04.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:04.595 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:04.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:04.648 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:04.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:04.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:04.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:04.789 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:04.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:04.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:04.911 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:04.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:04.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:05.012 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:05.036 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:05.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:05.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:05.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:04 smithi098 ceph-mon[97580]: pgmap v713: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 7167/310629 objects degraded (2.307%); 7776/310629 objects misplaced (2.503%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T10:15:05.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:04 smithi098 ceph-mon[100914]: pgmap v713: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 7167/310629 objects degraded (2.307%); 7776/310629 objects misplaced (2.503%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T10:15:05.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:04 smithi196 ceph-mon[84784]: pgmap v713: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 7167/310629 objects degraded (2.307%); 7776/310629 objects misplaced (2.503%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T10:15:05.176 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:05.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:05.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:05.279 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:05.323 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:05.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:05.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:05.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:05.471 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:05.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:05.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:05.573 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:05.611 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:05.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:05.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:05.713 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:05.739 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:05.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:05.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:05.841 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:05.862 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:05.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:05.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:05.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:05.988 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:05.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:05.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:06.089 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:06.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:05 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 7167/310629 objects degraded (2.307%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:15:06.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:05 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 7167/310629 objects degraded (2.307%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:15:06.111 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:06.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:06.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:06.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:05 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 7167/310629 objects degraded (2.307%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:15:06.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:06.233 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:06.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:06.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:06.335 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:06.357 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:06.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:06.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:06.459 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:06.481 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:06.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:06.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:06.582 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:06.606 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:06.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:06.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:06.707 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:06.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:06.742 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:06.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:06.822 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:15:06.822 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-21T10:15:06.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-21T10:15:06.822 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:15:06.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.822 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:267693 5.0 5:02a5fe86:::benchmark_data_smithi098_135187_object132055:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:15:06.822 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:59.398968+0000", 2024-09-21T10:15:06.823 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 426.15026300099998, 2024-09-21T10:15:06.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.363874005, 2024-09-21T10:15:06.823 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:15:06.823 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:15:06.823 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:06.823 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:15:06.823 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:15:06.823 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:15:06.823 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 267693 2024-09-21T10:15:06.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.823 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:15:06.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:15:06.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.398968+0000", 2024-09-21T10:15:06.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:15:06.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.398968+0000", 2024-09-21T10:15:06.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:15:06.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.398970+0000", 2024-09-21T10:15:06.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0310000000000001e-06 2024-09-21T10:15:06.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:15:06.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.398977+0000", 2024-09-21T10:15:06.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1559999999999998e-06 2024-09-21T10:15:06.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:15:06.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.398999+0000", 2024-09-21T10:15:06.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1103000000000001e-05 2024-09-21T10:15:06.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:15:06.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399008+0000", 2024-09-21T10:15:06.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6350000000000001e-06 2024-09-21T10:15:06.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399115+0000", 2024-09-21T10:15:06.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106959 2024-09-21T10:15:06.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399130+0000", 2024-09-21T10:15:06.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4918e-05 2024-09-21T10:15:06.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.372566+0000", 2024-09-21T10:15:06.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.97343550599999995 2024-09-21T10:15:06.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.372578+0000", 2024-09-21T10:15:06.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2269000000000001e-05 2024-09-21T10:15:06.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.705405+0000", 2024-09-21T10:15:06.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3328273150000001 2024-09-21T10:15:06.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:15:06.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.705537+0000", 2024-09-21T10:15:06.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000131891 2024-09-21T10:15:06.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,1,4", 2024-09-21T10:15:06.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.706214+0000", 2024-09-21T10:15:06.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00067717799999999996 2024-09-21T10:15:06.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.740214+0000", 2024-09-21T10:15:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034000056000000001 2024-09-21T10:15:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.740297+0000", 2024-09-21T10:15:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2787999999999994e-05 2024-09-21T10:15:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:15:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.741535+0000", 2024-09-21T10:15:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001237602 2024-09-21T10:15:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.762486+0000", 2024-09-21T10:15:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020951805 2024-09-21T10:15:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:15:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.762689+0000", 2024-09-21T10:15:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020263099999999999 2024-09-21T10:15:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:15:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.762842+0000", 2024-09-21T10:15:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015316200000000001 2024-09-21T10:15:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:15:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:267694 5.1 5:88acc045:::benchmark_data_smithi098_135187_object132056:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:15:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:59.399033+0000", 2024-09-21T10:15:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 426.15019830799997, 2024-09-21T10:15:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2681572970000001, 2024-09-21T10:15:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:15:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:15:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:15:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:15:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:15:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 267694 2024-09-21T10:15:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:15:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:15:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399033+0000", 2024-09-21T10:15:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:15:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399033+0000", 2024-09-21T10:15:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:15:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399034+0000", 2024-09-21T10:15:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7800000000000001e-07 2024-09-21T10:15:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:15:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399036+0000", 2024-09-21T10:15:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8590000000000001e-06 2024-09-21T10:15:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:15:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399045+0000", 2024-09-21T10:15:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8950000000000004e-06 2024-09-21T10:15:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:15:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399051+0000", 2024-09-21T10:15:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6540000000000004e-06 2024-09-21T10:15:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399172+0000", 2024-09-21T10:15:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000121465 2024-09-21T10:15:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.399182+0000", 2024-09-21T10:15:06.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9199999999999999e-06 2024-09-21T10:15:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.377429+0000", 2024-09-21T10:15:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.97824708500000002 2024-09-21T10:15:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.377440+0000", 2024-09-21T10:15:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0424999999999999e-05 2024-09-21T10:15:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.646211+0000", 2024-09-21T10:15:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2687708520000001 2024-09-21T10:15:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:15:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.646342+0000", 2024-09-21T10:15:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013128700000000001 2024-09-21T10:15:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4,7", 2024-09-21T10:15:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.646466+0000", 2024-09-21T10:15:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012438799999999999 2024-09-21T10:15:06.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.652333+0000", 2024-09-21T10:15:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058668069999999999 2024-09-21T10:15:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:15:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.661482+0000", 2024-09-21T10:15:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0091486180000000007 2024-09-21T10:15:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.662947+0000", 2024-09-21T10:15:06.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001465638 2024-09-21T10:15:06.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.666986+0000", 2024-09-21T10:15:06.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040387859999999999 2024-09-21T10:15:06.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:15:06.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.667053+0000", 2024-09-21T10:15:06.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6304999999999996e-05 2024-09-21T10:15:06.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:15:06.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.667190+0000", 2024-09-21T10:15:06.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000137735 2024-09-21T10:15:06.840 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.840 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:15:06.840 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.840 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:267704 5.1 5:8e7993d3:::benchmark_data_smithi098_135187_object132066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:15:06.840 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:07:59.421065+0000", 2024-09-21T10:15:06.840 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 426.12816672100001, 2024-09-21T10:15:06.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.250816414, 2024-09-21T10:15:06.840 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:15:06.841 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:15:06.841 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:06.841 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:15:06.841 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:15:06.841 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:15:06.841 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 267704 2024-09-21T10:15:06.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.841 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:15:06.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:15:06.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421065+0000", 2024-09-21T10:15:06.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:15:06.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421065+0000", 2024-09-21T10:15:06.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:15:06.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421066+0000", 2024-09-21T10:15:06.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.871e-06 2024-09-21T10:15:06.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:15:06.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421073+0000", 2024-09-21T10:15:06.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9059999999999996e-06 2024-09-21T10:15:06.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:15:06.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421091+0000", 2024-09-21T10:15:06.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7703e-05 2024-09-21T10:15:06.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:15:06.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421099+0000", 2024-09-21T10:15:06.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.464e-06 2024-09-21T10:15:06.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421173+0000", 2024-09-21T10:15:06.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4025999999999999e-05 2024-09-21T10:15:06.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:07:59.421180+0000", 2024-09-21T10:15:06.844 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9240000000000007e-06 2024-09-21T10:15:06.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.380372+0000", 2024-09-21T10:15:06.844 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.95919200400000004 2024-09-21T10:15:06.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:00.380384+0000", 2024-09-21T10:15:06.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1955e-05 2024-09-21T10:15:06.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.652368+0000", 2024-09-21T10:15:06.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2719833110000001 2024-09-21T10:15:06.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:15:06.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.652497+0000", 2024-09-21T10:15:06.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012972 2024-09-21T10:15:06.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4,7", 2024-09-21T10:15:06.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.652632+0000", 2024-09-21T10:15:06.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013412799999999999 2024-09-21T10:15:06.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:15:06.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.661488+0000", 2024-09-21T10:15:06.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0088560349999999999 2024-09-21T10:15:06.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.661596+0000", 2024-09-21T10:15:06.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108306 2024-09-21T10:15:06.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.667234+0000", 2024-09-21T10:15:06.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056377440000000001 2024-09-21T10:15:06.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.671567+0000", 2024-09-21T10:15:06.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043329320000000003 2024-09-21T10:15:06.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:15:06.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.671746+0000", 2024-09-21T10:15:06.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017953199999999999 2024-09-21T10:15:06.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:15:06.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:01.671881+0000", 2024-09-21T10:15:06.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013485299999999999 2024-09-21T10:15:06.848 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.848 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:15:06.848 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.848 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.44894.0:269031 5.0 5:098330c7:::benchmark_data_smithi098_135187_object133393:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-21T10:15:06.848 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:08:04.462832+0000", 2024-09-21T10:15:06.849 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 421.08639895499999, 2024-09-21T10:15:06.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.042491235, 2024-09-21T10:15:06.849 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:15:06.849 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:15:06.849 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:06.849 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:15:06.849 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.44894", 2024-09-21T10:15:06.849 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:15:06.849 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 269031 2024-09-21T10:15:06.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.849 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:15:06.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:15:06.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462832+0000", 2024-09-21T10:15:06.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:15:06.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462832+0000", 2024-09-21T10:15:06.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:15:06.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462834+0000", 2024-09-21T10:15:06.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.68e-06 2024-09-21T10:15:06.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:15:06.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462840+0000", 2024-09-21T10:15:06.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6670000000000001e-06 2024-09-21T10:15:06.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:15:06.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462857+0000", 2024-09-21T10:15:06.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7028999999999999e-05 2024-09-21T10:15:06.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:15:06.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462866+0000", 2024-09-21T10:15:06.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9080000000000001e-06 2024-09-21T10:15:06.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.852 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.852 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462933+0000", 2024-09-21T10:15:06.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7100999999999993e-05 2024-09-21T10:15:06.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.852 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.852 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:04.462943+0000", 2024-09-21T10:15:06.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0098999999999999e-05 2024-09-21T10:15:06.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.852 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.852 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.489116+0000", 2024-09-21T10:15:06.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0261731869999999 2024-09-21T10:15:06.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:15:06.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.489223+0000", 2024-09-21T10:15:06.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106709 2024-09-21T10:15:06.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:15:06.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.489331+0000", 2024-09-21T10:15:06.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010831399999999999 2024-09-21T10:15:06.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:15:06.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.498893+0000", 2024-09-21T10:15:06.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0095622350000000005 2024-09-21T10:15:06.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.500295+0000", 2024-09-21T10:15:06.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0014013249999999999 2024-09-21T10:15:06.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.505231+0000", 2024-09-21T10:15:06.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0049369000000000001 2024-09-21T10:15:06.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:15:06.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.505284+0000", 2024-09-21T10:15:06.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2510999999999999e-05 2024-09-21T10:15:06.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:15:06.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:08:05.505324+0000", 2024-09-21T10:15:06.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9570000000000002e-05 2024-09-21T10:15:06.855 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.855 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:15:06.855 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.856 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:207172 6.6 6:619aed3e:::benchmark_data_smithi098_169033_object84513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-21T10:15:06.856 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:11:53.076631+0000", 2024-09-21T10:15:06.856 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 192.47260010299999, 2024-09-21T10:15:06.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.230844351, 2024-09-21T10:15:06.856 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:15:06.856 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:15:06.856 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:06.856 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:15:06.856 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:15:06.856 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:15:06.856 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 207172 2024-09-21T10:15:06.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.856 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:15:06.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:15:06.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.076631+0000", 2024-09-21T10:15:06.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:15:06.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.076631+0000", 2024-09-21T10:15:06.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:15:06.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.076633+0000", 2024-09-21T10:15:06.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1780000000000002e-06 2024-09-21T10:15:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:15:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.076639+0000", 2024-09-21T10:15:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9259999999999997e-06 2024-09-21T10:15:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:15:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.076654+0000", 2024-09-21T10:15:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4229999999999999e-05 2024-09-21T10:15:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:15:06.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.076661+0000", 2024-09-21T10:15:06.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5900000000000002e-06 2024-09-21T10:15:06.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.076731+0000", 2024-09-21T10:15:06.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0034000000000001e-05 2024-09-21T10:15:06.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.076742+0000", 2024-09-21T10:15:06.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0703999999999999e-05 2024-09-21T10:15:06.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046148+0000", 2024-09-21T10:15:06.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.96940612900000001 2024-09-21T10:15:06.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046161+0000", 2024-09-21T10:15:06.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3327e-05 2024-09-21T10:15:06.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.294291+0000", 2024-09-21T10:15:06.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2481292340000001 2024-09-21T10:15:06.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:15:06.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.294395+0000", 2024-09-21T10:15:06.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010417900000000001 2024-09-21T10:15:06.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6,7", 2024-09-21T10:15:06.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.294482+0000", 2024-09-21T10:15:06.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7445e-05 2024-09-21T10:15:06.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.301624+0000", 2024-09-21T10:15:06.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0071413900000000001 2024-09-21T10:15:06.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:15:06.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.301633+0000", 2024-09-21T10:15:06.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6169999999999996e-06 2024-09-21T10:15:06.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.307326+0000", 2024-09-21T10:15:06.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056926199999999998 2024-09-21T10:15:06.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.863 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.307375+0000", 2024-09-21T10:15:06.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9635999999999997e-05 2024-09-21T10:15:06.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:15:06.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.307411+0000", 2024-09-21T10:15:06.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5543000000000002e-05 2024-09-21T10:15:06.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:15:06.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.307476+0000", 2024-09-21T10:15:06.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4568999999999998e-05 2024-09-21T10:15:06.865 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.865 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:15:06.865 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.866 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:207183 6.6 6:6414a58d:::benchmark_data_smithi098_169033_object84524:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-21T10:15:06.866 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:11:53.086677+0000", 2024-09-21T10:15:06.866 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 192.462554066, 2024-09-21T10:15:06.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2283806359999998, 2024-09-21T10:15:06.866 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:15:06.866 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:15:06.866 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:06.867 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:15:06.867 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:15:06.867 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:15:06.867 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 207183 2024-09-21T10:15:06.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.867 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:15:06.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:15:06.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.086677+0000", 2024-09-21T10:15:06.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:15:06.868 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.086677+0000", 2024-09-21T10:15:06.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:15:06.868 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.086678+0000", 2024-09-21T10:15:06.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3099999999999997e-07 2024-09-21T10:15:06.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:15:06.868 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.086681+0000", 2024-09-21T10:15:06.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2260000000000001e-06 2024-09-21T10:15:06.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:15:06.869 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.086689+0000", 2024-09-21T10:15:06.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1519999999999992e-06 2024-09-21T10:15:06.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.869 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:15:06.869 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.086695+0000", 2024-09-21T10:15:06.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9800000000000003e-06 2024-09-21T10:15:06.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.869 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.869 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.086753+0000", 2024-09-21T10:15:06.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8131999999999998e-05 2024-09-21T10:15:06.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.086762+0000", 2024-09-21T10:15:06.870 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7110000000000001e-06 2024-09-21T10:15:06.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046212+0000", 2024-09-21T10:15:06.870 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.95944979900000005 2024-09-21T10:15:06.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046221+0000", 2024-09-21T10:15:06.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8599999999999999e-06 2024-09-21T10:15:06.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.297010+0000", 2024-09-21T10:15:06.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.250789116 2024-09-21T10:15:06.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:15:06.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.297086+0000", 2024-09-21T10:15:06.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5871000000000006e-05 2024-09-21T10:15:06.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6,7", 2024-09-21T10:15:06.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.297174+0000", 2024-09-21T10:15:06.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8177999999999995e-05 2024-09-21T10:15:06.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.304453+0000", 2024-09-21T10:15:06.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0072791820000000004 2024-09-21T10:15:06.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:15:06.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.305910+0000", 2024-09-21T10:15:06.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001456854 2024-09-21T10:15:06.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.873 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.307508+0000", 2024-09-21T10:15:06.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001598185 2024-09-21T10:15:06.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.873 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.314945+0000", 2024-09-21T10:15:06.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0074369140000000002 2024-09-21T10:15:06.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:15:06.873 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.314982+0000", 2024-09-21T10:15:06.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6777999999999999e-05 2024-09-21T10:15:06.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:15:06.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.315058+0000", 2024-09-21T10:15:06.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6067000000000002e-05 2024-09-21T10:15:06.874 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.874 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:15:06.874 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.875 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:207211 6.6 6:65970d3a:::benchmark_data_smithi098_169033_object84552:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-21T10:15:06.875 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:11:53.165609+0000", 2024-09-21T10:15:06.875 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 192.383622718, 2024-09-21T10:15:06.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1496655379999998, 2024-09-21T10:15:06.875 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:15:06.875 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:15:06.875 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:06.875 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:15:06.875 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:15:06.875 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:15:06.875 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 207211 2024-09-21T10:15:06.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.876 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:15:06.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:15:06.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.165609+0000", 2024-09-21T10:15:06.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:15:06.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.165609+0000", 2024-09-21T10:15:06.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:15:06.877 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.165609+0000", 2024-09-21T10:15:06.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8000000000000001e-07 2024-09-21T10:15:06.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.877 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:15:06.877 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.165611+0000", 2024-09-21T10:15:06.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2019999999999998e-06 2024-09-21T10:15:06.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.877 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:15:06.877 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.165618+0000", 2024-09-21T10:15:06.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4999999999999996e-06 2024-09-21T10:15:06.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:15:06.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.165623+0000", 2024-09-21T10:15:06.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8679999999999998e-06 2024-09-21T10:15:06.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.165681+0000", 2024-09-21T10:15:06.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8572000000000001e-05 2024-09-21T10:15:06.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.165692+0000", 2024-09-21T10:15:06.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0519e-05 2024-09-21T10:15:06.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046279+0000", 2024-09-21T10:15:06.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.88058784999999995 2024-09-21T10:15:06.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046289+0000", 2024-09-21T10:15:06.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1670000000000005e-06 2024-09-21T10:15:06.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.298633+0000", 2024-09-21T10:15:06.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.252344116 2024-09-21T10:15:06.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:15:06.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.298708+0000", 2024-09-21T10:15:06.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4832000000000005e-05 2024-09-21T10:15:06.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6,7", 2024-09-21T10:15:06.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.298783+0000", 2024-09-21T10:15:06.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5204999999999996e-05 2024-09-21T10:15:06.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:15:06.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.305913+0000", 2024-09-21T10:15:06.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0071303139999999996 2024-09-21T10:15:06.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.307542+0000", 2024-09-21T10:15:06.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016289830000000001 2024-09-21T10:15:06.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.315133+0000", 2024-09-21T10:15:06.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0075910140000000001 2024-09-21T10:15:06.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.882 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.882 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.315177+0000", 2024-09-21T10:15:06.882 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:06.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3606999999999998e-05 2024-09-21T10:15:06.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:15:06.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.315208+0000", 2024-09-21T10:15:06.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1327e-05 2024-09-21T10:15:06.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:15:06.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.315274+0000", 2024-09-21T10:15:06.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6081999999999995e-05 2024-09-21T10:15:06.884 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.884 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:15:06.884 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.885 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:207214 6.6 6:6a147d09:::benchmark_data_smithi098_169033_object84555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-21T10:15:06.885 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:11:53.175320+0000", 2024-09-21T10:15:06.885 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 192.37391101, 2024-09-21T10:15:06.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.144742419, 2024-09-21T10:15:06.885 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:15:06.885 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:15:06.885 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:06.885 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:15:06.885 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:15:06.885 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:15:06.885 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 207214 2024-09-21T10:15:06.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.885 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:15:06.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.886 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:15:06.886 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.175320+0000", 2024-09-21T10:15:06.886 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.886 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:15:06.886 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.175320+0000", 2024-09-21T10:15:06.886 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.886 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:15:06.886 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.175322+0000", 2024-09-21T10:15:06.887 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5030000000000001e-06 2024-09-21T10:15:06.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.887 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:15:06.887 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.175326+0000", 2024-09-21T10:15:06.887 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7670000000000001e-06 2024-09-21T10:15:06.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.887 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:15:06.887 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.175336+0000", 2024-09-21T10:15:06.887 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.057e-05 2024-09-21T10:15:06.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.887 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:15:06.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.175341+0000", 2024-09-21T10:15:06.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.378e-06 2024-09-21T10:15:06.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.888 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.175402+0000", 2024-09-21T10:15:06.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0661000000000001e-05 2024-09-21T10:15:06.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.888 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.175412+0000", 2024-09-21T10:15:06.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.38e-06 2024-09-21T10:15:06.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046318+0000", 2024-09-21T10:15:06.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.87090624100000003 2024-09-21T10:15:06.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046322+0000", 2024-09-21T10:15:06.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1790000000000001e-06 2024-09-21T10:15:06.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.300117+0000", 2024-09-21T10:15:06.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2537953499999999 2024-09-21T10:15:06.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:15:06.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.300185+0000", 2024-09-21T10:15:06.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7362999999999999e-05 2024-09-21T10:15:06.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6,7", 2024-09-21T10:15:06.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.300257+0000", 2024-09-21T10:15:06.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2193999999999994e-05 2024-09-21T10:15:06.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:15:06.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.305917+0000", 2024-09-21T10:15:06.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056601009999999998 2024-09-21T10:15:06.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.891 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.891 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.311194+0000", 2024-09-21T10:15:06.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005277286 2024-09-21T10:15:06.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.891 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.891 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.315344+0000", 2024-09-21T10:15:06.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0041494440000000004 2024-09-21T10:15:06.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.319943+0000", 2024-09-21T10:15:06.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0045997449999999997 2024-09-21T10:15:06.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:15:06.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.319981+0000", 2024-09-21T10:15:06.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7355999999999999e-05 2024-09-21T10:15:06.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:15:06.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.320063+0000", 2024-09-21T10:15:06.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1901000000000001e-05 2024-09-21T10:15:06.892 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.893 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:15:06.893 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.893 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:207215 6.6 6:67c7f41a:::benchmark_data_smithi098_169033_object84556:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-21T10:15:06.893 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:11:53.176041+0000", 2024-09-21T10:15:06.893 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 192.37319023800001, 2024-09-21T10:15:06.893 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1443163849999998, 2024-09-21T10:15:06.893 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:15:06.893 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:15:06.893 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:06.893 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:15:06.893 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:15:06.893 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:15:06.894 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 207215 2024-09-21T10:15:06.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.894 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:15:06.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.894 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:15:06.894 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.176041+0000", 2024-09-21T10:15:06.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.894 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:15:06.894 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.176041+0000", 2024-09-21T10:15:06.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.895 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:15:06.895 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.176042+0000", 2024-09-21T10:15:06.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.003e-06 2024-09-21T10:15:06.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.895 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:15:06.895 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.176046+0000", 2024-09-21T10:15:06.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.968e-06 2024-09-21T10:15:06.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.895 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:15:06.895 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.176057+0000", 2024-09-21T10:15:06.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0699e-05 2024-09-21T10:15:06.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:15:06.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.176063+0000", 2024-09-21T10:15:06.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7000000000000002e-06 2024-09-21T10:15:06.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.176117+0000", 2024-09-21T10:15:06.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3820999999999999e-05 2024-09-21T10:15:06.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.176125+0000", 2024-09-21T10:15:06.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1589999999999993e-06 2024-09-21T10:15:06.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.897 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.897 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046347+0000", 2024-09-21T10:15:06.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.87022157300000003 2024-09-21T10:15:06.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.897 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.897 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046351+0000", 2024-09-21T10:15:06.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2139999999999998e-06 2024-09-21T10:15:06.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.897 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.897 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.301660+0000", 2024-09-21T10:15:06.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2553083840000001 2024-09-21T10:15:06.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.898 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:15:06.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.301728+0000", 2024-09-21T10:15:06.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8836000000000003e-05 2024-09-21T10:15:06.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.898 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6,7", 2024-09-21T10:15:06.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.301803+0000", 2024-09-21T10:15:06.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4969999999999995e-05 2024-09-21T10:15:06.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.899 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:15:06.899 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.307298+0000", 2024-09-21T10:15:06.899 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054946079999999998 2024-09-21T10:15:06.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.899 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.899 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.311329+0000", 2024-09-21T10:15:06.899 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040308499999999999 2024-09-21T10:15:06.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.899 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.899 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.320130+0000", 2024-09-21T10:15:06.899 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0088008129999999993 2024-09-21T10:15:06.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.900 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.900 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.320188+0000", 2024-09-21T10:15:06.900 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8489000000000002e-05 2024-09-21T10:15:06.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.900 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:15:06.900 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.320247+0000", 2024-09-21T10:15:06.900 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8390000000000002e-05 2024-09-21T10:15:06.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.900 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:15:06.900 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.320357+0000", 2024-09-21T10:15:06.900 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000110908 2024-09-21T10:15:06.901 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.901 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:15:06.901 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.901 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:207219 6.6 6:6df4633d:::benchmark_data_smithi098_169033_object84560:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-21T10:15:06.901 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:11:53.176933+0000", 2024-09-21T10:15:06.901 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 192.37229858500001, 2024-09-21T10:15:06.901 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1502119469999998, 2024-09-21T10:15:06.901 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:15:06.901 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:15:06.901 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:06.902 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:15:06.902 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:15:06.902 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:15:06.902 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 207219 2024-09-21T10:15:06.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.902 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:15:06.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:15:06.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.176933+0000", 2024-09-21T10:15:06.902 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:15:06.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.176933+0000", 2024-09-21T10:15:06.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:15:06.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.176934+0000", 2024-09-21T10:15:06.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7499999999999998e-07 2024-09-21T10:15:06.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:15:06.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.176938+0000", 2024-09-21T10:15:06.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2270000000000004e-06 2024-09-21T10:15:06.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:15:06.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.176949+0000", 2024-09-21T10:15:06.904 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0708e-05 2024-09-21T10:15:06.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:15:06.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.176955+0000", 2024-09-21T10:15:06.904 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7769999999999997e-06 2024-09-21T10:15:06.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.181058+0000", 2024-09-21T10:15:06.904 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0041025300000000001 2024-09-21T10:15:06.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.905 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.905 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.181069+0000", 2024-09-21T10:15:06.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1395e-05 2024-09-21T10:15:06.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.905 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.905 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046372+0000", 2024-09-21T10:15:06.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.86530233599999995 2024-09-21T10:15:06.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.905 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.905 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046375+0000", 2024-09-21T10:15:06.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7299999999999999e-06 2024-09-21T10:15:06.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.906 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.906 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.303013+0000", 2024-09-21T10:15:06.906 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2566375459999999 2024-09-21T10:15:06.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.906 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:15:06.906 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.303078+0000", 2024-09-21T10:15:06.906 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4840999999999998e-05 2024-09-21T10:15:06.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.907 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6,7", 2024-09-21T10:15:06.907 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.303152+0000", 2024-09-21T10:15:06.907 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4547000000000003e-05 2024-09-21T10:15:06.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.907 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:15:06.907 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.307301+0000", 2024-09-21T10:15:06.907 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0041489079999999998 2024-09-21T10:15:06.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.907 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.907 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.311374+0000", 2024-09-21T10:15:06.907 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004072783 2024-09-21T10:15:06.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.908 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.908 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.320420+0000", 2024-09-21T10:15:06.908 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0090457279999999994 2024-09-21T10:15:06.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.908 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.908 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.326812+0000", 2024-09-21T10:15:06.908 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0063923089999999997 2024-09-21T10:15:06.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.909 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:15:06.909 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.326847+0000", 2024-09-21T10:15:06.909 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4858999999999999e-05 2024-09-21T10:15:06.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.909 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:15:06.909 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.327145+0000", 2024-09-21T10:15:06.909 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00029774800000000002 2024-09-21T10:15:06.909 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.909 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:15:06.909 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.909 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:207222 6.6 6:6c39d6bf:::benchmark_data_smithi098_169033_object84563:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-21T10:15:06.910 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:11:53.180866+0000", 2024-09-21T10:15:06.910 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 192.36836521800001, 2024-09-21T10:15:06.910 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1465361569999999, 2024-09-21T10:15:06.910 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:15:06.910 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:15:06.910 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:06.910 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:15:06.910 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:15:06.910 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:15:06.910 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 207222 2024-09-21T10:15:06.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.910 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:15:06.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:15:06.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.180866+0000", 2024-09-21T10:15:06.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:15:06.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.180866+0000", 2024-09-21T10:15:06.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:15:06.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.180867+0000", 2024-09-21T10:15:06.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.232e-06 2024-09-21T10:15:06.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.912 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:15:06.912 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.180872+0000", 2024-09-21T10:15:06.912 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8010000000000003e-06 2024-09-21T10:15:06.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:15:06.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.180882+0000", 2024-09-21T10:15:06.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.013e-05 2024-09-21T10:15:06.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:15:06.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.180889+0000", 2024-09-21T10:15:06.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9199999999999998e-06 2024-09-21T10:15:06.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.914 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.181096+0000", 2024-09-21T10:15:06.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000206935 2024-09-21T10:15:06.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.915 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.915 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.181103+0000", 2024-09-21T10:15:06.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1019999999999999e-06 2024-09-21T10:15:06.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.915 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046394+0000", 2024-09-21T10:15:06.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.86529127900000002 2024-09-21T10:15:06.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046398+0000", 2024-09-21T10:15:06.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5700000000000001e-06 2024-09-21T10:15:06.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.917 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.917 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.304492+0000", 2024-09-21T10:15:06.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2580938129999999 2024-09-21T10:15:06.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.917 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:15:06.917 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.304567+0000", 2024-09-21T10:15:06.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4809999999999997e-05 2024-09-21T10:15:06.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.917 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6,7", 2024-09-21T10:15:06.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.304644+0000", 2024-09-21T10:15:06.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7449999999999999e-05 2024-09-21T10:15:06.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:15:06.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.307335+0000", 2024-09-21T10:15:06.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026906090000000001 2024-09-21T10:15:06.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.315385+0000", 2024-09-21T10:15:06.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0080504940000000001 2024-09-21T10:15:06.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.326707+0000", 2024-09-21T10:15:06.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011322211 2024-09-21T10:15:06.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.327228+0000", 2024-09-21T10:15:06.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00052079199999999996 2024-09-21T10:15:06.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:15:06.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.327286+0000", 2024-09-21T10:15:06.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7469999999999997e-05 2024-09-21T10:15:06.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:15:06.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.327402+0000", 2024-09-21T10:15:06.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116539 2024-09-21T10:15:06.920 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.920 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:15:06.920 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.920 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:207226 6.6 6:6bc21423:::benchmark_data_smithi098_169033_object84567:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-21T10:15:06.920 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:11:53.185816+0000", 2024-09-21T10:15:06.920 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 192.36341575200001, 2024-09-21T10:15:06.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.14198311, 2024-09-21T10:15:06.921 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:15:06.921 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:15:06.921 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:06.921 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:15:06.921 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:15:06.921 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:15:06.921 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 207226 2024-09-21T10:15:06.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.921 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:15:06.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.921 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:15:06.921 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.185816+0000", 2024-09-21T10:15:06.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:15:06.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.185816+0000", 2024-09-21T10:15:06.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:15:06.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.185817+0000", 2024-09-21T10:15:06.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.685e-06 2024-09-21T10:15:06.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:15:06.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.185822+0000", 2024-09-21T10:15:06.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2059999999999998e-06 2024-09-21T10:15:06.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:15:06.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.185834+0000", 2024-09-21T10:15:06.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1800999999999999e-05 2024-09-21T10:15:06.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:15:06.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.185842+0000", 2024-09-21T10:15:06.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5020000000000003e-06 2024-09-21T10:15:06.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.185917+0000", 2024-09-21T10:15:06.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4782000000000003e-05 2024-09-21T10:15:06.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.924 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.185926+0000", 2024-09-21T10:15:06.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4660000000000005e-06 2024-09-21T10:15:06.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.924 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046417+0000", 2024-09-21T10:15:06.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.86049092999999999 2024-09-21T10:15:06.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046421+0000", 2024-09-21T10:15:06.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6430000000000001e-06 2024-09-21T10:15:06.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.305935+0000", 2024-09-21T10:15:06.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2595139959999999 2024-09-21T10:15:06.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:15:06.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.306002+0000", 2024-09-21T10:15:06.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7287999999999997e-05 2024-09-21T10:15:06.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6,7", 2024-09-21T10:15:06.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.306073+0000", 2024-09-21T10:15:06.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1151999999999999e-05 2024-09-21T10:15:06.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:15:06.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.309043+0000", 2024-09-21T10:15:06.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029701850000000002 2024-09-21T10:15:06.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.315434+0000", 2024-09-21T10:15:06.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0063911819999999996 2024-09-21T10:15:06.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.326788+0000", 2024-09-21T10:15:06.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01135358 2024-09-21T10:15:06.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.327635+0000", 2024-09-21T10:15:06.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00084689900000000002 2024-09-21T10:15:06.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:15:06.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.327692+0000", 2024-09-21T10:15:06.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7250999999999998e-05 2024-09-21T10:15:06.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:15:06.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.327799+0000", 2024-09-21T10:15:06.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106562 2024-09-21T10:15:06.928 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.929 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:15:06.929 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.929 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:207239 6.6 6:623e6f6c:::benchmark_data_smithi098_169033_object84580:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-21T10:15:06.929 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:11:53.196228+0000", 2024-09-21T10:15:06.929 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 192.35300362999999, 2024-09-21T10:15:06.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1412196209999999, 2024-09-21T10:15:06.929 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:15:06.929 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:15:06.929 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:06.929 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:15:06.929 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:15:06.929 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:15:06.930 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 207239 2024-09-21T10:15:06.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.930 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:15:06.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:15:06.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.196228+0000", 2024-09-21T10:15:06.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:15:06.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.196228+0000", 2024-09-21T10:15:06.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:15:06.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.196228+0000", 2024-09-21T10:15:06.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4499999999999996e-07 2024-09-21T10:15:06.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:15:06.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.196231+0000", 2024-09-21T10:15:06.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9670000000000001e-06 2024-09-21T10:15:06.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:15:06.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.196241+0000", 2024-09-21T10:15:06.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0101e-05 2024-09-21T10:15:06.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:15:06.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.196248+0000", 2024-09-21T10:15:06.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7710000000000001e-06 2024-09-21T10:15:06.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.196312+0000", 2024-09-21T10:15:06.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3385000000000004e-05 2024-09-21T10:15:06.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.196321+0000", 2024-09-21T10:15:06.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1910000000000006e-06 2024-09-21T10:15:06.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046440+0000", 2024-09-21T10:15:06.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.85011945200000005 2024-09-21T10:15:06.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046444+0000", 2024-09-21T10:15:06.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6729999999999998e-06 2024-09-21T10:15:06.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.307575+0000", 2024-09-21T10:15:06.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.261131145 2024-09-21T10:15:06.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:15:06.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.307644+0000", 2024-09-21T10:15:06.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8706000000000003e-05 2024-09-21T10:15:06.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6,7", 2024-09-21T10:15:06.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.307717+0000", 2024-09-21T10:15:06.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3700000000000002e-05 2024-09-21T10:15:06.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:15:06.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.311342+0000", 2024-09-21T10:15:06.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036242050000000001 2024-09-21T10:15:06.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.315472+0000", 2024-09-21T10:15:06.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0041300429999999999 2024-09-21T10:15:06.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.327861+0000", 2024-09-21T10:15:06.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012389668 2024-09-21T10:15:06.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.337283+0000", 2024-09-21T10:15:06.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0094210930000000002 2024-09-21T10:15:06.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:15:06.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.337339+0000", 2024-09-21T10:15:06.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6870000000000003e-05 2024-09-21T10:15:06.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:15:06.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.337447+0000", 2024-09-21T10:15:06.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010790600000000001 2024-09-21T10:15:06.937 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.937 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:15:06.937 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.937 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:207242 6.6 6:6ed9f933:::benchmark_data_smithi098_169033_object84583:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-21T10:15:06.937 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:11:53.201295+0000", 2024-09-21T10:15:06.938 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 192.347936049, 2024-09-21T10:15:06.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.136333542, 2024-09-21T10:15:06.938 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:15:06.938 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:15:06.938 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:06.938 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:15:06.938 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:15:06.938 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:15:06.938 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 207242 2024-09-21T10:15:06.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.938 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:15:06.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:15:06.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.201295+0000", 2024-09-21T10:15:06.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:15:06.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.201295+0000", 2024-09-21T10:15:06.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:15:06.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.201296+0000", 2024-09-21T10:15:06.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9800000000000002e-07 2024-09-21T10:15:06.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:15:06.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.201300+0000", 2024-09-21T10:15:06.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6380000000000002e-06 2024-09-21T10:15:06.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:15:06.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.201308+0000", 2024-09-21T10:15:06.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2800000000000003e-06 2024-09-21T10:15:06.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:15:06.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.201314+0000", 2024-09-21T10:15:06.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1199999999999999e-06 2024-09-21T10:15:06.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.201379+0000", 2024-09-21T10:15:06.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5011999999999994e-05 2024-09-21T10:15:06.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.201388+0000", 2024-09-21T10:15:06.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3680000000000002e-06 2024-09-21T10:15:06.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046463+0000", 2024-09-21T10:15:06.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.84507531700000005 2024-09-21T10:15:06.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046467+0000", 2024-09-21T10:15:06.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6660000000000001e-06 2024-09-21T10:15:06.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.309090+0000", 2024-09-21T10:15:06.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2626237630000001 2024-09-21T10:15:06.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:15:06.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.309229+0000", 2024-09-21T10:15:06.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013898099999999999 2024-09-21T10:15:06.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6,7", 2024-09-21T10:15:06.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.309376+0000", 2024-09-21T10:15:06.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014645399999999999 2024-09-21T10:15:06.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:15:06.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.313194+0000", 2024-09-21T10:15:06.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003818526 2024-09-21T10:15:06.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.315520+0000", 2024-09-21T10:15:06.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023256150000000001 2024-09-21T10:15:06.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.335819+0000", 2024-09-21T10:15:06.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020298742000000002 2024-09-21T10:15:06.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.337486+0000", 2024-09-21T10:15:06.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016674789999999999 2024-09-21T10:15:06.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:15:06.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.337535+0000", 2024-09-21T10:15:06.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8739000000000002e-05 2024-09-21T10:15:06.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:15:06.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.337629+0000", 2024-09-21T10:15:06.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3943999999999994e-05 2024-09-21T10:15:06.947 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.947 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:15:06.947 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.947 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:207312 6.6 6:6bfe80b6:::benchmark_data_smithi098_169033_object84653:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-21T10:15:06.947 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:11:53.297250+0000", 2024-09-21T10:15:06.947 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 192.251981199, 2024-09-21T10:15:06.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0453261770000002, 2024-09-21T10:15:06.948 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:15:06.948 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:15:06.948 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:06.948 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:15:06.948 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:15:06.948 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:15:06.948 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 207312 2024-09-21T10:15:06.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.948 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:15:06.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:15:06.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.297250+0000", 2024-09-21T10:15:06.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:15:06.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.297250+0000", 2024-09-21T10:15:06.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:15:06.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.297251+0000", 2024-09-21T10:15:06.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.156e-06 2024-09-21T10:15:06.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:15:06.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.297256+0000", 2024-09-21T10:15:06.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5700000000000003e-06 2024-09-21T10:15:06.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:15:06.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.297265+0000", 2024-09-21T10:15:06.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.312e-06 2024-09-21T10:15:06.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:15:06.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.297271+0000", 2024-09-21T10:15:06.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1580000000000002e-06 2024-09-21T10:15:06.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.297332+0000", 2024-09-21T10:15:06.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1125999999999997e-05 2024-09-21T10:15:06.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.297341+0000", 2024-09-21T10:15:06.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7849999999999997e-06 2024-09-21T10:15:06.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046486+0000", 2024-09-21T10:15:06.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.749144861 2024-09-21T10:15:06.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046490+0000", 2024-09-21T10:15:06.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.563e-06 2024-09-21T10:15:06.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.311401+0000", 2024-09-21T10:15:06.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.264911777 2024-09-21T10:15:06.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:15:06.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.311473+0000", 2024-09-21T10:15:06.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1987e-05 2024-09-21T10:15:06.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6,7", 2024-09-21T10:15:06.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.311555+0000", 2024-09-21T10:15:06.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1614999999999997e-05 2024-09-21T10:15:06.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:15:06.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.315287+0000", 2024-09-21T10:15:06.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003731631 2024-09-21T10:15:06.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.320462+0000", 2024-09-21T10:15:06.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051757019999999999 2024-09-21T10:15:06.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.337665+0000", 2024-09-21T10:15:06.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017202558999999999 2024-09-21T10:15:06.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.342415+0000", 2024-09-21T10:15:06.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0047497529999999998 2024-09-21T10:15:06.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:15:06.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.342465+0000", 2024-09-21T10:15:06.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0633999999999997e-05 2024-09-21T10:15:06.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:15:06.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.342576+0000", 2024-09-21T10:15:06.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000110988 2024-09-21T10:15:06.955 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.955 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:15:06.955 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.956 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:207329 6.6 6:60f1c0b5:::benchmark_data_smithi098_169033_object84670:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-21T10:15:06.956 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:11:53.320130+0000", 2024-09-21T10:15:06.956 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 192.22910174099999, 2024-09-21T10:15:06.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0226281670000001, 2024-09-21T10:15:06.956 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:15:06.956 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:15:06.956 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:06.956 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:15:06.956 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:15:06.956 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:15:06.956 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 207329 2024-09-21T10:15:06.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.956 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:15:06.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:15:06.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.320130+0000", 2024-09-21T10:15:06.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:15:06.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.320130+0000", 2024-09-21T10:15:06.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:15:06.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.320131+0000", 2024-09-21T10:15:06.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.327e-06 2024-09-21T10:15:06.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:15:06.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.320135+0000", 2024-09-21T10:15:06.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5850000000000001e-06 2024-09-21T10:15:06.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:15:06.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.320149+0000", 2024-09-21T10:15:06.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3922e-05 2024-09-21T10:15:06.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:15:06.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.320156+0000", 2024-09-21T10:15:06.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9059999999999996e-06 2024-09-21T10:15:06.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.320226+0000", 2024-09-21T10:15:06.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9281000000000003e-05 2024-09-21T10:15:06.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.320235+0000", 2024-09-21T10:15:06.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9849999999999995e-06 2024-09-21T10:15:06.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046509+0000", 2024-09-21T10:15:06.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.72627488299999998 2024-09-21T10:15:06.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046513+0000", 2024-09-21T10:15:06.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6859999999999999e-06 2024-09-21T10:15:06.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.313226+0000", 2024-09-21T10:15:06.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2667123840000001 2024-09-21T10:15:06.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:15:06.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.313297+0000", 2024-09-21T10:15:06.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1724999999999995e-05 2024-09-21T10:15:06.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6,7", 2024-09-21T10:15:06.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.313375+0000", 2024-09-21T10:15:06.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8066999999999996e-05 2024-09-21T10:15:06.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:15:06.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.315291+0000", 2024-09-21T10:15:06.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001915637 2024-09-21T10:15:06.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.327466+0000", 2024-09-21T10:15:06.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012175478999999999 2024-09-21T10:15:06.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.337701+0000", 2024-09-21T10:15:06.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010234981000000001 2024-09-21T10:15:06.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.342616+0000", 2024-09-21T10:15:06.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0049145530000000003 2024-09-21T10:15:06.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:15:06.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.342663+0000", 2024-09-21T10:15:06.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.676e-05 2024-09-21T10:15:06.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:15:06.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.342758+0000", 2024-09-21T10:15:06.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5006000000000005e-05 2024-09-21T10:15:06.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.963 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:15:06.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.964 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:207348 6.6 6:63a56912:::benchmark_data_smithi098_169033_object84689:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-21T10:15:06.964 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:11:53.345743+0000", 2024-09-21T10:15:06.964 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 192.20348855200001, 2024-09-21T10:15:06.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9971924649999999, 2024-09-21T10:15:06.964 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:15:06.964 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:15:06.964 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:06.964 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:15:06.964 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:15:06.964 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:15:06.964 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 207348 2024-09-21T10:15:06.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.965 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:15:06.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:15:06.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.345743+0000", 2024-09-21T10:15:06.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:15:06.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.345743+0000", 2024-09-21T10:15:06.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:15:06.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.345744+0000", 2024-09-21T10:15:06.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.404e-06 2024-09-21T10:15:06.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:15:06.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.345749+0000", 2024-09-21T10:15:06.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3569999999999997e-06 2024-09-21T10:15:06.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:15:06.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.345764+0000", 2024-09-21T10:15:06.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4796e-05 2024-09-21T10:15:06.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:15:06.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.345773+0000", 2024-09-21T10:15:06.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4279999999999996e-06 2024-09-21T10:15:06.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.345849+0000", 2024-09-21T10:15:06.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6728000000000001e-05 2024-09-21T10:15:06.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.345860+0000", 2024-09-21T10:15:06.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0750999999999999e-05 2024-09-21T10:15:06.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046532+0000", 2024-09-21T10:15:06.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.70067173699999996 2024-09-21T10:15:06.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046536+0000", 2024-09-21T10:15:06.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5719999999999999e-06 2024-09-21T10:15:06.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.315554+0000", 2024-09-21T10:15:06.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.269018625 2024-09-21T10:15:06.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:15:06.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.315629+0000", 2024-09-21T10:15:06.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4931000000000005e-05 2024-09-21T10:15:06.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6,7", 2024-09-21T10:15:06.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.315710+0000", 2024-09-21T10:15:06.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0953999999999998e-05 2024-09-21T10:15:06.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:15:06.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.320372+0000", 2024-09-21T10:15:06.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046617760000000003 2024-09-21T10:15:06.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.327526+0000", 2024-09-21T10:15:06.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0071545200000000002 2024-09-21T10:15:06.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.337736+0000", 2024-09-21T10:15:06.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010209994 2024-09-21T10:15:06.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.342795+0000", 2024-09-21T10:15:06.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050584260000000004 2024-09-21T10:15:06.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:15:06.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.342840+0000", 2024-09-21T10:15:06.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5728e-05 2024-09-21T10:15:06.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:15:06.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.342935+0000", 2024-09-21T10:15:06.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4738e-05 2024-09-21T10:15:06.971 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.971 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:15:06.972 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.972 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:207364 6.6 6:6adb1094:::benchmark_data_smithi098_169033_object84705:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-21T10:15:06.972 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:11:53.372665+0000", 2024-09-21T10:15:06.972 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 192.17656596800001, 2024-09-21T10:15:06.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.970457873, 2024-09-21T10:15:06.972 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:15:06.972 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:15:06.972 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:06.972 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:15:06.972 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:15:06.972 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:15:06.973 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 207364 2024-09-21T10:15:06.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.973 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:15:06.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:15:06.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.372665+0000", 2024-09-21T10:15:06.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:15:06.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.372665+0000", 2024-09-21T10:15:06.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:15:06.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.372667+0000", 2024-09-21T10:15:06.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.595e-06 2024-09-21T10:15:06.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:15:06.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.372672+0000", 2024-09-21T10:15:06.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2989999999999996e-06 2024-09-21T10:15:06.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:15:06.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.372685+0000", 2024-09-21T10:15:06.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2612999999999999e-05 2024-09-21T10:15:06.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:15:06.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.372691+0000", 2024-09-21T10:15:06.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7440000000000004e-06 2024-09-21T10:15:06.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.372746+0000", 2024-09-21T10:15:06.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5129e-05 2024-09-21T10:15:06.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.372756+0000", 2024-09-21T10:15:06.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9599999999999995e-06 2024-09-21T10:15:06.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046555+0000", 2024-09-21T10:15:06.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.67379976100000005 2024-09-21T10:15:06.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046559+0000", 2024-09-21T10:15:06.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5549999999999998e-06 2024-09-21T10:15:06.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.317666+0000", 2024-09-21T10:15:06.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.271106694 2024-09-21T10:15:06.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:15:06.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.317749+0000", 2024-09-21T10:15:06.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3007e-05 2024-09-21T10:15:06.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6,7", 2024-09-21T10:15:06.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.317835+0000", 2024-09-21T10:15:06.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5895000000000003e-05 2024-09-21T10:15:06.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:15:06.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.320378+0000", 2024-09-21T10:15:06.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025437239999999998 2024-09-21T10:15:06.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.327582+0000", 2024-09-21T10:15:06.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0072037029999999997 2024-09-21T10:15:06.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.338009+0000", 2024-09-21T10:15:06.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010427118000000001 2024-09-21T10:15:06.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.342972+0000", 2024-09-21T10:15:06.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0049627270000000001 2024-09-21T10:15:06.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:15:06.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.343017+0000", 2024-09-21T10:15:06.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4798e-05 2024-09-21T10:15:06.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:15:06.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.343123+0000", 2024-09-21T10:15:06.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106551 2024-09-21T10:15:06.980 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.980 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:15:06.980 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.981 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:207391 6.6 6:6281fca3:::benchmark_data_smithi098_169033_object84732:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-21T10:15:06.981 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:11:53.433540+0000", 2024-09-21T10:15:06.981 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 192.11569163799999, 2024-09-21T10:15:06.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9098065289999999, 2024-09-21T10:15:06.981 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:15:06.981 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:15:06.981 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:06.981 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:15:06.981 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:15:06.981 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:15:06.981 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 207391 2024-09-21T10:15:06.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.981 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:15:06.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:15:06.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.433540+0000", 2024-09-21T10:15:06.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:15:06.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.433540+0000", 2024-09-21T10:15:06.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:15:06.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.433542+0000", 2024-09-21T10:15:06.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9810000000000002e-06 2024-09-21T10:15:06.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:15:06.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.433546+0000", 2024-09-21T10:15:06.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.758e-06 2024-09-21T10:15:06.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:15:06.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.433561+0000", 2024-09-21T10:15:06.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4280999999999999e-05 2024-09-21T10:15:06.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:15:06.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.433568+0000", 2024-09-21T10:15:06.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4000000000000003e-06 2024-09-21T10:15:06.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.433640+0000", 2024-09-21T10:15:06.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2177999999999999e-05 2024-09-21T10:15:06.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.433652+0000", 2024-09-21T10:15:06.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2044e-05 2024-09-21T10:15:06.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046579+0000", 2024-09-21T10:15:06.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.61292696700000004 2024-09-21T10:15:06.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046586+0000", 2024-09-21T10:15:06.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9199999999999998e-06 2024-09-21T10:15:06.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.320514+0000", 2024-09-21T10:15:06.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.273927614 2024-09-21T10:15:06.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:15:06.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.320653+0000", 2024-09-21T10:15:06.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000139536 2024-09-21T10:15:06.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6,7", 2024-09-21T10:15:06.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.320794+0000", 2024-09-21T10:15:06.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014110199999999999 2024-09-21T10:15:06.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:15:06.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.326747+0000", 2024-09-21T10:15:06.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0059529869999999999 2024-09-21T10:15:06.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.335333+0000", 2024-09-21T10:15:06.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0085853730000000003 2024-09-21T10:15:06.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.343161+0000", 2024-09-21T10:15:06.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0078282360000000006 2024-09-21T10:15:06.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.343199+0000", 2024-09-21T10:15:06.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8179999999999997e-05 2024-09-21T10:15:06.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:15:06.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.343245+0000", 2024-09-21T10:15:06.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5722e-05 2024-09-21T10:15:06.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:15:06.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.343346+0000", 2024-09-21T10:15:06.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010124999999999999 2024-09-21T10:15:06.988 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.988 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:15:06.988 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.988 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45263.0:207395 6.6 6:634aa997:::benchmark_data_smithi098_169033_object84736:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-21T10:15:06.989 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:11:53.449574+0000", 2024-09-21T10:15:06.989 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 192.09965724899999, 2024-09-21T10:15:06.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.894617733, 2024-09-21T10:15:06.989 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:15:06.989 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:15:06.989 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:06.989 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:15:06.989 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45263", 2024-09-21T10:15:06.989 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/3206174965", 2024-09-21T10:15:06.989 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 207395 2024-09-21T10:15:06.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.989 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:15:06.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:15:06.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.449574+0000", 2024-09-21T10:15:06.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:15:06.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.449574+0000", 2024-09-21T10:15:06.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:15:06.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:15:06.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.449575+0000", 2024-09-21T10:15:06.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9400000000000004e-07 2024-09-21T10:15:06.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:15:06.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.449579+0000", 2024-09-21T10:15:06.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6090000000000001e-06 2024-09-21T10:15:06.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:15:06.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.449586+0000", 2024-09-21T10:15:06.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4240000000000004e-06 2024-09-21T10:15:06.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:15:06.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.449591+0000", 2024-09-21T10:15:06.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6199999999999998e-06 2024-09-21T10:15:06.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.449637+0000", 2024-09-21T10:15:06.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6687000000000002e-05 2024-09-21T10:15:06.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:53.449645+0000", 2024-09-21T10:15:06.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.14e-06 2024-09-21T10:15:06.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046601+0000", 2024-09-21T10:15:06.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.59695511599999995 2024-09-21T10:15:06.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:15:06.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:54.046604+0000", 2024-09-21T10:15:06.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.782e-06 2024-09-21T10:15:06.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:15:06.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.323489+0000", 2024-09-21T10:15:06.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.276884385 2024-09-21T10:15:06.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:15:06.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.323602+0000", 2024-09-21T10:15:06.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000113705 2024-09-21T10:15:06.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6,7", 2024-09-21T10:15:06.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.323725+0000", 2024-09-21T10:15:06.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012281799999999999 2024-09-21T10:15:06.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:15:06.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.327175+0000", 2024-09-21T10:15:06.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034496779999999999 2024-09-21T10:15:06.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.335751+0000", 2024-09-21T10:15:06.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0085764750000000001 2024-09-21T10:15:06.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.343401+0000", 2024-09-21T10:15:06.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0076494420000000002 2024-09-21T10:15:06.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:06.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.344016+0000", 2024-09-21T10:15:06.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00061473599999999995 2024-09-21T10:15:06.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:15:06.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.344069+0000", 2024-09-21T10:15:06.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3674000000000001e-05 2024-09-21T10:15:06.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:15:06.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:15:06.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:15:06.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:11:55.344192+0000", 2024-09-21T10:15:06.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012254799999999999 2024-09-21T10:15:06.996 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.996 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:15:06.996 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.996 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:15:06.996 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:15:06.996 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:15:07.030 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:07.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:07.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:07.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:06 smithi098 ceph-mon[97580]: pgmap v714: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 136 op/s; 7167/304803 objects degraded (2.351%); 7776/304803 objects misplaced (2.551%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:15:07.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:06 smithi098 ceph-mon[100914]: pgmap v714: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 136 op/s; 7167/304803 objects degraded (2.351%); 7776/304803 objects misplaced (2.551%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:15:07.133 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:07.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:06 smithi196 ceph-mon[84784]: pgmap v714: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 136 op/s; 7167/304803 objects degraded (2.351%); 7776/304803 objects misplaced (2.551%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:15:07.173 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:07.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:07.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:07.275 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:07.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:07.320 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:07.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:07.421 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:07.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:07.447 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:07.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:07.548 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:07.573 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:07.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:07.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:07.675 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:07.697 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:07.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:07.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:07.799 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:07.843 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:07.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:07.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:07.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:07.969 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:07.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:07.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:08.071 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:08.096 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:08.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:08.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:08.197 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:08.231 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:08.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:08.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:08.333 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:08.374 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:08.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:08.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:08.475 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:08.501 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:08.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:08.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:15:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:08 smithi098 ceph-mon[97580]: pgmap v715: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.1 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 306 op/s; 7167/299289 objects degraded (2.395%); 7521/299289 objects misplaced (2.513%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:15:08.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:15:08.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:08.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:08 smithi098 ceph-mon[100914]: pgmap v715: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.1 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 306 op/s; 7167/299289 objects degraded (2.395%); 7521/299289 objects misplaced (2.513%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:15:08.602 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:08.626 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:08.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:08.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:08.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:15:08.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:08 smithi196 ceph-mon[84784]: pgmap v715: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.1 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 306 op/s; 7167/299289 objects degraded (2.395%); 7521/299289 objects misplaced (2.513%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:15:08.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:08.759 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:08.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:08.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:08.860 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:08.896 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:08.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:08.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:08.998 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:09.019 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:09.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:09.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:09.089 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:09.121 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:09.145 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:09.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:09.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:09.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:09.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:09.291 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:09.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:09.392 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:09.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:09.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:09.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:09.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:09.551 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:09.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:09.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:09.652 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:09.678 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:09.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:09.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:09.780 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:09.820 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:09.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:09.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:09.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:09.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:09.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:10.058 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:10.070 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:15:10.070 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:15:10.074 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:10.097 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:10.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:10.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:10.103 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:15:10.104 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:15:10.140 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:15:10.140 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:15:10.177 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:15:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:15:10.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:10.213 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:15:10.214 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:15:10.249 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:10.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:10.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:10.254 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:15:10.254 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:15:10.287 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:15:10.287 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:15:10.317 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:15:10.318 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:15:10.351 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:10.375 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:10.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:10.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:10.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:10.509 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:10.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:10.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:10.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:10 smithi098 ceph-mon[97580]: pgmap v716: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 781 op/s; 7167/294486 objects degraded (2.434%); 7521/294486 objects misplaced (2.554%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:15:10.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:10 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1454594155' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:15:10.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:10 smithi098 ceph-mon[100914]: pgmap v716: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 781 op/s; 7167/294486 objects degraded (2.434%); 7521/294486 objects misplaced (2.554%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:15:10.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:10 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1454594155' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:15:10.611 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:10 smithi196 ceph-mon[84784]: pgmap v716: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 781 op/s; 7167/294486 objects degraded (2.434%); 7521/294486 objects misplaced (2.554%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:15:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:10 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1454594155' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:15:10.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:15:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:15:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:15:10.646 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:10.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:10.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:10.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:10.789 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:10.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:10.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:10.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:10.922 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:10.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:10.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:10.969 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:15:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:15:10] "GET /metrics HTTP/1.1" 200 36581 "" "Prometheus/2.51.0" 2024-09-21T10:15:11.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:11.045 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:11.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:11.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:11.147 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:11.188 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:11.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:11.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:11.290 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:11.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:11.318 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:11.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:11.419 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:11.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:11.448 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:11.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:11.549 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:11.576 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:11.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:11.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:11.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:11 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 7167/294486 objects degraded (2.434%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:15:11.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:11 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 7167/294486 objects degraded (2.434%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:15:11.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:11 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 7167/294486 objects degraded (2.434%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:15:11.678 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:11.712 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:11.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:11.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:11.813 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:11.838 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:11.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:11.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:11.940 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:11.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:11.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:12.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:12.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:12.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:12.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:12.263 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:12.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:12.364 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:12.385 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:12.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:12.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:12.487 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:12.511 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:12.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:12.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:12.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:12 smithi098 ceph-mon[100914]: pgmap v717: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 772 op/s; 7167/292581 objects degraded (2.450%); 7521/292581 objects misplaced (2.571%); 1.4 MiB/s, 21 objects/s recovering 2024-09-21T10:15:12.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:12 smithi098 ceph-mon[97580]: pgmap v717: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 772 op/s; 7167/292581 objects degraded (2.450%); 7521/292581 objects misplaced (2.571%); 1.4 MiB/s, 21 objects/s recovering 2024-09-21T10:15:12.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:12.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:12 smithi196 ceph-mon[84784]: pgmap v717: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 772 op/s; 7167/292581 objects degraded (2.450%); 7521/292581 objects misplaced (2.571%); 1.4 MiB/s, 21 objects/s recovering 2024-09-21T10:15:12.652 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:12.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:12.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:12.753 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:12.795 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:12.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:12.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:12.897 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:12.931 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:12.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:12.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:13.033 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:13.059 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:13.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:13.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:13.161 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:13.200 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:13.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:13.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:13.302 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:13.345 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:13.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:13.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:13.447 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:13.470 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:15:13.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:13.507 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:13.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:13.608 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:13.648 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:13.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:13.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:13.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:13.784 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:15:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:13.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:13.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:13.893 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:13.918 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:13.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:13.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:14.020 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:14.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:14.056 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:14.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:14.157 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:14.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:14.192 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:14.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:14.293 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:14.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:14.318 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:14.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:14.419 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:14.459 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:14.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:14.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:14.561 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:14.586 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:14.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:14.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:14.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:14 smithi098 ceph-mon[97580]: pgmap v718: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.8 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 885 op/s; 7167/285021 objects degraded (2.515%); 7193/285021 objects misplaced (2.524%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:15:14.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:14 smithi098 ceph-mon[100914]: pgmap v718: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.8 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 885 op/s; 7167/285021 objects degraded (2.515%); 7193/285021 objects misplaced (2.524%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:15:14.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:14 smithi196 ceph-mon[84784]: pgmap v718: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.8 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 885 op/s; 7167/285021 objects degraded (2.515%); 7193/285021 objects misplaced (2.524%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:15:14.689 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:14.714 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:14.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:14.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:14.816 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:14.841 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:14.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:14.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:14.943 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:14.969 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:14.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:14.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:15.072 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:15.097 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:15.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:15.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:15.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:15.242 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:15.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:15.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:15.343 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:15.368 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:15.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:15.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:15.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:15.501 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:15.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:15.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:15.527 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:15:15.527 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:15:15.604 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:15.628 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:15.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:15.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:15.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:15.755 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:15.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:15.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:15.857 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:15.900 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:15.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:15.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:16.001 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:16.036 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:16.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:16.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:16.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:16.165 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:16.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:16.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:16.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:15:16.267 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:16.312 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:16.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:16.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:16.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:15:16.413 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:16.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:16.472 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:16.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:16.532 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:15:16.573 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:16.594 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:16.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:16.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:16.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:16 smithi196 ceph-mon[84784]: pgmap v719: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 886 op/s; 7167/278709 objects degraded (2.571%); 7193/278709 objects misplaced (2.581%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:15:16.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:16 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 7167/278709 objects degraded (2.571%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:15:16.695 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:16.719 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:16.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:16.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:16.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:16.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:16 smithi098 ceph-mon[97580]: pgmap v719: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 886 op/s; 7167/278709 objects degraded (2.571%); 7193/278709 objects misplaced (2.581%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:15:16.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:16 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 7167/278709 objects degraded (2.571%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:15:16.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:16 smithi098 ceph-mon[100914]: pgmap v719: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 886 op/s; 7167/278709 objects degraded (2.571%); 7193/278709 objects misplaced (2.581%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:15:16.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:16 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 7167/278709 objects degraded (2.571%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:15:16.855 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:16.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:16.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:16.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:16.979 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:16.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:16.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:17.080 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:17.101 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:17.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:17.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:17.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:17.224 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:17.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:17.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:17.326 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:17.353 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:17.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:17.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:17.428 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:15:17.455 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:17.482 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:17.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:17.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:17.584 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:17.606 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:17.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:17.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:17.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:17 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3938164410' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:15:17.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:17 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:15:17.708 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:17.729 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:17.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:17.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:17.831 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:17.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:17 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3938164410' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:15:17.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:17 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:15:17.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:17 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3938164410' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:15:17.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:17 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:15:17.857 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:17.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:17.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:17.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:18.000 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:18.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:18.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:18.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:18.127 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:18.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:18.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:18.228 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:18.254 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:18.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:18.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:18.355 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:18.387 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:18.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:18.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:18.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:18.525 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:18.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:18.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:18.533 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:15:18.533 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:15:18.533 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:18 smithi196 ceph-mon[84784]: pgmap v720: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 892 op/s; 7167/272652 objects degraded (2.629%); 6867/272652 objects misplaced (2.519%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:15:18.533 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:18 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:15:18.533 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:18 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:15:18.533 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:18 smithi196 ceph-mon[84784]: osdmap e240: 8 total, 8 up, 7 in 2024-09-21T10:15:18.628 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:18.665 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:18.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:18.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:18.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:18.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:18.811 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:18.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:18 smithi098 ceph-mon[97580]: pgmap v720: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 892 op/s; 7167/272652 objects degraded (2.629%); 6867/272652 objects misplaced (2.519%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:15:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:18 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:15:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:18 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:15:18.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:18 smithi098 ceph-mon[97580]: osdmap e240: 8 total, 8 up, 7 in 2024-09-21T10:15:18.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:18 smithi098 ceph-mon[100914]: pgmap v720: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 892 op/s; 7167/272652 objects degraded (2.629%); 6867/272652 objects misplaced (2.519%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:15:18.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:18 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:15:18.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:18 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:15:18.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:18 smithi098 ceph-mon[100914]: osdmap e240: 8 total, 8 up, 7 in 2024-09-21T10:15:18.912 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:18.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:18.946 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:18.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:19.047 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:19.073 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:19.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:19.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:19.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:19.200 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:19.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:19.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:19.288 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:15:19.302 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:19.348 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:19.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:19.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:19.450 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:19.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:19.486 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:19.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:19.587 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:19.604 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:15:19.626 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:19.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:19.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:19.728 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:19.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:19.767 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:19.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:19.868 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:19.892 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:19.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:19.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:19.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:20.050 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:20.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:20.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:20.152 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:20.190 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:20.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:20.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:20.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:20.329 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:20.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:20.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:20.358 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:15:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:15:20.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:15:20.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:20.467 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:20.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:20.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:20.569 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:20.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:20 smithi098 ceph-mon[100914]: pgmap v722: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.4 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 907 op/s; 7167/267267 objects degraded (2.682%); 6867/267267 objects misplaced (2.569%); 1.4 MiB/s, 21 objects/s recovering 2024-09-21T10:15:20.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:20 smithi098 ceph-mon[97580]: pgmap v722: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.4 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 907 op/s; 7167/267267 objects degraded (2.682%); 6867/267267 objects misplaced (2.569%); 1.4 MiB/s, 21 objects/s recovering 2024-09-21T10:15:20.609 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:20.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:20.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:20 smithi196 ceph-mon[84784]: pgmap v722: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.4 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 907 op/s; 7167/267267 objects degraded (2.682%); 6867/267267 objects misplaced (2.569%); 1.4 MiB/s, 21 objects/s recovering 2024-09-21T10:15:20.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:20.750 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:20.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:20.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:20.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:20.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:20.896 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:20.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:20.936 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:15:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:15:20] "GET /metrics HTTP/1.1" 200 36581 "" "Prometheus/2.51.0" 2024-09-21T10:15:20.996 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:21.024 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:21.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:21.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:21.126 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:21.149 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:21.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:21.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:21.250 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:21.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:21.274 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:21.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:21.314 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:21.375 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:21.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:21.448 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:21.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:21.466 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:15:21.466 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:15:21.539 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:21 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 7167/267267 objects degraded (2.682%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:15:21.539 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:21 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 7167/267267 objects degraded (2.682%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:15:21.549 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:21.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:21.615 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:21.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:21.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:21 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 7167/267267 objects degraded (2.682%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:15:21.716 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:21.752 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:21.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:21.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:21.854 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:21.879 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:21.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:21.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:21.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:22.027 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:22.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:22.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:22.127 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:15:22.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:22.175 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:22.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:22.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:22.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:22.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:22.312 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:22.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:22.373 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T10:15:22.414 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:22.440 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:22.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:22.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:22.457 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:15:22.542 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:22.568 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:22.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:22.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:22 smithi196 ceph-mon[84784]: pgmap v723: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 910 op/s; 7167/265248 objects degraded (2.702%); 6867/265248 objects misplaced (2.589%); 1.4 MiB/s, 21 objects/s recovering 2024-09-21T10:15:22.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:22 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1307718559' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:15:22.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:22 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:15:22.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:15:22.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:22.670 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:22.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:22.698 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:22.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:22.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:22.822 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:22.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:22.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:22.832 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:22 smithi098 ceph-mon[97580]: pgmap v723: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 910 op/s; 7167/265248 objects degraded (2.702%); 6867/265248 objects misplaced (2.589%); 1.4 MiB/s, 21 objects/s recovering 2024-09-21T10:15:22.832 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:22 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1307718559' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:15:22.832 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:22 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:15:22.832 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:15:22.832 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:22.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:22 smithi098 ceph-mon[100914]: pgmap v723: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 910 op/s; 7167/265248 objects degraded (2.702%); 6867/265248 objects misplaced (2.589%); 1.4 MiB/s, 21 objects/s recovering 2024-09-21T10:15:22.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:22 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1307718559' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:15:22.833 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:22 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:15:22.833 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:15:22.833 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:22.923 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:22.978 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:15:22.983 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:22.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:22.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:23.085 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:23.135 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:23.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:23.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:23.237 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:23.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:23.283 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:23.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:23.384 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:23.427 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:23.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:23.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:23.479 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:23 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:15:23.479 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:23 smithi098 ceph-mon[97580]: osdmap e241: 8 total, 8 up, 7 in 2024-09-21T10:15:23.479 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:23 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:15:23.479 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:23 smithi098 ceph-mon[100914]: osdmap e241: 8 total, 8 up, 7 in 2024-09-21T10:15:23.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T10:15:23.529 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:23.613 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:23.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:23.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:23.633 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:23 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:15:23.633 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:23 smithi196 ceph-mon[84784]: osdmap e241: 8 total, 8 up, 7 in 2024-09-21T10:15:23.714 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:23.749 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:23.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:23.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:23.851 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:23.875 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:23.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:23.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:23.976 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:24.003 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:24.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:24.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:24.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:24.147 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:24.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:24.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:24.246 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:24.248 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:24.305 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:24.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:24.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:24.334 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:15:24.334 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:15:24.410 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:24.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:24.458 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:24.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:24.559 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:24.611 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:24.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:24.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:24.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:24 smithi196 ceph-mon[84784]: pgmap v725: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 873 op/s; 7167/257733 objects degraded (2.781%); 6532/257733 objects misplaced (2.534%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:15:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:24 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:15:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:24 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:15:24.713 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:24.735 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:24.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:24.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:24.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:24.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:24 smithi098 ceph-mon[97580]: pgmap v725: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 873 op/s; 7167/257733 objects degraded (2.781%); 6532/257733 objects misplaced (2.534%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:15:24.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:24 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:15:24.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:24 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:15:24.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:24 smithi098 ceph-mon[100914]: pgmap v725: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 873 op/s; 7167/257733 objects degraded (2.781%); 6532/257733 objects misplaced (2.534%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:15:24.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:24 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:15:24.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:24 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:15:24.876 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:24.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:24.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:24.978 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:25.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:25.016 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:25.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:25.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:15:25.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:25.154 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:25.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:25.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:25.256 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:25.285 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:25.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:25.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:25.373 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:15:25.386 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:25.416 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:25.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:25.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:25.518 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:25.543 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:25.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:25.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:25.645 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:25.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:25.687 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:25.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:25.788 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:25.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:25.822 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:25.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:25.923 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:25.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:25.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:26.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:26.089 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:26.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:26.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:26.191 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:26.227 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:26.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:26.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:26.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:26.376 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:26.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:26.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:26.478 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:26.502 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:26.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:26.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:26.603 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:26.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:26 smithi196 ceph-mon[84784]: pgmap v726: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.1 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 872 op/s; 7167/251694 objects degraded (2.848%); 6532/251694 objects misplaced (2.595%); 887 KiB/s, 13 objects/s recovering 2024-09-21T10:15:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:26 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:15:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:26 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:15:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:26 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:15:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:26 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 7167/251694 objects degraded (2.848%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:15:26.644 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:26.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:26.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:26.746 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:26.772 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:26.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:26.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:26.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:26 smithi098 ceph-mon[97580]: pgmap v726: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.1 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 872 op/s; 7167/251694 objects degraded (2.848%); 6532/251694 objects misplaced (2.595%); 887 KiB/s, 13 objects/s recovering 2024-09-21T10:15:26.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:26 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:15:26.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:26 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:15:26.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:26 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:15:26.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:26 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 7167/251694 objects degraded (2.848%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:15:26.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:26 smithi098 ceph-mon[100914]: pgmap v726: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.1 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 872 op/s; 7167/251694 objects degraded (2.848%); 6532/251694 objects misplaced (2.595%); 887 KiB/s, 13 objects/s recovering 2024-09-21T10:15:26.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:26 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:15:26.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:26 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:15:26.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:26 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:15:26.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:26 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 7167/251694 objects degraded (2.848%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:15:26.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:26.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:26.931 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:26.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:27.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:27.033 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:27.057 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:27.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:27.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:27.159 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:27.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:27.194 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:27.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:27.295 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:27.334 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:27.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:27.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:27.436 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:27.466 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:27.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:27.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:27.568 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:27.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:27.597 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:27.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:27.626 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:15:27.626 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:15:27.698 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:27.723 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:27.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:27.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:27.825 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:27.850 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:27.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:27.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:27.952 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:27.992 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:27.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:27.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:28.094 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:28.120 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:28.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:28.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:28.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:28.258 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:28.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:28.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:28.306 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:15:28.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:28.400 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:28.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:28.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:28.502 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:28.540 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:28.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:28.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:28.625 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:15:28.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:28 smithi196 ceph-mon[84784]: pgmap v727: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 894 op/s; 7167/246042 objects degraded (2.913%); 6168/246042 objects misplaced (2.507%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:15:28.642 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:28.673 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:28.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:28.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:28.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:28 smithi098 ceph-mon[97580]: pgmap v727: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 894 op/s; 7167/246042 objects degraded (2.913%); 6168/246042 objects misplaced (2.507%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:15:28.705 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:28 smithi098 ceph-mon[100914]: pgmap v727: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 894 op/s; 7167/246042 objects degraded (2.913%); 6168/246042 objects misplaced (2.507%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:15:28.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:28.824 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:28.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:28.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:28.926 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:28.965 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:28.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:29.066 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:29.091 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:29.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:29.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:29.193 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:29.217 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:29.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:29.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:29.319 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:29.343 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:29.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:29.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:29.445 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:29.490 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:29.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:29.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:29.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:29.631 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:29.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:29.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:29.733 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:29.769 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:29.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:29.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:29.871 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:29.915 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:29.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:29.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:30.016 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:30.041 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:30.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:30.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:30.143 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:30.182 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:30.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:30.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:30.284 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:30.311 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:30.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:30.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:30.410 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:15:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:15:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:15:30.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:30.441 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:30.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:30.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:30.453 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:15:30.453 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:15:30.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:30.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:30.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:30.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:30 smithi098 ceph-mon[97580]: pgmap v728: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 884 op/s; 7167/240723 objects degraded (2.977%); 6168/240723 objects misplaced (2.562%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:15:30.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:30 smithi098 ceph-mon[100914]: pgmap v728: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 884 op/s; 7167/240723 objects degraded (2.977%); 6168/240723 objects misplaced (2.562%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:15:30.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:30.738 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:30.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:30.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:30.766 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:30 smithi196 ceph-mon[84784]: pgmap v728: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 884 op/s; 7167/240723 objects degraded (2.977%); 6168/240723 objects misplaced (2.562%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:15:30.840 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:30.880 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:30.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:30.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:30.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:30.986 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:15:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:15:30] "GET /metrics HTTP/1.1" 200 36583 "" "Prometheus/2.51.0" 2024-09-21T10:15:31.027 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:31.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:31.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:31.129 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:31.157 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:31.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:31.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:31.259 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:31.284 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:31.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:31.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:31.386 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:31.411 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:31.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:31.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:31.512 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:31.548 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:31.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:31.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:31.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:31.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:31.690 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:31.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:31.724 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:31 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 7167/240723 objects degraded (2.977%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:15:31.725 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:31 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 7167/240723 objects degraded (2.977%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:15:31.791 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:31.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:31.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:31.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:31.825 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:31 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 7167/240723 objects degraded (2.977%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:15:31.848 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:15:31.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:31.956 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:31.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:31.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:32.058 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:32.091 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:32.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:32.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:32.186 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:15:32.192 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:32.215 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:32.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:32.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:32.316 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:32.338 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:32.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:32.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:32.439 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:32.460 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:32.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:32.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:32.532 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:15:32.532 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:15:32.533 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:15:32.533 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:15:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:15:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:15:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:15:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:15:32.533 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:15:32.533 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:15:32.534 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:15:32.534 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:15:32.534 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:15:32.534 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:15:32.534 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:15:32.534 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:15:32.562 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:32.584 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:32.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:32.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:32.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:32 smithi196 ceph-mon[84784]: pgmap v729: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 885 op/s; 7167/238650 objects degraded (3.003%); 6168/238650 objects misplaced (2.585%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:15:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:32 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/979802638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:15:32.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:32.720 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:32.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:32.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:32.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:32.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:32 smithi098 ceph-mon[97580]: pgmap v729: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 885 op/s; 7167/238650 objects degraded (3.003%); 6168/238650 objects misplaced (2.585%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:15:32.926 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:32 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/979802638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:15:32.926 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:32 smithi098 ceph-mon[100914]: pgmap v729: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 885 op/s; 7167/238650 objects degraded (3.003%); 6168/238650 objects misplaced (2.585%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:15:32.926 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:32 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/979802638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:15:32.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:32.927 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:32.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:33.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:33.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:33.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:33.170 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:33.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:33.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:33.308 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:33.334 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:33.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:33.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:33.436 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:33.479 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:33.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:33.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:33.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:33.623 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:33.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:33.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:33.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:33.767 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:33.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:33.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:33.869 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:33.903 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:33.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:33.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:34.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:34.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:34.044 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:34.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:34.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:34.181 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:34.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:34.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:34.223 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:15:34.223 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:15:34.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.223 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:158943 7.d 7:b5b3beef:::benchmark_data_smithi098_197834_object42426:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-21T10:15:34.223 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:15:34.206058+0000", 2024-09-21T10:15:34.223 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.013868409, 2024-09-21T10:15:34.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013891618, 2024-09-21T10:15:34.224 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:34.224 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:15:34.224 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:34.224 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:15:34.224 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:15:34.224 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 158943 2024-09-21T10:15:34.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.224 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:34.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:34.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.206058+0000", 2024-09-21T10:15:34.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:34.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:34.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.206058+0000", 2024-09-21T10:15:34.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:34.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:34.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.206056+0000", 2024-09-21T10:15:34.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:15:34.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:34.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.206063+0000", 2024-09-21T10:15:34.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6170000000000003e-06 2024-09-21T10:15:34.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:34.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.206078+0000", 2024-09-21T10:15:34.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5432999999999999e-05 2024-09-21T10:15:34.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.226 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:34.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.206084+0000", 2024-09-21T10:15:34.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.519e-06 2024-09-21T10:15:34.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.226 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:34.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.208041+0000", 2024-09-21T10:15:34.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019577420000000002 2024-09-21T10:15:34.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.227 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:34.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.208170+0000", 2024-09-21T10:15:34.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012830900000000001 2024-09-21T10:15:34.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.227 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:15:34.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.208301+0000", 2024-09-21T10:15:34.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000130884 2024-09-21T10:15:34.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.227 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:34.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.211847+0000", 2024-09-21T10:15:34.228 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0035468140000000001 2024-09-21T10:15:34.228 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:34.228 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:34.228 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:34.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.228 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.228 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:158948 7.d 7:b5a93ce8:::benchmark_data_smithi098_197834_object42431:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-21T10:15:34.228 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:15:34.208121+0000", 2024-09-21T10:15:34.228 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.011805535000000001, 2024-09-21T10:15:34.228 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011885365, 2024-09-21T10:15:34.228 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:34.228 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:15:34.228 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:34.228 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:15:34.229 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:15:34.229 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 158948 2024-09-21T10:15:34.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.229 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:34.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:34.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.208121+0000", 2024-09-21T10:15:34.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:34.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:34.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.208121+0000", 2024-09-21T10:15:34.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:34.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:34.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.208120+0000", 2024-09-21T10:15:34.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:15:34.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:34.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.208125+0000", 2024-09-21T10:15:34.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8139999999999997e-06 2024-09-21T10:15:34.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:34.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.208136+0000", 2024-09-21T10:15:34.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0360000000000001e-05 2024-09-21T10:15:34.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:34.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.208140+0000", 2024-09-21T10:15:34.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0550000000000001e-06 2024-09-21T10:15:34.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:34.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.211887+0000", 2024-09-21T10:15:34.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0037467809999999998 2024-09-21T10:15:34.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:34.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.212023+0000", 2024-09-21T10:15:34.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013622600000000001 2024-09-21T10:15:34.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:15:34.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.212096+0000", 2024-09-21T10:15:34.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3385000000000003e-05 2024-09-21T10:15:34.232 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:34.232 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:34.232 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:34.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.232 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:158949 7.d 7:b9ea33ea:::benchmark_data_smithi098_197834_object42432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-21T10:15:34.233 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:15:34.208218+0000", 2024-09-21T10:15:34.233 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.011707959, 2024-09-21T10:15:34.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011826994, 2024-09-21T10:15:34.233 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:34.233 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:15:34.233 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:34.233 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:15:34.233 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:15:34.233 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 158949 2024-09-21T10:15:34.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.233 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:34.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:34.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.208218+0000", 2024-09-21T10:15:34.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:34.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.234 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:34.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.208218+0000", 2024-09-21T10:15:34.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:34.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.234 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:34.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.208218+0000", 2024-09-21T10:15:34.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:15:34.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:34.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.208222+0000", 2024-09-21T10:15:34.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1910000000000002e-06 2024-09-21T10:15:34.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:34.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.208238+0000", 2024-09-21T10:15:34.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6082e-05 2024-09-21T10:15:34.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:34.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.208242+0000", 2024-09-21T10:15:34.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7699999999999999e-06 2024-09-21T10:15:34.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:34.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.214856+0000", 2024-09-21T10:15:34.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0066140950000000004 2024-09-21T10:15:34.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:34.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.214992+0000", 2024-09-21T10:15:34.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013650399999999999 2024-09-21T10:15:34.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:15:34.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.215121+0000", 2024-09-21T10:15:34.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012900300000000001 2024-09-21T10:15:34.236 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:34.237 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:34.237 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:34.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.237 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:158954 7.d 7:b02c79fb:::benchmark_data_smithi098_197834_object42437:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-21T10:15:34.237 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:15:34.214374+0000", 2024-09-21T10:15:34.237 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0055518069999999997, 2024-09-21T10:15:34.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0057164629999999998, 2024-09-21T10:15:34.237 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:34.237 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:15:34.238 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:34.238 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:15:34.238 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:15:34.238 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 158954 2024-09-21T10:15:34.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.238 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:34.238 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.238 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:34.238 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.214374+0000", 2024-09-21T10:15:34.238 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:34.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.238 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.238 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:34.238 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.214374+0000", 2024-09-21T10:15:34.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:34.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:34.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.214373+0000", 2024-09-21T10:15:34.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:15:34.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:34.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.214380+0000", 2024-09-21T10:15:34.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6989999999999997e-06 2024-09-21T10:15:34.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:34.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.214394+0000", 2024-09-21T10:15:34.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4252e-05 2024-09-21T10:15:34.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.240 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.240 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:34.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.214401+0000", 2024-09-21T10:15:34.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8290000000000002e-06 2024-09-21T10:15:34.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.240 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.240 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:34.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.218663+0000", 2024-09-21T10:15:34.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042621359999999997 2024-09-21T10:15:34.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.241 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.241 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:34.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.218786+0000", 2024-09-21T10:15:34.241 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012291 2024-09-21T10:15:34.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.241 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.241 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:15:34.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.218935+0000", 2024-09-21T10:15:34.241 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014871900000000001 2024-09-21T10:15:34.241 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:34.241 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:34.241 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:34.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.241 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.241 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.55000.0:158943 7.d e241/202)", 2024-09-21T10:15:34.242 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:15:34.214781+0000", 2024-09-21T10:15:34.242 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0051448379999999997, 2024-09-21T10:15:34.242 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0053474739999999996, 2024-09-21T10:15:34.242 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:34.242 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "queued for pg", 2024-09-21T10:15:34.242 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:34.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:34.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.214781+0000", 2024-09-21T10:15:34.242 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:34.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:34.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.214781+0000", 2024-09-21T10:15:34.243 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:34.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.243 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:34.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.214781+0000", 2024-09-21T10:15:34.243 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:15:34.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.243 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:34.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.214786+0000", 2024-09-21T10:15:34.243 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8099999999999997e-06 2024-09-21T10:15:34.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.243 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:34.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.214797+0000", 2024-09-21T10:15:34.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1134e-05 2024-09-21T10:15:34.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.244 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:34.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.214802+0000", 2024-09-21T10:15:34.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1880000000000001e-06 2024-09-21T10:15:34.244 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:34.244 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:34.244 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:34.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.244 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:158940 7.d 7:b80f0fb3:::benchmark_data_smithi098_197834_object42423:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-21T10:15:34.244 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:15:34.203224+0000", 2024-09-21T10:15:34.244 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.016702268999999999, 2024-09-21T10:15:34.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016932890999999999, 2024-09-21T10:15:34.244 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:34.245 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:15:34.245 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:34.245 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:15:34.245 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:15:34.245 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 158940 2024-09-21T10:15:34.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.245 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:34.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.245 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:34.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.203224+0000", 2024-09-21T10:15:34.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:34.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.245 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:34.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.203224+0000", 2024-09-21T10:15:34.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:34.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:34.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.203223+0000", 2024-09-21T10:15:34.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:15:34.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:34.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.203232+0000", 2024-09-21T10:15:34.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5019999999999998e-06 2024-09-21T10:15:34.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:34.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.203243+0000", 2024-09-21T10:15:34.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1005000000000001e-05 2024-09-21T10:15:34.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:34.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.203248+0000", 2024-09-21T10:15:34.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3669999999999999e-06 2024-09-21T10:15:34.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:34.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.203293+0000", 2024-09-21T10:15:34.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5429000000000002e-05 2024-09-21T10:15:34.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.248 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:34.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.203417+0000", 2024-09-21T10:15:34.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012415200000000001 2024-09-21T10:15:34.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.248 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:15:34.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.203543+0000", 2024-09-21T10:15:34.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012537299999999999 2024-09-21T10:15:34.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.248 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:34.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.211803+0000", 2024-09-21T10:15:34.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0082606910000000006 2024-09-21T10:15:34.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:34.249 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:34.249 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:34.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:34.211841+0000", 2024-09-21T10:15:34.249 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7595000000000002e-05 2024-09-21T10:15:34.249 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:34.249 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:34.249 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:34.249 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:34.249 INFO:teuthology.orchestra.run.smithi196.stdout: ], 2024-09-21T10:15:34.249 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 6 2024-09-21T10:15:34.249 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:15:34.283 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:34.325 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:34.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:34.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:34.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:34.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:34.469 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:34.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:34.569 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:34.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:34 smithi098 ceph-mon[97580]: pgmap v730: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 871 op/s; 7167/230976 objects degraded (3.103%); 5775/230976 objects misplaced (2.500%); 1.4 MiB/s, 23 objects/s recovering 2024-09-21T10:15:34.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:34 smithi098 ceph-mon[100914]: pgmap v730: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 871 op/s; 7167/230976 objects degraded (3.103%); 5775/230976 objects misplaced (2.500%); 1.4 MiB/s, 23 objects/s recovering 2024-09-21T10:15:34.609 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:34.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:34.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:34.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:34 smithi196 ceph-mon[84784]: pgmap v730: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 871 op/s; 7167/230976 objects degraded (3.103%); 5775/230976 objects misplaced (2.500%); 1.4 MiB/s, 23 objects/s recovering 2024-09-21T10:15:34.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:34.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:34.750 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:34.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:34.851 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:34.878 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:34.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:34.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:34.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:35.025 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:15:35.042 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:35.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:35.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:35.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:35.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:35.186 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:35.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:35.287 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:35.328 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:35.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:35.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:35.351 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:15:35.429 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:35.438 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:15:35.468 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:35.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:35.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:35.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:35.597 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:35.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:35 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/764706610' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:15:35.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:35 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/764706610' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:15:35.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:35.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:35.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:35.724 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:35 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/764706610' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:15:35.747 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:35.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:35.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:35.883 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:35.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:35.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:35.985 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:36.011 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:36.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:36.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:36.113 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:36.139 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:36.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:36.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:36.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:36.277 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:36.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:36.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:36.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:36.425 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:36.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:36.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:36.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:36 smithi196 ceph-mon[84784]: pgmap v731: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 897 op/s; 7167/225411 objects degraded (3.180%); 5775/225411 objects misplaced (2.562%); 1.3 MiB/s, 21 objects/s recovering 2024-09-21T10:15:36.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:36 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:15:36.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:36 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/764706610' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:15:36.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:36 smithi196 ceph-mon[84784]: osdmap e242: 8 total, 8 up, 7 in 2024-09-21T10:15:36.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:36 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 7167/225411 objects degraded (3.180%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:15:36.527 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:36.566 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:36.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:36.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:36.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:36.705 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:36.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:36.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:36.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:36 smithi098 ceph-mon[97580]: pgmap v731: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 897 op/s; 7167/225411 objects degraded (3.180%); 5775/225411 objects misplaced (2.562%); 1.3 MiB/s, 21 objects/s recovering 2024-09-21T10:15:36.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:36 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:15:36.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:36 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/764706610' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:15:36.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:36 smithi098 ceph-mon[97580]: osdmap e242: 8 total, 8 up, 7 in 2024-09-21T10:15:36.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:36 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 7167/225411 objects degraded (3.180%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:15:36.772 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:36 smithi098 ceph-mon[100914]: pgmap v731: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 897 op/s; 7167/225411 objects degraded (3.180%); 5775/225411 objects misplaced (2.562%); 1.3 MiB/s, 21 objects/s recovering 2024-09-21T10:15:36.772 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:36 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:15:36.772 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:36 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/764706610' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:15:36.772 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:36 smithi098 ceph-mon[100914]: osdmap e242: 8 total, 8 up, 7 in 2024-09-21T10:15:36.772 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:36 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 7167/225411 objects degraded (3.180%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:15:36.807 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:36.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:36.833 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:36.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:36.934 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:36.959 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:36.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:36.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:37.061 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:37.090 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:37.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:37.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:37.106 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:15:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:15:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-21T10:15:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-21T10:15:37.106 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:15:37.192 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:37.231 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:37.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:37.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:37.333 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:37.358 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:37.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:37.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:37.460 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:37.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:37.484 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:37.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:37.518 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T10:15:37.585 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:37.612 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:37.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:37.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:37.714 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:37.739 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:37.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:37.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:37.840 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:37.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:37.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:37 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1266482122' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:15:37.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:37 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:15:37.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:37.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:37 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1266482122' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:15:37.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:37 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:15:37.856 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:37.857 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:37 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1266482122' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:15:37.857 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:37 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:15:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:37.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:37.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:37.978 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:38.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:38.019 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:38.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:38.021 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:15:38.090 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:15:38.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:38.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:38.176 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:38.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:38.277 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:38.308 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:38.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:38.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:38.348 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:15:38.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T10:15:38.410 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:38.437 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:38.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:38.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:38.538 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:38.562 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:38.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:38.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:38.627 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:38 smithi098 ceph-mon[97580]: pgmap v733: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 864 op/s; 7167/220107 objects degraded (3.256%); 5340/220107 objects misplaced (2.426%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:15:38.628 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:38 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:15:38.628 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:38 smithi098 ceph-mon[97580]: osdmap e243: 8 total, 8 up, 7 in 2024-09-21T10:15:38.628 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:38 smithi098 ceph-mon[100914]: pgmap v733: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 864 op/s; 7167/220107 objects degraded (3.256%); 5340/220107 objects misplaced (2.426%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:15:38.628 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:38 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:15:38.628 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:38 smithi098 ceph-mon[100914]: osdmap e243: 8 total, 8 up, 7 in 2024-09-21T10:15:38.664 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:38.687 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:38.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:38.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:38.752 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:38 smithi196 ceph-mon[84784]: pgmap v733: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 864 op/s; 7167/220107 objects degraded (3.256%); 5340/220107 objects misplaced (2.426%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:15:38.753 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:38 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:15:38.753 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:38 smithi196 ceph-mon[84784]: osdmap e243: 8 total, 8 up, 7 in 2024-09-21T10:15:38.789 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:38.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:38.846 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:38.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:38.856 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:38.928 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:38.947 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:38.973 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:38.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:38.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:39.075 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:39.133 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:39.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:39.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:39.237 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:39.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:39.283 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:39.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:39.384 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:39.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:39.417 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:39.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:39.435 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:15:39.518 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:39.544 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:39.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:39.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:39.645 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:39.671 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:39.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:39.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:39.773 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:39.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:39.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:39.923 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:39.966 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:39.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:39.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:40.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:40.098 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:40.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:40.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:40.200 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:40.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:40.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:40.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:40.368 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:40.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:40.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:40.381 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:15:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:15:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:15:40.382 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:15:40.382 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-21T10:15:40.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-21T10:15:40.382 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:15:40.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.382 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:267684 5.f 5:f923f360:::benchmark_data_smithi098_135187_object132046:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:15:40.382 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:07:59.386796+0000", 2024-09-21T10:15:40.382 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 460.99114974399998, 2024-09-21T10:15:40.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2285444640000001, 2024-09-21T10:15:40.382 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:40.382 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:40.382 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:40.382 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:15:40.383 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:15:40.383 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 267684 2024-09-21T10:15:40.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.383 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:40.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:40.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.386796+0000", 2024-09-21T10:15:40.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:40.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.386796+0000", 2024-09-21T10:15:40.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:40.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.386795+0000", 2024-09-21T10:15:40.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:15:40.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:40.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.386799+0000", 2024-09-21T10:15:40.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2699999999999998e-06 2024-09-21T10:15:40.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:40.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.386807+0000", 2024-09-21T10:15:40.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5639999999999999e-06 2024-09-21T10:15:40.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:40.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.386813+0000", 2024-09-21T10:15:40.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7450000000000001e-06 2024-09-21T10:15:40.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.389452+0000", 2024-09-21T10:15:40.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026394589999999998 2024-09-21T10:15:40.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:15:40.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.389460+0000", 2024-09-21T10:15:40.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1289999999999996e-06 2024-09-21T10:15:40.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:00.373649+0000", 2024-09-21T10:15:40.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.98418915200000001 2024-09-21T10:15:40.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:15:40.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:00.373662+0000", 2024-09-21T10:15:40.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.236e-05 2024-09-21T10:15:40.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.603922+0000", 2024-09-21T10:15:40.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2302606309999999 2024-09-21T10:15:40.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:40.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.604044+0000", 2024-09-21T10:15:40.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000121914 2024-09-21T10:15:40.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,4,5", 2024-09-21T10:15:40.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.604139+0000", 2024-09-21T10:15:40.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5185000000000004e-05 2024-09-21T10:15:40.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:40.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.614058+0000", 2024-09-21T10:15:40.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0099189120000000002 2024-09-21T10:15:40.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.614287+0000", 2024-09-21T10:15:40.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00022896200000000001 2024-09-21T10:15:40.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.614341+0000", 2024-09-21T10:15:40.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3458999999999998e-05 2024-09-21T10:15:40.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.615174+0000", 2024-09-21T10:15:40.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00083303000000000001 2024-09-21T10:15:40.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:15:40.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.615228+0000", 2024-09-21T10:15:40.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4710999999999998e-05 2024-09-21T10:15:40.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:15:40.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.615341+0000", 2024-09-21T10:15:40.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000112281 2024-09-21T10:15:40.390 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.390 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:40.390 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.390 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:267701 5.f 5:f834efde:::benchmark_data_smithi098_135187_object132063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:15:40.390 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:07:59.412832+0000", 2024-09-21T10:15:40.390 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 460.965113825, 2024-09-21T10:15:40.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.20275146, 2024-09-21T10:15:40.390 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:40.390 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:40.391 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:40.391 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:15:40.391 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:15:40.391 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 267701 2024-09-21T10:15:40.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.391 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:40.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:40.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.412832+0000", 2024-09-21T10:15:40.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:40.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.412832+0000", 2024-09-21T10:15:40.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:40.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.412831+0000", 2024-09-21T10:15:40.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:15:40.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:40.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.412837+0000", 2024-09-21T10:15:40.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6069999999999998e-06 2024-09-21T10:15:40.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:40.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.412848+0000", 2024-09-21T10:15:40.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0991e-05 2024-09-21T10:15:40.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:40.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.412853+0000", 2024-09-21T10:15:40.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0000000000000004e-06 2024-09-21T10:15:40.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.412913+0000", 2024-09-21T10:15:40.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0652999999999997e-05 2024-09-21T10:15:40.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:15:40.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.412922+0000", 2024-09-21T10:15:40.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2959999999999999e-06 2024-09-21T10:15:40.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:00.374014+0000", 2024-09-21T10:15:40.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.96109220699999998 2024-09-21T10:15:40.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:15:40.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:00.374023+0000", 2024-09-21T10:15:40.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2909999999999996e-06 2024-09-21T10:15:40.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.607188+0000", 2024-09-21T10:15:40.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.233165227 2024-09-21T10:15:40.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:40.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.607268+0000", 2024-09-21T10:15:40.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9999000000000004e-05 2024-09-21T10:15:40.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,4,5", 2024-09-21T10:15:40.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.607345+0000", 2024-09-21T10:15:40.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6730000000000006e-05 2024-09-21T10:15:40.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:40.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.614298+0000", 2024-09-21T10:15:40.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0069524610000000001 2024-09-21T10:15:40.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.614380+0000", 2024-09-21T10:15:40.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2667e-05 2024-09-21T10:15:40.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.615136+0000", 2024-09-21T10:15:40.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00075546800000000002 2024-09-21T10:15:40.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.615447+0000", 2024-09-21T10:15:40.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00031086200000000002 2024-09-21T10:15:40.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:15:40.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.615478+0000", 2024-09-21T10:15:40.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1129e-05 2024-09-21T10:15:40.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:15:40.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.615584+0000", 2024-09-21T10:15:40.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105895 2024-09-21T10:15:40.398 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.398 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:40.398 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.398 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:267706 5.f 5:fcecc627:::benchmark_data_smithi098_135187_object132068:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-21T10:15:40.398 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:07:59.426683+0000", 2024-09-21T10:15:40.398 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 460.951262692, 2024-09-21T10:15:40.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1988511480000001, 2024-09-21T10:15:40.398 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:40.398 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:40.399 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:40.399 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:15:40.399 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:15:40.399 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 267706 2024-09-21T10:15:40.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.399 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:40.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:40.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.426683+0000", 2024-09-21T10:15:40.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:40.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.426683+0000", 2024-09-21T10:15:40.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:40.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.426682+0000", 2024-09-21T10:15:40.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:15:40.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:40.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.426689+0000", 2024-09-21T10:15:40.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6709999999999999e-06 2024-09-21T10:15:40.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:40.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.426703+0000", 2024-09-21T10:15:40.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3968000000000001e-05 2024-09-21T10:15:40.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:40.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.426709+0000", 2024-09-21T10:15:40.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3909999999999997e-06 2024-09-21T10:15:40.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.426759+0000", 2024-09-21T10:15:40.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0587000000000002e-05 2024-09-21T10:15:40.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:15:40.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:07:59.426768+0000", 2024-09-21T10:15:40.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3680000000000002e-06 2024-09-21T10:15:40.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:00.374044+0000", 2024-09-21T10:15:40.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.947276377 2024-09-21T10:15:40.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:15:40.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:00.374051+0000", 2024-09-21T10:15:40.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8349999999999998e-06 2024-09-21T10:15:40.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.610059+0000", 2024-09-21T10:15:40.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2360081759999999 2024-09-21T10:15:40.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:40.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.610184+0000", 2024-09-21T10:15:40.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012485799999999999 2024-09-21T10:15:40.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,4,5", 2024-09-21T10:15:40.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.610310+0000", 2024-09-21T10:15:40.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012570699999999999 2024-09-21T10:15:40.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:40.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.614563+0000", 2024-09-21T10:15:40.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042535170000000001 2024-09-21T10:15:40.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.615411+0000", 2024-09-21T10:15:40.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00084803600000000004 2024-09-21T10:15:40.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.615693+0000", 2024-09-21T10:15:40.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00028183199999999998 2024-09-21T10:15:40.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.625367+0000", 2024-09-21T10:15:40.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0096744350000000003 2024-09-21T10:15:40.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:15:40.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.625422+0000", 2024-09-21T10:15:40.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4279e-05 2024-09-21T10:15:40.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:15:40.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:01.625534+0000", 2024-09-21T10:15:40.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000112825 2024-09-21T10:15:40.405 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.406 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:40.406 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.406 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:268913 5.f 5:fe84a087:::benchmark_data_smithi098_135187_object133275:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-21T10:15:40.406 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:08:03.451359+0000", 2024-09-21T10:15:40.406 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 456.92658697100001, 2024-09-21T10:15:40.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0111894130000001, 2024-09-21T10:15:40.406 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:40.406 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:40.406 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:40.406 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:15:40.406 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:15:40.407 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 268913 2024-09-21T10:15:40.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.407 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:40.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:40.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451359+0000", 2024-09-21T10:15:40.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:40.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451359+0000", 2024-09-21T10:15:40.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:40.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451357+0000", 2024-09-21T10:15:40.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:15:40.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:40.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451365+0000", 2024-09-21T10:15:40.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5190000000000003e-06 2024-09-21T10:15:40.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:40.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451383+0000", 2024-09-21T10:15:40.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8697000000000001e-05 2024-09-21T10:15:40.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:40.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451391+0000", 2024-09-21T10:15:40.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5619999999999998e-06 2024-09-21T10:15:40.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451437+0000", 2024-09-21T10:15:40.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6418999999999997e-05 2024-09-21T10:15:40.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:15:40.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451443+0000", 2024-09-21T10:15:40.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8320000000000002e-06 2024-09-21T10:15:40.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.453088+0000", 2024-09-21T10:15:40.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0016444179999999 2024-09-21T10:15:40.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:40.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.453176+0000", 2024-09-21T10:15:40.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8582e-05 2024-09-21T10:15:40.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:15:40.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.453250+0000", 2024-09-21T10:15:40.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4245999999999994e-05 2024-09-21T10:15:40.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:40.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462271+0000", 2024-09-21T10:15:40.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0090206030000000003 2024-09-21T10:15:40.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462311+0000", 2024-09-21T10:15:40.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9798000000000001e-05 2024-09-21T10:15:40.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462477+0000", 2024-09-21T10:15:40.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000165845 2024-09-21T10:15:40.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:15:40.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462526+0000", 2024-09-21T10:15:40.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9472999999999999e-05 2024-09-21T10:15:40.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:15:40.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462548+0000", 2024-09-21T10:15:40.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2235e-05 2024-09-21T10:15:40.412 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.412 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:40.412 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.413 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:268916 5.f 5:f65a8683:::benchmark_data_smithi098_135187_object133278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-21T10:15:40.413 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:08:03.451594+0000", 2024-09-21T10:15:40.413 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 456.92635234800002, 2024-09-21T10:15:40.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.011084791, 2024-09-21T10:15:40.413 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:40.413 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:40.413 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:40.413 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:15:40.413 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:15:40.413 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 268916 2024-09-21T10:15:40.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.413 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:40.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:40.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451594+0000", 2024-09-21T10:15:40.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:40.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451594+0000", 2024-09-21T10:15:40.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:40.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451593+0000", 2024-09-21T10:15:40.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:15:40.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:40.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451602+0000", 2024-09-21T10:15:40.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9290000000000004e-06 2024-09-21T10:15:40.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:40.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451613+0000", 2024-09-21T10:15:40.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1637999999999999e-05 2024-09-21T10:15:40.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:40.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451619+0000", 2024-09-21T10:15:40.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4729999999999999e-06 2024-09-21T10:15:40.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451664+0000", 2024-09-21T10:15:40.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5396999999999999e-05 2024-09-21T10:15:40.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:15:40.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.451670+0000", 2024-09-21T10:15:40.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0390000000000001e-06 2024-09-21T10:15:40.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.456144+0000", 2024-09-21T10:15:40.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.004474139 2024-09-21T10:15:40.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:40.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.456256+0000", 2024-09-21T10:15:40.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000111902 2024-09-21T10:15:40.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:15:40.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.456363+0000", 2024-09-21T10:15:40.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107057 2024-09-21T10:15:40.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462350+0000", 2024-09-21T10:15:40.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059866920000000001 2024-09-21T10:15:40.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:40.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462434+0000", 2024-09-21T10:15:40.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3663000000000002e-05 2024-09-21T10:15:40.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462635+0000", 2024-09-21T10:15:40.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000201184 2024-09-21T10:15:40.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:15:40.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462660+0000", 2024-09-21T10:15:40.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4604999999999999e-05 2024-09-21T10:15:40.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:15:40.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462678+0000", 2024-09-21T10:15:40.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8858000000000001e-05 2024-09-21T10:15:40.419 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.419 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:40.419 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.419 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:268925 5.f 5:f31b12d3:::benchmark_data_smithi098_135187_object133287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-21T10:15:40.420 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:08:03.462291+0000", 2024-09-21T10:15:40.420 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 456.91565530999998, 2024-09-21T10:15:40.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0005015799999999, 2024-09-21T10:15:40.420 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:40.420 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:40.420 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:40.420 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:15:40.420 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:15:40.420 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 268925 2024-09-21T10:15:40.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.420 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:40.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:40.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.462291+0000", 2024-09-21T10:15:40.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:40.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.462291+0000", 2024-09-21T10:15:40.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:40.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.462289+0000", 2024-09-21T10:15:40.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:15:40.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:40.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.462296+0000", 2024-09-21T10:15:40.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7329999999999997e-06 2024-09-21T10:15:40.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:40.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.462308+0000", 2024-09-21T10:15:40.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1691e-05 2024-09-21T10:15:40.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:40.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.462314+0000", 2024-09-21T10:15:40.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8130000000000001e-06 2024-09-21T10:15:40.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.462361+0000", 2024-09-21T10:15:40.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7240000000000002e-05 2024-09-21T10:15:40.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:15:40.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.462367+0000", 2024-09-21T10:15:40.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7509999999999997e-06 2024-09-21T10:15:40.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.458802+0000", 2024-09-21T10:15:40.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99643555699999997 2024-09-21T10:15:40.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:40.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.458919+0000", 2024-09-21T10:15:40.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000117297 2024-09-21T10:15:40.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:15:40.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.459025+0000", 2024-09-21T10:15:40.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105159 2024-09-21T10:15:40.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462592+0000", 2024-09-21T10:15:40.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00356719 2024-09-21T10:15:40.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:40.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462602+0000", 2024-09-21T10:15:40.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9890000000000004e-06 2024-09-21T10:15:40.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462718+0000", 2024-09-21T10:15:40.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000116588 2024-09-21T10:15:40.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:15:40.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462757+0000", 2024-09-21T10:15:40.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8271e-05 2024-09-21T10:15:40.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:15:40.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462792+0000", 2024-09-21T10:15:40.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.561e-05 2024-09-21T10:15:40.426 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.426 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:40.426 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.426 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:268939 5.f 5:f8ea3d45:::benchmark_data_smithi098_135187_object133301:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-21T10:15:40.426 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:08:03.483485+0000", 2024-09-21T10:15:40.426 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 456.89446057599997, 2024-09-21T10:15:40.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.98675756599999997, 2024-09-21T10:15:40.426 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:40.427 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:40.427 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:40.427 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:15:40.427 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:15:40.427 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 268939 2024-09-21T10:15:40.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.427 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:40.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:40.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.483485+0000", 2024-09-21T10:15:40.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:40.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.483485+0000", 2024-09-21T10:15:40.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:40.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.483483+0000", 2024-09-21T10:15:40.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:15:40.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:40.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.483492+0000", 2024-09-21T10:15:40.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1310000000000006e-06 2024-09-21T10:15:40.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:40.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.483510+0000", 2024-09-21T10:15:40.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8431e-05 2024-09-21T10:15:40.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:40.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.483516+0000", 2024-09-21T10:15:40.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0299999999999999e-06 2024-09-21T10:15:40.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.483561+0000", 2024-09-21T10:15:40.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4793999999999998e-05 2024-09-21T10:15:40.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:15:40.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.483571+0000", 2024-09-21T10:15:40.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0135e-05 2024-09-21T10:15:40.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462824+0000", 2024-09-21T10:15:40.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97925344000000003 2024-09-21T10:15:40.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:40.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462901+0000", 2024-09-21T10:15:40.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6489999999999994e-05 2024-09-21T10:15:40.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:15:40.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.462960+0000", 2024-09-21T10:15:40.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9107999999999997e-05 2024-09-21T10:15:40.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.470058+0000", 2024-09-21T10:15:40.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0070984150000000003 2024-09-21T10:15:40.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.470100+0000", 2024-09-21T10:15:40.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1520000000000002e-05 2024-09-21T10:15:40.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:40.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.470109+0000", 2024-09-21T10:15:40.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5559999999999996e-06 2024-09-21T10:15:40.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:15:40.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.470202+0000", 2024-09-21T10:15:40.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2547999999999996e-05 2024-09-21T10:15:40.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:15:40.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.470243+0000", 2024-09-21T10:15:40.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0985999999999997e-05 2024-09-21T10:15:40.432 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.432 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:40.432 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.433 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:268948 5.f 5:f12efb92:::benchmark_data_smithi098_135187_object133310:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-21T10:15:40.433 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:08:03.495650+0000", 2024-09-21T10:15:40.433 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 456.88229644099999, 2024-09-21T10:15:40.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99013958999999996, 2024-09-21T10:15:40.433 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:40.433 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:40.433 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:40.433 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:15:40.433 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:15:40.433 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 268948 2024-09-21T10:15:40.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.433 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:40.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:40.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.495650+0000", 2024-09-21T10:15:40.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:40.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.495650+0000", 2024-09-21T10:15:40.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:40.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.495648+0000", 2024-09-21T10:15:40.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:15:40.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:40.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.495656+0000", 2024-09-21T10:15:40.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6070000000000002e-06 2024-09-21T10:15:40.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:40.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.495670+0000", 2024-09-21T10:15:40.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4619e-05 2024-09-21T10:15:40.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:40.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.495676+0000", 2024-09-21T10:15:40.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9719999999999997e-06 2024-09-21T10:15:40.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.495726+0000", 2024-09-21T10:15:40.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9317999999999998e-05 2024-09-21T10:15:40.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:15:40.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.495736+0000", 2024-09-21T10:15:40.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0316000000000001e-05 2024-09-21T10:15:40.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.466400+0000", 2024-09-21T10:15:40.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97066425899999997 2024-09-21T10:15:40.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:40.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.466517+0000", 2024-09-21T10:15:40.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011711500000000001 2024-09-21T10:15:40.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:15:40.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.466628+0000", 2024-09-21T10:15:40.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000110221 2024-09-21T10:15:40.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.470299+0000", 2024-09-21T10:15:40.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003671278 2024-09-21T10:15:40.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.470333+0000", 2024-09-21T10:15:40.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3581999999999998e-05 2024-09-21T10:15:40.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:40.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.485700+0000", 2024-09-21T10:15:40.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015367159 2024-09-21T10:15:40.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:15:40.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.485747+0000", 2024-09-21T10:15:40.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6897000000000002e-05 2024-09-21T10:15:40.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:15:40.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.485789+0000", 2024-09-21T10:15:40.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2540999999999998e-05 2024-09-21T10:15:40.440 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.440 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:40.440 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.440 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:268977 5.f 5:f9611bc4:::benchmark_data_smithi098_135187_object133339:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-21T10:15:40.440 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:08:03.534671+0000", 2024-09-21T10:15:40.440 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 456.84327496899999, 2024-09-21T10:15:40.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.95125447399999996, 2024-09-21T10:15:40.440 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:40.440 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:40.441 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:40.441 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:15:40.441 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:15:40.441 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 268977 2024-09-21T10:15:40.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.441 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:40.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:40.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.534671+0000", 2024-09-21T10:15:40.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:40.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.534671+0000", 2024-09-21T10:15:40.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:40.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.534670+0000", 2024-09-21T10:15:40.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:15:40.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:40.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.534677+0000", 2024-09-21T10:15:40.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1019999999999999e-06 2024-09-21T10:15:40.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:40.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.534690+0000", 2024-09-21T10:15:40.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3586e-05 2024-09-21T10:15:40.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:40.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.534696+0000", 2024-09-21T10:15:40.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7559999999999996e-06 2024-09-21T10:15:40.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.534769+0000", 2024-09-21T10:15:40.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3295999999999998e-05 2024-09-21T10:15:40.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:15:40.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.534779+0000", 2024-09-21T10:15:40.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8330000000000006e-06 2024-09-21T10:15:40.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.470351+0000", 2024-09-21T10:15:40.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.93557197700000005 2024-09-21T10:15:40.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:40.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.470478+0000", 2024-09-21T10:15:40.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012648999999999999 2024-09-21T10:15:40.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:15:40.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.470578+0000", 2024-09-21T10:15:40.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010086699999999999 2024-09-21T10:15:40.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.478079+0000", 2024-09-21T10:15:40.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0075000980000000002 2024-09-21T10:15:40.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.478104+0000", 2024-09-21T10:15:40.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5689000000000001e-05 2024-09-21T10:15:40.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:40.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.485838+0000", 2024-09-21T10:15:40.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077339920000000003 2024-09-21T10:15:40.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.446 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:15:40.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.485881+0000", 2024-09-21T10:15:40.446 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3013999999999998e-05 2024-09-21T10:15:40.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.446 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:15:40.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.485925+0000", 2024-09-21T10:15:40.446 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4223e-05 2024-09-21T10:15:40.446 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.446 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:40.446 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.447 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:269009 5.f 5:f68d3502:::benchmark_data_smithi098_135187_object133371:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-21T10:15:40.447 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:08:03.580762+0000", 2024-09-21T10:15:40.447 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 456.797184023, 2024-09-21T10:15:40.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.90523814300000005, 2024-09-21T10:15:40.447 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:40.447 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:40.447 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:40.447 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:15:40.447 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:15:40.447 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 269009 2024-09-21T10:15:40.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.447 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:40.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:40.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.580762+0000", 2024-09-21T10:15:40.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:40.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.580762+0000", 2024-09-21T10:15:40.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:40.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.580761+0000", 2024-09-21T10:15:40.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:15:40.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:40.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.580768+0000", 2024-09-21T10:15:40.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1810000000000004e-06 2024-09-21T10:15:40.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:40.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.580781+0000", 2024-09-21T10:15:40.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3421000000000001e-05 2024-09-21T10:15:40.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:40.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.580787+0000", 2024-09-21T10:15:40.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6710000000000004e-06 2024-09-21T10:15:40.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.580824+0000", 2024-09-21T10:15:40.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7515999999999998e-05 2024-09-21T10:15:40.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:15:40.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.580832+0000", 2024-09-21T10:15:40.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2300000000000002e-06 2024-09-21T10:15:40.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.474329+0000", 2024-09-21T10:15:40.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.89349757200000002 2024-09-21T10:15:40.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:40.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.474434+0000", 2024-09-21T10:15:40.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104351 2024-09-21T10:15:40.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:15:40.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.474536+0000", 2024-09-21T10:15:40.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010213 2024-09-21T10:15:40.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.478132+0000", 2024-09-21T10:15:40.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0035959859999999998 2024-09-21T10:15:40.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.485833+0000", 2024-09-21T10:15:40.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077007810000000003 2024-09-21T10:15:40.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:40.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.485931+0000", 2024-09-21T10:15:40.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8894000000000006e-05 2024-09-21T10:15:40.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:15:40.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.485967+0000", 2024-09-21T10:15:40.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5966000000000001e-05 2024-09-21T10:15:40.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:15:40.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.486000+0000", 2024-09-21T10:15:40.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2727e-05 2024-09-21T10:15:40.453 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.453 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:40.453 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.453 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:269025 5.f 5:f5c512e1:::benchmark_data_smithi098_135187_object133387:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-21T10:15:40.453 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:08:03.627608+0000", 2024-09-21T10:15:40.453 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 456.75033787500001, 2024-09-21T10:15:40.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.85976989199999998, 2024-09-21T10:15:40.454 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:40.454 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:40.454 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:40.454 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:15:40.454 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:15:40.454 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 269025 2024-09-21T10:15:40.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.454 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:40.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:40.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.627608+0000", 2024-09-21T10:15:40.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:40.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.627608+0000", 2024-09-21T10:15:40.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:40.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.627607+0000", 2024-09-21T10:15:40.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:15:40.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:40.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.627615+0000", 2024-09-21T10:15:40.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8240000000000006e-06 2024-09-21T10:15:40.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:40.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.627627+0000", 2024-09-21T10:15:40.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1963999999999999e-05 2024-09-21T10:15:40.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:40.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.627634+0000", 2024-09-21T10:15:40.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.516e-06 2024-09-21T10:15:40.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.627672+0000", 2024-09-21T10:15:40.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8012000000000001e-05 2024-09-21T10:15:40.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:15:40.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.627679+0000", 2024-09-21T10:15:40.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3950000000000003e-06 2024-09-21T10:15:40.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.478150+0000", 2024-09-21T10:15:40.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.85047037999999997 2024-09-21T10:15:40.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:40.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.478251+0000", 2024-09-21T10:15:40.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000101469 2024-09-21T10:15:40.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:15:40.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.478357+0000", 2024-09-21T10:15:40.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105981 2024-09-21T10:15:40.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.485693+0000", 2024-09-21T10:15:40.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0073359719999999996 2024-09-21T10:15:40.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:40.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.486005+0000", 2024-09-21T10:15:40.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00031183900000000001 2024-09-21T10:15:40.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.487305+0000", 2024-09-21T10:15:40.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001300163 2024-09-21T10:15:40.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:15:40.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.487345+0000", 2024-09-21T10:15:40.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0055000000000002e-05 2024-09-21T10:15:40.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:15:40.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.487378+0000", 2024-09-21T10:15:40.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2793000000000003e-05 2024-09-21T10:15:40.460 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.460 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:40.460 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.460 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.44894.0:269029 5.f 5:f51cff0c:::benchmark_data_smithi098_135187_object133391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-21T10:15:40.460 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:08:03.645273+0000", 2024-09-21T10:15:40.460 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 456.73267310900002, 2024-09-21T10:15:40.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.84219354899999999, 2024-09-21T10:15:40.460 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:40.460 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:40.460 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:40.460 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.44894", 2024-09-21T10:15:40.460 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1104504876", 2024-09-21T10:15:40.461 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 269029 2024-09-21T10:15:40.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.461 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:40.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:40.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.645273+0000", 2024-09-21T10:15:40.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:40.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.645273+0000", 2024-09-21T10:15:40.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:40.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.645266+0000", 2024-09-21T10:15:40.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999933 2024-09-21T10:15:40.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:40.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.645284+0000", 2024-09-21T10:15:40.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7552999999999999e-05 2024-09-21T10:15:40.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:40.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.645317+0000", 2024-09-21T10:15:40.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3918999999999998e-05 2024-09-21T10:15:40.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:40.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.645335+0000", 2024-09-21T10:15:40.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7193e-05 2024-09-21T10:15:40.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.645454+0000", 2024-09-21T10:15:40.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119295 2024-09-21T10:15:40.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:15:40.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:03.645467+0000", 2024-09-21T10:15:40.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2969000000000001e-05 2024-09-21T10:15:40.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.481903+0000", 2024-09-21T10:15:40.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83643561499999997 2024-09-21T10:15:40.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:40.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.482031+0000", 2024-09-21T10:15:40.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012832499999999999 2024-09-21T10:15:40.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:15:40.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.482139+0000", 2024-09-21T10:15:40.465 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108121 2024-09-21T10:15:40.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.486032+0000", 2024-09-21T10:15:40.465 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0038929809999999998 2024-09-21T10:15:40.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:40.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.486037+0000", 2024-09-21T10:15:40.465 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3029999999999999e-06 2024-09-21T10:15:40.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.487407+0000", 2024-09-21T10:15:40.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001369459 2024-09-21T10:15:40.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:15:40.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.487440+0000", 2024-09-21T10:15:40.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2870000000000002e-05 2024-09-21T10:15:40.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:15:40.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:08:04.487466+0000", 2024-09-21T10:15:40.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6798999999999999e-05 2024-09-21T10:15:40.467 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.467 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:40.467 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.467 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.467 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:134299 7.c 7:32ac5e8c:::benchmark_data_smithi098_197834_object17782:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T10:15:40.467 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:15:06.412100+0000", 2024-09-21T10:15:40.467 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 33.965845645999998, 2024-09-21T10:15:40.467 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.11364442900000001, 2024-09-21T10:15:40.467 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:40.467 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:40.467 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:40.467 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:15:40.467 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:15:40.468 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 134299 2024-09-21T10:15:40.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.468 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:40.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:40.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:06.412100+0000", 2024-09-21T10:15:40.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:40.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:06.412100+0000", 2024-09-21T10:15:40.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:40.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:06.412100+0000", 2024-09-21T10:15:40.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:15:40.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:40.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:06.412102+0000", 2024-09-21T10:15:40.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4789999999999999e-06 2024-09-21T10:15:40.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:40.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:06.412108+0000", 2024-09-21T10:15:40.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9320000000000001e-06 2024-09-21T10:15:40.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:40.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:06.412112+0000", 2024-09-21T10:15:40.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.411e-06 2024-09-21T10:15:40.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:06.412155+0000", 2024-09-21T10:15:40.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3112000000000003e-05 2024-09-21T10:15:40.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:40.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:06.412283+0000", 2024-09-21T10:15:40.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012855399999999999 2024-09-21T10:15:40.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:15:40.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:06.412403+0000", 2024-09-21T10:15:40.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011974 2024-09-21T10:15:40.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:06.415601+0000", 2024-09-21T10:15:40.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0031980950000000002 2024-09-21T10:15:40.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:40.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:06.415741+0000", 2024-09-21T10:15:40.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013950599999999999 2024-09-21T10:15:40.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:06.525688+0000", 2024-09-21T10:15:40.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.10994699199999999 2024-09-21T10:15:40.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:15:40.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:06.525718+0000", 2024-09-21T10:15:40.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0719999999999997e-05 2024-09-21T10:15:40.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.473 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:15:40.473 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:06.525745+0000", 2024-09-21T10:15:40.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6322999999999999e-05 2024-09-21T10:15:40.473 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.473 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:40.473 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.473 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.473 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:136597 7.d 7:b147bd38:::benchmark_data_smithi098_197834_object20080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T10:15:40.473 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:15:08.942513+0000", 2024-09-21T10:15:40.473 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 31.435433132, 2024-09-21T10:15:40.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.114030988, 2024-09-21T10:15:40.473 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:40.473 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:40.474 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:40.474 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:15:40.474 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:15:40.474 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136597 2024-09-21T10:15:40.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.474 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:40.474 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.474 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:40.474 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:08.942513+0000", 2024-09-21T10:15:40.474 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.474 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.474 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:40.474 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:08.942513+0000", 2024-09-21T10:15:40.475 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:40.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:08.942512+0000", 2024-09-21T10:15:40.475 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:15:40.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:40.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:08.942518+0000", 2024-09-21T10:15:40.475 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9200000000000001e-06 2024-09-21T10:15:40.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:40.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:08.942531+0000", 2024-09-21T10:15:40.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3672000000000001e-05 2024-09-21T10:15:40.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:40.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:08.942537+0000", 2024-09-21T10:15:40.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5489999999999997e-06 2024-09-21T10:15:40.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:08.943868+0000", 2024-09-21T10:15:40.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0013309280000000001 2024-09-21T10:15:40.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:40.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:08.944019+0000", 2024-09-21T10:15:40.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015123400000000001 2024-09-21T10:15:40.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.477 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.477 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:15:40.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:08.944161+0000", 2024-09-21T10:15:40.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000141877 2024-09-21T10:15:40.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.477 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.477 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:40.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:08.951382+0000", 2024-09-21T10:15:40.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0072214360000000003 2024-09-21T10:15:40.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.477 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:08.954755+0000", 2024-09-21T10:15:40.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0033721290000000002 2024-09-21T10:15:40.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:09.056475+0000", 2024-09-21T10:15:40.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.101720212 2024-09-21T10:15:40.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:15:40.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:09.056507+0000", 2024-09-21T10:15:40.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1806e-05 2024-09-21T10:15:40.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.479 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:15:40.479 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:09.056544+0000", 2024-09-21T10:15:40.479 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7273e-05 2024-09-21T10:15:40.479 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.479 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:40.479 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.479 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.479 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:157417 7.d 7:bb2092b0:::benchmark_data_smithi098_197834_object40900:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-21T10:15:40.479 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:15:32.112456+0000", 2024-09-21T10:15:40.479 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.2654903480000002, 2024-09-21T10:15:40.479 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.123836614, 2024-09-21T10:15:40.479 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:40.480 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:40.480 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:40.480 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:15:40.480 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:15:40.480 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 157417 2024-09-21T10:15:40.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.480 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:40.480 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.480 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:40.480 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.112456+0000", 2024-09-21T10:15:40.480 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.480 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.480 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:40.480 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.112456+0000", 2024-09-21T10:15:40.481 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:40.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.112454+0000", 2024-09-21T10:15:40.481 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:15:40.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:40.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.112460+0000", 2024-09-21T10:15:40.481 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1e-06 2024-09-21T10:15:40.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:40.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.112475+0000", 2024-09-21T10:15:40.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4729000000000001e-05 2024-09-21T10:15:40.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.482 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:40.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.112481+0000", 2024-09-21T10:15:40.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.409e-06 2024-09-21T10:15:40.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.482 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.112525+0000", 2024-09-21T10:15:40.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3169e-05 2024-09-21T10:15:40.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:40.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.112603+0000", 2024-09-21T10:15:40.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8132000000000003e-05 2024-09-21T10:15:40.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:15:40.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.112668+0000", 2024-09-21T10:15:40.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5628000000000003e-05 2024-09-21T10:15:40.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:40.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.120583+0000", 2024-09-21T10:15:40.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0079145940000000005 2024-09-21T10:15:40.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.484 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.484 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.124497+0000", 2024-09-21T10:15:40.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039137529999999999 2024-09-21T10:15:40.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.484 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.484 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.236203+0000", 2024-09-21T10:15:40.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.111706138 2024-09-21T10:15:40.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.484 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:15:40.484 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.236254+0000", 2024-09-21T10:15:40.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1504e-05 2024-09-21T10:15:40.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.485 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:15:40.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.236292+0000", 2024-09-21T10:15:40.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7839000000000001e-05 2024-09-21T10:15:40.485 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.485 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:40.485 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.485 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:157427 7.d 7:b3143701:::benchmark_data_smithi098_197834_object40910:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-21T10:15:40.485 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:15:32.115973+0000", 2024-09-21T10:15:40.485 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.2619733160000006, 2024-09-21T10:15:40.486 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.120430486, 2024-09-21T10:15:40.486 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:40.486 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:40.486 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:40.486 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:15:40.486 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:15:40.486 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 157427 2024-09-21T10:15:40.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.486 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:40.486 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.486 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:40.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.115973+0000", 2024-09-21T10:15:40.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:40.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.115973+0000", 2024-09-21T10:15:40.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:40.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.115971+0000", 2024-09-21T10:15:40.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:15:40.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:40.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.115978+0000", 2024-09-21T10:15:40.488 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0339999999999999e-06 2024-09-21T10:15:40.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:40.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.115996+0000", 2024-09-21T10:15:40.488 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7819999999999999e-05 2024-09-21T10:15:40.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:40.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.116002+0000", 2024-09-21T10:15:40.488 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2060000000000004e-06 2024-09-21T10:15:40.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.489 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.116045+0000", 2024-09-21T10:15:40.489 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3133999999999997e-05 2024-09-21T10:15:40.489 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.489 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.489 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:40.489 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.116176+0000", 2024-09-21T10:15:40.489 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013051299999999999 2024-09-21T10:15:40.489 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.489 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.489 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:15:40.489 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.116317+0000", 2024-09-21T10:15:40.489 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000140743 2024-09-21T10:15:40.489 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.489 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.489 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:40.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.120702+0000", 2024-09-21T10:15:40.490 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:40.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004385707 2024-09-21T10:15:40.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.491 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.491 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.124599+0000", 2024-09-21T10:15:40.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0038968919999999999 2024-09-21T10:15:40.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.491 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.492 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.492 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.236329+0000", 2024-09-21T10:15:40.492 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.11173010899999999 2024-09-21T10:15:40.492 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.492 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.492 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:15:40.492 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.236369+0000", 2024-09-21T10:15:40.492 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9780999999999997e-05 2024-09-21T10:15:40.492 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.492 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.492 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:15:40.492 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.236403+0000", 2024-09-21T10:15:40.492 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3967000000000002e-05 2024-09-21T10:15:40.493 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.493 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:40.493 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.493 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.493 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.493 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:157492 7.d 7:bc80f3fd:::benchmark_data_smithi098_197834_object40975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-21T10:15:40.493 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:15:32.273499+0000", 2024-09-21T10:15:40.493 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.1044470319999995, 2024-09-21T10:15:40.493 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.33144177200000002, 2024-09-21T10:15:40.493 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:40.493 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:40.493 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:40.493 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:15:40.494 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:15:40.494 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 157492 2024-09-21T10:15:40.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.494 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:40.494 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:40.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.273499+0000", 2024-09-21T10:15:40.494 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.494 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:40.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.273499+0000", 2024-09-21T10:15:40.494 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.495 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:40.495 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.273498+0000", 2024-09-21T10:15:40.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:15:40.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.495 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:40.495 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.273506+0000", 2024-09-21T10:15:40.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.323e-06 2024-09-21T10:15:40.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.495 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:40.495 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.273518+0000", 2024-09-21T10:15:40.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2787e-05 2024-09-21T10:15:40.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.496 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.496 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:40.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.273523+0000", 2024-09-21T10:15:40.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4739999999999999e-06 2024-09-21T10:15:40.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.496 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.496 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.273571+0000", 2024-09-21T10:15:40.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7797000000000003e-05 2024-09-21T10:15:40.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.496 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.497 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:40.497 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.273643+0000", 2024-09-21T10:15:40.497 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2737999999999994e-05 2024-09-21T10:15:40.497 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.497 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.497 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:15:40.497 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.273730+0000", 2024-09-21T10:15:40.497 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6767000000000005e-05 2024-09-21T10:15:40.497 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.497 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.497 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:40.497 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.277311+0000", 2024-09-21T10:15:40.497 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0035806710000000001 2024-09-21T10:15:40.498 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.498 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.498 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.498 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.277676+0000", 2024-09-21T10:15:40.498 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00036562999999999998 2024-09-21T10:15:40.498 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.498 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.498 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.498 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.604887+0000", 2024-09-21T10:15:40.498 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.32721010099999998 2024-09-21T10:15:40.498 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.498 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.499 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:15:40.499 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.604918+0000", 2024-09-21T10:15:40.499 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1423000000000001e-05 2024-09-21T10:15:40.499 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.499 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.499 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:15:40.499 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.604941+0000", 2024-09-21T10:15:40.499 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2796e-05 2024-09-21T10:15:40.499 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.499 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:40.499 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.499 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.500 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.500 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:157496 7.d 7:b7a01abf:::benchmark_data_smithi098_197834_object40979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-21T10:15:40.500 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:15:32.278211+0000", 2024-09-21T10:15:40.500 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.0997346409999995, 2024-09-21T10:15:40.500 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.32681266599999997, 2024-09-21T10:15:40.500 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:40.500 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:40.500 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:40.500 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:15:40.500 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:15:40.500 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 157496 2024-09-21T10:15:40.500 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.500 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:40.501 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.501 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:40.501 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.278211+0000", 2024-09-21T10:15:40.501 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.501 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.501 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.501 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:40.501 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.278211+0000", 2024-09-21T10:15:40.501 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.501 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.501 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.501 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:40.502 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.278210+0000", 2024-09-21T10:15:40.502 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:15:40.502 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.502 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.502 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:40.502 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.278217+0000", 2024-09-21T10:15:40.502 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8900000000000001e-06 2024-09-21T10:15:40.502 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.502 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.502 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:40.502 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.278230+0000", 2024-09-21T10:15:40.502 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.331e-05 2024-09-21T10:15:40.502 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.503 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.503 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:40.503 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.278236+0000", 2024-09-21T10:15:40.503 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9519999999999999e-06 2024-09-21T10:15:40.503 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.503 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.503 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.503 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.278274+0000", 2024-09-21T10:15:40.503 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7956e-05 2024-09-21T10:15:40.503 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.503 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.503 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:40.503 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.278399+0000", 2024-09-21T10:15:40.503 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012527299999999999 2024-09-21T10:15:40.504 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.504 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.504 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:15:40.504 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.278522+0000", 2024-09-21T10:15:40.504 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012304300000000001 2024-09-21T10:15:40.504 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.504 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.504 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:40.504 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.281470+0000", 2024-09-21T10:15:40.504 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002947486 2024-09-21T10:15:40.504 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.504 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.504 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.504 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.282303+0000", 2024-09-21T10:15:40.505 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00083261499999999998 2024-09-21T10:15:40.505 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.505 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.505 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.505 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.604972+0000", 2024-09-21T10:15:40.505 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.32266949700000003 2024-09-21T10:15:40.505 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.505 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.505 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:15:40.505 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.605003+0000", 2024-09-21T10:15:40.505 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0815999999999998e-05 2024-09-21T10:15:40.505 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.505 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.505 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:15:40.506 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:32.605024+0000", 2024-09-21T10:15:40.506 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1155999999999999e-05 2024-09-21T10:15:40.506 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.506 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:40.506 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.506 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.506 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.506 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:164057 7.7 7:e56dc0b5:::benchmark_data_smithi098_197834_object47540:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-09-21T10:15:40.506 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:15:40.076937+0000", 2024-09-21T10:15:40.506 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.30100865199999999, 2024-09-21T10:15:40.506 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.12942624999999999, 2024-09-21T10:15:40.506 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:40.506 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:40.507 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:40.507 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:15:40.507 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:15:40.507 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 164057 2024-09-21T10:15:40.507 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.507 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:40.507 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.507 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:40.507 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.076937+0000", 2024-09-21T10:15:40.507 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.507 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.507 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.507 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:40.508 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.076937+0000", 2024-09-21T10:15:40.508 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.508 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.508 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.508 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:40.508 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.076936+0000", 2024-09-21T10:15:40.508 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:15:40.508 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.508 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.508 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:40.508 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.076942+0000", 2024-09-21T10:15:40.508 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8860000000000001e-06 2024-09-21T10:15:40.508 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.508 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.509 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:40.509 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.076953+0000", 2024-09-21T10:15:40.509 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0738e-05 2024-09-21T10:15:40.509 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.509 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.509 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:40.509 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.076958+0000", 2024-09-21T10:15:40.509 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7090000000000001e-06 2024-09-21T10:15:40.509 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.509 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.509 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.509 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.076994+0000", 2024-09-21T10:15:40.509 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6702000000000002e-05 2024-09-21T10:15:40.509 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.510 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.510 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:40.510 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.077080+0000", 2024-09-21T10:15:40.510 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5526999999999996e-05 2024-09-21T10:15:40.510 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.510 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.510 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:15:40.510 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.077147+0000", 2024-09-21T10:15:40.510 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6816999999999994e-05 2024-09-21T10:15:40.510 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.510 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.510 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:40.510 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.079524+0000", 2024-09-21T10:15:40.511 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023772569999999998 2024-09-21T10:15:40.511 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.511 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.511 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.511 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.086743+0000", 2024-09-21T10:15:40.511 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0072194590000000001 2024-09-21T10:15:40.511 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.511 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.511 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.511 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.206304+0000", 2024-09-21T10:15:40.511 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.119560504 2024-09-21T10:15:40.511 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.511 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.512 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:15:40.512 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.206338+0000", 2024-09-21T10:15:40.512 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4714e-05 2024-09-21T10:15:40.512 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.512 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.512 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:15:40.512 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.206364+0000", 2024-09-21T10:15:40.512 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5085000000000001e-05 2024-09-21T10:15:40.512 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.512 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:40.512 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.512 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.512 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.512 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:164067 7.17 7:e86f401b:::benchmark_data_smithi098_197834_object47550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-09-21T10:15:40.512 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:15:40.088396+0000", 2024-09-21T10:15:40.513 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.28954999599999998, 2024-09-21T10:15:40.513 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.117967564, 2024-09-21T10:15:40.513 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:40.513 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:15:40.513 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:40.513 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:15:40.513 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:15:40.513 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 164067 2024-09-21T10:15:40.513 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.513 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:40.513 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.513 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:40.513 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.088396+0000", 2024-09-21T10:15:40.513 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.513 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.514 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.514 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:40.514 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.088396+0000", 2024-09-21T10:15:40.514 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:40.514 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.514 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.514 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:40.514 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.088394+0000", 2024-09-21T10:15:40.514 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:15:40.514 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.514 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.514 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:40.514 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.088406+0000", 2024-09-21T10:15:40.514 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1623e-05 2024-09-21T10:15:40.515 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.515 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.515 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:40.515 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.088421+0000", 2024-09-21T10:15:40.515 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5259000000000001e-05 2024-09-21T10:15:40.515 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.515 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.515 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:40.515 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.088429+0000", 2024-09-21T10:15:40.515 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4379999999999998e-06 2024-09-21T10:15:40.515 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.515 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.515 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:40.516 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.088473+0000", 2024-09-21T10:15:40.516 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4119999999999998e-05 2024-09-21T10:15:40.516 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.516 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.516 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:40.516 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.088590+0000", 2024-09-21T10:15:40.516 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000117397 2024-09-21T10:15:40.516 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.516 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.516 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:15:40.516 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.088691+0000", 2024-09-21T10:15:40.516 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010108200000000001 2024-09-21T10:15:40.516 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.516 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.517 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:40.517 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.091389+0000", 2024-09-21T10:15:40.517 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026975150000000002 2024-09-21T10:15:40.517 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.517 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.517 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.517 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.172348+0000", 2024-09-21T10:15:40.517 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.080959594999999995 2024-09-21T10:15:40.517 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.517 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.517 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:40.517 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.206313+0000", 2024-09-21T10:15:40.517 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.033964530999999999 2024-09-21T10:15:40.518 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.518 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.518 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:15:40.518 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.206341+0000", 2024-09-21T10:15:40.518 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8068e-05 2024-09-21T10:15:40.518 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:40.518 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:40.518 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:15:40.518 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:40.206364+0000", 2024-09-21T10:15:40.518 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2629999999999998e-05 2024-09-21T10:15:40.518 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.518 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:40.518 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.519 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:40.519 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:40.519 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:15:40.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:40.536 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:40.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:40.636 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:40 smithi196 ceph-mon[84784]: pgmap v735: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 7167/214497 objects degraded (3.341%); 5340/214497 objects misplaced (2.490%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:15:40.665 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:40.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:40.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:40.767 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:40.797 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:40.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:40.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:40.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:15:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:15:40] "GET /metrics HTTP/1.1" 200 36584 "" "Prometheus/2.51.0" 2024-09-21T10:15:40.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:40 smithi098 ceph-mon[97580]: pgmap v735: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 7167/214497 objects degraded (3.341%); 5340/214497 objects misplaced (2.490%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:15:40.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:40 smithi098 ceph-mon[100914]: pgmap v735: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 7167/214497 objects degraded (3.341%); 5340/214497 objects misplaced (2.490%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:15:40.898 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:40.927 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:40.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:40.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:41.029 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:41.055 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:41.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:41.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:41.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:41.188 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:41.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:41.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:41.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:41.311 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:15:41.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:41.312 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:41.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:41.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:41.446 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:41.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:41.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:41.548 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:41.579 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:41.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:41.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:41.610 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:15:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:41 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 7167/214497 objects degraded (3.341%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:15:41.639 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 10:15:41 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=cleanup t=2024-09-21T10:15:41.367307539Z level=info msg="Completed cleanup jobs" duration=12.337825ms 2024-09-21T10:15:41.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:41.707 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:41.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:41.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:41.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:41.841 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:41 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 7167/214497 objects degraded (3.341%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:15:41.841 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:41 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 7167/214497 objects degraded (3.341%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:15:41.859 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:41.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:41.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:41.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:41.986 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:41.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:41.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:42.088 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:42.113 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:42.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:42.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:42.215 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:42.238 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:42.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:42.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:42.339 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:42.384 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:42.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:42.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:42.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:42.507 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:42.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:42.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:42.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:42 smithi098 ceph-mon[97580]: pgmap v736: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 772 op/s; 7167/212427 objects degraded (3.374%); 5340/212427 objects misplaced (2.514%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:15:42.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:42 smithi098 ceph-mon[100914]: pgmap v736: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 772 op/s; 7167/212427 objects degraded (3.374%); 5340/212427 objects misplaced (2.514%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:15:42.609 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:42.634 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:42.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:42.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:42.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:42.777 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:42.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:42.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:42.797 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:42 smithi196 ceph-mon[84784]: pgmap v736: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 772 op/s; 7167/212427 objects degraded (3.374%); 5340/212427 objects misplaced (2.514%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:15:42.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:42.907 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:42.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:42.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:43.008 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:43.036 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:43.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:43.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:43.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:43.185 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:43.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:43.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:43.287 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:43.318 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:43.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:43.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:43.387 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:15:43.387 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:15:43.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.388 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:166616 7.5 7:a04a4017:::benchmark_data_smithi098_197834_object50099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-09-21T10:15:43.388 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:15:43.379193+0000", 2024-09-21T10:15:43.388 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0054182240000000001, 2024-09-21T10:15:43.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0054388680000000003, 2024-09-21T10:15:43.388 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:43.388 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:15:43.388 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:43.388 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:15:43.389 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:15:43.389 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 166616 2024-09-21T10:15:43.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.389 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:43.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:43.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.379193+0000", 2024-09-21T10:15:43.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:43.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:43.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.379193+0000", 2024-09-21T10:15:43.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:43.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:43.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.379192+0000", 2024-09-21T10:15:43.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:15:43.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:43.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.379197+0000", 2024-09-21T10:15:43.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.831e-06 2024-09-21T10:15:43.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:43.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.379208+0000", 2024-09-21T10:15:43.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0644e-05 2024-09-21T10:15:43.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:43.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.379212+0000", 2024-09-21T10:15:43.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3170000000000003e-06 2024-09-21T10:15:43.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:43.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.379250+0000", 2024-09-21T10:15:43.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7716000000000003e-05 2024-09-21T10:15:43.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:43.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.379325+0000", 2024-09-21T10:15:43.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5233000000000003e-05 2024-09-21T10:15:43.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,2", 2024-09-21T10:15:43.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.379771+0000", 2024-09-21T10:15:43.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00044584300000000002 2024-09-21T10:15:43.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:15:43.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.382776+0000", 2024-09-21T10:15:43.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030050609999999998 2024-09-21T10:15:43.394 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:43.394 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:43.394 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:43.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.395 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:166621 7.15 7:a8188543:::benchmark_data_smithi098_197834_object50104:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-09-21T10:15:43.395 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:15:43.382898+0000", 2024-09-21T10:15:43.395 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0017138730000000001, 2024-09-21T10:15:43.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00178287, 2024-09-21T10:15:43.395 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:43.395 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:15:43.395 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:15:43.395 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:15:43.395 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:15:43.396 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 166621 2024-09-21T10:15:43.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.396 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:43.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:43.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.382898+0000", 2024-09-21T10:15:43.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:43.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:43.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.382898+0000", 2024-09-21T10:15:43.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:43.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:43.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.382896+0000", 2024-09-21T10:15:43.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:15:43.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:43.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.382905+0000", 2024-09-21T10:15:43.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7499999999999998e-06 2024-09-21T10:15:43.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:43.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.382920+0000", 2024-09-21T10:15:43.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4534e-05 2024-09-21T10:15:43.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:43.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.382926+0000", 2024-09-21T10:15:43.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6250000000000004e-06 2024-09-21T10:15:43.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:43.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.382981+0000", 2024-09-21T10:15:43.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5328999999999998e-05 2024-09-21T10:15:43.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:43.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.383131+0000", 2024-09-21T10:15:43.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014965700000000001 2024-09-21T10:15:43.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:15:43.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.383588+0000", 2024-09-21T10:15:43.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00045719199999999998 2024-09-21T10:15:43.399 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:43.399 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:43.399 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:43.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.400 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.55000.0:166622 7.16 e243/237 7:6ffbc4ee:::benchmark_data_smithi098_197834_object50105:head v 243'6539, mlcod=206'664)", 2024-09-21T10:15:43.400 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:15:43.383429+0000", 2024-09-21T10:15:43.400 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.001182935, 2024-09-21T10:15:43.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012904380000000001, 2024-09-21T10:15:43.400 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:15:43.400 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:15:43.400 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:15:43.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:15:43.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.383429+0000", 2024-09-21T10:15:43.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:43.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:15:43.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.383429+0000", 2024-09-21T10:15:43.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:15:43.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:15:43.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.383428+0000", 2024-09-21T10:15:43.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:15:43.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:15:43.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.383433+0000", 2024-09-21T10:15:43.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3680000000000001e-06 2024-09-21T10:15:43.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:15:43.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.383441+0000", 2024-09-21T10:15:43.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1119999999999996e-06 2024-09-21T10:15:43.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:15:43.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.383460+0000", 2024-09-21T10:15:43.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8185999999999998e-05 2024-09-21T10:15:43.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:15:43.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.383518+0000", 2024-09-21T10:15:43.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8687000000000002e-05 2024-09-21T10:15:43.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:15:43.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:15:43.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:15:43.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:15:43.383549+0000", 2024-09-21T10:15:43.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0620000000000002e-05 2024-09-21T10:15:43.403 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:43.404 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:15:43.404 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:43.404 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:15:43.404 INFO:teuthology.orchestra.run.smithi196.stdout: ], 2024-09-21T10:15:43.404 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 3 2024-09-21T10:15:43.404 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:15:43.419 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:43.446 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:43.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:43.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:43.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:43.587 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:43.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:43.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:43.689 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:43.715 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:43.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:43.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:43.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:43.843 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:43.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:43.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:43.945 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:43.982 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:43.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:43.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:44.083 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:44.110 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:44.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:44.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:44.120 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:15:44.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:44.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:44.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:44.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:44.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:44.429 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:44.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:44.476 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:15:44.530 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:44.560 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:44.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:44.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:44.662 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:44.686 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:44.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:44.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:44.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:44 smithi196 ceph-mon[84784]: pgmap v737: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 816 op/s; 7167/205812 objects degraded (3.482%); 4947/205812 objects misplaced (2.404%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:15:44.771 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:44 smithi098 ceph-mon[100914]: pgmap v737: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 816 op/s; 7167/205812 objects degraded (3.482%); 4947/205812 objects misplaced (2.404%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:15:44.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:44 smithi098 ceph-mon[97580]: pgmap v737: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 816 op/s; 7167/205812 objects degraded (3.482%); 4947/205812 objects misplaced (2.404%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:15:44.788 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:44.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:44.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:44.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:44.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:44.941 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:44.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:44.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:45.043 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:45.067 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:45.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:45.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:45.138 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:45.169 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:45.192 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:45.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:45.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:45.293 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:45.319 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:45.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:45.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:45.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:45.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:45.465 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:45.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:45.566 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:45.591 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:45.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:45.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:45.693 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:45.722 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:45.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:45.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:45.824 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:45.866 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:45.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:45.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:45.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:46.009 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:46.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:46.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:46.111 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:46.135 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:46.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:46.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:46.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:46.247 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:15:46.247 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:15:46.248 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-21T10:15:46.248 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-21T10:15:46.248 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:15:46.258 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:46.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:46.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:46.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:46.411 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:46.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:46.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:46.513 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:46.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:46.535 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:46.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:46.630 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:46 smithi196 ceph-mon[84784]: pgmap v738: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.1 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 840 op/s; 7167/200592 objects degraded (3.573%); 4947/200592 objects misplaced (2.466%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:15:46.630 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:46 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 7167/200592 objects degraded (3.573%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:15:46.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:46.657 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:46.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:46.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:46.759 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:46.782 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:46.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:46.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:46.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:46 smithi098 ceph-mon[97580]: pgmap v738: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.1 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 840 op/s; 7167/200592 objects degraded (3.573%); 4947/200592 objects misplaced (2.466%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:15:46.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:46 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 7167/200592 objects degraded (3.573%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:15:46.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:46 smithi098 ceph-mon[100914]: pgmap v738: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.1 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 840 op/s; 7167/200592 objects degraded (3.573%); 4947/200592 objects misplaced (2.466%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:15:46.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:46 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 7167/200592 objects degraded (3.573%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:15:46.884 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:46.922 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:46.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:46.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:46.942 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:15:47.024 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:47.055 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:47.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:47.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:47.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:47.197 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:47.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:47.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:47.268 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:15:47.299 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:47.320 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:47.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:47.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:47.422 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:47.447 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:47.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:47.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:47.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:47.590 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:47.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:47.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:47.692 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:47.724 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:47.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:47.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:47.826 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:47.873 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:47.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:47.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:47.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:48.009 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:48.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:48.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:48.111 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:48.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:48.135 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:48.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:48.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:48.260 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:48.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:48.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:48.362 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:48.386 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:48.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:48.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:48.459 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:15:48.459 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:15:48.488 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:48.497 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:15:48.497 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:15:48.512 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:48.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:48.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:48.532 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:15:48.532 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:15:48.572 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:15:48.572 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:15:48.600 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:15:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:15:48.614 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:48.627 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:15:48.627 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:15:48.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:48.641 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:48.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:48.657 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:15:48.657 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:15:48.688 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:15:48.689 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:15:48.709 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:48 smithi098 ceph-mon[97580]: pgmap v739: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 836 op/s; 7167/194994 objects degraded (3.675%); 4488/194994 objects misplaced (2.302%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:15:48.710 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:48 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3635264794' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:15:48.711 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:48 smithi098 ceph-mon[100914]: pgmap v739: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 836 op/s; 7167/194994 objects degraded (3.675%); 4488/194994 objects misplaced (2.302%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:15:48.711 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:48 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3635264794' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:15:48.742 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:48.766 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:48.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:48.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:48.846 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:48 smithi196 ceph-mon[84784]: pgmap v739: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 836 op/s; 7167/194994 objects degraded (3.675%); 4488/194994 objects misplaced (2.302%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:15:48.846 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:48 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3635264794' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:15:48.869 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:48.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:48.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:49.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:49.034 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:49.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:49.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:49.117 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:15:49.117 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:15:49.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:49.173 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:49.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:49.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:49.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:49.299 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:49.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:49.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:49.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:49.437 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:49.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:49.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:49.538 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:49.566 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:49.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:49.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:49.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:49.710 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:49.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:49.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:49.812 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:49.840 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:49.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:49.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:49.843 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:15:49.941 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:49.964 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:49.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:49.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:50.065 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:50.124 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:50.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:50.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:50.225 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:50.278 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:50.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:50.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:50.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:50.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:50.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:50.485 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:15:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:15:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:15:50.534 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:50.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:50.557 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:50.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:50.658 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:50.681 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:50.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:50.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:50.783 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:50.807 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:50.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:50.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:50.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:15:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:15:50] "GET /metrics HTTP/1.1" 200 36584 "" "Prometheus/2.51.0" 2024-09-21T10:15:50.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:50 smithi098 ceph-mon[100914]: pgmap v740: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 850 op/s; 7167/189852 objects degraded (3.775%); 4488/189852 objects misplaced (2.364%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:15:50.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:50 smithi098 ceph-mon[97580]: pgmap v740: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 850 op/s; 7167/189852 objects degraded (3.775%); 4488/189852 objects misplaced (2.364%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:15:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:50 smithi196 ceph-mon[84784]: pgmap v740: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 850 op/s; 7167/189852 objects degraded (3.775%); 4488/189852 objects misplaced (2.364%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:15:50.910 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:50.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:50.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:51.048 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:51.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:51.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:51.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:51.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:51.198 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:51.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:51.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:51.299 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:51.323 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:51.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:51.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:51.425 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:51.450 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:51.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:51.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:51.500 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T10:15:51.551 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:51.611 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:51.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:51.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:51.713 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:51.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:51.756 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:51.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:51.835 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:51 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 7167/189852 objects degraded (3.775%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:15:51.835 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:51 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1426870416' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:15:51.836 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:51 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:15:51.836 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:51 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 7167/189852 objects degraded (3.775%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:15:51.836 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:51 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1426870416' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:15:51.836 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:51 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:15:51.857 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:51.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:51 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 7167/189852 objects degraded (3.775%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:15:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:51 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1426870416' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:15:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:51 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:15:51.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:51.921 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:51.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:51.983 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:15:52.022 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:52.044 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:52.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:52.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:52.146 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:52.170 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:52.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:52.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:52.272 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:52.294 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:52.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:52.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:52.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T10:15:52.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:52.457 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:52.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:52.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:52.519 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:15:52.558 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:52.582 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:52.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:52.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:52.684 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:52.709 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:52.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:52.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:52.811 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:52.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:52 smithi098 ceph-mon[100914]: pgmap v741: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 736 op/s; 7167/187980 objects degraded (3.813%); 4488/187980 objects misplaced (2.387%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:15:52.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:52 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:15:52.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:52 smithi098 ceph-mon[100914]: osdmap e244: 8 total, 8 up, 7 in 2024-09-21T10:15:52.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:52 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1999184814' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:15:52.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:52 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:15:52.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:52.852 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:52 smithi098 ceph-mon[97580]: pgmap v741: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 736 op/s; 7167/187980 objects degraded (3.813%); 4488/187980 objects misplaced (2.387%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:15:52.852 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:52 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:15:52.853 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:52 smithi098 ceph-mon[97580]: osdmap e244: 8 total, 8 up, 7 in 2024-09-21T10:15:52.853 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:52 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1999184814' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:15:52.853 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:52 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:15:52.853 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:52.860 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:52.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:52.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:52.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:52 smithi196 ceph-mon[84784]: pgmap v741: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 736 op/s; 7167/187980 objects degraded (3.813%); 4488/187980 objects misplaced (2.387%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:15:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:52 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:15:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:52 smithi196 ceph-mon[84784]: osdmap e244: 8 total, 8 up, 7 in 2024-09-21T10:15:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:52 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1999184814' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:15:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:52 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:15:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:52.962 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:52.986 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:52.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:52.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:53.088 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:53.110 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:53.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:53.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:53.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:53.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:53.258 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:53.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:53.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:53.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:53.402 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:53.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:53.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:53.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:53.549 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:53.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:53.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:53.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:53.675 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:53.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:53.776 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:53.802 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:53.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:53.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:53.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:53 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:15:53.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:53 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:15:53.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:53 smithi098 ceph-mon[97580]: osdmap e245: 8 total, 8 up, 7 in 2024-09-21T10:15:53.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:53 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:15:53.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:53 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:15:53.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:53 smithi098 ceph-mon[100914]: osdmap e245: 8 total, 8 up, 7 in 2024-09-21T10:15:53.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:53 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:15:53.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:53 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:15:53.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:53 smithi196 ceph-mon[84784]: osdmap e245: 8 total, 8 up, 7 in 2024-09-21T10:15:53.904 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:53.928 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:53.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:53.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:54.029 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:54.053 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:54.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:54.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:54.155 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:54.192 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:54.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:54.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:54.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:54.317 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:54.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:54.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:54.419 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:54.441 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:54.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:54.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:54.543 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:54.582 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:54.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:54.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:54.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:54.704 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:54.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:54.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:54.805 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:54.833 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:54.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:54.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:54.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:54 smithi098 ceph-mon[97580]: pgmap v744: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 818 op/s; 7167/180939 objects degraded (3.961%); 4100/180939 objects misplaced (2.266%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:15:54.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:54 smithi098 ceph-mon[100914]: pgmap v744: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 818 op/s; 7167/180939 objects degraded (3.961%); 4100/180939 objects misplaced (2.266%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:15:54.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:54 smithi196 ceph-mon[84784]: pgmap v744: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 818 op/s; 7167/180939 objects degraded (3.961%); 4100/180939 objects misplaced (2.266%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:15:54.935 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:54.966 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:54.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:54.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:55.068 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:55.091 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:55.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:55.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:55.193 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:55.214 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:55.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:55.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:55.315 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:55.342 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:55.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:55.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:55.444 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:55.472 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:55.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:55.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:55.574 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:55.599 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:55.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:55.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:55.700 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:55.724 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:55.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:55.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:55.826 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:55.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:55.857 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:55.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:55.958 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:55.985 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:55.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:55.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:56.087 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:56.112 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:56.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:56.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:56.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:56.249 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:56.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:56.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:56.350 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:56.374 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:56.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:56.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:56.476 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:56.500 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:56.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:56.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:56.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:56.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:56 smithi196 ceph-mon[84784]: pgmap v745: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 807 op/s; 7167/175593 objects degraded (4.082%); 4100/175593 objects misplaced (2.335%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:15:56.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:56 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 7167/175593 objects degraded (4.082%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:15:56.640 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:56.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:56.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:56.742 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:56.766 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:56.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:56.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:56.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:56 smithi098 ceph-mon[97580]: pgmap v745: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 807 op/s; 7167/175593 objects degraded (4.082%); 4100/175593 objects misplaced (2.335%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:15:56.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:56 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 7167/175593 objects degraded (4.082%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:15:56.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:56 smithi098 ceph-mon[100914]: pgmap v745: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 807 op/s; 7167/175593 objects degraded (4.082%); 4100/175593 objects misplaced (2.335%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:15:56.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:56 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 7167/175593 objects degraded (4.082%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:15:56.868 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:56.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:56.911 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:56.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:57.012 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:57.046 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:57.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:57.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:57.148 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:57.172 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:57.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:57.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:57.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:57.299 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:57.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:57.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:57.401 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:57.425 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:57.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:57.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:57.527 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:57.551 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:57.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:57.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:57.653 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:57.698 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:57.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:57.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:57.800 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:57.824 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:57.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:57.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:57.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:57.950 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:57.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:57.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:58.052 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:58.076 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:58.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:58.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:58.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:58.201 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:58.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:58.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:58.303 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:58.349 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:58.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:58.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:58.451 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:58.477 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:58.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:58.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:58.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:58.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:58.622 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:58.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:58.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:58.760 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:58.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:58.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:58.844 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:15:58 smithi098 ceph-mon[97580]: pgmap v746: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 815 op/s; 7167/170286 objects degraded (4.209%); 3660/170286 objects misplaced (2.149%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:15:58.844 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:15:58 smithi098 ceph-mon[100914]: pgmap v746: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 815 op/s; 7167/170286 objects degraded (4.209%); 3660/170286 objects misplaced (2.149%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:15:58.862 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:15:58 smithi196 ceph-mon[84784]: pgmap v746: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 815 op/s; 7167/170286 objects degraded (4.209%); 3660/170286 objects misplaced (2.149%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:15:58.901 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:58.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:58.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:59.003 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:59.028 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:59.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:59.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:59.129 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:59.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:15:59.154 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:59.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:59.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:59.256 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:59.281 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:59.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:59.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:59.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:59.437 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:15:59.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:59.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:59.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:59.564 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:59.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:59.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:59.666 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:59.688 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:59.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:59.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:59.790 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:59.813 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:59.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:59.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:59.870 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:15:59.870 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:15:59.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:59.938 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:15:59.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:59.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:00.040 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:00.064 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:00.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:00.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:00.166 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:00.193 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:00.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:00.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:00.295 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:00.321 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:00.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:00.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:00.424 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:00.450 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:00.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:00.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:00.552 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:00.561 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:16:00.570 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:16:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:16:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:16:00.576 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:00.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:00.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:00.678 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:00.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:00.749 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:00.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:00.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:16:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:16:00] "GET /metrics HTTP/1.1" 200 36568 "" "Prometheus/2.51.0" 2024-09-21T10:16:00.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:00 smithi098 ceph-mon[97580]: pgmap v747: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.4 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 950 op/s; 7167/165159 objects degraded (4.339%); 3660/165159 objects misplaced (2.216%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:16:00.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:00 smithi098 ceph-mon[100914]: pgmap v747: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.4 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 950 op/s; 7167/165159 objects degraded (4.339%); 3660/165159 objects misplaced (2.216%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:16:00.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:00 smithi196 ceph-mon[84784]: pgmap v747: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.4 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 950 op/s; 7167/165159 objects degraded (4.339%); 3660/165159 objects misplaced (2.216%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:16:00.894 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:00.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:00.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:00.995 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:01.020 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:01.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:01.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:01.121 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:01.147 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:01.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:01.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:01.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:01.306 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:01.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:01.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:01.408 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:01.435 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:01.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:01.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:01.537 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:01.559 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:01.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:01.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:01.661 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:01.687 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:01.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:01.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:01.789 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:01.813 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:01.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:01.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:01.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:01 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 7167/165159 objects degraded (4.339%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:01.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:01 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 7167/165159 objects degraded (4.339%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:01.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:01 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 7167/165159 objects degraded (4.339%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:01.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:01.939 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:01.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:01.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:02.041 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:02.066 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:02.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:02.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:02.168 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:02.190 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:02.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:02.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:02.256 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:16:02.256 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:16:02.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:02.299 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:16:02.299 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:16:02.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:02.337 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:02.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:02.345 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:16:02.345 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:16:02.379 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:16:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:16:02.408 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:16:02.408 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:16:02.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:02.443 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:16:02.443 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:16:02.462 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:02.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:02.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:02.473 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:16:02.474 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:16:02.504 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:16:02.504 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:16:02.564 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:02.588 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:02.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:02.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:02.690 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:02.734 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:02.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:02.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:02.836 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:02.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:02 smithi098 ceph-mon[97580]: pgmap v748: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 794 op/s; 7167/163095 objects degraded (4.394%); 3660/163095 objects misplaced (2.244%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:16:02.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:02 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3101019954' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:16:02.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:02 smithi098 ceph-mon[100914]: pgmap v748: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 794 op/s; 7167/163095 objects degraded (4.394%); 3660/163095 objects misplaced (2.244%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:16:02.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:02 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3101019954' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:16:02.857 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:02.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:02.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:02.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:02 smithi196 ceph-mon[84784]: pgmap v748: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 794 op/s; 7167/163095 objects degraded (4.394%); 3660/163095 objects misplaced (2.244%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:16:02.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:02 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3101019954' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:16:02.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:02.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:02.994 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:02.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:03.095 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:03.137 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:03.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:03.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:03.238 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:03.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:03.276 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:03.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:03.377 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:03.402 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:03.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:03.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:03.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:03.549 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:03.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:03.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:03.560 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:03.560 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:03.639 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T10:16:03.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:03.698 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:03.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:03.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:03.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:03.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:03.834 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:03.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:03.875 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:03 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2522904658' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:16:03.875 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:03 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:16:03.875 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:03 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2522904658' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:16:03.875 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:03 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:16:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:03 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2522904658' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:16:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:03 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:16:03.935 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:03.958 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:03.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:03.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:04.059 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:04.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:04.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:04.184 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:04.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:04.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:04.309 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:04.331 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:04.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:04.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:04.433 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:04.456 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:04.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:04.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:04.558 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:04.581 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:04.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:04.683 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:04.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:04.705 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:04.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:04.805 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:04.827 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:04.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:04.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:04.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:04 smithi098 ceph-mon[97580]: pgmap v749: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 821 op/s; 7167/155937 objects degraded (4.596%); 3207/155937 objects misplaced (2.057%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:16:04.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:04 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:16:04.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:04 smithi098 ceph-mon[97580]: osdmap e246: 8 total, 8 up, 7 in 2024-09-21T10:16:04.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:04 smithi098 ceph-mon[100914]: pgmap v749: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 821 op/s; 7167/155937 objects degraded (4.596%); 3207/155937 objects misplaced (2.057%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:16:04.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:04 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:16:04.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:04 smithi098 ceph-mon[100914]: osdmap e246: 8 total, 8 up, 7 in 2024-09-21T10:16:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:04 smithi196 ceph-mon[84784]: pgmap v749: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 821 op/s; 7167/155937 objects degraded (4.596%); 3207/155937 objects misplaced (2.057%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:16:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:04 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:16:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:04 smithi196 ceph-mon[84784]: osdmap e246: 8 total, 8 up, 7 in 2024-09-21T10:16:04.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:04.950 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:04.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:04.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:05.052 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:05.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:05.074 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:05.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:05.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:05.197 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:05.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:05.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:05.299 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:05.321 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:05.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:05.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:05.447 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:05.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:05.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:05.550 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:05.572 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:05.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:05.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:05.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:05.697 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:05.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:05.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:05.786 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:05 smithi098 ceph-mon[97580]: osdmap e247: 8 total, 8 up, 7 in 2024-09-21T10:16:05.787 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:05 smithi098 ceph-mon[100914]: osdmap e247: 8 total, 8 up, 7 in 2024-09-21T10:16:05.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:05.820 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:05.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:05.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:05.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:05 smithi196 ceph-mon[84784]: osdmap e247: 8 total, 8 up, 7 in 2024-09-21T10:16:05.922 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:05.943 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:05.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:05.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:06.045 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:06.068 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:06.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:06.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:06.170 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:06.194 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:06.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:06.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:06.296 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:06.321 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:06.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:06.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:06.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:06.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:06.461 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:06.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:06.562 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:06.586 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:06.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:06.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:06.687 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:06.712 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:06.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:06.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:06.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:06.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:06.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:06 smithi196 ceph-mon[84784]: pgmap v752: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 740 op/s; 7167/152511 objects degraded (4.699%); 3207/152511 objects misplaced (2.103%); 1.2 MiB/s, 18 objects/s recovering 2024-09-21T10:16:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:06 smithi196 ceph-mon[84784]: 7.1 starting backfill to osd.3 from (0'0,0'0] MAX to 245'7229 2024-09-21T10:16:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:06 smithi196 ceph-mon[84784]: osdmap e248: 8 total, 8 up, 7 in 2024-09-21T10:16:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:06 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 7167/152511 objects degraded (4.699%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:06.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:06.974 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:06.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:06.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:07.076 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:07.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:06 smithi098 ceph-mon[97580]: pgmap v752: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 740 op/s; 7167/152511 objects degraded (4.699%); 3207/152511 objects misplaced (2.103%); 1.2 MiB/s, 18 objects/s recovering 2024-09-21T10:16:07.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:06 smithi098 ceph-mon[97580]: 7.1 starting backfill to osd.3 from (0'0,0'0] MAX to 245'7229 2024-09-21T10:16:07.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:06 smithi098 ceph-mon[97580]: osdmap e248: 8 total, 8 up, 7 in 2024-09-21T10:16:07.094 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:06 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 7167/152511 objects degraded (4.699%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:07.094 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:06 smithi098 ceph-mon[100914]: pgmap v752: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 740 op/s; 7167/152511 objects degraded (4.699%); 3207/152511 objects misplaced (2.103%); 1.2 MiB/s, 18 objects/s recovering 2024-09-21T10:16:07.094 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:06 smithi098 ceph-mon[100914]: 7.1 starting backfill to osd.3 from (0'0,0'0] MAX to 245'7229 2024-09-21T10:16:07.094 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:06 smithi098 ceph-mon[100914]: osdmap e248: 8 total, 8 up, 7 in 2024-09-21T10:16:07.094 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:06 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 7167/152511 objects degraded (4.699%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:07.115 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:07.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:07.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:07.217 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:07.241 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:07.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:07.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:07.343 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:07.382 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:07.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:07.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:07.459 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:16:07.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:07.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:07.553 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:07.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:07.654 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:07.676 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:07.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:07.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:07.778 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:07.800 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:07.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:07.842 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:07.843 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:07.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:07.949 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:07.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:07.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:08.051 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:08.076 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:08.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:08.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:08.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:08.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:08.206 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:08.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:08.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:08.308 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:08.331 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:08.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:08.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:08.433 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:08.455 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:08.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:08.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:08.557 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:08.589 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:08.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:08.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:08.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:08.839 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T10:16:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:08.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:08.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:08.896 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:08 smithi098 ceph-mon[97580]: pgmap v754: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 remapped+peering, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 828 op/s; 10818/148182 objects degraded (7.300%); 3440/148182 objects misplaced (2.321%); 6.1 MiB/s, 97 objects/s recovering 2024-09-21T10:16:08.901 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:08 smithi098 ceph-mon[100914]: pgmap v754: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 remapped+peering, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 828 op/s; 10818/148182 objects degraded (7.300%); 3440/148182 objects misplaced (2.321%); 6.1 MiB/s, 97 objects/s recovering 2024-09-21T10:16:08.990 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:09.015 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:09.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:09.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:09.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:08 smithi196 ceph-mon[84784]: pgmap v754: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 remapped+peering, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 828 op/s; 10818/148182 objects degraded (7.300%); 3440/148182 objects misplaced (2.321%); 6.1 MiB/s, 97 objects/s recovering 2024-09-21T10:16:09.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:09.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:09.192 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:09.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:09.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:09.313 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:09.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:09.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:09.415 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:09.440 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:09.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:09.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:09.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:09.580 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:09.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:09.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:09.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:09.707 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:09.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:09.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:09.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:09.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:09.850 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:09.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:09.951 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:09.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:09.997 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:09.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:10.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:10.125 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:10.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:10.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:10.227 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:10.254 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:10.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:10.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:10.355 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:10.381 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:10.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:10.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:10.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:16:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:16:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:16:10.483 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:10.506 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:10.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:10.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:10.608 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:10.652 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:10.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:10.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:10.753 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:10.789 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:10.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:10.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:10.891 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:10.911 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:10 smithi098 ceph-mon[97580]: pgmap v755: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 remapped+peering, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 577 op/s; 10818/145536 objects degraded (7.433%); 3440/145536 objects misplaced (2.364%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:16:10.911 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:16:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:16:10] "GET /metrics HTTP/1.1" 200 36559 "" "Prometheus/2.51.0" 2024-09-21T10:16:10.912 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:10 smithi098 ceph-mon[100914]: pgmap v755: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 remapped+peering, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 577 op/s; 10818/145536 objects degraded (7.433%); 3440/145536 objects misplaced (2.364%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:16:10.917 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:10.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:10.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:10.930 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:10 smithi196 ceph-mon[84784]: pgmap v755: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 remapped+peering, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 577 op/s; 10818/145536 objects degraded (7.433%); 3440/145536 objects misplaced (2.364%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:16:11.018 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:11.041 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:11.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:11.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:11.143 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:11.187 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:11.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:11.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:11.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:11.326 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:11.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:11.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:11.427 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:11.450 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:11.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:11.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:11.552 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:11.577 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:11.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:11.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:11.667 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:16:11.678 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:11.703 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:11.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:11.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:11.805 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:11.828 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:11.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:11.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:11.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:11 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 10818/145536 objects degraded (7.433%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:11.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:11 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3298527516' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:16:11.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:11 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:16:11.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:11 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 10818/145536 objects degraded (7.433%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:11.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:11 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3298527516' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:16:11.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:11 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:16:11.930 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:11.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:11.957 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:11.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:12.058 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:12.102 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:12.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:12.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:12.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:11 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 10818/145536 objects degraded (7.433%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:12.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:11 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3298527516' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:16:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:11 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:16:12.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:12.225 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:12.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:12.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:12.327 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:12.348 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:12.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:12.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:12.450 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:12.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:12.472 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:12.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:12.572 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:12.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:12.594 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:12.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:12.695 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:12.719 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:12.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:12.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:12.821 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:12.846 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:12.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:12.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:12.948 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:12.987 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:12.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:12.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:13.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:13.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:12 smithi098 ceph-mon[97580]: pgmap v756: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 413 op/s; 14388/142977 objects degraded (10.063%); 3440/142977 objects misplaced (2.406%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:16:13.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:12 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:16:13.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:12 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:16:13.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:12 smithi098 ceph-mon[97580]: osdmap e249: 8 total, 8 up, 7 in 2024-09-21T10:16:13.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:12 smithi098 ceph-mon[100914]: pgmap v756: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 413 op/s; 14388/142977 objects degraded (10.063%); 3440/142977 objects misplaced (2.406%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:16:13.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:12 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:16:13.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:12 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:16:13.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:12 smithi098 ceph-mon[100914]: osdmap e249: 8 total, 8 up, 7 in 2024-09-21T10:16:13.127 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:13.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:13.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:13.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:12 smithi196 ceph-mon[84784]: pgmap v756: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 413 op/s; 14388/142977 objects degraded (10.063%); 3440/142977 objects misplaced (2.406%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:16:13.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:12 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:16:13.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:12 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:16:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:12 smithi196 ceph-mon[84784]: osdmap e249: 8 total, 8 up, 7 in 2024-09-21T10:16:13.229 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:13.253 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:13.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:13.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:13.355 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:13.394 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:13.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:13.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:13.496 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:13.518 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:13.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:13.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:13.620 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:13.645 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:13.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:13.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:13.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:13.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:13.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:13.889 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:13.923 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:13.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:13.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:14.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:14.062 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:14.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:14.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:14.164 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:14.211 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:14.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:14.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:14.312 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:14.336 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:14.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:14.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:14.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:14.463 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:14.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:14.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:14.564 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:14.590 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:14.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:14.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:14.692 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:14.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:14.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:14.836 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:14.860 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:14.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:14.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:14.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:15.002 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:15.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:15.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:15.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:14 smithi098 ceph-mon[97580]: pgmap v758: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 618 op/s; 14388/137667 objects degraded (10.451%); 2774/137667 objects misplaced (2.015%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T10:16:15.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:14 smithi098 ceph-mon[100914]: pgmap v758: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 618 op/s; 14388/137667 objects degraded (10.451%); 2774/137667 objects misplaced (2.015%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T10:16:15.104 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:15.129 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:15.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:15.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:15.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:14 smithi196 ceph-mon[84784]: pgmap v758: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 618 op/s; 14388/137667 objects degraded (10.451%); 2774/137667 objects misplaced (2.015%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T10:16:15.231 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:15.265 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:15.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:15.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:15.367 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:15.393 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:15.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:15.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:15.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:15.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:15.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:15.636 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:15.699 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:15.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:15.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:15.801 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:15.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:15.849 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:15.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:15.950 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:15.990 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:15.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:15.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:16.078 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:16.092 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:16.095 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:16.117 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:16.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:16.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:16.219 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:16.271 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:16.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:16.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:16.373 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:16.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:16.411 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:16.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:16.512 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:16.547 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:16.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:16.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:16.649 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:16.677 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:16.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:16.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:16.779 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:16.802 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:16.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:16.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:16 smithi098 ceph-mon[97580]: pgmap v759: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 553 op/s; 14388/134529 objects degraded (10.695%); 2774/134529 objects misplaced (2.062%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T10:16:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:16 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 14388/134529 objects degraded (10.695%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:16.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:16 smithi098 ceph-mon[100914]: pgmap v759: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 553 op/s; 14388/134529 objects degraded (10.695%); 2774/134529 objects misplaced (2.062%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T10:16:16.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:16 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 14388/134529 objects degraded (10.695%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:16.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:16.946 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:16.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:16.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:17.048 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:17.087 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:17.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:17.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:17.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:16 smithi196 ceph-mon[84784]: pgmap v759: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 553 op/s; 14388/134529 objects degraded (10.695%); 2774/134529 objects misplaced (2.062%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T10:16:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:16 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 14388/134529 objects degraded (10.695%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:17.189 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:17.233 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:17.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:17.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:17.334 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:17.372 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:17.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:17.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:17.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:17.497 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:17.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:17.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:17.598 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:17.622 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:17.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:17.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:17.669 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:16:17.724 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:17.747 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:17.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:17.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:17.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:17.897 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:17.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:17.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:17.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:18.047 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:18.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:18.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:18.148 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:18.174 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:18.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:18.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:18.276 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:18.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:18.342 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:18.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:18.443 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:18.466 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:18.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:18.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:18.568 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:18.620 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:18.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:18.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:18.721 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:18.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:18.769 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:18.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:18 smithi098 ceph-mon[97580]: pgmap v760: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.6 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 693 op/s; 14388/127353 objects degraded (11.298%); 2116/127353 objects misplaced (1.662%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:16:18.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:18 smithi098 ceph-mon[100914]: pgmap v760: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.6 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 693 op/s; 14388/127353 objects degraded (11.298%); 2116/127353 objects misplaced (1.662%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:16:18.870 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:18.894 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:18.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:18.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:18.996 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:19.019 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:19.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:19.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:19.121 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:19.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:18 smithi196 ceph-mon[84784]: pgmap v760: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.6 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 693 op/s; 14388/127353 objects degraded (11.298%); 2116/127353 objects misplaced (1.662%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:16:19.145 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:19.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:19.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:19.180 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:16:19.180 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:16:19.180 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-21T10:16:19.180 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-21T10:16:19.180 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:16:19.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:19.290 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:19.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:19.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:19.392 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:19.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:19.431 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:19.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:19.532 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:19.556 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:19.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:19.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:19.658 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:19.683 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:19.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:19.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:19.784 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:19.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:19.839 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:19.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:19.940 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:19.963 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:19.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:19.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:20.065 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:20.088 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:20.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:20.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:20.190 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:20.214 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:20.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:20.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:20.315 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:20.342 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:20.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:20.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:20.444 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:20.468 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:20.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:20.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:20.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:20.595 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:20.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:20.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:20.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:16:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:16:20.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:16:20.653 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:16:20.697 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:20.728 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:16:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:16:20] "GET /metrics HTTP/1.1" 200 36559 "" "Prometheus/2.51.0" 2024-09-21T10:16:20.728 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:20 smithi098 ceph-mon[100914]: pgmap v761: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 764 op/s; 14388/122586 objects degraded (11.737%); 2116/122586 objects misplaced (1.726%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:16:20.728 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:20 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1391677793' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:16:20.729 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:20 smithi098 ceph-mon[97580]: pgmap v761: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 764 op/s; 14388/122586 objects degraded (11.737%); 2116/122586 objects misplaced (1.726%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:16:20.729 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:20 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1391677793' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:16:20.749 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:20.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:20.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:20.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:20.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:20.888 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:20.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:20.989 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:21.014 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:20 smithi196 ceph-mon[84784]: pgmap v761: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 764 op/s; 14388/122586 objects degraded (11.737%); 2116/122586 objects misplaced (1.726%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:16:21.015 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:20 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1391677793' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:16:21.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:21.015 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:21.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:21.081 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:16:21.081 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:16:21.081 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:16:21.081 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:16:21.081 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:16:21.081 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:16:21.081 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:16:21.081 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:16:21.082 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:16:21.082 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:16:21.082 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:16:21.082 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:16:21.082 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:16:21.082 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:16:21.083 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:16:21.083 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:16:21.116 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:21.155 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:21.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:21.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:21.259 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:21.284 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:21.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:21.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:21.386 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:21.411 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:21.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:21.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:21.513 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:21.553 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:21.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:21.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:21.654 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:21.680 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:21.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:21.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:21.694 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:21 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 14388/122586 objects degraded (11.737%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:21.782 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:21.824 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:21.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:21.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:21.925 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:21.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:21.968 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:21.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:22.062 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:21 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 14388/122586 objects degraded (11.737%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:22.070 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:22.094 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:22.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:22.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:22.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:21 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 14388/122586 objects degraded (11.737%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:22.196 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:22.199 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:22.254 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:22.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:22.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:22.356 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:22.381 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:22.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:22.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:22.482 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:22.504 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:22.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:22.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:22.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:22.653 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:22.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:22.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:22.754 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:22.791 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:22.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:22.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:22.893 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:22.917 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:22.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:22.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:23.019 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:23.040 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:23.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:23.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:22 smithi098 ceph-mon[97580]: pgmap v762: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 775 op/s; 14388/119682 objects degraded (12.022%); 2116/119682 objects misplaced (1.768%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:16:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:16:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:22 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1494519206' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:16:23.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:22 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:16:23.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:22 smithi098 ceph-mon[100914]: pgmap v762: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 775 op/s; 14388/119682 objects degraded (12.022%); 2116/119682 objects misplaced (1.768%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:16:23.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:16:23.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:23.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:22 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1494519206' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:16:23.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:22 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:16:23.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:22 smithi196 ceph-mon[84784]: pgmap v762: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 775 op/s; 14388/119682 objects degraded (12.022%); 2116/119682 objects misplaced (1.768%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:16:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:16:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:22 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1494519206' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:16:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:22 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:16:23.142 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:23.167 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:23.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:23.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:23.247 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T10:16:23.268 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:23.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:23.307 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:23.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:23.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:23.445 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:23.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:23.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:23.546 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:23.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:23.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:23.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:23.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:23.712 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:23.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:23.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:23.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:23.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:23.849 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:23.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:23.910 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:16:23.950 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:24.015 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:24.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:24.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:24.117 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:24.140 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:24.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:24.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:24.242 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:24.265 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:24.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:24.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:24.368 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:24.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:24.405 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:24.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:24.505 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:24.528 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:24.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:24.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:24.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:24 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:16:24.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:24 smithi098 ceph-mon[97580]: osdmap e250: 8 total, 8 up, 7 in 2024-09-21T10:16:24.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:24 smithi098 ceph-mon[97580]: pgmap v764: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 784 op/s; 14388/114117 objects degraded (12.608%); 1360/114117 objects misplaced (1.192%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:16:24.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:24 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:16:24.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:24 smithi098 ceph-mon[100914]: osdmap e250: 8 total, 8 up, 7 in 2024-09-21T10:16:24.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:24 smithi098 ceph-mon[100914]: pgmap v764: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 784 op/s; 14388/114117 objects degraded (12.608%); 1360/114117 objects misplaced (1.192%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:16:24.630 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:24.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:24 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:16:24.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:24 smithi196 ceph-mon[84784]: osdmap e250: 8 total, 8 up, 7 in 2024-09-21T10:16:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:24 smithi196 ceph-mon[84784]: pgmap v764: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 784 op/s; 14388/114117 objects degraded (12.608%); 1360/114117 objects misplaced (1.192%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:16:24.665 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:24.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:24.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:24.768 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:24.793 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:24.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:24.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:24.895 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:24.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:24.923 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:24.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:25.024 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:25.045 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:25.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:25.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:25.047 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T10:16:25.147 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:25.168 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:25.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:25.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:25.270 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:25.294 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:25.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:25.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:25.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:25 smithi098 ceph-mon[97580]: osdmap e251: 8 total, 8 up, 7 in 2024-09-21T10:16:25.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:25 smithi098 ceph-mon[100914]: osdmap e251: 8 total, 8 up, 7 in 2024-09-21T10:16:25.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:25.457 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:25.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:25.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:25.559 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:25.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:25.608 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:25.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:25.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:25 smithi196 ceph-mon[84784]: osdmap e251: 8 total, 8 up, 7 in 2024-09-21T10:16:25.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:25.758 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:25.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:25.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:25.860 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:25.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:25.906 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:25.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:26.007 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:26.036 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:26.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:26.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:26.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:26.191 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:26.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:26.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:26.290 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:16:26.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:26.363 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:26.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:26.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:26.465 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:26.493 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:26.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:26.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:26.594 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:26.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:26 smithi098 ceph-mon[97580]: osdmap e252: 8 total, 8 up, 7 in 2024-09-21T10:16:26.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:26 smithi098 ceph-mon[97580]: pgmap v767: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.3 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 660 op/s; 14388/110682 objects degraded (12.999%); 1360/110682 objects misplaced (1.229%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:16:26.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:26 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 14388/114117 objects degraded (12.608%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:26.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:26 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:16:26.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:26 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3822074813' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:16:26.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:26 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:16:26.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:26 smithi098 ceph-mon[100914]: osdmap e252: 8 total, 8 up, 7 in 2024-09-21T10:16:26.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:26 smithi098 ceph-mon[100914]: pgmap v767: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.3 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 660 op/s; 14388/110682 objects degraded (12.999%); 1360/110682 objects misplaced (1.229%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:16:26.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:26 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 14388/114117 objects degraded (12.608%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:26.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:26 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:16:26.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:26 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3822074813' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:16:26.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:26 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:16:26.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:26.638 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:26.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:26 smithi196 ceph-mon[84784]: osdmap e252: 8 total, 8 up, 7 in 2024-09-21T10:16:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:26 smithi196 ceph-mon[84784]: pgmap v767: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.3 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 660 op/s; 14388/110682 objects degraded (12.999%); 1360/110682 objects misplaced (1.229%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:16:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:26 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 14388/114117 objects degraded (12.608%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:26 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:16:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:26 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3822074813' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:16:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:26 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:16:26.740 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:26.768 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:26.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:26.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:26.870 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:26.921 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:26.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:26.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:27.023 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:27.060 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:27.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:27.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:27.162 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:27.203 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:27.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:27.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:27.305 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:27.329 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:27.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:27.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:27.430 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:27.456 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:27.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:27.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:27.511 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:27 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:16:27.511 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:27 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:16:27.511 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:27 smithi098 ceph-mon[100914]: osdmap e253: 8 total, 8 up, 7 in 2024-09-21T10:16:27.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:27 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:16:27.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:27 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:16:27.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:27 smithi098 ceph-mon[97580]: osdmap e253: 8 total, 8 up, 7 in 2024-09-21T10:16:27.535 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:16:27.535 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:16:27.557 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:27.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:27.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:27.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:27 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:16:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:27 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:16:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:27 smithi196 ceph-mon[84784]: osdmap e253: 8 total, 8 up, 7 in 2024-09-21T10:16:27.718 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:27.746 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:27.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:27.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:27.848 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:27.875 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:27.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:27.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:27.977 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:28.006 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:28.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:28.108 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:28.145 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:28.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:28.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:28.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:28.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:28.282 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:28.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:28.383 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:28.410 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:28.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:28.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:28.512 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:28.555 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:28.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:28.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:28.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:28 smithi098 ceph-mon[97580]: pgmap v769: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 513 op/s; 14388/107850 objects degraded (13.341%); 1360/107850 objects misplaced (1.261%) 2024-09-21T10:16:28.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:28 smithi098 ceph-mon[100914]: pgmap v769: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 513 op/s; 14388/107850 objects degraded (13.341%); 1360/107850 objects misplaced (1.261%) 2024-09-21T10:16:28.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:28 smithi196 ceph-mon[84784]: pgmap v769: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 513 op/s; 14388/107850 objects degraded (13.341%); 1360/107850 objects misplaced (1.261%) 2024-09-21T10:16:28.657 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:28.698 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:28.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:28.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:28.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:28.826 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:28.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:28.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:28.927 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:28.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:28.968 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:28.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:29.069 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:29.101 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:29.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:29.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:29.203 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:29.226 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:29.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:29.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:29.323 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:16:29.328 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:29.354 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:29.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:29.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:29.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:29.477 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:29.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:29.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:29.580 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:29.602 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:29.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:29.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:29.637 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:16:29.706 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:29.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:29.749 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:29.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:29.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:29.901 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:29.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:29.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:30.003 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:30.053 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:30.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:30.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:30.154 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:30.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:30.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:30.300 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:30.327 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:30.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:30.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:30.430 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:30.455 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:30.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:30.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:30.520 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:30 smithi196 ceph-mon[84784]: pgmap v770: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.0 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 826 op/s; 12992/99231 objects degraded (13.093%); 1854/99231 objects misplaced (1.868%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T10:16:30.521 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:16:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:16:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:16:30.556 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:30.585 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:30.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:30.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:30.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:30 smithi098 ceph-mon[97580]: pgmap v770: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.0 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 826 op/s; 12992/99231 objects degraded (13.093%); 1854/99231 objects misplaced (1.868%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T10:16:30.660 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:30 smithi098 ceph-mon[100914]: pgmap v770: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.0 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 826 op/s; 12992/99231 objects degraded (13.093%); 1854/99231 objects misplaced (1.868%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T10:16:30.687 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:30.711 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:30.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:30.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:30.723 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:30.813 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:30.837 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:30.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:30.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:30.914 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:16:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:16:30] "GET /metrics HTTP/1.1" 200 36560 "" "Prometheus/2.51.0" 2024-09-21T10:16:30.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:30.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:30.965 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:30.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:31.066 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:31.105 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:31.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:31.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:31.207 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:31.240 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:31.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:31.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:31.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:31.369 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:31.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:31.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:31.471 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:31.513 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:31.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:31.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:31.615 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:31.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:31 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 12992/99231 objects degraded (13.093%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:31 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:16:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:31 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:16:31.639 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:31.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:31.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:31.741 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:31.761 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:31.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:31.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:31.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:31 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 12992/99231 objects degraded (13.093%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:31.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:31 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:16:31.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:31 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:16:31.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:31 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 12992/99231 objects degraded (13.093%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:31.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:31 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:16:31.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:31 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:16:31.863 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:31.903 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:31.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:31.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:32.005 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:32.031 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:32.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:32.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:32.133 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:32.165 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:32.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:32.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:32.267 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:32.291 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:32.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:32.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:32.393 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:32.421 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:32.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:32.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:32.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:32.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:32.551 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:32.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:32.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:32.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:32.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:32.787 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:32.811 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:32.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:32.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:32.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:32.939 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:32.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:32.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:33.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:33.081 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:33.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:33.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:33.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:33.225 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:33.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:33.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:33.327 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:33.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:32 smithi098 ceph-mon[97580]: pgmap v771: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.0 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 678 op/s; 12992/96336 objects degraded (13.486%); 1854/96336 objects misplaced (1.925%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:16:33.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:32 smithi098 ceph-mon[100914]: pgmap v771: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.0 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 678 op/s; 12992/96336 objects degraded (13.486%); 1854/96336 objects misplaced (1.925%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:16:33.349 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:33.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:33.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:33.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:32 smithi196 ceph-mon[84784]: pgmap v771: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.0 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 678 op/s; 12992/96336 objects degraded (13.486%); 1854/96336 objects misplaced (1.925%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:16:33.451 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:33.477 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:33.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:33.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:33.579 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:33.600 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:33.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:33.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:33.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:33.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:33.731 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:33.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:33.832 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:33.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:33.875 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:33.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:33.976 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:34.002 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:34.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:34.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:34.104 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:34.130 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:34.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:34.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:34.232 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:34.253 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:34.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:34.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:34.355 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:34.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:34.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:34.497 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:34.524 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:34.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:34.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:34.626 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:34.651 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:34.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:34.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:34.753 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:34.780 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:34.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:34.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:34.882 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:34.914 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:34.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:34.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:35.015 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:35.040 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:35.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:35.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:35.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:35.175 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:35.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:35.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:35.187 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:34 smithi098 ceph-mon[97580]: pgmap v772: 99 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.8 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 838 op/s; 12992/90447 objects degraded (14.364%); 1714/90447 objects misplaced (1.895%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:16:35.187 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:34 smithi098 ceph-mon[100914]: pgmap v772: 99 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.8 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 838 op/s; 12992/90447 objects degraded (14.364%); 1714/90447 objects misplaced (1.895%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:16:35.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:35.319 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:35.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:35.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:35.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:34 smithi196 ceph-mon[84784]: pgmap v772: 99 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.8 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 838 op/s; 12992/90447 objects degraded (14.364%); 1714/90447 objects misplaced (1.895%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:16:35.421 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:35.447 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:35.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:35.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:35.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:35.586 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:35.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:35.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:35.688 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:35.709 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:35.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:35.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:35.811 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:35.836 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:35.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:35.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:35.938 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:35.964 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:35.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:35.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:36.066 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:36.088 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:36.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:36.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:36.190 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:36.226 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:36.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:36.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:36.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:35 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 12992/90447 objects degraded (14.364%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:36.260 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:35 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 12992/90447 objects degraded (14.364%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:36.328 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:36.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:36.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:35 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 12992/90447 objects degraded (14.364%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:36.465 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:36.489 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:36.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:36.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:36.518 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:36.518 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:36.591 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:36.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:36.644 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:36.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:36.745 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:36.770 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:36.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:36.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:36.872 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:36.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:36.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:37.021 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:37.046 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:37.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:37.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:37.147 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:37.170 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:37.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:37.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:37.271 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:37.310 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:36 smithi098 ceph-mon[100914]: pgmap v773: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.8 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 791 op/s; 12992/86922 objects degraded (14.947%); 1461/86922 objects misplaced (1.681%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T10:16:37.311 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:36 smithi098 ceph-mon[97580]: pgmap v773: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.8 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 791 op/s; 12992/86922 objects degraded (14.947%); 1461/86922 objects misplaced (1.681%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T10:16:37.322 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:37.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:37.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:37.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:36 smithi196 ceph-mon[84784]: pgmap v773: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.8 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 791 op/s; 12992/86922 objects degraded (14.947%); 1461/86922 objects misplaced (1.681%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T10:16:37.424 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:37.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:37.452 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:37.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:37.553 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:37.580 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:37.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:37.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:37.682 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:37.704 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:37.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:37.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:37.806 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:37.855 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:37.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:37.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:37.957 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:37.993 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:37.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:37.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:38.095 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:38.120 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:38.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:38.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:38.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:38.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:38.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:38.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:38.403 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:38.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:38.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:38.480 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:16:38.480 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:16:38.480 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:16:38.480 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:16:38.492 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:16:38.492 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:16:38.504 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:38.527 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:38.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:38.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:38.540 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:16:38.541 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:16:38.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:16:38.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:38.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:38 smithi098 ceph-mon[97580]: pgmap v774: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.7 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 729 op/s; 12992/83727 objects degraded (15.517%); 1461/83727 objects misplaced (1.745%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:16:38.541 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:16:38.542 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:38.542 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:38 smithi098 ceph-mon[100914]: pgmap v774: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.7 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 729 op/s; 12992/83727 objects degraded (15.517%); 1461/83727 objects misplaced (1.745%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:16:38.574 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:16:38.574 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:16:38.604 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:16:38.605 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:16:38.628 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:38.637 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:16:38.637 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:16:38.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:16:38.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:38.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:38 smithi196 ceph-mon[84784]: pgmap v774: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.7 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 729 op/s; 12992/83727 objects degraded (15.517%); 1461/83727 objects misplaced (1.745%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:16:38.667 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:38.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:38.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:38.670 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:16:38.670 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:16:38.770 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:38.795 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:38.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:38.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:38.897 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:38.922 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:38.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:38.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:39.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:39.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:39.073 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:39.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:39.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:39.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:39.200 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:39.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:39.301 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:39.324 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:39.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:39.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:39.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:39.489 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:39.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:39.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:39.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:39 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2321276581' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:16:39.590 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:39.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:39 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2321276581' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:16:39.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:39 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2321276581' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:16:39.620 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:39.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:39.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:39.722 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:39.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:39.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:39.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:39.851 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:39.878 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:39.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:39.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:39.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:40.009 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:40.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:40.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:40.110 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:40.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:40.139 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:40.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:40.240 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:40.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:40.316 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:40.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:40.417 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:40.443 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:40.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:40.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:40.533 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:40 smithi098 ceph-mon[100914]: pgmap v775: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.5 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 956 op/s; 12992/73392 objects degraded (17.702%); 1038/73392 objects misplaced (1.414%); 4.6 MiB/s, 72 objects/s recovering 2024-09-21T10:16:40.536 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:40 smithi098 ceph-mon[97580]: pgmap v775: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.5 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 956 op/s; 12992/73392 objects degraded (17.702%); 1038/73392 objects misplaced (1.414%); 4.6 MiB/s, 72 objects/s recovering 2024-09-21T10:16:40.545 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:40.598 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:40.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:40.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:40.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:40 smithi196 ceph-mon[84784]: pgmap v775: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.5 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 956 op/s; 12992/73392 objects degraded (17.702%); 1038/73392 objects misplaced (1.414%); 4.6 MiB/s, 72 objects/s recovering 2024-09-21T10:16:40.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:16:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:16:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:16:40.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:40.740 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:40.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:40.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:40.842 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:40.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:16:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:16:40] "GET /metrics HTTP/1.1" 200 36557 "" "Prometheus/2.51.0" 2024-09-21T10:16:40.871 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:40.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:40.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:40.972 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:40.997 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:40.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:40.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:41.099 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:41.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:41.150 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:41.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:41.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:41.275 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:41.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:41.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:41.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:41.416 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:41.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:41.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:41.519 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:41.544 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:41.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:41.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:41.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:41 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 12992/73392 objects degraded (17.702%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:41.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:41 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 12992/73392 objects degraded (17.702%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:41.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:41 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 12992/73392 objects degraded (17.702%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:41.646 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:41.673 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:41.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:41.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:41.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:16:41.775 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:41.798 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:41.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:41.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:41.899 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:41.923 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:41.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:41.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:42.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:42.068 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:42.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:42.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:42.170 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:42.212 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:42.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:42.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:16:42.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:42.366 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:42.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:42.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:42.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:42.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:42.512 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:42.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:42.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:42 smithi098 ceph-mon[97580]: pgmap v776: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.5 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 12992/71910 objects degraded (18.067%); 1038/71910 objects misplaced (1.443%); 2.9 MiB/s, 45 objects/s recovering 2024-09-21T10:16:42.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:42 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3600467631' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:16:42.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:42 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:16:42.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:42 smithi098 ceph-mon[100914]: pgmap v776: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.5 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 12992/71910 objects degraded (18.067%); 1038/71910 objects misplaced (1.443%); 2.9 MiB/s, 45 objects/s recovering 2024-09-21T10:16:42.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:42 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3600467631' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:16:42.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:42 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:16:42.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:42.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:42 smithi196 ceph-mon[84784]: pgmap v776: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.5 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 12992/71910 objects degraded (18.067%); 1038/71910 objects misplaced (1.443%); 2.9 MiB/s, 45 objects/s recovering 2024-09-21T10:16:42.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:42 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3600467631' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:16:42.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:42 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:16:42.653 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:42.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:42.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:42.755 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:42.796 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:42.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:42.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:42.898 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:42.921 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:42.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:42.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:43.023 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:43.049 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:43.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:43.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:43.151 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:43.176 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:43.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:43.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:43.277 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:43.301 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:43.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:43.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:43.403 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:43.427 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:43.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:43.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:43.529 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:43.564 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:43 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:16:43.564 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:43 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:16:43.564 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:43 smithi098 ceph-mon[100914]: osdmap e254: 8 total, 8 up, 7 in 2024-09-21T10:16:43.565 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:43 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:16:43.565 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:43 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:16:43.565 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:43 smithi098 ceph-mon[97580]: osdmap e254: 8 total, 8 up, 7 in 2024-09-21T10:16:43.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:43.591 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:43.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:43.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:43 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:16:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:43 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:16:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:43 smithi196 ceph-mon[84784]: osdmap e254: 8 total, 8 up, 7 in 2024-09-21T10:16:43.692 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:43.733 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:43.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:43.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:43.834 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:43.860 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:43.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:43.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:43.962 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:44.003 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:44.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:44.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:44.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:44.141 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:44.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:44.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:44.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:44.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:44.304 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:44.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:44.405 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:44.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:44.454 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:44.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:44.510 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:44 smithi098 ceph-mon[97580]: pgmap v778: 99 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.4 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 797 op/s; 12992/66510 objects degraded (19.534%); 939/66510 objects misplaced (1.412%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T10:16:44.510 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:44 smithi098 ceph-mon[100914]: pgmap v778: 99 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.4 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 797 op/s; 12992/66510 objects degraded (19.534%); 939/66510 objects misplaced (1.412%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T10:16:44.554 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:44.578 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:44.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:44.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:44.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:44 smithi196 ceph-mon[84784]: pgmap v778: 99 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.4 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 797 op/s; 12992/66510 objects degraded (19.534%); 939/66510 objects misplaced (1.412%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T10:16:44.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:44.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:44.742 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:44.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:44.842 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:44.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:44.892 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:44.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:44.966 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:16:44.966 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:16:44.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:16:44.966 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.55000.0:214350 7.d e254/202 7:b16a3f3c:::benchmark_data_smithi098_197834_object97833:head v 254'13235, mlcod=254'13233)", 2024-09-21T10:16:44.966 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:16:44.949500+0000", 2024-09-21T10:16:44.966 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.013876414, 2024-09-21T10:16:44.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013905501000000001, 2024-09-21T10:16:44.966 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:16:44.966 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:16:44.966 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:16:44.966 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:16:44.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:16:44.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:16:44.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:44.949500+0000", 2024-09-21T10:16:44.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:16:44.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:16:44.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:16:44.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:16:44.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:44.949500+0000", 2024-09-21T10:16:44.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:16:44.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:16:44.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:16:44.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:16:44.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:44.949501+0000", 2024-09-21T10:16:44.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4099999999999999e-07 2024-09-21T10:16:44.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:16:44.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:16:44.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:16:44.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:44.949506+0000", 2024-09-21T10:16:44.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4260000000000002e-06 2024-09-21T10:16:44.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:16:44.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:16:44.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:16:44.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:44.949515+0000", 2024-09-21T10:16:44.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5699999999999993e-06 2024-09-21T10:16:44.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:16:44.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:16:44.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:16:44.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:44.949520+0000", 2024-09-21T10:16:44.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.13e-06 2024-09-21T10:16:44.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:16:44.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:16:44.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:16:44.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:44.954655+0000", 2024-09-21T10:16:44.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051348890000000001 2024-09-21T10:16:44.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:16:44.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:16:44.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:16:44.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:44.954698+0000", 2024-09-21T10:16:44.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3192000000000003e-05 2024-09-21T10:16:44.969 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:16:44.969 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:16:44.970 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:16:44.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:16:44.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:16:44.970 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:214344 7.0 7:02b46456:::benchmark_data_smithi098_197834_object97827:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-09-21T10:16:44.970 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:16:44.943819+0000", 2024-09-21T10:16:44.970 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.019557475000000001, 2024-09-21T10:16:44.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019648675000000001, 2024-09-21T10:16:44.970 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:16:44.970 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:16:44.970 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:16:44.970 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:16:44.970 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:16:44.970 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:16:44.971 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 214344 2024-09-21T10:16:44.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:16:44.971 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:16:44.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:16:44.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:16:44.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:44.943819+0000", 2024-09-21T10:16:44.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:16:44.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:16:44.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:16:44.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:16:44.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:44.943819+0000", 2024-09-21T10:16:44.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:16:44.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:16:44.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:16:44.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:16:44.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:44.943821+0000", 2024-09-21T10:16:44.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9989999999999998e-06 2024-09-21T10:16:44.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:16:44.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:16:44.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:16:44.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:44.943828+0000", 2024-09-21T10:16:44.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9280000000000004e-06 2024-09-21T10:16:44.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:16:44.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:16:44.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:16:44.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:44.943843+0000", 2024-09-21T10:16:44.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4985e-05 2024-09-21T10:16:44.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:16:44.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:16:44.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:16:44.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:44.943853+0000", 2024-09-21T10:16:44.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7780000000000002e-06 2024-09-21T10:16:44.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:16:44.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:16:44.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:16:44.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:44.943921+0000", 2024-09-21T10:16:44.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7960999999999995e-05 2024-09-21T10:16:44.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:16:44.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:16:44.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:16:44.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:44.944051+0000", 2024-09-21T10:16:44.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013017700000000001 2024-09-21T10:16:44.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:16:44.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:16:44.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:16:44.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:44.944541+0000", 2024-09-21T10:16:44.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00049040599999999998 2024-09-21T10:16:44.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:16:44.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:16:44.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:16:44.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:44.954384+0000", 2024-09-21T10:16:44.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0098430189999999997 2024-09-21T10:16:44.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:16:44.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:16:44.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:16:44.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:44.954398+0000", 2024-09-21T10:16:44.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3467e-05 2024-09-21T10:16:44.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:16:44.975 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:16:44.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:16:44.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:16:44.975 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-21T10:16:44.975 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 2 2024-09-21T10:16:44.975 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:16:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:44.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:45.015 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:45.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:45.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:45.117 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:45.141 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:45.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:45.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:45.242 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:45.303 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:45.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:45.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:45.318 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T10:16:45.343 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:16:45.405 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:45.428 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:45.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:45.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:45.530 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:45.553 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:45.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:45.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:45.571 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:45 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2306705943' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:16:45.571 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:45 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:16:45.571 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:45 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2306705943' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:16:45.571 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:45 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:16:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:45 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2306705943' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:16:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:45 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:16:45.655 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:45.680 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:45.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:45.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:45.781 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:45.803 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:45.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:45.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:45.905 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:45.930 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:45.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:45.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:46.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:46.062 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:46.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:46.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:46.164 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:46.185 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:46.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:46.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:46.287 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:46.309 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:46.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:46.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:46.411 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:46.432 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:46.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:46.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:46.534 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:46.556 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:46.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:46.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:46.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:46 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:16:46.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:46 smithi098 ceph-mon[97580]: osdmap e255: 8 total, 8 up, 7 in 2024-09-21T10:16:46.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:46 smithi098 ceph-mon[97580]: pgmap v780: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.3 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 917 op/s; 12992/61674 objects degraded (21.066%); 663/61674 objects misplaced (1.075%); 4.5 MiB/s, 74 objects/s recovering 2024-09-21T10:16:46.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:46 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 12992/66510 objects degraded (19.534%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:46.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:46 smithi098 ceph-mon[97580]: osdmap e256: 8 total, 8 up, 7 in 2024-09-21T10:16:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:46 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:16:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:46 smithi098 ceph-mon[100914]: osdmap e255: 8 total, 8 up, 7 in 2024-09-21T10:16:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:46 smithi098 ceph-mon[100914]: pgmap v780: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.3 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 917 op/s; 12992/61674 objects degraded (21.066%); 663/61674 objects misplaced (1.075%); 4.5 MiB/s, 74 objects/s recovering 2024-09-21T10:16:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:46 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 12992/66510 objects degraded (19.534%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:46 smithi098 ceph-mon[100914]: osdmap e256: 8 total, 8 up, 7 in 2024-09-21T10:16:46.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:46 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:16:46.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:46 smithi196 ceph-mon[84784]: osdmap e255: 8 total, 8 up, 7 in 2024-09-21T10:16:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:46 smithi196 ceph-mon[84784]: pgmap v780: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.3 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 917 op/s; 12992/61674 objects degraded (21.066%); 663/61674 objects misplaced (1.075%); 4.5 MiB/s, 74 objects/s recovering 2024-09-21T10:16:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:46 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 12992/66510 objects degraded (19.534%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:46 smithi196 ceph-mon[84784]: osdmap e256: 8 total, 8 up, 7 in 2024-09-21T10:16:46.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:46.679 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:46.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:46.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:46.781 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:46.802 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:46.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:46.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:46.904 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:46.925 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:46.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:46.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:47.027 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:47.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:47.071 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:47.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:47.172 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:47.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:47.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:47.308 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:47.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T10:16:47.371 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:47.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:47.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:47.473 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:47.530 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:47.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:47.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:47.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:47 smithi098 ceph-mon[97580]: 7.1 continuing backfill to osd.3 from (206'1492,254'8225] MIN to 254'8225 2024-09-21T10:16:47.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:47 smithi098 ceph-mon[97580]: 7.1 starting backfill to osd.6 from (0'0,0'0] MAX to 254'8225 2024-09-21T10:16:47.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:47 smithi098 ceph-mon[97580]: osdmap e257: 8 total, 8 up, 7 in 2024-09-21T10:16:47.538 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:47 smithi098 ceph-mon[100914]: 7.1 continuing backfill to osd.3 from (206'1492,254'8225] MIN to 254'8225 2024-09-21T10:16:47.539 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:47 smithi098 ceph-mon[100914]: 7.1 starting backfill to osd.6 from (0'0,0'0] MAX to 254'8225 2024-09-21T10:16:47.539 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:47 smithi098 ceph-mon[100914]: osdmap e257: 8 total, 8 up, 7 in 2024-09-21T10:16:47.632 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:47.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:47 smithi196 ceph-mon[84784]: 7.1 continuing backfill to osd.3 from (206'1492,254'8225] MIN to 254'8225 2024-09-21T10:16:47.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:47 smithi196 ceph-mon[84784]: 7.1 starting backfill to osd.6 from (0'0,0'0] MAX to 254'8225 2024-09-21T10:16:47.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:47 smithi196 ceph-mon[84784]: osdmap e257: 8 total, 8 up, 7 in 2024-09-21T10:16:47.689 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:47.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:47.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:47.790 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:47.817 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:47.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:47.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:47.919 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:47.942 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:47.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:47.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:48.044 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:48.089 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:48.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:48.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:48.190 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:48.215 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:48.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:48.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:48.316 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:48.340 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:48.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:48.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:48.442 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:48.466 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:48.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:48.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:48.568 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:48.593 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:48.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:48.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:48 smithi098 ceph-mon[97580]: 7.12 starting backfill to osd.6 from (0'0,0'0] MAX to 254'8034 2024-09-21T10:16:48.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:48 smithi098 ceph-mon[97580]: pgmap v783: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.2 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 769 op/s; 12992/60309 objects degraded (21.542%); 663/60309 objects misplaced (1.099%); 3.4 MiB/s, 60 objects/s recovering 2024-09-21T10:16:48.600 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:48 smithi098 ceph-mon[100914]: 7.12 starting backfill to osd.6 from (0'0,0'0] MAX to 254'8034 2024-09-21T10:16:48.600 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:48 smithi098 ceph-mon[100914]: pgmap v783: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.2 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 769 op/s; 12992/60309 objects degraded (21.542%); 663/60309 objects misplaced (1.099%); 3.4 MiB/s, 60 objects/s recovering 2024-09-21T10:16:48.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:48 smithi196 ceph-mon[84784]: 7.12 starting backfill to osd.6 from (0'0,0'0] MAX to 254'8034 2024-09-21T10:16:48.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:48 smithi196 ceph-mon[84784]: pgmap v783: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.2 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 769 op/s; 12992/60309 objects degraded (21.542%); 663/60309 objects misplaced (1.099%); 3.4 MiB/s, 60 objects/s recovering 2024-09-21T10:16:48.695 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:48.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:48.755 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:48.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:48.856 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:48.880 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:48.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:48.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:48.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:49.015 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:49.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:49.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:49.117 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:49.143 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:49.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:49.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:49.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:49.283 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:49.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:49.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:49.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:49.404 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:16:49.423 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:49.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:49.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:49.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:49.549 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:49.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:49.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:49.651 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:49.673 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:49.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:49.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:49.774 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:49.820 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:49.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:49.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:49.922 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:49.946 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:49.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:49.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:50.009 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:16:50.010 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:16:50.045 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:16:50.045 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:16:50.048 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:50.081 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:16:50.081 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:16:50.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:50.092 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:50.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:50.127 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:16:50.127 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:16:50.157 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:16:50.157 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:16:50.187 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:16:50.188 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:16:50.193 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:50.216 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:50.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:50.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:50.229 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:16:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:16:50.258 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:16:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:16:50.317 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:50.342 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:50.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:50.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:50.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:50.486 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:50.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:50.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:50.511 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:50.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:50 smithi196 ceph-mon[84784]: pgmap v784: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.1 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 832 op/s; 12756/51501 objects degraded (24.768%); 1655/51501 objects misplaced (3.214%); 2.9 MiB/s, 136 objects/s recovering 2024-09-21T10:16:50.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:50 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2495702667' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:16:50.530 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:16:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:16:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:16:50.590 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:50.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:50 smithi098 ceph-mon[97580]: pgmap v784: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.1 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 832 op/s; 12756/51501 objects degraded (24.768%); 1655/51501 objects misplaced (3.214%); 2.9 MiB/s, 136 objects/s recovering 2024-09-21T10:16:50.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:50 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2495702667' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:16:50.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:50 smithi098 ceph-mon[100914]: pgmap v784: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.1 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 832 op/s; 12756/51501 objects degraded (24.768%); 1655/51501 objects misplaced (3.214%); 2.9 MiB/s, 136 objects/s recovering 2024-09-21T10:16:50.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:50 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2495702667' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:16:50.615 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:50.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:50.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:50.717 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:50.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:50.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:50.874 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:50.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:50.922 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:50.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:50.929 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:16:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:16:50] "GET /metrics HTTP/1.1" 200 36557 "" "Prometheus/2.51.0" 2024-09-21T10:16:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:51.023 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:51.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:51.079 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:51.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:51.180 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:51.225 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:51.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:51.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:51.327 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:51.355 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:51.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:51.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:51.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:51.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:51.500 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:51.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:51.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:51 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 12756/51501 objects degraded (24.768%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:51.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:51 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 12756/51501 objects degraded (24.768%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:51.601 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:51.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:51.622 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:51.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:51.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:51 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 12756/51501 objects degraded (24.768%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:51.724 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:51.745 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:51.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:51.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:51.847 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:51.886 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:51.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:51.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:51.988 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:52.036 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:52.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:52.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:52.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:52.163 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:52.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:52.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:52.312 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:52.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:52.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:52.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:52.473 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:52.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:52.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:52.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:52.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:52 smithi098 ceph-mon[97580]: pgmap v785: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.0 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 632 op/s; 12756/50256 objects degraded (25.382%); 1655/50256 objects misplaced (3.293%); 0 B/s, 86 objects/s recovering 2024-09-21T10:16:52.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:52.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:52 smithi098 ceph-mon[100914]: pgmap v785: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.0 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 632 op/s; 12756/50256 objects degraded (25.382%); 1655/50256 objects misplaced (3.293%); 0 B/s, 86 objects/s recovering 2024-09-21T10:16:52.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:52.621 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:52.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:52.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:52.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:52 smithi196 ceph-mon[84784]: pgmap v785: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.0 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 632 op/s; 12756/50256 objects degraded (25.382%); 1655/50256 objects misplaced (3.293%); 0 B/s, 86 objects/s recovering 2024-09-21T10:16:52.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:52.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:52.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:52.763 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:52.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:52.864 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:52.886 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:52.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:52.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:52.978 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:16:52.989 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:53.012 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:53.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:53.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:53.114 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:53.136 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:53.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:53.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:53.238 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:53.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:53.320 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:53.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:53.421 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:53.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:53.469 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:53.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:53.569 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:53.605 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:53.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:53.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:53.707 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:53.757 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:53.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:53.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:53.858 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:53.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:53.884 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:53.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:53.985 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:54.010 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:54.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:54.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:54.112 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:54.135 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:54.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:54.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:54.237 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:54.264 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:54.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:54.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:54.366 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:54.399 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:54.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:54.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:54.501 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:54.537 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:54.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:54.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:54.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:54 smithi196 ceph-mon[84784]: pgmap v786: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 935 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 704 op/s; 12756/44748 objects degraded (28.506%); 1263/44748 objects misplaced (2.822%); 0 B/s, 64 objects/s recovering 2024-09-21T10:16:54.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:54 smithi098 ceph-mon[97580]: pgmap v786: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 935 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 704 op/s; 12756/44748 objects degraded (28.506%); 1263/44748 objects misplaced (2.822%); 0 B/s, 64 objects/s recovering 2024-09-21T10:16:54.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:54 smithi098 ceph-mon[100914]: pgmap v786: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 935 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 704 op/s; 12756/44748 objects degraded (28.506%); 1263/44748 objects misplaced (2.822%); 0 B/s, 64 objects/s recovering 2024-09-21T10:16:54.663 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:54.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:54.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:54.765 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:54.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:54.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:54.911 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:54.938 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:54.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:54.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:55.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:55.080 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:55.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:55.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:55.182 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:55.216 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:55.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:55.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:55.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:55.357 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:55.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:55.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:55.458 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:55.483 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:55.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:55.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:55.584 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:55.607 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:55.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:55.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:55.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:55.762 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:55.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:55.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:55.864 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:55.889 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:55.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:55.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:55.991 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:56.017 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:56.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:56.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:56.118 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:56.158 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:56.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:56.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:56.260 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:56.284 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:56.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:56.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:56.386 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:56.391 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:56 smithi098 ceph-mon[97580]: pgmap v787: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 841 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 695 op/s; 12461/40224 objects degraded (30.979%); 1263/40224 objects misplaced (3.140%); 0 B/s, 84 objects/s recovering 2024-09-21T10:16:56.391 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:56 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 12461/40224 objects degraded (30.979%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:56.392 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:56 smithi098 ceph-mon[100914]: pgmap v787: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 841 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 695 op/s; 12461/40224 objects degraded (30.979%); 1263/40224 objects misplaced (3.140%); 0 B/s, 84 objects/s recovering 2024-09-21T10:16:56.393 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:56 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 12461/40224 objects degraded (30.979%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:56.411 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:56.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:56.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:56.512 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:56.538 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:56.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:56.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:56 smithi196 ceph-mon[84784]: pgmap v787: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 841 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 695 op/s; 12461/40224 objects degraded (30.979%); 1263/40224 objects misplaced (3.140%); 0 B/s, 84 objects/s recovering 2024-09-21T10:16:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:56 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 12461/40224 objects degraded (30.979%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:16:56.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:56.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:56.695 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:56.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:56.796 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:56.820 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:56.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:56.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:56.921 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:56.945 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:56.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:56.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:57.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:57.086 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:57.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:57.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:57.188 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:57.212 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:57.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:57.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:57.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:57.361 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:57.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:57.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:57.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:57.505 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:57.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:57.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:57.607 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:57.632 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:57.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:57.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:57.733 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:57.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:57.770 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:57.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:57.871 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:57.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:57.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:58.022 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:58.062 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:58.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:58.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:58.165 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:58.219 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:58.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:58.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:58.321 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:58.343 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:58.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:58.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:58.389 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T10:16:58.392 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:16:58.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:58.496 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:58.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:58.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:58.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:58.632 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:58.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:58.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:58.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:58 smithi196 ceph-mon[84784]: pgmap v788: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 807 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 681 op/s; 12461/38598 objects degraded (32.284%); 1263/38598 objects misplaced (3.272%); 0 B/s, 76 objects/s recovering 2024-09-21T10:16:58.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:58 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/451946287' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:16:58.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:58 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/751061257' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:16:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:58 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:16:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:58 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:16:58.734 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:58.760 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:58.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:58.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:58.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:58 smithi098 ceph-mon[97580]: pgmap v788: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 807 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 681 op/s; 12461/38598 objects degraded (32.284%); 1263/38598 objects misplaced (3.272%); 0 B/s, 76 objects/s recovering 2024-09-21T10:16:58.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:58 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/451946287' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:16:58.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:58 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/751061257' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:16:58.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:58 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:16:58.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:58 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:16:58.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:58 smithi098 ceph-mon[100914]: pgmap v788: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 807 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 681 op/s; 12461/38598 objects degraded (32.284%); 1263/38598 objects misplaced (3.272%); 0 B/s, 76 objects/s recovering 2024-09-21T10:16:58.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:58 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/451946287' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:16:58.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:58 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/751061257' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:16:58.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:58 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:16:58.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:58 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:16:58.861 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:58.919 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:58.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:58.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:59.021 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:59.045 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:59.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:59.147 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:59.187 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:59.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:59.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:59.288 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:59.314 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:59.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:59.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:59.416 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:59.441 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:59.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:59.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:59.543 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:59.568 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:59.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:59.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:59.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:59 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:16:59.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:59 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/451946287' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:16:59.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:59 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/751061257' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:16:59.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:59 smithi098 ceph-mon[97580]: osdmap e258: 8 total, 8 up, 7 in 2024-09-21T10:16:59.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:59 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:16:59.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:59 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:16:59.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:16:59 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:16:59.601 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:59 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:16:59.601 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:59 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/451946287' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:16:59.601 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:59 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/751061257' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:16:59.601 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:59 smithi098 ceph-mon[100914]: osdmap e258: 8 total, 8 up, 7 in 2024-09-21T10:16:59.601 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:59 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:16:59.602 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:59 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:16:59.602 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:16:59 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:16:59.602 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:59.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:59 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:16:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:59 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/451946287' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:16:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:59 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/751061257' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:16:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:59 smithi196 ceph-mon[84784]: osdmap e258: 8 total, 8 up, 7 in 2024-09-21T10:16:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:59 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:16:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:59 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:16:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:16:59 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:16:59.670 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:59.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:59.699 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:59.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:59.800 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:59.824 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:16:59.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:59.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:59.925 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:59.970 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:16:59.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:59.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:00.071 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:00.114 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:00.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:00.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:00.215 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:00.261 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:00.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:00.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:00.363 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:00.399 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:00.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:00.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:00.501 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:00.547 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:00.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:00.628 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:17:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:17:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:17:00.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:00.686 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:00.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:00.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:00.788 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:00.813 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:00.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:00.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:00.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:17:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:17:00] "GET /metrics HTTP/1.1" 200 36528 "" "Prometheus/2.51.0" 2024-09-21T10:17:00.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:00 smithi098 ceph-mon[97580]: pgmap v790: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 582 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 789 op/s; 12138/27816 objects degraded (43.637%); 859/27816 objects misplaced (3.088%); 0 B/s, 61 objects/s recovering 2024-09-21T10:17:00.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:00 smithi098 ceph-mon[100914]: pgmap v790: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 582 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 789 op/s; 12138/27816 objects degraded (43.637%); 859/27816 objects misplaced (3.088%); 0 B/s, 61 objects/s recovering 2024-09-21T10:17:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:00 smithi196 ceph-mon[84784]: pgmap v790: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 582 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 789 op/s; 12138/27816 objects degraded (43.637%); 859/27816 objects misplaced (3.088%); 0 B/s, 61 objects/s recovering 2024-09-21T10:17:00.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:00.945 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:00.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:00.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:01.047 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:01.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:01.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:01.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:01.173 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:01.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:01.208 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:01.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:01.309 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:01.330 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:01.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:01.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:01.432 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:01.453 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:01.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:01.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:01.555 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:01.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:01.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:01.709 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:01.731 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:01.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:01.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:01.833 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:01.855 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:01.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:01.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:01.957 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:01.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:01.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:02.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:01 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 12138/27816 objects degraded (43.637%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:17:02.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:01 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 12138/27816 objects degraded (43.637%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:17:02.099 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:02.137 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:02.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:02.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:02.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:01 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 12138/27816 objects degraded (43.637%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:17:02.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:02.274 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:02.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:02.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:02.375 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:02.402 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:02.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:02.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:02.533 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:02.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:02.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:02.634 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:02.656 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:02.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:02.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:02.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:02.790 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:02.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:02.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:02.892 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:02.917 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:02.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:02.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:03.019 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:03.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:03.058 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:03.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:03.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:02 smithi098 ceph-mon[97580]: pgmap v791: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 550 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 798 op/s; 12138/26289 objects degraded (46.171%); 859/26289 objects misplaced (3.268%); 0 B/s, 61 objects/s recovering 2024-09-21T10:17:03.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:02 smithi098 ceph-mon[100914]: pgmap v791: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 550 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 798 op/s; 12138/26289 objects degraded (46.171%); 859/26289 objects misplaced (3.268%); 0 B/s, 61 objects/s recovering 2024-09-21T10:17:03.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:02 smithi196 ceph-mon[84784]: pgmap v791: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 550 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 798 op/s; 12138/26289 objects degraded (46.171%); 859/26289 objects misplaced (3.268%); 0 B/s, 61 objects/s recovering 2024-09-21T10:17:03.160 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:03.186 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:03.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:03.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:03.240 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T10:17:03.240 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:17:03.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:03.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:03.366 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:03.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:03.467 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:03.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:03.514 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:03.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:03.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:03.667 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:03.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:03.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:03.769 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:03.794 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:03.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:03.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:03.895 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:03.920 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:03.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:03.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:03.952 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:17:03.952 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:17:04.021 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:04.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:04.073 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:04.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:04.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:04.197 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:04.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:04.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:04.299 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:04.337 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:04.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:04.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:04.439 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:04.465 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:04.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:04.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:04.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:04.601 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:04.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:04.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:04.702 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:04.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:04.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:04.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:04.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:04.851 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:04.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:04.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:04.953 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:04.979 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:04.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:04.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:05.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:05.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:04 smithi098 ceph-mon[97580]: pgmap v792: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 440 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 791 op/s; 12138/20985 objects degraded (57.841%); 517/20985 objects misplaced (2.464%); 0 B/s, 61 objects/s recovering 2024-09-21T10:17:05.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:04 smithi098 ceph-mon[100914]: pgmap v792: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 440 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 791 op/s; 12138/20985 objects degraded (57.841%); 517/20985 objects misplaced (2.464%); 0 B/s, 61 objects/s recovering 2024-09-21T10:17:05.104 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:05.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:05.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:04 smithi196 ceph-mon[84784]: pgmap v792: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 440 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 791 op/s; 12138/20985 objects degraded (57.841%); 517/20985 objects misplaced (2.464%); 0 B/s, 61 objects/s recovering 2024-09-21T10:17:05.206 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:05.230 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:05.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:05.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:05.332 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:05.367 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:05.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:05.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:05.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:05.493 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:05.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:05.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:05.595 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:05.620 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:05.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:05.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:05.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:17:05.721 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:05.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:05.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:05.912 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:05.937 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:05.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:05.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:06.039 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:06.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:06.066 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:06.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:06.166 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:06.169 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:17:06.191 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:06.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:06.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:06.293 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:06.315 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:06.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:06.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:06.417 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:06.448 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:06.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:06.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:06.550 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:06.572 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:06.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:06.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:06.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:06.695 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:06.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:06.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:06.797 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:06.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:06.819 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:06.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:06.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:06 smithi098 ceph-mon[97580]: pgmap v793: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 341 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 799 op/s; 11706/16215 objects degraded (72.192%); 517/16215 objects misplaced (3.188%); 0 B/s, 75 objects/s recovering 2024-09-21T10:17:06.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:06 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 11706/16215 objects degraded (72.192%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:17:06.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:06 smithi098 ceph-mon[100914]: pgmap v793: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 341 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 799 op/s; 11706/16215 objects degraded (72.192%); 517/16215 objects misplaced (3.188%); 0 B/s, 75 objects/s recovering 2024-09-21T10:17:06.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:06 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 11706/16215 objects degraded (72.192%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:17:06.920 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:06.941 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:06.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:06.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:07.042 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:07.089 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:07.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:07.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:07.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:06 smithi196 ceph-mon[84784]: pgmap v793: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 341 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 799 op/s; 11706/16215 objects degraded (72.192%); 517/16215 objects misplaced (3.188%); 0 B/s, 75 objects/s recovering 2024-09-21T10:17:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:06 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 11706/16215 objects degraded (72.192%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:17:07.191 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:07.230 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:07.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:07.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:07.331 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:07.363 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:07.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:07.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:07.465 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:07.489 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:07.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:07.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:07.591 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:07.634 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:07.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:07.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:07.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:07.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:07.805 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:07.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:07.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:07.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:07.906 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:07.929 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:07.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:07.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:08.031 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:08.055 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:08.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:08.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:08.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:08.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:08.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:08.201 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:08.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:08.302 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:08.337 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:08.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:08.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:08.439 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:08.465 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:08.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:08.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:08.529 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:17:08.529 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:17:08.566 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:17:08.566 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:17:08.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:08.602 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:17:08.602 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:17:08.623 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:08.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:08.633 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:17:08.633 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:17:08.665 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:17:08.665 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:17:08.696 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:17:08.696 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:17:08.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:08.729 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:17:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:17:08.756 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:17:08.756 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:17:08.776 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:08.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:08.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:08.878 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:08.919 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:08.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:08.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:09.021 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:09.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:08 smithi196 ceph-mon[84784]: pgmap v794: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 308 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 797 op/s; 11706/14661 objects degraded (79.844%); 517/14661 objects misplaced (3.526%); 0 B/s, 75 objects/s recovering 2024-09-21T10:17:09.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:08 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3209732737' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:09.057 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:09.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:09.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:09.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:08 smithi098 ceph-mon[97580]: pgmap v794: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 308 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 797 op/s; 11706/14661 objects degraded (79.844%); 517/14661 objects misplaced (3.526%); 0 B/s, 75 objects/s recovering 2024-09-21T10:17:09.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:08 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3209732737' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:09.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:08 smithi098 ceph-mon[100914]: pgmap v794: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 308 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 797 op/s; 11706/14661 objects degraded (79.844%); 517/14661 objects misplaced (3.526%); 0 B/s, 75 objects/s recovering 2024-09-21T10:17:09.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:08 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3209732737' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:09.158 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:09.183 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:09.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:09.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:09.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:09.317 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:09.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:09.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:09.419 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:09.454 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:09.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:09.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:09.556 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:09.580 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:09.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:09.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:09.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:09.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:09.722 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:09.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:09.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:09.846 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:09.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:09.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:09.940 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Removed 116514 objects 2024-09-21T10:17:09.940 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Clean up completed and total clean up time :146.676 2024-09-21T10:17:09.948 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:09.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:09.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:10.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:10.093 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:10.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:10.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:10.197 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:10.236 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:10.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:10.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:10.286 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:10.337 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:10.359 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:10.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:10.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:10.461 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:10.481 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:10.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:10.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:10.583 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:10.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:10.606 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:10.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:10.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:17:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:17:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:17:10.707 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:10.727 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:10.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:10.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:10.830 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:10.853 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:10.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:10.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:10.943 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:10 smithi098 ceph-mon[97580]: pgmap v795: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 113 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 734 op/s; 11245/5295 objects degraded (212.370%); 187/5295 objects misplaced (3.532%); 0 B/s, 81 objects/s recovering 2024-09-21T10:17:10.943 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:17:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:17:10] "GET /metrics HTTP/1.1" 200 36524 "" "Prometheus/2.51.0" 2024-09-21T10:17:10.944 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:10 smithi098 ceph-mon[100914]: pgmap v795: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 113 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 734 op/s; 11245/5295 objects degraded (212.370%); 187/5295 objects misplaced (3.532%); 0 B/s, 81 objects/s recovering 2024-09-21T10:17:10.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:10.976 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:10.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:10.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:11.077 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:11.099 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:11.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:11.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:11.138 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-21T10:17:11.139 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-21T10:17:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:10 smithi196 ceph-mon[84784]: pgmap v795: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 113 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 734 op/s; 11245/5295 objects degraded (212.370%); 187/5295 objects misplaced (3.532%); 0 B/s, 81 objects/s recovering 2024-09-21T10:17:11.200 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:11.222 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:11.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:11.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:11.323 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:11.345 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:11.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:11.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:11.447 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:11.468 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:11.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:11.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:11.570 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:11.592 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:11.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:11.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:11.694 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:11.740 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:11.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:11.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:11.841 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:11.863 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:11.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:11.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:11.904 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:11 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 11245/5295 objects degraded (212.370%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:17:11.904 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:11 smithi098 ceph-mon[97580]: osdmap e259: 8 total, 8 up, 7 in 2024-09-21T10:17:11.904 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:11 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 11245/5295 objects degraded (212.370%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:17:11.904 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:11 smithi098 ceph-mon[100914]: osdmap e259: 8 total, 8 up, 7 in 2024-09-21T10:17:11.965 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:11.986 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:11.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:11.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:12.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:12.110 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:12.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:12.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:11 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 11245/5295 objects degraded (212.370%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:17:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:11 smithi196 ceph-mon[84784]: osdmap e259: 8 total, 8 up, 7 in 2024-09-21T10:17:12.212 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:12.232 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:12.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:12.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:12.334 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:12.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:12.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:12.457 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:12.478 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:12.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:12.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:12.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:12.605 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:12.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:12.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:12.707 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:12.728 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:12.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:12.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:12.829 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:12.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:17:12.850 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:12.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:12.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:12.952 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:12.976 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:12.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:12.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:13.078 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:13.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:12 smithi098 ceph-mon[97580]: pgmap v797: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 90 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 736 op/s; 11245/4185 objects degraded (268.698%); 187/4185 objects misplaced (4.468%); 0 B/s, 89 objects/s recovering 2024-09-21T10:17:13.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:12 smithi098 ceph-mon[97580]: osdmap e260: 8 total, 8 up, 7 in 2024-09-21T10:17:13.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:12 smithi098 ceph-mon[100914]: pgmap v797: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 90 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 736 op/s; 11245/4185 objects degraded (268.698%); 187/4185 objects misplaced (4.468%); 0 B/s, 89 objects/s recovering 2024-09-21T10:17:13.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:12 smithi098 ceph-mon[100914]: osdmap e260: 8 total, 8 up, 7 in 2024-09-21T10:17:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:13.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:13.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:12 smithi196 ceph-mon[84784]: pgmap v797: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 90 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 736 op/s; 11245/4185 objects degraded (268.698%); 187/4185 objects misplaced (4.468%); 0 B/s, 89 objects/s recovering 2024-09-21T10:17:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:12 smithi196 ceph-mon[84784]: osdmap e260: 8 total, 8 up, 7 in 2024-09-21T10:17:13.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:13.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:13.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:13.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:13.349 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:13.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:13.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:13.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:13.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:13.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:13.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:13.598 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:13.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:13.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:13.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:13.722 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:13.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:13.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:13.824 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:13.847 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:13.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:13.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:13.949 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:13.970 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:13.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:13.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:14.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:14.093 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:14.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:14.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:14.195 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:14.218 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:14.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:14.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:14.319 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:14.341 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:14.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:14.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:14.442 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:14.464 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:14.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:14.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:14.565 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:14.587 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:14.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:14.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:14.688 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:14.714 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:14.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:14.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:14.816 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:14.838 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:14.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:14.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:14.940 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:14.966 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:14.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:14.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:15.067 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:15.092 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:15.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:15.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:15.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:14 smithi098 ceph-mon[97580]: pgmap v799: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 38 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 606 op/s; 11245/1698 objects degraded (662.250%); 0 B/s, 57 objects/s recovering 2024-09-21T10:17:15.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:14 smithi098 ceph-mon[97580]: 7.15 scrub starts 2024-09-21T10:17:15.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:14 smithi098 ceph-mon[97580]: 7.15 scrub ok 2024-09-21T10:17:15.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:14 smithi098 ceph-mon[100914]: pgmap v799: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 38 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 606 op/s; 11245/1698 objects degraded (662.250%); 0 B/s, 57 objects/s recovering 2024-09-21T10:17:15.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:14 smithi098 ceph-mon[100914]: 7.15 scrub starts 2024-09-21T10:17:15.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:14 smithi098 ceph-mon[100914]: 7.15 scrub ok 2024-09-21T10:17:15.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:14 smithi196 ceph-mon[84784]: pgmap v799: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 38 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 606 op/s; 11245/1698 objects degraded (662.250%); 0 B/s, 57 objects/s recovering 2024-09-21T10:17:15.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:14 smithi196 ceph-mon[84784]: 7.15 scrub starts 2024-09-21T10:17:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:14 smithi196 ceph-mon[84784]: 7.15 scrub ok 2024-09-21T10:17:15.194 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:15.215 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:15.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:15.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:15.317 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:15.343 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:15.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:15.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:15.445 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:15.470 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:15.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:15.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:15.571 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:15.593 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:15.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:15.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:15.695 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:15.720 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:17:15.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:15.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:15.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:15.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:15.846 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:15.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:15.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:15.948 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:15.974 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:15.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:15.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:16.075 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:16.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:15 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3783891008' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:17:16.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:15 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 11245/1698 objects degraded (662.250%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:17:16.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:15 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3783891008' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:17:16.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:15 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 11245/1698 objects degraded (662.250%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:17:16.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:16.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:16.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:16.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:15 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3783891008' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:17:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:15 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 11245/1698 objects degraded (662.250%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:17:16.203 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:16.228 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:16.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:16.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:16.330 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:16.353 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:16.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:16.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:16.455 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:16.477 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:16.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:16.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:16.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:16.602 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:16.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:16.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:16.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:16.726 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:16.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:16.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:16.827 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:16 smithi098 ceph-mon[97580]: pgmap v800: 99 pgs: 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 9.7 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 598 op/s; 10759/330 objects degraded (3260.303%); 0 B/s, 118 objects/s recovering 2024-09-21T10:17:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:16 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:17:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:16 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3783891008' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:17:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:16 smithi098 ceph-mon[97580]: osdmap e261: 8 total, 8 up, 7 in 2024-09-21T10:17:16.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:16 smithi098 ceph-mon[100914]: pgmap v800: 99 pgs: 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 9.7 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 598 op/s; 10759/330 objects degraded (3260.303%); 0 B/s, 118 objects/s recovering 2024-09-21T10:17:16.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:16 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:17:16.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:16 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3783891008' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:17:16.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:16 smithi098 ceph-mon[100914]: osdmap e261: 8 total, 8 up, 7 in 2024-09-21T10:17:16.853 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:16.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:16.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:16.955 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:16.976 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:16.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:16.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:17.078 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:17.099 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:17.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:17.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:16 smithi196 ceph-mon[84784]: pgmap v800: 99 pgs: 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 9.7 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 598 op/s; 10759/330 objects degraded (3260.303%); 0 B/s, 118 objects/s recovering 2024-09-21T10:17:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:16 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:17:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:16 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3783891008' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:17:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:16 smithi196 ceph-mon[84784]: osdmap e261: 8 total, 8 up, 7 in 2024-09-21T10:17:17.201 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:17.227 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:17.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:17.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:17.328 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:17.351 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:17.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:17.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:17.453 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:17.474 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:17.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:17.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:17.576 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:17.601 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:17.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:17.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:17.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:17.726 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:17.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:17.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:17.827 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:17.849 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:17.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:17.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:17.951 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:17.973 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:17.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:18.075 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:18.100 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:18.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:18.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:18.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:18.225 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:18.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:18.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:18.326 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:18.351 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:18.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:18.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:18.453 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:18.479 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:18.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:18.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:18.580 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:18.605 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:18.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:18.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:18.707 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:18.729 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:18.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:18.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:18.831 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:18 smithi098 ceph-mon[97580]: pgmap v802: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 9.7 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 195 op/s; 8718/330 objects degraded (2641.818%); 0 B/s, 238 objects/s recovering 2024-09-21T10:17:18.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:18 smithi098 ceph-mon[100914]: pgmap v802: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 9.7 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 195 op/s; 8718/330 objects degraded (2641.818%); 0 B/s, 238 objects/s recovering 2024-09-21T10:17:18.857 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:18.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:18.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:18.959 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:18.984 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:18.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:18.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:19.085 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:19.109 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:19.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:19.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:19.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:18 smithi196 ceph-mon[84784]: pgmap v802: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 9.7 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 195 op/s; 8718/330 objects degraded (2641.818%); 0 B/s, 238 objects/s recovering 2024-09-21T10:17:19.211 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:19.236 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:19.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:19.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:19.337 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:19.363 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:19.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:19.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:19.464 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:19.490 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:19.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:19.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:19.592 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:19.617 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:19.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:19.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:19.719 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:19.744 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:19.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:19.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:19.846 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:19.871 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:19.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:19.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:19.973 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:19.997 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:19.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:19.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:20.098 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:20.121 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:20.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:20.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:20.223 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:20.247 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:20.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:20.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:20.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:20.373 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:20.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:20.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:20.474 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:20.498 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:20.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:20.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:20.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:20.624 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:20.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:20.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:20.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:17:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:17:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:17:20.726 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:20.749 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:20.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:20.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:20.850 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:20.874 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:20.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:20.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:20.977 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:21.000 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:21.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:21.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:21.013 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:17:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:17:20] "GET /metrics HTTP/1.1" 200 36524 "" "Prometheus/2.51.0" 2024-09-21T10:17:21.014 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:20 smithi098 ceph-mon[97580]: pgmap v803: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 9.7 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 8718/330 objects degraded (2641.818%) 2024-09-21T10:17:21.014 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:20 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 8718/330 objects degraded (2641.818%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:17:21.014 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:20 smithi098 ceph-mon[100914]: pgmap v803: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 9.7 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 8718/330 objects degraded (2641.818%) 2024-09-21T10:17:21.014 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:20 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 8718/330 objects degraded (2641.818%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:17:21.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:21.123 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:21.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:21.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:20 smithi196 ceph-mon[84784]: pgmap v803: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 9.7 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 8718/330 objects degraded (2641.818%) 2024-09-21T10:17:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:20 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 8718/330 objects degraded (2641.818%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:17:21.166 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:21.225 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:21.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:21.250 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:21.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:21.351 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:21.374 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:21.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:21.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:21.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:21.499 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:21.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:21.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:21.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:21.623 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:21.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:21.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:21.724 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:21.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:21.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:21.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:21.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:21.874 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:21.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:21.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:21.975 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:22.001 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:22.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:22.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:17:22.085 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:17:22.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:22.126 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:22.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:22.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:22.227 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:22.252 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:22.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:22.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:22.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:22.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:22.379 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:22.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:22.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:22.503 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:22.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:22.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:22.605 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:22.631 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:22.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:22.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:22.732 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:22.756 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:22.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:22.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:22 smithi098 ceph-mon[97580]: pgmap v804: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 9.7 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 8166/330 objects degraded (2474.545%) 2024-09-21T10:17:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:22 smithi098 ceph-mon[97580]: from='client.45731 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:17:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:17:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:22.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:22 smithi098 ceph-mon[100914]: pgmap v804: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 9.7 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 8166/330 objects degraded (2474.545%) 2024-09-21T10:17:22.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:22 smithi098 ceph-mon[100914]: from='client.45731 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:17:22.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:17:22.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:22.858 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:22.883 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:22.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:22.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:22.984 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:23.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:23.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:23.109 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:23.134 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:23.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:23.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:23.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:22 smithi196 ceph-mon[84784]: pgmap v804: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 9.7 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 8166/330 objects degraded (2474.545%) 2024-09-21T10:17:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:22 smithi196 ceph-mon[84784]: from='client.45731 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:17:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:17:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:23.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:23.262 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:23.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:23.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:23.364 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:23.388 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:23.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:23.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:23.490 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:23.516 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:23.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:23.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:23.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:23.641 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:23.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:23.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:23.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:23.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:23.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:23.867 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:23.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:23.890 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:23.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:23.991 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:24.017 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:24.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:24.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:24.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:24.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:24.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:24.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:24.267 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:24.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:24.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:24.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:24.394 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:24.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:24.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:24.496 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:24.517 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:24.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:24.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:24.619 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:24.636 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":804,"stamp":"2024-09-21T10:17:21.332343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10164173,"num_objects":110,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110,"num_whiteouts":0,"num_read":6289,"num_read_kb":13765,"num_write":202728,"num_write_kb":5714476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12738,"num_bytes_recovered":661530854,"num_keys_recovered":30,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":175945,"ondisk_log_size":175945,"up":297,"acting":294,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":308,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":3988752,"kb_used_data":1034816,"kb_used_omap":33,"kb_used_meta":2953822,"kb_avail":652227312,"statfs":{"total":671965249536,"available":667880767488,"internally_reserved":0,"allocated":1059651584,"data_stored":1050071690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34649,"internal_metadata":3024713895},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-84115480,"num_objects":-1285,"num_object_clones":0,"num_object_copies":-3855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3079,"num_objects_misplaced":-187,"num_objects_unfound":0,"num_objects_dirty":-1285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-5962,"num_write_kb":-232224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-551,"num_bytes_recovered":-71172096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.619662"},"pg_stats":[{"pgid":"7.17","version":"258'8796","reported_seq":8953,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.810474+0000","last_change":"2024-09-21T10:13:44.684354+0000","last_active":"2024-09-21T10:17:11.810474+0000","last_peered":"2024-09-21T10:17:11.810474+0000","last_clean":"2024-09-21T10:17:11.810474+0000","last_became_active":"2024-09-21T10:13:36.527577+0000","last_became_peered":"2024-09-21T10:13:36.527577+0000","last_unstale":"2024-09-21T10:17:11.810474+0000","last_undegraded":"2024-09-21T10:17:11.810474+0000","last_fullsized":"2024-09-21T10:17:11.810474+0000","mapping_epoch":209,"log_start":"206'973","ondisk_log_start":"206'973","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":5,"last_scrub":"213'3271","last_scrub_stamp":"2024-09-21T10:13:44.684263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:44.684263+0000","objects_scrubbed":1783,"log_size":7823,"log_dups_size":0,"ondisk_log_size":7823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:11:57.613857+0000","scrub_duration":0.165485878,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5525,"num_write_kb":119744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":393,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.727869+0000","last_change":"2024-09-21T10:02:14.097991+0000","last_active":"2024-09-21T10:17:15.727869+0000","last_peered":"2024-09-21T10:17:15.727869+0000","last_clean":"2024-09-21T10:17:15.727869+0000","last_became_active":"2024-09-21T10:02:14.097407+0000","last_became_peered":"2024-09-21T10:02:14.097407+0000","last_unstale":"2024-09-21T10:17:15.727869+0000","last_undegraded":"2024-09-21T10:17:15.727869+0000","last_fullsized":"2024-09-21T10:17:15.727869+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:18:42.098886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":276,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.714115+0000","last_change":"2024-09-21T10:08:00.406360+0000","last_active":"2024-09-21T10:17:11.714115+0000","last_peered":"2024-09-21T10:17:11.714115+0000","last_clean":"2024-09-21T10:17:11.714115+0000","last_became_active":"2024-09-21T10:08:00.380042+0000","last_became_peered":"2024-09-21T10:08:00.380042+0000","last_unstale":"2024-09-21T10:17:11.714115+0000","last_undegraded":"2024-09-21T10:17:11.714115+0000","last_fullsized":"2024-09-21T10:17:11.714115+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:49:46.544051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.16","version":"258'8539","reported_seq":8688,"reported_epoch":261,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:17:16.748779+0000","last_change":"2024-09-21T10:14:57.827443+0000","last_active":"2024-09-21T10:17:16.748779+0000","last_peered":"2024-09-21T10:17:16.748779+0000","last_clean":"2024-09-21T10:14:54.690226+0000","last_became_active":"2024-09-21T10:14:57.819744+0000","last_became_peered":"2024-09-21T10:14:57.819744+0000","last_unstale":"2024-09-21T10:17:16.748779+0000","last_undegraded":"2024-09-21T10:14:57.724005+0000","last_fullsized":"2024-09-21T10:14:57.703932+0000","mapping_epoch":237,"log_start":"207'2153","ondisk_log_start":"207'2153","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":6386,"log_dups_size":0,"ondisk_log_size":6386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7098,"num_write_kb":227840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,7","objects":-3171}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":182,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.714270+0000","last_change":"2024-09-21T10:07:43.789751+0000","last_active":"2024-09-21T10:17:11.714270+0000","last_peered":"2024-09-21T10:17:11.714270+0000","last_clean":"2024-09-21T10:17:11.714270+0000","last_became_active":"2024-09-21T10:07:43.789448+0000","last_became_peered":"2024-09-21T10:07:43.789448+0000","last_unstale":"2024-09-21T10:17:11.714270+0000","last_undegraded":"2024-09-21T10:17:11.714270+0000","last_fullsized":"2024-09-21T10:17:11.714270+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:36.586942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":389,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:16.748329+0000","last_change":"2024-09-21T10:02:14.098949+0000","last_active":"2024-09-21T10:17:16.748329+0000","last_peered":"2024-09-21T10:17:16.748329+0000","last_clean":"2024-09-21T10:17:16.748329+0000","last_became_active":"2024-09-21T10:02:14.098646+0000","last_became_peered":"2024-09-21T10:02:14.098646+0000","last_unstale":"2024-09-21T10:17:16.748329+0000","last_undegraded":"2024-09-21T10:17:16.748329+0000","last_fullsized":"2024-09-21T10:17:16.748329+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:50:56.862529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"258'8723","reported_seq":16091,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:13.682718+0000","last_change":"2024-09-21T10:17:13.682666+0000","last_active":"2024-09-21T10:17:13.682718+0000","last_peered":"2024-09-21T10:17:13.682718+0000","last_clean":"2024-09-21T10:17:13.682718+0000","last_became_active":"2024-09-21T10:17:11.734748+0000","last_became_peered":"2024-09-21T10:17:11.734748+0000","last_unstale":"2024-09-21T10:17:13.682718+0000","last_undegraded":"2024-09-21T10:17:13.682718+0000","last_fullsized":"2024-09-21T10:17:13.682718+0000","mapping_epoch":259,"log_start":"206'1720","ondisk_log_start":"206'1720","created":209,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":5,"last_scrub":"258'8723","last_scrub_stamp":"2024-09-21T10:17:13.682594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:17:13.682594+0000","objects_scrubbed":0,"log_size":7003,"log_dups_size":0,"ondisk_log_size":7003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:39.820580+0000","scrub_duration":0.0047462720000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":389,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:16.750053+0000","last_change":"2024-09-21T10:02:14.098205+0000","last_active":"2024-09-21T10:17:16.750053+0000","last_peered":"2024-09-21T10:17:16.750053+0000","last_clean":"2024-09-21T10:17:16.750053+0000","last_became_active":"2024-09-21T10:02:14.097911+0000","last_became_peered":"2024-09-21T10:02:14.097911+0000","last_unstale":"2024-09-21T10:17:16.750053+0000","last_undegraded":"2024-09-21T10:17:16.750053+0000","last_fullsized":"2024-09-21T10:17:16.750053+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:55.971191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":141,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.810576+0000","last_change":"2024-09-21T10:08:00.385671+0000","last_active":"2024-09-21T10:17:11.810576+0000","last_peered":"2024-09-21T10:17:11.810576+0000","last_clean":"2024-09-21T10:17:11.810576+0000","last_became_active":"2024-09-21T10:08:00.385368+0000","last_became_peered":"2024-09-21T10:08:00.385368+0000","last_unstale":"2024-09-21T10:17:11.810576+0000","last_undegraded":"2024-09-21T10:17:11.810576+0000","last_fullsized":"2024-09-21T10:17:11.810576+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:21.799149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.14","version":"258'8814","reported_seq":21804,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.727577+0000","last_change":"2024-09-21T10:16:25.295721+0000","last_active":"2024-09-21T10:17:15.727577+0000","last_peered":"2024-09-21T10:17:15.727577+0000","last_clean":"2024-09-21T10:17:15.727577+0000","last_became_active":"2024-09-21T10:16:25.275905+0000","last_became_peered":"2024-09-21T10:16:25.275905+0000","last_unstale":"2024-09-21T10:17:15.727577+0000","last_undegraded":"2024-09-21T10:17:15.727577+0000","last_fullsized":"2024-09-21T10:17:15.727577+0000","mapping_epoch":251,"log_start":"207'2513","ondisk_log_start":"207'2513","created":209,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":6301,"log_dups_size":0,"ondisk_log_size":6301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:17:30.243532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7382,"num_write_kb":236416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3208,"num_bytes_recovered":210239488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":393,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.729524+0000","last_change":"2024-09-21T10:02:14.097817+0000","last_active":"2024-09-21T10:17:15.729524+0000","last_peered":"2024-09-21T10:17:15.729524+0000","last_clean":"2024-09-21T10:17:15.729524+0000","last_became_active":"2024-09-21T10:02:14.097466+0000","last_became_peered":"2024-09-21T10:02:14.097466+0000","last_unstale":"2024-09-21T10:17:15.729524+0000","last_undegraded":"2024-09-21T10:17:15.729524+0000","last_fullsized":"2024-09-21T10:17:15.729524+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:57.883118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":233,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.810382+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:17:11.810382+0000","last_peered":"2024-09-21T10:17:11.810382+0000","last_clean":"2024-09-21T10:17:11.810382+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:17:11.810382+0000","last_undegraded":"2024-09-21T10:17:11.810382+0000","last_fullsized":"2024-09-21T10:17:11.810382+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"258'14616","reported_seq":14756,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.731641+0000","last_change":"2024-09-21T10:13:00.065993+0000","last_active":"2024-09-21T10:17:15.731641+0000","last_peered":"2024-09-21T10:17:15.731641+0000","last_clean":"2024-09-21T10:17:15.731641+0000","last_became_active":"2024-09-21T10:13:00.065574+0000","last_became_peered":"2024-09-21T10:13:00.065574+0000","last_unstale":"2024-09-21T10:17:15.731641+0000","last_undegraded":"2024-09-21T10:17:15.731641+0000","last_fullsized":"2024-09-21T10:17:15.731641+0000","mapping_epoch":202,"log_start":"239'8639","ondisk_log_start":"239'8639","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":5977,"log_dups_size":0,"ondisk_log_size":5977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:53:02.159144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14616,"num_write_kb":467712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":269,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.714454+0000","last_change":"2024-09-21T10:08:00.392499+0000","last_active":"2024-09-21T10:17:11.714454+0000","last_peered":"2024-09-21T10:17:11.714454+0000","last_clean":"2024-09-21T10:17:11.714454+0000","last_became_active":"2024-09-21T10:08:00.379165+0000","last_became_peered":"2024-09-21T10:08:00.379165+0000","last_unstale":"2024-09-21T10:17:11.714454+0000","last_undegraded":"2024-09-21T10:17:11.714454+0000","last_fullsized":"2024-09-21T10:17:11.714454+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:18.933144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":125,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.713815+0000","last_change":"2024-09-21T10:07:43.794677+0000","last_active":"2024-09-21T10:17:11.713815+0000","last_peered":"2024-09-21T10:17:11.713815+0000","last_clean":"2024-09-21T10:17:11.713815+0000","last_became_active":"2024-09-21T10:07:43.794348+0000","last_became_peered":"2024-09-21T10:07:43.794348+0000","last_unstale":"2024-09-21T10:17:11.713815+0000","last_undegraded":"2024-09-21T10:17:11.713815+0000","last_fullsized":"2024-09-21T10:17:11.713815+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:11.159014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"258'14630","reported_seq":14770,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.727506+0000","last_change":"2024-09-21T10:13:00.064683+0000","last_active":"2024-09-21T10:17:15.727506+0000","last_peered":"2024-09-21T10:17:15.727506+0000","last_clean":"2024-09-21T10:17:15.727506+0000","last_became_active":"2024-09-21T10:13:00.064382+0000","last_became_peered":"2024-09-21T10:13:00.064382+0000","last_unstale":"2024-09-21T10:17:15.727506+0000","last_undegraded":"2024-09-21T10:17:15.727506+0000","last_fullsized":"2024-09-21T10:17:15.727506+0000","mapping_epoch":202,"log_start":"239'8416","ondisk_log_start":"239'8416","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":6214,"log_dups_size":0,"ondisk_log_size":6214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:11.591410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14630,"num_write_kb":468160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":270,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.714496+0000","last_change":"2024-09-21T10:08:00.412449+0000","last_active":"2024-09-21T10:17:11.714496+0000","last_peered":"2024-09-21T10:17:11.714496+0000","last_clean":"2024-09-21T10:17:11.714496+0000","last_became_active":"2024-09-21T10:08:00.386530+0000","last_became_peered":"2024-09-21T10:08:00.386530+0000","last_unstale":"2024-09-21T10:17:11.714496+0000","last_undegraded":"2024-09-21T10:17:11.714496+0000","last_fullsized":"2024-09-21T10:17:11.714496+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:12:50.581176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":315,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.731630+0000","last_change":"2024-09-21T10:08:18.475007+0000","last_active":"2024-09-21T10:17:15.731630+0000","last_peered":"2024-09-21T10:17:15.731630+0000","last_clean":"2024-09-21T10:17:15.731630+0000","last_became_active":"2024-09-21T10:08:18.474730+0000","last_became_peered":"2024-09-21T10:08:18.474730+0000","last_unstale":"2024-09-21T10:17:15.731630+0000","last_undegraded":"2024-09-21T10:17:15.731630+0000","last_fullsized":"2024-09-21T10:17:15.731630+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:35.455790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.9","version":"258'8574","reported_seq":8722,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:16.748241+0000","last_change":"2024-09-21T10:13:49.895960+0000","last_active":"2024-09-21T10:17:16.748241+0000","last_peered":"2024-09-21T10:17:16.748241+0000","last_clean":"2024-09-21T10:17:16.748241+0000","last_became_active":"2024-09-21T10:13:36.526784+0000","last_became_peered":"2024-09-21T10:13:36.526784+0000","last_unstale":"2024-09-21T10:17:16.748241+0000","last_undegraded":"2024-09-21T10:17:16.748241+0000","last_fullsized":"2024-09-21T10:17:16.748241+0000","mapping_epoch":202,"log_start":"207'2315","ondisk_log_start":"207'2315","created":202,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":6259,"log_dups_size":0,"ondisk_log_size":6259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":720896,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7151,"num_write_kb":229184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":210,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.714496+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:17:11.714496+0000","last_peered":"2024-09-21T10:17:11.714496+0000","last_clean":"2024-09-21T10:17:11.714496+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:17:11.714496+0000","last_undegraded":"2024-09-21T10:17:11.714496+0000","last_fullsized":"2024-09-21T10:17:11.714496+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":224,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.810585+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:17:11.810585+0000","last_peered":"2024-09-21T10:17:11.810585+0000","last_clean":"2024-09-21T10:17:11.810585+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:17:11.810585+0000","last_undegraded":"2024-09-21T10:17:11.810585+0000","last_fullsized":"2024-09-21T10:17:11.810585+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"258'8766","reported_seq":8938,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.910121+0000","last_change":"2024-09-21T10:13:40.147612+0000","last_active":"2024-09-21T10:17:15.910121+0000","last_peered":"2024-09-21T10:17:15.910121+0000","last_clean":"2024-09-21T10:17:15.910121+0000","last_became_active":"2024-09-21T10:13:36.522799+0000","last_became_peered":"2024-09-21T10:13:36.522799+0000","last_unstale":"2024-09-21T10:17:15.910121+0000","last_undegraded":"2024-09-21T10:17:15.910121+0000","last_fullsized":"2024-09-21T10:17:15.910121+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'3074","last_scrub_stamp":"2024-09-21T10:13:40.147275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:40.147275+0000","objects_scrubbed":1635,"log_size":8766,"log_dups_size":0,"ondisk_log_size":8766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:57:47.053800+0000","scrub_duration":458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5705,"num_write_kb":130240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":299,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.727938+0000","last_change":"2024-09-21T10:07:43.793008+0000","last_active":"2024-09-21T10:17:15.727938+0000","last_peered":"2024-09-21T10:17:15.727938+0000","last_clean":"2024-09-21T10:17:15.727938+0000","last_became_active":"2024-09-21T10:07:43.792683+0000","last_became_peered":"2024-09-21T10:07:43.792683+0000","last_unstale":"2024-09-21T10:17:15.727938+0000","last_undegraded":"2024-09-21T10:17:15.727938+0000","last_fullsized":"2024-09-21T10:17:15.727938+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:46:13.137478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":410,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:16.748425+0000","last_change":"2024-09-21T10:08:00.389645+0000","last_active":"2024-09-21T10:17:16.748425+0000","last_peered":"2024-09-21T10:17:16.748425+0000","last_clean":"2024-09-21T10:17:16.748425+0000","last_became_active":"2024-09-21T10:08:00.386076+0000","last_became_peered":"2024-09-21T10:08:00.386076+0000","last_unstale":"2024-09-21T10:17:16.748425+0000","last_undegraded":"2024-09-21T10:17:16.748425+0000","last_fullsized":"2024-09-21T10:17:16.748425+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:58:10.646600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"258'14788","reported_seq":14928,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:16.747930+0000","last_change":"2024-09-21T10:13:00.064670+0000","last_active":"2024-09-21T10:17:16.747930+0000","last_peered":"2024-09-21T10:17:16.747930+0000","last_clean":"2024-09-21T10:17:16.747930+0000","last_became_active":"2024-09-21T10:13:00.063176+0000","last_became_peered":"2024-09-21T10:13:00.063176+0000","last_unstale":"2024-09-21T10:17:16.747930+0000","last_undegraded":"2024-09-21T10:17:16.747930+0000","last_fullsized":"2024-09-21T10:17:16.747930+0000","mapping_epoch":202,"log_start":"239'8700","ondisk_log_start":"239'8700","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":6088,"log_dups_size":0,"ondisk_log_size":6088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:14:56.420455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14788,"num_write_kb":473216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":424,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.727733+0000","last_change":"2024-09-21T10:08:00.395898+0000","last_active":"2024-09-21T10:17:15.727733+0000","last_peered":"2024-09-21T10:17:15.727733+0000","last_clean":"2024-09-21T10:17:15.727733+0000","last_became_active":"2024-09-21T10:08:00.390462+0000","last_became_peered":"2024-09-21T10:08:00.390462+0000","last_unstale":"2024-09-21T10:17:15.727733+0000","last_undegraded":"2024-09-21T10:17:15.727733+0000","last_fullsized":"2024-09-21T10:17:15.727733+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:29:01.877089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":311,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.729239+0000","last_change":"2024-09-21T10:08:00.383016+0000","last_active":"2024-09-21T10:17:15.729239+0000","last_peered":"2024-09-21T10:17:15.729239+0000","last_clean":"2024-09-21T10:17:15.729239+0000","last_became_active":"2024-09-21T10:08:00.382708+0000","last_became_peered":"2024-09-21T10:08:00.382708+0000","last_unstale":"2024-09-21T10:17:15.729239+0000","last_undegraded":"2024-09-21T10:17:15.729239+0000","last_fullsized":"2024-09-21T10:17:15.729239+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:30:52.797259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"258'14468","reported_seq":14608,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.910586+0000","last_change":"2024-09-21T10:13:00.067640+0000","last_active":"2024-09-21T10:17:15.910586+0000","last_peered":"2024-09-21T10:17:15.910586+0000","last_clean":"2024-09-21T10:17:15.910586+0000","last_became_active":"2024-09-21T10:13:00.067380+0000","last_became_peered":"2024-09-21T10:13:00.067380+0000","last_unstale":"2024-09-21T10:17:15.910586+0000","last_undegraded":"2024-09-21T10:17:15.910586+0000","last_fullsized":"2024-09-21T10:17:15.910586+0000","mapping_epoch":202,"log_start":"225'6131","ondisk_log_start":"225'6131","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":8337,"log_dups_size":0,"ondisk_log_size":8337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:55:46.885222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14468,"num_write_kb":462976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":255,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.810669+0000","last_change":"2024-09-21T10:08:00.379604+0000","last_active":"2024-09-21T10:17:11.810669+0000","last_peered":"2024-09-21T10:17:11.810669+0000","last_clean":"2024-09-21T10:17:11.810669+0000","last_became_active":"2024-09-21T10:08:00.379301+0000","last_became_peered":"2024-09-21T10:08:00.379301+0000","last_unstale":"2024-09-21T10:17:11.810669+0000","last_undegraded":"2024-09-21T10:17:11.810669+0000","last_fullsized":"2024-09-21T10:17:11.810669+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:32:41.963929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":322,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.910733+0000","last_change":"2024-09-21T10:08:00.387420+0000","last_active":"2024-09-21T10:17:15.910733+0000","last_peered":"2024-09-21T10:17:15.910733+0000","last_clean":"2024-09-21T10:17:15.910733+0000","last_became_active":"2024-09-21T10:08:00.386843+0000","last_became_peered":"2024-09-21T10:08:00.386843+0000","last_unstale":"2024-09-21T10:17:15.910733+0000","last_undegraded":"2024-09-21T10:17:15.910733+0000","last_fullsized":"2024-09-21T10:17:15.910733+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:51:00.654836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"258'14358","reported_seq":14426,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.810661+0000","last_change":"2024-09-21T10:13:00.067618+0000","last_active":"2024-09-21T10:17:11.810661+0000","last_peered":"2024-09-21T10:17:11.810661+0000","last_clean":"2024-09-21T10:17:11.810661+0000","last_became_active":"2024-09-21T10:13:00.067306+0000","last_became_peered":"2024-09-21T10:13:00.067306+0000","last_unstale":"2024-09-21T10:17:11.810661+0000","last_undegraded":"2024-09-21T10:17:11.810661+0000","last_fullsized":"2024-09-21T10:17:11.810661+0000","mapping_epoch":202,"log_start":"231'7453","ondisk_log_start":"231'7453","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":6905,"log_dups_size":0,"ondisk_log_size":6905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:45:24.816636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14358,"num_write_kb":459393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":306,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.910038+0000","last_change":"2024-09-21T10:08:00.382496+0000","last_active":"2024-09-21T10:17:15.910038+0000","last_peered":"2024-09-21T10:17:15.910038+0000","last_clean":"2024-09-21T10:17:15.910038+0000","last_became_active":"2024-09-21T10:08:00.382180+0000","last_became_peered":"2024-09-21T10:08:00.382180+0000","last_unstale":"2024-09-21T10:17:15.910038+0000","last_undegraded":"2024-09-21T10:17:15.910038+0000","last_fullsized":"2024-09-21T10:17:15.910038+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:16.074736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":194,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.810345+0000","last_change":"2024-09-21T10:08:38.906801+0000","last_active":"2024-09-21T10:17:11.810345+0000","last_peered":"2024-09-21T10:17:11.810345+0000","last_clean":"2024-09-21T10:17:11.810345+0000","last_became_active":"2024-09-21T10:08:38.906487+0000","last_became_peered":"2024-09-21T10:08:38.906487+0000","last_unstale":"2024-09-21T10:17:11.810345+0000","last_undegraded":"2024-09-21T10:17:11.810345+0000","last_fullsized":"2024-09-21T10:17:11.810345+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:31.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"258'14562","reported_seq":14629,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.810808+0000","last_change":"2024-09-21T10:13:00.063591+0000","last_active":"2024-09-21T10:17:11.810808+0000","last_peered":"2024-09-21T10:17:11.810808+0000","last_clean":"2024-09-21T10:17:11.810808+0000","last_became_active":"2024-09-21T10:13:00.063262+0000","last_became_peered":"2024-09-21T10:13:00.063262+0000","last_unstale":"2024-09-21T10:17:11.810808+0000","last_undegraded":"2024-09-21T10:17:11.810808+0000","last_fullsized":"2024-09-21T10:17:11.810808+0000","mapping_epoch":202,"log_start":"232'7728","ondisk_log_start":"232'7728","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":6834,"log_dups_size":0,"ondisk_log_size":6834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:52:08.153986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14562,"num_write_kb":465984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":394,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.727834+0000","last_change":"2024-09-21T10:02:14.097462+0000","last_active":"2024-09-21T10:17:15.727834+0000","last_peered":"2024-09-21T10:17:15.727834+0000","last_clean":"2024-09-21T10:17:15.727834+0000","last_became_active":"2024-09-21T10:02:14.097167+0000","last_became_peered":"2024-09-21T10:02:14.097167+0000","last_unstale":"2024-09-21T10:17:15.727834+0000","last_undegraded":"2024-09-21T10:17:15.727834+0000","last_fullsized":"2024-09-21T10:17:15.727834+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:54.922731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":399,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.732291+0000","last_change":"2024-09-21T10:08:00.378569+0000","last_active":"2024-09-21T10:17:15.732291+0000","last_peered":"2024-09-21T10:17:15.732291+0000","last_clean":"2024-09-21T10:17:15.732291+0000","last_became_active":"2024-09-21T10:08:00.378015+0000","last_became_peered":"2024-09-21T10:08:00.378015+0000","last_unstale":"2024-09-21T10:17:15.732291+0000","last_undegraded":"2024-09-21T10:17:15.732291+0000","last_fullsized":"2024-09-21T10:17:15.732291+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:58.121005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"258'8721","reported_seq":8897,"reported_epoch":261,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:17:16.748838+0000","last_change":"2024-09-21T10:16:06.043524+0000","last_active":"2024-09-21T10:17:16.748838+0000","last_peered":"2024-09-21T10:17:16.748838+0000","last_clean":"2024-09-21T10:16:03.596201+0000","last_became_active":"2024-09-21T10:16:06.022079+0000","last_became_peered":"2024-09-21T10:16:06.022079+0000","last_unstale":"2024-09-21T10:17:16.748838+0000","last_undegraded":"2024-09-21T10:16:05.657401+0000","last_fullsized":"2024-09-21T10:16:05.629959+0000","mapping_epoch":247,"log_start":"207'2586","ondisk_log_start":"207'2586","created":202,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'3132","last_scrub_stamp":"2024-09-21T10:13:41.424720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:41.424720+0000","objects_scrubbed":1704,"log_size":6135,"log_dups_size":0,"ondisk_log_size":6135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":522,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3651,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5598,"num_write_kb":124608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,7","objects":-1593}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":343,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.713960+0000","last_change":"2024-09-21T09:59:54.265556+0000","last_active":"2024-09-21T10:17:11.713960+0000","last_peered":"2024-09-21T10:17:11.713960+0000","last_clean":"2024-09-21T10:17:11.713960+0000","last_became_active":"2024-09-21T09:59:54.265033+0000","last_became_peered":"2024-09-21T09:59:54.265033+0000","last_unstale":"2024-09-21T10:17:11.713960+0000","last_undegraded":"2024-09-21T10:17:11.713960+0000","last_fullsized":"2024-09-21T10:17:11.713960+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T09:48:34.850872+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:40.449756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":686,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.910541+0000","last_change":"2024-09-21T10:07:43.790490+0000","last_active":"2024-09-21T10:17:15.910541+0000","last_peered":"2024-09-21T10:17:15.910541+0000","last_clean":"2024-09-21T10:17:15.910541+0000","last_became_active":"2024-09-21T10:07:43.789986+0000","last_became_peered":"2024-09-21T10:07:43.789986+0000","last_unstale":"2024-09-21T10:17:15.910541+0000","last_undegraded":"2024-09-21T10:17:15.910541+0000","last_fullsized":"2024-09-21T10:17:15.910541+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:08.740356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":481,"num_read_kb":1580,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":348,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.727674+0000","last_change":"2024-09-21T10:02:14.099961+0000","last_active":"2024-09-21T10:17:15.727674+0000","last_peered":"2024-09-21T10:17:15.727674+0000","last_clean":"2024-09-21T10:17:15.727674+0000","last_became_active":"2024-09-21T10:02:14.099067+0000","last_became_peered":"2024-09-21T10:02:14.099067+0000","last_unstale":"2024-09-21T10:17:15.727674+0000","last_undegraded":"2024-09-21T10:17:15.727674+0000","last_fullsized":"2024-09-21T10:17:15.727674+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:41.037532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":237,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.810955+0000","last_change":"2024-09-21T10:08:00.386135+0000","last_active":"2024-09-21T10:17:11.810955+0000","last_peered":"2024-09-21T10:17:11.810955+0000","last_clean":"2024-09-21T10:17:11.810955+0000","last_became_active":"2024-09-21T10:08:00.385796+0000","last_became_peered":"2024-09-21T10:08:00.385796+0000","last_unstale":"2024-09-21T10:17:11.810955+0000","last_undegraded":"2024-09-21T10:17:11.810955+0000","last_fullsized":"2024-09-21T10:17:11.810955+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:27.054699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"258'8740","reported_seq":8962,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.714190+0000","last_change":"2024-09-21T10:14:53.771164+0000","last_active":"2024-09-21T10:17:11.714190+0000","last_peered":"2024-09-21T10:17:11.714190+0000","last_clean":"2024-09-21T10:17:11.714190+0000","last_became_active":"2024-09-21T10:13:36.529528+0000","last_became_peered":"2024-09-21T10:13:36.529528+0000","last_unstale":"2024-09-21T10:17:11.714190+0000","last_undegraded":"2024-09-21T10:17:11.714190+0000","last_fullsized":"2024-09-21T10:17:11.714190+0000","mapping_epoch":202,"log_start":"207'1815","ondisk_log_start":"207'1815","created":202,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'5397","last_scrub_stamp":"2024-09-21T10:14:53.771071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:14:53.771071+0000","objects_scrubbed":3357,"log_size":6925,"log_dups_size":0,"ondisk_log_size":6925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:42:22.675965+0000","scrub_duration":1.2314094090000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3357,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":606,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.713949+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:17:11.713949+0000","last_peered":"2024-09-21T10:17:11.713949+0000","last_clean":"2024-09-21T10:17:11.713949+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:17:11.713949+0000","last_undegraded":"2024-09-21T10:17:11.713949+0000","last_fullsized":"2024-09-21T10:17:11.713949+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":411,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:16.748615+0000","last_change":"2024-09-21T10:08:00.383928+0000","last_active":"2024-09-21T10:17:16.748615+0000","last_peered":"2024-09-21T10:17:16.748615+0000","last_clean":"2024-09-21T10:17:16.748615+0000","last_became_active":"2024-09-21T10:08:00.383625+0000","last_became_peered":"2024-09-21T10:08:00.383625+0000","last_unstale":"2024-09-21T10:17:16.748615+0000","last_undegraded":"2024-09-21T10:17:16.748615+0000","last_fullsized":"2024-09-21T10:17:16.748615+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:39.207283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":401,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:16.750151+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:17:16.750151+0000","last_peered":"2024-09-21T10:17:16.750151+0000","last_clean":"2024-09-21T10:17:16.750151+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:17:16.750151+0000","last_undegraded":"2024-09-21T10:17:16.750151+0000","last_fullsized":"2024-09-21T10:17:16.750151+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"258'8720","reported_seq":8892,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:16.748283+0000","last_change":"2024-09-21T10:13:44.137191+0000","last_active":"2024-09-21T10:17:16.748283+0000","last_peered":"2024-09-21T10:17:16.748283+0000","last_clean":"2024-09-21T10:17:16.748283+0000","last_became_active":"2024-09-21T10:13:36.520394+0000","last_became_peered":"2024-09-21T10:13:36.520394+0000","last_unstale":"2024-09-21T10:17:16.748283+0000","last_undegraded":"2024-09-21T10:17:16.748283+0000","last_fullsized":"2024-09-21T10:17:16.748283+0000","mapping_epoch":202,"log_start":"207'2571","ondisk_log_start":"207'2571","created":202,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'3166","last_scrub_stamp":"2024-09-21T10:13:44.137119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:44.137119+0000","objects_scrubbed":1736,"log_size":6149,"log_dups_size":0,"ondisk_log_size":6149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:49:32.175480+0000","scrub_duration":264,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5558,"num_write_kb":122304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":1138,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.910129+0000","last_change":"2024-09-21T10:02:14.100143+0000","last_active":"2024-09-21T10:17:15.910129+0000","last_peered":"2024-09-21T10:17:15.910129+0000","last_clean":"2024-09-21T10:17:15.910129+0000","last_became_active":"2024-09-21T10:02:14.099586+0000","last_became_peered":"2024-09-21T10:02:14.099586+0000","last_unstale":"2024-09-21T10:17:15.910129+0000","last_undegraded":"2024-09-21T10:17:15.910129+0000","last_fullsized":"2024-09-21T10:17:15.910129+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:34.064652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":273,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.731763+0000","last_change":"2024-09-21T10:08:18.470748+0000","last_active":"2024-09-21T10:17:15.731763+0000","last_peered":"2024-09-21T10:17:15.731763+0000","last_clean":"2024-09-21T10:17:15.731763+0000","last_became_active":"2024-09-21T10:08:18.470227+0000","last_became_peered":"2024-09-21T10:08:18.470227+0000","last_unstale":"2024-09-21T10:17:15.731763+0000","last_undegraded":"2024-09-21T10:17:15.731763+0000","last_fullsized":"2024-09-21T10:17:15.731763+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:16.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":395,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.910733+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:17:15.910733+0000","last_peered":"2024-09-21T10:17:15.910733+0000","last_clean":"2024-09-21T10:17:15.910733+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:17:15.910733+0000","last_undegraded":"2024-09-21T10:17:15.910733+0000","last_fullsized":"2024-09-21T10:17:15.910733+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"258'8896","reported_seq":9049,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.810555+0000","last_change":"2024-09-21T10:13:40.065182+0000","last_active":"2024-09-21T10:17:11.810555+0000","last_peered":"2024-09-21T10:17:11.810555+0000","last_clean":"2024-09-21T10:17:11.810555+0000","last_became_active":"2024-09-21T10:13:36.518471+0000","last_became_peered":"2024-09-21T10:13:36.518471+0000","last_unstale":"2024-09-21T10:17:11.810555+0000","last_undegraded":"2024-09-21T10:17:11.810555+0000","last_fullsized":"2024-09-21T10:17:11.810555+0000","mapping_epoch":202,"log_start":"206'1224","ondisk_log_start":"206'1224","created":202,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'3140","last_scrub_stamp":"2024-09-21T10:13:40.065082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:40.065082+0000","objects_scrubbed":1666,"log_size":7672,"log_dups_size":0,"ondisk_log_size":7672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:34.697455+0000","scrub_duration":0.54074375200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5768,"num_write_kb":131264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":871,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.731990+0000","last_change":"2024-09-21T10:07:43.794767+0000","last_active":"2024-09-21T10:17:15.731990+0000","last_peered":"2024-09-21T10:17:15.731990+0000","last_clean":"2024-09-21T10:17:15.731990+0000","last_became_active":"2024-09-21T10:07:43.794062+0000","last_became_peered":"2024-09-21T10:07:43.794062+0000","last_unstale":"2024-09-21T10:17:15.731990+0000","last_undegraded":"2024-09-21T10:17:15.731990+0000","last_fullsized":"2024-09-21T10:17:15.731990+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:51.977955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":297,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:16.748217+0000","last_change":"2024-09-21T10:08:00.389176+0000","last_active":"2024-09-21T10:17:16.748217+0000","last_peered":"2024-09-21T10:17:16.748217+0000","last_clean":"2024-09-21T10:17:16.748217+0000","last_became_active":"2024-09-21T10:08:00.383854+0000","last_became_peered":"2024-09-21T10:08:00.383854+0000","last_unstale":"2024-09-21T10:17:16.748217+0000","last_undegraded":"2024-09-21T10:17:16.748217+0000","last_fullsized":"2024-09-21T10:17:16.748217+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:46.952216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":311,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.910468+0000","last_change":"2024-09-21T10:07:43.792153+0000","last_active":"2024-09-21T10:17:15.910468+0000","last_peered":"2024-09-21T10:17:15.910468+0000","last_clean":"2024-09-21T10:17:15.910468+0000","last_became_active":"2024-09-21T10:07:43.791770+0000","last_became_peered":"2024-09-21T10:07:43.791770+0000","last_unstale":"2024-09-21T10:17:15.910468+0000","last_undegraded":"2024-09-21T10:17:15.910468+0000","last_fullsized":"2024-09-21T10:17:15.910468+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:52.506941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"258'8807","reported_seq":8987,"reported_epoch":261,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:17:16.748084+0000","last_change":"2024-09-21T10:16:45.836003+0000","last_active":"2024-09-21T10:17:16.748084+0000","last_peered":"2024-09-21T10:17:16.748084+0000","last_clean":"2024-09-21T10:16:03.424996+0000","last_became_active":"2024-09-21T10:16:45.790612+0000","last_became_peered":"2024-09-21T10:16:45.790612+0000","last_unstale":"2024-09-21T10:17:16.748084+0000","last_undegraded":"2024-09-21T10:17:16.748084+0000","last_fullsized":"2024-09-21T10:17:16.748084+0000","mapping_epoch":255,"log_start":"207'2680","ondisk_log_start":"207'2680","created":202,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'3209","last_scrub_stamp":"2024-09-21T10:13:42.905547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:42.905547+0000","objects_scrubbed":1742,"log_size":6127,"log_dups_size":0,"ondisk_log_size":6127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":186,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5598,"num_write_kb":123392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[0,1,7],"avail_no_missing":["0","1","3","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":316,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.729264+0000","last_change":"2024-09-21T10:07:43.791405+0000","last_active":"2024-09-21T10:17:15.729264+0000","last_peered":"2024-09-21T10:17:15.729264+0000","last_clean":"2024-09-21T10:17:15.729264+0000","last_became_active":"2024-09-21T10:07:43.791078+0000","last_became_peered":"2024-09-21T10:07:43.791078+0000","last_unstale":"2024-09-21T10:17:15.729264+0000","last_undegraded":"2024-09-21T10:17:15.729264+0000","last_fullsized":"2024-09-21T10:17:15.729264+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:17:38.072612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":978,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.731870+0000","last_change":"2024-09-21T10:08:18.468995+0000","last_active":"2024-09-21T10:17:15.731870+0000","last_peered":"2024-09-21T10:17:15.731870+0000","last_clean":"2024-09-21T10:17:15.731870+0000","last_became_active":"2024-09-21T10:08:18.468243+0000","last_became_peered":"2024-09-21T10:08:18.468243+0000","last_unstale":"2024-09-21T10:17:15.731870+0000","last_undegraded":"2024-09-21T10:17:15.731870+0000","last_fullsized":"2024-09-21T10:17:15.731870+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:45.520968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":297,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.732504+0000","last_change":"2024-09-21T10:08:18.468595+0000","last_active":"2024-09-21T10:17:15.732504+0000","last_peered":"2024-09-21T10:17:15.732504+0000","last_clean":"2024-09-21T10:17:15.732504+0000","last_became_active":"2024-09-21T10:08:18.468296+0000","last_became_peered":"2024-09-21T10:08:18.468296+0000","last_unstale":"2024-09-21T10:17:15.732504+0000","last_undegraded":"2024-09-21T10:17:15.732504+0000","last_fullsized":"2024-09-21T10:17:15.732504+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:42.748130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"258'8760","reported_seq":8931,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.731432+0000","last_change":"2024-09-21T10:13:40.091216+0000","last_active":"2024-09-21T10:17:15.731432+0000","last_peered":"2024-09-21T10:17:15.731432+0000","last_clean":"2024-09-21T10:17:15.731432+0000","last_became_active":"2024-09-21T10:13:36.519364+0000","last_became_peered":"2024-09-21T10:13:36.519364+0000","last_unstale":"2024-09-21T10:17:15.731432+0000","last_undegraded":"2024-09-21T10:17:15.731432+0000","last_fullsized":"2024-09-21T10:17:15.731432+0000","mapping_epoch":202,"log_start":"210'2864","ondisk_log_start":"210'2864","created":202,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'3004","last_scrub_stamp":"2024-09-21T10:13:40.090769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:40.090769+0000","objects_scrubbed":1593,"log_size":5896,"log_dups_size":0,"ondisk_log_size":5896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:38:37.362274+0000","scrub_duration":445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5763,"num_write_kb":133440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":273,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.731769+0000","last_change":"2024-09-21T10:08:18.469907+0000","last_active":"2024-09-21T10:17:15.731769+0000","last_peered":"2024-09-21T10:17:15.731769+0000","last_clean":"2024-09-21T10:17:15.731769+0000","last_became_active":"2024-09-21T10:08:18.469443+0000","last_became_peered":"2024-09-21T10:08:18.469443+0000","last_unstale":"2024-09-21T10:17:15.731769+0000","last_undegraded":"2024-09-21T10:17:15.731769+0000","last_fullsized":"2024-09-21T10:17:15.731769+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:15.645803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":656,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.714527+0000","last_change":"2024-09-21T10:08:00.419641+0000","last_active":"2024-09-21T10:17:11.714527+0000","last_peered":"2024-09-21T10:17:11.714527+0000","last_clean":"2024-09-21T10:17:11.714527+0000","last_became_active":"2024-09-21T10:08:00.387050+0000","last_became_peered":"2024-09-21T10:08:00.387050+0000","last_unstale":"2024-09-21T10:17:11.714527+0000","last_undegraded":"2024-09-21T10:17:11.714527+0000","last_fullsized":"2024-09-21T10:17:11.714527+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:00:32.881619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":379,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:16.748402+0000","last_change":"2024-09-21T09:59:54.267798+0000","last_active":"2024-09-21T10:17:16.748402+0000","last_peered":"2024-09-21T10:17:16.748402+0000","last_clean":"2024-09-21T10:17:16.748402+0000","last_became_active":"2024-09-21T09:59:54.265615+0000","last_became_peered":"2024-09-21T09:59:54.265615+0000","last_unstale":"2024-09-21T10:17:16.748402+0000","last_undegraded":"2024-09-21T10:17:16.748402+0000","last_fullsized":"2024-09-21T10:17:16.748402+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:30.363872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"258'8826","reported_seq":8998,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.727578+0000","last_change":"2024-09-21T10:13:40.433495+0000","last_active":"2024-09-21T10:17:15.727578+0000","last_peered":"2024-09-21T10:17:15.727578+0000","last_clean":"2024-09-21T10:17:15.727578+0000","last_became_active":"2024-09-21T10:13:36.518368+0000","last_became_peered":"2024-09-21T10:13:36.518368+0000","last_unstale":"2024-09-21T10:17:15.727578+0000","last_undegraded":"2024-09-21T10:17:15.727578+0000","last_fullsized":"2024-09-21T10:17:15.727578+0000","mapping_epoch":202,"log_start":"207'2529","ondisk_log_start":"207'2529","created":202,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'3047","last_scrub_stamp":"2024-09-21T10:13:40.433382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:40.433382+0000","objects_scrubbed":1631,"log_size":6297,"log_dups_size":0,"ondisk_log_size":6297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:29:52.573342+0000","scrub_duration":476,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5797,"num_write_kb":133312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":391,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.729625+0000","last_change":"2024-09-21T10:02:14.101943+0000","last_active":"2024-09-21T10:17:15.729625+0000","last_peered":"2024-09-21T10:17:15.729625+0000","last_clean":"2024-09-21T10:17:15.729625+0000","last_became_active":"2024-09-21T10:02:14.101488+0000","last_became_peered":"2024-09-21T10:02:14.101488+0000","last_unstale":"2024-09-21T10:17:15.729625+0000","last_undegraded":"2024-09-21T10:17:15.729625+0000","last_fullsized":"2024-09-21T10:17:15.729625+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:52:27.866414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":642,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.732425+0000","last_change":"2024-09-21T10:08:18.472876+0000","last_active":"2024-09-21T10:17:15.732425+0000","last_peered":"2024-09-21T10:17:15.732425+0000","last_clean":"2024-09-21T10:17:15.732425+0000","last_became_active":"2024-09-21T10:08:18.472385+0000","last_became_peered":"2024-09-21T10:08:18.472385+0000","last_unstale":"2024-09-21T10:17:15.732425+0000","last_undegraded":"2024-09-21T10:17:15.732425+0000","last_fullsized":"2024-09-21T10:17:15.732425+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:33.109956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":210,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.910070+0000","last_change":"2024-09-21T10:08:38.797352+0000","last_active":"2024-09-21T10:17:15.910070+0000","last_peered":"2024-09-21T10:17:15.910070+0000","last_clean":"2024-09-21T10:17:15.910070+0000","last_became_active":"2024-09-21T10:08:38.796950+0000","last_became_peered":"2024-09-21T10:08:38.796950+0000","last_unstale":"2024-09-21T10:17:15.910070+0000","last_undegraded":"2024-09-21T10:17:15.910070+0000","last_fullsized":"2024-09-21T10:17:15.910070+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:02.646321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2","version":"258'8599","reported_seq":8770,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.727536+0000","last_change":"2024-09-21T10:13:38.781080+0000","last_active":"2024-09-21T10:17:15.727536+0000","last_peered":"2024-09-21T10:17:15.727536+0000","last_clean":"2024-09-21T10:17:15.727536+0000","last_became_active":"2024-09-21T10:13:36.518804+0000","last_became_peered":"2024-09-21T10:13:36.518804+0000","last_unstale":"2024-09-21T10:17:15.727536+0000","last_undegraded":"2024-09-21T10:17:15.727536+0000","last_fullsized":"2024-09-21T10:17:15.727536+0000","mapping_epoch":202,"log_start":"207'2087","ondisk_log_start":"207'2087","created":202,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'2979","last_scrub_stamp":"2024-09-21T10:13:38.780919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:38.780919+0000","objects_scrubbed":1566,"log_size":6512,"log_dups_size":0,"ondisk_log_size":6512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:59:50.428195+0000","scrub_duration":769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5632,"num_write_kb":130112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":427,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:16.749996+0000","last_change":"2024-09-21T10:08:00.390436+0000","last_active":"2024-09-21T10:17:16.749996+0000","last_peered":"2024-09-21T10:17:16.749996+0000","last_clean":"2024-09-21T10:17:16.749996+0000","last_became_active":"2024-09-21T10:08:00.381835+0000","last_became_peered":"2024-09-21T10:08:00.381835+0000","last_unstale":"2024-09-21T10:17:16.749996+0000","last_undegraded":"2024-09-21T10:17:16.749996+0000","last_fullsized":"2024-09-21T10:17:16.749996+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:01:35.697356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":487,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.810698+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:17:11.810698+0000","last_peered":"2024-09-21T10:17:11.810698+0000","last_clean":"2024-09-21T10:17:11.810698+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:17:11.810698+0000","last_undegraded":"2024-09-21T10:17:11.810698+0000","last_fullsized":"2024-09-21T10:17:11.810698+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":125,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.810478+0000","last_change":"2024-09-21T10:08:00.385774+0000","last_active":"2024-09-21T10:17:11.810478+0000","last_peered":"2024-09-21T10:17:11.810478+0000","last_clean":"2024-09-21T10:17:11.810478+0000","last_became_active":"2024-09-21T10:08:00.385445+0000","last_became_peered":"2024-09-21T10:08:00.385445+0000","last_unstale":"2024-09-21T10:17:11.810478+0000","last_undegraded":"2024-09-21T10:17:11.810478+0000","last_fullsized":"2024-09-21T10:17:11.810478+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:42.442190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.10","version":"258'8564","reported_seq":18906,"reported_epoch":261,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T10:17:20.514598+0000","last_change":"2024-09-21T10:17:10.694697+0000","last_active":"2024-09-21T10:17:20.514598+0000","last_peered":"2024-09-21T10:17:20.514598+0000","last_clean":"2024-09-21T10:16:03.602521+0000","last_became_active":"2024-09-21T10:16:06.034220+0000","last_became_peered":"2024-09-21T10:16:06.034220+0000","last_unstale":"2024-09-21T10:17:20.514598+0000","last_undegraded":"2024-09-21T10:16:05.649875+0000","last_fullsized":"2024-09-21T10:16:05.628112+0000","mapping_epoch":247,"log_start":"207'2532","ondisk_log_start":"207'2532","created":209,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":6032,"log_dups_size":0,"ondisk_log_size":6032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":589824,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7151,"num_write_kb":229120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4260,"num_bytes_recovered":109182976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[1,2],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2","objects":-968}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":184,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.811084+0000","last_change":"2024-09-21T10:08:00.379661+0000","last_active":"2024-09-21T10:17:11.811084+0000","last_peered":"2024-09-21T10:17:11.811084+0000","last_clean":"2024-09-21T10:17:11.811084+0000","last_became_active":"2024-09-21T10:08:00.379301+0000","last_became_peered":"2024-09-21T10:08:00.379301+0000","last_unstale":"2024-09-21T10:17:11.811084+0000","last_undegraded":"2024-09-21T10:17:11.811084+0000","last_fullsized":"2024-09-21T10:17:11.811084+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:15.649523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":208,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.714070+0000","last_change":"2024-09-21T10:08:00.410731+0000","last_active":"2024-09-21T10:17:11.714070+0000","last_peered":"2024-09-21T10:17:11.714070+0000","last_clean":"2024-09-21T10:17:11.714070+0000","last_became_active":"2024-09-21T10:08:00.380324+0000","last_became_peered":"2024-09-21T10:08:00.380324+0000","last_unstale":"2024-09-21T10:17:11.714070+0000","last_undegraded":"2024-09-21T10:17:11.714070+0000","last_fullsized":"2024-09-21T10:17:11.714070+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:47:34.960221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.11","version":"258'8966","reported_seq":14439,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.713796+0000","last_change":"2024-09-21T10:14:00.826698+0000","last_active":"2024-09-21T10:17:11.713796+0000","last_peered":"2024-09-21T10:17:11.713796+0000","last_clean":"2024-09-21T10:17:11.713796+0000","last_became_active":"2024-09-21T10:14:00.823724+0000","last_became_peered":"2024-09-21T10:14:00.823724+0000","last_unstale":"2024-09-21T10:17:11.713796+0000","last_undegraded":"2024-09-21T10:17:11.713796+0000","last_fullsized":"2024-09-21T10:17:11.713796+0000","mapping_epoch":221,"log_start":"207'2241","ondisk_log_start":"207'2241","created":209,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":5,"last_scrub":"211'3233","last_scrub_stamp":"2024-09-21T10:13:42.082966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:42.082966+0000","objects_scrubbed":1774,"log_size":6725,"log_dups_size":0,"ondisk_log_size":6725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:48:10.553472+0000","scrub_duration":372,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5738,"num_write_kb":126848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1780,"num_bytes_recovered":116654080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":272,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.732045+0000","last_change":"2024-09-21T10:08:18.470378+0000","last_active":"2024-09-21T10:17:15.732045+0000","last_peered":"2024-09-21T10:17:15.732045+0000","last_clean":"2024-09-21T10:17:15.732045+0000","last_became_active":"2024-09-21T10:08:18.469482+0000","last_became_peered":"2024-09-21T10:08:18.469482+0000","last_unstale":"2024-09-21T10:17:15.732045+0000","last_undegraded":"2024-09-21T10:17:15.732045+0000","last_fullsized":"2024-09-21T10:17:15.732045+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:35.628724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":277,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.714352+0000","last_change":"2024-09-21T10:08:00.403606+0000","last_active":"2024-09-21T10:17:11.714352+0000","last_peered":"2024-09-21T10:17:11.714352+0000","last_clean":"2024-09-21T10:17:11.714352+0000","last_became_active":"2024-09-21T10:08:00.379330+0000","last_became_peered":"2024-09-21T10:08:00.379330+0000","last_unstale":"2024-09-21T10:17:11.714352+0000","last_undegraded":"2024-09-21T10:17:11.714352+0000","last_fullsized":"2024-09-21T10:17:11.714352+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:07.226027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"258'8599","reported_seq":13926,"reported_epoch":261,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:17:16.748311+0000","last_change":"2024-09-21T10:16:46.961566+0000","last_active":"2024-09-21T10:17:16.748311+0000","last_peered":"2024-09-21T10:17:16.748311+0000","last_clean":"2024-09-21T10:16:45.268576+0000","last_became_active":"2024-09-21T10:16:46.946209+0000","last_became_peered":"2024-09-21T10:16:46.946209+0000","last_unstale":"2024-09-21T10:17:16.748311+0000","last_undegraded":"2024-09-21T10:17:16.748311+0000","last_fullsized":"2024-09-21T10:17:16.748311+0000","mapping_epoch":256,"log_start":"207'2295","ondisk_log_start":"207'2295","created":209,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":5,"last_scrub":"211'3189","last_scrub_stamp":"2024-09-21T10:13:43.205779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:43.205779+0000","objects_scrubbed":1704,"log_size":6304,"log_dups_size":0,"ondisk_log_size":6304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":220,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5410,"num_write_kb":118592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1715,"num_bytes_recovered":112394240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[0,2,7],"avail_no_missing":["0","2","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":173,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.714409+0000","last_change":"2024-09-21T10:07:43.791830+0000","last_active":"2024-09-21T10:17:11.714409+0000","last_peered":"2024-09-21T10:17:11.714409+0000","last_clean":"2024-09-21T10:17:11.714409+0000","last_became_active":"2024-09-21T10:07:43.791314+0000","last_became_peered":"2024-09-21T10:07:43.791314+0000","last_unstale":"2024-09-21T10:17:11.714409+0000","last_undegraded":"2024-09-21T10:17:11.714409+0000","last_fullsized":"2024-09-21T10:17:11.714409+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:37.086629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":348,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.727495+0000","last_change":"2024-09-21T10:02:14.100451+0000","last_active":"2024-09-21T10:17:15.727495+0000","last_peered":"2024-09-21T10:17:15.727495+0000","last_clean":"2024-09-21T10:17:15.727495+0000","last_became_active":"2024-09-21T10:02:14.100160+0000","last_became_peered":"2024-09-21T10:02:14.100160+0000","last_unstale":"2024-09-21T10:17:15.727495+0000","last_undegraded":"2024-09-21T10:17:15.727495+0000","last_fullsized":"2024-09-21T10:17:15.727495+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:28.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"258'8654","reported_seq":13957,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:16.748327+0000","last_change":"2024-09-21T10:14:26.673462+0000","last_active":"2024-09-21T10:17:16.748327+0000","last_peered":"2024-09-21T10:17:16.748327+0000","last_clean":"2024-09-21T10:17:16.748327+0000","last_became_active":"2024-09-21T10:14:26.667042+0000","last_became_peered":"2024-09-21T10:14:26.667042+0000","last_unstale":"2024-09-21T10:17:16.748327+0000","last_undegraded":"2024-09-21T10:17:16.748327+0000","last_fullsized":"2024-09-21T10:17:16.748327+0000","mapping_epoch":226,"log_start":"207'2449","ondisk_log_start":"207'2449","created":209,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":6205,"log_dups_size":0,"ondisk_log_size":6205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2228224,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7222,"num_write_kb":232192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1716,"num_bytes_recovered":112459776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":389,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:16.748624+0000","last_change":"2024-09-21T10:02:14.096098+0000","last_active":"2024-09-21T10:17:16.748624+0000","last_peered":"2024-09-21T10:17:16.748624+0000","last_clean":"2024-09-21T10:17:16.748624+0000","last_became_active":"2024-09-21T10:02:14.095811+0000","last_became_peered":"2024-09-21T10:02:14.095811+0000","last_unstale":"2024-09-21T10:17:16.748624+0000","last_undegraded":"2024-09-21T10:17:16.748624+0000","last_fullsized":"2024-09-21T10:17:16.748624+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:54.619333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":393,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.727740+0000","last_change":"2024-09-21T10:02:14.100712+0000","last_active":"2024-09-21T10:17:15.727740+0000","last_peered":"2024-09-21T10:17:15.727740+0000","last_clean":"2024-09-21T10:17:15.727740+0000","last_became_active":"2024-09-21T10:02:14.100427+0000","last_became_peered":"2024-09-21T10:02:14.100427+0000","last_unstale":"2024-09-21T10:17:15.727740+0000","last_undegraded":"2024-09-21T10:17:15.727740+0000","last_fullsized":"2024-09-21T10:17:15.727740+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:16:42.350642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":224,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.810622+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:17:11.810622+0000","last_peered":"2024-09-21T10:17:11.810622+0000","last_clean":"2024-09-21T10:17:11.810622+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:17:11.810622+0000","last_undegraded":"2024-09-21T10:17:11.810622+0000","last_fullsized":"2024-09-21T10:17:11.810622+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":411,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.727812+0000","last_change":"2024-09-21T10:02:14.097369+0000","last_active":"2024-09-21T10:17:15.727812+0000","last_peered":"2024-09-21T10:17:15.727812+0000","last_clean":"2024-09-21T10:17:15.727812+0000","last_became_active":"2024-09-21T10:02:14.097067+0000","last_became_peered":"2024-09-21T10:02:14.097067+0000","last_unstale":"2024-09-21T10:17:15.727812+0000","last_undegraded":"2024-09-21T10:17:15.727812+0000","last_fullsized":"2024-09-21T10:17:15.727812+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:50:48.123813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":404,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:16.747910+0000","last_change":"2024-09-21T10:16:04.628509+0000","last_active":"2024-09-21T10:17:16.747910+0000","last_peered":"2024-09-21T10:17:16.747910+0000","last_clean":"2024-09-21T10:17:16.747910+0000","last_became_active":"2024-09-21T10:16:04.628325+0000","last_became_peered":"2024-09-21T10:16:04.628325+0000","last_unstale":"2024-09-21T10:17:16.747910+0000","last_undegraded":"2024-09-21T10:17:16.747910+0000","last_fullsized":"2024-09-21T10:17:16.747910+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:06:30.017700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":398,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.731719+0000","last_change":"2024-09-21T10:08:00.381058+0000","last_active":"2024-09-21T10:17:15.731719+0000","last_peered":"2024-09-21T10:17:15.731719+0000","last_clean":"2024-09-21T10:17:15.731719+0000","last_became_active":"2024-09-21T10:08:00.380573+0000","last_became_peered":"2024-09-21T10:08:00.380573+0000","last_unstale":"2024-09-21T10:17:15.731719+0000","last_undegraded":"2024-09-21T10:17:15.731719+0000","last_fullsized":"2024-09-21T10:17:15.731719+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:01:04.137425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":414,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.727923+0000","last_change":"2024-09-21T10:08:00.387321+0000","last_active":"2024-09-21T10:17:15.727923+0000","last_peered":"2024-09-21T10:17:15.727923+0000","last_clean":"2024-09-21T10:17:15.727923+0000","last_became_active":"2024-09-21T10:08:00.386935+0000","last_became_peered":"2024-09-21T10:08:00.386935+0000","last_unstale":"2024-09-21T10:17:15.727923+0000","last_undegraded":"2024-09-21T10:17:15.727923+0000","last_fullsized":"2024-09-21T10:17:15.727923+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:57.706313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":218,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.732444+0000","last_change":"2024-09-21T10:08:18.471282+0000","last_active":"2024-09-21T10:17:15.732444+0000","last_peered":"2024-09-21T10:17:15.732444+0000","last_clean":"2024-09-21T10:17:15.732444+0000","last_became_active":"2024-09-21T10:08:18.470952+0000","last_became_peered":"2024-09-21T10:08:18.470952+0000","last_unstale":"2024-09-21T10:17:15.732444+0000","last_undegraded":"2024-09-21T10:17:15.732444+0000","last_fullsized":"2024-09-21T10:17:15.732444+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:25.597081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":44,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.910213+0000","last_change":"2024-09-21T10:15:52.579747+0000","last_active":"2024-09-21T10:17:15.910213+0000","last_peered":"2024-09-21T10:17:15.910213+0000","last_clean":"2024-09-21T10:17:15.910213+0000","last_became_active":"2024-09-21T10:15:52.579386+0000","last_became_peered":"2024-09-21T10:15:52.579386+0000","last_unstale":"2024-09-21T10:17:15.910213+0000","last_undegraded":"2024-09-21T10:17:15.910213+0000","last_fullsized":"2024-09-21T10:17:15.910213+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:41:42.022025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":402,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:16.748285+0000","last_change":"2024-09-21T10:08:00.386104+0000","last_active":"2024-09-21T10:17:16.748285+0000","last_peered":"2024-09-21T10:17:16.748285+0000","last_clean":"2024-09-21T10:17:16.748285+0000","last_became_active":"2024-09-21T10:08:00.381484+0000","last_became_peered":"2024-09-21T10:08:00.381484+0000","last_unstale":"2024-09-21T10:17:16.748285+0000","last_undegraded":"2024-09-21T10:17:16.748285+0000","last_fullsized":"2024-09-21T10:17:16.748285+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:39.190191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"258'8760","reported_seq":8932,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.910002+0000","last_change":"2024-09-21T10:13:42.089302+0000","last_active":"2024-09-21T10:17:15.910002+0000","last_peered":"2024-09-21T10:17:15.910002+0000","last_clean":"2024-09-21T10:17:15.910002+0000","last_became_active":"2024-09-21T10:13:36.538007+0000","last_became_peered":"2024-09-21T10:13:36.538007+0000","last_unstale":"2024-09-21T10:17:15.910002+0000","last_undegraded":"2024-09-21T10:17:15.910002+0000","last_fullsized":"2024-09-21T10:17:15.910002+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":5,"last_scrub":"211'3195","last_scrub_stamp":"2024-09-21T10:13:42.089240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:13:42.089240+0000","objects_scrubbed":1718,"log_size":8760,"log_dups_size":0,"ondisk_log_size":8760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:54:58.578963+0000","scrub_duration":332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5566,"num_write_kb":123136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":328,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.727706+0000","last_change":"2024-09-21T10:08:00.381149+0000","last_active":"2024-09-21T10:17:15.727706+0000","last_peered":"2024-09-21T10:17:15.727706+0000","last_clean":"2024-09-21T10:17:15.727706+0000","last_became_active":"2024-09-21T10:08:00.380795+0000","last_became_peered":"2024-09-21T10:08:00.380795+0000","last_unstale":"2024-09-21T10:17:15.727706+0000","last_undegraded":"2024-09-21T10:17:15.727706+0000","last_fullsized":"2024-09-21T10:17:15.727706+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:50:46.793182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":229,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.732004+0000","last_change":"2024-09-21T10:08:18.466882+0000","last_active":"2024-09-21T10:17:15.732004+0000","last_peered":"2024-09-21T10:17:15.732004+0000","last_clean":"2024-09-21T10:17:15.732004+0000","last_became_active":"2024-09-21T10:08:18.466470+0000","last_became_peered":"2024-09-21T10:08:18.466470+0000","last_unstale":"2024-09-21T10:17:15.732004+0000","last_undegraded":"2024-09-21T10:17:15.732004+0000","last_fullsized":"2024-09-21T10:17:15.732004+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:51:31.020051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.19","version":"258'8654","reported_seq":8802,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:16.748067+0000","last_change":"2024-09-21T10:13:47.865419+0000","last_active":"2024-09-21T10:17:16.748067+0000","last_peered":"2024-09-21T10:17:16.748067+0000","last_clean":"2024-09-21T10:17:16.748067+0000","last_became_active":"2024-09-21T10:13:36.535264+0000","last_became_peered":"2024-09-21T10:13:36.535264+0000","last_unstale":"2024-09-21T10:17:16.748067+0000","last_undegraded":"2024-09-21T10:17:16.748067+0000","last_fullsized":"2024-09-21T10:17:16.748067+0000","mapping_epoch":209,"log_start":"207'2473","ondisk_log_start":"207'2473","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:12:59.040434+0000","last_clean_scrub_stamp":"2024-09-21T10:12:59.040434+0000","objects_scrubbed":0,"log_size":6181,"log_dups_size":0,"ondisk_log_size":6181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7231,"num_write_kb":231040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":328,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.727757+0000","last_change":"2024-09-21T10:08:00.389061+0000","last_active":"2024-09-21T10:17:15.727757+0000","last_peered":"2024-09-21T10:17:15.727757+0000","last_clean":"2024-09-21T10:17:15.727757+0000","last_became_active":"2024-09-21T10:08:00.388657+0000","last_became_peered":"2024-09-21T10:08:00.388657+0000","last_unstale":"2024-09-21T10:17:15.727757+0000","last_undegraded":"2024-09-21T10:17:15.727757+0000","last_fullsized":"2024-09-21T10:17:15.727757+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:58:17.239589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":136,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.810862+0000","last_change":"2024-09-21T10:08:00.394350+0000","last_active":"2024-09-21T10:17:11.810862+0000","last_peered":"2024-09-21T10:17:11.810862+0000","last_clean":"2024-09-21T10:17:11.810862+0000","last_became_active":"2024-09-21T10:08:00.387232+0000","last_became_peered":"2024-09-21T10:08:00.387232+0000","last_unstale":"2024-09-21T10:17:11.810862+0000","last_undegraded":"2024-09-21T10:17:11.810862+0000","last_fullsized":"2024-09-21T10:17:11.810862+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:56.160018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":414,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.727666+0000","last_change":"2024-09-21T10:02:14.099799+0000","last_active":"2024-09-21T10:17:15.727666+0000","last_peered":"2024-09-21T10:17:15.727666+0000","last_clean":"2024-09-21T10:17:15.727666+0000","last_became_active":"2024-09-21T10:02:14.099375+0000","last_became_peered":"2024-09-21T10:02:14.099375+0000","last_unstale":"2024-09-21T10:17:15.727666+0000","last_undegraded":"2024-09-21T10:17:15.727666+0000","last_fullsized":"2024-09-21T10:17:15.727666+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:13.407638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":400,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:15.729479+0000","last_change":"2024-09-21T10:02:14.099439+0000","last_active":"2024-09-21T10:17:15.729479+0000","last_peered":"2024-09-21T10:17:15.729479+0000","last_clean":"2024-09-21T10:17:15.729479+0000","last_became_active":"2024-09-21T10:02:14.099117+0000","last_became_peered":"2024-09-21T10:02:14.099117+0000","last_unstale":"2024-09-21T10:17:15.729479+0000","last_undegraded":"2024-09-21T10:17:15.729479+0000","last_fullsized":"2024-09-21T10:17:15.729479+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:39.681817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":199,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-21T10:17:11.810248+0000","last_change":"2024-09-21T10:08:00.381529+0000","last_active":"2024-09-21T10:17:11.810248+0000","last_peered":"2024-09-21T10:17:11.810248+0000","last_clean":"2024-09-21T10:17:11.810248+0000","last_became_active":"2024-09-21T10:08:00.381061+0000","last_became_peered":"2024-09-21T10:08:00.381061+0000","last_unstale":"2024-09-21T10:17:11.810248+0000","last_undegraded":"2024-09-21T10:17:11.810248+0000","last_fullsized":"2024-09-21T10:17:11.810248+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:56:58.775586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":394,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-21T10:17:16.748621+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:17:16.748621+0000","last_peered":"2024-09-21T10:17:16.748621+0000","last_clean":"2024-09-21T10:17:16.748621+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:17:16.748621+0000","last_undegraded":"2024-09-21T10:17:16.748621+0000","last_fullsized":"2024-09-21T10:17:16.748621+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":5373952,"num_objects":82,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201672,"num_write_kb":5700225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12679,"num_bytes_recovered":660930560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1022164992,"data_stored":1022164992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":174813,"ondisk_log_size":174813,"up":78,"acting":75,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":9926,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":21480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5982,"num_read_kb":13005,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397830,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":411512,"kb_used_data":105836,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93333640,"statfs":{"total":95995035648,"available":95573647360,"internally_reserved":0,"allocated":108376064,"data_stored":106993556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594058,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":413108,"kb_used_data":107432,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93332044,"statfs":{"total":95995035648,"available":95572013056,"internally_reserved":0,"allocated":110010368,"data_stored":108631863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790288,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":619296,"kb_used_data":313620,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93125856,"statfs":{"total":95995035648,"available":95360876544,"internally_reserved":0,"allocated":321146880,"data_stored":319783788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986517,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":101,"seq":433791697079,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":535640,"kb_used_data":3216,"kb_used_omap":5,"kb_used_meta":532410,"kb_avail":93209512,"statfs":{"total":95995035648,"available":95446540288,"internally_reserved":0,"allocated":3293184,"data_stored":1933946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":545188624},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860606,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":557932,"kb_used_data":7328,"kb_used_omap":8,"kb_used_meta":550583,"kb_avail":93187220,"statfs":{"total":95995035648,"available":95423713280,"internally_reserved":0,"allocated":7503872,"data_stored":6133516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":563797868},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024133,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":613656,"kb_used_data":208524,"kb_used_omap":7,"kb_used_meta":405112,"kb_avail":93131496,"statfs":{"total":95995035648,"available":95366651904,"internally_reserved":0,"allocated":213528576,"data_stored":212172763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":414835250},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187667,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":837608,"kb_used_data":288860,"kb_used_omap":8,"kb_used_meta":548727,"kb_avail":92907544,"statfs":{"total":95995035648,"available":95137325056,"internally_reserved":0,"allocated":295792640,"data_stored":294422258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":561897457},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":291897344,"data_stored":291897344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":210239488,"data_stored":210239488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":313655296,"data_stored":313655296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":106102784,"data_stored":106102784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":100270080,"data_stored":100270080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:17:24.641 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:17:24.643 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:24.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:24.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:24.745 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:24.771 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:24.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:24.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:24.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:24.897 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:24 smithi098 ceph-mon[100914]: pgmap v805: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 9.7 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 8166/330 objects degraded (2474.545%) 2024-09-21T10:17:24.897 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:24 smithi098 ceph-mon[100914]: osdmap e262: 8 total, 8 up, 7 in 2024-09-21T10:17:24.898 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:24 smithi098 ceph-mon[97580]: pgmap v805: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 9.7 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 8166/330 objects degraded (2474.545%) 2024-09-21T10:17:24.898 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:24 smithi098 ceph-mon[97580]: osdmap e262: 8 total, 8 up, 7 in 2024-09-21T10:17:24.902 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:24.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:24.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:25.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:25.028 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:25.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:25.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:25.130 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:25.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:24 smithi196 ceph-mon[84784]: pgmap v805: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 9.7 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 8166/330 objects degraded (2474.545%) 2024-09-21T10:17:25.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:24 smithi196 ceph-mon[84784]: osdmap e262: 8 total, 8 up, 7 in 2024-09-21T10:17:25.156 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:25.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:25.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:25.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:25.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:25.281 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:25.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:25.382 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:25.408 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:25.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:25.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:25.510 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:25.579 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:25.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:25.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:25.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:25.706 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:25.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:25.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:25.807 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:25.833 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:25.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:25.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:25.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:17:25.934 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:25.996 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:25.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:25.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:26.025 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:25 smithi098 ceph-mon[97580]: osdmap e263: 8 total, 8 up, 7 in 2024-09-21T10:17:26.025 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:25 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 8166/330 objects degraded (2474.545%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:17:26.026 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:25 smithi098 ceph-mon[100914]: osdmap e263: 8 total, 8 up, 7 in 2024-09-21T10:17:26.026 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:25 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 8166/330 objects degraded (2474.545%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:17:26.044 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:17:26.044 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-21T10:17:26.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-21T10:17:26.044 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:17:26.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.044 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:55911 7.12 7:4baf3ad5:::benchmark_data_smithi098_197834_object55910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e216)", 2024-09-21T10:17:26.044 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:46.682951+0000", 2024-09-21T10:17:26.044 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 219.35701047699999, 2024-09-21T10:17:26.045 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1336062870000001, 2024-09-21T10:17:26.045 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:17:26.045 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:17:26.045 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:26.045 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:17:26.045 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:17:26.045 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:17:26.045 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 55911 2024-09-21T10:17:26.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.045 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:17:26.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.045 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:17:26.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.682951+0000", 2024-09-21T10:17:26.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:17:26.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.682951+0000", 2024-09-21T10:17:26.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:17:26.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.682952+0000", 2024-09-21T10:17:26.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6600000000000002e-07 2024-09-21T10:17:26.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:17:26.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.682982+0000", 2024-09-21T10:17:26.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0660000000000001e-05 2024-09-21T10:17:26.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:17:26.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.682990+0000", 2024-09-21T10:17:26.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0199999999999994e-06 2024-09-21T10:17:26.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:17:26.048 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.682996+0000", 2024-09-21T10:17:26.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7819999999999999e-06 2024-09-21T10:17:26.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.048 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683568+0000", 2024-09-21T10:17:26.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00057152199999999998 2024-09-21T10:17:26.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:17:26.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683593+0000", 2024-09-21T10:17:26.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4927999999999999e-05 2024-09-21T10:17:26.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.811635+0000", 2024-09-21T10:17:26.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1280425940000001 2024-09-21T10:17:26.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:17:26.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.811781+0000", 2024-09-21T10:17:26.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014603399999999999 2024-09-21T10:17:26.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,2,7", 2024-09-21T10:17:26.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.811942+0000", 2024-09-21T10:17:26.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016044900000000001 2024-09-21T10:17:26.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.815289+0000", 2024-09-21T10:17:26.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033467589999999999 2024-09-21T10:17:26.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.815337+0000", 2024-09-21T10:17:26.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8523000000000003e-05 2024-09-21T10:17:26.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.051 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.815412+0000", 2024-09-21T10:17:26.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4906000000000004e-05 2024-09-21T10:17:26.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.051 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:17:26.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.815428+0000", 2024-09-21T10:17:26.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6243e-05 2024-09-21T10:17:26.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.052 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:17:26.052 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.815488+0000", 2024-09-21T10:17:26.052 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9803000000000003e-05 2024-09-21T10:17:26.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.052 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:17:26.052 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.816558+0000", 2024-09-21T10:17:26.052 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010695979999999999 2024-09-21T10:17:26.052 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.052 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:17:26.052 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.052 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:55942 7.12 7:4d78df3c:::benchmark_data_smithi098_197834_object55941:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e216)", 2024-09-21T10:17:26.053 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:46.683007+0000", 2024-09-21T10:17:26.053 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 219.356954301, 2024-09-21T10:17:26.053 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.138718578, 2024-09-21T10:17:26.053 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:17:26.053 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:17:26.053 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:26.053 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:17:26.053 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:17:26.053 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:17:26.053 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 55942 2024-09-21T10:17:26.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.053 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:17:26.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.054 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:17:26.054 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683007+0000", 2024-09-21T10:17:26.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.054 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:17:26.054 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683007+0000", 2024-09-21T10:17:26.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.054 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:17:26.054 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683008+0000", 2024-09-21T10:17:26.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.82e-07 2024-09-21T10:17:26.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:17:26.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683044+0000", 2024-09-21T10:17:26.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5522000000000003e-05 2024-09-21T10:17:26.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:17:26.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683051+0000", 2024-09-21T10:17:26.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6599999999999995e-06 2024-09-21T10:17:26.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:17:26.056 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683059+0000", 2024-09-21T10:17:26.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8739999999999995e-06 2024-09-21T10:17:26.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.056 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.056 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683617+0000", 2024-09-21T10:17:26.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00055830400000000003 2024-09-21T10:17:26.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.056 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:17:26.056 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683624+0000", 2024-09-21T10:17:26.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8639999999999998e-06 2024-09-21T10:17:26.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.057 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.057 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.814033+0000", 2024-09-21T10:17:26.057 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.130409182 2024-09-21T10:17:26.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.057 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:17:26.057 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.814144+0000", 2024-09-21T10:17:26.057 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000111016 2024-09-21T10:17:26.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.057 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,2,7", 2024-09-21T10:17:26.057 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.814303+0000", 2024-09-21T10:17:26.058 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000158658 2024-09-21T10:17:26.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.058 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.058 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.817891+0000", 2024-09-21T10:17:26.058 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035880180000000001 2024-09-21T10:17:26.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.058 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.058 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.817931+0000", 2024-09-21T10:17:26.058 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0247000000000003e-05 2024-09-21T10:17:26.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.059 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.059 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.817980+0000", 2024-09-21T10:17:26.059 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8877e-05 2024-09-21T10:17:26.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.060 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:17:26.060 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.820549+0000", 2024-09-21T10:17:26.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025688709999999999 2024-09-21T10:17:26.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.060 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:17:26.060 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.820602+0000", 2024-09-21T10:17:26.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2955999999999999e-05 2024-09-21T10:17:26.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.060 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:17:26.060 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.821726+0000", 2024-09-21T10:17:26.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001124047 2024-09-21T10:17:26.061 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.061 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:17:26.061 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.061 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56000 7.12 7:4c9ff66d:::benchmark_data_smithi098_197834_object55999:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e216)", 2024-09-21T10:17:26.061 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:46.683069+0000", 2024-09-21T10:17:26.061 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 219.35689272900001, 2024-09-21T10:17:26.061 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.14174486, 2024-09-21T10:17:26.061 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:17:26.061 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:17:26.061 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:26.061 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:17:26.062 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:17:26.062 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:17:26.062 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56000 2024-09-21T10:17:26.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.062 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:17:26.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.063 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:17:26.063 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683069+0000", 2024-09-21T10:17:26.063 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.063 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:17:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683069+0000", 2024-09-21T10:17:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:17:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683070+0000", 2024-09-21T10:17:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9900000000000001e-07 2024-09-21T10:17:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:17:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683098+0000", 2024-09-21T10:17:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8010000000000001e-05 2024-09-21T10:17:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:17:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683473+0000", 2024-09-21T10:17:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037528399999999998 2024-09-21T10:17:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:17:26.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683484+0000", 2024-09-21T10:17:26.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1134999999999999e-05 2024-09-21T10:17:26.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.066 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683646+0000", 2024-09-21T10:17:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000161803 2024-09-21T10:17:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:17:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683652+0000", 2024-09-21T10:17:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.285e-06 2024-09-21T10:17:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.816611+0000", 2024-09-21T10:17:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1329588850000001 2024-09-21T10:17:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:17:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.816749+0000", 2024-09-21T10:17:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000137634 2024-09-21T10:17:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,2,7", 2024-09-21T10:17:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.816908+0000", 2024-09-21T10:17:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015897200000000001 2024-09-21T10:17:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:17:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.823573+0000", 2024-09-21T10:17:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0066655170000000001 2024-09-21T10:17:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.823627+0000", 2024-09-21T10:17:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4073000000000002e-05 2024-09-21T10:17:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.823678+0000", 2024-09-21T10:17:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0606999999999998e-05 2024-09-21T10:17:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.823761+0000", 2024-09-21T10:17:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3055999999999999e-05 2024-09-21T10:17:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:17:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.823823+0000", 2024-09-21T10:17:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2609999999999999e-05 2024-09-21T10:17:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:17:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.824814+0000", 2024-09-21T10:17:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0009904899999999999 2024-09-21T10:17:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:17:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56021 7.12 7:48079405:::benchmark_data_smithi098_197834_object56020:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e216)", 2024-09-21T10:17:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:46.683497+0000", 2024-09-21T10:17:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 219.35646485300001, 2024-09-21T10:17:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.142497563, 2024-09-21T10:17:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:17:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:17:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:17:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:17:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:17:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56021 2024-09-21T10:17:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:17:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:17:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683497+0000", 2024-09-21T10:17:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:17:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683497+0000", 2024-09-21T10:17:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:17:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683498+0000", 2024-09-21T10:17:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6499999999999998e-07 2024-09-21T10:17:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:17:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683845+0000", 2024-09-21T10:17:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00034733300000000002 2024-09-21T10:17:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:17:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683856+0000", 2024-09-21T10:17:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0698e-05 2024-09-21T10:17:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:17:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683864+0000", 2024-09-21T10:17:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7880000000000007e-06 2024-09-21T10:17:26.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.075 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.075 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683914+0000", 2024-09-21T10:17:26.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0831000000000002e-05 2024-09-21T10:17:26.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.075 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:17:26.075 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683923+0000", 2024-09-21T10:17:26.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6370000000000005e-06 2024-09-21T10:17:26.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.075 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.818008+0000", 2024-09-21T10:17:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1340853660000001 2024-09-21T10:17:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:17:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.818171+0000", 2024-09-21T10:17:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016250600000000001 2024-09-21T10:17:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,2,7", 2024-09-21T10:17:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.818364+0000", 2024-09-21T10:17:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019306299999999999 2024-09-21T10:17:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:17:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.823581+0000", 2024-09-21T10:17:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052169900000000003 2024-09-21T10:17:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.823718+0000", 2024-09-21T10:17:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013740900000000001 2024-09-21T10:17:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.824876+0000", 2024-09-21T10:17:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011575489999999999 2024-09-21T10:17:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.824983+0000", 2024-09-21T10:17:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106717 2024-09-21T10:17:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:17:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.825034+0000", 2024-09-21T10:17:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1637e-05 2024-09-21T10:17:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:17:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.825995+0000", 2024-09-21T10:17:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00096017400000000005 2024-09-21T10:17:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:17:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56026 7.12 7:4ab16f7b:::benchmark_data_smithi098_197834_object56025:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e216)", 2024-09-21T10:17:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:46.683881+0000", 2024-09-21T10:17:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 219.35608058599999, 2024-09-21T10:17:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1488375770000001, 2024-09-21T10:17:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:17:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:17:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:17:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:17:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:17:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56026 2024-09-21T10:17:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:17:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:17:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683881+0000", 2024-09-21T10:17:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:17:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683881+0000", 2024-09-21T10:17:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:17:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683882+0000", 2024-09-21T10:17:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.37e-07 2024-09-21T10:17:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:17:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683903+0000", 2024-09-21T10:17:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0897999999999999e-05 2024-09-21T10:17:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:17:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683911+0000", 2024-09-21T10:17:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3100000000000001e-06 2024-09-21T10:17:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:17:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683917+0000", 2024-09-21T10:17:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.015e-06 2024-09-21T10:17:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683947+0000", 2024-09-21T10:17:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9377999999999999e-05 2024-09-21T10:17:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:17:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683954+0000", 2024-09-21T10:17:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9569999999999996e-06 2024-09-21T10:17:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.819273+0000", 2024-09-21T10:17:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.135319789 2024-09-21T10:17:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:17:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.819410+0000", 2024-09-21T10:17:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013691300000000001 2024-09-21T10:17:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,2,7", 2024-09-21T10:17:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.819582+0000", 2024-09-21T10:17:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017130499999999999 2024-09-21T10:17:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:17:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.823588+0000", 2024-09-21T10:17:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040066779999999996 2024-09-21T10:17:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.824927+0000", 2024-09-21T10:17:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013390159999999999 2024-09-21T10:17:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.826046+0000", 2024-09-21T10:17:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001119232 2024-09-21T10:17:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.831404+0000", 2024-09-21T10:17:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053579430000000004 2024-09-21T10:17:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:17:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.831460+0000", 2024-09-21T10:17:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5345e-05 2024-09-21T10:17:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:17:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.832719+0000", 2024-09-21T10:17:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0012590609999999999 2024-09-21T10:17:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:17:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56052 7.12 7:4e34fde1:::benchmark_data_smithi098_197834_object56051:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e216)", 2024-09-21T10:17:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:46.683926+0000", 2024-09-21T10:17:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 219.35603629799999, 2024-09-21T10:17:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1505354080000001, 2024-09-21T10:17:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:17:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:17:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:17:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:17:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:17:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56052 2024-09-21T10:17:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:17:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:17:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683926+0000", 2024-09-21T10:17:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:17:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683926+0000", 2024-09-21T10:17:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:17:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683926+0000", 2024-09-21T10:17:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4099999999999999e-07 2024-09-21T10:17:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:17:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683944+0000", 2024-09-21T10:17:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7546000000000001e-05 2024-09-21T10:17:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:17:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683948+0000", 2024-09-21T10:17:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1119999999999998e-06 2024-09-21T10:17:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:17:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683951+0000", 2024-09-21T10:17:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9689999999999999e-06 2024-09-21T10:17:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683975+0000", 2024-09-21T10:17:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.446e-05 2024-09-21T10:17:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:17:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683982+0000", 2024-09-21T10:17:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7510000000000002e-06 2024-09-21T10:17:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.821767+0000", 2024-09-21T10:17:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1377853950000001 2024-09-21T10:17:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:17:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.821909+0000", 2024-09-21T10:17:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014206599999999999 2024-09-21T10:17:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,2,7", 2024-09-21T10:17:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.822161+0000", 2024-09-21T10:17:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00025163699999999998 2024-09-21T10:17:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:17:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.823640+0000", 2024-09-21T10:17:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001479099 2024-09-21T10:17:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.826112+0000", 2024-09-21T10:17:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024723779999999999 2024-09-21T10:17:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.826175+0000", 2024-09-21T10:17:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2310000000000005e-05 2024-09-21T10:17:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.832783+0000", 2024-09-21T10:17:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0066086299999999999 2024-09-21T10:17:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:17:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.832845+0000", 2024-09-21T10:17:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1750999999999999e-05 2024-09-21T10:17:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:17:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.834461+0000", 2024-09-21T10:17:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016157630000000001 2024-09-21T10:17:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:17:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56056 7.12 7:4825925f:::benchmark_data_smithi098_197834_object56055:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e216)", 2024-09-21T10:17:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:46.683955+0000", 2024-09-21T10:17:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 219.356006877, 2024-09-21T10:17:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.154793937, 2024-09-21T10:17:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:17:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:17:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:17:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:17:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:17:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56056 2024-09-21T10:17:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:17:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:17:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683955+0000", 2024-09-21T10:17:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:17:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683955+0000", 2024-09-21T10:17:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:17:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683955+0000", 2024-09-21T10:17:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8500000000000002e-07 2024-09-21T10:17:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:17:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683972+0000", 2024-09-21T10:17:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7084000000000001e-05 2024-09-21T10:17:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:17:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683979+0000", 2024-09-21T10:17:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8859999999999998e-06 2024-09-21T10:17:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:17:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683986+0000", 2024-09-21T10:17:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7599999999999997e-06 2024-09-21T10:17:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684032+0000", 2024-09-21T10:17:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5825000000000003e-05 2024-09-21T10:17:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:17:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684040+0000", 2024-09-21T10:17:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9470000000000003e-06 2024-09-21T10:17:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.826221+0000", 2024-09-21T10:17:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.142181391 2024-09-21T10:17:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:17:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.826530+0000", 2024-09-21T10:17:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00030879900000000001 2024-09-21T10:17:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,2,7", 2024-09-21T10:17:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.827101+0000", 2024-09-21T10:17:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00057068600000000004 2024-09-21T10:17:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.831358+0000", 2024-09-21T10:17:26.100 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0042571299999999996 2024-09-21T10:17:26.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.100 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.100 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.834514+0000", 2024-09-21T10:17:26.100 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003156491 2024-09-21T10:17:26.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.101 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.834558+0000", 2024-09-21T10:17:26.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3667000000000001e-05 2024-09-21T10:17:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:17:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.837525+0000", 2024-09-21T10:17:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029666860000000001 2024-09-21T10:17:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:17:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.837577+0000", 2024-09-21T10:17:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2751999999999999e-05 2024-09-21T10:17:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:17:26.103 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.838749+0000", 2024-09-21T10:17:26.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011715479999999999 2024-09-21T10:17:26.103 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.103 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:17:26.103 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.103 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56072 7.12 7:4e7822f0:::benchmark_data_smithi098_197834_object56071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e216)", 2024-09-21T10:17:26.103 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:46.683997+0000", 2024-09-21T10:17:26.103 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 219.355964363, 2024-09-21T10:17:26.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.157511188, 2024-09-21T10:17:26.103 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:17:26.103 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:17:26.103 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:26.104 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:17:26.104 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:17:26.104 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:17:26.104 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56072 2024-09-21T10:17:26.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.104 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:17:26.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.104 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:17:26.104 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683997+0000", 2024-09-21T10:17:26.104 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.104 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:17:26.104 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683997+0000", 2024-09-21T10:17:26.104 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.105 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:17:26.105 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.683998+0000", 2024-09-21T10:17:26.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6100000000000001e-07 2024-09-21T10:17:26.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.105 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:26.106 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:17:26.106 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684022+0000", 2024-09-21T10:17:26.106 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3674999999999999e-05 2024-09-21T10:17:26.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.106 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:17:26.106 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684028+0000", 2024-09-21T10:17:26.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.866e-06 2024-09-21T10:17:26.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.107 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:17:26.107 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684033+0000", 2024-09-21T10:17:26.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7870000000000001e-06 2024-09-21T10:17:26.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.107 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.107 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684067+0000", 2024-09-21T10:17:26.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4273999999999997e-05 2024-09-21T10:17:26.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:17:26.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684074+0000", 2024-09-21T10:17:26.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4819999999999999e-06 2024-09-21T10:17:26.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.830118+0000", 2024-09-21T10:17:26.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.146043777 2024-09-21T10:17:26.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:17:26.109 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.830254+0000", 2024-09-21T10:17:26.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000136048 2024-09-21T10:17:26.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.109 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,2,7", 2024-09-21T10:17:26.109 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.830404+0000", 2024-09-21T10:17:26.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014972299999999999 2024-09-21T10:17:26.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.109 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.109 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.834602+0000", 2024-09-21T10:17:26.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0041983799999999998 2024-09-21T10:17:26.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.110 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.110 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.837509+0000", 2024-09-21T10:17:26.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029074999999999999 2024-09-21T10:17:26.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.110 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:17:26.110 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.838766+0000", 2024-09-21T10:17:26.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001256343 2024-09-21T10:17:26.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.111 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.840382+0000", 2024-09-21T10:17:26.111 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016159939999999999 2024-09-21T10:17:26.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:17:26.111 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.840426+0000", 2024-09-21T10:17:26.111 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4561000000000002e-05 2024-09-21T10:17:26.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:17:26.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.841509+0000", 2024-09-21T10:17:26.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001082317 2024-09-21T10:17:26.112 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.112 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:17:26.112 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.112 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56089 7.12 7:48948ce1:::benchmark_data_smithi098_197834_object56088:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e216)", 2024-09-21T10:17:26.112 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:46.684043+0000", 2024-09-21T10:17:26.112 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 219.35591908999999, 2024-09-21T10:17:26.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.16994569, 2024-09-21T10:17:26.112 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:17:26.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:17:26.112 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:26.113 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:17:26.113 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:17:26.113 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:17:26.113 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56089 2024-09-21T10:17:26.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.113 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:17:26.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.113 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:17:26.113 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684043+0000", 2024-09-21T10:17:26.113 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.113 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:17:26.113 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684043+0000", 2024-09-21T10:17:26.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.114 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:17:26.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684043+0000", 2024-09-21T10:17:26.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6899999999999998e-07 2024-09-21T10:17:26.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.114 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:17:26.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684064+0000", 2024-09-21T10:17:26.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0899000000000001e-05 2024-09-21T10:17:26.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.114 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:17:26.115 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684071+0000", 2024-09-21T10:17:26.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4289999999999998e-06 2024-09-21T10:17:26.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.115 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:17:26.115 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684076+0000", 2024-09-21T10:17:26.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1540000000000001e-06 2024-09-21T10:17:26.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.115 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.115 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684131+0000", 2024-09-21T10:17:26.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5053999999999998e-05 2024-09-21T10:17:26.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:17:26.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684139+0000", 2024-09-21T10:17:26.116 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4999999999999999e-06 2024-09-21T10:17:26.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.834635+0000", 2024-09-21T10:17:26.116 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.150495582 2024-09-21T10:17:26.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:17:26.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.834778+0000", 2024-09-21T10:17:26.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014284000000000001 2024-09-21T10:17:26.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,2,7", 2024-09-21T10:17:26.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.835087+0000", 2024-09-21T10:17:26.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00030898500000000001 2024-09-21T10:17:26.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.840340+0000", 2024-09-21T10:17:26.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052529109999999999 2024-09-21T10:17:26.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.841622+0000", 2024-09-21T10:17:26.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001282212 2024-09-21T10:17:26.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:17:26.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.844632+0000", 2024-09-21T10:17:26.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030107440000000001 2024-09-21T10:17:26.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.852620+0000", 2024-09-21T10:17:26.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0079876679999999999 2024-09-21T10:17:26.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.119 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:17:26.119 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.852678+0000", 2024-09-21T10:17:26.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7386000000000002e-05 2024-09-21T10:17:26.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.119 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:17:26.119 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.853988+0000", 2024-09-21T10:17:26.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013108569999999999 2024-09-21T10:17:26.119 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.119 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:17:26.119 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.120 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56107 7.12 7:4fda650d:::benchmark_data_smithi098_197834_object56106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e216)", 2024-09-21T10:17:26.120 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:46.684085+0000", 2024-09-21T10:17:26.120 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 219.35587727800001, 2024-09-21T10:17:26.120 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1737462409999999, 2024-09-21T10:17:26.120 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:17:26.120 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:17:26.120 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:26.120 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:17:26.120 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:17:26.120 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:17:26.120 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56107 2024-09-21T10:17:26.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.121 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:17:26.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:17:26.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684085+0000", 2024-09-21T10:17:26.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:17:26.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684085+0000", 2024-09-21T10:17:26.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:17:26.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684085+0000", 2024-09-21T10:17:26.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0800000000000005e-07 2024-09-21T10:17:26.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:17:26.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684113+0000", 2024-09-21T10:17:26.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.764e-05 2024-09-21T10:17:26.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:17:26.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684120+0000", 2024-09-21T10:17:26.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2599999999999999e-06 2024-09-21T10:17:26.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:17:26.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684125+0000", 2024-09-21T10:17:26.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5670000000000002e-06 2024-09-21T10:17:26.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684165+0000", 2024-09-21T10:17:26.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9020000000000002e-05 2024-09-21T10:17:26.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:17:26.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684173+0000", 2024-09-21T10:17:26.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1489999999999994e-06 2024-09-21T10:17:26.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.125 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.836173+0000", 2024-09-21T10:17:26.125 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.152000133 2024-09-21T10:17:26.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:17:26.125 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.836329+0000", 2024-09-21T10:17:26.125 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015607699999999999 2024-09-21T10:17:26.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,2,7", 2024-09-21T10:17:26.125 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.836487+0000", 2024-09-21T10:17:26.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015813900000000001 2024-09-21T10:17:26.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.126 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.126 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.841560+0000", 2024-09-21T10:17:26.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050729049999999999 2024-09-21T10:17:26.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.126 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.126 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.841723+0000", 2024-09-21T10:17:26.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000162795 2024-09-21T10:17:26.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.127 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:17:26.127 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.844639+0000", 2024-09-21T10:17:26.127 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029163930000000002 2024-09-21T10:17:26.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.127 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.127 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.856472+0000", 2024-09-21T10:17:26.127 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011832743 2024-09-21T10:17:26.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.127 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:17:26.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.856531+0000", 2024-09-21T10:17:26.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9324999999999999e-05 2024-09-21T10:17:26.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:17:26.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.857831+0000", 2024-09-21T10:17:26.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0012995870000000001 2024-09-21T10:17:26.128 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.128 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:17:26.128 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.128 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56160 7.12 7:49bec15a:::benchmark_data_smithi098_197834_object56159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e216)", 2024-09-21T10:17:26.129 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:46.684135+0000", 2024-09-21T10:17:26.129 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 219.35582658000001, 2024-09-21T10:17:26.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1747192499999999, 2024-09-21T10:17:26.129 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:17:26.129 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:17:26.129 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:26.129 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:17:26.129 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:17:26.129 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:17:26.129 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56160 2024-09-21T10:17:26.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.129 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:17:26.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:17:26.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684135+0000", 2024-09-21T10:17:26.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:17:26.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684135+0000", 2024-09-21T10:17:26.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:17:26.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684136+0000", 2024-09-21T10:17:26.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7199999999999999e-07 2024-09-21T10:17:26.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.131 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:17:26.131 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684186+0000", 2024-09-21T10:17:26.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9942999999999999e-05 2024-09-21T10:17:26.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.131 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:17:26.131 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684194+0000", 2024-09-21T10:17:26.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4409999999999993e-06 2024-09-21T10:17:26.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.131 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:17:26.131 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684200+0000", 2024-09-21T10:17:26.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5979999999999996e-06 2024-09-21T10:17:26.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.132 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.132 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684238+0000", 2024-09-21T10:17:26.133 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8705999999999998e-05 2024-09-21T10:17:26.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.133 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:17:26.133 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684246+0000", 2024-09-21T10:17:26.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.537e-06 2024-09-21T10:17:26.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.134 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.134 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.838818+0000", 2024-09-21T10:17:26.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.154572218 2024-09-21T10:17:26.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.134 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:17:26.134 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.838987+0000", 2024-09-21T10:17:26.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016894499999999999 2024-09-21T10:17:26.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.135 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,2,7", 2024-09-21T10:17:26.135 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.839211+0000", 2024-09-21T10:17:26.135 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022417100000000001 2024-09-21T10:17:26.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.135 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.135 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.841676+0000", 2024-09-21T10:17:26.135 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024643429999999999 2024-09-21T10:17:26.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.135 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.135 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.844683+0000", 2024-09-21T10:17:26.135 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030076360000000002 2024-09-21T10:17:26.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.136 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:17:26.136 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.844698+0000", 2024-09-21T10:17:26.136 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.523e-05 2024-09-21T10:17:26.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.136 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.136 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.857883+0000", 2024-09-21T10:17:26.136 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01318478 2024-09-21T10:17:26.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.136 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:17:26.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.857942+0000", 2024-09-21T10:17:26.137 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8629999999999999e-05 2024-09-21T10:17:26.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.137 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:17:26.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.858854+0000", 2024-09-21T10:17:26.137 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:26.138 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00091259999999999996 2024-09-21T10:17:26.138 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.138 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:17:26.138 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.139 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56198 7.12 7:4bbf861e:::benchmark_data_smithi098_197834_object56197:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e216)", 2024-09-21T10:17:26.139 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:46.684210+0000", 2024-09-21T10:17:26.139 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 219.35575180000001, 2024-09-21T10:17:26.139 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1755239799999999, 2024-09-21T10:17:26.139 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:17:26.139 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:17:26.139 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:26.139 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:17:26.139 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:17:26.139 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:17:26.139 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56198 2024-09-21T10:17:26.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.140 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:17:26.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.140 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:17:26.140 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684210+0000", 2024-09-21T10:17:26.140 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.140 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:17:26.140 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684210+0000", 2024-09-21T10:17:26.140 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.140 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:17:26.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684211+0000", 2024-09-21T10:17:26.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5700000000000002e-07 2024-09-21T10:17:26.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.141 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:17:26.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684254+0000", 2024-09-21T10:17:26.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3600999999999998e-05 2024-09-21T10:17:26.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.141 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:17:26.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684262+0000", 2024-09-21T10:17:26.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4070000000000004e-06 2024-09-21T10:17:26.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:17:26.142 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684267+0000", 2024-09-21T10:17:26.142 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8340000000000004e-06 2024-09-21T10:17:26.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.142 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684310+0000", 2024-09-21T10:17:26.142 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2338e-05 2024-09-21T10:17:26.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.143 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:17:26.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684318+0000", 2024-09-21T10:17:26.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9130000000000003e-06 2024-09-21T10:17:26.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.143 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.841753+0000", 2024-09-21T10:17:26.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.157435024 2024-09-21T10:17:26.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.143 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:17:26.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.841933+0000", 2024-09-21T10:17:26.144 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018057699999999999 2024-09-21T10:17:26.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.144 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,2,7", 2024-09-21T10:17:26.144 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.842477+0000", 2024-09-21T10:17:26.144 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00054410600000000004 2024-09-21T10:17:26.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.144 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:17:26.144 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.844704+0000", 2024-09-21T10:17:26.144 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022268840000000002 2024-09-21T10:17:26.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.144 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.144 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.845042+0000", 2024-09-21T10:17:26.145 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000337743 2024-09-21T10:17:26.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.145 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.145 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.846360+0000", 2024-09-21T10:17:26.145 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001317546 2024-09-21T10:17:26.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.145 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.145 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.858920+0000", 2024-09-21T10:17:26.145 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012560907 2024-09-21T10:17:26.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.145 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:17:26.146 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.858982+0000", 2024-09-21T10:17:26.146 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1501000000000006e-05 2024-09-21T10:17:26.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.146 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:17:26.146 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.859734+0000", 2024-09-21T10:17:26.146 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00075204199999999997 2024-09-21T10:17:26.146 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.146 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:17:26.146 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.147 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56225 7.12 7:4f909546:::benchmark_data_smithi098_197834_object56224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e216)", 2024-09-21T10:17:26.147 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:46.684279+0000", 2024-09-21T10:17:26.147 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 219.355682422, 2024-09-21T10:17:26.147 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1770254959999999, 2024-09-21T10:17:26.147 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:17:26.147 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:17:26.147 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:26.147 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:17:26.147 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:17:26.147 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:17:26.147 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56225 2024-09-21T10:17:26.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.147 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:17:26.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:17:26.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684279+0000", 2024-09-21T10:17:26.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:17:26.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684279+0000", 2024-09-21T10:17:26.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:17:26.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684280+0000", 2024-09-21T10:17:26.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7599999999999997e-07 2024-09-21T10:17:26.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.149 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:17:26.149 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684322+0000", 2024-09-21T10:17:26.149 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2503000000000003e-05 2024-09-21T10:17:26.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.149 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:17:26.149 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684330+0000", 2024-09-21T10:17:26.149 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6760000000000007e-06 2024-09-21T10:17:26.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.150 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:17:26.150 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684335+0000", 2024-09-21T10:17:26.150 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3040000000000004e-06 2024-09-21T10:17:26.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.150 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.150 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684374+0000", 2024-09-21T10:17:26.150 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8868000000000001e-05 2024-09-21T10:17:26.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.150 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:17:26.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.684381+0000", 2024-09-21T10:17:26.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8170000000000001e-06 2024-09-21T10:17:26.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.151 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.843508+0000", 2024-09-21T10:17:26.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.159127223 2024-09-21T10:17:26.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.151 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:17:26.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.843644+0000", 2024-09-21T10:17:26.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013598899999999999 2024-09-21T10:17:26.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,2,7", 2024-09-21T10:17:26.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.843792+0000", 2024-09-21T10:17:26.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014812899999999999 2024-09-21T10:17:26.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.846402+0000", 2024-09-21T10:17:26.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026091209999999998 2024-09-21T10:17:26.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:17:26.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.847550+0000", 2024-09-21T10:17:26.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011485480000000001 2024-09-21T10:17:26.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.153 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.153 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.847588+0000", 2024-09-21T10:17:26.153 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7997999999999998e-05 2024-09-21T10:17:26.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.153 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.153 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.859833+0000", 2024-09-21T10:17:26.153 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012244596999999999 2024-09-21T10:17:26.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.153 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:17:26.154 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.859883+0000", 2024-09-21T10:17:26.154 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0677000000000003e-05 2024-09-21T10:17:26.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.154 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:17:26.154 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.861305+0000", 2024-09-21T10:17:26.154 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00142147 2024-09-21T10:17:26.154 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.154 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:17:26.154 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.155 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:56332 7.12 7:4a86aa4d:::benchmark_data_smithi098_197834_object56331:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-09-21T10:17:26.155 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:13:46.704257+0000", 2024-09-21T10:17:26.155 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 219.33570493600001, 2024-09-21T10:17:26.155 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.159000279, 2024-09-21T10:17:26.155 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:17:26.155 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:17:26.155 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:26.155 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:17:26.155 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:17:26.155 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:17:26.155 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 56332 2024-09-21T10:17:26.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.155 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:17:26.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.156 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:17:26.156 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.704257+0000", 2024-09-21T10:17:26.156 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.156 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:17:26.156 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.704257+0000", 2024-09-21T10:17:26.156 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.156 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:17:26.156 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.704258+0000", 2024-09-21T10:17:26.156 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5889999999999999e-06 2024-09-21T10:17:26.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.157 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:17:26.157 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.704312+0000", 2024-09-21T10:17:26.157 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3872000000000002e-05 2024-09-21T10:17:26.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.157 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:17:26.157 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.704326+0000", 2024-09-21T10:17:26.157 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3569e-05 2024-09-21T10:17:26.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.157 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:17:26.157 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.704334+0000", 2024-09-21T10:17:26.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6450000000000005e-06 2024-09-21T10:17:26.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.158 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.158 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.704378+0000", 2024-09-21T10:17:26.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4705e-05 2024-09-21T10:17:26.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.158 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:17:26.158 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:46.704384+0000", 2024-09-21T10:17:26.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8980000000000001e-06 2024-09-21T10:17:26.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.158 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.159 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.845077+0000", 2024-09-21T10:17:26.159 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1406930609999999 2024-09-21T10:17:26.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.159 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:17:26.159 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.845231+0000", 2024-09-21T10:17:26.159 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015337400000000001 2024-09-21T10:17:26.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.159 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,2,7", 2024-09-21T10:17:26.159 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.845378+0000", 2024-09-21T10:17:26.159 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000147334 2024-09-21T10:17:26.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.160 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.160 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.847625+0000", 2024-09-21T10:17:26.160 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022470010000000002 2024-09-21T10:17:26.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.160 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:17:26.160 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.848257+0000", 2024-09-21T10:17:26.160 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00063221999999999996 2024-09-21T10:17:26.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.160 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.160 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.850047+0000", 2024-09-21T10:17:26.160 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0017899529999999999 2024-09-21T10:17:26.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.161 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.161 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.861360+0000", 2024-09-21T10:17:26.161 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011312439000000001 2024-09-21T10:17:26.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.161 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:17:26.161 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.861506+0000", 2024-09-21T10:17:26.161 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014644699999999999 2024-09-21T10:17:26.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.161 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:17:26.161 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:13:47.863257+0000", 2024-09-21T10:17:26.162 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0017511720000000001 2024-09-21T10:17:26.162 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.162 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:17:26.162 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.162 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:183384 7.10 7:08a2bd4f:::benchmark_data_smithi098_197834_object66867:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e247)", 2024-09-21T10:17:26.162 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:16:04.625356+0000", 2024-09-21T10:17:26.162 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 81.414606173999999, 2024-09-21T10:17:26.162 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4766592080000001, 2024-09-21T10:17:26.162 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:17:26.162 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:17:26.163 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-21T10:17:26.163 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:17:26.163 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:17:26.163 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:17:26.163 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 183384 2024-09-21T10:17:26.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.163 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:17:26.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.163 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:17:26.163 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625356+0000", 2024-09-21T10:17:26.163 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.164 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:17:26.164 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625356+0000", 2024-09-21T10:17:26.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.164 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:17:26.164 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625357+0000", 2024-09-21T10:17:26.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5850000000000001e-06 2024-09-21T10:17:26.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.164 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:17:26.164 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625363+0000", 2024-09-21T10:17:26.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8560000000000003e-06 2024-09-21T10:17:26.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.165 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:17:26.165 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625376+0000", 2024-09-21T10:17:26.165 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2745000000000001e-05 2024-09-21T10:17:26.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.165 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:17:26.165 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625385+0000", 2024-09-21T10:17:26.165 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8410000000000006e-06 2024-09-21T10:17:26.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.165 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.165 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625473+0000", 2024-09-21T10:17:26.166 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8282999999999995e-05 2024-09-21T10:17:26.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.166 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:17:26.166 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625482+0000", 2024-09-21T10:17:26.166 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3700000000000001e-06 2024-09-21T10:17:26.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.166 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.166 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.052517+0000", 2024-09-21T10:17:26.166 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.427034554 2024-09-21T10:17:26.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.166 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for missing object", 2024-09-21T10:17:26.167 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.052656+0000", 2024-09-21T10:17:26.167 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013865500000000001 2024-09-21T10:17:26.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.167 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.167 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.101924+0000", 2024-09-21T10:17:26.167 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.049268924999999998 2024-09-21T10:17:26.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.168 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:17:26.168 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.102015+0000", 2024-09-21T10:17:26.168 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0394000000000002e-05 2024-09-21T10:17:26.168 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.168 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:17:26.168 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.169 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:183402 7.10 7:0fd4b20d:::benchmark_data_smithi098_197834_object66885:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e247)", 2024-09-21T10:17:26.169 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:16:04.625403+0000", 2024-09-21T10:17:26.169 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 81.414558955999993, 2024-09-21T10:17:26.169 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.467456573, 2024-09-21T10:17:26.170 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:17:26.170 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:17:26.170 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:26.170 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:17:26.170 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:17:26.170 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:17:26.171 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 183402 2024-09-21T10:17:26.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.171 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:17:26.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.171 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:17:26.171 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625403+0000", 2024-09-21T10:17:26.172 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.172 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:17:26.172 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625403+0000", 2024-09-21T10:17:26.172 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.172 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:17:26.173 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625403+0000", 2024-09-21T10:17:26.173 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.99e-07 2024-09-21T10:17:26.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.173 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:17:26.173 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625406+0000", 2024-09-21T10:17:26.173 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7269999999999999e-06 2024-09-21T10:17:26.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.173 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:17:26.173 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625413+0000", 2024-09-21T10:17:26.173 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8850000000000001e-06 2024-09-21T10:17:26.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.174 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:17:26.174 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625419+0000", 2024-09-21T10:17:26.174 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5099999999999998e-06 2024-09-21T10:17:26.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.174 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.174 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625505+0000", 2024-09-21T10:17:26.174 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6426000000000002e-05 2024-09-21T10:17:26.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.174 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:17:26.174 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625511+0000", 2024-09-21T10:17:26.175 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1310000000000003e-06 2024-09-21T10:17:26.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.175 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.175 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.054457+0000", 2024-09-21T10:17:26.175 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4289461880000001 2024-09-21T10:17:26.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.175 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:17:26.175 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.054624+0000", 2024-09-21T10:17:26.175 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000166785 2024-09-21T10:17:26.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:17:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.055132+0000", 2024-09-21T10:17:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00050767599999999998 2024-09-21T10:17:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.063905+0000", 2024-09-21T10:17:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0087736229999999995 2024-09-21T10:17:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:17:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.067250+0000", 2024-09-21T10:17:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033444999999999998 2024-09-21T10:17:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.091569+0000", 2024-09-21T10:17:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.024319552000000001 2024-09-21T10:17:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:17:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.091658+0000", 2024-09-21T10:17:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8708000000000005e-05 2024-09-21T10:17:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:17:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.092859+0000", 2024-09-21T10:17:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0012012629999999999 2024-09-21T10:17:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:17:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:183425 7.10 7:0e439a58:::benchmark_data_smithi098_197834_object66908:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e247)", 2024-09-21T10:17:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:16:04.625426+0000", 2024-09-21T10:17:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 81.414535905999998, 2024-09-21T10:17:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4683116520000001, 2024-09-21T10:17:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:17:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:17:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:17:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:17:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:17:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 183425 2024-09-21T10:17:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:17:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:17:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625426+0000", 2024-09-21T10:17:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:17:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625426+0000", 2024-09-21T10:17:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:17:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625426+0000", 2024-09-21T10:17:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0800000000000005e-07 2024-09-21T10:17:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:17:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625428+0000", 2024-09-21T10:17:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.08e-06 2024-09-21T10:17:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:17:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625439+0000", 2024-09-21T10:17:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0040999999999999e-05 2024-09-21T10:17:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:17:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625444+0000", 2024-09-21T10:17:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-21T10:17:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625532+0000", 2024-09-21T10:17:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7598999999999999e-05 2024-09-21T10:17:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:17:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625538+0000", 2024-09-21T10:17:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8799999999999996e-06 2024-09-21T10:17:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.057949+0000", 2024-09-21T10:17:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4324113439999999 2024-09-21T10:17:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:17:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.058077+0000", 2024-09-21T10:17:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012833199999999999 2024-09-21T10:17:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:17:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.058560+0000", 2024-09-21T10:17:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048303700000000002 2024-09-21T10:17:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:17:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.067257+0000", 2024-09-21T10:17:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0086964520000000003 2024-09-21T10:17:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.070481+0000", 2024-09-21T10:17:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032244119999999999 2024-09-21T10:17:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.092909+0000", 2024-09-21T10:17:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.02242767 2024-09-21T10:17:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:17:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.092963+0000", 2024-09-21T10:17:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4605999999999999e-05 2024-09-21T10:17:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:17:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.093738+0000", 2024-09-21T10:17:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00077408100000000003 2024-09-21T10:17:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:17:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:183521 7.10 7:0eaed8ba:::benchmark_data_smithi098_197834_object67004:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e247)", 2024-09-21T10:17:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:16:04.625461+0000", 2024-09-21T10:17:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 81.414500829000005, 2024-09-21T10:17:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4690343239999999, 2024-09-21T10:17:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:17:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:17:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:17:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:17:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:17:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 183521 2024-09-21T10:17:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:17:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:17:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625461+0000", 2024-09-21T10:17:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:17:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625461+0000", 2024-09-21T10:17:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:17:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625462+0000", 2024-09-21T10:17:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8800000000000002e-07 2024-09-21T10:17:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:17:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625464+0000", 2024-09-21T10:17:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.526e-06 2024-09-21T10:17:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:17:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625473+0000", 2024-09-21T10:17:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2760000000000006e-06 2024-09-21T10:17:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:17:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625479+0000", 2024-09-21T10:17:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0990000000000004e-06 2024-09-21T10:17:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625558+0000", 2024-09-21T10:17:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8733999999999995e-05 2024-09-21T10:17:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:17:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625564+0000", 2024-09-21T10:17:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0610000000000001e-06 2024-09-21T10:17:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.060861+0000", 2024-09-21T10:17:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.435296865 2024-09-21T10:17:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:17:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.060959+0000", 2024-09-21T10:17:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7464000000000001e-05 2024-09-21T10:17:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:17:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.061391+0000", 2024-09-21T10:17:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00043200999999999998 2024-09-21T10:17:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:17:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.070432+0000", 2024-09-21T10:17:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00904107 2024-09-21T10:17:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.070539+0000", 2024-09-21T10:17:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107613 2024-09-21T10:17:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.093784+0000", 2024-09-21T10:17:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.023244396000000001 2024-09-21T10:17:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:17:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.093836+0000", 2024-09-21T10:17:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2797999999999998e-05 2024-09-21T10:17:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:17:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.094495+0000", 2024-09-21T10:17:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00065882400000000002 2024-09-21T10:17:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:17:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:183542 7.10 7:0aad07f0:::benchmark_data_smithi098_197834_object67025:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e247)", 2024-09-21T10:17:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:16:04.625487+0000", 2024-09-21T10:17:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 81.414475296000006, 2024-09-21T10:17:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.469794703, 2024-09-21T10:17:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:17:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:17:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:17:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:17:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:17:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 183542 2024-09-21T10:17:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:17:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:17:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625487+0000", 2024-09-21T10:17:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:17:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625487+0000", 2024-09-21T10:17:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:17:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625487+0000", 2024-09-21T10:17:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.15e-07 2024-09-21T10:17:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:17:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625489+0000", 2024-09-21T10:17:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1220000000000002e-06 2024-09-21T10:17:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:17:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625496+0000", 2024-09-21T10:17:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0380000000000002e-06 2024-09-21T10:17:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:17:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625502+0000", 2024-09-21T10:17:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9449999999999998e-06 2024-09-21T10:17:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625585+0000", 2024-09-21T10:17:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2830999999999993e-05 2024-09-21T10:17:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:17:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625591+0000", 2024-09-21T10:17:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1550000000000004e-06 2024-09-21T10:17:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.063953+0000", 2024-09-21T10:17:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.438361497 2024-09-21T10:17:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:17:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.064064+0000", 2024-09-21T10:17:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011127 2024-09-21T10:17:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.198 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:17:26.198 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.064526+0000", 2024-09-21T10:17:26.198 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000462523 2024-09-21T10:17:26.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.198 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:17:26.198 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.070495+0000", 2024-09-21T10:17:26.198 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0059688450000000004 2024-09-21T10:17:26.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.198 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.199 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:26.199 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.073177+0000", 2024-09-21T10:17:26.199 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026820920000000001 2024-09-21T10:17:26.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.199 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.199 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.094546+0000", 2024-09-21T10:17:26.200 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.021368726000000001 2024-09-21T10:17:26.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.200 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.200 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:17:26.200 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.094602+0000", 2024-09-21T10:17:26.200 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6008000000000003e-05 2024-09-21T10:17:26.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.200 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.200 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:17:26.200 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.095281+0000", 2024-09-21T10:17:26.200 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00067923600000000005 2024-09-21T10:17:26.200 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.200 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:17:26.200 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.201 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:183546 7.10 7:083dfd64:::benchmark_data_smithi098_197834_object67029:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e247)", 2024-09-21T10:17:26.201 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:16:04.625508+0000", 2024-09-21T10:17:26.201 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 81.414453680999998, 2024-09-21T10:17:26.201 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.470515864, 2024-09-21T10:17:26.201 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:17:26.201 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:17:26.201 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:26.201 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:17:26.201 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:17:26.201 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:17:26.201 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 183546 2024-09-21T10:17:26.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.202 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:17:26.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.202 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:17:26.202 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625508+0000", 2024-09-21T10:17:26.202 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.202 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:17:26.202 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625508+0000", 2024-09-21T10:17:26.202 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:17:26.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.203 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:17:26.203 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625509+0000", 2024-09-21T10:17:26.203 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2300000000000002e-07 2024-09-21T10:17:26.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.203 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:17:26.203 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625510+0000", 2024-09-21T10:17:26.203 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9580000000000001e-06 2024-09-21T10:17:26.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.203 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:17:26.203 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625516+0000", 2024-09-21T10:17:26.203 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9519999999999999e-06 2024-09-21T10:17:26.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.204 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:17:26.204 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625522+0000", 2024-09-21T10:17:26.204 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.101e-06 2024-09-21T10:17:26.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.204 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.204 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625612+0000", 2024-09-21T10:17:26.204 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0529e-05 2024-09-21T10:17:26.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.204 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:17:26.204 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.625618+0000", 2024-09-21T10:17:26.205 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9780000000000002e-06 2024-09-21T10:17:26.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.205 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:17:26.205 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.067291+0000", 2024-09-21T10:17:26.205 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.441672767 2024-09-21T10:17:26.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.205 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:17:26.205 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.067439+0000", 2024-09-21T10:17:26.205 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000147708 2024-09-21T10:17:26.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.205 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:17:26.206 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.067881+0000", 2024-09-21T10:17:26.206 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044199300000000001 2024-09-21T10:17:26.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.206 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:17:26.206 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.071397+0000", 2024-09-21T10:17:26.206 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035164699999999998 2024-09-21T10:17:26.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.206 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.206 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.073219+0000", 2024-09-21T10:17:26.206 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018217999999999999 2024-09-21T10:17:26.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.207 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:26.207 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.095335+0000", 2024-09-21T10:17:26.207 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.022116278 2024-09-21T10:17:26.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.207 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:17:26.207 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.095389+0000", 2024-09-21T10:17:26.207 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3949999999999997e-05 2024-09-21T10:17:26.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:17:26.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:17:26.207 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:17:26.207 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.096024+0000", 2024-09-21T10:17:26.207 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00063495700000000001 2024-09-21T10:17:26.208 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.208 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:17:26.208 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.208 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:17:26.208 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:17:26.208 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:17:26.209 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:25 smithi196 ceph-mon[84784]: osdmap e263: 8 total, 8 up, 7 in 2024-09-21T10:17:26.209 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:25 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 8166/330 objects degraded (2474.545%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:17:26.233 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:26.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:26.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:26.335 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:26.361 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:26.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:26.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:26.462 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:26.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:26.489 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:26.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:26.590 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:26.614 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:26.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:26.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:26.737 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:26.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:26.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:26.757 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'unique_pool_2' removed 2024-09-21T10:17:26.840 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:26.863 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:26.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:26.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:26.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:26.988 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:26.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:26.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:27.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:26 smithi098 ceph-mon[97580]: pgmap v808: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 9.7 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 8166/330 objects degraded (2474.545%) 2024-09-21T10:17:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:26 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/722076552' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:17:27.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:26 smithi098 ceph-mon[100914]: pgmap v808: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 9.7 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 8166/330 objects degraded (2474.545%) 2024-09-21T10:17:27.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:26 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/722076552' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:17:27.113 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:27.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:27.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:27.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:26 smithi196 ceph-mon[84784]: pgmap v808: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 9.7 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 8166/330 objects degraded (2474.545%) 2024-09-21T10:17:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:26 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/722076552' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:17:27.215 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:27.249 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:27.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:27.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:27.351 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:27.372 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:27.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:27.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:27.474 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:27.497 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:27.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:27.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:27.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:27.621 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:27.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:27.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:27.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:27.744 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:27.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:27.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:27.846 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:27.869 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:27.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:27.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:27.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:27.993 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:27.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:27.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:28.095 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:28.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:27 smithi098 ceph-mon[97580]: 7.10 scrub starts 2024-09-21T10:17:28.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:27 smithi098 ceph-mon[97580]: 7.10 scrub ok 2024-09-21T10:17:28.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:27 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/722076552' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:17:28.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:27 smithi098 ceph-mon[97580]: osdmap e264: 8 total, 8 up, 7 in 2024-09-21T10:17:28.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:27 smithi098 ceph-mon[100914]: 7.10 scrub starts 2024-09-21T10:17:28.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:27 smithi098 ceph-mon[100914]: 7.10 scrub ok 2024-09-21T10:17:28.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:27 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/722076552' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:17:28.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:27 smithi098 ceph-mon[100914]: osdmap e264: 8 total, 8 up, 7 in 2024-09-21T10:17:28.116 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:28.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:28.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:28.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:27 smithi196 ceph-mon[84784]: 7.10 scrub starts 2024-09-21T10:17:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:27 smithi196 ceph-mon[84784]: 7.10 scrub ok 2024-09-21T10:17:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:27 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/722076552' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:17:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:27 smithi196 ceph-mon[84784]: osdmap e264: 8 total, 8 up, 7 in 2024-09-21T10:17:28.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:28.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:28.239 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:28.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:28.340 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:28.362 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:28.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:28.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:28.464 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:28.486 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:28.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:28.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:28.587 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:28.609 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:28.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:28.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:28.710 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:28.735 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:28.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:28.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:28.771 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:17:28.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:28.858 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:28.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:28.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:28.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:28.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:28.982 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:28.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:29.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:28 smithi098 ceph-mon[97580]: pgmap v810: 73 pgs: 73 active+clean; 4.6 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:29.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:28 smithi098 ceph-mon[97580]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8166/330 objects degraded (2474.545%), 3 pgs degraded, 3 pgs undersized) 2024-09-21T10:17:29.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:28 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:17:29.061 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:28 smithi098 ceph-mon[100914]: pgmap v810: 73 pgs: 73 active+clean; 4.6 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:29.061 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:28 smithi098 ceph-mon[100914]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8166/330 objects degraded (2474.545%), 3 pgs degraded, 3 pgs undersized) 2024-09-21T10:17:29.061 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:28 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:17:29.084 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:29.107 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:29.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:29.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:28 smithi196 ceph-mon[84784]: pgmap v810: 73 pgs: 73 active+clean; 4.6 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:28 smithi196 ceph-mon[84784]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8166/330 objects degraded (2474.545%), 3 pgs degraded, 3 pgs undersized) 2024-09-21T10:17:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:28 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:17:29.209 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:29.233 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:29.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:29.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:29.334 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:29.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:29.357 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:29.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:29.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:29.480 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:29.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:29.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:29.582 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:29.608 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:29.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:29.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:29.710 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:29.735 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:29.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:29.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:29.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:29.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:29.860 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:29.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:29.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:29.987 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:29.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:29.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:30.089 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:30.095 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:17:30.096 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:17:30.096 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-21T10:17:30.096 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_3 16 2024-09-21T10:17:30.115 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:30.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:30.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:30.216 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:30.241 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:30.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:30.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:30.316 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:30.343 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:30.364 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:30.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:30.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:30.465 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:30.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:30.488 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:30.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:30.500 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:17:30.501 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:17:30.589 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:30.615 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:30.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:30.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:30.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:17:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:17:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:17:30.716 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:30.738 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:30.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:30.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:30.765 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:17:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:17:30] "GET /metrics HTTP/1.1" 200 36519 "" "Prometheus/2.51.0" 2024-09-21T10:17:30.840 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:30.866 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:30.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:30.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:30.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:30.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:30.990 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:30.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:31.091 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:31.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:30 smithi098 ceph-mon[97580]: pgmap v811: 73 pgs: 73 active+clean; 4.6 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:31.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:30 smithi098 ceph-mon[100914]: pgmap v811: 73 pgs: 73 active+clean; 4.6 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:31.112 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:31.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:31.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:31.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:30 smithi196 ceph-mon[84784]: pgmap v811: 73 pgs: 73 active+clean; 4.6 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:31.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:31.234 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:31.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:31.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:31.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:31.357 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:31.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:31.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:31.459 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:31.480 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:31.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:31.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:31.582 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:31.608 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:31.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:31.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:31.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:31.733 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:31.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:31.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:31.835 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:31.861 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:31.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:31.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:31.962 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:31.989 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:31.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:31.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:32.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:32.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:31 smithi098 ceph-mon[97580]: from='client.35838 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:17:32.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:31 smithi098 ceph-mon[100914]: from='client.35838 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:17:32.115 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:32.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:32.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:32.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:31 smithi196 ceph-mon[84784]: from='client.35838 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:17:32.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:32.243 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:32.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:32.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:32.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:32.370 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:32.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:32.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:32.472 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:32.497 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:32.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:32.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:32.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:32.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:32.626 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:32.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:32.727 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:32.735 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:32.752 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:32.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:32.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:32.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:32.878 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:32.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:32.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:32.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:33.005 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:33.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:33.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:33.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:32 smithi098 ceph-mon[97580]: pgmap v812: 73 pgs: 73 active+clean; 4.6 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:33.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:32 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:17:33.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:32 smithi098 ceph-mon[100914]: pgmap v812: 73 pgs: 73 active+clean; 4.6 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:33.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:32 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:17:33.107 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:33.131 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:33.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:33.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:33.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:32 smithi196 ceph-mon[84784]: pgmap v812: 73 pgs: 73 active+clean; 4.6 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:33.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:32 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:17:33.233 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:33.259 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:33.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:33.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:33.361 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:33.386 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:33.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:33.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:33.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:33.512 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:33.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:33.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:33.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:33.637 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:33.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:33.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:33.739 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:33.765 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:33.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:33.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:33.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:33.892 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:33.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:33.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:33.912 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":811,"stamp":"2024-09-21T10:17:29.334055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6288,"num_read_kb":13764,"num_write":1056,"num_write_kb":14251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":600294,"num_keys_recovered":30,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1132,"ondisk_log_size":1132,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":280,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":3965984,"kb_used_data":983744,"kb_used_omap":33,"kb_used_meta":2980190,"kb_avail":652250080,"statfs":{"total":671965249536,"available":667904081920,"internally_reserved":0,"allocated":1007353856,"data_stored":997616669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34649,"internal_metadata":3051714727},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":69,"apply_latency_ms":69,"commit_latency_ns":69000000,"apply_latency_ns":69000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5373952,"num_objects":-82,"num_object_clones":0,"num_object_copies":-246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-8166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-82,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-201672,"num_write_kb":-5700225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-12679,"num_bytes_recovered":-660930560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001263"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":399,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.068547+0000","last_change":"2024-09-21T10:02:14.097991+0000","last_active":"2024-09-21T10:17:27.068547+0000","last_peered":"2024-09-21T10:17:27.068547+0000","last_clean":"2024-09-21T10:17:27.068547+0000","last_became_active":"2024-09-21T10:02:14.097407+0000","last_became_peered":"2024-09-21T10:02:14.097407+0000","last_unstale":"2024-09-21T10:17:27.068547+0000","last_undegraded":"2024-09-21T10:17:27.068547+0000","last_fullsized":"2024-09-21T10:17:27.068547+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:18:42.098886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":279,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.826425+0000","last_change":"2024-09-21T10:08:00.406360+0000","last_active":"2024-09-21T10:17:24.826425+0000","last_peered":"2024-09-21T10:17:24.826425+0000","last_clean":"2024-09-21T10:17:24.826425+0000","last_became_active":"2024-09-21T10:08:00.380042+0000","last_became_peered":"2024-09-21T10:08:00.380042+0000","last_unstale":"2024-09-21T10:17:24.826425+0000","last_undegraded":"2024-09-21T10:17:24.826425+0000","last_fullsized":"2024-09-21T10:17:24.826425+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:49:46.544051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":185,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.826592+0000","last_change":"2024-09-21T10:07:43.789751+0000","last_active":"2024-09-21T10:17:24.826592+0000","last_peered":"2024-09-21T10:17:24.826592+0000","last_clean":"2024-09-21T10:17:24.826592+0000","last_became_active":"2024-09-21T10:07:43.789448+0000","last_became_peered":"2024-09-21T10:07:43.789448+0000","last_unstale":"2024-09-21T10:17:24.826592+0000","last_undegraded":"2024-09-21T10:17:24.826592+0000","last_fullsized":"2024-09-21T10:17:24.826592+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:36.586942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":395,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.044643+0000","last_change":"2024-09-21T10:02:14.098949+0000","last_active":"2024-09-21T10:17:27.044643+0000","last_peered":"2024-09-21T10:17:27.044643+0000","last_clean":"2024-09-21T10:17:27.044643+0000","last_became_active":"2024-09-21T10:02:14.098646+0000","last_became_peered":"2024-09-21T10:02:14.098646+0000","last_unstale":"2024-09-21T10:17:27.044643+0000","last_undegraded":"2024-09-21T10:17:27.044643+0000","last_fullsized":"2024-09-21T10:17:27.044643+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:50:56.862529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":395,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.045691+0000","last_change":"2024-09-21T10:02:14.098205+0000","last_active":"2024-09-21T10:17:27.045691+0000","last_peered":"2024-09-21T10:17:27.045691+0000","last_clean":"2024-09-21T10:17:27.045691+0000","last_became_active":"2024-09-21T10:02:14.097911+0000","last_became_peered":"2024-09-21T10:02:14.097911+0000","last_unstale":"2024-09-21T10:17:27.045691+0000","last_undegraded":"2024-09-21T10:17:27.045691+0000","last_fullsized":"2024-09-21T10:17:27.045691+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:55.971191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":144,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:25.256188+0000","last_change":"2024-09-21T10:08:00.385671+0000","last_active":"2024-09-21T10:17:25.256188+0000","last_peered":"2024-09-21T10:17:25.256188+0000","last_clean":"2024-09-21T10:17:25.256188+0000","last_became_active":"2024-09-21T10:08:00.385368+0000","last_became_peered":"2024-09-21T10:08:00.385368+0000","last_unstale":"2024-09-21T10:17:25.256188+0000","last_undegraded":"2024-09-21T10:17:25.256188+0000","last_fullsized":"2024-09-21T10:17:25.256188+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:21.799149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":399,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.073901+0000","last_change":"2024-09-21T10:02:14.097817+0000","last_active":"2024-09-21T10:17:27.073901+0000","last_peered":"2024-09-21T10:17:27.073901+0000","last_clean":"2024-09-21T10:17:27.073901+0000","last_became_active":"2024-09-21T10:02:14.097466+0000","last_became_peered":"2024-09-21T10:02:14.097466+0000","last_unstale":"2024-09-21T10:17:27.073901+0000","last_undegraded":"2024-09-21T10:17:27.073901+0000","last_fullsized":"2024-09-21T10:17:27.073901+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:57.883118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":236,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:25.255326+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:17:25.255326+0000","last_peered":"2024-09-21T10:17:25.255326+0000","last_clean":"2024-09-21T10:17:25.255326+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:17:25.255326+0000","last_undegraded":"2024-09-21T10:17:25.255326+0000","last_fullsized":"2024-09-21T10:17:25.255326+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":272,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.826915+0000","last_change":"2024-09-21T10:08:00.392499+0000","last_active":"2024-09-21T10:17:24.826915+0000","last_peered":"2024-09-21T10:17:24.826915+0000","last_clean":"2024-09-21T10:17:24.826915+0000","last_became_active":"2024-09-21T10:08:00.379165+0000","last_became_peered":"2024-09-21T10:08:00.379165+0000","last_unstale":"2024-09-21T10:17:24.826915+0000","last_undegraded":"2024-09-21T10:17:24.826915+0000","last_fullsized":"2024-09-21T10:17:24.826915+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:18.933144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":128,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.826599+0000","last_change":"2024-09-21T10:07:43.794677+0000","last_active":"2024-09-21T10:17:24.826599+0000","last_peered":"2024-09-21T10:17:24.826599+0000","last_clean":"2024-09-21T10:17:24.826599+0000","last_became_active":"2024-09-21T10:07:43.794348+0000","last_became_peered":"2024-09-21T10:07:43.794348+0000","last_unstale":"2024-09-21T10:17:24.826599+0000","last_undegraded":"2024-09-21T10:17:24.826599+0000","last_fullsized":"2024-09-21T10:17:24.826599+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:11.159014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":273,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.826868+0000","last_change":"2024-09-21T10:08:00.412449+0000","last_active":"2024-09-21T10:17:24.826868+0000","last_peered":"2024-09-21T10:17:24.826868+0000","last_clean":"2024-09-21T10:17:24.826868+0000","last_became_active":"2024-09-21T10:08:00.386530+0000","last_became_peered":"2024-09-21T10:08:00.386530+0000","last_unstale":"2024-09-21T10:17:24.826868+0000","last_undegraded":"2024-09-21T10:17:24.826868+0000","last_fullsized":"2024-09-21T10:17:24.826868+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:12:50.581176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":319,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.740473+0000","last_change":"2024-09-21T10:08:18.475007+0000","last_active":"2024-09-21T10:17:24.740473+0000","last_peered":"2024-09-21T10:17:24.740473+0000","last_clean":"2024-09-21T10:17:24.740473+0000","last_became_active":"2024-09-21T10:08:18.474730+0000","last_became_peered":"2024-09-21T10:08:18.474730+0000","last_unstale":"2024-09-21T10:17:24.740473+0000","last_undegraded":"2024-09-21T10:17:24.740473+0000","last_fullsized":"2024-09-21T10:17:24.740473+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:35.455790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":213,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.826759+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:17:24.826759+0000","last_peered":"2024-09-21T10:17:24.826759+0000","last_clean":"2024-09-21T10:17:24.826759+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:17:24.826759+0000","last_undegraded":"2024-09-21T10:17:24.826759+0000","last_fullsized":"2024-09-21T10:17:24.826759+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":227,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:25.255615+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:17:25.255615+0000","last_peered":"2024-09-21T10:17:25.255615+0000","last_clean":"2024-09-21T10:17:25.255615+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:17:25.255615+0000","last_undegraded":"2024-09-21T10:17:25.255615+0000","last_fullsized":"2024-09-21T10:17:25.255615+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":305,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.069147+0000","last_change":"2024-09-21T10:07:43.793008+0000","last_active":"2024-09-21T10:17:27.069147+0000","last_peered":"2024-09-21T10:17:27.069147+0000","last_clean":"2024-09-21T10:17:27.069147+0000","last_became_active":"2024-09-21T10:07:43.792683+0000","last_became_peered":"2024-09-21T10:07:43.792683+0000","last_unstale":"2024-09-21T10:17:27.069147+0000","last_undegraded":"2024-09-21T10:17:27.069147+0000","last_fullsized":"2024-09-21T10:17:27.069147+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:46:13.137478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":416,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.038554+0000","last_change":"2024-09-21T10:08:00.389645+0000","last_active":"2024-09-21T10:17:27.038554+0000","last_peered":"2024-09-21T10:17:27.038554+0000","last_clean":"2024-09-21T10:17:27.038554+0000","last_became_active":"2024-09-21T10:08:00.386076+0000","last_became_peered":"2024-09-21T10:08:00.386076+0000","last_unstale":"2024-09-21T10:17:27.038554+0000","last_undegraded":"2024-09-21T10:17:27.038554+0000","last_fullsized":"2024-09-21T10:17:27.038554+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:58:10.646600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":430,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.061951+0000","last_change":"2024-09-21T10:08:00.395898+0000","last_active":"2024-09-21T10:17:27.061951+0000","last_peered":"2024-09-21T10:17:27.061951+0000","last_clean":"2024-09-21T10:17:27.061951+0000","last_became_active":"2024-09-21T10:08:00.390462+0000","last_became_peered":"2024-09-21T10:08:00.390462+0000","last_unstale":"2024-09-21T10:17:27.061951+0000","last_undegraded":"2024-09-21T10:17:27.061951+0000","last_fullsized":"2024-09-21T10:17:27.061951+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:29:01.877089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":317,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.069231+0000","last_change":"2024-09-21T10:08:00.383016+0000","last_active":"2024-09-21T10:17:27.069231+0000","last_peered":"2024-09-21T10:17:27.069231+0000","last_clean":"2024-09-21T10:17:27.069231+0000","last_became_active":"2024-09-21T10:08:00.382708+0000","last_became_peered":"2024-09-21T10:08:00.382708+0000","last_unstale":"2024-09-21T10:17:27.069231+0000","last_undegraded":"2024-09-21T10:17:27.069231+0000","last_fullsized":"2024-09-21T10:17:27.069231+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:30:52.797259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":258,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:25.255814+0000","last_change":"2024-09-21T10:08:00.379604+0000","last_active":"2024-09-21T10:17:25.255814+0000","last_peered":"2024-09-21T10:17:25.255814+0000","last_clean":"2024-09-21T10:17:25.255814+0000","last_became_active":"2024-09-21T10:08:00.379301+0000","last_became_peered":"2024-09-21T10:08:00.379301+0000","last_unstale":"2024-09-21T10:17:25.255814+0000","last_undegraded":"2024-09-21T10:17:25.255814+0000","last_fullsized":"2024-09-21T10:17:25.255814+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:32:41.963929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":328,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.032084+0000","last_change":"2024-09-21T10:08:00.387420+0000","last_active":"2024-09-21T10:17:27.032084+0000","last_peered":"2024-09-21T10:17:27.032084+0000","last_clean":"2024-09-21T10:17:27.032084+0000","last_became_active":"2024-09-21T10:08:00.386843+0000","last_became_peered":"2024-09-21T10:08:00.386843+0000","last_unstale":"2024-09-21T10:17:27.032084+0000","last_undegraded":"2024-09-21T10:17:27.032084+0000","last_fullsized":"2024-09-21T10:17:27.032084+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:51:00.654836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":312,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.031408+0000","last_change":"2024-09-21T10:08:00.382496+0000","last_active":"2024-09-21T10:17:27.031408+0000","last_peered":"2024-09-21T10:17:27.031408+0000","last_clean":"2024-09-21T10:17:27.031408+0000","last_became_active":"2024-09-21T10:08:00.382180+0000","last_became_peered":"2024-09-21T10:08:00.382180+0000","last_unstale":"2024-09-21T10:17:27.031408+0000","last_undegraded":"2024-09-21T10:17:27.031408+0000","last_fullsized":"2024-09-21T10:17:27.031408+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:16.074736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":197,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:25.255511+0000","last_change":"2024-09-21T10:08:38.906801+0000","last_active":"2024-09-21T10:17:25.255511+0000","last_peered":"2024-09-21T10:17:25.255511+0000","last_clean":"2024-09-21T10:17:25.255511+0000","last_became_active":"2024-09-21T10:08:38.906487+0000","last_became_peered":"2024-09-21T10:08:38.906487+0000","last_unstale":"2024-09-21T10:17:25.255511+0000","last_undegraded":"2024-09-21T10:17:25.255511+0000","last_fullsized":"2024-09-21T10:17:25.255511+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:31.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":401,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.058796+0000","last_change":"2024-09-21T10:02:14.097462+0000","last_active":"2024-09-21T10:17:27.058796+0000","last_peered":"2024-09-21T10:17:27.058796+0000","last_clean":"2024-09-21T10:17:27.058796+0000","last_became_active":"2024-09-21T10:02:14.097167+0000","last_became_peered":"2024-09-21T10:02:14.097167+0000","last_unstale":"2024-09-21T10:17:27.058796+0000","last_undegraded":"2024-09-21T10:17:27.058796+0000","last_fullsized":"2024-09-21T10:17:27.058796+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:54.922731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":403,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.741965+0000","last_change":"2024-09-21T10:08:00.378569+0000","last_active":"2024-09-21T10:17:24.741965+0000","last_peered":"2024-09-21T10:17:24.741965+0000","last_clean":"2024-09-21T10:17:24.741965+0000","last_became_active":"2024-09-21T10:08:00.378015+0000","last_became_peered":"2024-09-21T10:08:00.378015+0000","last_unstale":"2024-09-21T10:17:24.741965+0000","last_undegraded":"2024-09-21T10:17:24.741965+0000","last_fullsized":"2024-09-21T10:17:24.741965+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:58.121005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":346,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.826891+0000","last_change":"2024-09-21T09:59:54.265556+0000","last_active":"2024-09-21T10:17:24.826891+0000","last_peered":"2024-09-21T10:17:24.826891+0000","last_clean":"2024-09-21T10:17:24.826891+0000","last_became_active":"2024-09-21T09:59:54.265033+0000","last_became_peered":"2024-09-21T09:59:54.265033+0000","last_unstale":"2024-09-21T10:17:24.826891+0000","last_undegraded":"2024-09-21T10:17:24.826891+0000","last_fullsized":"2024-09-21T10:17:24.826891+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T09:48:34.850872+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:40.449756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":692,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.032456+0000","last_change":"2024-09-21T10:07:43.790490+0000","last_active":"2024-09-21T10:17:27.032456+0000","last_peered":"2024-09-21T10:17:27.032456+0000","last_clean":"2024-09-21T10:17:27.032456+0000","last_became_active":"2024-09-21T10:07:43.789986+0000","last_became_peered":"2024-09-21T10:07:43.789986+0000","last_unstale":"2024-09-21T10:17:27.032456+0000","last_undegraded":"2024-09-21T10:17:27.032456+0000","last_fullsized":"2024-09-21T10:17:27.032456+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:08.740356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":481,"num_read_kb":1580,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":354,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.072869+0000","last_change":"2024-09-21T10:02:14.099961+0000","last_active":"2024-09-21T10:17:27.072869+0000","last_peered":"2024-09-21T10:17:27.072869+0000","last_clean":"2024-09-21T10:17:27.072869+0000","last_became_active":"2024-09-21T10:02:14.099067+0000","last_became_peered":"2024-09-21T10:02:14.099067+0000","last_unstale":"2024-09-21T10:17:27.072869+0000","last_undegraded":"2024-09-21T10:17:27.072869+0000","last_fullsized":"2024-09-21T10:17:27.072869+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:41.037532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":240,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:25.256207+0000","last_change":"2024-09-21T10:08:00.386135+0000","last_active":"2024-09-21T10:17:25.256207+0000","last_peered":"2024-09-21T10:17:25.256207+0000","last_clean":"2024-09-21T10:17:25.256207+0000","last_became_active":"2024-09-21T10:08:00.385796+0000","last_became_peered":"2024-09-21T10:08:00.385796+0000","last_unstale":"2024-09-21T10:17:25.256207+0000","last_undegraded":"2024-09-21T10:17:25.256207+0000","last_fullsized":"2024-09-21T10:17:25.256207+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:27.054699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":609,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.826844+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:17:24.826844+0000","last_peered":"2024-09-21T10:17:24.826844+0000","last_clean":"2024-09-21T10:17:24.826844+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:17:24.826844+0000","last_undegraded":"2024-09-21T10:17:24.826844+0000","last_fullsized":"2024-09-21T10:17:24.826844+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":417,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.039355+0000","last_change":"2024-09-21T10:08:00.383928+0000","last_active":"2024-09-21T10:17:27.039355+0000","last_peered":"2024-09-21T10:17:27.039355+0000","last_clean":"2024-09-21T10:17:27.039355+0000","last_became_active":"2024-09-21T10:08:00.383625+0000","last_became_peered":"2024-09-21T10:08:00.383625+0000","last_unstale":"2024-09-21T10:17:27.039355+0000","last_undegraded":"2024-09-21T10:17:27.039355+0000","last_fullsized":"2024-09-21T10:17:27.039355+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:39.207283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":407,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.045879+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:17:27.045879+0000","last_peered":"2024-09-21T10:17:27.045879+0000","last_clean":"2024-09-21T10:17:27.045879+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:17:27.045879+0000","last_undegraded":"2024-09-21T10:17:27.045879+0000","last_fullsized":"2024-09-21T10:17:27.045879+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":1144,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.031481+0000","last_change":"2024-09-21T10:02:14.100143+0000","last_active":"2024-09-21T10:17:27.031481+0000","last_peered":"2024-09-21T10:17:27.031481+0000","last_clean":"2024-09-21T10:17:27.031481+0000","last_became_active":"2024-09-21T10:02:14.099586+0000","last_became_peered":"2024-09-21T10:02:14.099586+0000","last_unstale":"2024-09-21T10:17:27.031481+0000","last_undegraded":"2024-09-21T10:17:27.031481+0000","last_fullsized":"2024-09-21T10:17:27.031481+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:34.064652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":277,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.741252+0000","last_change":"2024-09-21T10:08:18.470748+0000","last_active":"2024-09-21T10:17:24.741252+0000","last_peered":"2024-09-21T10:17:24.741252+0000","last_clean":"2024-09-21T10:17:24.741252+0000","last_became_active":"2024-09-21T10:08:18.470227+0000","last_became_peered":"2024-09-21T10:08:18.470227+0000","last_unstale":"2024-09-21T10:17:24.741252+0000","last_undegraded":"2024-09-21T10:17:24.741252+0000","last_fullsized":"2024-09-21T10:17:24.741252+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:16.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":401,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.032380+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:17:27.032380+0000","last_peered":"2024-09-21T10:17:27.032380+0000","last_clean":"2024-09-21T10:17:27.032380+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:17:27.032380+0000","last_undegraded":"2024-09-21T10:17:27.032380+0000","last_fullsized":"2024-09-21T10:17:27.032380+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":875,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.740637+0000","last_change":"2024-09-21T10:07:43.794767+0000","last_active":"2024-09-21T10:17:24.740637+0000","last_peered":"2024-09-21T10:17:24.740637+0000","last_clean":"2024-09-21T10:17:24.740637+0000","last_became_active":"2024-09-21T10:07:43.794062+0000","last_became_peered":"2024-09-21T10:07:43.794062+0000","last_unstale":"2024-09-21T10:17:24.740637+0000","last_undegraded":"2024-09-21T10:17:24.740637+0000","last_fullsized":"2024-09-21T10:17:24.740637+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:51.977955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":304,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.036172+0000","last_change":"2024-09-21T10:08:00.389176+0000","last_active":"2024-09-21T10:17:27.036172+0000","last_peered":"2024-09-21T10:17:27.036172+0000","last_clean":"2024-09-21T10:17:27.036172+0000","last_became_active":"2024-09-21T10:08:00.383854+0000","last_became_peered":"2024-09-21T10:08:00.383854+0000","last_unstale":"2024-09-21T10:17:27.036172+0000","last_undegraded":"2024-09-21T10:17:27.036172+0000","last_fullsized":"2024-09-21T10:17:27.036172+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:46.952216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":317,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.032726+0000","last_change":"2024-09-21T10:07:43.792153+0000","last_active":"2024-09-21T10:17:27.032726+0000","last_peered":"2024-09-21T10:17:27.032726+0000","last_clean":"2024-09-21T10:17:27.032726+0000","last_became_active":"2024-09-21T10:07:43.791770+0000","last_became_peered":"2024-09-21T10:07:43.791770+0000","last_unstale":"2024-09-21T10:17:27.032726+0000","last_undegraded":"2024-09-21T10:17:27.032726+0000","last_fullsized":"2024-09-21T10:17:27.032726+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:52.506941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":322,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.062232+0000","last_change":"2024-09-21T10:07:43.791405+0000","last_active":"2024-09-21T10:17:27.062232+0000","last_peered":"2024-09-21T10:17:27.062232+0000","last_clean":"2024-09-21T10:17:27.062232+0000","last_became_active":"2024-09-21T10:07:43.791078+0000","last_became_peered":"2024-09-21T10:07:43.791078+0000","last_unstale":"2024-09-21T10:17:27.062232+0000","last_undegraded":"2024-09-21T10:17:27.062232+0000","last_fullsized":"2024-09-21T10:17:27.062232+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:17:38.072612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":982,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.741418+0000","last_change":"2024-09-21T10:08:18.468995+0000","last_active":"2024-09-21T10:17:24.741418+0000","last_peered":"2024-09-21T10:17:24.741418+0000","last_clean":"2024-09-21T10:17:24.741418+0000","last_became_active":"2024-09-21T10:08:18.468243+0000","last_became_peered":"2024-09-21T10:08:18.468243+0000","last_unstale":"2024-09-21T10:17:24.741418+0000","last_undegraded":"2024-09-21T10:17:24.741418+0000","last_fullsized":"2024-09-21T10:17:24.741418+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:45.520968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":301,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.740866+0000","last_change":"2024-09-21T10:08:18.468595+0000","last_active":"2024-09-21T10:17:24.740866+0000","last_peered":"2024-09-21T10:17:24.740866+0000","last_clean":"2024-09-21T10:17:24.740866+0000","last_became_active":"2024-09-21T10:08:18.468296+0000","last_became_peered":"2024-09-21T10:08:18.468296+0000","last_unstale":"2024-09-21T10:17:24.740866+0000","last_undegraded":"2024-09-21T10:17:24.740866+0000","last_fullsized":"2024-09-21T10:17:24.740866+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:42.748130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":277,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.741323+0000","last_change":"2024-09-21T10:08:18.469907+0000","last_active":"2024-09-21T10:17:24.741323+0000","last_peered":"2024-09-21T10:17:24.741323+0000","last_clean":"2024-09-21T10:17:24.741323+0000","last_became_active":"2024-09-21T10:08:18.469443+0000","last_became_peered":"2024-09-21T10:08:18.469443+0000","last_unstale":"2024-09-21T10:17:24.741323+0000","last_undegraded":"2024-09-21T10:17:24.741323+0000","last_fullsized":"2024-09-21T10:17:24.741323+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:15.645803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":659,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.826757+0000","last_change":"2024-09-21T10:08:00.419641+0000","last_active":"2024-09-21T10:17:24.826757+0000","last_peered":"2024-09-21T10:17:24.826757+0000","last_clean":"2024-09-21T10:17:24.826757+0000","last_became_active":"2024-09-21T10:08:00.387050+0000","last_became_peered":"2024-09-21T10:08:00.387050+0000","last_unstale":"2024-09-21T10:17:24.826757+0000","last_undegraded":"2024-09-21T10:17:24.826757+0000","last_fullsized":"2024-09-21T10:17:24.826757+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:00:32.881619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":385,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.038468+0000","last_change":"2024-09-21T09:59:54.267798+0000","last_active":"2024-09-21T10:17:27.038468+0000","last_peered":"2024-09-21T10:17:27.038468+0000","last_clean":"2024-09-21T10:17:27.038468+0000","last_became_active":"2024-09-21T09:59:54.265615+0000","last_became_peered":"2024-09-21T09:59:54.265615+0000","last_unstale":"2024-09-21T10:17:27.038468+0000","last_undegraded":"2024-09-21T10:17:27.038468+0000","last_fullsized":"2024-09-21T10:17:27.038468+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:30.363872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":398,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.074081+0000","last_change":"2024-09-21T10:02:14.101943+0000","last_active":"2024-09-21T10:17:27.074081+0000","last_peered":"2024-09-21T10:17:27.074081+0000","last_clean":"2024-09-21T10:17:27.074081+0000","last_became_active":"2024-09-21T10:02:14.101488+0000","last_became_peered":"2024-09-21T10:02:14.101488+0000","last_unstale":"2024-09-21T10:17:27.074081+0000","last_undegraded":"2024-09-21T10:17:27.074081+0000","last_fullsized":"2024-09-21T10:17:27.074081+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:52:27.866414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":646,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.741860+0000","last_change":"2024-09-21T10:08:18.472876+0000","last_active":"2024-09-21T10:17:24.741860+0000","last_peered":"2024-09-21T10:17:24.741860+0000","last_clean":"2024-09-21T10:17:24.741860+0000","last_became_active":"2024-09-21T10:08:18.472385+0000","last_became_peered":"2024-09-21T10:08:18.472385+0000","last_unstale":"2024-09-21T10:17:24.741860+0000","last_undegraded":"2024-09-21T10:17:24.741860+0000","last_fullsized":"2024-09-21T10:17:24.741860+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:33.109956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":216,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.031314+0000","last_change":"2024-09-21T10:08:38.797352+0000","last_active":"2024-09-21T10:17:27.031314+0000","last_peered":"2024-09-21T10:17:27.031314+0000","last_clean":"2024-09-21T10:17:27.031314+0000","last_became_active":"2024-09-21T10:08:38.796950+0000","last_became_peered":"2024-09-21T10:08:38.796950+0000","last_unstale":"2024-09-21T10:17:27.031314+0000","last_undegraded":"2024-09-21T10:17:27.031314+0000","last_fullsized":"2024-09-21T10:17:27.031314+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:02.646321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":433,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.045580+0000","last_change":"2024-09-21T10:08:00.390436+0000","last_active":"2024-09-21T10:17:27.045580+0000","last_peered":"2024-09-21T10:17:27.045580+0000","last_clean":"2024-09-21T10:17:27.045580+0000","last_became_active":"2024-09-21T10:08:00.381835+0000","last_became_peered":"2024-09-21T10:08:00.381835+0000","last_unstale":"2024-09-21T10:17:27.045580+0000","last_undegraded":"2024-09-21T10:17:27.045580+0000","last_fullsized":"2024-09-21T10:17:27.045580+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:01:35.697356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":490,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:25.256112+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:17:25.256112+0000","last_peered":"2024-09-21T10:17:25.256112+0000","last_clean":"2024-09-21T10:17:25.256112+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:17:25.256112+0000","last_undegraded":"2024-09-21T10:17:25.256112+0000","last_fullsized":"2024-09-21T10:17:25.256112+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":128,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:25.256162+0000","last_change":"2024-09-21T10:08:00.385774+0000","last_active":"2024-09-21T10:17:25.256162+0000","last_peered":"2024-09-21T10:17:25.256162+0000","last_clean":"2024-09-21T10:17:25.256162+0000","last_became_active":"2024-09-21T10:08:00.385445+0000","last_became_peered":"2024-09-21T10:08:00.385445+0000","last_unstale":"2024-09-21T10:17:25.256162+0000","last_undegraded":"2024-09-21T10:17:25.256162+0000","last_fullsized":"2024-09-21T10:17:25.256162+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:42.442190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":187,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:25.255948+0000","last_change":"2024-09-21T10:08:00.379661+0000","last_active":"2024-09-21T10:17:25.255948+0000","last_peered":"2024-09-21T10:17:25.255948+0000","last_clean":"2024-09-21T10:17:25.255948+0000","last_became_active":"2024-09-21T10:08:00.379301+0000","last_became_peered":"2024-09-21T10:08:00.379301+0000","last_unstale":"2024-09-21T10:17:25.255948+0000","last_undegraded":"2024-09-21T10:17:25.255948+0000","last_fullsized":"2024-09-21T10:17:25.255948+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:15.649523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":211,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.826422+0000","last_change":"2024-09-21T10:08:00.410731+0000","last_active":"2024-09-21T10:17:24.826422+0000","last_peered":"2024-09-21T10:17:24.826422+0000","last_clean":"2024-09-21T10:17:24.826422+0000","last_became_active":"2024-09-21T10:08:00.380324+0000","last_became_peered":"2024-09-21T10:08:00.380324+0000","last_unstale":"2024-09-21T10:17:24.826422+0000","last_undegraded":"2024-09-21T10:17:24.826422+0000","last_fullsized":"2024-09-21T10:17:24.826422+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:47:34.960221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":276,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.740725+0000","last_change":"2024-09-21T10:08:18.470378+0000","last_active":"2024-09-21T10:17:24.740725+0000","last_peered":"2024-09-21T10:17:24.740725+0000","last_clean":"2024-09-21T10:17:24.740725+0000","last_became_active":"2024-09-21T10:08:18.469482+0000","last_became_peered":"2024-09-21T10:08:18.469482+0000","last_unstale":"2024-09-21T10:17:24.740725+0000","last_undegraded":"2024-09-21T10:17:24.740725+0000","last_fullsized":"2024-09-21T10:17:24.740725+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:35.628724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":280,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.826626+0000","last_change":"2024-09-21T10:08:00.403606+0000","last_active":"2024-09-21T10:17:24.826626+0000","last_peered":"2024-09-21T10:17:24.826626+0000","last_clean":"2024-09-21T10:17:24.826626+0000","last_became_active":"2024-09-21T10:08:00.379330+0000","last_became_peered":"2024-09-21T10:08:00.379330+0000","last_unstale":"2024-09-21T10:17:24.826626+0000","last_undegraded":"2024-09-21T10:17:24.826626+0000","last_fullsized":"2024-09-21T10:17:24.826626+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:07.226027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":176,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.826758+0000","last_change":"2024-09-21T10:07:43.791830+0000","last_active":"2024-09-21T10:17:24.826758+0000","last_peered":"2024-09-21T10:17:24.826758+0000","last_clean":"2024-09-21T10:17:24.826758+0000","last_became_active":"2024-09-21T10:07:43.791314+0000","last_became_peered":"2024-09-21T10:07:43.791314+0000","last_unstale":"2024-09-21T10:17:24.826758+0000","last_undegraded":"2024-09-21T10:17:24.826758+0000","last_fullsized":"2024-09-21T10:17:24.826758+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:37.086629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":354,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.070070+0000","last_change":"2024-09-21T10:02:14.100451+0000","last_active":"2024-09-21T10:17:27.070070+0000","last_peered":"2024-09-21T10:17:27.070070+0000","last_clean":"2024-09-21T10:17:27.070070+0000","last_became_active":"2024-09-21T10:02:14.100160+0000","last_became_peered":"2024-09-21T10:02:14.100160+0000","last_unstale":"2024-09-21T10:17:27.070070+0000","last_undegraded":"2024-09-21T10:17:27.070070+0000","last_fullsized":"2024-09-21T10:17:27.070070+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:28.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":395,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.043572+0000","last_change":"2024-09-21T10:02:14.096098+0000","last_active":"2024-09-21T10:17:27.043572+0000","last_peered":"2024-09-21T10:17:27.043572+0000","last_clean":"2024-09-21T10:17:27.043572+0000","last_became_active":"2024-09-21T10:02:14.095811+0000","last_became_peered":"2024-09-21T10:02:14.095811+0000","last_unstale":"2024-09-21T10:17:27.043572+0000","last_undegraded":"2024-09-21T10:17:27.043572+0000","last_fullsized":"2024-09-21T10:17:27.043572+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:54.619333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":399,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.058617+0000","last_change":"2024-09-21T10:02:14.100712+0000","last_active":"2024-09-21T10:17:27.058617+0000","last_peered":"2024-09-21T10:17:27.058617+0000","last_clean":"2024-09-21T10:17:27.058617+0000","last_became_active":"2024-09-21T10:02:14.100427+0000","last_became_peered":"2024-09-21T10:02:14.100427+0000","last_unstale":"2024-09-21T10:17:27.058617+0000","last_undegraded":"2024-09-21T10:17:27.058617+0000","last_fullsized":"2024-09-21T10:17:27.058617+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:16:42.350642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":227,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:25.256033+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:17:25.256033+0000","last_peered":"2024-09-21T10:17:25.256033+0000","last_clean":"2024-09-21T10:17:25.256033+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:17:25.256033+0000","last_undegraded":"2024-09-21T10:17:25.256033+0000","last_fullsized":"2024-09-21T10:17:25.256033+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":417,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.058711+0000","last_change":"2024-09-21T10:02:14.097369+0000","last_active":"2024-09-21T10:17:27.058711+0000","last_peered":"2024-09-21T10:17:27.058711+0000","last_clean":"2024-09-21T10:17:27.058711+0000","last_became_active":"2024-09-21T10:02:14.097067+0000","last_became_peered":"2024-09-21T10:02:14.097067+0000","last_unstale":"2024-09-21T10:17:27.058711+0000","last_undegraded":"2024-09-21T10:17:27.058711+0000","last_fullsized":"2024-09-21T10:17:27.058711+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:50:48.123813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":410,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.045844+0000","last_change":"2024-09-21T10:16:04.628509+0000","last_active":"2024-09-21T10:17:27.045844+0000","last_peered":"2024-09-21T10:17:27.045844+0000","last_clean":"2024-09-21T10:17:27.045844+0000","last_became_active":"2024-09-21T10:16:04.628325+0000","last_became_peered":"2024-09-21T10:16:04.628325+0000","last_unstale":"2024-09-21T10:17:27.045844+0000","last_undegraded":"2024-09-21T10:17:27.045844+0000","last_fullsized":"2024-09-21T10:17:27.045844+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:06:30.017700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":402,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.740475+0000","last_change":"2024-09-21T10:08:00.381058+0000","last_active":"2024-09-21T10:17:24.740475+0000","last_peered":"2024-09-21T10:17:24.740475+0000","last_clean":"2024-09-21T10:17:24.740475+0000","last_became_active":"2024-09-21T10:08:00.380573+0000","last_became_peered":"2024-09-21T10:08:00.380573+0000","last_unstale":"2024-09-21T10:17:24.740475+0000","last_undegraded":"2024-09-21T10:17:24.740475+0000","last_fullsized":"2024-09-21T10:17:24.740475+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:01:04.137425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":420,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.062055+0000","last_change":"2024-09-21T10:08:00.387321+0000","last_active":"2024-09-21T10:17:27.062055+0000","last_peered":"2024-09-21T10:17:27.062055+0000","last_clean":"2024-09-21T10:17:27.062055+0000","last_became_active":"2024-09-21T10:08:00.386935+0000","last_became_peered":"2024-09-21T10:08:00.386935+0000","last_unstale":"2024-09-21T10:17:27.062055+0000","last_undegraded":"2024-09-21T10:17:27.062055+0000","last_fullsized":"2024-09-21T10:17:27.062055+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:57.706313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":222,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.740954+0000","last_change":"2024-09-21T10:08:18.471282+0000","last_active":"2024-09-21T10:17:24.740954+0000","last_peered":"2024-09-21T10:17:24.740954+0000","last_clean":"2024-09-21T10:17:24.740954+0000","last_became_active":"2024-09-21T10:08:18.470952+0000","last_became_peered":"2024-09-21T10:08:18.470952+0000","last_unstale":"2024-09-21T10:17:24.740954+0000","last_undegraded":"2024-09-21T10:17:24.740954+0000","last_fullsized":"2024-09-21T10:17:24.740954+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:25.597081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":50,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.032385+0000","last_change":"2024-09-21T10:15:52.579747+0000","last_active":"2024-09-21T10:17:27.032385+0000","last_peered":"2024-09-21T10:17:27.032385+0000","last_clean":"2024-09-21T10:17:27.032385+0000","last_became_active":"2024-09-21T10:15:52.579386+0000","last_became_peered":"2024-09-21T10:15:52.579386+0000","last_unstale":"2024-09-21T10:17:27.032385+0000","last_undegraded":"2024-09-21T10:17:27.032385+0000","last_fullsized":"2024-09-21T10:17:27.032385+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:41:42.022025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":409,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.036319+0000","last_change":"2024-09-21T10:08:00.386104+0000","last_active":"2024-09-21T10:17:27.036319+0000","last_peered":"2024-09-21T10:17:27.036319+0000","last_clean":"2024-09-21T10:17:27.036319+0000","last_became_active":"2024-09-21T10:08:00.381484+0000","last_became_peered":"2024-09-21T10:08:00.381484+0000","last_unstale":"2024-09-21T10:17:27.036319+0000","last_undegraded":"2024-09-21T10:17:27.036319+0000","last_fullsized":"2024-09-21T10:17:27.036319+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:39.190191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":334,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.061864+0000","last_change":"2024-09-21T10:08:00.381149+0000","last_active":"2024-09-21T10:17:27.061864+0000","last_peered":"2024-09-21T10:17:27.061864+0000","last_clean":"2024-09-21T10:17:27.061864+0000","last_became_active":"2024-09-21T10:08:00.380795+0000","last_became_peered":"2024-09-21T10:08:00.380795+0000","last_unstale":"2024-09-21T10:17:27.061864+0000","last_undegraded":"2024-09-21T10:17:27.061864+0000","last_fullsized":"2024-09-21T10:17:27.061864+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:50:46.793182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":233,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:24.741587+0000","last_change":"2024-09-21T10:08:18.466882+0000","last_active":"2024-09-21T10:17:24.741587+0000","last_peered":"2024-09-21T10:17:24.741587+0000","last_clean":"2024-09-21T10:17:24.741587+0000","last_became_active":"2024-09-21T10:08:18.466470+0000","last_became_peered":"2024-09-21T10:08:18.466470+0000","last_unstale":"2024-09-21T10:17:24.741587+0000","last_undegraded":"2024-09-21T10:17:24.741587+0000","last_fullsized":"2024-09-21T10:17:24.741587+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:51:31.020051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":334,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.072997+0000","last_change":"2024-09-21T10:08:00.389061+0000","last_active":"2024-09-21T10:17:27.072997+0000","last_peered":"2024-09-21T10:17:27.072997+0000","last_clean":"2024-09-21T10:17:27.072997+0000","last_became_active":"2024-09-21T10:08:00.388657+0000","last_became_peered":"2024-09-21T10:08:00.388657+0000","last_unstale":"2024-09-21T10:17:27.072997+0000","last_undegraded":"2024-09-21T10:17:27.072997+0000","last_fullsized":"2024-09-21T10:17:27.072997+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:58:17.239589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":139,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:25.256148+0000","last_change":"2024-09-21T10:08:00.394350+0000","last_active":"2024-09-21T10:17:25.256148+0000","last_peered":"2024-09-21T10:17:25.256148+0000","last_clean":"2024-09-21T10:17:25.256148+0000","last_became_active":"2024-09-21T10:08:00.387232+0000","last_became_peered":"2024-09-21T10:08:00.387232+0000","last_unstale":"2024-09-21T10:17:25.256148+0000","last_undegraded":"2024-09-21T10:17:25.256148+0000","last_fullsized":"2024-09-21T10:17:25.256148+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:56.160018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":420,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.070379+0000","last_change":"2024-09-21T10:02:14.099799+0000","last_active":"2024-09-21T10:17:27.070379+0000","last_peered":"2024-09-21T10:17:27.070379+0000","last_clean":"2024-09-21T10:17:27.070379+0000","last_became_active":"2024-09-21T10:02:14.099375+0000","last_became_peered":"2024-09-21T10:02:14.099375+0000","last_unstale":"2024-09-21T10:17:27.070379+0000","last_undegraded":"2024-09-21T10:17:27.070379+0000","last_fullsized":"2024-09-21T10:17:27.070379+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:13.407638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":406,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.062867+0000","last_change":"2024-09-21T10:02:14.099439+0000","last_active":"2024-09-21T10:17:27.062867+0000","last_peered":"2024-09-21T10:17:27.062867+0000","last_clean":"2024-09-21T10:17:27.062867+0000","last_became_active":"2024-09-21T10:02:14.099117+0000","last_became_peered":"2024-09-21T10:02:14.099117+0000","last_unstale":"2024-09-21T10:17:27.062867+0000","last_undegraded":"2024-09-21T10:17:27.062867+0000","last_fullsized":"2024-09-21T10:17:27.062867+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:39.681817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":202,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-21T10:17:25.254942+0000","last_change":"2024-09-21T10:08:00.381529+0000","last_active":"2024-09-21T10:17:25.254942+0000","last_peered":"2024-09-21T10:17:25.254942+0000","last_clean":"2024-09-21T10:17:25.254942+0000","last_became_active":"2024-09-21T10:08:00.381061+0000","last_became_peered":"2024-09-21T10:08:00.381061+0000","last_unstale":"2024-09-21T10:17:25.254942+0000","last_undegraded":"2024-09-21T10:17:25.254942+0000","last_fullsized":"2024-09-21T10:17:25.254942+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:56:58.775586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":400,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-21T10:17:27.043455+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:17:27.043455+0000","last_peered":"2024-09-21T10:17:27.043455+0000","last_clean":"2024-09-21T10:17:27.043455+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:17:27.043455+0000","last_undegraded":"2024-09-21T10:17:27.043455+0000","last_fullsized":"2024-09-21T10:17:27.043455+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":9926,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":21480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5982,"num_read_kb":13005,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397832,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":405928,"kb_used_data":100252,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93339224,"statfs":{"total":95995035648,"available":95579365376,"internally_reserved":0,"allocated":102658048,"data_stored":101251941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594060,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":395876,"kb_used_data":90200,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93349276,"statfs":{"total":95995035648,"available":95589658624,"internally_reserved":0,"allocated":92364800,"data_stored":90962696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790290,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":615636,"kb_used_data":309956,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93129516,"statfs":{"total":95995035648,"available":95364624384,"internally_reserved":0,"allocated":317394944,"data_stored":316008253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986519,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":101,"seq":433791697081,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":553740,"kb_used_data":3264,"kb_used_omap":5,"kb_used_meta":550458,"kb_avail":93191412,"statfs":{"total":95995035648,"available":95428005888,"internally_reserved":0,"allocated":3342336,"data_stored":1959499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":563669776},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860608,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":566304,"kb_used_data":7376,"kb_used_omap":8,"kb_used_meta":558903,"kb_avail":93178848,"statfs":{"total":95995035648,"available":95415140352,"internally_reserved":0,"allocated":7553024,"data_stored":6159069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":572317548},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024134,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":607996,"kb_used_data":202860,"kb_used_omap":7,"kb_used_meta":405112,"kb_avail":93137156,"statfs":{"total":95995035648,"available":95372447744,"internally_reserved":0,"allocated":207728640,"data_stored":206357128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":414835250},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187669,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":820504,"kb_used_data":269836,"kb_used_omap":8,"kb_used_meta":548727,"kb_avail":92924648,"statfs":{"total":95995035648,"available":95154839552,"internally_reserved":0,"allocated":276312064,"data_stored":274918083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":561897457},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:17:33.915 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T10:17:33.916 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:17:33.916 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T10:17:33.916 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 1.000000 2024-09-21T10:17:33.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 3 1 2024-09-21T10:17:33.994 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:34.020 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:34.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:34.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:34.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:34.147 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:34.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:34.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:34.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:34.271 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:34.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:34.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:34.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:34.394 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:34.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:34.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:34.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:34.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:34.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:34.618 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:34.644 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:34.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:34.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:34.746 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:34.771 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:34.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:34.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:34.873 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:34.897 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:34.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:34.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:34.999 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:35.024 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:35.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:35.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:35.032 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:35.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:34 smithi098 ceph-mon[97580]: pgmap v813: 73 pgs: 73 active+clean; 4.6 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:35.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:34 smithi098 ceph-mon[100914]: pgmap v813: 73 pgs: 73 active+clean; 4.6 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:35.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:34 smithi196 ceph-mon[84784]: pgmap v813: 73 pgs: 73 active+clean; 4.6 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:35.147 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:35.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:35.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:35.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:35.273 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:35.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:35.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:35.375 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:35.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:35.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:35.501 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:35.523 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:35.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:35.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:35.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:35.646 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:35.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:35.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:35.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:35.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:35.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:35.810 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:35.873 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:35.899 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:35.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:35.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:36.001 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:36.027 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:36.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:36.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:36.129 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:36.154 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:36.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:36.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:36.256 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:36.281 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:36.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:36.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:36.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:36.405 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:36.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:36.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:36.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:36.528 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:36.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:36.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:36.629 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:36.655 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:36.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:36.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:36.756 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:36.777 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:36.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:36.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:36.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:36.904 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:36.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:36.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:37.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:37.026 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:36 smithi098 ceph-mon[97580]: pgmap v814: 73 pgs: 73 active+clean; 4.6 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:37.026 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:36 smithi098 ceph-mon[100914]: pgmap v814: 73 pgs: 73 active+clean; 4.6 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:37.030 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:37.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:37.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:37.061 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:17:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:17:37.075 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:17:37.075 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:17:37.093 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:17:37.093 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:17:37.109 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:17:37.109 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:17:37.124 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:17:37.125 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:17:37.132 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:37.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:36 smithi196 ceph-mon[84784]: pgmap v814: 73 pgs: 73 active+clean; 4.6 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:37.139 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:17:37.140 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:17:37.155 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:17:37.155 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:17:37.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:37.157 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:37.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:37.171 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:17:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:17:37.258 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:37.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:37.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:37.381 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:37.405 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:37.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:37.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:37.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:37.528 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:37.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:37.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:37.630 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:37.655 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:37.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:37.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:37.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:37.778 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:37.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:37.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:37.880 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:37.906 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:37.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:37.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:38.007 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:38.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:37 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2763656044' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:38.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:17:38.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:38.031 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:37 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2763656044' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:38.031 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:17:38.032 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:38.032 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:38.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:38.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:38.134 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:38.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:37 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2763656044' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:38.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:17:38.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:38.158 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:38.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:38.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:38.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:38.283 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:38.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:38.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:38.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:38.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:38.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:38.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:38.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:38.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:38.632 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:38.658 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:38.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:38.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:38.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:38.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:38.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:38.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:38.908 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:38.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:38.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:39.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:39.031 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:39.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:39.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:39.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:38 smithi196 ceph-mon[84784]: pgmap v815: 73 pgs: 73 active+clean; 4.6 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:39.045 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:38 smithi098 ceph-mon[97580]: pgmap v815: 73 pgs: 73 active+clean; 4.6 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:39.045 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:38 smithi098 ceph-mon[100914]: pgmap v815: 73 pgs: 73 active+clean; 4.6 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:39.133 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:39.154 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:39.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:39.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:39.256 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:39.277 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:39.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:39.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:39.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:39.401 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:39.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:39.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:39.502 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:39.525 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:39.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:39.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:39.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:39.649 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:39.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:39.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:39.751 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:39.756 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:17:39.756 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:17:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:39.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:39.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:39.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:39.895 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:39.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:39.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:39.919 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:17:39.998 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:40.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:40.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:40.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:40.143 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:40.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:40.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:40.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:40.265 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:40.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:40.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:40.367 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:40.388 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:40.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:40.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:40.490 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:40.502 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:17:40.511 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:40.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:40.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:40.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:40.634 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:40.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:40.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:40.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:17:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:17:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:17:40.736 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:40.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:40.759 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:40.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:40.763 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:17:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:17:40] "GET /metrics HTTP/1.1" 200 34166 "" "Prometheus/2.51.0" 2024-09-21T10:17:40.860 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:40.936 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:40.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:40.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:41.038 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:41.059 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:41.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:41.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:41.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:40 smithi098 ceph-mon[97580]: pgmap v816: 73 pgs: 73 active+clean; 4.6 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:41.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:40 smithi098 ceph-mon[100914]: pgmap v816: 73 pgs: 73 active+clean; 4.6 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:41.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:40 smithi196 ceph-mon[84784]: pgmap v816: 73 pgs: 73 active+clean; 4.6 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:41.161 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:41.183 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:41.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:41.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:41.285 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:41.307 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:41.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:41.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:41.409 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:41.437 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:41.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:41.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:41.539 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:41.565 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:41.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:41.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:41.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:41.693 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:41.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:41.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:41.794 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:41.820 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:41.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:41.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:41.923 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:41.948 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:41.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:41.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:42.050 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:42.076 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:42.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:42.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:42.178 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:42.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:42.202 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:42.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:42.303 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:42.328 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:42.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:42.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:42.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:42.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:42.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:42.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:42.581 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:42.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:42.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:42.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:42.704 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:42.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:42.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:42.789 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'unique_pool_3' created 2024-09-21T10:17:42.806 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:42.830 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:42.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:42.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:42.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:42 smithi098 ceph-mon[97580]: pgmap v817: 73 pgs: 73 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:42.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:42 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/605810851' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T10:17:42.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:42 smithi098 ceph-mon[100914]: pgmap v817: 73 pgs: 73 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:42.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:42 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/605810851' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T10:17:42.932 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:42.953 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:42.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:42.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:43.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:43.076 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:43.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:43.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:42 smithi196 ceph-mon[84784]: pgmap v817: 73 pgs: 73 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:42 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/605810851' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T10:17:43.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:43.204 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:43.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:43.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:43.306 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:43.331 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:43.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:43.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:43.432 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:43.458 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:43.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:43.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:43.465 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:43.560 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:43.585 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:43.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:43.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:43.688 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:43.713 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:43.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:43.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:43.796 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:17:43.814 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:43.839 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:43.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:43.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:43.936 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-21T10:17:43.940 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:43.965 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:43.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:43.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:44.066 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:44.088 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:44.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:44.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:44.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:43 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/605810851' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-21T10:17:44.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:43 smithi098 ceph-mon[97580]: osdmap e265: 8 total, 8 up, 7 in 2024-09-21T10:17:44.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:43 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1860562058' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:17:44.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:43 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:17:44.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:43 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:17:44.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:43 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:17:44.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:43 smithi098 ceph-mon[97580]: osdmap e266: 8 total, 8 up, 7 in 2024-09-21T10:17:44.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:43 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/605810851' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-21T10:17:44.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:43 smithi098 ceph-mon[100914]: osdmap e265: 8 total, 8 up, 7 in 2024-09-21T10:17:44.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:43 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1860562058' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:17:44.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:43 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:17:44.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:43 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:17:44.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:43 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:17:44.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:43 smithi098 ceph-mon[100914]: osdmap e266: 8 total, 8 up, 7 in 2024-09-21T10:17:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:43 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/605810851' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-21T10:17:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:43 smithi196 ceph-mon[84784]: osdmap e265: 8 total, 8 up, 7 in 2024-09-21T10:17:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:43 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1860562058' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:17:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:43 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:17:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:43 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:17:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:43 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:17:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:43 smithi196 ceph-mon[84784]: osdmap e266: 8 total, 8 up, 7 in 2024-09-21T10:17:44.189 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:44.215 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:44.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:44.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:44.317 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:44.339 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:44.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:44.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:44.440 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:44.467 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:44.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:44.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:44.569 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:44.590 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:44.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:44.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:44.691 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:44.717 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:44.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:44.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:44.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:44.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:44.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:44.920 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:44 smithi098 ceph-mon[97580]: pgmap v819: 89 pgs: 4 creating+peering, 12 unknown, 73 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:44.920 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:44 smithi098 ceph-mon[97580]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:17:44.920 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:44 smithi098 ceph-mon[97580]: osdmap e267: 8 total, 8 up, 7 in 2024-09-21T10:17:44.920 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:44 smithi098 ceph-mon[100914]: pgmap v819: 89 pgs: 4 creating+peering, 12 unknown, 73 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:44.920 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:44 smithi098 ceph-mon[100914]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:17:44.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:44 smithi098 ceph-mon[100914]: osdmap e267: 8 total, 8 up, 7 in 2024-09-21T10:17:44.943 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:44.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:44.971 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:44.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:45.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:45.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:45.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:45.116 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:44 smithi196 ceph-mon[84784]: pgmap v819: 89 pgs: 4 creating+peering, 12 unknown, 73 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:44 smithi196 ceph-mon[84784]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:17:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:44 smithi196 ceph-mon[84784]: osdmap e267: 8 total, 8 up, 7 in 2024-09-21T10:17:45.150 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:45.196 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:45.217 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:45.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:45.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:45.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:45.339 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:45.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:45.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:45.441 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:45.467 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:45.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:45.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:45.568 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:45.594 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:45.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:45.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:45.695 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:45.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:45.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:45.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:45.820 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-21T10:17:45.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:45.848 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:45.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:45.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:45.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:45.974 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:45.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:45.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:46.076 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:46.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:45 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2118708256' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:17:46.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:45 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:17:46.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:45 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2118708256' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:17:46.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:45 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:17:46.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:46.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:46.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:46.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:45 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2118708256' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:17:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:45 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:17:46.202 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:46.228 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:46.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:46.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:46.330 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:46.352 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:46.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:46.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:46.454 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:46.480 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:46.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:46.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:46.582 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:46.607 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:46.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:46.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:46.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:17:46.709 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:46.734 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:46.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:46.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:46.835 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:46.860 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:46.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:46.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:46.962 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:46.967 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:46 smithi098 ceph-mon[100914]: pgmap v822: 89 pgs: 1 creating+activating, 7 creating+peering, 8 unknown, 73 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:46.967 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:46 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:17:46.967 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:46 smithi098 ceph-mon[100914]: osdmap e268: 8 total, 8 up, 7 in 2024-09-21T10:17:46.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:46 smithi098 ceph-mon[97580]: pgmap v822: 89 pgs: 1 creating+activating, 7 creating+peering, 8 unknown, 73 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:46.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:46 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:17:46.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:46 smithi098 ceph-mon[97580]: osdmap e268: 8 total, 8 up, 7 in 2024-09-21T10:17:46.986 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:46.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:46.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:47.088 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:47.112 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:47.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:47.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:47.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:46 smithi196 ceph-mon[84784]: pgmap v822: 89 pgs: 1 creating+activating, 7 creating+peering, 8 unknown, 73 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:47.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:46 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:17:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:46 smithi196 ceph-mon[84784]: osdmap e268: 8 total, 8 up, 7 in 2024-09-21T10:17:47.214 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:47.238 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:47.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:47.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:47.340 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:47.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:47.365 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:47.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:47.466 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:47.491 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:47.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:47.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:47.592 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:47.617 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:47.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:47.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:47.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:47.742 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:47.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:47.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:47.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:47.870 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:47.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:47.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:47.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:47.995 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:47.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:47.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:48.097 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:48.121 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:48.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:48.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:48.223 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:48.247 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:48.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:48.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:48.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:48.371 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:48.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:48.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:48.473 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:48.498 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:48.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:48.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:48.601 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:48.626 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:48.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:48.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:48.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:48.754 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:48.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:48.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:48.856 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:48.882 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:48.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:48.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:48.984 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:49.007 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:49.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:49.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:49.033 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:48 smithi098 ceph-mon[100914]: pgmap v824: 89 pgs: 1 creating+activating, 7 creating+peering, 4 unknown, 77 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:49.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:48 smithi098 ceph-mon[97580]: pgmap v824: 89 pgs: 1 creating+activating, 7 creating+peering, 4 unknown, 77 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:49.109 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:49.132 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:49.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:49.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:48 smithi196 ceph-mon[84784]: pgmap v824: 89 pgs: 1 creating+activating, 7 creating+peering, 4 unknown, 77 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:49.187 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:49.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:49.256 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:49.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:49.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:49.279 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:17:49.280 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:17:49.297 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:17:49.298 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:17:49.313 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:17:49.314 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:17:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:17:49.329 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:17:49.345 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:17:49.345 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:17:49.358 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:49.361 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:17:49.361 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:17:49.377 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:17:49.377 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:17:49.384 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:49.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:49.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:49.392 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:17:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:17:49.486 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:49.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:49.509 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:49.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:49.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:49.631 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:49.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:49.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:49.732 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:49.759 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:49.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:49.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:49.860 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:49.881 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:49.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:49.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:49.983 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:50.005 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:50.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:50.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:50.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:49 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3577929524' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:50.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:49 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3577929524' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:50.107 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:50.128 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:50.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:50.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:50.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:49 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3577929524' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:50.231 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:50.252 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:50.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:50.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:50.353 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:50.376 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:50.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:50.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:50.478 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:50.500 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:50.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:50.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:50.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:50.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:50.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:50.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:17:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:17:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:17:50.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:50.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:50.746 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:50.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:50.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:17:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:17:50] "GET /metrics HTTP/1.1" 200 34166 "" "Prometheus/2.51.0" 2024-09-21T10:17:50.847 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:50.872 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:50.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:50.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:50.973 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:50.997 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:50.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:50.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:51.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:50 smithi098 ceph-mon[97580]: pgmap v825: 89 pgs: 89 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:51.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:50 smithi098 ceph-mon[100914]: pgmap v825: 89 pgs: 89 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:51.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:51.122 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:51.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:51.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:51.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:50 smithi196 ceph-mon[84784]: pgmap v825: 89 pgs: 89 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:51.224 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:51.246 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:51.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:51.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:51.348 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:51.374 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:51.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:51.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:51.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:51.498 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:51.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:51.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:51.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:51.625 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:51.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:51.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:51.726 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:51.747 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:51.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:51.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:51.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:51.870 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:51.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:51.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:51.969 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:17:51.970 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:17:51.970 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T10:17:51.970 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.859167 2024-09-21T10:17:51.970 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 0 0.8591666130377936 2024-09-21T10:17:51.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:51.997 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:51.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:51.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:52.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:52.124 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:52.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:52.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:52.226 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:52.249 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:52.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:52.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:52.351 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:52.376 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:52.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:52.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:52.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:52.500 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:52.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:52.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:52.602 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:52.628 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:52.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:52.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:52.665 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:17:52.666 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:17:52.666 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-21T10:17:52.666 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-21T10:17:52.666 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:17:52.730 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:52.752 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:52.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:52.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-21T10:17:52.854 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:52.880 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:52.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:52.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:52.949 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:17:52.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:52 smithi098 ceph-mon[97580]: pgmap v826: 89 pgs: 89 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:52.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:52 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2199236972' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:17:52.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:52 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:17:52.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:17:52.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:52.972 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:52 smithi098 ceph-mon[100914]: pgmap v826: 89 pgs: 89 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:52.972 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:52 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2199236972' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:17:52.972 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:52 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:17:52.972 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:17:52.972 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:52.981 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:53.006 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:53.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:53.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:53.108 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:53.133 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:53.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:53.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:53.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:52 smithi196 ceph-mon[84784]: pgmap v826: 89 pgs: 89 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:53.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:52 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2199236972' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:17:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:52 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:17:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:17:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:53.235 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:53.261 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:53.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:53.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:53.363 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:53.385 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:53.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:53.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:53.487 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:53.512 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:53.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:53.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:53.529 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:17:53.613 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:53.638 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:53.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:53.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:53.739 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:53.761 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:53.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:53.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:53.862 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:53.883 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:53.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:53.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:53.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:54.006 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:54.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:54.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:54.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:53 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:17:54.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:53 smithi098 ceph-mon[97580]: osdmap e269: 8 total, 8 up, 7 in 2024-09-21T10:17:54.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:53 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:17:54.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:53 smithi098 ceph-mon[100914]: osdmap e269: 8 total, 8 up, 7 in 2024-09-21T10:17:54.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:54.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:54.132 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:54.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:54.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:53 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:17:54.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:53 smithi196 ceph-mon[84784]: osdmap e269: 8 total, 8 up, 7 in 2024-09-21T10:17:54.233 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:54.258 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:54.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:54.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:54.360 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:54.385 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:54.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:54.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:54.487 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:54.512 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:54.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:54.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:54.636 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:54.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:54.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:54.737 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:54.764 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:54.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:54.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:54.865 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:54.890 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:54.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:54.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:54.974 DEBUG:teuthology.orchestra.run.smithi098:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-21T10:17:54.977 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:17:54.977 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:17:54.982 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:54 smithi098 ceph-mon[97580]: pgmap v828: 89 pgs: 89 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:54.982 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:54 smithi098 ceph-mon[97580]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:17:54.983 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:54 smithi098 ceph-mon[100914]: pgmap v828: 89 pgs: 89 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:54.983 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:54 smithi098 ceph-mon[100914]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:17:54.992 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:55.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:55.018 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:55.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:55.120 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:55.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:54 smithi196 ceph-mon[84784]: pgmap v828: 89 pgs: 89 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:17:55.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:54 smithi196 ceph-mon[84784]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:17:55.143 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:55.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:55.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:55.144 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:hints = 1 2024-09-21T10:17:55.144 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:17:55.144 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Object prefix: benchmark_data_smithi098_226587 2024-09-21T10:17:55.145 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:17:55.145 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:17:55.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:55.268 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:55.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:55.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:55.370 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:55.409 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:55.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:55.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:55.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:55.551 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:55.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:55.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:55.653 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:55.680 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:55.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:55.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:55.782 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:55.808 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:55.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:55.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:55.830 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:55.833 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:55.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:55 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2449419197' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:17:55.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:55 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:17:55.868 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:17:55.910 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:55.937 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:55.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:55.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:56.039 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:56.069 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:56.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:56.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:56.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:55 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2449419197' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:17:56.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:55 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:17:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:55 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2449419197' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:17:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:55 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:17:56.144 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 1 16 3218 3202 200.107 200.125 0.00168508 0.00477932 2024-09-21T10:17:56.171 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:56.209 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:56.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:56.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:56.311 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:56.338 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:56.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:56.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:56.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:56.476 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:56.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:56.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:56.578 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:56.606 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:56.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:56.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:56.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:56.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:56.745 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:56.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:56.846 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:56.883 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:56.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:56.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:56.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:56 smithi098 ceph-mon[97580]: pgmap v829: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:17:56.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:56 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:17:56.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:56 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:17:56.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:56 smithi098 ceph-mon[97580]: osdmap e270: 8 total, 8 up, 7 in 2024-09-21T10:17:56.933 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:56 smithi098 ceph-mon[100914]: pgmap v829: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:17:56.933 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:56 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:17:56.933 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:56 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:17:56.933 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:56 smithi098 ceph-mon[100914]: osdmap e270: 8 total, 8 up, 7 in 2024-09-21T10:17:56.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:57.018 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:57.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:57.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:57.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:57.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:56 smithi196 ceph-mon[84784]: pgmap v829: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail 2024-09-21T10:17:57.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:56 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:17:57.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:56 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:17:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:56 smithi196 ceph-mon[84784]: osdmap e270: 8 total, 8 up, 7 in 2024-09-21T10:17:57.144 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 2 16 6378 6362 198.794 197.5 0.00334844 0.0050192 2024-09-21T10:17:57.158 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:57.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:57.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:57.261 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:57.286 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:57.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:57.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:57.341 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:17:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:17:57.378 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:17:57.378 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:17:57.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:57.413 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:17:57.413 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:17:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:57.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:57.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:57.452 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:17:57.452 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:17:57.483 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:17:57.485 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:17:57.515 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:17:57.515 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:17:57.527 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:57.543 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:17:57.543 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:17:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:57.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:57.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:57.578 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:17:57.579 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:17:57.666 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:57.693 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:57.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:57.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:57.795 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:57.834 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:57.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:57.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:57.936 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:57.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:57.978 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:57.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:58.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:58.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:57 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2279588046' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:58.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:57 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:17:58.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:57 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2279588046' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:58.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:57 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:17:58.115 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:58.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:58.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:58.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:57 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2279588046' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:58.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:57 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:17:58.144 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 3 16 9191 9175 191.128 175.812 0.00180449 0.00522229 2024-09-21T10:17:58.217 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:58.245 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:58.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:58.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:58.347 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:58.373 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:58.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:58.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:58.475 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:58.511 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:58.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:58.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:58.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:58.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:58.650 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:58.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:58.751 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:58.778 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:58.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:58.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:58.880 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:58.908 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:58.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:58.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:59.010 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:58 smithi098 ceph-mon[100914]: pgmap v831: 89 pgs: 89 active+clean; 27 MiB data, 3.3 GiB used, 622 GiB / 626 GiB avail; 2.8 MiB/s wr, 44 op/s 2024-09-21T10:17:59.010 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:17:58 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:17:59.010 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:58 smithi098 ceph-mon[97580]: pgmap v831: 89 pgs: 89 active+clean; 27 MiB data, 3.3 GiB used, 622 GiB / 626 GiB avail; 2.8 MiB/s wr, 44 op/s 2024-09-21T10:17:59.010 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:17:58 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:17:59.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:59.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:59.054 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:59.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:59.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:58 smithi196 ceph-mon[84784]: pgmap v831: 89 pgs: 89 active+clean; 27 MiB data, 3.3 GiB used, 622 GiB / 626 GiB avail; 2.8 MiB/s wr, 44 op/s 2024-09-21T10:17:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:17:58 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:17:59.144 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 4 16 12010 11994 187.388 176.188 0.00399562 0.00532816 2024-09-21T10:17:59.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:59.196 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:59.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:59.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:59.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:59.335 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:59.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:59.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:59.436 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:59.474 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:17:59.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:59.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:59.576 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:59.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:59.608 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:59.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:59.709 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:59.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:59.740 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:59.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:59.841 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:59.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:59.875 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:17:59.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:59.976 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:00.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:00.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:00.116 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:00.141 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:00.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:00.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:00.144 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 5 16 14818 14802 185.008 175.5 0.00729641 0.00539879 2024-09-21T10:18:00.172 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:18:00.243 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:00.270 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:00.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:00.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:00.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:00 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:00.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:00 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:00.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:00 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:00.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:00 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:00.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:00.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:00 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:00.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:00 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:00.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:18:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:18:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:18:00.398 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:00.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:00.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:00.501 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:00.528 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:00.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:00.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:00.630 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:00.675 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:00.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:00.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:00.777 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:00.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:00.812 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:00.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:00.842 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:18:00.887 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:18:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:18:00] "GET /metrics HTTP/1.1" 200 36456 "" "Prometheus/2.51.0" 2024-09-21T10:18:00.913 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:00.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:00.958 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:00.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:01.058 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:01.082 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:01.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:01.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:01.146 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 6 16 17050 17034 177.421 139.5 0.00441034 0.00562543 2024-09-21T10:18:01.184 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:01.204 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:01.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:01.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:01.306 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:01.336 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:01.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:01.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:01.438 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:01.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:01.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:01.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:01 smithi098 ceph-mon[97580]: pgmap v832: 89 pgs: 89 active+clean; 450 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 56 MiB/s wr, 889 op/s 2024-09-21T10:18:01.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:01 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:01.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:01 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:01.499 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:01 smithi098 ceph-mon[100914]: pgmap v832: 89 pgs: 89 active+clean; 450 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 56 MiB/s wr, 889 op/s 2024-09-21T10:18:01.499 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:01 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:01.499 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:01 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:01.583 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:01.609 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:01.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:01.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:01.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:01 smithi196 ceph-mon[84784]: pgmap v832: 89 pgs: 89 active+clean; 450 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 56 MiB/s wr, 889 op/s 2024-09-21T10:18:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:01 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:01 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:01.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:01.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:01.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:01.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:01.890 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:01.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:01.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:01.992 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:02.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:02.033 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:02.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:02.134 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:02.146 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 7 16 19152 19136 170.838 131.375 0.00396702 0.00584763 2024-09-21T10:18:02.171 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:02.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:02.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:02.272 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:02.306 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:02.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:02.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:02.407 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:02.443 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:02.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:02.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:02.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:02.565 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:02.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:02.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:02.667 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:02.696 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:02.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:02.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:02.798 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:02.837 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:02.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:02.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:02.938 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:02.960 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:02.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:02.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:03.062 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:03.099 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:03.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:03.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:03.145 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 8 16 21211 21195 165.568 128.688 0.00481961 0.00603334 2024-09-21T10:18:03.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:03.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:03.237 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:03.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:03.338 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:03.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:03.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:03.464 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:03.493 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:03.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:03.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:03.596 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:03.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:03 smithi098 ceph-mon[97580]: pgmap v833: 89 pgs: 89 active+clean; 672 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:18:03.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:03 smithi098 ceph-mon[100914]: pgmap v833: 89 pgs: 89 active+clean; 672 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:18:03.624 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:03.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:03.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:03.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:03 smithi196 ceph-mon[84784]: pgmap v833: 89 pgs: 89 active+clean; 672 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:18:03.726 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:03.752 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:03.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:03.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:03.855 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:03.882 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:03.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:03.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:03.984 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:04.005 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:04.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:04.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:04.107 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:04.137 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:04.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:04.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:04.145 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 9 16 22842 22826 158.497 101.938 0.00679407 0.00630256 2024-09-21T10:18:04.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:04.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:04.277 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:04.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:04.335 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:18:04.336 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:18:04.378 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:04.399 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:04.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:04.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:04.500 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:04.526 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:04.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:04.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:04.630 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:04.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:04.655 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:04.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:04.756 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:04.783 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:04.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:04.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:04.885 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:04.916 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:04.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:04.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:05.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:05.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:05.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:05.145 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 10 16 24816 24800 154.984 123.375 0.00662762 0.00644616 2024-09-21T10:18:05.154 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:05.181 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:05.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:05.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:05.283 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:05.315 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:05.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:05.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:05.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:05 smithi196 ceph-mon[84784]: pgmap v834: 89 pgs: 89 active+clean; 1.0 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-21T10:18:05.417 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:05.439 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:05.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:05.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:05.541 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:05.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:05.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:05.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:05.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:05 smithi098 ceph-mon[97580]: pgmap v834: 89 pgs: 89 active+clean; 1.0 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-21T10:18:05.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:05 smithi098 ceph-mon[100914]: pgmap v834: 89 pgs: 89 active+clean; 1.0 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-21T10:18:05.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:05.702 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:05.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:05.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:05.804 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:05.830 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:05.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:05.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:05.932 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:05.960 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:05.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:05.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:06.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:06.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:06.093 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:06.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:06.145 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 11 16 26675 26659 151.456 116.188 0.0143847 0.00659703 2024-09-21T10:18:06.194 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:06.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:06.231 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:06.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:06.332 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:06.357 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:06.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:06.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:06.458 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:06.479 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:06.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:06.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:06.581 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:06.609 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:06.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:06.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:06.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:06 smithi196 ceph-mon[84784]: pgmap v835: 89 pgs: 89 active+clean; 1.2 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-09-21T10:18:06.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:06 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:06 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:06.711 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:06.739 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:06.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:06.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:06.795 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:06.842 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:06.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:06 smithi098 ceph-mon[97580]: pgmap v835: 89 pgs: 89 active+clean; 1.2 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-09-21T10:18:06.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:06 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:06.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:06 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:06.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:06 smithi098 ceph-mon[100914]: pgmap v835: 89 pgs: 89 active+clean; 1.2 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-09-21T10:18:06.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:06 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:06.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:06 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:06.864 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:06.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:06.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:06.966 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:07.009 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:07.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:07.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:07.111 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:07.145 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 12 16 28588 28572 148.798 119.562 0.00673655 0.0067147 2024-09-21T10:18:07.153 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:07.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:07.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:07.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:07.283 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:07.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:07.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:07.385 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:07.412 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:07.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:07.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:07.514 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:07.545 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:07.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:07.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:07.646 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:07.684 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:07.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:07.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:07.786 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:07.815 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:07.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:07.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:07.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:07.954 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:07.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:07.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:08.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:08.077 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:08.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:08.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:08.145 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 13 16 30339 30323 145.769 109.438 0.00676004 0.00685485 2024-09-21T10:18:08.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:08.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:08.217 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:08.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:08.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:08.346 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:08.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:08.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:08.447 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:08.480 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:08.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:08.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:08.582 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:08 smithi098 ceph-mon[97580]: pgmap v836: 89 pgs: 89 active+clean; 1.4 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-09-21T10:18:08.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:08.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:08.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:08 smithi098 ceph-mon[100914]: pgmap v836: 89 pgs: 89 active+clean; 1.4 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-09-21T10:18:08.607 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:08.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:08.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:08.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:08 smithi196 ceph-mon[84784]: pgmap v836: 89 pgs: 89 active+clean; 1.4 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-09-21T10:18:08.709 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:08.738 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:08.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:08.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:08.840 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:08.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:08.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:08.982 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:09.008 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:09.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:09.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:09.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:09.131 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:09.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:09.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:09.145 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 14 16 32076 32060 143.111 108.562 0.00217298 0.00693069 2024-09-21T10:18:09.233 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.0 primary-affinity to 0.859167 (8563062) 2024-09-21T10:18:09.276 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:09.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:09.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:09.378 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:09.406 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:09.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:09.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:09.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:09.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:09 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3174091145' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.8591666130377936} : dispatch 2024-09-21T10:18:09.558 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:09.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:09.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:09.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:09 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3174091145' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.8591666130377936} : dispatch 2024-09-21T10:18:09.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:09 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3174091145' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.8591666130377936} : dispatch 2024-09-21T10:18:09.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:09.690 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:09.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:09.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:09.792 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:09.822 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:09.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:09.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:09.923 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:09.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:09.963 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:09.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:10.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:10.105 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:10.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:10.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:10.145 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 15 16 33738 33722 140.494 103.875 0.00709349 0.00711264 2024-09-21T10:18:10.206 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:10.234 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:10.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:10.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:10.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:10.358 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:10.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:10.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:10 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3174091145' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.8591666130377936}]': finished 2024-09-21T10:18:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:10 smithi196 ceph-mon[84784]: osdmap e271: 8 total, 8 up, 7 in 2024-09-21T10:18:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:10 smithi196 ceph-mon[84784]: pgmap v838: 89 pgs: 89 active+clean; 1.8 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-09-21T10:18:10.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:18:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:18:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:18:10.459 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:10.493 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:10.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:10.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:10.595 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:10.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:10 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3174091145' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.8591666130377936}]': finished 2024-09-21T10:18:10.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:10 smithi098 ceph-mon[97580]: osdmap e271: 8 total, 8 up, 7 in 2024-09-21T10:18:10.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:10 smithi098 ceph-mon[97580]: pgmap v838: 89 pgs: 89 active+clean; 1.8 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-09-21T10:18:10.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:10 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3174091145' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.8591666130377936}]': finished 2024-09-21T10:18:10.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:10 smithi098 ceph-mon[100914]: osdmap e271: 8 total, 8 up, 7 in 2024-09-21T10:18:10.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:10 smithi098 ceph-mon[100914]: pgmap v838: 89 pgs: 89 active+clean; 1.8 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-09-21T10:18:10.635 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:10.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:10.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:10.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:10.772 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:10.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:10.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:10.874 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:10.908 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:10.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:10.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:11.009 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:11.031 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:11.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:11.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:11.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:18:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:18:10] "GET /metrics HTTP/1.1" 200 36513 "" "Prometheus/2.51.0" 2024-09-21T10:18:11.132 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:11.145 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 16 16 35328 35312 137.924 99.375 0.203718 0.00716746 2024-09-21T10:18:11.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:11.157 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:11.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:11.258 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:11.298 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:11.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:11.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:11.400 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:11.428 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:11.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:11.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:11.530 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:11.556 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:11.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:11.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:11.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:11 smithi098 ceph-mon[97580]: osdmap e272: 8 total, 8 up, 7 in 2024-09-21T10:18:11.628 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:11 smithi098 ceph-mon[100914]: osdmap e272: 8 total, 8 up, 7 in 2024-09-21T10:18:11.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:11 smithi196 ceph-mon[84784]: osdmap e272: 8 total, 8 up, 7 in 2024-09-21T10:18:11.658 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:11.682 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:11.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:11.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:11.784 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:11.811 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:11.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:11.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:11.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:11.941 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:11.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:11.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:12.020 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:18:12.043 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:12.071 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:12.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:12.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:12.146 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 17 16 37167 37151 136.571 114.938 0.00706005 0.00731715 2024-09-21T10:18:12.173 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:12.234 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:12.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:12.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:12.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:12.382 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:12.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:12.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:12.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:12.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:12.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:12.566 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:12 smithi098 ceph-mon[97580]: pgmap v840: 89 pgs: 89 active+clean; 1.9 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-21T10:18:12.566 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:12 smithi098 ceph-mon[100914]: pgmap v840: 89 pgs: 89 active+clean; 1.9 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-21T10:18:12.595 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:12.629 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:12.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:12 smithi196 ceph-mon[84784]: pgmap v840: 89 pgs: 89 active+clean; 1.9 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-21T10:18:12.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:12.668 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:12.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:12.769 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:12.798 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:12.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:12.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:12.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:12.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:12.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:13.037 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:13.065 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:13.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:13.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:13.146 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 18 16 38957 38941 135.199 111.875 0.00452927 0.0073906 2024-09-21T10:18:13.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:13.202 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:13.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:13.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:13.304 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:13.333 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:13.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:13.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:13.435 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:13.461 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:13.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:13.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:13.562 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:13.590 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:13.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:13.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:13.692 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:13.716 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:13.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:13.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:13.818 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:13.853 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:13.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:13.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:13.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:13.983 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:13.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:13.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:14.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:14.122 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:14.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:14.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:14.146 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 19 16 40776 40760 134.066 113.688 0.00912997 0.00745356 2024-09-21T10:18:14.224 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:14.252 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:14.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:14.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:14.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:14.395 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:14.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:14.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:14.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:14.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:14.532 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:14.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:14.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:14 smithi098 ceph-mon[97580]: pgmap v841: 89 pgs: 89 active+clean; 2.2 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 124 MiB/s wr, 1.99k op/s 2024-09-21T10:18:14.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:14 smithi098 ceph-mon[100914]: pgmap v841: 89 pgs: 89 active+clean; 2.2 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 124 MiB/s wr, 1.99k op/s 2024-09-21T10:18:14.633 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:14.670 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:14.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:14.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:14.772 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:14.793 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:14.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:14.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:14 smithi196 ceph-mon[84784]: pgmap v841: 89 pgs: 89 active+clean; 2.2 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 124 MiB/s wr, 1.99k op/s 2024-09-21T10:18:14.895 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:14.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:14.926 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:14.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:15.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:15.062 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:15.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:15.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:15.146 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:18:15.145017+0000 min lat: 0.00125976 max lat: 0.217359 avg lat: 0.00753046 2024-09-21T10:18:15.146 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:18:15.146 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 20 16 42479 42463 132.684 106.438 0.0116405 0.00753046 2024-09-21T10:18:15.163 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:15.199 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:15.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:15.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:15.301 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:15.329 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:15.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:15.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:15.430 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:15.459 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:15.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:15.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:15.561 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:15.596 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:15.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:15.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:15.668 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:15.698 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:15.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:15.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:15.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:15.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:15.857 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:15.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:15.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:15.958 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:15.987 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:15.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:15.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:16.089 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:16.117 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:16.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:16.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:16.146 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 21 16 44230 44214 131.577 109.438 0.00176823 0.0075877 2024-09-21T10:18:16.218 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:16.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:16.258 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:16.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:16.358 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:16.394 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:16.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:16.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:16.496 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:16.519 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:16.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:16.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:16.621 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:16.650 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:16.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:16.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:16.752 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:16.772 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:16.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:16.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:16 smithi098 ceph-mon[97580]: pgmap v842: 89 pgs: 89 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-21T10:18:16.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:16 smithi098 ceph-mon[100914]: pgmap v842: 89 pgs: 89 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-21T10:18:16.874 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:16 smithi196 ceph-mon[84784]: pgmap v842: 89 pgs: 89 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-21T10:18:16.912 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:16.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:16.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:17.013 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:17.042 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:17.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:17.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:17.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:17.147 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 22 16 45726 45710 129.846 93.5 0.00207249 0.00766718 2024-09-21T10:18:17.170 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:17.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:17.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:17.272 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:17.298 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:17.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:17.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:17.400 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:17.432 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:17.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:17.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:17.533 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:17.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:17.556 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:17.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:17.657 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:17.682 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:17.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:17.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:17.784 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:17.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:17.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:17.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:17.914 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:17.968 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:17.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:17.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:18.070 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:18.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:18.117 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:18.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:18.146 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 23 16 47154 47138 128.078 89.25 0.00891746 0.0078024 2024-09-21T10:18:18.218 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:18.243 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:18.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:18.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:18.345 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:18.368 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:18.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:18.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:18.472 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:18.509 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:18.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:18.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:18.610 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:18.637 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:18.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:18.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:18.699 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:18 smithi098 ceph-mon[97580]: pgmap v843: 89 pgs: 89 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-21T10:18:18.699 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:18 smithi098 ceph-mon[100914]: pgmap v843: 89 pgs: 89 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-21T10:18:18.739 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:18.776 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:18.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:18.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:18.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:18.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:18 smithi196 ceph-mon[84784]: pgmap v843: 89 pgs: 89 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-21T10:18:18.907 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:18.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:18.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:19.008 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:19.045 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:19.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:19.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:19.146 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:19.147 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 24 16 48677 48661 126.707 95.1875 0.0073711 0.00788637 2024-09-21T10:18:19.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:19.186 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:19.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:19.287 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:19.316 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:19.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:19.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:19.373 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:18:19.373 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:18:19.373 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 5, 6, 7] 2024-09-21T10:18:19.373 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd out 3 2024-09-21T10:18:19.419 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:19.443 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:19.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:19.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:19.446 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:18:19.545 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:19.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:19.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:19.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:19.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:19.715 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:19.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:19.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:19.816 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:19.842 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:19.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:19.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:19.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:19 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2274392393' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:18:19.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:19 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:18:19.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:19 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2274392393' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:18:19.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:19 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:18:19.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:19 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2274392393' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:18:19.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:19 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:18:19.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:19.973 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:19.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:19.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:20.075 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:20.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:20.116 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:20.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:20.147 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 25 16 50154 50138 125.331 92.3125 0.00982522 0.00797291 2024-09-21T10:18:20.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:20.263 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:20.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:20.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:20.366 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:20.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:20.411 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:20.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:20.511 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:20.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:20.538 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:20.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:20 smithi196 ceph-mon[84784]: pgmap v844: 89 pgs: 89 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-09-21T10:18:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:20 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:18:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:20 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:18:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:20 smithi196 ceph-mon[84784]: osdmap e273: 8 total, 8 up, 7 in 2024-09-21T10:18:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:20 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4111792669' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:18:20.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:18:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:18:20.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:18:20.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:20.678 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:20.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:20.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:20.722 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:20 smithi098 ceph-mon[97580]: pgmap v844: 89 pgs: 89 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-09-21T10:18:20.722 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:20 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:18:20.722 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:20 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:18:20.722 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:20 smithi098 ceph-mon[97580]: osdmap e273: 8 total, 8 up, 7 in 2024-09-21T10:18:20.723 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:20 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4111792669' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:18:20.723 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:18:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:18:20] "GET /metrics HTTP/1.1" 200 36513 "" "Prometheus/2.51.0" 2024-09-21T10:18:20.723 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:20 smithi098 ceph-mon[100914]: pgmap v844: 89 pgs: 89 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-09-21T10:18:20.723 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:20 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:18:20.723 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:20 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:18:20.723 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:20 smithi098 ceph-mon[100914]: osdmap e273: 8 total, 8 up, 7 in 2024-09-21T10:18:20.723 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:20 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4111792669' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:18:20.780 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:20.819 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:20.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:20.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:20.913 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:18:20.913 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:18:20.913 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:18:20.913 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:18:20.913 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:18:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:18:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:18:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:18:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:18:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:18:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:18:20.914 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:18:20.914 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:18:20.914 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:18:20.915 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:18:20.915 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:18:20.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:20.958 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:20.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:20.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:21.060 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:21.094 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:21.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:21.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:21.147 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 26 16 51721 51705 124.277 97.9375 0.0125122 0.00804152 2024-09-21T10:18:21.196 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:21.226 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:21.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:21.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:21.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:21.366 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:21.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:21.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:21.468 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:21.496 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:21.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:21.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:21.597 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:21.647 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:21.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:21.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:21.749 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:21.776 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:21.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:21.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:21.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:21.906 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:21.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:21.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:22.008 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:22.029 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:22.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:22.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:22.131 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:22.147 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 27 16 53114 53098 122.899 87.0625 0.00572341 0.00813123 2024-09-21T10:18:22.157 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:22.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:22.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:22.259 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:22.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:22.293 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:22.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:22.394 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:22.428 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:22.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:22.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:22.530 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:22.556 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:22.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:22.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:22.657 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:22.698 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:22.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:22.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:22.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:18:22.795 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:22 smithi098 ceph-mon[97580]: pgmap v846: 89 pgs: 89 active+clean; 2.9 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-21T10:18:22.795 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:22.795 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:22 smithi098 ceph-mon[100914]: pgmap v846: 89 pgs: 89 active+clean; 2.9 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-21T10:18:22.795 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:22.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:22.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:22.852 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:22.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:22 smithi196 ceph-mon[84784]: pgmap v846: 89 pgs: 89 active+clean; 2.9 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-21T10:18:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:22.953 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:22.979 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:22.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:22.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:23.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:23.109 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:23.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:23.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:23.147 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 28 16 54500 54484 121.603 86.625 0.00774027 0.00821867 2024-09-21T10:18:23.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:23.248 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:23.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:23.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:23.349 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:23.387 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:23.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:23.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:23.488 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:23.509 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:23.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:23.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:23.611 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:23.650 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:23.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:23.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:23.752 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:23.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:23.781 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:23.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:23.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:23.903 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:23.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:23.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:24.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:24.040 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:24.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:24.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:24.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:24.147 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 29 16 55521 55505 119.608 63.8125 0.0178986 0.00835469 2024-09-21T10:18:24.177 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:24.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:24.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:24.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:24.302 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:24.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:24.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:24.404 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:24.433 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:24.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:24.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:24.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:24.561 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:24.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:24.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:24.663 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:24.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:24.698 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:24.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:24.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:24.830 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:24.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:24.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:24 smithi098 ceph-mon[97580]: pgmap v847: 89 pgs: 89 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:18:24.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:24 smithi098 ceph-mon[100914]: pgmap v847: 89 pgs: 89 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:18:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:24 smithi196 ceph-mon[84784]: pgmap v847: 89 pgs: 89 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:18:24.932 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:24.971 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:24.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:24.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:25.073 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:25.107 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:25.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:25.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:25.147 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 30 16 56809 56793 118.305 80.5 0.0203857 0.00844702 2024-09-21T10:18:25.209 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:25.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:25.238 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:25.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:25.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:25.366 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:25.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:25.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:25.468 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:25.492 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:25.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:25.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:25.593 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:25.619 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:25.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:25.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:25.721 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:25.746 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:25.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:25.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:25.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:25.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:25.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:25.998 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:26.033 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:26.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:26.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:26.061 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:18:26.061 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:18:26.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.062 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:58021 8.0 8:0e0b3fca:::benchmark_data_smithi098_226587_object58020:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e273)", 2024-09-21T10:18:26.062 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:26.040559+0000", 2024-09-21T10:18:26.062 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.017932378999999998, 2024-09-21T10:18:26.062 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017965037999999999, 2024-09-21T10:18:26.062 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:18:26.062 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:18:26.062 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:18:26.062 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:18:26.062 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:18:26.062 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:18:26.062 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 58021 2024-09-21T10:18:26.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.063 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:18:26.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.063 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:18:26.063 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.040559+0000", 2024-09-21T10:18:26.063 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:18:26.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.063 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:18:26.063 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.040559+0000", 2024-09-21T10:18:26.063 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:18:26.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.063 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:18:26.063 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.040561+0000", 2024-09-21T10:18:26.063 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.573e-06 2024-09-21T10:18:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:18:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.040584+0000", 2024-09-21T10:18:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2609999999999999e-05 2024-09-21T10:18:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:18:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.040593+0000", 2024-09-21T10:18:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8090000000000004e-06 2024-09-21T10:18:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:18:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.040600+0000", 2024-09-21T10:18:26.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4869999999999999e-06 2024-09-21T10:18:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:18:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.040859+0000", 2024-09-21T10:18:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00025867199999999999 2024-09-21T10:18:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:18:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.040978+0000", 2024-09-21T10:18:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011951900000000001 2024-09-21T10:18:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:18:26.065 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.041103+0000", 2024-09-21T10:18:26.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000124556 2024-09-21T10:18:26.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.066 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:26.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.043643+0000", 2024-09-21T10:18:26.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002540535 2024-09-21T10:18:26.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.066 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:26.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.049569+0000", 2024-09-21T10:18:26.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0059261460000000002 2024-09-21T10:18:26.066 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:18:26.066 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:18:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:18:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:58022 8.0 8:0b2b9fca:::benchmark_data_smithi098_226587_object58021:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e273)", 2024-09-21T10:18:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:26.042664+0000", 2024-09-21T10:18:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.015827733, 2024-09-21T10:18:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015941882000000001, 2024-09-21T10:18:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:18:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:18:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:18:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:18:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:18:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:18:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 58022 2024-09-21T10:18:26.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:18:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:18:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.042664+0000", 2024-09-21T10:18:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:18:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:18:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.042664+0000", 2024-09-21T10:18:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:18:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:18:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.042666+0000", 2024-09-21T10:18:26.068 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5910000000000001e-06 2024-09-21T10:18:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:18:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.042687+0000", 2024-09-21T10:18:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0939000000000001e-05 2024-09-21T10:18:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:18:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.042699+0000", 2024-09-21T10:18:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2401e-05 2024-09-21T10:18:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:18:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.042706+0000", 2024-09-21T10:18:26.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2400000000000001e-06 2024-09-21T10:18:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:18:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.043670+0000", 2024-09-21T10:18:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000963448 2024-09-21T10:18:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:18:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.045893+0000", 2024-09-21T10:18:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002223549 2024-09-21T10:18:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:18:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.046031+0000", 2024-09-21T10:18:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013727100000000001 2024-09-21T10:18:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.050788+0000", 2024-09-21T10:18:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0047579459999999999 2024-09-21T10:18:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:18:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:18:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:18:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:58024 8.8 8:1ede85a9:::benchmark_data_smithi098_226587_object58023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e273)", 2024-09-21T10:18:26.071 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:26.042721+0000", 2024-09-21T10:18:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.015770448999999999, 2024-09-21T10:18:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01595624, 2024-09-21T10:18:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:18:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:18:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:18:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:18:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:18:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:18:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 58024 2024-09-21T10:18:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:18:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.072 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:18:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.042721+0000", 2024-09-21T10:18:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:18:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:18:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.042721+0000", 2024-09-21T10:18:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:18:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:18:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.042722+0000", 2024-09-21T10:18:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9500000000000003e-07 2024-09-21T10:18:26.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:18:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.042741+0000", 2024-09-21T10:18:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9015999999999999e-05 2024-09-21T10:18:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:18:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.042747+0000", 2024-09-21T10:18:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6050000000000003e-06 2024-09-21T10:18:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:18:26.074 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.042752+0000", 2024-09-21T10:18:26.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9030000000000003e-06 2024-09-21T10:18:26.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.075 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:18:26.075 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.043880+0000", 2024-09-21T10:18:26.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011272540000000001 2024-09-21T10:18:26.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:18:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.044295+0000", 2024-09-21T10:18:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00041491800000000001 2024-09-21T10:18:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,7", 2024-09-21T10:18:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.044427+0000", 2024-09-21T10:18:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000132387 2024-09-21T10:18:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.050828+0000", 2024-09-21T10:18:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0064014559999999998 2024-09-21T10:18:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.051362+0000", 2024-09-21T10:18:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000533499 2024-09-21T10:18:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:18:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:18:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:18:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:58028 8.0 8:0eaa2e5d:::benchmark_data_smithi098_226587_object58027:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e273)", 2024-09-21T10:18:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:26.047178+0000", 2024-09-21T10:18:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.011314189000000001, 2024-09-21T10:18:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011558624999999999, 2024-09-21T10:18:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:18:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:18:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:18:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:18:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:18:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:18:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 58028 2024-09-21T10:18:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:18:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:18:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.047178+0000", 2024-09-21T10:18:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:18:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:18:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.047178+0000", 2024-09-21T10:18:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:18:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:18:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.047179+0000", 2024-09-21T10:18:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.44e-06 2024-09-21T10:18:26.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:18:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.047202+0000", 2024-09-21T10:18:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2586999999999999e-05 2024-09-21T10:18:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:18:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.047212+0000", 2024-09-21T10:18:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0402e-05 2024-09-21T10:18:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:18:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.047219+0000", 2024-09-21T10:18:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5239999999999997e-06 2024-09-21T10:18:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:18:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.050858+0000", 2024-09-21T10:18:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036398759999999998 2024-09-21T10:18:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:18:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.050991+0000", 2024-09-21T10:18:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000132578 2024-09-21T10:18:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:18:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.051474+0000", 2024-09-21T10:18:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000483432 2024-09-21T10:18:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:18:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:18:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:18:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.45794.0:58028 8.0 e273/265)", 2024-09-21T10:18:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:26.053190+0000", 2024-09-21T10:18:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0053018809999999996, 2024-09-21T10:18:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055771919999999999, 2024-09-21T10:18:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:18:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:18:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-21T10:18:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:18:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:18:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.053190+0000", 2024-09-21T10:18:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:18:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:18:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.053190+0000", 2024-09-21T10:18:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:18:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:18:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.053191+0000", 2024-09-21T10:18:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1099999999999995e-07 2024-09-21T10:18:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:18:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.053198+0000", 2024-09-21T10:18:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8499999999999994e-06 2024-09-21T10:18:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:18:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.053207+0000", 2024-09-21T10:18:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7860000000000002e-06 2024-09-21T10:18:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:18:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.053214+0000", 2024-09-21T10:18:26.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3450000000000002e-06 2024-09-21T10:18:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:18:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:18:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:18:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:58034 8.b 8:d4cd1cf9:::benchmark_data_smithi098_226587_object58033:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e273)", 2024-09-21T10:18:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:26.053328+0000", 2024-09-21T10:18:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0051638430000000004, 2024-09-21T10:18:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005462738, 2024-09-21T10:18:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:18:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:18:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:18:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:18:26.085 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:18:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:18:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 58034 2024-09-21T10:18:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:18:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:18:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.053328+0000", 2024-09-21T10:18:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:18:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:18:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.053328+0000", 2024-09-21T10:18:26.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:18:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:18:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.053329+0000", 2024-09-21T10:18:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2619999999999999e-06 2024-09-21T10:18:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:18:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.053352+0000", 2024-09-21T10:18:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2305000000000001e-05 2024-09-21T10:18:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:18:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.053361+0000", 2024-09-21T10:18:26.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8239999999999995e-06 2024-09-21T10:18:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:18:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.053368+0000", 2024-09-21T10:18:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0149999999999998e-06 2024-09-21T10:18:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:18:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.053409+0000", 2024-09-21T10:18:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0460000000000002e-05 2024-09-21T10:18:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:18:26.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.053547+0000", 2024-09-21T10:18:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013829200000000001 2024-09-21T10:18:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:18:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.053694+0000", 2024-09-21T10:18:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014714799999999999 2024-09-21T10:18:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:18:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.056181+0000", 2024-09-21T10:18:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024865040000000001 2024-09-21T10:18:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:18:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:18:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:18:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:58039 8.0 8.ff25e870 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e273)", 2024-09-21T10:18:26.090 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:26.055120+0000", 2024-09-21T10:18:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.003371972, 2024-09-21T10:18:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037053910000000002, 2024-09-21T10:18:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:18:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:18:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-21T10:18:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:18:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:18:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:18:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 58039 2024-09-21T10:18:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.091 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:18:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:18:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.055120+0000", 2024-09-21T10:18:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:18:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:18:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.055120+0000", 2024-09-21T10:18:26.092 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:18:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:18:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.055122+0000", 2024-09-21T10:18:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9889999999999999e-06 2024-09-21T10:18:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:18:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.055145+0000", 2024-09-21T10:18:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3161000000000001e-05 2024-09-21T10:18:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:18:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.055162+0000", 2024-09-21T10:18:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6969e-05 2024-09-21T10:18:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:18:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.055169+0000", 2024-09-21T10:18:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3830000000000003e-06 2024-09-21T10:18:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:18:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:18:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:18:26.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.45794.0:58032 8.2 e273/265 8:40ad610a:::benchmark_data_smithi098_226587_object58031:head v 273'3566, mlcod=273'3564)", 2024-09-21T10:18:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:26.056420+0000", 2024-09-21T10:18:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.002072253, 2024-09-21T10:18:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024300110000000002, 2024-09-21T10:18:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:18:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:18:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:18:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:18:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:18:26.095 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.056420+0000", 2024-09-21T10:18:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:18:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:18:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.056420+0000", 2024-09-21T10:18:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:18:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:18:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.056421+0000", 2024-09-21T10:18:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2550000000000001e-06 2024-09-21T10:18:26.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:18:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.056445+0000", 2024-09-21T10:18:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4062000000000001e-05 2024-09-21T10:18:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:18:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.056460+0000", 2024-09-21T10:18:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5061e-05 2024-09-21T10:18:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:18:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.056467+0000", 2024-09-21T10:18:26.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.007e-06 2024-09-21T10:18:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:18:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.056544+0000", 2024-09-21T10:18:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6804000000000005e-05 2024-09-21T10:18:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:18:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.056564+0000", 2024-09-21T10:18:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0247000000000001e-05 2024-09-21T10:18:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:18:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:18:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:18:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:58043 8.0 8.c55ffeb0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e273)", 2024-09-21T10:18:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:26.058742+0000", 2024-09-21T10:18:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4294967295.9997501, 2024-09-21T10:18:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014411099999999999, 2024-09-21T10:18:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:18:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:18:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-21T10:18:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:18:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:18:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:18:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 58043 2024-09-21T10:18:26.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.100 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:18:26.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.100 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:18:26.100 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.058742+0000", 2024-09-21T10:18:26.100 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:18:26.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.100 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:18:26.100 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.058742+0000", 2024-09-21T10:18:26.100 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:18:26.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:18:26.101 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.058743+0000", 2024-09-21T10:18:26.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.576e-06 2024-09-21T10:18:26.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:18:26.101 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.058766+0000", 2024-09-21T10:18:26.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3107000000000001e-05 2024-09-21T10:18:26.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:18:26.101 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.058781+0000", 2024-09-21T10:18:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.429e-05 2024-09-21T10:18:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:18:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:18:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:18:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:26.058788+0000", 2024-09-21T10:18:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4610000000000002e-06 2024-09-21T10:18:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:18:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:18:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:18:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:18:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-21T10:18:26.102 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 9 2024-09-21T10:18:26.102 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:18:26.136 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:26.150 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 31 16 58151 58135 117.185 83.875 0.0068486 0.00852806 2024-09-21T10:18:26.165 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:26.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:26.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:26.266 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:26.292 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:26.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:26.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:26.393 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:26.424 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:26.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:26.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:26.526 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:26.571 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:26.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:26.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:26.673 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:26.702 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:26.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:26.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:26.807 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:26.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:26 smithi098 ceph-mon[97580]: pgmap v848: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-21T10:18:26.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:26 smithi098 ceph-mon[100914]: pgmap v848: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-21T10:18:26.849 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:26.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:26.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:26 smithi196 ceph-mon[84784]: pgmap v848: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-21T10:18:26.951 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:26.990 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:26.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:26.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:27.019 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:27.061 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:18:27.093 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:27.097 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:27.152 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 32 16 59580 59564 116.31 89.3125 0.0121895 0.00859126 2024-09-21T10:18:27.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:27.153 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:27.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:27.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:27.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:27.297 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:27.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:27.398 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:27.425 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:27.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:27.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:27.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:18:27.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:27.549 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:27.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:27.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:27.650 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:27.678 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:27.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:27.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:27.779 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:27.804 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:27.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:27.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:27.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:27.944 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:27.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:27.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:28.046 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:28.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:28.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:28.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:28.151 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 33 16 60889 60873 115.265 81.8125 0.00807728 0.00867098 2024-09-21T10:18:28.175 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:28.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:28.221 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:28.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:28.322 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:28.347 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:28.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:28.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:28.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:28.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:28.497 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:28.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:28.598 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:28.624 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:28.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:28.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:28.722 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:28 smithi098 ceph-mon[97580]: pgmap v849: 89 pgs: 89 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-21T10:18:28.723 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:28 smithi098 ceph-mon[100914]: pgmap v849: 89 pgs: 89 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-21T10:18:28.726 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:28.763 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:28.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:28.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:28.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:28 smithi196 ceph-mon[84784]: pgmap v849: 89 pgs: 89 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-21T10:18:28.905 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:28.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:28.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:29.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:29.034 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:29.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:29.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:29.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:29.152 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 34 16 62313 62297 114.491 89 0.00650266 0.00872926 2024-09-21T10:18:29.172 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:29.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:29.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:29.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:29.300 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:29.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:29.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:29.402 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:29.432 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:29.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:29.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:29.534 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:29.571 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:29.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:29.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:29.673 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:29.699 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:29.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:29.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:29.801 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:29.839 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:29.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:29.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:29.941 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:29.981 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:29.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:29.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:30.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:30.111 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:30.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:30.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:30.152 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 35 16 63682 63666 113.664 85.5625 0.010502 0.00879178 2024-09-21T10:18:30.213 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:30.256 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:30.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:30.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:30.359 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:30.389 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:30.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:30.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:30.491 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:30.519 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:30.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:30.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:30.620 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:30 smithi196 ceph-mon[84784]: pgmap v850: 89 pgs: 89 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-21T10:18:30.640 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:18:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:18:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:18:30.649 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:30.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:30.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:30.736 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:30.751 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:30.801 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:30.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:30.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:30.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:30 smithi098 ceph-mon[97580]: pgmap v850: 89 pgs: 89 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-21T10:18:30.849 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:18:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:18:30] "GET /metrics HTTP/1.1" 200 36519 "" "Prometheus/2.51.0" 2024-09-21T10:18:30.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:30 smithi098 ceph-mon[100914]: pgmap v850: 89 pgs: 89 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-21T10:18:30.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:30.943 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:30.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:30.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:31.045 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:31.068 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:31.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:31.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:31.152 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 36 16 64996 64980 112.788 82.125 0.0160556 0.00886121 2024-09-21T10:18:31.170 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:31.196 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:31.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:31.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:31.298 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:31.325 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:31.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:31.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:31.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:31.461 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:31.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:31.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:31.504 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:31.563 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:31.610 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:31.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:31.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:31.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:31.759 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:31.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:31.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:31.861 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:31.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:31.911 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:31.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:31.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:31 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a[97576]: 2024-09-21T10:18:31.811+0000 7f65561f9640 -1 mon.a@0(leader).osd e273 definitely_dead 0 2024-09-21T10:18:32.013 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:32.047 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:32.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:32.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:32.152 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 37 16 66123 66107 111.643 70.4375 0.0114836 0.00895187 2024-09-21T10:18:32.186 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:32.221 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:32.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:32.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:32.322 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:32.361 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:32.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:32.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:32.461 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:18:32.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:32.465 INFO:teuthology.orchestra.run.smithi098.stderr:marked out osd.3. 2024-09-21T10:18:32.514 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:32.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:32.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:32.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:32 smithi098 ceph-mon[97580]: pgmap v851: 89 pgs: 89 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 82 MiB/s wr, 1.30k op/s 2024-09-21T10:18:32.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:32 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3459442316' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-21T10:18:32.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:32 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-21T10:18:32.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:32 smithi098 ceph-mon[97580]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-21T10:18:32.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:32 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3991706380' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:18:32.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:32 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:18:32.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:32 smithi098 ceph-mon[100914]: pgmap v851: 89 pgs: 89 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 82 MiB/s wr, 1.30k op/s 2024-09-21T10:18:32.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:32 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3459442316' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-21T10:18:32.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:32 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-21T10:18:32.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:32 smithi098 ceph-mon[100914]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-21T10:18:32.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:32 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3991706380' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:18:32.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:32 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:18:32.617 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:32.638 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:32.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:32.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:32.739 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:32.760 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:32.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:32.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:32.862 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:32.883 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:32.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:32.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:32.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:32 smithi196 ceph-mon[84784]: pgmap v851: 89 pgs: 89 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 82 MiB/s wr, 1.30k op/s 2024-09-21T10:18:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:32 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3459442316' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-21T10:18:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:32 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-21T10:18:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:32 smithi196 ceph-mon[84784]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-21T10:18:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:32 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3991706380' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:18:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:32 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:18:32.985 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:33.006 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:33.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:33.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:33.108 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:33.129 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:33.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:33.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:33.152 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 38 16 66553 66537 109.413 26.875 0.00259394 0.00897559 2024-09-21T10:18:33.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:33.254 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:33.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:33.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:33.355 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:33.377 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:33.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:33.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:33.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:33.506 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:33.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:33.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:33.608 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:33.629 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:33.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:33.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:33.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:33.752 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:33.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:33.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:33.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:33 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:18:33.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:33 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:18:33.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:33 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-21T10:18:33.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:33 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:18:33.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:33 smithi098 ceph-mon[100914]: osdmap e274: 8 total, 8 up, 6 in 2024-09-21T10:18:33.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:33 smithi098 ceph-mon[100914]: osdmap e275: 8 total, 8 up, 6 in 2024-09-21T10:18:33.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:33 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:18:33.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:33 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:18:33.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:33 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-21T10:18:33.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:33 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:18:33.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:33 smithi098 ceph-mon[97580]: osdmap e274: 8 total, 8 up, 6 in 2024-09-21T10:18:33.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:33 smithi098 ceph-mon[97580]: osdmap e275: 8 total, 8 up, 6 in 2024-09-21T10:18:33.854 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:33.876 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:33.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:33.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:33.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:33 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:18:33.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:33 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:18:33.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:33 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-21T10:18:33.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:33 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:18:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:33 smithi196 ceph-mon[84784]: osdmap e274: 8 total, 8 up, 6 in 2024-09-21T10:18:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:33 smithi196 ceph-mon[84784]: osdmap e275: 8 total, 8 up, 6 in 2024-09-21T10:18:33.978 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:34.002 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:34.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:34.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:34.103 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:34.129 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:34.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:34.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:34.152 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 39 16 66553 66537 106.607 0 - 0.00897559 2024-09-21T10:18:34.231 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:34.253 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:34.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:34.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:34.355 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:34.381 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:34.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:34.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:34.482 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:34.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:34.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:34.622 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:34.645 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:34.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:34.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:34.747 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:34.770 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:34.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:34.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:34.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:34 smithi098 ceph-mon[97580]: pgmap v853: 89 pgs: 10 peering, 79 active+clean; 3.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 48 MiB/s wr, 772 op/s 2024-09-21T10:18:34.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:34 smithi098 ceph-mon[97580]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:18:34.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:34 smithi098 ceph-mon[97580]: osdmap e276: 8 total, 8 up, 6 in 2024-09-21T10:18:34.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:34 smithi098 ceph-mon[100914]: pgmap v853: 89 pgs: 10 peering, 79 active+clean; 3.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 48 MiB/s wr, 772 op/s 2024-09-21T10:18:34.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:34 smithi098 ceph-mon[100914]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:18:34.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:34 smithi098 ceph-mon[100914]: osdmap e276: 8 total, 8 up, 6 in 2024-09-21T10:18:34.872 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:34 smithi196 ceph-mon[84784]: pgmap v853: 89 pgs: 10 peering, 79 active+clean; 3.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 48 MiB/s wr, 772 op/s 2024-09-21T10:18:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:34 smithi196 ceph-mon[84784]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:18:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:34 smithi196 ceph-mon[84784]: osdmap e276: 8 total, 8 up, 6 in 2024-09-21T10:18:34.898 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:34.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:34.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:35.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:35.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:35.022 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:35.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:35.123 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:35.144 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:35.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:35.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:35.152 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:18:35.151402+0000 min lat: 0.00125976 max lat: 2.26409 avg lat: 0.00933835 2024-09-21T10:18:35.152 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:18:35.152 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 40 16 66691 66675 104.158 4.3125 0.0043091 0.00933835 2024-09-21T10:18:35.246 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:35.267 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:35.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:35.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:35.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:35.390 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:35.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:35.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:35.492 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:35.518 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:35.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:35.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:35.620 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:35.641 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:35.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:35.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:35.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:35.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:35.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:35.867 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:35.892 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:35.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:35.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:35.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:36.019 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:36.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:36.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:36.121 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:36.142 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:36.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:36.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:36.152 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 41 16 66691 66675 101.618 0 - 0.00933835 2024-09-21T10:18:36.244 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:36.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:36.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:36.370 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:36.391 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:36.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:36.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:36.493 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:36.520 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:36.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:36.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:36.622 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:36.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:36.644 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:36.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:36.746 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:36.768 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:36.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:36.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:36.835 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:36 smithi098 ceph-mon[97580]: pgmap v856: 89 pgs: 2 remapped+peering, 22 peering, 65 active+clean; 3.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 12 MiB/s wr, 188 op/s 2024-09-21T10:18:36.835 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:36 smithi098 ceph-mon[97580]: osdmap e277: 8 total, 8 up, 6 in 2024-09-21T10:18:36.835 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:36 smithi098 ceph-mon[100914]: pgmap v856: 89 pgs: 2 remapped+peering, 22 peering, 65 active+clean; 3.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 12 MiB/s wr, 188 op/s 2024-09-21T10:18:36.836 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:36 smithi098 ceph-mon[100914]: osdmap e277: 8 total, 8 up, 6 in 2024-09-21T10:18:36.870 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:36.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:36 smithi196 ceph-mon[84784]: pgmap v856: 89 pgs: 2 remapped+peering, 22 peering, 65 active+clean; 3.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 12 MiB/s wr, 188 op/s 2024-09-21T10:18:36.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:36 smithi196 ceph-mon[84784]: osdmap e277: 8 total, 8 up, 6 in 2024-09-21T10:18:36.892 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:36.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:36.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:36.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:36.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:18:37.017 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:37.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:37.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:37.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:37.148 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:37.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:37.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:37.152 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 42 16 66691 66675 99.1987 0 - 0.00933835 2024-09-21T10:18:37.250 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:37.276 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:37.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:37.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:37.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:37.400 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:37.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:37.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:37.501 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:37.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:37.524 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:37.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:37.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:37.648 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:37.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:37.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:37.750 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:37.757 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:37 smithi098 ceph-mon[97580]: osdmap e278: 8 total, 8 up, 6 in 2024-09-21T10:18:37.757 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:37.757 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:37.758 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:37 smithi098 ceph-mon[100914]: osdmap e278: 8 total, 8 up, 6 in 2024-09-21T10:18:37.758 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:37.758 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:37.773 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:37.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:37.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:37.875 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:37.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:37 smithi196 ceph-mon[84784]: osdmap e278: 8 total, 8 up, 6 in 2024-09-21T10:18:37.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:37.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:37.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:37.897 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:37.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:37.998 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:38.020 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:38.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:38.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:38.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:38.144 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:38.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:38.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:38.152 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 43 16 66691 66675 96.892 0 - 0.00933835 2024-09-21T10:18:38.246 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:38.268 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:38.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:38.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:38.369 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:38.393 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:38.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:38.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:38.494 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:38.515 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:38.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:38.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:38.617 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:38.641 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:38.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:38.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:38.743 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:38.764 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:38.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:38.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:38.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:38.892 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:38.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:38.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:38.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:39.018 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:39.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:39.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:39.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:39.145 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:39.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:39.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:39.152 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 44 16 66691 66675 94.6901 0 - 0.00933835 2024-09-21T10:18:39.188 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7] out_osds: [4, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:18:39.188 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:18:39.188 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-21T10:18:39.188 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:18:39.188 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:18:39.189 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-21T10:18:39.247 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:39.269 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:39.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:39.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:39.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:39 smithi098 ceph-mon[97580]: pgmap v859: 89 pgs: 2 remapped+peering, 22 peering, 65 active+clean; 3.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 26 MiB/s wr, 412 op/s 2024-09-21T10:18:39.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:39 smithi098 ceph-mon[100914]: pgmap v859: 89 pgs: 2 remapped+peering, 22 peering, 65 active+clean; 3.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 26 MiB/s wr, 412 op/s 2024-09-21T10:18:39.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:39.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:39.407 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:39.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:39.441 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:18:39.441 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:18:39.441 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-21T10:18:39.441 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-21T10:18:39.441 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:18:39.508 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:39.530 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:39.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:39.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:39.632 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:39.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:39 smithi196 ceph-mon[84784]: pgmap v859: 89 pgs: 2 remapped+peering, 22 peering, 65 active+clean; 3.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 26 MiB/s wr, 412 op/s 2024-09-21T10:18:39.653 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:39.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:39.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:39.755 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:39.778 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:39.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:39.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:39.880 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:39.901 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:39.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:39.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:40.003 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:40.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:40.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:40.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:40.151 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:40.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:40.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:40.153 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 45 16 66691 66675 92.5861 0 - 0.00933835 2024-09-21T10:18:40.253 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:40.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:40.275 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:40.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:40.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:18:40.376 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:40.397 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:40.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:40.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:40.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:40.527 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:40.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:40.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:40 smithi098 ceph-mon[97580]: pgmap v860: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 85 active+clean; 4.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 52 MiB/s wr, 834 op/s; 8227/196323 objects degraded (4.191%); 8085/196323 objects misplaced (4.118%); 2.4 MiB/s, 40 objects/s recovering 2024-09-21T10:18:40.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:40 smithi098 ceph-mon[97580]: Health check failed: Degraded data redundancy: 8227/196323 objects degraded (4.191%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:18:40.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:40 smithi098 ceph-mon[97580]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 9 pgs peering) 2024-09-21T10:18:40.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:40 smithi098 ceph-mon[100914]: pgmap v860: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 85 active+clean; 4.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 52 MiB/s wr, 834 op/s; 8227/196323 objects degraded (4.191%); 8085/196323 objects misplaced (4.118%); 2.4 MiB/s, 40 objects/s recovering 2024-09-21T10:18:40.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:40 smithi098 ceph-mon[100914]: Health check failed: Degraded data redundancy: 8227/196323 objects degraded (4.191%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:18:40.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:40 smithi098 ceph-mon[100914]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 9 pgs peering) 2024-09-21T10:18:40.629 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:40 smithi196 ceph-mon[84784]: pgmap v860: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 85 active+clean; 4.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 52 MiB/s wr, 834 op/s; 8227/196323 objects degraded (4.191%); 8085/196323 objects misplaced (4.118%); 2.4 MiB/s, 40 objects/s recovering 2024-09-21T10:18:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:40 smithi196 ceph-mon[84784]: Health check failed: Degraded data redundancy: 8227/196323 objects degraded (4.191%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:18:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:40 smithi196 ceph-mon[84784]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 9 pgs peering) 2024-09-21T10:18:40.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:18:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:18:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:18:40.656 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:40.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:40.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:40.758 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:40.779 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:40.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:40.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:40.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:40.905 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:40.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:40.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:41.007 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:41.028 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:41.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:41.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:41.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:18:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:18:40] "GET /metrics HTTP/1.1" 200 36519 "" "Prometheus/2.51.0" 2024-09-21T10:18:41.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:41.153 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 46 16 66691 66675 90.5736 0 - 0.00933835 2024-09-21T10:18:41.155 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:41.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:41.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:41.257 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:41.279 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:41.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:41.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:41.380 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:41.381 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:18:41.381 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:18:41.424 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:41.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:41.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:41.425 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:41.436 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:41.443 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:41.513 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:18:41.513 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:18:41.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:41.532 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:18:41.533 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:18:41.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:41.548 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:41.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:41.556 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:18:41.556 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:18:41.578 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:18:41.578 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:18:41.601 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:18:41.602 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:18:41.627 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:18:41.627 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:18:41.649 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:41.655 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:18:41.655 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:18:41.677 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:41.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:41.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:41.779 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:41.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:41.811 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:41.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:41.912 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:41.946 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:41.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:41.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:42.048 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:42.085 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:42.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:42.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:42.153 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 47 16 67231 67215 89.3647 4.82143 0.0188115 0.0111842 2024-09-21T10:18:42.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:42.226 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:42.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:42.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:42.328 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:42.361 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:42.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:42.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:42.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:42 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:42.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:42 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:42 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3995408815' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:18:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:42 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:18:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:42 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:18:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:42 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:42.462 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:42.485 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:42 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:42.485 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:42 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:42.485 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:42 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3995408815' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:18:42.485 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:42 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:18:42.485 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:42 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:18:42.485 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:42 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:42.486 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:42 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:42.486 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:42 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:42.486 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:42 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3995408815' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:18:42.486 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:42 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:18:42.486 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:42 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:18:42.486 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:42 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:42.491 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:42.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:42.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:42.592 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:42.619 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:42.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:42.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:42.721 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:42.758 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:42.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:42.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:42.860 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:42.889 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:42.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:42.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:42.991 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:43.026 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:43.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:43.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:43.130 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:43.153 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 48 16 68263 68247 88.8466 64.5 0.0247204 0.0112492 2024-09-21T10:18:43.171 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:43.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:43.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:43.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:43.309 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:43.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:43.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:43.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:43 smithi098 ceph-mon[97580]: pgmap v861: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 85 active+clean; 4.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 39 MiB/s wr, 625 op/s; 8227/196323 objects degraded (4.191%); 8085/196323 objects misplaced (4.118%); 2.0 MiB/s, 34 objects/s recovering 2024-09-21T10:18:43.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:43 smithi098 ceph-mon[100914]: pgmap v861: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 85 active+clean; 4.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 39 MiB/s wr, 625 op/s; 8227/196323 objects degraded (4.191%); 8085/196323 objects misplaced (4.118%); 2.0 MiB/s, 34 objects/s recovering 2024-09-21T10:18:43.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:43.439 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:43.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:43.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:43.540 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:43.575 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:43.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:43.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:43.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:43 smithi196 ceph-mon[84784]: pgmap v861: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 85 active+clean; 4.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 39 MiB/s wr, 625 op/s; 8227/196323 objects degraded (4.191%); 8085/196323 objects misplaced (4.118%); 2.0 MiB/s, 34 objects/s recovering 2024-09-21T10:18:43.677 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:43.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:43.718 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:43.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:43.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:43.860 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:43.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:43.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:43.962 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:43.995 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:43.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:43.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:44.096 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:44.122 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:44.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:44.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:44.154 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 49 16 69266 69250 88.3127 62.6875 0.0109824 0.0113105 2024-09-21T10:18:44.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:44.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:44.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:44.360 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:44.390 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:44.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:44.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:44.492 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:44.520 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:44.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:44.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:44.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:18:44.622 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:44.680 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:44.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:44.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:44.804 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:44.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:44.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:44.906 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:44.952 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:44.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:44.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:45.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:45.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:45.089 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:45.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:45.153 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 50 16 70161 70145 87.6651 55.9375 0.00451469 0.0113977 2024-09-21T10:18:45.189 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:45.218 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:45.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:45.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:45.320 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:45.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:45 smithi098 ceph-mon[97580]: pgmap v862: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 activating+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 46 MiB/s wr, 739 op/s; 20033/201126 objects degraded (9.960%); 7730/201126 objects misplaced (3.843%); 9.3 MiB/s, 151 objects/s recovering 2024-09-21T10:18:45.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:45 smithi098 ceph-mon[100914]: pgmap v862: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 activating+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 46 MiB/s wr, 739 op/s; 20033/201126 objects degraded (9.960%); 7730/201126 objects misplaced (3.843%); 9.3 MiB/s, 151 objects/s recovering 2024-09-21T10:18:45.350 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:45.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:45.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:45.452 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:45.482 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:45.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:45.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:45.584 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:45.612 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:45.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:45.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:45.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:45 smithi196 ceph-mon[84784]: pgmap v862: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 activating+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 46 MiB/s wr, 739 op/s; 20033/201126 objects degraded (9.960%); 7730/201126 objects misplaced (3.843%); 9.3 MiB/s, 151 objects/s recovering 2024-09-21T10:18:45.714 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:45.746 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:45.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:45.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:45.847 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:45.884 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:45.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:45.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:45.986 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:46.014 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:46.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:46.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:46.116 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:46.141 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:46.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:46.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:46.153 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 51 16 71140 71124 87.1459 61.1875 0.00735453 0.0114672 2024-09-21T10:18:46.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:46.276 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:46.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:46.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:46.410 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:46.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:46.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:46.512 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:46.538 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:46.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:46.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:46.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:46 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 20033/201126 objects degraded (9.960%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:18:46.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:46 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 20033/201126 objects degraded (9.960%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:18:46.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:46 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 20033/201126 objects degraded (9.960%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:18:46.640 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:46.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:46.678 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:46.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:46.779 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:46.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:46.815 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:46.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:46.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:46.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:46.956 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:46.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:47.057 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:47.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:47.088 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:47.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:47.153 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 52 16 72078 72062 86.5974 58.625 0.0580264 0.0115408 2024-09-21T10:18:47.189 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:47.217 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:47.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:47.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:47.319 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:47.351 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:47.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:47.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:47.453 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:47.506 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:47.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:47.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:47.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:47 smithi098 ceph-mon[97580]: pgmap v863: 89 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 42 MiB/s wr, 664 op/s; 15942/203073 objects degraded (7.850%); 11763/203073 objects misplaced (5.792%); 7.9 MiB/s, 128 objects/s recovering 2024-09-21T10:18:47.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:47 smithi098 ceph-mon[100914]: pgmap v863: 89 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 42 MiB/s wr, 664 op/s; 15942/203073 objects degraded (7.850%); 11763/203073 objects misplaced (5.792%); 7.9 MiB/s, 128 objects/s recovering 2024-09-21T10:18:47.609 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:47.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:47 smithi196 ceph-mon[84784]: pgmap v863: 89 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 42 MiB/s wr, 664 op/s; 15942/203073 objects degraded (7.850%); 11763/203073 objects misplaced (5.792%); 7.9 MiB/s, 128 objects/s recovering 2024-09-21T10:18:47.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:47.639 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:47.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:47.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:47.776 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:47.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:47.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:47.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:47.906 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:47.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:47.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:48.008 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:48.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:48.044 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:48.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:48.145 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:48.153 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 53 16 72995 72979 86.0448 57.3125 0.00855337 0.0116164 2024-09-21T10:18:48.173 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:48.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:48.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:48.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:48.304 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:48.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:48.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:48.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:48.442 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:48.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:48.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:48.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:48.572 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:48.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:48.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:48.674 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:48.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:48.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:48.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:48.837 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:48.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:48.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:48.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:48 smithi098 ceph-mon[97580]: pgmap v864: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 48 MiB/s wr, 769 op/s; 19443/208488 objects degraded (9.326%); 8252/208488 objects misplaced (3.958%); 7.2 MiB/s, 117 objects/s recovering 2024-09-21T10:18:48.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:48 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:48.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:48 smithi098 ceph-mon[100914]: pgmap v864: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 48 MiB/s wr, 769 op/s; 19443/208488 objects degraded (9.326%); 8252/208488 objects misplaced (3.958%); 7.2 MiB/s, 117 objects/s recovering 2024-09-21T10:18:48.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:48 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:48 smithi196 ceph-mon[84784]: pgmap v864: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 48 MiB/s wr, 769 op/s; 19443/208488 objects degraded (9.326%); 8252/208488 objects misplaced (3.958%); 7.2 MiB/s, 117 objects/s recovering 2024-09-21T10:18:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:48 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:48.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:48.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:48.969 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:48.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:49.070 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:49.096 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:49.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:49.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:49.153 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 54 16 73839 73823 85.4282 52.75 0.0130696 0.0116987 2024-09-21T10:18:49.198 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:49.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:49.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:49.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:49.332 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:49.360 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:49.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:49.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:49.462 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:49.500 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:49.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:49.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:49.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:49.641 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:49.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:49.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:49.743 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:49.772 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:49.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:49.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:49.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:49.904 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:49.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:49.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:50.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:50.027 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:50.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:50.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:50.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:50.154 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 55 16 74722 74706 84.8783 55.1875 0.0193101 0.0117764 2024-09-21T10:18:50.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:50.177 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:50.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:50.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:50.319 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:50.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:50.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:50.421 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:50.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:50.458 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:50.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:50.559 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:50.580 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:50.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:50.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:50.586 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:18:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:18:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:18:50.682 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:50.710 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:50.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:50.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:50.812 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:50.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:50 smithi098 ceph-mon[97580]: pgmap v865: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 59 MiB/s wr, 942 op/s; 19443/217299 objects degraded (8.948%); 8115/217299 objects misplaced (3.734%); 7.3 MiB/s, 118 objects/s recovering 2024-09-21T10:18:50.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:18:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:18:50] "GET /metrics HTTP/1.1" 200 36519 "" "Prometheus/2.51.0" 2024-09-21T10:18:50.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:50 smithi098 ceph-mon[100914]: pgmap v865: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 59 MiB/s wr, 942 op/s; 19443/217299 objects degraded (8.948%); 8115/217299 objects misplaced (3.734%); 7.3 MiB/s, 118 objects/s recovering 2024-09-21T10:18:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:50.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:50.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:50 smithi196 ceph-mon[84784]: pgmap v865: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 59 MiB/s wr, 942 op/s; 19443/217299 objects degraded (8.948%); 8115/217299 objects misplaced (3.734%); 7.3 MiB/s, 118 objects/s recovering 2024-09-21T10:18:50.952 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:50.984 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:50.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:50.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:51.085 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:51.119 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:51.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:51.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:51.154 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 56 16 75578 75562 84.3179 53.5 0.0159624 0.0118537 2024-09-21T10:18:51.221 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:51.249 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:51.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:51.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:51.351 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:51.379 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:51.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:51.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:51.481 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:51.510 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:51.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:51.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:51.612 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:51.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:51.643 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:51.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:51.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:51.784 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:51.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:51.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:51.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:51 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/217299 objects degraded (8.948%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:18:51.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:51 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/217299 objects degraded (8.948%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:18:51.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:51 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/217299 objects degraded (8.948%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:18:51.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:51.922 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:51.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:52.023 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:52.058 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:52.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:52.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:52.154 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 57 16 76581 76565 83.9384 62.6875 0.0400792 0.0119063 2024-09-21T10:18:52.159 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:52.196 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:52.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:52.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:52.298 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:52.325 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:52.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:52.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:52.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:52.477 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:52.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:52.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:52.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:52.605 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:52.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:52.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:52.706 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:52.750 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:52.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:52.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:52.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:52.878 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:52.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:52.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:52.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:53.014 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:53.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:53.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:53.116 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:53.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:53.156 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:53.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:53.156 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 58 16 77444 77428 83.4195 53.9375 0.0103789 0.0119797 2024-09-21T10:18:53.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:53.291 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:53.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:53.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:53.393 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:53.425 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:53.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:53.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:53.527 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:53.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:53.558 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:53.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:53.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:53 smithi098 ceph-mon[97580]: pgmap v866: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 36 MiB/s wr, 582 op/s; 19443/217299 objects degraded (8.948%); 8115/217299 objects misplaced (3.734%); 6.1 MiB/s, 97 objects/s recovering 2024-09-21T10:18:53.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:53.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:53.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:53 smithi098 ceph-mon[100914]: pgmap v866: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 36 MiB/s wr, 582 op/s; 19443/217299 objects degraded (8.948%); 8115/217299 objects misplaced (3.734%); 6.1 MiB/s, 97 objects/s recovering 2024-09-21T10:18:53.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:53.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:53.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:53 smithi196 ceph-mon[84784]: pgmap v866: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 36 MiB/s wr, 582 op/s; 19443/217299 objects degraded (8.948%); 8115/217299 objects misplaced (3.734%); 6.1 MiB/s, 97 objects/s recovering 2024-09-21T10:18:53.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:18:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:53.659 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:53.702 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:53.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:53.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:53.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:53.841 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:53.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:53.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:53.943 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:53.974 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:53.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:53.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:54.076 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:54.103 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:54.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:54.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:54.155 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 59 16 78294 78278 82.906 53.125 0.00935593 0.0120559 2024-09-21T10:18:54.205 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:54.231 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:54.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:54.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:54.333 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:54.361 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:54.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:54.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:54.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:54.507 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:54.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:54.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:54.609 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:54.639 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:54.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:54.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:54.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:54.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:54.786 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:54.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:54.887 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:54.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:54.926 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:54.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:55.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:55.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:55.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:55.155 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:18:55.154301+0000 min lat: 0.00125976 max lat: 9.22765 avg lat: 0.0121334 2024-09-21T10:18:55.155 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:18:55.156 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 60 16 79103 79087 82.3669 50.5625 0.00844476 0.0121334 2024-09-21T10:18:55.168 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:55.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:55.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:55.306 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:55.328 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:55.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:55.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:55.430 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:55.458 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:55.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:55.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:55.560 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:55.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:55 smithi098 ceph-mon[97580]: pgmap v867: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 53 MiB/s wr, 840 op/s; 19443/226575 objects degraded (8.581%); 8115/226575 objects misplaced (3.582%); 6.1 MiB/s, 97 objects/s recovering 2024-09-21T10:18:55.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:55 smithi098 ceph-mon[100914]: pgmap v867: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 53 MiB/s wr, 840 op/s; 19443/226575 objects degraded (8.581%); 8115/226575 objects misplaced (3.582%); 6.1 MiB/s, 97 objects/s recovering 2024-09-21T10:18:55.599 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:55.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:55.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:55.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:55 smithi196 ceph-mon[84784]: pgmap v867: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 53 MiB/s wr, 840 op/s; 19443/226575 objects degraded (8.581%); 8115/226575 objects misplaced (3.582%); 6.1 MiB/s, 97 objects/s recovering 2024-09-21T10:18:55.701 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:55.738 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:55.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:55.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:55.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:55.870 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:55.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:55.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:55.972 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:56.009 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:56.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:56.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:56.111 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:56.132 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:56.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:56.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:56.155 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 61 16 80022 80006 81.9581 57.4375 0.00684659 0.0121941 2024-09-21T10:18:56.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:56.269 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:56.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:56.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:56.371 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:56.407 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:56.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:56.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:56.509 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:56.547 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:56.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:56.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:56 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/226575 objects degraded (8.581%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:18:56.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:56 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/226575 objects degraded (8.581%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:18:56.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:56 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/226575 objects degraded (8.581%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:18:56.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:56.687 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:56.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:56.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:56.788 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:56.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:56.834 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:56.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:56.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:56.971 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:56.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:56.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:57.073 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:57.100 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:57.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:57.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:57.155 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 62 16 80941 80925 81.5626 57.4375 0.0354883 0.0122539 2024-09-21T10:18:57.202 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:57.232 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:57.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:57.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:57.334 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:57.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:57.374 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:57.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:57.474 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:57.504 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:57 smithi098 ceph-mon[100914]: pgmap v868: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.7 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 51 MiB/s wr, 823 op/s; 19443/230763 objects degraded (8.426%); 7985/230763 objects misplaced (3.460%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:18:57.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:57 smithi098 ceph-mon[97580]: pgmap v868: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.7 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 51 MiB/s wr, 823 op/s; 19443/230763 objects degraded (8.426%); 7985/230763 objects misplaced (3.460%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:18:57.524 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:57.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:57.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:57.626 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:57.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:57 smithi196 ceph-mon[84784]: pgmap v868: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.7 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 51 MiB/s wr, 823 op/s; 19443/230763 objects degraded (8.426%); 7985/230763 objects misplaced (3.460%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:18:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:57.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:57.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:57.828 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:57.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:57.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:57.969 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:57.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:57.999 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:57.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:58.100 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:58.136 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:58.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:58.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:58.156 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 63 16 81831 81815 81.1509 55.625 0.010507 0.0123176 2024-09-21T10:18:58.237 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:58.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:58.278 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:58.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:58.279 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:58.329 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:18:58.329 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:18:58.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:18:58 smithi098 ceph-mon[97580]: pgmap v869: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 57 MiB/s wr, 904 op/s; 19443/235635 objects degraded (8.251%); 7985/235635 objects misplaced (3.389%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:18:58.330 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:18:58 smithi098 ceph-mon[100914]: pgmap v869: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 57 MiB/s wr, 904 op/s; 19443/235635 objects degraded (8.251%); 7985/235635 objects misplaced (3.389%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:18:58.379 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:58.407 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:58.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:58.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:58.509 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:58.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:58.550 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:58.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:18:58 smithi196 ceph-mon[84784]: pgmap v869: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 57 MiB/s wr, 904 op/s; 19443/235635 objects degraded (8.251%); 7985/235635 objects misplaced (3.389%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:18:58.651 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:58.680 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:58.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:58.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:58.781 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:58.819 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:58.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:58.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:58.920 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:58.956 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:58.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:58.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:59.057 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:59.090 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:59.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:59.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:59.156 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 64 16 82708 82692 80.7393 54.8125 0.0160994 0.0123792 2024-09-21T10:18:59.191 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:59.228 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:59.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:59.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:59.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:18:59.330 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:59.359 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:59.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:59.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:59.461 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:59.504 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:59.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:59.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:59.606 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:59.631 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:18:59.642 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:59.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:59.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:59.744 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:59.775 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:18:59.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:59.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:59.876 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:59.916 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:18:59.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:59.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:00.017 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:00.047 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:00.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:00.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:00.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:00.156 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 65 16 83565 83549 80.3211 53.5625 0.00418693 0.0124429 2024-09-21T10:19:00.195 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:00.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:00.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:00.297 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:00.329 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:00.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:00.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:00.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:19:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:19:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:19:00.431 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:00.434 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:19:00.473 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:00.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:00.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:00.513 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:19:00.513 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "" 2024-09-21T10:19:00.513 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:19:00.576 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:00.604 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:00.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:00.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:00.706 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:00.738 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:00.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:00.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:00.835 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:00 smithi098 ceph-mon[97580]: pgmap v870: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 62 MiB/s wr, 985 op/s; 19443/243963 objects degraded (7.970%); 7859/243963 objects misplaced (3.221%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T10:19:00.836 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:00 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/697770201' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:19:00.836 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:00 smithi098 ceph-mon[100914]: pgmap v870: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 62 MiB/s wr, 985 op/s; 19443/243963 objects degraded (7.970%); 7859/243963 objects misplaced (3.221%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T10:19:00.836 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:00 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/697770201' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:19:00.836 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:19:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:19:00] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-21T10:19:00.841 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:00.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:00.876 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:00.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:00 smithi196 ceph-mon[84784]: pgmap v870: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 62 MiB/s wr, 985 op/s; 19443/243963 objects degraded (7.970%); 7859/243963 objects misplaced (3.221%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T10:19:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:00 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/697770201' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:19:00.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:01.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:01.011 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:01.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:01.112 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:01.153 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:01.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:01.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:01.156 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 66 16 84493 84477 79.9829 58 0.00848943 0.0124961 2024-09-21T10:19:01.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:01.303 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:01.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:01.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:01.404 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:01.440 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:01.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:01.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:01.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:01.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:01.580 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:01.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:01 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:19:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:01 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/697770201' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:19:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:01 smithi098 ceph-mon[97580]: osdmap e279: 8 total, 8 up, 6 in 2024-09-21T10:19:01.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:01 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/243963 objects degraded (7.970%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:19:01.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:01 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:19:01.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:01 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/697770201' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:19:01.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:01 smithi098 ceph-mon[100914]: osdmap e279: 8 total, 8 up, 6 in 2024-09-21T10:19:01.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:01 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/243963 objects degraded (7.970%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:19:01.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:01.711 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:01 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:19:01.711 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:01 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/697770201' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:19:01.711 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:01 smithi196 ceph-mon[84784]: osdmap e279: 8 total, 8 up, 6 in 2024-09-21T10:19:01.711 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:01 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/243963 objects degraded (7.970%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:19:01.719 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:01.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:01.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:01.821 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:01.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:01.856 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:01.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:01.957 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:01.994 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:01.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:01.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:02.095 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:02.117 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:19:02.117 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:19:02.133 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:02.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:02.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:02.156 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 67 16 85422 85406 79.6556 58.0625 0.0285204 0.012548 2024-09-21T10:19:02.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:02.275 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:02.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:02.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:02.376 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:02.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:02.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:02.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:02.511 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:02.539 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:02.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:02.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:02.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:02 smithi098 ceph-mon[97580]: pgmap v872: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 56 MiB/s wr, 888 op/s; 19443/243963 objects degraded (7.970%); 7859/243963 objects misplaced (3.221%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:19:02.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:02 smithi098 ceph-mon[100914]: pgmap v872: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 56 MiB/s wr, 888 op/s; 19443/243963 objects degraded (7.970%); 7859/243963 objects misplaced (3.221%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:19:02.641 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:02.675 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:02.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:02.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:02.777 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:02.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:02.804 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:02.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:02.835 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:02 smithi196 ceph-mon[84784]: pgmap v872: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 56 MiB/s wr, 888 op/s; 19443/243963 objects degraded (7.970%); 7859/243963 objects misplaced (3.221%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:19:02.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:02.947 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:19:02.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:02.953 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:02.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:03.054 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:03.098 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:03.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:03.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:03.156 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 68 16 86324 86308 79.3132 56.375 0.0158764 0.012601 2024-09-21T10:19:03.199 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:03.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:03.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:03.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:03.332 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:03.373 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:03.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:03.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:03.382 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:19:03.474 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:03.512 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:03.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:03.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:03.547 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:19:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:19:03.583 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:19:03.583 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:19:03.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:03.623 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:19:03.623 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:19:03.640 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:03.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:03.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:03.644 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:19:03.644 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:19:03.673 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:19:03.673 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:19:03.698 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:19:03.698 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:19:03.731 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:19:03.731 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:19:03.742 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:03.762 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:03.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:03.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:03.806 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:19:03.806 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:19:03.864 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:03.902 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:03.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:03.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:04.003 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:04.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:04.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:04.140 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:04.156 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 69 16 87084 87068 78.8522 47.5 0.0154709 0.0126769 2024-09-21T10:19:04.180 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:04.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:04.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:04.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:04.306 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:19:04.310 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:04.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:04.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:04.412 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:04.450 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:04.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:04.551 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:04.582 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:04.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:04.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:04 smithi098 ceph-mon[100914]: pgmap v873: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 56 MiB/s wr, 890 op/s; 19443/253284 objects degraded (7.676%); 7859/253284 objects misplaced (3.103%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:19:04.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:04 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3212799990' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:04.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:04 smithi098 ceph-mon[97580]: pgmap v873: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 56 MiB/s wr, 890 op/s; 19443/253284 objects degraded (7.676%); 7859/253284 objects misplaced (3.103%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:19:04.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:04 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3212799990' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:04.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:04 smithi196 ceph-mon[84784]: pgmap v873: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 56 MiB/s wr, 890 op/s; 19443/253284 objects degraded (7.676%); 7859/253284 objects misplaced (3.103%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:19:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:04 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3212799990' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:04.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:04.716 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:04.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:04.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:04.818 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:04.854 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:04.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:04.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:04.956 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:04.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:04.995 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:04.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:05.096 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:05.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:05.130 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:05.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:05.156 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 70 16 87893 87877 78.448 50.5625 0.0648134 0.0127391 2024-09-21T10:19:05.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:05.265 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:05.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:05.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:05.367 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:05.405 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:05.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:05.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:05.506 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:05.542 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:05.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:05.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:05.644 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:05.682 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:05.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:05.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:05.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:05.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:05.917 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:05.924 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:19:05.924 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:19:05.924 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-21T10:19:05.925 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-21T10:19:05.925 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:19:05.954 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:05.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:05.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:06.056 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:06.090 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:06.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:06.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:06.156 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 71 16 88777 88761 78.1213 55.25 0.00652075 0.0127945 2024-09-21T10:19:06.191 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:06.229 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:06.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:06.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:06.293 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7] out_osds: [4, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:19:06.293 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:19:06.293 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-21T10:19:06.293 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:19:06.293 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:19:06.294 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-09-21T10:19:06.332 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:06.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:06.358 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:06.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:06.358 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:06.459 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:06.507 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:06 smithi098 ceph-mon[100914]: pgmap v874: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 55 MiB/s wr, 885 op/s; 19443/257325 objects degraded (7.556%); 7734/257325 objects misplaced (3.006%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:19:06.507 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:06 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/257325 objects degraded (7.556%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:19:06.507 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:06 smithi098 ceph-mon[97580]: pgmap v874: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 55 MiB/s wr, 885 op/s; 19443/257325 objects degraded (7.556%); 7734/257325 objects misplaced (3.006%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:19:06.507 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:06 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/257325 objects degraded (7.556%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:19:06.527 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:06.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:06.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:06.629 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:06.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:06 smithi196 ceph-mon[84784]: pgmap v874: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 55 MiB/s wr, 885 op/s; 19443/257325 objects degraded (7.556%); 7734/257325 objects misplaced (3.006%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:19:06.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:06 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/257325 objects degraded (7.556%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:19:06.662 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:06.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:06.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:06.764 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:06.791 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:06.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:06.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:06.893 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:06.922 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:19:06.930 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:06.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:06.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:07.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:07.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:07.067 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:07.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:07.156 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 72 16 89585 89569 77.7376 50.5 0.0262524 0.0128583 2024-09-21T10:19:07.168 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:07.215 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:07.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:07.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:07.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:07.317 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:19:07.359 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:07.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:07.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:07.461 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:07.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:07.505 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:07.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:07.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:07.639 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:07.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:07.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:07.741 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:07.778 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:07.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:07.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:07.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:07.903 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:07.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:07.927 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:07.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:08.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:08.072 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:08.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:08.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:08.157 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 73 16 90408 90392 77.3765 51.4375 0.0142549 0.0129181 2024-09-21T10:19:08.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:08.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:08.215 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:08.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:08.300 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:19:08.317 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:08.348 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:08.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:08.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:08.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:19:08.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:08.408 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:19:08.408 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:08.450 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:08.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:08.481 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:08.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:08.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:08.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:08.621 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:08.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:08.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:19:08.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:08.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:08.812 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:08.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:08.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:08.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:08.942 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:08.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:08.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:09.044 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:09.073 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:09.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:09.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:09.157 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 74 16 91182 91166 76.9845 48.375 0.00483211 0.0129823 2024-09-21T10:19:09.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:09.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:09.211 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:09.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:09.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:09.353 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:09.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:09.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:09.455 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:09.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:09.488 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:09.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:09.500 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:19:09.500 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-21T10:19:09.500 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-21T10:19:09.500 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [] 2024-09-21T10:19:09.500 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:19:09.510 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:09 smithi098 ceph-mon[97580]: pgmap v875: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 55 MiB/s wr, 882 op/s; 19443/262119 objects degraded (7.418%); 7734/262119 objects misplaced (2.951%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:19:09.510 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:09 smithi098 ceph-mon[100914]: pgmap v875: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 55 MiB/s wr, 882 op/s; 19443/262119 objects degraded (7.418%); 7734/262119 objects misplaced (2.951%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:19:09.517 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:09 smithi196 ceph-mon[84784]: pgmap v875: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 55 MiB/s wr, 882 op/s; 19443/262119 objects degraded (7.418%); 7734/262119 objects misplaced (2.951%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:19:09.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:09.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:09.634 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:09.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:09.735 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:09.758 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:09.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:09.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:09.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:09.905 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:09.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:09.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:10.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:10.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:10.033 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:10.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:10.134 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:10.157 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 75 16 91890 91874 76.5478 44.25 0.0265701 0.013057 2024-09-21T10:19:10.170 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:10.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:10.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:10.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:10.312 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:10.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:10.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:10.413 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:10.437 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:19:10.453 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:10.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:10.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:10.474 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:10 smithi196 ceph-mon[84784]: pgmap v876: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 54 MiB/s wr, 863 op/s; 19443/269874 objects degraded (7.204%); 7617/269874 objects misplaced (2.822%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:19:10.474 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:19:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:19:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:19:10.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:10.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:10 smithi098 ceph-mon[97580]: pgmap v876: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 54 MiB/s wr, 863 op/s; 19443/269874 objects degraded (7.204%); 7617/269874 objects misplaced (2.822%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:19:10.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:10 smithi098 ceph-mon[100914]: pgmap v876: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 54 MiB/s wr, 863 op/s; 19443/269874 objects degraded (7.204%); 7617/269874 objects misplaced (2.822%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:19:10.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:10.607 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:10.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:10.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:10.746 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:10.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:10.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:10.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:10.861 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:19:10.885 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:10.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:10.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:10.987 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:11.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:11.021 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:11.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:11.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:19:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:19:10] "GET /metrics HTTP/1.1" 200 36544 "" "Prometheus/2.51.0" 2024-09-21T10:19:11.122 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:11.154 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:11.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:11.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:11.157 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 76 16 92708 92692 76.2133 51.125 0.00917894 0.0131147 2024-09-21T10:19:11.256 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:11.296 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:11.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:11.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:11.398 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:11.425 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:11.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:11.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:11.527 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:11.571 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:11.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:11.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:11.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:11 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/269874 objects degraded (7.204%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:19:11.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:11 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/269874 objects degraded (7.204%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:19:11.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:11 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/269874 objects degraded (7.204%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:19:11.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:11.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:11.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:11.811 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:11.840 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:11.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:11.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:11.941 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:11.968 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:11.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:11.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:12.070 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:12.112 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:12.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:12.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:12.158 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 77 16 93514 93498 75.8777 50.375 0.00911935 0.013172 2024-09-21T10:19:12.213 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:12.248 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:12.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:12.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:12.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:12 smithi196 ceph-mon[84784]: pgmap v877: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 49 MiB/s wr, 789 op/s; 19443/269874 objects degraded (7.204%); 7617/269874 objects misplaced (2.822%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:19:12.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:12.381 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:12.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:12.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:12.483 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:12.514 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:12.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:12.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:12.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:12 smithi098 ceph-mon[97580]: pgmap v877: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 49 MiB/s wr, 789 op/s; 19443/269874 objects degraded (7.204%); 7617/269874 objects misplaced (2.822%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:19:12.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:12 smithi098 ceph-mon[100914]: pgmap v877: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 49 MiB/s wr, 789 op/s; 19443/269874 objects degraded (7.204%); 7617/269874 objects misplaced (2.822%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:19:12.616 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:12.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:12.655 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:12.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:12.742 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:19:12.742 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:19:12.742 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.742 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:93944 8.a e279/265 8:581a393a:::benchmark_data_smithi098_226587_object93943:head v 279'5941, mlcod=279'5939)", 2024-09-21T10:19:12.742 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:12.733267+0000", 2024-09-21T10:19:12.742 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.005342599, 2024-09-21T10:19:12.742 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0053747639999999998, 2024-09-21T10:19:12.742 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:12.742 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:19:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.733267+0000", 2024-09-21T10:19:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.733267+0000", 2024-09-21T10:19:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.733265+0000", 2024-09-21T10:19:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:19:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.733331+0000", 2024-09-21T10:19:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5591000000000003e-05 2024-09-21T10:19:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.733351+0000", 2024-09-21T10:19:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0471000000000001e-05 2024-09-21T10:19:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.733360+0000", 2024-09-21T10:19:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3690000000000007e-06 2024-09-21T10:19:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.735402+0000", 2024-09-21T10:19:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002042837 2024-09-21T10:19:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.735422+0000", 2024-09-21T10:19:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9777000000000001e-05 2024-09-21T10:19:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:93925 8.3 e279/265 8:c00a1600:::benchmark_data_smithi098_226587_object93924:head v 279'5910, mlcod=279'5908)", 2024-09-21T10:19:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:12.735370+0000", 2024-09-21T10:19:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0032393809999999999, 2024-09-21T10:19:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003336145, 2024-09-21T10:19:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:19:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.735370+0000", 2024-09-21T10:19:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.735370+0000", 2024-09-21T10:19:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.735370+0000", 2024-09-21T10:19:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:19:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.735414+0000", 2024-09-21T10:19:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4039999999999998e-05 2024-09-21T10:19:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.735428+0000", 2024-09-21T10:19:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3968000000000001e-05 2024-09-21T10:19:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.735434+0000", 2024-09-21T10:19:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2550000000000003e-06 2024-09-21T10:19:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.735588+0000", 2024-09-21T10:19:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015413300000000001 2024-09-21T10:19:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.735609+0000", 2024-09-21T10:19:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0965e-05 2024-09-21T10:19:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:93945 8.9 e279/265 8:9efe2068:::benchmark_data_smithi098_226587_object93944:head v 279'5808, mlcod=279'5806)", 2024-09-21T10:19:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:12.735474+0000", 2024-09-21T10:19:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0031354690000000001, 2024-09-21T10:19:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003291988, 2024-09-21T10:19:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:19:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.735474+0000", 2024-09-21T10:19:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.735474+0000", 2024-09-21T10:19:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.735473+0000", 2024-09-21T10:19:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:19:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.735497+0000", 2024-09-21T10:19:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3600000000000001e-05 2024-09-21T10:19:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.735506+0000", 2024-09-21T10:19:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1600000000000004e-06 2024-09-21T10:19:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.735511+0000", 2024-09-21T10:19:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.707e-06 2024-09-21T10:19:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.735554+0000", 2024-09-21T10:19:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2858999999999997e-05 2024-09-21T10:19:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.735570+0000", 2024-09-21T10:19:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6299000000000001e-05 2024-09-21T10:19:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:93929 8.3 e279/265 8:cad43192:::benchmark_data_smithi098_226587_object93928:head v 279'5911, mlcod=279'5908)", 2024-09-21T10:19:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:12.737874+0000", 2024-09-21T10:19:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.00073506999999999999, 2024-09-21T10:19:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00094345799999999999, 2024-09-21T10:19:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:19:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.737874+0000", 2024-09-21T10:19:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.737874+0000", 2024-09-21T10:19:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.737869+0000", 2024-09-21T10:19:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999948 2024-09-21T10:19:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.737983+0000", 2024-09-21T10:19:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114063 2024-09-21T10:19:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.738006+0000", 2024-09-21T10:19:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2328999999999999e-05 2024-09-21T10:19:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.756 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:12.756 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.738012+0000", 2024-09-21T10:19:12.756 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1279999999999996e-06 2024-09-21T10:19:12.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.756 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.756 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:12.756 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.738694+0000", 2024-09-21T10:19:12.756 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00068234800000000005 2024-09-21T10:19:12.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:12.756 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:12.756 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:12.756 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.738713+0000", 2024-09-21T10:19:12.756 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9201000000000001e-05 2024-09-21T10:19:12.756 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:12.757 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:12.757 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:12.757 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:12.757 INFO:teuthology.orchestra.run.smithi196.stdout: ], 2024-09-21T10:19:12.757 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 4 2024-09-21T10:19:12.757 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:19:12.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:12.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:12.799 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:12.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:12.900 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:12.940 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:12.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:12.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:13.042 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:13.076 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:13.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:13.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:13.158 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 78 16 94309 94293 75.5419 49.6875 0.00730474 0.013232 2024-09-21T10:19:13.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:13.212 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:13.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:13.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:19:13.314 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:13.340 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:13.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:13.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:13.362 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:13.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:13 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3934827126' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:19:13.442 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:13.481 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:13.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:13.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:13.485 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:19:13.583 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:13.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:13 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3934827126' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:19:13.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:13 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3934827126' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:19:13.619 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:13.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:13.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:13.721 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:13.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:13.751 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:13.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:13.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:13.869 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:19:13.891 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:13.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:13.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:13.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:14.026 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:14.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:14.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:14.127 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:14.158 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 79 16 95118 95102 75.2257 50.5625 0.0358791 0.013287 2024-09-21T10:19:14.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:14.161 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:14.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:14.262 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:14.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:14.297 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:14.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:14.398 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:14.437 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:14.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:14.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:14.539 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:14.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:14.575 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:14.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:14.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:14 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:19:14.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:14 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3934827126' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:19:14.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:14 smithi098 ceph-mon[100914]: osdmap e280: 8 total, 8 up, 6 in 2024-09-21T10:19:14.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:14 smithi098 ceph-mon[100914]: pgmap v879: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 51 MiB/s wr, 819 op/s; 19443/277890 objects degraded (6.997%); 7617/277890 objects misplaced (2.741%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:19:14.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:14 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:19:14.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:14 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3934827126' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:19:14.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:14 smithi098 ceph-mon[97580]: osdmap e280: 8 total, 8 up, 6 in 2024-09-21T10:19:14.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:14 smithi098 ceph-mon[97580]: pgmap v879: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 51 MiB/s wr, 819 op/s; 19443/277890 objects degraded (6.997%); 7617/277890 objects misplaced (2.741%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:19:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:14 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:19:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:14 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3934827126' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:19:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:14 smithi196 ceph-mon[84784]: osdmap e280: 8 total, 8 up, 6 in 2024-09-21T10:19:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:14 smithi196 ceph-mon[84784]: pgmap v879: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 51 MiB/s wr, 819 op/s; 19443/277890 objects degraded (6.997%); 7617/277890 objects misplaced (2.741%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:19:14.677 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:14.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:14.718 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:14.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:14.819 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:14.851 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:14.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:14.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:14.952 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:14.974 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:14.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:14.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:15.075 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:15.097 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:15.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:15.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:15.158 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:19:15.157100+0000 min lat: 0.00125976 max lat: 9.22765 avg lat: 0.0133276 2024-09-21T10:19:15.158 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:19:15.158 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 80 16 95773 95757 74.7971 40.9375 0.0148317 0.0133276 2024-09-21T10:19:15.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:15.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:15.223 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:15.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:15.324 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:15.371 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:15.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:15.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:15.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:15.505 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:15.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:15.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:15.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:15.641 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:15.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:15.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:15.743 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:15.777 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:15.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:15.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:15.875 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:19:15.875 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:19:15.875 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-21T10:19:15.876 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-21T10:19:15.876 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:19:15.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:15.908 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:15.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:15.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:16.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:16.049 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:16.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:16.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:16.151 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:16.158 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 81 16 96541 96525 74.4662 48 0.00771658 0.0134226 2024-09-21T10:19:16.182 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:16.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:16.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:16.284 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:16.314 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:16.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:16.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:16.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:16.454 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:16.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:16.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:16.525 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:16 smithi196 ceph-mon[84784]: pgmap v880: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 50 MiB/s wr, 807 op/s; 19443/281577 objects degraded (6.905%); 7482/281577 objects misplaced (2.657%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:19:16.525 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:16 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/281577 objects degraded (6.905%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:19:16.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:16.565 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:19:16.587 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:16.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:16.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:16.611 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:19:16.611 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:19:16.611 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:19:16.689 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:16.741 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:16.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:16.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:16.842 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:16 smithi098 ceph-mon[97580]: pgmap v880: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 50 MiB/s wr, 807 op/s; 19443/281577 objects degraded (6.905%); 7482/281577 objects misplaced (2.657%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:19:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:16 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/281577 objects degraded (6.905%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:19:16.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:16 smithi098 ceph-mon[100914]: pgmap v880: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 50 MiB/s wr, 807 op/s; 19443/281577 objects degraded (6.905%); 7482/281577 objects misplaced (2.657%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:19:16.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:16 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/281577 objects degraded (6.905%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:19:16.879 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:16.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:16.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:16.984 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:16.992 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:19:17.022 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:17.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:17.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:17.124 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:17.153 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:17.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:17.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:17.158 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 82 16 97383 97367 74.1998 52.625 0.0133168 0.0134718 2024-09-21T10:19:17.256 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:17.295 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:17.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:17.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:17.397 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:17.436 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:17.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:17.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:17.538 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:17.582 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:17.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:17.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:17.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:17.721 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:17.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:17.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:17.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:17.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:17.866 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:17.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:17.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:18.005 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:18.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:18.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:18.107 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:18.143 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:18.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:18.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:18.158 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 83 16 98228 98212 73.9421 52.8125 0.00974679 0.0135176 2024-09-21T10:19:18.246 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:18.287 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:18.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:18.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:18.388 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:18.432 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:18.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:18.502 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:18 smithi196 ceph-mon[84784]: pgmap v881: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 49 MiB/s wr, 791 op/s; 19443/285882 objects degraded (6.801%); 7482/285882 objects misplaced (2.617%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:19:18.534 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:18.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:18.578 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:18.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:18.679 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:18.715 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:18.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:18.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:18.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:18.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:18 smithi098 ceph-mon[97580]: pgmap v881: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 49 MiB/s wr, 791 op/s; 19443/285882 objects degraded (6.801%); 7482/285882 objects misplaced (2.617%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:19:18.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:18 smithi098 ceph-mon[100914]: pgmap v881: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 49 MiB/s wr, 791 op/s; 19443/285882 objects degraded (6.801%); 7482/285882 objects misplaced (2.617%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:19:18.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:18.853 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:18.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:18.925 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:19:18.925 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-21T10:19:18.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-21T10:19:18.925 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:19:18.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.925 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45263.0:210850 6.c e191/165 6:3e7fce8f:::benchmark_data_smithi098_169033_object88191:head v 191'13334, mlcod=191'13332)", 2024-09-21T10:19:18.925 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:11:59.223541+0000", 2024-09-21T10:19:18.925 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 439.69665050499998, 2024-09-21T10:19:18.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.069683151999999998, 2024-09-21T10:19:18.925 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:18.925 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:18.925 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:18.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.926 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:18.926 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.223541+0000", 2024-09-21T10:19:18.926 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.926 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:18.926 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.223541+0000", 2024-09-21T10:19:18.926 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.926 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:18.926 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.223540+0000", 2024-09-21T10:19:18.926 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:19:18.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.927 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.927 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:18.927 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.223551+0000", 2024-09-21T10:19:18.927 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0942e-05 2024-09-21T10:19:18.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.927 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.927 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:18.927 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.223564+0000", 2024-09-21T10:19:18.927 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3358e-05 2024-09-21T10:19:18.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.927 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.927 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:18.927 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.223571+0000", 2024-09-21T10:19:18.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4450000000000001e-06 2024-09-21T10:19:18.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:18.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.223630+0000", 2024-09-21T10:19:18.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9052999999999999e-05 2024-09-21T10:19:18.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:18.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.223650+0000", 2024-09-21T10:19:18.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0165000000000001e-05 2024-09-21T10:19:18.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:18.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.292938+0000", 2024-09-21T10:19:18.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.069288649999999993 2024-09-21T10:19:18.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:18.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.293224+0000", 2024-09-21T10:19:18.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00028573999999999998 2024-09-21T10:19:18.929 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.929 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:18.929 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.930 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.930 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45263.0:210858 6.4 e191/165 6:2e378941:::benchmark_data_smithi098_169033_object88199:head v 191'13137, mlcod=191'13134)", 2024-09-21T10:19:18.930 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:11:59.226574+0000", 2024-09-21T10:19:18.930 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 439.69361702200001, 2024-09-21T10:19:18.930 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.074140533999999994, 2024-09-21T10:19:18.930 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:18.930 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:18.930 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:18.930 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.930 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:18.931 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.226574+0000", 2024-09-21T10:19:18.931 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.931 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.931 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:18.931 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.226574+0000", 2024-09-21T10:19:18.931 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.931 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.931 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:18.931 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.226574+0000", 2024-09-21T10:19:18.931 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:19:18.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.932 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.932 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:18.932 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.226580+0000", 2024-09-21T10:19:18.932 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3509999999999998e-06 2024-09-21T10:19:18.932 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.932 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.932 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:18.932 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.226588+0000", 2024-09-21T10:19:18.932 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4880000000000001e-06 2024-09-21T10:19:18.932 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.932 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.932 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:18.932 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.226591+0000", 2024-09-21T10:19:18.932 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5439999999999999e-06 2024-09-21T10:19:18.932 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.933 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:18.933 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.226622+0000", 2024-09-21T10:19:18.933 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0788999999999999e-05 2024-09-21T10:19:18.933 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.933 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:18.933 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.226634+0000", 2024-09-21T10:19:18.933 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1858999999999999e-05 2024-09-21T10:19:18.933 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.933 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:18.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.300690+0000", 2024-09-21T10:19:18.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.074055837999999999 2024-09-21T10:19:18.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:18.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.300715+0000", 2024-09-21T10:19:18.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5230000000000001e-05 2024-09-21T10:19:18.934 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.934 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:18.934 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.934 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45263.0:210856 6.c e191/165 6:38316d98:::benchmark_data_smithi098_169033_object88197:head v 191'13335, mlcod=191'13332)", 2024-09-21T10:19:18.934 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:11:59.227110+0000", 2024-09-21T10:19:18.935 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 439.69308132200001, 2024-09-21T10:19:18.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.073625235999999997, 2024-09-21T10:19:18.935 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:18.935 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:18.935 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:18.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:18.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.227110+0000", 2024-09-21T10:19:18.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:18.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.227110+0000", 2024-09-21T10:19:18.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:18.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.227110+0000", 2024-09-21T10:19:18.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:19:18.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:18.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.227118+0000", 2024-09-21T10:19:18.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0330000000000008e-06 2024-09-21T10:19:18.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:18.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.227126+0000", 2024-09-21T10:19:18.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8320000000000006e-06 2024-09-21T10:19:18.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.937 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.937 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:18.937 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.227129+0000", 2024-09-21T10:19:18.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9190000000000001e-06 2024-09-21T10:19:18.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.937 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.937 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:18.937 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.227159+0000", 2024-09-21T10:19:18.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9598999999999999e-05 2024-09-21T10:19:18.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.937 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.937 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:18.937 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.227175+0000", 2024-09-21T10:19:18.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.552e-05 2024-09-21T10:19:18.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:18.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.300718+0000", 2024-09-21T10:19:18.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.073543735999999998 2024-09-21T10:19:18.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:18.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.300735+0000", 2024-09-21T10:19:18.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7122999999999999e-05 2024-09-21T10:19:18.938 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.938 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:18.938 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.939 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.939 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45263.0:210862 6.4 e191/165 6:2ddce9f0:::benchmark_data_smithi098_169033_object88203:head v 191'13138, mlcod=191'13134)", 2024-09-21T10:19:18.939 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:11:59.230467+0000", 2024-09-21T10:19:18.939 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 439.689724329, 2024-09-21T10:19:18.939 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.070306395999999993, 2024-09-21T10:19:18.939 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:18.940 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:18.940 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:18.940 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.940 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:18.940 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.230467+0000", 2024-09-21T10:19:18.940 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.940 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.940 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.940 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:18.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.230467+0000", 2024-09-21T10:19:18.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:18.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.230466+0000", 2024-09-21T10:19:18.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:19:18.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:18.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.230473+0000", 2024-09-21T10:19:18.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0439999999999998e-06 2024-09-21T10:19:18.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:18.942 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.285816+0000", 2024-09-21T10:19:18.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.055342449000000002 2024-09-21T10:19:18.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:18.942 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.285830+0000", 2024-09-21T10:19:18.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4657e-05 2024-09-21T10:19:18.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:18.942 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.289329+0000", 2024-09-21T10:19:18.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034981719999999999 2024-09-21T10:19:18.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.943 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.943 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:18.943 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.289354+0000", 2024-09-21T10:19:18.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5901000000000002e-05 2024-09-21T10:19:18.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.943 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.943 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:18.943 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.300737+0000", 2024-09-21T10:19:18.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011382534 2024-09-21T10:19:18.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.943 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.943 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:18.943 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:11:59.300774+0000", 2024-09-21T10:19:18.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6532999999999997e-05 2024-09-21T10:19:18.944 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.944 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:18.944 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.944 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.944 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:65364 8.4 e273/265 8:2f211808:::benchmark_data_smithi098_226587_object65363:head v 273'4085, mlcod=273'4083)", 2024-09-21T10:19:18.944 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:18:31.429081+0000", 2024-09-21T10:19:18.944 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 47.491110311, 2024-09-21T10:19:18.944 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.035249097, 2024-09-21T10:19:18.944 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:18.944 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:18.944 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:18.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:18.944 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:18:31.429081+0000", 2024-09-21T10:19:18.945 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.945 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.945 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.945 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:18.945 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:18:31.429081+0000", 2024-09-21T10:19:18.945 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.945 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.945 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.945 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:18.945 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:18:31.429080+0000", 2024-09-21T10:19:18.945 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:19:18.945 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.945 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.945 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:18.945 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:18:31.459819+0000", 2024-09-21T10:19:18.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030738801 2024-09-21T10:19:18.946 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.946 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.946 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:18.946 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:18:31.459830+0000", 2024-09-21T10:19:18.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0811e-05 2024-09-21T10:19:18.946 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.946 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.946 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:18.946 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:18:31.459835+0000", 2024-09-21T10:19:18.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6489999999999996e-06 2024-09-21T10:19:18.946 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.946 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.946 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:18.947 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:18:31.460006+0000", 2024-09-21T10:19:18.947 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017054200000000001 2024-09-21T10:19:18.947 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.947 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.947 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:18.947 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:18:31.460023+0000", 2024-09-21T10:19:18.947 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6677999999999999e-05 2024-09-21T10:19:18.947 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.947 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.947 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:18.947 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:18:31.464257+0000", 2024-09-21T10:19:18.947 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042347160000000003 2024-09-21T10:19:18.947 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.947 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:18.948 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:18:31.464330+0000", 2024-09-21T10:19:18.948 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2837999999999996e-05 2024-09-21T10:19:18.949 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.950 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:18.950 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.950 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:87094 8.c e279/275 8:38884160:::benchmark_data_smithi098_226587_object87093:head v 279'5470, mlcod=0'0)", 2024-09-21T10:19:18.950 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:04.174922+0000", 2024-09-21T10:19:18.950 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 14.745269622, 2024-09-21T10:19:18.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.039272176999999998, 2024-09-21T10:19:18.950 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:18.950 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:18.950 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:18.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.950 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:18.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.174922+0000", 2024-09-21T10:19:18.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:18.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.174922+0000", 2024-09-21T10:19:18.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:18.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.174921+0000", 2024-09-21T10:19:18.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:19:18.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.952 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:18.952 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.174954+0000", 2024-09-21T10:19:18.953 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3120999999999997e-05 2024-09-21T10:19:18.953 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.953 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:18.953 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.174965+0000", 2024-09-21T10:19:18.953 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0839000000000001e-05 2024-09-21T10:19:18.953 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.953 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:18.953 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.174970+0000", 2024-09-21T10:19:18.953 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2000000000000002e-06 2024-09-21T10:19:18.953 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:18.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.186536+0000", 2024-09-21T10:19:18.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011566108 2024-09-21T10:19:18.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:18.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.186561+0000", 2024-09-21T10:19:18.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4266000000000001e-05 2024-09-21T10:19:18.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:18.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.214155+0000", 2024-09-21T10:19:18.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.027594693999999999 2024-09-21T10:19:18.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.955 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:18.955 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.214194+0000", 2024-09-21T10:19:18.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8727999999999999e-05 2024-09-21T10:19:18.955 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.955 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:18.955 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.955 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:87097 8.c e279/275 8:30e3f56d:::benchmark_data_smithi098_226587_object87096:head v 279'5471, mlcod=0'0)", 2024-09-21T10:19:18.955 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:04.178135+0000", 2024-09-21T10:19:18.956 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 14.742056485999999, 2024-09-21T10:19:18.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.036244964999999997, 2024-09-21T10:19:18.956 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:18.956 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:18.956 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:18.956 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.956 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:18.956 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.178135+0000", 2024-09-21T10:19:18.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.956 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.956 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.956 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:18.956 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.178135+0000", 2024-09-21T10:19:18.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.957 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:18.957 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.178135+0000", 2024-09-21T10:19:18.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:19:18.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.957 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:18.957 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.178163+0000", 2024-09-21T10:19:18.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8645000000000001e-05 2024-09-21T10:19:18.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:18.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.178169+0000", 2024-09-21T10:19:18.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8980000000000001e-06 2024-09-21T10:19:18.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:18.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.178172+0000", 2024-09-21T10:19:18.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2100000000000002e-06 2024-09-21T10:19:18.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:18.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.192906+0000", 2024-09-21T10:19:18.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014733642999999999 2024-09-21T10:19:18.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.959 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:18.959 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.192958+0000", 2024-09-21T10:19:18.959 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1761999999999997e-05 2024-09-21T10:19:18.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.959 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:18.959 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.214198+0000", 2024-09-21T10:19:18.959 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.021240467999999998 2024-09-21T10:19:18.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.959 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:18.959 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.214380+0000", 2024-09-21T10:19:18.960 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000181604 2024-09-21T10:19:18.960 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.960 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:18.960 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.960 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.960 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.960 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:87419 8.4 e279/265 8:21db75ec:::benchmark_data_smithi098_226587_object87418:head v 279'5455, mlcod=279'5452)", 2024-09-21T10:19:18.960 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:04.595476+0000", 2024-09-21T10:19:18.960 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 14.324715238, 2024-09-21T10:19:18.960 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037655933000000003, 2024-09-21T10:19:18.960 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:18.960 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:18.960 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:18.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.961 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:18.961 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.595476+0000", 2024-09-21T10:19:18.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.961 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.961 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:18.961 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.595476+0000", 2024-09-21T10:19:18.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.961 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.961 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:18.961 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.595476+0000", 2024-09-21T10:19:18.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:19:18.962 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.962 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.962 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:18.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.595513+0000", 2024-09-21T10:19:18.962 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7722000000000002e-05 2024-09-21T10:19:18.962 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.962 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.962 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:18.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.595528+0000", 2024-09-21T10:19:18.962 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4392e-05 2024-09-21T10:19:18.962 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.962 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.962 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:18.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.595533+0000", 2024-09-21T10:19:18.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4079999999999997e-06 2024-09-21T10:19:18.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.963 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:18.963 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.595746+0000", 2024-09-21T10:19:18.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00021283399999999999 2024-09-21T10:19:18.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.963 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:18.963 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.595765+0000", 2024-09-21T10:19:18.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8527999999999999e-05 2024-09-21T10:19:18.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.964 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.964 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:18.964 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.633105+0000", 2024-09-21T10:19:18.964 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037340456000000001 2024-09-21T10:19:18.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.964 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.964 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:18.964 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.633132+0000", 2024-09-21T10:19:18.964 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7192999999999999e-05 2024-09-21T10:19:18.964 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.964 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:18.964 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.964 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.965 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:87404 8.c e279/275 8:397eaefd:::benchmark_data_smithi098_226587_object87403:head v 279'5493, mlcod=0'0)", 2024-09-21T10:19:18.965 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:04.595591+0000", 2024-09-21T10:19:18.965 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 14.324599996, 2024-09-21T10:19:18.965 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037569685999999998, 2024-09-21T10:19:18.965 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:18.965 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:18.965 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:18.965 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.965 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:18.965 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.595591+0000", 2024-09-21T10:19:18.965 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.965 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.965 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.966 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:18.966 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.595591+0000", 2024-09-21T10:19:18.966 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.966 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.966 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.966 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:18.966 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.595591+0000", 2024-09-21T10:19:18.966 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:19:18.966 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.966 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.966 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:18.966 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.595621+0000", 2024-09-21T10:19:18.966 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0477e-05 2024-09-21T10:19:18.967 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.967 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:18.967 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.595632+0000", 2024-09-21T10:19:18.967 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0546e-05 2024-09-21T10:19:18.967 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.967 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:18.967 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.595636+0000", 2024-09-21T10:19:18.967 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1849999999999997e-06 2024-09-21T10:19:18.967 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.967 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:18.967 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.595681+0000", 2024-09-21T10:19:18.968 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5235000000000003e-05 2024-09-21T10:19:18.968 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.968 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.968 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:18.968 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.595730+0000", 2024-09-21T10:19:18.968 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8798999999999998e-05 2024-09-21T10:19:18.968 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.968 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.968 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:18.968 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.633135+0000", 2024-09-21T10:19:18.968 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037404785000000003 2024-09-21T10:19:18.968 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.968 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.969 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:18.969 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.633161+0000", 2024-09-21T10:19:18.969 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6166999999999999e-05 2024-09-21T10:19:18.969 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.969 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:18.969 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.969 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.969 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.969 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:87415 8.c e279/275 8:32e7f505:::benchmark_data_smithi098_226587_object87414:head v 279'5494, mlcod=0'0)", 2024-09-21T10:19:18.969 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:04.598776+0000", 2024-09-21T10:19:18.969 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 14.321415607, 2024-09-21T10:19:18.969 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.034407167000000002, 2024-09-21T10:19:18.969 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:18.970 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:18.970 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:18.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:18.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.598776+0000", 2024-09-21T10:19:18.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.970 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:18.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.598776+0000", 2024-09-21T10:19:18.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.970 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.971 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:18.971 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.598775+0000", 2024-09-21T10:19:18.971 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:19:18.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.971 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:18.971 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.598817+0000", 2024-09-21T10:19:18.971 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2194999999999998e-05 2024-09-21T10:19:18.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.971 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:18.971 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.598833+0000", 2024-09-21T10:19:18.971 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5279999999999999e-05 2024-09-21T10:19:18.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.972 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:18.972 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.598838+0000", 2024-09-21T10:19:18.972 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4870000000000002e-06 2024-09-21T10:19:18.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.972 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:18.972 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.602833+0000", 2024-09-21T10:19:18.972 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039946749999999996 2024-09-21T10:19:18.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.972 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:18.972 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.602858+0000", 2024-09-21T10:19:18.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5420000000000001e-05 2024-09-21T10:19:18.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.973 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.973 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:18.973 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.633163+0000", 2024-09-21T10:19:18.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030305113000000002 2024-09-21T10:19:18.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.973 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.973 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:18.973 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:04.633183+0000", 2024-09-21T10:19:18.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9845999999999999e-05 2024-09-21T10:19:18.973 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.974 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:18.974 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.974 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.974 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:91368 8.c e279/275 8:3ce5bdbb:::benchmark_data_smithi098_226587_object91367:head v 279'5734, mlcod=0'0)", 2024-09-21T10:19:18.974 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:09.396933+0000", 2024-09-21T10:19:18.974 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.5232589030000003, 2024-09-21T10:19:18.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.035592356999999998, 2024-09-21T10:19:18.974 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:18.974 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:18.974 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:18.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.974 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:18.975 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:09.396933+0000", 2024-09-21T10:19:18.975 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.975 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.975 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.975 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:18.975 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:09.396933+0000", 2024-09-21T10:19:18.975 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.975 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.975 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.975 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:18.975 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:09.396931+0000", 2024-09-21T10:19:18.975 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:19:18.975 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.975 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.976 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:18.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:09.397001+0000", 2024-09-21T10:19:18.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9350000000000005e-05 2024-09-21T10:19:18.976 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.976 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:18.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:09.397014+0000", 2024-09-21T10:19:18.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3388e-05 2024-09-21T10:19:18.976 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.976 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:18.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:09.397021+0000", 2024-09-21T10:19:18.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8240000000000002e-06 2024-09-21T10:19:18.977 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.977 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:18.977 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:09.407108+0000", 2024-09-21T10:19:18.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010087196 2024-09-21T10:19:18.977 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.977 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:18.977 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:09.407132+0000", 2024-09-21T10:19:18.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4273999999999998e-05 2024-09-21T10:19:18.977 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.977 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:18.977 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:09.432486+0000", 2024-09-21T10:19:18.978 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.025353530999999999 2024-09-21T10:19:18.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.978 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.978 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:18.978 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:09.432525+0000", 2024-09-21T10:19:18.978 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9054000000000003e-05 2024-09-21T10:19:18.978 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.978 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:18.978 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.978 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.978 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:92348 8.c e279/275 8:3a708625:::benchmark_data_smithi098_226587_object92347:head v 279'5793, mlcod=0'0)", 2024-09-21T10:19:18.978 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:10.741694+0000", 2024-09-21T10:19:18.978 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.1784976139999994, 2024-09-21T10:19:18.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.043361987999999997, 2024-09-21T10:19:18.979 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:18.979 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:18.979 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:18.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:18.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.741694+0000", 2024-09-21T10:19:18.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:18.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.741694+0000", 2024-09-21T10:19:18.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.980 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:18.980 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.741693+0000", 2024-09-21T10:19:18.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:19:18.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.980 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:18.980 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.741734+0000", 2024-09-21T10:19:18.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1356000000000002e-05 2024-09-21T10:19:18.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.980 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:18.980 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.741744+0000", 2024-09-21T10:19:18.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6830000000000003e-06 2024-09-21T10:19:18.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:18.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.741749+0000", 2024-09-21T10:19:18.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2070000000000003e-06 2024-09-21T10:19:18.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:18.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.754205+0000", 2024-09-21T10:19:18.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012455730999999999 2024-09-21T10:19:18.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:18.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.754262+0000", 2024-09-21T10:19:18.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7166000000000001e-05 2024-09-21T10:19:18.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:18.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.784961+0000", 2024-09-21T10:19:18.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030698494 2024-09-21T10:19:18.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:18.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.785056+0000", 2024-09-21T10:19:18.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5082999999999997e-05 2024-09-21T10:19:18.982 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.983 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:18.983 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.983 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.983 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:92353 8.4 e279/265 8:2615a936:::benchmark_data_smithi098_226587_object92352:head v 279'5755, mlcod=279'5751)", 2024-09-21T10:19:18.983 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:10.742948+0000", 2024-09-21T10:19:18.983 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.1772438279999999, 2024-09-21T10:19:18.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037203992999999998, 2024-09-21T10:19:18.983 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:18.983 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:18.983 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:18.983 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.983 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:18.984 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.742948+0000", 2024-09-21T10:19:18.984 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.984 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.984 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:18.984 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.742948+0000", 2024-09-21T10:19:18.984 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.984 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.984 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:18.984 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.742947+0000", 2024-09-21T10:19:18.984 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:19:18.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.984 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.984 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:18.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.743185+0000", 2024-09-21T10:19:18.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00023841699999999999 2024-09-21T10:19:18.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:18.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.743199+0000", 2024-09-21T10:19:18.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3947e-05 2024-09-21T10:19:18.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:18.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.743205+0000", 2024-09-21T10:19:18.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3750000000000002e-06 2024-09-21T10:19:18.986 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.751413+0000", 2024-09-21T10:19:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0082084370000000007 2024-09-21T10:19:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.751431+0000", 2024-09-21T10:19:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8102e-05 2024-09-21T10:19:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.780098+0000", 2024-09-21T10:19:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.028666598000000001 2024-09-21T10:19:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.780152+0000", 2024-09-21T10:19:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3718999999999998e-05 2024-09-21T10:19:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:92357 8.c e279/275 8:3ec8e5c5:::benchmark_data_smithi098_226587_object92356:head v 279'5794, mlcod=0'0)", 2024-09-21T10:19:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:10.746458+0000", 2024-09-21T10:19:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.1737331340000008, 2024-09-21T10:19:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.038655241, 2024-09-21T10:19:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.746458+0000", 2024-09-21T10:19:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.746458+0000", 2024-09-21T10:19:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.746457+0000", 2024-09-21T10:19:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:19:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.746498+0000", 2024-09-21T10:19:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1088999999999999e-05 2024-09-21T10:19:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.746511+0000", 2024-09-21T10:19:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2889e-05 2024-09-21T10:19:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.746518+0000", 2024-09-21T10:19:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2389999999999999e-06 2024-09-21T10:19:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.764305+0000", 2024-09-21T10:19:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017787164000000001 2024-09-21T10:19:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.764331+0000", 2024-09-21T10:19:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5871e-05 2024-09-21T10:19:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.785062+0000", 2024-09-21T10:19:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.020731191 2024-09-21T10:19:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.785114+0000", 2024-09-21T10:19:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1756999999999999e-05 2024-09-21T10:19:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:92359 8.4 e279/265 8:267e0b2b:::benchmark_data_smithi098_226587_object92358:head v 279'5756, mlcod=279'5751)", 2024-09-21T10:19:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:10.746531+0000", 2024-09-21T10:19:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.1736608410000002, 2024-09-21T10:19:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0386402, 2024-09-21T10:19:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.746531+0000", 2024-09-21T10:19:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.746531+0000", 2024-09-21T10:19:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.746530+0000", 2024-09-21T10:19:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:19:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.746561+0000", 2024-09-21T10:19:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0365000000000002e-05 2024-09-21T10:19:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.746568+0000", 2024-09-21T10:19:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0439999999999998e-06 2024-09-21T10:19:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.746571+0000", 2024-09-21T10:19:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7799999999999998e-06 2024-09-21T10:19:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.771016+0000", 2024-09-21T10:19:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.024444398999999999 2024-09-21T10:19:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.771041+0000", 2024-09-21T10:19:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5582e-05 2024-09-21T10:19:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.785118+0000", 2024-09-21T10:19:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014076740000000001 2024-09-21T10:19:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:10.785171+0000", 2024-09-21T10:19:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2729999999999998e-05 2024-09-21T10:19:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:94165 8.c e279/275 8:37ee5c62:::benchmark_data_smithi098_226587_object94164:head v 279'5906, mlcod=0'0)", 2024-09-21T10:19:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:12.996002+0000", 2024-09-21T10:19:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 5.9241898930000003, 2024-09-21T10:19:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037605618, 2024-09-21T10:19:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.996002+0000", 2024-09-21T10:19:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:18.999 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.996002+0000", 2024-09-21T10:19:18.999 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:18.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:18.999 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.996001+0000", 2024-09-21T10:19:18.999 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:19:18.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:18.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:18.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:18.999 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.996045+0000", 2024-09-21T10:19:18.999 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4189e-05 2024-09-21T10:19:18.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.000 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.000 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:19.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.996053+0000", 2024-09-21T10:19:19.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1529999999999997e-06 2024-09-21T10:19:19.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.000 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.000 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:19.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.996059+0000", 2024-09-21T10:19:19.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5840000000000002e-06 2024-09-21T10:19:19.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.000 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.000 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:19.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.996110+0000", 2024-09-21T10:19:19.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0908000000000001e-05 2024-09-21T10:19:19.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.001 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:19.001 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:12.996126+0000", 2024-09-21T10:19:19.001 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6025999999999999e-05 2024-09-21T10:19:19.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.001 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:19.001 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:13.033551+0000", 2024-09-21T10:19:19.001 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037425184 2024-09-21T10:19:19.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.001 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:13.033607+0000", 2024-09-21T10:19:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6378999999999997e-05 2024-09-21T10:19:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:98785 8.c e280/275 8:31404c44:::benchmark_data_smithi098_226587_object98784:head v 280'6222, mlcod=0'0)", 2024-09-21T10:19:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:18.804899+0000", 2024-09-21T10:19:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.115291986, 2024-09-21T10:19:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.044358476000000001, 2024-09-21T10:19:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.804899+0000", 2024-09-21T10:19:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.804899+0000", 2024-09-21T10:19:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.804898+0000", 2024-09-21T10:19:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:19:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.804948+0000", 2024-09-21T10:19:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9554e-05 2024-09-21T10:19:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.804965+0000", 2024-09-21T10:19:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7034999999999999e-05 2024-09-21T10:19:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.804971+0000", 2024-09-21T10:19:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2539999999999998e-06 2024-09-21T10:19:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.806532+0000", 2024-09-21T10:19:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001561123 2024-09-21T10:19:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.806555+0000", 2024-09-21T10:19:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3150999999999999e-05 2024-09-21T10:19:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.849232+0000", 2024-09-21T10:19:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.042676256000000003 2024-09-21T10:19:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.849258+0000", 2024-09-21T10:19:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6327000000000001e-05 2024-09-21T10:19:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:98799 8.4 e280/265 8:2e3b5ee7:::benchmark_data_smithi098_226587_object98798:head v 280'6156, mlcod=280'6151)", 2024-09-21T10:19:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:18.804984+0000", 2024-09-21T10:19:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.11520704499999999, 2024-09-21T10:19:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.044244115000000001, 2024-09-21T10:19:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.804984+0000", 2024-09-21T10:19:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.804984+0000", 2024-09-21T10:19:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.804984+0000", 2024-09-21T10:19:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:19:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:19.009 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.805023+0000", 2024-09-21T10:19:19.009 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8797000000000001e-05 2024-09-21T10:19:19.009 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.009 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.009 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:19.009 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.805029+0000", 2024-09-21T10:19:19.009 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5930000000000002e-06 2024-09-21T10:19:19.009 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.009 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.009 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:19.009 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.805033+0000", 2024-09-21T10:19:19.010 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.427e-06 2024-09-21T10:19:19.010 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.010 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.010 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:19.010 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.806572+0000", 2024-09-21T10:19:19.010 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015386950000000001 2024-09-21T10:19:19.010 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.010 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.010 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:19.010 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.806592+0000", 2024-09-21T10:19:19.010 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9936e-05 2024-09-21T10:19:19.010 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.010 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.011 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:19.011 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.849184+0000", 2024-09-21T10:19:19.011 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.042592597000000003 2024-09-21T10:19:19.011 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.011 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.011 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:19.011 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.849229+0000", 2024-09-21T10:19:19.011 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4409999999999997e-05 2024-09-21T10:19:19.011 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:19.011 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:19.011 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:19.011 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.011 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.012 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:98790 8.c e280/275 8:3edcf424:::benchmark_data_smithi098_226587_object98789:head v 280'6223, mlcod=0'0)", 2024-09-21T10:19:19.012 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:18.808768+0000", 2024-09-21T10:19:19.012 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.111423457, 2024-09-21T10:19:19.012 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.040513079, 2024-09-21T10:19:19.012 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:19.012 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:19.012 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:19.012 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.012 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:19.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.808768+0000", 2024-09-21T10:19:19.012 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:19.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.012 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.012 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:19.013 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.808768+0000", 2024-09-21T10:19:19.013 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:19.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.013 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.013 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:19.013 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.808767+0000", 2024-09-21T10:19:19.013 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:19:19.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.013 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.013 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:19.013 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.808818+0000", 2024-09-21T10:19:19.013 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1405e-05 2024-09-21T10:19:19.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.013 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.014 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:19.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.808833+0000", 2024-09-21T10:19:19.014 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4572e-05 2024-09-21T10:19:19.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.014 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.014 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:19.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.808839+0000", 2024-09-21T10:19:19.014 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1310000000000003e-06 2024-09-21T10:19:19.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.014 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.014 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:19.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.812105+0000", 2024-09-21T10:19:19.014 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032664009999999999 2024-09-21T10:19:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.812129+0000", 2024-09-21T10:19:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3706e-05 2024-09-21T10:19:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.849262+0000", 2024-09-21T10:19:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037132765999999998 2024-09-21T10:19:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.849281+0000", 2024-09-21T10:19:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9208e-05 2024-09-21T10:19:19.016 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:19.016 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:19.016 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:19.016 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.016 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.016 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:98803 8.c e280/275 8:314adc6a:::benchmark_data_smithi098_226587_object98802:head v 280'6224, mlcod=0'0)", 2024-09-21T10:19:19.016 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:18.812803+0000", 2024-09-21T10:19:19.016 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.107388478, 2024-09-21T10:19:19.016 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.036499420999999997, 2024-09-21T10:19:19.016 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:19.016 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:19.016 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:19.016 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.017 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:19.017 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.812803+0000", 2024-09-21T10:19:19.017 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:19.017 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.017 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.017 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:19.017 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.812803+0000", 2024-09-21T10:19:19.017 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:19.017 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.017 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.017 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:19.017 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.812802+0000", 2024-09-21T10:19:19.017 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:19:19.017 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.018 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.018 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:19.018 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.812836+0000", 2024-09-21T10:19:19.018 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4115000000000001e-05 2024-09-21T10:19:19.018 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.018 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.018 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:19.018 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.812850+0000", 2024-09-21T10:19:19.018 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3574000000000001e-05 2024-09-21T10:19:19.018 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.018 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.018 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:19.018 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.812855+0000", 2024-09-21T10:19:19.018 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2159999999999997e-06 2024-09-21T10:19:19.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.019 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.019 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:19.019 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.818800+0000", 2024-09-21T10:19:19.019 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059453570000000001 2024-09-21T10:19:19.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.019 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.019 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:19.019 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.818824+0000", 2024-09-21T10:19:19.019 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.319e-05 2024-09-21T10:19:19.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.019 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.019 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:19.019 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.849283+0000", 2024-09-21T10:19:19.019 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030459389 2024-09-21T10:19:19.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:19.020 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:19.020 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:19.020 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:18.849302+0000", 2024-09-21T10:19:19.020 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9417999999999999e-05 2024-09-21T10:19:19.020 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:19.020 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:19.020 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:19.020 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:19.020 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:19.020 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:19:19.021 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:19.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:19.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:19.123 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:19.160 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 84 15 99097 99082 73.7078 54.375 0.032084 0.0135616 2024-09-21T10:19:19.168 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:19.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:19.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:19.270 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:19.309 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:19.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:19.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:19.410 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:19.447 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:19.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:19.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:19.549 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:19.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:19.581 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:19.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:19.680 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:19:19.681 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:19:19.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:19.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:19.720 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:19.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:19.725 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:19:19.726 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:19:19.762 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:19:19.763 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:19:19.796 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:19:19.796 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:19:19.821 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:19.827 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:19:19.827 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:19:19.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:19:19.853 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:19.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:19.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:19.859 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:19:19.859 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:19:19.885 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:19:19.893 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:19:19.894 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:19:19.926 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:19:19.926 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:19:19.955 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:20.003 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:20.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:20.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:20.105 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:20.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:20.142 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:20.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:20.161 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 85 16 99899 99883 73.4296 50.0625 0.00866752 0.0136128 2024-09-21T10:19:20.243 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:20.279 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:20.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:20.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:20.353 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:19:20.381 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:20.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:20.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:20.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:20.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:20 smithi196 ceph-mon[84784]: pgmap v882: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 49 MiB/s wr, 780 op/s; 19443/293316 objects degraded (6.629%); 7326/293316 objects misplaced (2.498%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T10:19:20.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:20 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3325457230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:20.488 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:19:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:19:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:19:20.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:20.563 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:20.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:20.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:20.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:20 smithi098 ceph-mon[97580]: pgmap v882: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 49 MiB/s wr, 780 op/s; 19443/293316 objects degraded (6.629%); 7326/293316 objects misplaced (2.498%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T10:19:20.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:20 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3325457230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:20.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:20 smithi098 ceph-mon[100914]: pgmap v882: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 49 MiB/s wr, 780 op/s; 19443/293316 objects degraded (6.629%); 7326/293316 objects misplaced (2.498%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T10:19:20.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:20 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3325457230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:20.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:20.697 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:20.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:20.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:20.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:20.833 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:20.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:20.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:20.936 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:20.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:20.979 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:20.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:21.080 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:21.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:19:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:19:20] "GET /metrics HTTP/1.1" 200 36544 "" "Prometheus/2.51.0" 2024-09-21T10:19:21.115 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:21.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:21.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:21.160 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 86 16 100680 100664 73.1434 48.8125 0.0261347 0.013665 2024-09-21T10:19:21.217 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:21.252 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:21.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:21.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:21.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:21.383 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:21.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:21.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:21.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:21.518 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:21.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:21.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:21.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:21 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/293316 objects degraded (6.629%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:19:21.621 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:21.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:21.645 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:21.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:21.746 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:21.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:21.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:21.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:21 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/293316 objects degraded (6.629%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:19:21.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:21 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/293316 objects degraded (6.629%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:19:21.885 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:21.945 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:21.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:21.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:22.047 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:22.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:22.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:22.160 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:19:22.160 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:19:22.160 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 87 16 101465 101449 72.8665 49.0625 0.016415 0.0137177 2024-09-21T10:19:22.183 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:22.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:22.212 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:22.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:22.313 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:22.357 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:22.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:22.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:22.437 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7] out_osds: [4, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:19:22.438 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T10:19:22.458 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:22.491 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:22.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:22.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:22.593 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:22.614 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:22.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:22.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:22.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:22.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:22.739 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:22.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:22.749 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:22 smithi196 ceph-mon[84784]: pgmap v883: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 49 MiB/s wr, 781 op/s; 19443/293316 objects degraded (6.629%); 7326/293316 objects misplaced (2.498%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T10:19:22.749 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:22.790 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:19:22.840 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:22 smithi098 ceph-mon[97580]: pgmap v883: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 49 MiB/s wr, 781 op/s; 19443/293316 objects degraded (6.629%); 7326/293316 objects misplaced (2.498%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T10:19:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:22.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:22 smithi098 ceph-mon[100914]: pgmap v883: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 49 MiB/s wr, 781 op/s; 19443/293316 objects degraded (6.629%); 7326/293316 objects misplaced (2.498%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T10:19:22.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:22.868 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:22.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:22.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:22.970 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:23.012 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:23.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:23.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:23.114 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:23.137 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:19:23.159 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:23.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:23.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:23.160 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 88 16 102051 102035 72.4547 36.625 0.0061826 0.0137945 2024-09-21T10:19:23.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:23.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:23.305 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:23.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:23.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:23.443 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:23.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:23.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:23.531 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:23.545 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:23.578 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:23.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:23.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:23.588 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:23.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:23.738 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:23.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:23.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:23.840 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:23.873 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:23.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:23.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:23.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:24.026 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:24.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:24.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:24.129 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:24.160 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 89 16 102820 102804 72.1806 48.0625 0.0185307 0.0138485 2024-09-21T10:19:24.162 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:24.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:24.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:24.266 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:24.306 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:24.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:24.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:24.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:24.441 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:24.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:24.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:24.543 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:24.577 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:24.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:24.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:24.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:24 smithi098 ceph-mon[97580]: pgmap v884: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.1 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 51 MiB/s wr, 809 op/s; 19443/301611 objects degraded (6.446%); 7326/301611 objects misplaced (2.429%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:19:24.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:24 smithi098 ceph-mon[100914]: pgmap v884: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.1 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 51 MiB/s wr, 809 op/s; 19443/301611 objects degraded (6.446%); 7326/301611 objects misplaced (2.429%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:19:24.679 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:24.715 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:24.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:24.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:24.763 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:24 smithi196 ceph-mon[84784]: pgmap v884: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.1 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 51 MiB/s wr, 809 op/s; 19443/301611 objects degraded (6.446%); 7326/301611 objects misplaced (2.429%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:19:24.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:24.860 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:24.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:24.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:24.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:25.002 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:25.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:25.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:25.041 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:19:25.041 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:19:25.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:25.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:25.147 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:25.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:25.160 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 90 1 103667 103666 71.9772 53.875 0.0208288 0.0138878 2024-09-21T10:19:25.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:25.270 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:25.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:25.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:25.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:25.393 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:25.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:25.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:25.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:19:25.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:25.517 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:25.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:25.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:25.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:25.643 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:25.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:25.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:25.744 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:25.767 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:25.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:25.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:25.783 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:19:25.869 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:25.891 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:25.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:25.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:25.992 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:26.015 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:26.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:26.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:26.075 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:19:26.116 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:26.137 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:26.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:26.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:26.160 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Total time run: 90.0221 2024-09-21T10:19:26.161 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Total writes made: 103667 2024-09-21T10:19:26.161 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Write size: 65536 2024-09-21T10:19:26.161 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Object size: 65536 2024-09-21T10:19:26.161 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Bandwidth (MB/sec): 71.9733 2024-09-21T10:19:26.161 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Stddev Bandwidth: 42.9409 2024-09-21T10:19:26.161 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Max bandwidth (MB/sec): 200.125 2024-09-21T10:19:26.161 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Min bandwidth (MB/sec): 0 2024-09-21T10:19:26.161 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Average IOPS: 1151 2024-09-21T10:19:26.161 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Stddev IOPS: 687.057 2024-09-21T10:19:26.161 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Max IOPS: 3202 2024-09-21T10:19:26.161 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Min IOPS: 0 2024-09-21T10:19:26.161 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Average Latency(s): 0.0138879 2024-09-21T10:19:26.161 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Stddev Latency(s): 0.0984973 2024-09-21T10:19:26.161 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Max latency(s): 9.22765 2024-09-21T10:19:26.161 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Min latency(s): 0.00125976 2024-09-21T10:19:26.165 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:19:26.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:26.274 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:26.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:26.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:26.376 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:26.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:26.397 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:26.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:26.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:26.531 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:26.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:26.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:26.633 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:26.665 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:26.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:26.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:26.767 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:26.798 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:26.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:26.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:26.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:26 smithi098 ceph-mon[97580]: pgmap v885: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 48 MiB/s wr, 762 op/s; 19443/305340 objects degraded (6.368%); 7199/305340 objects misplaced (2.358%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:19:26.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:26 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/305340 objects degraded (6.368%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:19:26.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:26 smithi098 ceph-mon[100914]: pgmap v885: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 48 MiB/s wr, 762 op/s; 19443/305340 objects degraded (6.368%); 7199/305340 objects misplaced (2.358%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:19:26.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:26 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/305340 objects degraded (6.368%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:19:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:26 smithi196 ceph-mon[84784]: pgmap v885: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 48 MiB/s wr, 762 op/s; 19443/305340 objects degraded (6.368%); 7199/305340 objects misplaced (2.358%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:19:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:26 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/305340 objects degraded (6.368%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:19:26.900 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:26.927 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:26.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:26.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:27.029 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:27.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:27.068 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:27.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:27.169 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:27.199 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:27.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:27.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:27.301 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:27.340 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:27.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:27.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:27.442 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:27.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:27.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:27.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:27.618 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:27.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:27.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:27.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:27.763 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:27.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:27.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:27.865 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:27.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:27.894 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:27.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:27.954 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:27.995 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:28.022 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:28.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:28.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:28.124 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:28.154 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:28.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:28.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:28.202 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:19:28.202 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:19:28.256 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:28.285 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:28.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:28.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:28.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:28.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:28.432 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:28.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:28.533 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:28.561 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:28.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:28.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:28.663 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:28.693 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:28.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:28.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:28.738 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:28 smithi098 ceph-mon[97580]: pgmap v886: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 46 MiB/s wr, 751 op/s; 19443/308001 objects degraded (6.313%); 7199/308001 objects misplaced (2.337%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:19:28.738 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:28 smithi098 ceph-mon[100914]: pgmap v886: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 46 MiB/s wr, 751 op/s; 19443/308001 objects degraded (6.313%); 7199/308001 objects misplaced (2.337%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:19:28.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:28.838 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:28.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:28.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:28 smithi196 ceph-mon[84784]: pgmap v886: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 46 MiB/s wr, 751 op/s; 19443/308001 objects degraded (6.313%); 7199/308001 objects misplaced (2.337%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:19:28.940 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:28.973 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:28.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:28.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:29.011 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:19:29.075 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:29.107 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:29.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:29.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:29.208 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:29.243 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:29.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:29.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:29.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:29.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:29.390 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:29.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:29.398 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:19:29.478 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:19:29.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:29.545 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:29.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:29.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:29.647 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:29.680 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:29.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:29.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:29.729 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:29 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4233998796' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:19:29.781 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:29.810 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:29 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4233998796' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:19:29.811 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:29 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4233998796' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:19:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:29.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:29.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:29.932 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:29.978 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:29.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:29.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:30.080 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:30.116 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:30.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:30.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:30.218 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:30.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:30.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:30.355 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:30.404 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:30.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:30.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:30.477 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:19:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:19:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:19:30.506 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:30.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:30 smithi098 ceph-mon[97580]: pgmap v887: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 85 B/s rd, 44 MiB/s wr, 801 op/s; 19443/307437 objects degraded (6.324%); 6955/307437 objects misplaced (2.262%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:19:30.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:30 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:19:30.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:30 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4233998796' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:19:30.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:30 smithi098 ceph-mon[97580]: osdmap e281: 8 total, 8 up, 6 in 2024-09-21T10:19:30.511 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:30 smithi098 ceph-mon[100914]: pgmap v887: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 85 B/s rd, 44 MiB/s wr, 801 op/s; 19443/307437 objects degraded (6.324%); 6955/307437 objects misplaced (2.262%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:19:30.511 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:30 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:19:30.511 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:30 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4233998796' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:19:30.511 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:30 smithi098 ceph-mon[100914]: osdmap e281: 8 total, 8 up, 6 in 2024-09-21T10:19:30.537 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:30.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:30.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:30.639 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:30.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:30.671 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:30.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:30.731 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:30 smithi196 ceph-mon[84784]: pgmap v887: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 85 B/s rd, 44 MiB/s wr, 801 op/s; 19443/307437 objects degraded (6.324%); 6955/307437 objects misplaced (2.262%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:19:30.731 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:30 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:19:30.731 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:30 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4233998796' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:19:30.732 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:30 smithi196 ceph-mon[84784]: osdmap e281: 8 total, 8 up, 6 in 2024-09-21T10:19:30.772 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:30.805 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:30.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:30.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:30.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:19:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:19:30] "GET /metrics HTTP/1.1" 200 36543 "" "Prometheus/2.51.0" 2024-09-21T10:19:30.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:30.951 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:30.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:30.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:31.053 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:31.098 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:31.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:31.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:31.199 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:31.227 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:31.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:31.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:31.233 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:19:31.234 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:19:31.269 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:19:31.269 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:19:31.303 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:19:31.303 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:19:31.328 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:31.336 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:19:31.337 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:19:31.366 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:19:31.367 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:19:31.376 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:31.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:31.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:31.406 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:19:31.407 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:19:31.436 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:19:31.436 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:19:31.472 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:19:31.472 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:19:31.477 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:31.508 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:31.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:31.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:31.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:31.647 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:31.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:31.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:31.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:31 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/307437 objects degraded (6.324%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:19:31.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:31 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3785029910' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:31.658 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:19:31.658 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:19:31.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.658 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.45794.0:108225 8.d 8:bf907d54:::benchmark_data_smithi098_226587_object4555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-21T10:19:31.659 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:31.637445+0000", 2024-09-21T10:19:31.659 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.018229535000000002, 2024-09-21T10:19:31.659 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018261792999999998, 2024-09-21T10:19:31.659 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:31.659 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:19:31.659 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:31.659 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.45794", 2024-09-21T10:19:31.659 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:19:31.659 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 108225 2024-09-21T10:19:31.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.659 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:31.659 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.659 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:31.659 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.637445+0000", 2024-09-21T10:19:31.660 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:31.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.660 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.660 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:31.660 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.637445+0000", 2024-09-21T10:19:31.660 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:31.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.660 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.660 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:31.660 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.637443+0000", 2024-09-21T10:19:31.660 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:19:31.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.660 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.660 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:31.660 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.637455+0000", 2024-09-21T10:19:31.661 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1626e-05 2024-09-21T10:19:31.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.661 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:31.661 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.637472+0000", 2024-09-21T10:19:31.661 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6770999999999999e-05 2024-09-21T10:19:31.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.661 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:31.661 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.637478+0000", 2024-09-21T10:19:31.661 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3659999999999997e-06 2024-09-21T10:19:31.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.661 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:31.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.638127+0000", 2024-09-21T10:19:31.662 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00064915199999999995 2024-09-21T10:19:31.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.662 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.662 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:31.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.638540+0000", 2024-09-21T10:19:31.662 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00041290199999999998 2024-09-21T10:19:31.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.662 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.662 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:19:31.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.639005+0000", 2024-09-21T10:19:31.662 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00046479399999999998 2024-09-21T10:19:31.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.662 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:19:31.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.644031+0000", 2024-09-21T10:19:31.663 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0050266410000000001 2024-09-21T10:19:31.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:31.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.648170+0000", 2024-09-21T10:19:31.663 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041387450000000001 2024-09-21T10:19:31.663 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:31.663 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:31.663 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:31.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.663 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.45794.0:108226 8.d 8:bb2cfcd9:::benchmark_data_smithi098_226587_object4556:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-21T10:19:31.664 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:31.637989+0000", 2024-09-21T10:19:31.664 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.017685257999999999, 2024-09-21T10:19:31.664 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017795495000000001, 2024-09-21T10:19:31.664 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:31.664 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:19:31.664 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:31.664 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.45794", 2024-09-21T10:19:31.664 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:19:31.664 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 108226 2024-09-21T10:19:31.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.664 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:31.664 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.664 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:31.664 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.637989+0000", 2024-09-21T10:19:31.664 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:31.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.665 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.665 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:31.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.637989+0000", 2024-09-21T10:19:31.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:31.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.665 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.665 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:31.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.637989+0000", 2024-09-21T10:19:31.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:19:31.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.665 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.665 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:31.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.637993+0000", 2024-09-21T10:19:31.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3610000000000003e-06 2024-09-21T10:19:31.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.666 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:31.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.638003+0000", 2024-09-21T10:19:31.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0225e-05 2024-09-21T10:19:31.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.666 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:31.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.638009+0000", 2024-09-21T10:19:31.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5720000000000002e-06 2024-09-21T10:19:31.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.667 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:31.667 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.644053+0000", 2024-09-21T10:19:31.667 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0060444019999999999 2024-09-21T10:19:31.667 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.667 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.667 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:31.667 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.644183+0000", 2024-09-21T10:19:31.667 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012982400000000001 2024-09-21T10:19:31.667 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.667 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.667 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:19:31.667 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.644305+0000", 2024-09-21T10:19:31.667 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000122061 2024-09-21T10:19:31.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.668 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.668 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:19:31.668 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.648470+0000", 2024-09-21T10:19:31.668 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041650130000000004 2024-09-21T10:19:31.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.668 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.668 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:31.668 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.648713+0000", 2024-09-21T10:19:31.668 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000242749 2024-09-21T10:19:31.668 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:31.668 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:31.668 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:31.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.668 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.668 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.45794.0:108237 8.d 8:ba3ea1b6:::benchmark_data_smithi098_226587_object4567:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-21T10:19:31.669 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:31.652889+0000", 2024-09-21T10:19:31.669 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0027854910000000002, 2024-09-21T10:19:31.669 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029653570000000001, 2024-09-21T10:19:31.669 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:31.669 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:19:31.669 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:31.669 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.45794", 2024-09-21T10:19:31.669 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:19:31.669 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 108237 2024-09-21T10:19:31.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.669 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:31.669 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.669 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:31.669 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.652889+0000", 2024-09-21T10:19:31.670 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:31.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.670 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.670 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:31.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.652889+0000", 2024-09-21T10:19:31.670 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:31.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.670 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.670 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:31.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.652887+0000", 2024-09-21T10:19:31.670 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:19:31.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.670 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.671 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:31.671 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.652897+0000", 2024-09-21T10:19:31.671 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3060000000000003e-06 2024-09-21T10:19:31.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.671 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.671 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:31.671 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.652918+0000", 2024-09-21T10:19:31.671 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1682999999999999e-05 2024-09-21T10:19:31.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.671 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.671 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:31.671 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.652925+0000", 2024-09-21T10:19:31.671 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.131e-06 2024-09-21T10:19:31.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.672 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.672 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:31.672 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.653003+0000", 2024-09-21T10:19:31.672 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8135999999999999e-05 2024-09-21T10:19:31.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.672 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.672 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:31.672 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.653168+0000", 2024-09-21T10:19:31.672 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016425500000000001 2024-09-21T10:19:31.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.673 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.673 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:19:31.673 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.653294+0000", 2024-09-21T10:19:31.673 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012636599999999999 2024-09-21T10:19:31.673 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:31.673 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:31.673 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:31.673 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.673 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.673 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:108221 8.b e281/275 8:de989177:::benchmark_data_smithi098_226587_object4551:head v 281'6767, mlcod=270'624)", 2024-09-21T10:19:31.673 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:31.653375+0000", 2024-09-21T10:19:31.673 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0022996050000000001, 2024-09-21T10:19:31.673 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025464799999999998, 2024-09-21T10:19:31.673 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:31.674 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:19:31.674 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:31.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.674 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:31.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.653375+0000", 2024-09-21T10:19:31.674 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:31.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.674 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:31.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.653375+0000", 2024-09-21T10:19:31.674 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:31.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.675 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:31.675 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.653374+0000", 2024-09-21T10:19:31.675 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:19:31.675 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.675 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.675 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:31.675 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.653383+0000", 2024-09-21T10:19:31.675 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8079999999999994e-06 2024-09-21T10:19:31.675 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.675 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.675 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:31.675 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.653404+0000", 2024-09-21T10:19:31.675 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0789e-05 2024-09-21T10:19:31.676 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.676 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.676 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:31.676 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.653414+0000", 2024-09-21T10:19:31.676 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0183000000000001e-05 2024-09-21T10:19:31.676 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.676 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.676 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:31.676 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.654746+0000", 2024-09-21T10:19:31.676 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001332359 2024-09-21T10:19:31.676 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.676 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.676 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:31.676 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.654844+0000", 2024-09-21T10:19:31.676 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8162e-05 2024-09-21T10:19:31.677 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:31.677 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:31.677 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:31.677 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.677 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.677 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:108236 8.6 e281/265 8:69cbfcd9:::benchmark_data_smithi098_226587_object4566:head v 281'6693, mlcod=281'6691)", 2024-09-21T10:19:31.677 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:31.654005+0000", 2024-09-21T10:19:31.677 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0016696580000000001, 2024-09-21T10:19:31.677 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019650449999999999, 2024-09-21T10:19:31.677 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:31.677 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:19:31.677 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:31.677 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.677 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:31.678 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.654005+0000", 2024-09-21T10:19:31.678 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:31.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.678 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.678 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:31.678 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.654005+0000", 2024-09-21T10:19:31.678 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:31.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.678 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.678 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:31.678 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.654004+0000", 2024-09-21T10:19:31.678 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:19:31.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.678 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.678 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:31.679 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.654010+0000", 2024-09-21T10:19:31.679 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8050000000000003e-06 2024-09-21T10:19:31.679 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.679 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.679 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:31.679 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.654023+0000", 2024-09-21T10:19:31.679 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2456999999999999e-05 2024-09-21T10:19:31.679 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.679 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.679 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:31.679 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.654027+0000", 2024-09-21T10:19:31.679 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7659999999999998e-06 2024-09-21T10:19:31.680 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.680 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.680 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:31.680 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.654086+0000", 2024-09-21T10:19:31.680 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9095000000000003e-05 2024-09-21T10:19:31.680 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:31.680 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:31.680 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:31.680 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:31.654113+0000", 2024-09-21T10:19:31.680 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6636e-05 2024-09-21T10:19:31.680 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:31.680 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:31.680 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:31.680 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:31.681 INFO:teuthology.orchestra.run.smithi196.stdout: ], 2024-09-21T10:19:31.681 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 5 2024-09-21T10:19:31.681 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:19:31.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:31.779 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:31.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:31.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:31.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:31 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/307437 objects degraded (6.324%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:19:31.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:31 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3785029910' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:31.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:31 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/307437 objects degraded (6.324%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:19:31.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:31 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3785029910' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:31.880 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:31.903 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:31.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:31.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:32.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:32.041 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:32.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:32.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:32.143 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:32.168 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:32.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:32.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:32.270 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:32.304 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:32.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:32.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:32.361 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:19:32.406 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:32.459 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:32.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:32.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:32.510 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T10:19:32.561 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:32.604 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:32.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:32.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:32.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:32 smithi196 ceph-mon[84784]: pgmap v889: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 102 B/s rd, 37 MiB/s wr, 713 op/s; 19443/307437 objects degraded (6.324%); 6955/307437 objects misplaced (2.262%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T10:19:32.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:32 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3191293504' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:19:32.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:32 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:19:32.706 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:32.740 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:32.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:32.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:32.789 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:19:32.841 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:32.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:32 smithi098 ceph-mon[97580]: pgmap v889: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 102 B/s rd, 37 MiB/s wr, 713 op/s; 19443/307437 objects degraded (6.324%); 6955/307437 objects misplaced (2.262%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T10:19:32.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:32 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3191293504' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:19:32.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:32 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:19:32.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:32 smithi098 ceph-mon[100914]: pgmap v889: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 102 B/s rd, 37 MiB/s wr, 713 op/s; 19443/307437 objects degraded (6.324%); 6955/307437 objects misplaced (2.262%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T10:19:32.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:32 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3191293504' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:19:32.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:32 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:19:32.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:32.877 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:32.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:32.978 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:32.983 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:19:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:33.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:33.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:33.116 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:33.159 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:33.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:33.261 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:33.293 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:33.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:33.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:33.394 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:33.433 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:33.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:33.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:33.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:33.568 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:33.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:33.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:33.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:33 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:19:33.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:33 smithi196 ceph-mon[84784]: osdmap e282: 8 total, 8 up, 6 in 2024-09-21T10:19:33.670 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:33.706 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:33.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:33.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:33.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:33.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:33 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:19:33.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:33 smithi098 ceph-mon[100914]: osdmap e282: 8 total, 8 up, 6 in 2024-09-21T10:19:33.848 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:33.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:33 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:19:33.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:33 smithi098 ceph-mon[97580]: osdmap e282: 8 total, 8 up, 6 in 2024-09-21T10:19:33.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:33.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:33.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T10:19:33.950 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:34.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:34.047 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:34.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:34.084 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:34.148 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:34.175 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:34.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:34.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:34.276 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:34.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:34.309 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:34.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:34.410 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:34.437 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:34.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:34.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:34.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:34.569 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:34.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:34.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:34.670 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:34.704 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:34.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:34.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:34.806 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:34.830 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:34.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:34.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:34.834 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:34 smithi196 ceph-mon[84784]: pgmap v891: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 127 B/s rd, 15 MiB/s wr, 747 op/s; 19443/298890 objects degraded (6.505%); 6955/298890 objects misplaced (2.327%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T10:19:34.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:34 smithi098 ceph-mon[97580]: pgmap v891: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 127 B/s rd, 15 MiB/s wr, 747 op/s; 19443/298890 objects degraded (6.505%); 6955/298890 objects misplaced (2.327%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T10:19:34.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:34 smithi098 ceph-mon[100914]: pgmap v891: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 127 B/s rd, 15 MiB/s wr, 747 op/s; 19443/298890 objects degraded (6.505%); 6955/298890 objects misplaced (2.327%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T10:19:34.932 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:34.980 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:34.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:34.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:35.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:35.111 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:35.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:35.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:35.213 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:35.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:19:35.255 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:35.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:35.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:35.357 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:35.433 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:35.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:35.456 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:35.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:35.557 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:35.597 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:35.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:35.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:35.699 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:35.731 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:35.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:35.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:35.737 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:19:35.737 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:19:35.737 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-21T10:19:35.737 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-21T10:19:35.737 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:19:35.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:35.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:35.869 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:35.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:35.970 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:36.011 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:36.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:36.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:36.113 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:36.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:36.148 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:36.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:36.249 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:36.278 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:36.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:36.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:36.380 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:36.418 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:19:36.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:36.420 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:36.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:36.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:36.546 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:19:36.555 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:36.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:36.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:36.568 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:36 smithi098 ceph-mon[97580]: pgmap v892: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 127 B/s rd, 7.2 MiB/s wr, 766 op/s; 19443/295128 objects degraded (6.588%); 6819/295128 objects misplaced (2.311%); 3.0 MiB/s, 47 objects/s recovering 2024-09-21T10:19:36.568 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:36 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/295128 objects degraded (6.588%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:19:36.568 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:36 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1813884391' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:19:36.568 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:36 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:19:36.568 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:36 smithi098 ceph-mon[100914]: pgmap v892: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 127 B/s rd, 7.2 MiB/s wr, 766 op/s; 19443/295128 objects degraded (6.588%); 6819/295128 objects misplaced (2.311%); 3.0 MiB/s, 47 objects/s recovering 2024-09-21T10:19:36.568 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:36 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/295128 objects degraded (6.588%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:19:36.569 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:36 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1813884391' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:19:36.569 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:36 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:19:36.657 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:36.696 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:36.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:36.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:36.794 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:19:36.798 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:36.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:36.838 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:36.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:36.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:36 smithi196 ceph-mon[84784]: pgmap v892: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 127 B/s rd, 7.2 MiB/s wr, 766 op/s; 19443/295128 objects degraded (6.588%); 6819/295128 objects misplaced (2.311%); 3.0 MiB/s, 47 objects/s recovering 2024-09-21T10:19:36.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:36 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/295128 objects degraded (6.588%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:19:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:36 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1813884391' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:19:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:36 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:19:36.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:36.989 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:36.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:36.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:37.091 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:37.124 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:37.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:37.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:37.226 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:37.270 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:37.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:37.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:37.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:37.405 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:37.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:37.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:37.506 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:37.544 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:37.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:37.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:37.645 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:37.699 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:37.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:37.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:37.801 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:37.804 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:37 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:19:37.804 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:37 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:19:37.804 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:37 smithi098 ceph-mon[97580]: osdmap e283: 8 total, 8 up, 6 in 2024-09-21T10:19:37.804 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:19:37.804 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:37.805 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:37 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:19:37.805 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:37 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:19:37.805 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:37 smithi098 ceph-mon[100914]: osdmap e283: 8 total, 8 up, 6 in 2024-09-21T10:19:37.805 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:19:37.805 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:37.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:37.831 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:37.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:37 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:19:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:37 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:19:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:37 smithi196 ceph-mon[84784]: osdmap e283: 8 total, 8 up, 6 in 2024-09-21T10:19:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:19:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:37.932 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:37.964 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:37.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:37.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:38.066 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:38.095 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:38.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:38.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:38.196 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:38.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:38.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:38.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:38.399 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:38.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:38.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:38.501 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:38.531 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:38.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:38.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:38.632 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:38.663 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:38.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:38.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:38.765 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:38.798 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:38.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:38.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:38.853 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:38.885 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:19:38.885 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-21T10:19:38.885 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-21T10:19:38.885 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:19:38.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.885 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:55406 7.11 7:8973fea9:::benchmark_data_smithi098_197834_object55405:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-21T10:19:38.885 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:13:43.756897+0000", 2024-09-21T10:19:38.886 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 355.122180743, 2024-09-21T10:19:38.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2483234169999999, 2024-09-21T10:19:38.886 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:38.886 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:38.886 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:38.886 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:19:38.886 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:19:38.886 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 55406 2024-09-21T10:19:38.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.886 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:38.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.886 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:38.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.756897+0000", 2024-09-21T10:19:38.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:38.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.756897+0000", 2024-09-21T10:19:38.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:38.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.756895+0000", 2024-09-21T10:19:38.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:19:38.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:38.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.757047+0000", 2024-09-21T10:19:38.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000151359 2024-09-21T10:19:38.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:38.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.757060+0000", 2024-09-21T10:19:38.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3158e-05 2024-09-21T10:19:38.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:38.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.757064+0000", 2024-09-21T10:19:38.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7520000000000004e-06 2024-09-21T10:19:38.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.757111+0000", 2024-09-21T10:19:38.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6539000000000003e-05 2024-09-21T10:19:38.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.757121+0000", 2024-09-21T10:19:38.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0447e-05 2024-09-21T10:19:38.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677439+0000", 2024-09-21T10:19:38.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.92031749100000004 2024-09-21T10:19:38.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.890 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677449+0000", 2024-09-21T10:19:38.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0059e-05 2024-09-21T10:19:38.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.890 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.890 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:45.996507+0000", 2024-09-21T10:19:38.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3190579410000001 2024-09-21T10:19:38.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.890 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:38.890 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:45.996650+0000", 2024-09-21T10:19:38.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014312799999999999 2024-09-21T10:19:38.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:19:38.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:45.996835+0000", 2024-09-21T10:19:38.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00018520099999999999 2024-09-21T10:19:38.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:19:38.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.000187+0000", 2024-09-21T10:19:38.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0033520080000000001 2024-09-21T10:19:38.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.000974+0000", 2024-09-21T10:19:38.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00078717999999999995 2024-09-21T10:19:38.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.004097+0000", 2024-09-21T10:19:38.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0031225979999999999 2024-09-21T10:19:38.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:38.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.004130+0000", 2024-09-21T10:19:38.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3303999999999998e-05 2024-09-21T10:19:38.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:38.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.005220+0000", 2024-09-21T10:19:38.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010896840000000001 2024-09-21T10:19:38.893 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.893 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:38.893 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.893 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:55408 7.11 7:898dbbe2:::benchmark_data_smithi098_197834_object55407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-21T10:19:38.893 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:13:43.757912+0000", 2024-09-21T10:19:38.893 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 355.12116540400001, 2024-09-21T10:19:38.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2481479279999999, 2024-09-21T10:19:38.893 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:38.893 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:38.894 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:38.894 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:19:38.894 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:19:38.894 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 55408 2024-09-21T10:19:38.894 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.894 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:38.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:38.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.757912+0000", 2024-09-21T10:19:38.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.894 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:38.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.757912+0000", 2024-09-21T10:19:38.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.895 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:38.895 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.757911+0000", 2024-09-21T10:19:38.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:19:38.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.895 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:38.895 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.758019+0000", 2024-09-21T10:19:38.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107135 2024-09-21T10:19:38.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.895 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:38.896 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.758025+0000", 2024-09-21T10:19:38.896 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9639999999999997e-06 2024-09-21T10:19:38.896 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.896 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.896 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:38.896 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.758029+0000", 2024-09-21T10:19:38.896 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5279999999999999e-06 2024-09-21T10:19:38.896 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.896 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.896 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.896 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.758096+0000", 2024-09-21T10:19:38.896 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6988000000000003e-05 2024-09-21T10:19:38.896 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.896 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.896 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.897 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.758102+0000", 2024-09-21T10:19:38.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3029999999999996e-06 2024-09-21T10:19:38.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.897 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.897 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677615+0000", 2024-09-21T10:19:38.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.91951281200000001 2024-09-21T10:19:38.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.897 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.897 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677623+0000", 2024-09-21T10:19:38.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3410000000000003e-06 2024-09-21T10:19:38.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.897 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:45.998401+0000", 2024-09-21T10:19:38.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3207773190000001 2024-09-21T10:19:38.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:38.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:45.998492+0000", 2024-09-21T10:19:38.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1407000000000001e-05 2024-09-21T10:19:38.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:19:38.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:45.998589+0000", 2024-09-21T10:19:38.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6668000000000003e-05 2024-09-21T10:19:38.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:19:38.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.000927+0000", 2024-09-21T10:19:38.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023384780000000002 2024-09-21T10:19:38.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.001022+0000", 2024-09-21T10:19:38.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5004e-05 2024-09-21T10:19:38.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.900 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.005274+0000", 2024-09-21T10:19:38.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042519480000000002 2024-09-21T10:19:38.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.900 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.900 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:38.900 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.005313+0000", 2024-09-21T10:19:38.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8371999999999998e-05 2024-09-21T10:19:38.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.900 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.900 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:38.900 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.006060+0000", 2024-09-21T10:19:38.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00074726199999999995 2024-09-21T10:19:38.900 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.901 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:38.901 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.901 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:55450 7.11 7:8a8f55f9:::benchmark_data_smithi098_197834_object55449:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-21T10:19:38.901 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:13:43.786722+0000", 2024-09-21T10:19:38.901 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 355.09235529099999, 2024-09-21T10:19:38.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2212804739999998, 2024-09-21T10:19:38.901 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:38.901 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:38.901 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:38.901 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:19:38.902 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:19:38.902 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 55450 2024-09-21T10:19:38.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.902 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:38.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:38.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.786722+0000", 2024-09-21T10:19:38.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:38.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.786722+0000", 2024-09-21T10:19:38.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.903 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.903 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:38.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.786721+0000", 2024-09-21T10:19:38.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:19:38.903 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.903 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:38.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.786835+0000", 2024-09-21T10:19:38.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000113991 2024-09-21T10:19:38.903 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.903 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:38.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.786845+0000", 2024-09-21T10:19:38.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9280000000000005e-06 2024-09-21T10:19:38.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.904 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.904 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:38.904 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.786851+0000", 2024-09-21T10:19:38.904 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0900000000000001e-06 2024-09-21T10:19:38.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.904 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.904 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.904 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.786908+0000", 2024-09-21T10:19:38.904 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7497999999999997e-05 2024-09-21T10:19:38.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.904 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.904 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.905 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.786916+0000", 2024-09-21T10:19:38.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0290000000000005e-06 2024-09-21T10:19:38.905 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.905 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.905 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.905 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677650+0000", 2024-09-21T10:19:38.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.89073383100000003 2024-09-21T10:19:38.905 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.905 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.905 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.905 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677655+0000", 2024-09-21T10:19:38.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5210000000000004e-06 2024-09-21T10:19:38.905 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.905 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.905 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.906 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:45.999264+0000", 2024-09-21T10:19:38.906 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3216095270000001 2024-09-21T10:19:38.906 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.906 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.906 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:38.906 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:45.999386+0000", 2024-09-21T10:19:38.906 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000121192 2024-09-21T10:19:38.906 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.906 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.906 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:19:38.906 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:45.999537+0000", 2024-09-21T10:19:38.906 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015125199999999999 2024-09-21T10:19:38.906 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.906 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.907 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:19:38.907 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.001032+0000", 2024-09-21T10:19:38.907 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014955820000000001 2024-09-21T10:19:38.907 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.907 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.907 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.907 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.004067+0000", 2024-09-21T10:19:38.907 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030350500000000001 2024-09-21T10:19:38.907 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.907 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.907 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.907 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.007158+0000", 2024-09-21T10:19:38.907 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030909790000000002 2024-09-21T10:19:38.907 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.908 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.908 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:38.908 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.007199+0000", 2024-09-21T10:19:38.908 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0232999999999999e-05 2024-09-21T10:19:38.908 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.908 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.908 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:38.908 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.008003+0000", 2024-09-21T10:19:38.908 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00080395800000000002 2024-09-21T10:19:38.908 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.908 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:38.908 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.908 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.908 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.908 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:55472 7.11 7:8b981b06:::benchmark_data_smithi098_197834_object55471:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-21T10:19:38.909 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:13:43.797560+0000", 2024-09-21T10:19:38.909 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 355.08151717999999, 2024-09-21T10:19:38.909 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2112651310000002, 2024-09-21T10:19:38.909 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:38.909 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:38.909 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:38.909 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:19:38.909 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:19:38.909 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 55472 2024-09-21T10:19:38.909 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.909 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:38.909 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.909 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:38.909 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.797560+0000", 2024-09-21T10:19:38.910 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.910 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.910 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.910 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:38.910 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.797560+0000", 2024-09-21T10:19:38.910 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.910 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.910 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.910 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:38.910 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.797559+0000", 2024-09-21T10:19:38.910 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:19:38.910 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.910 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.911 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:38.911 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.797597+0000", 2024-09-21T10:19:38.911 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8365000000000003e-05 2024-09-21T10:19:38.911 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.911 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.911 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:38.911 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.797612+0000", 2024-09-21T10:19:38.911 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4567e-05 2024-09-21T10:19:38.911 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.911 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.911 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:38.911 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.797622+0000", 2024-09-21T10:19:38.911 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0456e-05 2024-09-21T10:19:38.911 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.912 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.912 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.912 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.797672+0000", 2024-09-21T10:19:38.912 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9434999999999997e-05 2024-09-21T10:19:38.912 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.912 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.912 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.912 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.797682+0000", 2024-09-21T10:19:38.912 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0318e-05 2024-09-21T10:19:38.912 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.912 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:38.913 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.914 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.914 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677680+0000", 2024-09-21T10:19:38.914 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.87999812499999996 2024-09-21T10:19:38.914 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.914 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.914 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.914 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677687+0000", 2024-09-21T10:19:38.914 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9820000000000002e-06 2024-09-21T10:19:38.914 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.914 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.914 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.914 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.000218+0000", 2024-09-21T10:19:38.914 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.322531254 2024-09-21T10:19:38.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:38.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.000307+0000", 2024-09-21T10:19:38.915 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8635999999999996e-05 2024-09-21T10:19:38.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:19:38.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.000406+0000", 2024-09-21T10:19:38.915 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8945999999999998e-05 2024-09-21T10:19:38.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:19:38.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.002466+0000", 2024-09-21T10:19:38.916 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020606090000000001 2024-09-21T10:19:38.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.916 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.916 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.916 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.007127+0000", 2024-09-21T10:19:38.916 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046607339999999997 2024-09-21T10:19:38.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.916 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.916 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.916 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.008062+0000", 2024-09-21T10:19:38.916 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00093477600000000001 2024-09-21T10:19:38.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.916 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.916 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:38.916 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.008115+0000", 2024-09-21T10:19:38.917 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2685000000000001e-05 2024-09-21T10:19:38.917 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.917 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.917 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:38.917 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.008825+0000", 2024-09-21T10:19:38.917 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00071077399999999995 2024-09-21T10:19:38.917 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.917 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:38.917 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.917 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.917 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.917 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:55518 7.11 7:8fe9d082:::benchmark_data_smithi098_197834_object55517:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-21T10:19:38.917 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:13:43.832082+0000", 2024-09-21T10:19:38.918 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 355.04699584700001, 2024-09-21T10:19:38.918 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1816013430000001, 2024-09-21T10:19:38.918 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:38.918 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:38.918 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:38.918 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:19:38.918 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:19:38.918 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 55518 2024-09-21T10:19:38.918 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.918 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:38.918 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.918 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:38.918 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.832082+0000", 2024-09-21T10:19:38.918 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.919 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.919 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.919 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:38.919 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.832082+0000", 2024-09-21T10:19:38.919 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.919 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.919 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.919 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:38.919 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.832080+0000", 2024-09-21T10:19:38.919 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:19:38.919 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.919 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.919 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:38.919 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.832227+0000", 2024-09-21T10:19:38.919 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014749699999999999 2024-09-21T10:19:38.920 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.920 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.920 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:38.920 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.832240+0000", 2024-09-21T10:19:38.920 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2454e-05 2024-09-21T10:19:38.920 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.920 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.920 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:38.920 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.832246+0000", 2024-09-21T10:19:38.920 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0240000000000003e-06 2024-09-21T10:19:38.920 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.920 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.920 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.920 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.832296+0000", 2024-09-21T10:19:38.921 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0318000000000002e-05 2024-09-21T10:19:38.921 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.921 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.832306+0000", 2024-09-21T10:19:38.921 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0164e-05 2024-09-21T10:19:38.921 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.921 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677737+0000", 2024-09-21T10:19:38.921 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.84543046899999996 2024-09-21T10:19:38.921 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.921 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677743+0000", 2024-09-21T10:19:38.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5470000000000002e-06 2024-09-21T10:19:38.922 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.922 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.922 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.001047+0000", 2024-09-21T10:19:38.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3233035319999999 2024-09-21T10:19:38.922 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.922 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:38.922 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.001122+0000", 2024-09-21T10:19:38.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4595e-05 2024-09-21T10:19:38.922 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.922 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:19:38.923 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.001216+0000", 2024-09-21T10:19:38.923 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4889000000000006e-05 2024-09-21T10:19:38.923 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.923 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:19:38.923 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.004025+0000", 2024-09-21T10:19:38.923 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002808166 2024-09-21T10:19:38.923 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.923 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.923 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.008948+0000", 2024-09-21T10:19:38.923 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049232160000000002 2024-09-21T10:19:38.923 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.923 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.923 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.012478+0000", 2024-09-21T10:19:38.924 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0035298109999999999 2024-09-21T10:19:38.924 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.924 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.924 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:38.924 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.012519+0000", 2024-09-21T10:19:38.924 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1593e-05 2024-09-21T10:19:38.924 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.924 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.924 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:38.924 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.013683+0000", 2024-09-21T10:19:38.924 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001163662 2024-09-21T10:19:38.924 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.924 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:38.925 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.925 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.925 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:55535 7.11 7:8c5840ea:::benchmark_data_smithi098_197834_object55534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-21T10:19:38.925 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:13:43.842295+0000", 2024-09-21T10:19:38.925 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 355.03678265399998, 2024-09-21T10:19:38.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1724636749999999, 2024-09-21T10:19:38.925 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:38.925 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:38.925 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:38.925 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:19:38.925 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:19:38.925 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 55535 2024-09-21T10:19:38.925 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.925 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:38.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.926 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:38.926 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.842295+0000", 2024-09-21T10:19:38.926 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.926 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:38.926 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.842295+0000", 2024-09-21T10:19:38.926 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.926 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:38.926 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.842294+0000", 2024-09-21T10:19:38.926 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:19:38.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.927 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:38.927 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.842420+0000", 2024-09-21T10:19:38.927 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012685100000000001 2024-09-21T10:19:38.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.927 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.927 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:38.927 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.842432+0000", 2024-09-21T10:19:38.927 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1100999999999999e-05 2024-09-21T10:19:38.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.927 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.927 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:38.927 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.842437+0000", 2024-09-21T10:19:38.927 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2970000000000003e-06 2024-09-21T10:19:38.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.842477+0000", 2024-09-21T10:19:38.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0611999999999997e-05 2024-09-21T10:19:38.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.842487+0000", 2024-09-21T10:19:38.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1090000000000004e-06 2024-09-21T10:19:38.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677764+0000", 2024-09-21T10:19:38.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83527696799999995 2024-09-21T10:19:38.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677767+0000", 2024-09-21T10:19:38.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.461e-06 2024-09-21T10:19:38.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.001729+0000", 2024-09-21T10:19:38.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.323962077 2024-09-21T10:19:38.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:38.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.001848+0000", 2024-09-21T10:19:38.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011905600000000001 2024-09-21T10:19:38.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.930 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.930 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:19:38.930 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.001954+0000", 2024-09-21T10:19:38.930 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105565 2024-09-21T10:19:38.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.930 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.930 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:19:38.930 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.005235+0000", 2024-09-21T10:19:38.930 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003281651 2024-09-21T10:19:38.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.930 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.930 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.931 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.008996+0000", 2024-09-21T10:19:38.931 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0037602930000000001 2024-09-21T10:19:38.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.931 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.931 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.931 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.013729+0000", 2024-09-21T10:19:38.931 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047330059999999997 2024-09-21T10:19:38.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.931 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.931 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:38.931 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.013782+0000", 2024-09-21T10:19:38.931 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3090000000000002e-05 2024-09-21T10:19:38.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.932 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.932 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:38.932 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.014758+0000", 2024-09-21T10:19:38.932 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00097667900000000005 2024-09-21T10:19:38.932 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.932 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:38.932 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.932 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.932 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.932 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:55537 7.11 7:8b80ba5f:::benchmark_data_smithi098_197834_object55536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-21T10:19:38.932 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:13:43.844111+0000", 2024-09-21T10:19:38.932 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 355.03496629699998, 2024-09-21T10:19:38.932 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.171681317, 2024-09-21T10:19:38.933 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:38.933 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:38.933 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:38.933 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:19:38.933 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:19:38.933 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 55537 2024-09-21T10:19:38.933 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.933 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:38.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.933 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:38.933 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844111+0000", 2024-09-21T10:19:38.933 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.933 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.933 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:38.933 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844111+0000", 2024-09-21T10:19:38.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:38.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844110+0000", 2024-09-21T10:19:38.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:19:38.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:38.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844326+0000", 2024-09-21T10:19:38.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00021639700000000001 2024-09-21T10:19:38.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:38.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844335+0000", 2024-09-21T10:19:38.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4470000000000006e-06 2024-09-21T10:19:38.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:38.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844340+0000", 2024-09-21T10:19:38.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7999999999999998e-06 2024-09-21T10:19:38.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844381+0000", 2024-09-21T10:19:38.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0890000000000003e-05 2024-09-21T10:19:38.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844389+0000", 2024-09-21T10:19:38.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4400000000000005e-06 2024-09-21T10:19:38.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677794+0000", 2024-09-21T10:19:38.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83340498200000002 2024-09-21T10:19:38.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.937 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.937 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677801+0000", 2024-09-21T10:19:38.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6959999999999999e-06 2024-09-21T10:19:38.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.937 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.937 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.937 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.002504+0000", 2024-09-21T10:19:38.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3247034090000001 2024-09-21T10:19:38.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.937 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.937 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:38.937 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.002600+0000", 2024-09-21T10:19:38.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6107999999999995e-05 2024-09-21T10:19:38.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:19:38.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.002697+0000", 2024-09-21T10:19:38.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6932e-05 2024-09-21T10:19:38.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:19:38.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.005242+0000", 2024-09-21T10:19:38.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025451430000000001 2024-09-21T10:19:38.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.014826+0000", 2024-09-21T10:19:38.939 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0095836979999999999 2024-09-21T10:19:38.939 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.939 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.939 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.939 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.014908+0000", 2024-09-21T10:19:38.939 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2025000000000002e-05 2024-09-21T10:19:38.939 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.939 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.939 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:38.939 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.014961+0000", 2024-09-21T10:19:38.939 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2889000000000001e-05 2024-09-21T10:19:38.939 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.939 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.939 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:38.940 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.015792+0000", 2024-09-21T10:19:38.940 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00083145400000000003 2024-09-21T10:19:38.940 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.940 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:38.940 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.940 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.940 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.940 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:55539 7.11 7:8eebc2e8:::benchmark_data_smithi098_197834_object55538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-21T10:19:38.940 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:13:43.844573+0000", 2024-09-21T10:19:38.940 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 355.03450407600002, 2024-09-21T10:19:38.940 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1757300549999998, 2024-09-21T10:19:38.940 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:38.940 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:38.940 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:38.941 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:19:38.941 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:19:38.941 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 55539 2024-09-21T10:19:38.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.941 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:38.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:38.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844573+0000", 2024-09-21T10:19:38.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:38.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844573+0000", 2024-09-21T10:19:38.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:38.942 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844573+0000", 2024-09-21T10:19:38.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:19:38.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:38.942 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844653+0000", 2024-09-21T10:19:38.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0142999999999995e-05 2024-09-21T10:19:38.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:38.942 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844664+0000", 2024-09-21T10:19:38.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1620999999999999e-05 2024-09-21T10:19:38.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.943 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.943 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:38.943 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844670+0000", 2024-09-21T10:19:38.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7570000000000001e-06 2024-09-21T10:19:38.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.943 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.943 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.943 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844710+0000", 2024-09-21T10:19:38.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9780999999999997e-05 2024-09-21T10:19:38.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.943 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.944 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844718+0000", 2024-09-21T10:19:38.944 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0420000000000003e-06 2024-09-21T10:19:38.944 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.944 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677821+0000", 2024-09-21T10:19:38.944 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83310285299999998 2024-09-21T10:19:38.944 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.944 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677824+0000", 2024-09-21T10:19:38.944 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2959999999999999e-06 2024-09-21T10:19:38.944 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.945 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.003258+0000", 2024-09-21T10:19:38.945 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.325434038 2024-09-21T10:19:38.945 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.945 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.945 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:38.945 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.003396+0000", 2024-09-21T10:19:38.945 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013825799999999999 2024-09-21T10:19:38.945 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.945 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.945 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:19:38.945 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.003552+0000", 2024-09-21T10:19:38.945 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015607699999999999 2024-09-21T10:19:38.945 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.945 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.945 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:19:38.946 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.007088+0000", 2024-09-21T10:19:38.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0035353870000000001 2024-09-21T10:19:38.946 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.946 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.946 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.946 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.015837+0000", 2024-09-21T10:19:38.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.008748881 2024-09-21T10:19:38.946 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.946 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.946 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.946 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.019617+0000", 2024-09-21T10:19:38.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0037801419999999998 2024-09-21T10:19:38.946 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.946 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.946 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:38.947 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.019673+0000", 2024-09-21T10:19:38.947 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6329000000000002e-05 2024-09-21T10:19:38.947 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.947 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.947 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:38.947 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.020303+0000", 2024-09-21T10:19:38.947 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00063017499999999996 2024-09-21T10:19:38.947 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.947 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:38.947 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.947 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.947 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.947 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:55586 7.11 7:8ac9c414:::benchmark_data_smithi098_197834_object55585:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-21T10:19:38.947 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:13:43.877450+0000", 2024-09-21T10:19:38.947 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 355.00162701099998, 2024-09-21T10:19:38.948 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1437025329999999, 2024-09-21T10:19:38.948 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:38.948 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:38.948 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:38.948 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:19:38.948 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:19:38.948 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 55586 2024-09-21T10:19:38.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.948 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:38.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:38.948 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.877450+0000", 2024-09-21T10:19:38.948 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.949 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:38.949 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.877450+0000", 2024-09-21T10:19:38.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.949 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:38.949 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.877449+0000", 2024-09-21T10:19:38.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:19:38.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.950 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:38.950 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.877505+0000", 2024-09-21T10:19:38.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5727999999999999e-05 2024-09-21T10:19:38.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.950 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:38.950 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.877515+0000", 2024-09-21T10:19:38.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9180000000000006e-06 2024-09-21T10:19:38.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.950 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:38.950 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.877520+0000", 2024-09-21T10:19:38.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3650000000000003e-06 2024-09-21T10:19:38.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.877558+0000", 2024-09-21T10:19:38.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8013999999999999e-05 2024-09-21T10:19:38.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.877567+0000", 2024-09-21T10:19:38.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9039999999999998e-06 2024-09-21T10:19:38.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.952 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677836+0000", 2024-09-21T10:19:38.952 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80026863000000004 2024-09-21T10:19:38.952 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.952 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.952 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.952 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677839+0000", 2024-09-21T10:19:38.952 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.044e-06 2024-09-21T10:19:38.952 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.952 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.952 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.952 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.006105+0000", 2024-09-21T10:19:38.952 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3282661060000001 2024-09-21T10:19:38.952 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.952 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.953 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:38.953 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.006239+0000", 2024-09-21T10:19:38.953 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013374799999999999 2024-09-21T10:19:38.953 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.953 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:19:38.953 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.006343+0000", 2024-09-21T10:19:38.953 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010435800000000001 2024-09-21T10:19:38.953 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.953 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:19:38.953 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.008909+0000", 2024-09-21T10:19:38.953 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002565459 2024-09-21T10:19:38.953 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.015898+0000", 2024-09-21T10:19:38.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0069897850000000001 2024-09-21T10:19:38.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.020402+0000", 2024-09-21T10:19:38.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045034910000000001 2024-09-21T10:19:38.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:38.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.020446+0000", 2024-09-21T10:19:38.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3784e-05 2024-09-21T10:19:38.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.955 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:38.955 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.021153+0000", 2024-09-21T10:19:38.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00070732399999999995 2024-09-21T10:19:38.955 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.955 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:38.955 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.955 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:126295 7.15 7:a83ab893:::benchmark_data_smithi098_197834_object9778:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-21T10:19:38.955 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:14:53.662164+0000", 2024-09-21T10:19:38.955 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 285.21691356100001, 2024-09-21T10:19:38.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.316171792, 2024-09-21T10:19:38.956 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:38.956 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:19:38.956 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:38.956 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:19:38.956 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:19:38.956 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 126295 2024-09-21T10:19:38.956 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.956 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:38.956 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.956 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:38.956 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.662164+0000", 2024-09-21T10:19:38.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.956 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.957 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:38.957 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.662164+0000", 2024-09-21T10:19:38.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.957 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:38.957 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.662163+0000", 2024-09-21T10:19:38.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:19:38.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.957 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:38.957 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.662166+0000", 2024-09-21T10:19:38.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.791e-06 2024-09-21T10:19:38.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:38.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.662172+0000", 2024-09-21T10:19:38.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.105e-06 2024-09-21T10:19:38.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:38.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.662176+0000", 2024-09-21T10:19:38.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0840000000000002e-06 2024-09-21T10:19:38.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.663493+0000", 2024-09-21T10:19:38.959 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0013171050000000001 2024-09-21T10:19:38.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.959 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:38.959 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.663622+0000", 2024-09-21T10:19:38.959 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012832499999999999 2024-09-21T10:19:38.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.959 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,2", 2024-09-21T10:19:38.959 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.663722+0000", 2024-09-21T10:19:38.959 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9994000000000005e-05 2024-09-21T10:19:38.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.959 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:19:38.959 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.665921+0000", 2024-09-21T10:19:38.960 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021989739999999998 2024-09-21T10:19:38.960 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.960 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.960 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.960 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.674333+0000", 2024-09-21T10:19:38.960 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0084118720000000008 2024-09-21T10:19:38.960 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.960 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.960 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.960 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.685186+0000", 2024-09-21T10:19:38.960 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010853115 2024-09-21T10:19:38.960 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.960 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.960 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.961 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.685196+0000", 2024-09-21T10:19:38.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0404000000000001e-05 2024-09-21T10:19:38.961 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.961 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.961 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687220+0000", 2024-09-21T10:19:38.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0020234910000001 2024-09-21T10:19:38.961 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.961 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.961 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687230+0000", 2024-09-21T10:19:38.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0789e-05 2024-09-21T10:19:38.961 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.962 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.972981+0000", 2024-09-21T10:19:38.962 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2857503000000001 2024-09-21T10:19:38.962 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.962 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.962 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for missing object", 2024-09-21T10:19:38.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.973091+0000", 2024-09-21T10:19:38.962 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000110454 2024-09-21T10:19:38.962 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.962 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.962 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.978279+0000", 2024-09-21T10:19:38.962 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051879990000000004 2024-09-21T10:19:38.962 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.963 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:38.963 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.978336+0000", 2024-09-21T10:19:38.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6446000000000001e-05 2024-09-21T10:19:38.963 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.963 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:38.963 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.963 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:126344 7.15 7:af8722a8:::benchmark_data_smithi098_197834_object9827:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-09-21T10:19:38.963 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:14:53.717412+0000", 2024-09-21T10:19:38.963 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 285.161665196, 2024-09-21T10:19:38.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2618246119999998, 2024-09-21T10:19:38.963 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:38.963 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:38.963 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:38.964 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:19:38.964 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:19:38.964 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 126344 2024-09-21T10:19:38.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.964 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:38.964 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.964 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:38.964 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717412+0000", 2024-09-21T10:19:38.964 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.964 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.964 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:38.964 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717412+0000", 2024-09-21T10:19:38.964 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.965 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.965 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:38.965 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717411+0000", 2024-09-21T10:19:38.965 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:19:38.965 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.965 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.965 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:38.965 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717417+0000", 2024-09-21T10:19:38.965 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9109999999999998e-06 2024-09-21T10:19:38.965 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.965 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.965 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:38.965 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717428+0000", 2024-09-21T10:19:38.965 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.102e-05 2024-09-21T10:19:38.965 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.965 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.966 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:38.966 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717432+0000", 2024-09-21T10:19:38.966 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3869999999999997e-06 2024-09-21T10:19:38.966 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.966 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.966 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.966 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717479+0000", 2024-09-21T10:19:38.966 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6888000000000002e-05 2024-09-21T10:19:38.966 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.966 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.966 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.966 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717488+0000", 2024-09-21T10:19:38.966 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1619999999999997e-06 2024-09-21T10:19:38.966 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.967 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.967 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687267+0000", 2024-09-21T10:19:38.967 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.969778794 2024-09-21T10:19:38.967 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.968 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.968 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687272+0000", 2024-09-21T10:19:38.968 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2220000000000001e-06 2024-09-21T10:19:38.968 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.969 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.969 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.969 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.974032+0000", 2024-09-21T10:19:38.969 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.28675984 2024-09-21T10:19:38.969 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.969 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.969 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:38.969 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.974120+0000", 2024-09-21T10:19:38.969 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8318000000000004e-05 2024-09-21T10:19:38.969 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.969 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.969 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:19:38.969 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.974212+0000", 2024-09-21T10:19:38.969 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2009000000000006e-05 2024-09-21T10:19:38.969 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.978367+0000", 2024-09-21T10:19:38.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041543960000000003 2024-09-21T10:19:38.970 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:19:38.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.978435+0000", 2024-09-21T10:19:38.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8034000000000006e-05 2024-09-21T10:19:38.970 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.978471+0000", 2024-09-21T10:19:38.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6238000000000001e-05 2024-09-21T10:19:38.970 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.971 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:38.971 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.978501+0000", 2024-09-21T10:19:38.971 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9850999999999999e-05 2024-09-21T10:19:38.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.971 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:38.971 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.979237+0000", 2024-09-21T10:19:38.971 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00073583399999999999 2024-09-21T10:19:38.971 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.971 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:38.971 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.972 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:126345 7.15 7:a9aca293:::benchmark_data_smithi098_197834_object9828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-09-21T10:19:38.972 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:14:53.717858+0000", 2024-09-21T10:19:38.972 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 285.16121983900001, 2024-09-21T10:19:38.972 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2720590949999999, 2024-09-21T10:19:38.972 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:38.972 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:38.972 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:38.972 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:19:38.972 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:19:38.972 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 126345 2024-09-21T10:19:38.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.972 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:38.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.973 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:38.973 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717858+0000", 2024-09-21T10:19:38.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.973 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.973 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:38.973 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717858+0000", 2024-09-21T10:19:38.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.973 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.973 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:38.973 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717857+0000", 2024-09-21T10:19:38.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:19:38.974 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.974 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:38.974 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717861+0000", 2024-09-21T10:19:38.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8680000000000001e-06 2024-09-21T10:19:38.974 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.974 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:38.974 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717879+0000", 2024-09-21T10:19:38.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7989e-05 2024-09-21T10:19:38.974 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.975 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:38.975 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717884+0000", 2024-09-21T10:19:38.975 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6010000000000002e-06 2024-09-21T10:19:38.975 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.975 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.975 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.975 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717920+0000", 2024-09-21T10:19:38.975 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5914000000000002e-05 2024-09-21T10:19:38.975 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.975 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.975 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.975 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717928+0000", 2024-09-21T10:19:38.975 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5700000000000004e-06 2024-09-21T10:19:38.975 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.976 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687305+0000", 2024-09-21T10:19:38.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.969376825 2024-09-21T10:19:38.976 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.976 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687310+0000", 2024-09-21T10:19:38.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9019999999999998e-06 2024-09-21T10:19:38.976 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.976 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.975661+0000", 2024-09-21T10:19:38.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2883512429999999 2024-09-21T10:19:38.977 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.977 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:38.977 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.975770+0000", 2024-09-21T10:19:38.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109239 2024-09-21T10:19:38.977 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.977 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:19:38.977 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.975847+0000", 2024-09-21T10:19:38.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7354999999999994e-05 2024-09-21T10:19:38.977 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.978 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:19:38.978 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.978443+0000", 2024-09-21T10:19:38.978 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025957739999999999 2024-09-21T10:19:38.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.978 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.978 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.978 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.979284+0000", 2024-09-21T10:19:38.978 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000840459 2024-09-21T10:19:38.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.978 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.978 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.988637+0000", 2024-09-21T10:19:38.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0093530850000000006 2024-09-21T10:19:38.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:38.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.988669+0000", 2024-09-21T10:19:38.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2756999999999998e-05 2024-09-21T10:19:38.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:38.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.989917+0000", 2024-09-21T10:19:38.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012471909999999999 2024-09-21T10:19:38.979 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.979 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:38.979 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.980 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:126354 7.15 7:a9f8f369:::benchmark_data_smithi098_197834_object9837:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-09-21T10:19:38.980 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:14:53.724439+0000", 2024-09-21T10:19:38.980 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 285.154638476, 2024-09-21T10:19:38.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2705321569999999, 2024-09-21T10:19:38.980 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:38.980 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:38.980 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:38.980 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:19:38.980 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:19:38.980 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 126354 2024-09-21T10:19:38.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.980 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:38.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:38.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.724439+0000", 2024-09-21T10:19:38.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:38.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.724439+0000", 2024-09-21T10:19:38.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:38.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.724438+0000", 2024-09-21T10:19:38.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:19:38.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:38.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.724443+0000", 2024-09-21T10:19:38.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5470000000000004e-06 2024-09-21T10:19:38.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:38.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.724453+0000", 2024-09-21T10:19:38.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8430000000000005e-06 2024-09-21T10:19:38.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:38.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.724459+0000", 2024-09-21T10:19:38.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5720000000000002e-06 2024-09-21T10:19:38.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.983 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.983 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.983 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.724486+0000", 2024-09-21T10:19:38.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7474000000000002e-05 2024-09-21T10:19:38.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.983 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.983 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.983 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.724491+0000", 2024-09-21T10:19:38.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4160000000000003e-06 2024-09-21T10:19:38.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.983 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.983 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.983 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687339+0000", 2024-09-21T10:19:38.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.96284762499999998 2024-09-21T10:19:38.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.984 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.984 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.984 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687344+0000", 2024-09-21T10:19:38.984 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3159999999999996e-06 2024-09-21T10:19:38.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.984 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.984 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.984 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.979308+0000", 2024-09-21T10:19:38.984 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.291963355 2024-09-21T10:19:38.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.984 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.984 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:38.984 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.979390+0000", 2024-09-21T10:19:38.984 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1928000000000006e-05 2024-09-21T10:19:38.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:19:38.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.979466+0000", 2024-09-21T10:19:38.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6395000000000003e-05 2024-09-21T10:19:38.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:19:38.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.983649+0000", 2024-09-21T10:19:38.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041824210000000004 2024-09-21T10:19:38.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.988605+0000", 2024-09-21T10:19:38.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004956556 2024-09-21T10:19:38.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.994039+0000", 2024-09-21T10:19:38.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0054335190000000004 2024-09-21T10:19:38.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:38.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.994071+0000", 2024-09-21T10:19:38.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2288e-05 2024-09-21T10:19:38.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:38.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.994971+0000", 2024-09-21T10:19:38.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00090020300000000003 2024-09-21T10:19:38.987 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.987 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:38.987 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.987 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:126362 7.15 7:ab907e32:::benchmark_data_smithi098_197834_object9845:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-09-21T10:19:38.987 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:14:53.731370+0000", 2024-09-21T10:19:38.987 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 285.14770722600002, 2024-09-21T10:19:38.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2648341489999999, 2024-09-21T10:19:38.987 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:38.987 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:38.987 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:38.988 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:19:38.988 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:19:38.988 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 126362 2024-09-21T10:19:38.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.988 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:38.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:38.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.731370+0000", 2024-09-21T10:19:38.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:38.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.731370+0000", 2024-09-21T10:19:38.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:38.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.731368+0000", 2024-09-21T10:19:38.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:19:38.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:38.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.731376+0000", 2024-09-21T10:19:38.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4479999999999997e-06 2024-09-21T10:19:38.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:38.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.731388+0000", 2024-09-21T10:19:38.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1684e-05 2024-09-21T10:19:38.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.990 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:38.990 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.731392+0000", 2024-09-21T10:19:38.990 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6949999999999999e-06 2024-09-21T10:19:38.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.990 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.990 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.731429+0000", 2024-09-21T10:19:38.990 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6405000000000001e-05 2024-09-21T10:19:38.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.990 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.731439+0000", 2024-09-21T10:19:38.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8230000000000006e-06 2024-09-21T10:19:38.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.991 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.991 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687364+0000", 2024-09-21T10:19:38.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.95592560199999999 2024-09-21T10:19:38.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.991 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.991 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687368+0000", 2024-09-21T10:19:38.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5370000000000002e-06 2024-09-21T10:19:38.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.992 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.980449+0000", 2024-09-21T10:19:38.992 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2930808410000001 2024-09-21T10:19:38.992 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.992 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:38.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.980563+0000", 2024-09-21T10:19:38.992 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011435800000000001 2024-09-21T10:19:38.992 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.992 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:19:38.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.980648+0000", 2024-09-21T10:19:38.992 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4917e-05 2024-09-21T10:19:38.992 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.993 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:19:38.993 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.983656+0000", 2024-09-21T10:19:38.993 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030083100000000002 2024-09-21T10:19:38.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.993 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.993 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.993 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.989968+0000", 2024-09-21T10:19:38.993 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006311924 2024-09-21T10:19:38.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.993 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.993 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:38.993 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.995064+0000", 2024-09-21T10:19:38.993 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0050958599999999998 2024-09-21T10:19:38.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.993 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.994 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:38.994 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.995114+0000", 2024-09-21T10:19:38.994 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9855000000000003e-05 2024-09-21T10:19:38.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.994 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.994 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:38.994 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.996204+0000", 2024-09-21T10:19:38.994 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010905579999999999 2024-09-21T10:19:38.994 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.994 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:38.994 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:38.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.994 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.994 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:126420 7.15 7:aea4d8a7:::benchmark_data_smithi098_197834_object9903:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-09-21T10:19:38.995 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:14:53.782086+0000", 2024-09-21T10:19:38.995 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 285.09699124600002, 2024-09-21T10:19:38.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.216431933, 2024-09-21T10:19:38.995 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:38.995 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:38.995 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:38.995 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:19:38.995 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:19:38.995 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 126420 2024-09-21T10:19:38.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.995 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:38.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.995 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:38.995 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.782086+0000", 2024-09-21T10:19:38.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.996 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.996 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:38.996 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.782086+0000", 2024-09-21T10:19:38.996 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:38.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.996 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.996 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:38.996 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.782084+0000", 2024-09-21T10:19:38.996 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:19:38.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.996 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.996 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:38.996 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.782091+0000", 2024-09-21T10:19:38.996 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7889999999999997e-06 2024-09-21T10:19:38.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.997 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.997 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:38.997 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.782105+0000", 2024-09-21T10:19:38.997 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3903000000000001e-05 2024-09-21T10:19:38.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.997 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.997 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:38.997 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.782111+0000", 2024-09-21T10:19:38.997 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0140000000000004e-06 2024-09-21T10:19:38.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.997 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.997 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.997 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.782159+0000", 2024-09-21T10:19:38.997 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8111000000000001e-05 2024-09-21T10:19:38.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.998 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.998 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.782169+0000", 2024-09-21T10:19:38.998 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.448e-06 2024-09-21T10:19:38.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.998 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.998 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687385+0000", 2024-09-21T10:19:38.998 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.90521664599999996 2024-09-21T10:19:38.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.998 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:38.998 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687389+0000", 2024-09-21T10:19:38.999 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4599999999999999e-06 2024-09-21T10:19:38.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:38.999 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.981530+0000", 2024-09-21T10:19:38.999 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.294141193 2024-09-21T10:19:38.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:38.999 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.981627+0000", 2024-09-21T10:19:38.999 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6613999999999994e-05 2024-09-21T10:19:38.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:38.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:38.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:19:39.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.981706+0000", 2024-09-21T10:19:39.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9658000000000002e-05 2024-09-21T10:19:39.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.000 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.000 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:19:39.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.985669+0000", 2024-09-21T10:19:39.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039624200000000004 2024-09-21T10:19:39.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.000 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.000 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:39.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.990000+0000", 2024-09-21T10:19:39.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043309350000000002 2024-09-21T10:19:39.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.000 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.001 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:39.001 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.997553+0000", 2024-09-21T10:19:39.001 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0075531210000000003 2024-09-21T10:19:39.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.001 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:39.001 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.997585+0000", 2024-09-21T10:19:39.001 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2069000000000001e-05 2024-09-21T10:19:39.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.001 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:39.001 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.998518+0000", 2024-09-21T10:19:39.001 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00093336299999999999 2024-09-21T10:19:39.001 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:39.002 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:39.002 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:39.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.002 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.002 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:126426 7.15 7:aff43bec:::benchmark_data_smithi098_197834_object9909:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-09-21T10:19:39.002 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:14:53.790525+0000", 2024-09-21T10:19:39.002 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 285.08855284200001, 2024-09-21T10:19:39.002 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2095328049999998, 2024-09-21T10:19:39.002 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:39.002 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:39.002 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:39.002 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:19:39.002 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:19:39.002 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 126426 2024-09-21T10:19:39.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.003 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:39.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:39.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.790525+0000", 2024-09-21T10:19:39.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:39.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:39.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.790525+0000", 2024-09-21T10:19:39.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:39.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:39.004 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.790523+0000", 2024-09-21T10:19:39.004 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:19:39.004 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.004 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.004 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:39.004 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.790529+0000", 2024-09-21T10:19:39.004 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6329999999999999e-06 2024-09-21T10:19:39.004 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.004 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.004 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:39.004 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.790542+0000", 2024-09-21T10:19:39.004 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2525e-05 2024-09-21T10:19:39.004 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.004 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.005 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:39.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.790548+0000", 2024-09-21T10:19:39.005 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7049999999999996e-06 2024-09-21T10:19:39.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.005 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.005 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:39.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.790600+0000", 2024-09-21T10:19:39.005 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2506999999999997e-05 2024-09-21T10:19:39.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.005 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.005 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:39.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.790610+0000", 2024-09-21T10:19:39.005 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8570000000000007e-06 2024-09-21T10:19:39.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.006 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:39.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687405+0000", 2024-09-21T10:19:39.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.89679551000000002 2024-09-21T10:19:39.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.006 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:39.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687409+0000", 2024-09-21T10:19:39.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.258e-06 2024-09-21T10:19:39.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.006 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:39.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.982677+0000", 2024-09-21T10:19:39.007 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.295268587 2024-09-21T10:19:39.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.007 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:39.007 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.982762+0000", 2024-09-21T10:19:39.007 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4469999999999996e-05 2024-09-21T10:19:39.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.007 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:19:39.007 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.982864+0000", 2024-09-21T10:19:39.007 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000101946 2024-09-21T10:19:39.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.007 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:19:39.008 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.986623+0000", 2024-09-21T10:19:39.008 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003759255 2024-09-21T10:19:39.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.008 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.008 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:39.008 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.991755+0000", 2024-09-21T10:19:39.008 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051318320000000002 2024-09-21T10:19:39.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.008 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.008 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:39.008 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.998586+0000", 2024-09-21T10:19:39.008 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0068313940000000002 2024-09-21T10:19:39.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.008 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.009 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:39.009 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.998633+0000", 2024-09-21T10:19:39.009 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6925000000000002e-05 2024-09-21T10:19:39.009 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.009 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.009 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:39.009 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:56.000057+0000", 2024-09-21T10:19:39.009 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014242549999999999 2024-09-21T10:19:39.009 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:39.009 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:39.009 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:39.009 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.009 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.009 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:126444 7.15 7:a889100f:::benchmark_data_smithi098_197834_object9927:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-09-21T10:19:39.010 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:14:53.808084+0000", 2024-09-21T10:19:39.010 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 285.07099323800003, 2024-09-21T10:19:39.010 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1976205769999999, 2024-09-21T10:19:39.010 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:39.010 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:39.010 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:39.010 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:19:39.010 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:19:39.010 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 126444 2024-09-21T10:19:39.010 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.010 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:39.010 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.010 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:39.011 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.808084+0000", 2024-09-21T10:19:39.011 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:39.011 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.011 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.011 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:39.011 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.808084+0000", 2024-09-21T10:19:39.011 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:39.011 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.011 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.011 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:39.011 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.808082+0000", 2024-09-21T10:19:39.011 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:19:39.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.012 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.012 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:39.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.808091+0000", 2024-09-21T10:19:39.012 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1559999999999995e-06 2024-09-21T10:19:39.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.012 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.012 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:39.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.808110+0000", 2024-09-21T10:19:39.012 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9683000000000001e-05 2024-09-21T10:19:39.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.012 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.012 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:39.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.808123+0000", 2024-09-21T10:19:39.013 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2422999999999999e-05 2024-09-21T10:19:39.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.013 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.013 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:39.013 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.808228+0000", 2024-09-21T10:19:39.013 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105097 2024-09-21T10:19:39.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.013 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.013 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:39.013 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.808234+0000", 2024-09-21T10:19:39.013 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8619999999999999e-06 2024-09-21T10:19:39.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.013 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.013 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:39.013 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687425+0000", 2024-09-21T10:19:39.013 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.87919187700000001 2024-09-21T10:19:39.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.014 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.014 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:39.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687429+0000", 2024-09-21T10:19:39.014 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.36e-06 2024-09-21T10:19:39.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.014 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.014 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:39.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.983691+0000", 2024-09-21T10:19:39.014 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2962624059999999 2024-09-21T10:19:39.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.014 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.014 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:39.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.983786+0000", 2024-09-21T10:19:39.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5231999999999999e-05 2024-09-21T10:19:39.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.015 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:19:39.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.983890+0000", 2024-09-21T10:19:39.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010302300000000001 2024-09-21T10:19:39.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.015 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:19:39.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.986626+0000", 2024-09-21T10:19:39.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027369120000000002 2024-09-21T10:19:39.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.015 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:39.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.991788+0000", 2024-09-21T10:19:39.016 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051614230000000001 2024-09-21T10:19:39.016 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.016 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.018 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:39.018 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:56.003715+0000", 2024-09-21T10:19:39.018 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011927589000000001 2024-09-21T10:19:39.018 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.018 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.018 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:39.018 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:56.003773+0000", 2024-09-21T10:19:39.018 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7102000000000003e-05 2024-09-21T10:19:39.018 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.018 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.018 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:39.018 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:56.005705+0000", 2024-09-21T10:19:39.019 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001932198 2024-09-21T10:19:39.019 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:39.019 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:39.019 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:39.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.019 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.019 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:126522 7.15 7:a82e3069:::benchmark_data_smithi098_197834_object10005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-09-21T10:19:39.019 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:14:53.885910+0000", 2024-09-21T10:19:39.019 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 284.99316748500001, 2024-09-21T10:19:39.019 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1211003129999999, 2024-09-21T10:19:39.019 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:39.019 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:39.019 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:39.019 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:19:39.019 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:19:39.020 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 126522 2024-09-21T10:19:39.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.020 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:39.020 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.020 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:39.020 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.885910+0000", 2024-09-21T10:19:39.020 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:39.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.020 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.020 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:39.020 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.885910+0000", 2024-09-21T10:19:39.020 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:39.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.020 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.021 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:39.021 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.885909+0000", 2024-09-21T10:19:39.021 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:19:39.021 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.021 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.021 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:39.021 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.885914+0000", 2024-09-21T10:19:39.021 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.853e-06 2024-09-21T10:19:39.021 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.021 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.021 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:39.021 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.885923+0000", 2024-09-21T10:19:39.021 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4690000000000003e-06 2024-09-21T10:19:39.021 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.022 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.022 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:39.022 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.885927+0000", 2024-09-21T10:19:39.022 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1489999999999996e-06 2024-09-21T10:19:39.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.022 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.022 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:39.022 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.885970+0000", 2024-09-21T10:19:39.022 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3260000000000003e-05 2024-09-21T10:19:39.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.022 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.022 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:39.022 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.885978+0000", 2024-09-21T10:19:39.023 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1400000000000002e-06 2024-09-21T10:19:39.023 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.023 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.023 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:39.023 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687445+0000", 2024-09-21T10:19:39.023 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80146784000000004 2024-09-21T10:19:39.023 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.023 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.023 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:39.023 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687449+0000", 2024-09-21T10:19:39.023 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.264e-06 2024-09-21T10:19:39.023 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.023 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.024 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:39.024 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.984716+0000", 2024-09-21T10:19:39.024 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2972670179999999 2024-09-21T10:19:39.024 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.024 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.024 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:39.024 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.984800+0000", 2024-09-21T10:19:39.024 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4655999999999997e-05 2024-09-21T10:19:39.024 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.024 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.024 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:19:39.024 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.984900+0000", 2024-09-21T10:19:39.025 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9747999999999994e-05 2024-09-21T10:19:39.025 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.025 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.025 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:19:39.025 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.987585+0000", 2024-09-21T10:19:39.025 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026850649999999999 2024-09-21T10:19:39.025 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.025 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.025 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:39.025 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.991822+0000", 2024-09-21T10:19:39.025 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004237088 2024-09-21T10:19:39.025 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.025 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.025 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:39.026 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:56.005774+0000", 2024-09-21T10:19:39.026 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013951882000000001 2024-09-21T10:19:39.026 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.026 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.026 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:39.026 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:56.005836+0000", 2024-09-21T10:19:39.026 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1477999999999997e-05 2024-09-21T10:19:39.026 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.026 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.026 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:39.026 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:56.007010+0000", 2024-09-21T10:19:39.026 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001174551 2024-09-21T10:19:39.026 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:39.026 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:39.027 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:39.027 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.027 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.027 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:126540 7.15 7:a9cf6911:::benchmark_data_smithi098_197834_object10023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-09-21T10:19:39.027 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:14:53.908019+0000", 2024-09-21T10:19:39.027 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 284.97105830599997, 2024-09-21T10:19:39.027 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1021127829999999, 2024-09-21T10:19:39.027 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:39.027 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:39.027 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:39.027 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:19:39.027 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:19:39.027 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 126540 2024-09-21T10:19:39.027 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.027 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:39.028 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.028 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:39.028 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.908019+0000", 2024-09-21T10:19:39.028 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:39.028 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.028 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.028 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:39.028 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.908019+0000", 2024-09-21T10:19:39.028 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:39.028 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.028 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.028 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:39.028 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.908019+0000", 2024-09-21T10:19:39.028 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:19:39.028 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.029 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.029 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:39.029 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.908022+0000", 2024-09-21T10:19:39.029 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1109999999999999e-06 2024-09-21T10:19:39.029 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.029 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.029 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:39.029 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.908028+0000", 2024-09-21T10:19:39.029 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3300000000000004e-06 2024-09-21T10:19:39.029 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.029 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.029 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:39.029 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.908032+0000", 2024-09-21T10:19:39.029 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2350000000000001e-06 2024-09-21T10:19:39.029 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.030 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.030 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:39.030 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.908073+0000", 2024-09-21T10:19:39.030 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1137999999999998e-05 2024-09-21T10:19:39.030 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.030 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.030 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:39.030 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.908080+0000", 2024-09-21T10:19:39.030 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6309999999999997e-06 2024-09-21T10:19:39.030 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.030 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.030 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:39.030 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687465+0000", 2024-09-21T10:19:39.031 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.77938554400000004 2024-09-21T10:19:39.031 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.031 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.031 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:19:39.031 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687469+0000", 2024-09-21T10:19:39.031 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3100000000000001e-06 2024-09-21T10:19:39.031 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.031 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.031 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:39.031 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.985687+0000", 2024-09-21T10:19:39.031 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2982183919999999 2024-09-21T10:19:39.031 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.031 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.031 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:39.032 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.985771+0000", 2024-09-21T10:19:39.032 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4031999999999998e-05 2024-09-21T10:19:39.032 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.032 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.032 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:19:39.032 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.985858+0000", 2024-09-21T10:19:39.032 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6815000000000001e-05 2024-09-21T10:19:39.032 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.032 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.032 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:19:39.032 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.988566+0000", 2024-09-21T10:19:39.032 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027079109999999999 2024-09-21T10:19:39.032 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.032 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.032 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:39.033 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.991852+0000", 2024-09-21T10:19:39.033 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003286527 2024-09-21T10:19:39.033 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.033 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.033 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:39.033 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:56.008824+0000", 2024-09-21T10:19:39.033 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016971198 2024-09-21T10:19:39.033 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.033 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.033 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:39.033 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:56.008888+0000", 2024-09-21T10:19:39.033 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3833999999999999e-05 2024-09-21T10:19:39.033 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.033 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.033 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:39.033 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:56.010132+0000", 2024-09-21T10:19:39.034 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012443560000000001 2024-09-21T10:19:39.034 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:39.034 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:39.034 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:39.034 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.034 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.034 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.45794.0:86883 8.d 8:b38f116c:::benchmark_data_smithi098_226587_object86882:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-09-21T10:19:39.034 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:03.884291+0000", 2024-09-21T10:19:39.034 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 34.994786787000002, 2024-09-21T10:19:39.034 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.077876780000000007, 2024-09-21T10:19:39.034 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:19:39.034 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:39.034 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:19:39.034 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.45794", 2024-09-21T10:19:39.035 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:19:39.035 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 86883 2024-09-21T10:19:39.035 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.035 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:19:39.035 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.035 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:19:39.035 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:03.884291+0000", 2024-09-21T10:19:39.035 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:39.035 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.035 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.035 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:19:39.035 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:03.884291+0000", 2024-09-21T10:19:39.035 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:19:39.035 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.036 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.036 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:19:39.036 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:03.884289+0000", 2024-09-21T10:19:39.036 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:19:39.036 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.036 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.036 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:19:39.036 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:03.884354+0000", 2024-09-21T10:19:39.036 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4476999999999999e-05 2024-09-21T10:19:39.036 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.036 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.036 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:19:39.036 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:03.884368+0000", 2024-09-21T10:19:39.036 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4387e-05 2024-09-21T10:19:39.037 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.037 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.037 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:19:39.037 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:03.884375+0000", 2024-09-21T10:19:39.037 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2300000000000002e-06 2024-09-21T10:19:39.037 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.037 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.037 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:19:39.037 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:03.884430+0000", 2024-09-21T10:19:39.037 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5312000000000002e-05 2024-09-21T10:19:39.037 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.037 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.037 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:19:39.037 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:03.884560+0000", 2024-09-21T10:19:39.038 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012998900000000001 2024-09-21T10:19:39.038 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.038 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.038 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:19:39.038 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:03.884690+0000", 2024-09-21T10:19:39.038 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012977799999999999 2024-09-21T10:19:39.038 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.038 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.038 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:39.038 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:03.891238+0000", 2024-09-21T10:19:39.038 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006547878 2024-09-21T10:19:39.038 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.038 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.038 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:19:39.039 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:03.891244+0000", 2024-09-21T10:19:39.039 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8839999999999999e-06 2024-09-21T10:19:39.039 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.039 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.039 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:39.039 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:03.962072+0000", 2024-09-21T10:19:39.039 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.070827996000000004 2024-09-21T10:19:39.039 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.039 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.039 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:19:39.039 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:03.962128+0000", 2024-09-21T10:19:39.039 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5646999999999998e-05 2024-09-21T10:19:39.039 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:19:39.040 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:19:39.040 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:19:39.040 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:03.962167+0000", 2024-09-21T10:19:39.040 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9762999999999999e-05 2024-09-21T10:19:39.040 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:39.040 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:39.040 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:39.040 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:19:39.040 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:19:39.040 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:19:39.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:39.042 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:39.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:39.143 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:39.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:39.189 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:39.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:39.290 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:39.307 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:39 smithi098 ceph-mon[97580]: pgmap v894: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.9 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 706 op/s; 19443/290682 objects degraded (6.689%); 6819/290682 objects misplaced (2.346%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:19:39.308 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:39 smithi098 ceph-mon[100914]: pgmap v894: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.9 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 706 op/s; 19443/290682 objects degraded (6.689%); 6819/290682 objects misplaced (2.346%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:19:39.326 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:39.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:39.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:39.427 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:39.462 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:39.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:39.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:39.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:39 smithi196 ceph-mon[84784]: pgmap v894: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.9 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 706 op/s; 19443/290682 objects degraded (6.689%); 6819/290682 objects misplaced (2.346%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:19:39.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:39.601 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:39.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:39.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:39.703 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:39.728 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:39.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:39.830 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:39.863 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:39.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:39.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:39.965 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:39.998 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:39.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:39.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:40.100 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:40.139 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:19:40.155 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:40.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:40.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:40.244 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:19:40.258 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:40.282 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T10:19:40.357 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:40.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:40.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:40.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:40 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/817553307' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:19:40.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:40 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:19:40.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:19:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:19:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:19:40.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:40.501 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:40.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:40.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:40.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:40 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/817553307' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:19:40.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:40 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:19:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:40 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/817553307' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:19:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:40 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:19:40.604 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:40.635 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:40.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:40.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:40.737 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:40.768 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:40.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:40.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:40.870 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:40.904 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:40.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:40.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:41.006 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:41.043 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:41.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:41.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:41.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:19:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:19:40] "GET /metrics HTTP/1.1" 200 36544 "" "Prometheus/2.51.0" 2024-09-21T10:19:41.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:41.191 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T10:19:41.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:41.210 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:41.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:41.311 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:41.356 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:41.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:41.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:41.435 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:41 smithi098 ceph-mon[97580]: pgmap v895: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 1.02k op/s; 19443/282972 objects degraded (6.871%); 6680/282972 objects misplaced (2.361%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T10:19:41.436 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:41 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:19:41.436 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:41 smithi098 ceph-mon[97580]: osdmap e284: 8 total, 8 up, 6 in 2024-09-21T10:19:41.436 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:41 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/282972 objects degraded (6.871%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:19:41.436 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:41 smithi098 ceph-mon[100914]: pgmap v895: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 1.02k op/s; 19443/282972 objects degraded (6.871%); 6680/282972 objects misplaced (2.361%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T10:19:41.436 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:41 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:19:41.436 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:41 smithi098 ceph-mon[100914]: osdmap e284: 8 total, 8 up, 6 in 2024-09-21T10:19:41.436 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:41 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/282972 objects degraded (6.871%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:19:41.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:41.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:41.500 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:41.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:41.601 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:41.626 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:41.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:41.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:41.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:41 smithi196 ceph-mon[84784]: pgmap v895: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 1.02k op/s; 19443/282972 objects degraded (6.871%); 6680/282972 objects misplaced (2.361%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T10:19:41.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:41 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:19:41.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:41 smithi196 ceph-mon[84784]: osdmap e284: 8 total, 8 up, 6 in 2024-09-21T10:19:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:41 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/282972 objects degraded (6.871%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:19:41.727 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:41.753 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:41.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:41.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:41.855 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:41.886 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:41.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:41.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:41.889 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:41.973 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:19:41.973 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:19:41.988 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:42.015 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:19:42.016 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:19:42.052 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:19:42.052 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:19:42.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:42.058 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:42.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:42.085 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:19:42.085 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:19:42.116 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:19:42.116 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:19:42.152 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:19:42.152 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:19:42.158 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:42.179 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:19:42.179 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:19:42.203 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:42.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:42.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:42.210 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:19:42.210 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:19:42.304 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:42.340 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:42.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:42.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:42.441 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:42.474 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:42.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:42.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:42.576 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:42.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:42 smithi098 ceph-mon[100914]: pgmap v897: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 663 op/s; 19443/282972 objects degraded (6.871%); 6680/282972 objects misplaced (2.361%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T10:19:42.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:42 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:19:42.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:42 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3669407241' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:42.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:42 smithi098 ceph-mon[97580]: pgmap v897: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 663 op/s; 19443/282972 objects degraded (6.871%); 6680/282972 objects misplaced (2.361%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T10:19:42.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:42 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:19:42.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:42 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3669407241' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:42.604 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:42.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:42.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:42.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:42 smithi196 ceph-mon[84784]: pgmap v897: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 663 op/s; 19443/282972 objects degraded (6.871%); 6680/282972 objects misplaced (2.361%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T10:19:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:42 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:19:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:42 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3669407241' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:42.706 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:42.740 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:42.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:42.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:42.842 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:42.881 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:42.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:42.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:42.910 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:42.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:43.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:43.019 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:43.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:43.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:43.150 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:43.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:43.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:43.252 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:43.299 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:43.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:43.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:43.401 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:43.444 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:43.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:43.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:43.546 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:43.579 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:43.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:43.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:43.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:43.715 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:43.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:43.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:43.816 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:43.854 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:43.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:43.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:43.956 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:43.992 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:43.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:43.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:44.094 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:44.133 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:44.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:44.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:44.235 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:44.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:44.288 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:44.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:44.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:44.426 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:44.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:44.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:44.527 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:44.562 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:44.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:44.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:44.664 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:44.695 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:44 smithi098 ceph-mon[97580]: pgmap v898: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 834 op/s; 19443/275085 objects degraded (7.068%); 6680/275085 objects misplaced (2.428%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:19:44.695 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:44 smithi098 ceph-mon[100914]: pgmap v898: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 834 op/s; 19443/275085 objects degraded (7.068%); 6680/275085 objects misplaced (2.428%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:19:44.705 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:44 smithi196 ceph-mon[84784]: pgmap v898: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 834 op/s; 19443/275085 objects degraded (7.068%); 6680/275085 objects misplaced (2.428%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:19:44.708 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:44.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:44.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:44.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:44.851 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:44.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:44.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:44.953 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:44.985 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:44.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:44.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:45.086 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:45.111 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:45.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:45.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:45.213 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:45.250 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:45.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:45.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:45.352 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:45.386 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:45.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:45.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:45.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:45.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:45.527 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:45.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:45.628 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:45.671 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:45.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:45.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:45.773 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:45.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:45.818 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:45.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:45.919 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:45.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:45.966 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:45.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:46.067 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:46.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:46.105 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:46.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:46.205 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:46.257 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:46.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:46.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:46.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:46.396 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:46.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:46.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:46.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:46.515 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:19:46.515 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:19:46.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:46.541 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:46.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:46.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:19:46.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:46 smithi098 ceph-mon[100914]: pgmap v899: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 896 op/s; 19443/271347 objects degraded (7.165%); 6538/271347 objects misplaced (2.409%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:19:46.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:46 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/271347 objects degraded (7.165%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:19:46.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:46 smithi098 ceph-mon[97580]: pgmap v899: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 896 op/s; 19443/271347 objects degraded (7.165%); 6538/271347 objects misplaced (2.409%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:19:46.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:46 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/271347 objects degraded (7.165%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:19:46.642 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:46.676 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:46.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:46.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:46.778 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:46.806 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:46.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:46.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:46.845 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:46 smithi196 ceph-mon[84784]: pgmap v899: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 896 op/s; 19443/271347 objects degraded (7.165%); 6538/271347 objects misplaced (2.409%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:19:46.845 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:46 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/271347 objects degraded (7.165%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:19:46.909 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:46.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:46.947 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:46.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:47.048 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:47.081 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:47.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:47.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:47.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:47.220 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:47.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:47.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:47.322 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:47.348 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:47.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:47.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:47.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:47.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:47.492 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:47.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:47.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:47.626 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:47.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:47.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:47.727 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:47.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:47.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:47.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:47.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:47.889 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:47.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:47.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:47.991 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:48.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:48.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:48.132 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:48.167 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:48.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:48.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:48.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:48.299 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:48.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:48.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:48.400 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:48.435 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:48.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:48.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:48.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:48.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:48.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:48.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:48 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:19:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:48 smithi196 ceph-mon[84784]: pgmap v900: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 796 op/s; 19443/266772 objects degraded (7.288%); 6538/266772 objects misplaced (2.451%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:19:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:48 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:19:48.659 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:48.700 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:48.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:48.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:48.801 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:48.832 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:48.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:48.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:48.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:48 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:19:48.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:48 smithi098 ceph-mon[97580]: pgmap v900: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 796 op/s; 19443/266772 objects degraded (7.288%); 6538/266772 objects misplaced (2.451%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:19:48.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:48 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:19:48.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:48 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:19:48.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:48 smithi098 ceph-mon[100914]: pgmap v900: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 796 op/s; 19443/266772 objects degraded (7.288%); 6538/266772 objects misplaced (2.451%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:19:48.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:48 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:19:48.934 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:48.965 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:48.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:48.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:49.067 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:49.103 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:49.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:49.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:49.205 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:49.234 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:49.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:49.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:49.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:49.376 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:49.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:49.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:49.478 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:49.499 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:49.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:49.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:49.600 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:49.627 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:49.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:49.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:49.729 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:49.766 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:49.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:49.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:49.868 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:49.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:49.903 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:49.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:50.004 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:50.038 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:50.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:50.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:50.139 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:50.187 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:50.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:50.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:50.289 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:50.330 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:50.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:50.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:50.432 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:50.446 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:19:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:19:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:19:50.467 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:50.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:50.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:50.569 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:50.596 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:50.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:50.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:50.698 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:50.719 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:50.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:50.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:50.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:50.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:19:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:19:50] "GET /metrics HTTP/1.1" 200 36544 "" "Prometheus/2.51.0" 2024-09-21T10:19:50.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:50 smithi098 ceph-mon[97580]: pgmap v901: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 801 op/s; 19443/258921 objects degraded (7.509%); 6382/258921 objects misplaced (2.465%); 1.9 MiB/s, 29 objects/s recovering 2024-09-21T10:19:50.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:50 smithi098 ceph-mon[100914]: pgmap v901: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 801 op/s; 19443/258921 objects degraded (7.509%); 6382/258921 objects misplaced (2.465%); 1.9 MiB/s, 29 objects/s recovering 2024-09-21T10:19:50.854 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:50.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:50.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:50 smithi196 ceph-mon[84784]: pgmap v901: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 801 op/s; 19443/258921 objects degraded (7.509%); 6382/258921 objects misplaced (2.465%); 1.9 MiB/s, 29 objects/s recovering 2024-09-21T10:19:50.956 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:50.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:50.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:51.095 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:51.127 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:51.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:51.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:51.229 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:51.252 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:51.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:51.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:51.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:51.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:51.384 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:51.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:51.484 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:51.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:51.519 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:51.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:51.620 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:51.654 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:51.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:51.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:51.756 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:51.785 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:51.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:51.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:51.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:51 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/258921 objects degraded (7.509%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:19:51.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:51 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/258921 objects degraded (7.509%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:19:51.886 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:51 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/258921 objects degraded (7.509%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:19:51.911 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:51.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:51.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:52.012 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:52.051 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:52.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:52.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:52.153 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:52.176 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:52.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:52.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:52.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:52.316 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:52.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:52.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:52.418 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:52.457 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:52.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:52.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:52.559 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:52.604 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:52.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:52.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:52.706 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:52.742 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:52.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:52.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:52.843 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:52.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:52 smithi098 ceph-mon[97580]: pgmap v902: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 721 op/s; 19443/258921 objects degraded (7.509%); 6382/258921 objects misplaced (2.465%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:19:52.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:52.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:52 smithi098 ceph-mon[100914]: pgmap v902: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 721 op/s; 19443/258921 objects degraded (7.509%); 6382/258921 objects misplaced (2.465%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:19:52.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:52.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:52.874 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:52.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:52.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:52 smithi196 ceph-mon[84784]: pgmap v902: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 721 op/s; 19443/258921 objects degraded (7.509%); 6382/258921 objects misplaced (2.465%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:19:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:52.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:53.007 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:53.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:53.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:53.109 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:53.142 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:53.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:53.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:53.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:53.274 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:53.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:53.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:53.375 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:53.412 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:53.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:53.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:53.514 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:53.556 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:53.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:53.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:53.657 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:53.692 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:53.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:53.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:53.794 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:53.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:53.825 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:53.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:53.926 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:53.958 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:53.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:53.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:54.059 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:54.092 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:54.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:54.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:54.194 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:54.230 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:54.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:54.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:54.331 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:54.363 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:54.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:54.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:54.464 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:54.498 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:54.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:54.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:54.600 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:54.643 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:54.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:54.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:54.706 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:19:54.744 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:54.776 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:54.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:54.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:54.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:54 smithi098 ceph-mon[97580]: pgmap v903: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 897 op/s; 19443/250641 objects degraded (7.757%); 6382/250641 objects misplaced (2.546%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T10:19:54.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:54 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/317265130' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:19:54.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:54 smithi098 ceph-mon[100914]: pgmap v903: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 897 op/s; 19443/250641 objects degraded (7.757%); 6382/250641 objects misplaced (2.546%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T10:19:54.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:54 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/317265130' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:19:54.878 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:54.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:54 smithi196 ceph-mon[84784]: pgmap v903: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 897 op/s; 19443/250641 objects degraded (7.757%); 6382/250641 objects misplaced (2.546%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T10:19:54.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:54 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/317265130' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:19:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:54.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:54.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:55.015 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:55.052 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:55.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:55.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:55.153 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:55.184 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:55.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:55.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:55.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:55.319 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:55.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:55.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:55.421 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:55.451 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:55.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:55.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:55.552 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:55.584 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:55.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:55.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:55.686 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:55.719 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:55.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:55.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:55.821 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:55.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:55 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:19:55.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:55 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/317265130' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:19:55.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:55 smithi098 ceph-mon[97580]: osdmap e285: 8 total, 8 up, 6 in 2024-09-21T10:19:55.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:55 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:19:55.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:55 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/317265130' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:19:55.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:55 smithi098 ceph-mon[100914]: osdmap e285: 8 total, 8 up, 6 in 2024-09-21T10:19:55.849 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:55.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:55.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:55.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:55 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:19:55.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:55 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/317265130' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:19:55.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:55 smithi196 ceph-mon[84784]: osdmap e285: 8 total, 8 up, 6 in 2024-09-21T10:19:55.951 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:55.983 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:55.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:55.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:56.085 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:56.123 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:56.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:56.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:56.225 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:56.265 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:56.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:56.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:56.367 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:56.399 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:56.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:56.501 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:56.529 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:56.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:56.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:56.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:56 smithi098 ceph-mon[97580]: pgmap v905: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.0 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 810 op/s; 19443/247017 objects degraded (7.871%); 6212/247017 objects misplaced (2.515%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T10:19:56.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:56 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/247017 objects degraded (7.871%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:19:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:56 smithi098 ceph-mon[100914]: pgmap v905: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.0 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 810 op/s; 19443/247017 objects degraded (7.871%); 6212/247017 objects misplaced (2.515%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T10:19:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:56 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/247017 objects degraded (7.871%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:19:56.631 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:56.667 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:56.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:56.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:56.769 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:56.804 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:56.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:56.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:56.826 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:19:56.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:56 smithi196 ceph-mon[84784]: pgmap v905: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.0 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 810 op/s; 19443/247017 objects degraded (7.871%); 6212/247017 objects misplaced (2.515%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T10:19:56.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:56 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/247017 objects degraded (7.871%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:19:56.907 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:56.933 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:56.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:56.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:57.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:57.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:57.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:57.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:57.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:57.245 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:57.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:57.346 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:57.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:57.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:57.512 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:57.545 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:57.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:57.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:57.647 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:57.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:57.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:57.784 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:57.818 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:57.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:57.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:57.920 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:57.956 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:57.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:57.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:58.058 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:58.098 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:58.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:58.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:58.199 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:58.234 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:58.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:58.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:58.335 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:58.356 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:58.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:58.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:58.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:19:58.458 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:58.481 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:58.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:58.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:58.583 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:58.609 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:58.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:58.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:58.711 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:58.732 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:58.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:58.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:58.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:19:58 smithi098 ceph-mon[97580]: pgmap v906: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 810 op/s; 19443/242439 objects degraded (8.020%); 6212/242439 objects misplaced (2.562%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T10:19:58.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:19:58 smithi098 ceph-mon[100914]: pgmap v906: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 810 op/s; 19443/242439 objects degraded (8.020%); 6212/242439 objects misplaced (2.562%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T10:19:58.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:58.876 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:58.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:19:58 smithi196 ceph-mon[84784]: pgmap v906: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 810 op/s; 19443/242439 objects degraded (8.020%); 6212/242439 objects misplaced (2.562%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T10:19:58.977 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:58.994 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:59.002 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:59.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:59.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:59.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:59.147 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:59.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:59.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:59.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:59.290 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:59.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:59.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:59.392 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:59.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:59.424 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:59.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:59.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:59.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:59.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:59.656 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:59.692 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:59.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:59.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:59.793 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:59.826 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:19:59.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:59.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:59.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:59.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:59.969 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:19:59.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:00.008 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:00.070 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:00.107 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:00.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:00.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:00.208 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:00.235 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:00.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:00.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:00.337 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:00.374 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:00.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:00.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:00.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:00.495 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:20:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:20:00.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:20:00.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:00.540 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:00.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:00.641 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:00.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:00.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:00.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:00.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:00.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:00.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:20:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:20:00] "GET /metrics HTTP/1.1" 200 36543 "" "Prometheus/2.51.0" 2024-09-21T10:20:00.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:00 smithi098 ceph-mon[97580]: pgmap v907: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.8 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 806 op/s; 19443/234711 objects degraded (8.284%); 6014/234711 objects misplaced (2.562%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:20:00.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:00 smithi098 ceph-mon[97580]: Health detail: HEALTH_WARN noscrub flag(s) set; Degraded data redundancy: 19443/234711 objects degraded (8.284%), 5 pgs degraded, 6 pgs undersized 2024-09-21T10:20:00.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:00 smithi098 ceph-mon[97580]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-21T10:20:00.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:00 smithi098 ceph-mon[97580]: [WRN] PG_DEGRADED: Degraded data redundancy: 19443/234711 objects degraded (8.284%), 5 pgs degraded, 6 pgs undersized 2024-09-21T10:20:00.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:00 smithi098 ceph-mon[97580]: pg 8.0 is stuck undersized for 78s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,5] 2024-09-21T10:20:00.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:00 smithi098 ceph-mon[97580]: pg 8.2 is stuck undersized for 84s, current state active+recovery_wait+undersized+degraded+remapped, last acting [0,5] 2024-09-21T10:20:00.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:00 smithi098 ceph-mon[97580]: pg 8.7 is stuck undersized for 84s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,6] 2024-09-21T10:20:00.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:00 smithi098 ceph-mon[97580]: pg 8.b is stuck undersized for 84s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,2] 2024-09-21T10:20:00.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:00 smithi098 ceph-mon[97580]: pg 8.c is stuck undersized for 84s, current state active+recovery_wait+undersized+degraded+remapped, last acting [0,7] 2024-09-21T10:20:00.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:00 smithi098 ceph-mon[97580]: pg 8.f is stuck undersized for 84s, current state active+recovering+undersized+remapped, last acting [2,1] 2024-09-21T10:20:00.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:00 smithi098 ceph-mon[100914]: pgmap v907: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.8 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 806 op/s; 19443/234711 objects degraded (8.284%); 6014/234711 objects misplaced (2.562%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:20:00.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:00 smithi098 ceph-mon[100914]: Health detail: HEALTH_WARN noscrub flag(s) set; Degraded data redundancy: 19443/234711 objects degraded (8.284%), 5 pgs degraded, 6 pgs undersized 2024-09-21T10:20:00.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:00 smithi098 ceph-mon[100914]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-21T10:20:00.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:00 smithi098 ceph-mon[100914]: [WRN] PG_DEGRADED: Degraded data redundancy: 19443/234711 objects degraded (8.284%), 5 pgs degraded, 6 pgs undersized 2024-09-21T10:20:00.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:00 smithi098 ceph-mon[100914]: pg 8.0 is stuck undersized for 78s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,5] 2024-09-21T10:20:00.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:00 smithi098 ceph-mon[100914]: pg 8.2 is stuck undersized for 84s, current state active+recovery_wait+undersized+degraded+remapped, last acting [0,5] 2024-09-21T10:20:00.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:00 smithi098 ceph-mon[100914]: pg 8.7 is stuck undersized for 84s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,6] 2024-09-21T10:20:00.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:00 smithi098 ceph-mon[100914]: pg 8.b is stuck undersized for 84s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,2] 2024-09-21T10:20:00.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:00 smithi098 ceph-mon[100914]: pg 8.c is stuck undersized for 84s, current state active+recovery_wait+undersized+degraded+remapped, last acting [0,7] 2024-09-21T10:20:00.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:00 smithi098 ceph-mon[100914]: pg 8.f is stuck undersized for 84s, current state active+recovering+undersized+remapped, last acting [2,1] 2024-09-21T10:20:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:00 smithi196 ceph-mon[84784]: pgmap v907: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.8 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 806 op/s; 19443/234711 objects degraded (8.284%); 6014/234711 objects misplaced (2.562%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:20:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:00 smithi196 ceph-mon[84784]: Health detail: HEALTH_WARN noscrub flag(s) set; Degraded data redundancy: 19443/234711 objects degraded (8.284%), 5 pgs degraded, 6 pgs undersized 2024-09-21T10:20:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:00 smithi196 ceph-mon[84784]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-21T10:20:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:00 smithi196 ceph-mon[84784]: [WRN] PG_DEGRADED: Degraded data redundancy: 19443/234711 objects degraded (8.284%), 5 pgs degraded, 6 pgs undersized 2024-09-21T10:20:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:00 smithi196 ceph-mon[84784]: pg 8.0 is stuck undersized for 78s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,5] 2024-09-21T10:20:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:00 smithi196 ceph-mon[84784]: pg 8.2 is stuck undersized for 84s, current state active+recovery_wait+undersized+degraded+remapped, last acting [0,5] 2024-09-21T10:20:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:00 smithi196 ceph-mon[84784]: pg 8.7 is stuck undersized for 84s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,6] 2024-09-21T10:20:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:00 smithi196 ceph-mon[84784]: pg 8.b is stuck undersized for 84s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,2] 2024-09-21T10:20:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:00 smithi196 ceph-mon[84784]: pg 8.c is stuck undersized for 84s, current state active+recovery_wait+undersized+degraded+remapped, last acting [0,7] 2024-09-21T10:20:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:00 smithi196 ceph-mon[84784]: pg 8.f is stuck undersized for 84s, current state active+recovering+undersized+remapped, last acting [2,1] 2024-09-21T10:20:00.924 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:00.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:00.958 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:00.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:01.059 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:01.088 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:01.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:01.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:01.189 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:01.218 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:01.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:01.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:01.319 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:01.348 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:01.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:01.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:01.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:01.511 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:01.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:01.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:01.612 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:01.644 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:01.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:01.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:01.695 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:01 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/234711 objects degraded (8.284%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:01.695 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:01 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/234711 objects degraded (8.284%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:01.745 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:01.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:01.780 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:01.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:01.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:01.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:01 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/234711 objects degraded (8.284%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:01.909 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:01.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:01.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:02.011 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:02.042 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:02.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:02.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:02.143 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:02.177 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:02.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:02.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:02.279 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:02.323 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:02.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:02.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:02.424 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:02.463 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:02.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:02.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:02.535 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T10:20:02.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:02.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:02.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:02.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:02.749 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:02.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:02.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:02.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:02 smithi098 ceph-mon[97580]: pgmap v908: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.8 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 806 op/s; 19443/234711 objects degraded (8.284%); 6014/234711 objects misplaced (2.562%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:20:02.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:02 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2865232419' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:20:02.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:02 smithi098 ceph-mon[100914]: pgmap v908: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.8 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 806 op/s; 19443/234711 objects degraded (8.284%); 6014/234711 objects misplaced (2.562%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:20:02.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:02 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2865232419' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:20:02.851 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:02.877 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:02.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:02.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:02.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:02 smithi196 ceph-mon[84784]: pgmap v908: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.8 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 806 op/s; 19443/234711 objects degraded (8.284%); 6014/234711 objects misplaced (2.562%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:20:02.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:02 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2865232419' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:20:02.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:03.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:03.021 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:03.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:03.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:03.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:03.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:03.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:03.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:03.305 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:03.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:03.406 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:03.438 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:03.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:03.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:03.540 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:03.574 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:03.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:03.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:03.675 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:03.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:03.724 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:03.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:03.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:03.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:03 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2865232419' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:20:03.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:03 smithi098 ceph-mon[100914]: osdmap e286: 8 total, 8 up, 6 in 2024-09-21T10:20:03.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:03 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2865232419' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:20:03.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:03 smithi098 ceph-mon[97580]: osdmap e286: 8 total, 8 up, 6 in 2024-09-21T10:20:03.863 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:03.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:03.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:03 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2865232419' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:20:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:03 smithi196 ceph-mon[84784]: osdmap e286: 8 total, 8 up, 6 in 2024-09-21T10:20:03.965 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:04.010 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:04.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:04.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:04.112 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:04.149 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:04.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:04.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:04.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:04.283 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:04.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:04.384 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:04.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T10:20:04.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:04.442 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:04.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:04.543 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:04.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:04.588 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:04.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:04.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:04 smithi098 ceph-mon[97580]: pgmap v910: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 762 op/s; 19443/226635 objects degraded (8.579%); 6014/226635 objects misplaced (2.654%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:20:04.611 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:04 smithi098 ceph-mon[100914]: pgmap v910: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 762 op/s; 19443/226635 objects degraded (8.579%); 6014/226635 objects misplaced (2.654%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:20:04.690 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:04.727 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:04.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:04.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:04.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:04.884 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:04.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:04.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:04.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:04 smithi196 ceph-mon[84784]: pgmap v910: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 762 op/s; 19443/226635 objects degraded (8.579%); 6014/226635 objects misplaced (2.654%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:20:04.986 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:05.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:05.031 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:05.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:05.132 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:05.159 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:05.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:05.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:05.203 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:20:05.204 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:20:05.204 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:20:05.204 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:20:05.235 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:20:05.235 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:20:05.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:05.281 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:20:05.281 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:20:05.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:05.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:05.314 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:20:05.315 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:20:05.321 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:20:05.321 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:20:05.321 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-21T10:20:05.321 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-21T10:20:05.321 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:20:05.342 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:20:05.343 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:20:05.382 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:20:05.382 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:20:05.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:05.413 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:20:05.414 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:20:05.429 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:05.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:05.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:05.531 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:05.563 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:05.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:05.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:05.664 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:05.704 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:05.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:05.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:05.806 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:05.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:05 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/722480521' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:20:05.848 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:05.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:05 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/722480521' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:20:05.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:05.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:05.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:05 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/722480521' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:20:05.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:05.981 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:05.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:05.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:06.083 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:06.118 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:06.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:06.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:06.219 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:06.240 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:06.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:06.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:06.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:06.373 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:06.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:06.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:06.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:06.502 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:06.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:06.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:06.603 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:06.642 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:06.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:06.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:06.744 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:06.765 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:06.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:06.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:06.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:06 smithi098 ceph-mon[100914]: pgmap v911: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 804 op/s; 19443/222867 objects degraded (8.724%); 5858/222867 objects misplaced (2.628%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:20:06.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:06 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/222867 objects degraded (8.724%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:06.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:06 smithi098 ceph-mon[97580]: pgmap v911: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 804 op/s; 19443/222867 objects degraded (8.724%); 5858/222867 objects misplaced (2.628%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:20:06.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:06 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/222867 objects degraded (8.724%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:06.867 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:06.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:06 smithi196 ceph-mon[84784]: pgmap v911: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 804 op/s; 19443/222867 objects degraded (8.724%); 5858/222867 objects misplaced (2.628%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:20:06.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:06 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/222867 objects degraded (8.724%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:06.902 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:06.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:06.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:07.003 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:07.030 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:07.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:07.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:07.134 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:07.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:07.173 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:07.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:07.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:07.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:07.306 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:07.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:07.407 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:07.440 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:07.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:07.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:07.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:07.577 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:07.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:07.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:07.678 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:07.714 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:07.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:07.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:07.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:07.854 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:07.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:07.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:07.955 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:07.990 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:07.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:07.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:08.091 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:08.125 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:08.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:08.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:08.227 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:08.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:08.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:08.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:08.401 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:08.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:08.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:08.503 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:08.542 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:08.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:08.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:20:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:08.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:20:08.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:08.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:20:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:08.644 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:08.681 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:08.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:08.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:08.783 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:08.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:08.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:08.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:08.913 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:08.945 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:08.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:08.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:09.046 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:09.080 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:09.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:09.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:09.182 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:09.216 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:09.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:09.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:09.317 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:09.351 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:09.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:09.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:09.453 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:09.485 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:09.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:09.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:09.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:09 smithi196 ceph-mon[84784]: pgmap v912: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.4 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 806 op/s; 19443/218232 objects degraded (8.909%); 5858/218232 objects misplaced (2.684%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:20:09.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:09.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:09 smithi098 ceph-mon[97580]: pgmap v912: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.4 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 806 op/s; 19443/218232 objects degraded (8.909%); 5858/218232 objects misplaced (2.684%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:20:09.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:09 smithi098 ceph-mon[100914]: pgmap v912: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.4 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 806 op/s; 19443/218232 objects degraded (8.909%); 5858/218232 objects misplaced (2.684%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:20:09.632 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:09.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:09.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:09.734 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:09.770 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:09.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:09.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:09.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:20:09.872 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:09.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:09.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:10.014 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:10.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:10.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:10.133 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:10.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:10.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:10.271 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:20:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:20:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:20:10.278 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:10.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:10.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:10.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:10.414 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:10.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:10.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:10.516 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:10.554 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:10.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:10.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:10.636 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:10.656 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:10.720 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:10.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:10.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:10.822 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:10.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:20:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:20:10] "GET /metrics HTTP/1.1" 200 36546 "" "Prometheus/2.51.0" 2024-09-21T10:20:10.856 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:10.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:10.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:10.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:10.996 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:10.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:10.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:11.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:11.145 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:11.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:11.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:11.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:11.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:11.288 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:11.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:11.388 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:11.421 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:11.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:11.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:11.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:11.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:11.576 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:11.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:11.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:11 smithi098 ceph-mon[97580]: pgmap v913: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 754 op/s; 19443/212070 objects degraded (9.168%); 5858/212070 objects misplaced (2.762%); 998 KiB/s, 15 objects/s recovering 2024-09-21T10:20:11.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:11 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/212070 objects degraded (9.168%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:11.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:11 smithi098 ceph-mon[100914]: pgmap v913: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 754 op/s; 19443/212070 objects degraded (9.168%); 5858/212070 objects misplaced (2.762%); 998 KiB/s, 15 objects/s recovering 2024-09-21T10:20:11.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:11 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/212070 objects degraded (9.168%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:11.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:11 smithi196 ceph-mon[84784]: pgmap v913: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 754 op/s; 19443/212070 objects degraded (9.168%); 5858/212070 objects misplaced (2.762%); 998 KiB/s, 15 objects/s recovering 2024-09-21T10:20:11.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:11 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/212070 objects degraded (9.168%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:11.677 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:11.708 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:11.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:11.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:11.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:11.846 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:11.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:11.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:11.948 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:11.985 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:11.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:11.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:12.086 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:12.113 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:12.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:12.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:12.215 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:12.253 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:12.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:12.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:12.355 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:12.400 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:12.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:12.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:12.501 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:12.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:12.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:12.636 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:12.643 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:20:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:12.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:12.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:12.773 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:12.801 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:12.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:12.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:12.902 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:12.930 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:12.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:12.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:13.031 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:13.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:13.059 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:13.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:13.160 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:13.188 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:13.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:13.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:13.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:13.339 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:13.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:13.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:13.441 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:13.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:13.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:13.576 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:13.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:13 smithi098 ceph-mon[97580]: pgmap v914: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 805 op/s; 19443/210549 objects degraded (9.234%); 5711/210549 objects misplaced (2.712%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T10:20:13.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:13 smithi098 ceph-mon[100914]: pgmap v914: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 805 op/s; 19443/210549 objects degraded (9.234%); 5711/210549 objects misplaced (2.712%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T10:20:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:13.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:13.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:13.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:13 smithi196 ceph-mon[84784]: pgmap v914: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 805 op/s; 19443/210549 objects degraded (9.234%); 5711/210549 objects misplaced (2.712%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T10:20:13.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:13.756 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:13.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:13.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:13.857 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:13.862 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:13.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:13.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:14.014 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:14.063 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:14.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:14.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:14.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:14.196 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:14.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:14.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:14.298 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:14.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:14.326 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:14.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:14.427 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:14.461 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:14.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:14.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:14.563 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:14.591 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:14.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:14.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:14.614 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:14.693 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:14.719 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:14.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:14.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:14.821 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:14.856 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:14.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:14.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:14.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:14.995 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:14.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:14.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:15.097 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:15.132 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:15.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:15.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:15.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:15.277 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:15.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:15.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:15.379 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:15.402 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:15.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:15.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:15.504 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:15.533 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:15.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:15.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:15.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:15 smithi098 ceph-mon[97580]: pgmap v915: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 858 op/s; 19443/202368 objects degraded (9.608%); 5711/202368 objects misplaced (2.822%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:20:15.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:15 smithi098 ceph-mon[100914]: pgmap v915: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 858 op/s; 19443/202368 objects degraded (9.608%); 5711/202368 objects misplaced (2.822%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:20:15.634 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:15.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:15 smithi196 ceph-mon[84784]: pgmap v915: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 858 op/s; 19443/202368 objects degraded (9.608%); 5711/202368 objects misplaced (2.822%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:20:15.673 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:15.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:15.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:15.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:15.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:15.814 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:15.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:15.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:15.963 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:15.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:15.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:16.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:16.101 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:16.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:16.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:16.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:16.242 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:16.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:16.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:16.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:16.374 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:16.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:16.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:16.476 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:16.499 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:16.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:16.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:16.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:16 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/202368 objects degraded (9.608%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:16.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:16 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/202368 objects degraded (9.608%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:16.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:16.634 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:16.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:16.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:16.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:16 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/202368 objects degraded (9.608%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:16.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:16.776 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:16.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:16.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:16.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:16.905 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:16.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:16.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:17.006 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:17.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:17.045 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:17.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:17.146 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:17.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:17.192 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:17.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:17.279 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T10:20:17.292 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:17.316 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:17.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:17.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:17.418 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:17.457 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:17.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:17.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:17.559 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:17.585 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:17.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:17.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:17.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:17 smithi098 ceph-mon[97580]: pgmap v916: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 776 op/s; 19443/198669 objects degraded (9.787%); 5536/198669 objects misplaced (2.787%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T10:20:17.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:17 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/239022466' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:20:17.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:17 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:20:17.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:17 smithi098 ceph-mon[100914]: pgmap v916: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 776 op/s; 19443/198669 objects degraded (9.787%); 5536/198669 objects misplaced (2.787%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T10:20:17.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:17 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/239022466' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:20:17.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:17 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:20:17.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:17 smithi196 ceph-mon[84784]: pgmap v916: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 776 op/s; 19443/198669 objects degraded (9.787%); 5536/198669 objects misplaced (2.787%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T10:20:17.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:17 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/239022466' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:20:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:17 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:20:17.687 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:17.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:17.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:17.818 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:17.856 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:17.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:17.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:17.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:17.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:17.998 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:17.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:18.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:18.128 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:18.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:18.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:18.229 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:18.257 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:18.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:18.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:18.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:18.408 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:18.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:18.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:18.510 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:18.541 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:18.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:18.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:18.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:18 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:20:18.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:18 smithi098 ceph-mon[97580]: osdmap e287: 8 total, 8 up, 6 in 2024-09-21T10:20:18.599 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:18 smithi098 ceph-mon[97580]: pgmap v918: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 805 op/s; 19443/194046 objects degraded (10.020%); 5536/194046 objects misplaced (2.853%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T10:20:18.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:18 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:20:18.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:18 smithi098 ceph-mon[100914]: osdmap e287: 8 total, 8 up, 6 in 2024-09-21T10:20:18.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:18 smithi098 ceph-mon[100914]: pgmap v918: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 805 op/s; 19443/194046 objects degraded (10.020%); 5536/194046 objects misplaced (2.853%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T10:20:18.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:18 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:20:18.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:18 smithi196 ceph-mon[84784]: osdmap e287: 8 total, 8 up, 6 in 2024-09-21T10:20:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:18 smithi196 ceph-mon[84784]: pgmap v918: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 805 op/s; 19443/194046 objects degraded (10.020%); 5536/194046 objects misplaced (2.853%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T10:20:18.643 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:18.675 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:18.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:18.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:18.777 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:18.806 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:18.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:18.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:18.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:18.950 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:18.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:18.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:19.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T10:20:19.051 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:19.114 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:19.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:19.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:19.216 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:19.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:19.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:19.309 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:20:19.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:19.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:19.441 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:19.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:19.541 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:19.572 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:19.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:19.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:19.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:19 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4064377489' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:20:19.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:19 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:20:19.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:19 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4064377489' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:20:19.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:19 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:20:19.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:19 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4064377489' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:20:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:19 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:20:19.674 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:19.717 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:19.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:19.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:19.819 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:19.851 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:20:19.851 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-21T10:20:19.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-21T10:20:19.852 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:20:19.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.852 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:126668 7.16 7:68e12100:::benchmark_data_smithi098_197834_object10151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:20:19.852 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:14:56.015607+0000", 2024-09-21T10:20:19.852 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 323.83081428999998, 2024-09-21T10:20:19.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8170184630000001, 2024-09-21T10:20:19.852 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:20:19.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:20:19.852 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:20:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:20:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:20:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 126668 2024-09-21T10:20:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:20:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:20:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.015607+0000", 2024-09-21T10:20:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:20:19.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.015607+0000", 2024-09-21T10:20:19.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:20:19.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.015609+0000", 2024-09-21T10:20:19.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.533e-06 2024-09-21T10:20:19.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:20:19.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.015614+0000", 2024-09-21T10:20:19.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2140000000000004e-06 2024-09-21T10:20:19.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:20:19.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.015624+0000", 2024-09-21T10:20:19.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.815e-06 2024-09-21T10:20:19.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:20:19.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.015630+0000", 2024-09-21T10:20:19.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1369999999999999e-06 2024-09-21T10:20:19.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.017016+0000", 2024-09-21T10:20:19.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001386013 2024-09-21T10:20:19.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.856 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.856 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.017025+0000", 2024-09-21T10:20:19.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1409999999999994e-06 2024-09-21T10:20:19.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.856 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.856 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.699261+0000", 2024-09-21T10:20:19.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.68223615599999998 2024-09-21T10:20:19.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.699273+0000", 2024-09-21T10:20:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1814000000000001e-05 2024-09-21T10:20:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.825506+0000", 2024-09-21T10:20:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.126233045 2024-09-21T10:20:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:20:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.825671+0000", 2024-09-21T10:20:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016523300000000001 2024-09-21T10:20:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:20:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.825793+0000", 2024-09-21T10:20:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012155 2024-09-21T10:20:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.830734+0000", 2024-09-21T10:20:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0049411480000000002 2024-09-21T10:20:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.830771+0000", 2024-09-21T10:20:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6936e-05 2024-09-21T10:20:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:20:19.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.830783+0000", 2024-09-21T10:20:19.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2553e-05 2024-09-21T10:20:19.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:20:19.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.830821+0000", 2024-09-21T10:20:19.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7963999999999998e-05 2024-09-21T10:20:19.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:20:19.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.832626+0000", 2024-09-21T10:20:19.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001804211 2024-09-21T10:20:19.859 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:20:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:126698 7.16 7:6d3aa5f6:::benchmark_data_smithi098_197834_object10181:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:20:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:14:56.033118+0000", 2024-09-21T10:20:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 323.81330386600001, 2024-09-21T10:20:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.801116078, 2024-09-21T10:20:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:20:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:20:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:20:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:20:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:20:19.861 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 126698 2024-09-21T10:20:19.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.861 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:20:19.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:20:19.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.033118+0000", 2024-09-21T10:20:19.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:20:19.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.033118+0000", 2024-09-21T10:20:19.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:20:19.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.033119+0000", 2024-09-21T10:20:19.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2890000000000001e-06 2024-09-21T10:20:19.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:20:19.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.033123+0000", 2024-09-21T10:20:19.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6410000000000001e-06 2024-09-21T10:20:19.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:20:19.863 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.033135+0000", 2024-09-21T10:20:19.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1348999999999999e-05 2024-09-21T10:20:19.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:20:19.863 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.033142+0000", 2024-09-21T10:20:19.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2019999999999998e-06 2024-09-21T10:20:19.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.863 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.033216+0000", 2024-09-21T10:20:19.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3530999999999998e-05 2024-09-21T10:20:19.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.033225+0000", 2024-09-21T10:20:19.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0799999999999995e-06 2024-09-21T10:20:19.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.699403+0000", 2024-09-21T10:20:19.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.66617875199999999 2024-09-21T10:20:19.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.699413+0000", 2024-09-21T10:20:19.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9110000000000005e-06 2024-09-21T10:20:19.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.828983+0000", 2024-09-21T10:20:19.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1295696719999999 2024-09-21T10:20:19.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:20:19.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.829080+0000", 2024-09-21T10:20:19.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6639999999999996e-05 2024-09-21T10:20:19.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:20:19.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.829178+0000", 2024-09-21T10:20:19.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8247999999999999e-05 2024-09-21T10:20:19.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.832692+0000", 2024-09-21T10:20:19.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035141450000000002 2024-09-21T10:20:19.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.832758+0000", 2024-09-21T10:20:19.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5729999999999996e-05 2024-09-21T10:20:19.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:20:19.868 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.832774+0000", 2024-09-21T10:20:19.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6767000000000001e-05 2024-09-21T10:20:19.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:20:19.868 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.832827+0000", 2024-09-21T10:20:19.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2122e-05 2024-09-21T10:20:19.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:20:19.868 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.834234+0000", 2024-09-21T10:20:19.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0014069989999999999 2024-09-21T10:20:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:20:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:126813 7.16 7:6a5b5ef2:::benchmark_data_smithi098_197834_object10296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:20:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:14:56.141981+0000", 2024-09-21T10:20:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 323.70444053199998, 2024-09-21T10:20:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.783015048, 2024-09-21T10:20:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:20:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:20:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:20:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:20:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:20:19.870 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 126813 2024-09-21T10:20:19.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.870 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:20:19.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:20:19.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.141981+0000", 2024-09-21T10:20:19.870 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:20:19.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.141981+0000", 2024-09-21T10:20:19.870 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:20:19.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.141983+0000", 2024-09-21T10:20:19.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.933e-06 2024-09-21T10:20:19.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:20:19.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.141988+0000", 2024-09-21T10:20:19.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5559999999999998e-06 2024-09-21T10:20:19.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:20:19.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.142009+0000", 2024-09-21T10:20:19.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0928999999999999e-05 2024-09-21T10:20:19.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:20:19.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.142018+0000", 2024-09-21T10:20:19.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2150000000000007e-06 2024-09-21T10:20:19.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.145154+0000", 2024-09-21T10:20:19.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031358969999999999 2024-09-21T10:20:19.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.873 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.145164+0000", 2024-09-21T10:20:19.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9289999999999993e-06 2024-09-21T10:20:19.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.873 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.699455+0000", 2024-09-21T10:20:19.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.55429031399999995 2024-09-21T10:20:19.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.699463+0000", 2024-09-21T10:20:19.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6349999999999995e-06 2024-09-21T10:20:19.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.834271+0000", 2024-09-21T10:20:19.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1348073839999999 2024-09-21T10:20:19.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:20:19.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.834381+0000", 2024-09-21T10:20:19.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000110729 2024-09-21T10:20:19.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:20:19.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.834478+0000", 2024-09-21T10:20:19.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6676999999999996e-05 2024-09-21T10:20:19.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.875 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.840122+0000", 2024-09-21T10:20:19.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056437630000000004 2024-09-21T10:20:19.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.875 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:20:19.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.840204+0000", 2024-09-21T10:20:19.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2526000000000002e-05 2024-09-21T10:20:19.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.922789+0000", 2024-09-21T10:20:19.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.082584802999999998 2024-09-21T10:20:19.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:20:19.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.922856+0000", 2024-09-21T10:20:19.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6940000000000006e-05 2024-09-21T10:20:19.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:20:19.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.924996+0000", 2024-09-21T10:20:19.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0021398179999999999 2024-09-21T10:20:19.876 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.877 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:20:19.877 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.877 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:126858 7.16 7:6a6893ad:::benchmark_data_smithi098_197834_object10341:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:20:19.877 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:14:56.182825+0000", 2024-09-21T10:20:19.877 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 323.66359683899998, 2024-09-21T10:20:19.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.746662014, 2024-09-21T10:20:19.877 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:20:19.877 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:20:19.877 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:19.877 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:20:19.877 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:20:19.877 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:20:19.877 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 126858 2024-09-21T10:20:19.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.878 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:20:19.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:20:19.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182825+0000", 2024-09-21T10:20:19.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:20:19.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182825+0000", 2024-09-21T10:20:19.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:20:19.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182826+0000", 2024-09-21T10:20:19.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0950000000000001e-06 2024-09-21T10:20:19.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:20:19.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182830+0000", 2024-09-21T10:20:19.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6600000000000003e-06 2024-09-21T10:20:19.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:20:19.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182840+0000", 2024-09-21T10:20:19.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0182e-05 2024-09-21T10:20:19.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:20:19.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182847+0000", 2024-09-21T10:20:19.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6760000000000001e-06 2024-09-21T10:20:19.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182960+0000", 2024-09-21T10:20:19.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000112698 2024-09-21T10:20:19.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182969+0000", 2024-09-21T10:20:19.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8410000000000006e-06 2024-09-21T10:20:19.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.699504+0000", 2024-09-21T10:20:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.516535623 2024-09-21T10:20:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.699510+0000", 2024-09-21T10:20:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5709999999999997e-06 2024-09-21T10:20:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.836487+0000", 2024-09-21T10:20:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.136976695 2024-09-21T10:20:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:20:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.836645+0000", 2024-09-21T10:20:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000158247 2024-09-21T10:20:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:20:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.836788+0000", 2024-09-21T10:20:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014280099999999999 2024-09-21T10:20:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.883 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.883 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.840186+0000", 2024-09-21T10:20:19.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003398098 2024-09-21T10:20:19.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.883 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:20:19.883 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.840212+0000", 2024-09-21T10:20:19.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5944000000000001e-05 2024-09-21T10:20:19.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.883 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.925064+0000", 2024-09-21T10:20:19.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.084852219000000006 2024-09-21T10:20:19.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:20:19.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.925132+0000", 2024-09-21T10:20:19.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7625000000000004e-05 2024-09-21T10:20:19.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:20:19.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.929487+0000", 2024-09-21T10:20:19.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043550389999999998 2024-09-21T10:20:19.885 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.885 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:20:19.885 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.885 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:126859 7.16 7:6e5030ca:::benchmark_data_smithi098_197834_object10342:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:20:19.885 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:14:56.182867+0000", 2024-09-21T10:20:19.885 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 323.66355475500001, 2024-09-21T10:20:19.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7560969719999999, 2024-09-21T10:20:19.885 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:20:19.885 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:20:19.885 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:19.886 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:20:19.886 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:20:19.886 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:20:19.886 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 126859 2024-09-21T10:20:19.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.886 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:20:19.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.886 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:20:19.886 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182867+0000", 2024-09-21T10:20:19.886 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.886 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:20:19.887 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182867+0000", 2024-09-21T10:20:19.887 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.887 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:20:19.887 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182867+0000", 2024-09-21T10:20:19.887 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.63e-07 2024-09-21T10:20:19.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.887 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:20:19.887 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182869+0000", 2024-09-21T10:20:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9080000000000002e-06 2024-09-21T10:20:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:20:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182875+0000", 2024-09-21T10:20:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0900000000000001e-06 2024-09-21T10:20:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:20:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182880+0000", 2024-09-21T10:20:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8879999999999996e-06 2024-09-21T10:20:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182991+0000", 2024-09-21T10:20:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000110759 2024-09-21T10:20:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182997+0000", 2024-09-21T10:20:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0719999999999996e-06 2024-09-21T10:20:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.699575+0000", 2024-09-21T10:20:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.51657803499999999 2024-09-21T10:20:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.699579+0000", 2024-09-21T10:20:19.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7859999999999998e-06 2024-09-21T10:20:19.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.838504+0000", 2024-09-21T10:20:19.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1389254849999999 2024-09-21T10:20:19.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:20:19.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.838616+0000", 2024-09-21T10:20:19.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000111607 2024-09-21T10:20:19.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.891 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:20:19.891 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.838700+0000", 2024-09-21T10:20:19.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4171000000000004e-05 2024-09-21T10:20:19.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.891 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.891 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.840240+0000", 2024-09-21T10:20:19.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001540144 2024-09-21T10:20:19.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.891 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:20:19.891 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.841268+0000", 2024-09-21T10:20:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010283899999999999 2024-09-21T10:20:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.936801+0000", 2024-09-21T10:20:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.095532814999999993 2024-09-21T10:20:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:20:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.936855+0000", 2024-09-21T10:20:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3912000000000002e-05 2024-09-21T10:20:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:20:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.938964+0000", 2024-09-21T10:20:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0021084469999999998 2024-09-21T10:20:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:20:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:183398 7.6 7:677fe744:::benchmark_data_smithi098_197834_object66881:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-09-21T10:20:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:16:03.629420+0000", 2024-09-21T10:20:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 256.21700140399997, 2024-09-21T10:20:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.423753906, 2024-09-21T10:20:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:20:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:20:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:20:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:20:19.894 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:20:19.894 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 183398 2024-09-21T10:20:19.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.894 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:20:19.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.894 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:20:19.894 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.629420+0000", 2024-09-21T10:20:19.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:20:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.629420+0000", 2024-09-21T10:20:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:20:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.629420+0000", 2024-09-21T10:20:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7e-07 2024-09-21T10:20:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:20:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.629423+0000", 2024-09-21T10:20:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1229999999999998e-06 2024-09-21T10:20:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:20:19.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.629429+0000", 2024-09-21T10:20:19.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0909999999999998e-06 2024-09-21T10:20:19.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:20:19.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.629434+0000", 2024-09-21T10:20:19.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8409999999999999e-06 2024-09-21T10:20:19.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.631093+0000", 2024-09-21T10:20:19.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016591920000000001 2024-09-21T10:20:19.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.897 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-09-21T10:20:19.897 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.631098+0000", 2024-09-21T10:20:19.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7230000000000001e-06 2024-09-21T10:20:19.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.897 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.897 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.662137+0000", 2024-09-21T10:20:19.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.031038204999999999 2024-09-21T10:20:19.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.897 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.662145+0000", 2024-09-21T10:20:19.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7700000000000007e-06 2024-09-21T10:20:19.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.898 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628715+0000", 2024-09-21T10:20:19.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.96656951999999996 2024-09-21T10:20:19.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.898 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628725+0000", 2024-09-21T10:20:19.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6139999999999998e-06 2024-09-21T10:20:19.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.899 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.899 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.039822+0000", 2024-09-21T10:20:19.899 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4110971880000001 2024-09-21T10:20:19.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.899 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:20:19.899 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.039986+0000", 2024-09-21T10:20:19.899 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016436000000000001 2024-09-21T10:20:19.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.899 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:20:19.899 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.040543+0000", 2024-09-21T10:20:19.899 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00055650899999999995 2024-09-21T10:20:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.044937+0000", 2024-09-21T10:20:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043948149999999998 2024-09-21T10:20:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:20:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.048484+0000", 2024-09-21T10:20:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035470599999999999 2024-09-21T10:20:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.051634+0000", 2024-09-21T10:20:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031494790000000002 2024-09-21T10:20:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:20:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.051697+0000", 2024-09-21T10:20:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3421000000000001e-05 2024-09-21T10:20:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:20:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.053174+0000", 2024-09-21T10:20:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001476525 2024-09-21T10:20:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:20:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.902 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:183404 7.6 7:60ad48eb:::benchmark_data_smithi098_197834_object66887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-09-21T10:20:19.902 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:16:03.654829+0000", 2024-09-21T10:20:19.902 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 256.19159274399999, 2024-09-21T10:20:19.902 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4193103439999999, 2024-09-21T10:20:19.902 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:20:19.902 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:20:19.902 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:19.902 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:20:19.902 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:20:19.902 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:20:19.902 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 183404 2024-09-21T10:20:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:20:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:20:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.654829+0000", 2024-09-21T10:20:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:20:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.654829+0000", 2024-09-21T10:20:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:20:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.654830+0000", 2024-09-21T10:20:19.904 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.53e-06 2024-09-21T10:20:19.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:20:19.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.654835+0000", 2024-09-21T10:20:19.904 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9919999999999998e-06 2024-09-21T10:20:19.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:20:19.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.654847+0000", 2024-09-21T10:20:19.904 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1412e-05 2024-09-21T10:20:19.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.905 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:20:19.905 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.654853+0000", 2024-09-21T10:20:19.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9079999999999998e-06 2024-09-21T10:20:19.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.905 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.905 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.662168+0000", 2024-09-21T10:20:19.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.007314643 2024-09-21T10:20:19.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.905 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.905 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.662175+0000", 2024-09-21T10:20:19.906 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7109999999999998e-06 2024-09-21T10:20:19.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.906 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.906 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628748+0000", 2024-09-21T10:20:19.906 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.96657321299999999 2024-09-21T10:20:19.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.906 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.906 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628756+0000", 2024-09-21T10:20:19.906 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6520000000000006e-06 2024-09-21T10:20:19.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.907 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.907 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.045075+0000", 2024-09-21T10:20:19.907 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4163190370000001 2024-09-21T10:20:19.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.907 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:20:19.907 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.045262+0000", 2024-09-21T10:20:19.907 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018714900000000001 2024-09-21T10:20:19.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.908 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:20:19.908 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.045819+0000", 2024-09-21T10:20:19.908 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00055741799999999995 2024-09-21T10:20:19.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.908 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.908 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.053244+0000", 2024-09-21T10:20:19.908 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0074251730000000002 2024-09-21T10:20:19.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.908 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:20:19.908 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.053318+0000", 2024-09-21T10:20:19.908 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3857999999999997e-05 2024-09-21T10:20:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.072458+0000", 2024-09-21T10:20:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019139417999999998 2024-09-21T10:20:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:20:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.072519+0000", 2024-09-21T10:20:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1099000000000006e-05 2024-09-21T10:20:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:20:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.074139+0000", 2024-09-21T10:20:19.910 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016201309999999999 2024-09-21T10:20:19.910 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.910 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:20:19.910 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:19.911 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:19.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:19.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.912 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:183421 7.6 7:631fb813:::benchmark_data_smithi098_197834_object66904:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-09-21T10:20:19.912 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:16:03.673007+0000", 2024-09-21T10:20:19.912 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 256.17341413600002, 2024-09-21T10:20:19.912 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4134710660000001, 2024-09-21T10:20:19.912 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:20:19.912 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:20:19.912 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:19.912 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:20:19.912 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:20:19.912 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:20:19.912 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 183421 2024-09-21T10:20:19.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.913 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:20:19.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:20:19.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.673007+0000", 2024-09-21T10:20:19.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:20:19.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.673007+0000", 2024-09-21T10:20:19.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.914 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:20:19.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.673008+0000", 2024-09-21T10:20:19.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3900000000000005e-07 2024-09-21T10:20:19.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.914 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:20:19.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.673010+0000", 2024-09-21T10:20:19.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1519999999999999e-06 2024-09-21T10:20:19.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.914 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:20:19.915 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.673016+0000", 2024-09-21T10:20:19.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2899999999999999e-06 2024-09-21T10:20:19.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.915 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:20:19.915 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.673021+0000", 2024-09-21T10:20:19.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4870000000000004e-06 2024-09-21T10:20:19.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.673092+0000", 2024-09-21T10:20:19.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1606000000000005e-05 2024-09-21T10:20:19.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.673100+0000", 2024-09-21T10:20:19.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9840000000000001e-06 2024-09-21T10:20:19.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628779+0000", 2024-09-21T10:20:19.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.95567843699999999 2024-09-21T10:20:19.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.917 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.917 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628786+0000", 2024-09-21T10:20:19.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1119999999999998e-06 2024-09-21T10:20:19.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.917 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.917 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.048533+0000", 2024-09-21T10:20:19.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.419747353 2024-09-21T10:20:19.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:20:19.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.048699+0000", 2024-09-21T10:20:19.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000165923 2024-09-21T10:20:19.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:20:19.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.049206+0000", 2024-09-21T10:20:19.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00050717899999999996 2024-09-21T10:20:19.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.053304+0000", 2024-09-21T10:20:19.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040978960000000002 2024-09-21T10:20:19.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:20:19.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.053325+0000", 2024-09-21T10:20:19.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0358999999999999e-05 2024-09-21T10:20:19.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.084799+0000", 2024-09-21T10:20:19.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.031474087999999997 2024-09-21T10:20:19.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:20:19.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.084928+0000", 2024-09-21T10:20:19.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012917699999999999 2024-09-21T10:20:19.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:20:19.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.086478+0000", 2024-09-21T10:20:19.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001550484 2024-09-21T10:20:19.920 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.920 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:20:19.920 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.920 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:183441 7.6 7:60d4bafe:::benchmark_data_smithi098_197834_object66924:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-09-21T10:20:19.920 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:16:03.704441+0000", 2024-09-21T10:20:19.920 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 256.14198044099999, 2024-09-21T10:20:19.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3832941619999999, 2024-09-21T10:20:19.921 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:20:19.921 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:20:19.921 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:19.921 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:20:19.921 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:20:19.921 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:20:19.921 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 183441 2024-09-21T10:20:19.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.921 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:20:19.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.921 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:20:19.921 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.704441+0000", 2024-09-21T10:20:19.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:20:19.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.704441+0000", 2024-09-21T10:20:19.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:20:19.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.704442+0000", 2024-09-21T10:20:19.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5900000000000005e-07 2024-09-21T10:20:19.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:20:19.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.704446+0000", 2024-09-21T10:20:19.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2629999999999997e-06 2024-09-21T10:20:19.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:20:19.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.704458+0000", 2024-09-21T10:20:19.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1457999999999999e-05 2024-09-21T10:20:19.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:20:19.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.704464+0000", 2024-09-21T10:20:19.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8600000000000004e-06 2024-09-21T10:20:19.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.705054+0000", 2024-09-21T10:20:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00058917300000000002 2024-09-21T10:20:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.705063+0000", 2024-09-21T10:20:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2699999999999993e-06 2024-09-21T10:20:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628813+0000", 2024-09-21T10:20:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.92374978600000002 2024-09-21T10:20:19.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628820+0000", 2024-09-21T10:20:19.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9519999999999997e-06 2024-09-21T10:20:19.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.053354+0000", 2024-09-21T10:20:19.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.424534186 2024-09-21T10:20:19.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:20:19.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.053483+0000", 2024-09-21T10:20:19.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012914799999999999 2024-09-21T10:20:19.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:20:19.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.054026+0000", 2024-09-21T10:20:19.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00054301499999999999 2024-09-21T10:20:19.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.060726+0000", 2024-09-21T10:20:19.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0066995049999999997 2024-09-21T10:20:19.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:20:19.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.060740+0000", 2024-09-21T10:20:19.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4569000000000001e-05 2024-09-21T10:20:19.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.086532+0000", 2024-09-21T10:20:19.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.025792356999999998 2024-09-21T10:20:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:20:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.086588+0000", 2024-09-21T10:20:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5507000000000002e-05 2024-09-21T10:20:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:20:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.087735+0000", 2024-09-21T10:20:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011471540000000001 2024-09-21T10:20:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:20:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:183484 7.6 7:670a48b1:::benchmark_data_smithi098_197834_object66967:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-09-21T10:20:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:16:03.754909+0000", 2024-09-21T10:20:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 256.09151201399999, 2024-09-21T10:20:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.333857107, 2024-09-21T10:20:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:20:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:20:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:20:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:20:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:20:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 183484 2024-09-21T10:20:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:20:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:20:19.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.754909+0000", 2024-09-21T10:20:19.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:20:19.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.754909+0000", 2024-09-21T10:20:19.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:20:19.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.754911+0000", 2024-09-21T10:20:19.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5400000000000001e-06 2024-09-21T10:20:19.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:20:19.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.754917+0000", 2024-09-21T10:20:19.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7400000000000001e-06 2024-09-21T10:20:19.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:20:19.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.754933+0000", 2024-09-21T10:20:19.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6416999999999999e-05 2024-09-21T10:20:19.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:20:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.754941+0000", 2024-09-21T10:20:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9719999999999992e-06 2024-09-21T10:20:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.755487+0000", 2024-09-21T10:20:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000545474 2024-09-21T10:20:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.755496+0000", 2024-09-21T10:20:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1130000000000007e-06 2024-09-21T10:20:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628840+0000", 2024-09-21T10:20:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.87334427299999995 2024-09-21T10:20:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628844+0000", 2024-09-21T10:20:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0199999999999996e-06 2024-09-21T10:20:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.057341+0000", 2024-09-21T10:20:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.428497514 2024-09-21T10:20:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:20:19.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.057502+0000", 2024-09-21T10:20:19.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016084699999999999 2024-09-21T10:20:19.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:20:19.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.058030+0000", 2024-09-21T10:20:19.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00052808200000000003 2024-09-21T10:20:19.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.072370+0000", 2024-09-21T10:20:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014339849 2024-09-21T10:20:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:20:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.074251+0000", 2024-09-21T10:20:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018803139999999999 2024-09-21T10:20:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.087785+0000", 2024-09-21T10:20:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013534934 2024-09-21T10:20:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:20:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.087833+0000", 2024-09-21T10:20:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7547999999999999e-05 2024-09-21T10:20:19.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:20:19.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.088766+0000", 2024-09-21T10:20:19.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00093347000000000005 2024-09-21T10:20:19.936 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.936 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:20:19.936 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.936 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:183536 7.6 7:60ea16cd:::benchmark_data_smithi098_197834_object67019:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-09-21T10:20:19.937 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:16:03.824423+0000", 2024-09-21T10:20:19.937 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 256.02199831299998, 2024-09-21T10:20:19.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.265316457, 2024-09-21T10:20:19.937 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:20:19.937 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:20:19.937 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:19.937 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:20:19.937 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:20:19.937 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:20:19.937 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 183536 2024-09-21T10:20:19.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.937 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:20:19.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:20:19.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.824423+0000", 2024-09-21T10:20:19.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:20:19.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.824423+0000", 2024-09-21T10:20:19.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:20:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.824425+0000", 2024-09-21T10:20:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.72e-06 2024-09-21T10:20:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:20:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.824430+0000", 2024-09-21T10:20:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3959999999999997e-06 2024-09-21T10:20:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:20:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.824445+0000", 2024-09-21T10:20:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4331999999999999e-05 2024-09-21T10:20:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:20:19.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.824453+0000", 2024-09-21T10:20:19.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0269999999999995e-06 2024-09-21T10:20:19.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.824517+0000", 2024-09-21T10:20:19.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4252999999999996e-05 2024-09-21T10:20:19.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.824527+0000", 2024-09-21T10:20:19.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0684e-05 2024-09-21T10:20:19.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628856+0000", 2024-09-21T10:20:19.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.80432857999999996 2024-09-21T10:20:19.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628860+0000", 2024-09-21T10:20:19.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4589999999999998e-06 2024-09-21T10:20:19.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.060765+0000", 2024-09-21T10:20:19.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.431905904 2024-09-21T10:20:19.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:20:19.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.060929+0000", 2024-09-21T10:20:19.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016366699999999999 2024-09-21T10:20:19.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:20:19.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.061444+0000", 2024-09-21T10:20:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00051481699999999999 2024-09-21T10:20:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.072415+0000", 2024-09-21T10:20:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010971553 2024-09-21T10:20:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:20:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.074259+0000", 2024-09-21T10:20:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001843807 2024-09-21T10:20:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.088812+0000", 2024-09-21T10:20:19.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014552235 2024-09-21T10:20:19.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:20:19.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.088859+0000", 2024-09-21T10:20:19.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7005000000000001e-05 2024-09-21T10:20:19.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:20:19.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.089740+0000", 2024-09-21T10:20:19.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088101800000000001 2024-09-21T10:20:19.944 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.944 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:20:19.945 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.945 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:183551 7.6 7:6005932d:::benchmark_data_smithi098_197834_object67034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-09-21T10:20:19.945 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:16:03.864376+0000", 2024-09-21T10:20:19.945 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 255.982045645, 2024-09-21T10:20:19.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.226529432, 2024-09-21T10:20:19.945 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:20:19.945 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:20:19.945 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:19.945 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:20:19.945 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:20:19.945 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:20:19.946 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 183551 2024-09-21T10:20:19.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.946 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:20:19.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:20:19.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.864376+0000", 2024-09-21T10:20:19.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:20:19.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.864376+0000", 2024-09-21T10:20:19.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:20:19.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.864378+0000", 2024-09-21T10:20:19.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.469e-06 2024-09-21T10:20:19.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:20:19.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.864388+0000", 2024-09-21T10:20:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2990000000000002e-06 2024-09-21T10:20:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:20:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.864404+0000", 2024-09-21T10:20:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6912e-05 2024-09-21T10:20:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:20:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.864416+0000", 2024-09-21T10:20:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1532000000000001e-05 2024-09-21T10:20:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.864509+0000", 2024-09-21T10:20:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3158000000000005e-05 2024-09-21T10:20:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.864521+0000", 2024-09-21T10:20:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1640999999999999e-05 2024-09-21T10:20:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628925+0000", 2024-09-21T10:20:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.76440443999999996 2024-09-21T10:20:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628933+0000", 2024-09-21T10:20:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9619999999999993e-06 2024-09-21T10:20:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.064395+0000", 2024-09-21T10:20:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4354620650000001 2024-09-21T10:20:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:20:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.064538+0000", 2024-09-21T10:20:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014270799999999999 2024-09-21T10:20:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:20:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.065083+0000", 2024-09-21T10:20:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00054525600000000004 2024-09-21T10:20:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.074235+0000", 2024-09-21T10:20:19.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0091516800000000006 2024-09-21T10:20:19.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:20:19.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.074265+0000", 2024-09-21T10:20:19.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0219999999999999e-05 2024-09-21T10:20:19.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.089842+0000", 2024-09-21T10:20:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015576865000000001 2024-09-21T10:20:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:20:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.089937+0000", 2024-09-21T10:20:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5376000000000003e-05 2024-09-21T10:20:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:20:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.090905+0000", 2024-09-21T10:20:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00096784900000000001 2024-09-21T10:20:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:20:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.953 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:66523 8.2 8:405f6d1b:::benchmark_data_smithi098_226587_object66522:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e273)", 2024-09-21T10:20:19.953 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:32.447947+0000", 2024-09-21T10:20:19.953 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 107.398473954, 2024-09-21T10:20:19.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1806429519999999, 2024-09-21T10:20:19.953 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:20:19.953 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:20:19.953 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-21T10:20:19.953 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:20:19.953 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:20:19.953 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:20:19.953 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66523 2024-09-21T10:20:19.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.954 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:20:19.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:20:19.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.447947+0000", 2024-09-21T10:20:19.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:20:19.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.447947+0000", 2024-09-21T10:20:19.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:20:19.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.447949+0000", 2024-09-21T10:20:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.471e-06 2024-09-21T10:20:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:20:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.447973+0000", 2024-09-21T10:20:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4209000000000001e-05 2024-09-21T10:20:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:20:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.447998+0000", 2024-09-21T10:20:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5290999999999999e-05 2024-09-21T10:20:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:20:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.448006+0000", 2024-09-21T10:20:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.729e-06 2024-09-21T10:20:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.448058+0000", 2024-09-21T10:20:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2318999999999998e-05 2024-09-21T10:20:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:20:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.448181+0000", 2024-09-21T10:20:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000122908 2024-09-21T10:20:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:20:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.448276+0000", 2024-09-21T10:20:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4541000000000002e-05 2024-09-21T10:20:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:20:19.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.449994+0000", 2024-09-21T10:20:19.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001718352 2024-09-21T10:20:19.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.450401+0000", 2024-09-21T10:20:19.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00040696500000000002 2024-09-21T10:20:19.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.480193+0000", 2024-09-21T10:20:19.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029792268 2024-09-21T10:20:19.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.480205+0000", 2024-09-21T10:20:19.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1619e-05 2024-09-21T10:20:19.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.470348+0000", 2024-09-21T10:20:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.99014276800000001 2024-09-21T10:20:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.470363+0000", 2024-09-21T10:20:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4776999999999999e-05 2024-09-21T10:20:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.628376+0000", 2024-09-21T10:20:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1580132670000001 2024-09-21T10:20:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:20:19.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.628590+0000", 2024-09-21T10:20:19.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00021446799999999999 2024-09-21T10:20:19.960 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.960 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:20:19.960 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.960 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:66530 8.8 8.76e2c0e8 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-21T10:20:19.960 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:32.460424+0000", 2024-09-21T10:20:19.960 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 107.38599757, 2024-09-21T10:20:19.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.001489912, 2024-09-21T10:20:19.960 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:20:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:20:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-21T10:20:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:20:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:20:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:20:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66530 2024-09-21T10:20:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:20:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:20:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.460424+0000", 2024-09-21T10:20:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:20:19.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.460424+0000", 2024-09-21T10:20:19.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:20:19.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.460424+0000", 2024-09-21T10:20:19.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9399999999999995e-07 2024-09-21T10:20:19.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:20:19.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.460446+0000", 2024-09-21T10:20:19.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1838999999999999e-05 2024-09-21T10:20:19.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:20:19.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.460454+0000", 2024-09-21T10:20:19.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6830000000000008e-06 2024-09-21T10:20:19.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:20:19.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.460460+0000", 2024-09-21T10:20:19.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2190000000000001e-06 2024-09-21T10:20:19.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.461837+0000", 2024-09-21T10:20:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013771300000000001 2024-09-21T10:20:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.461845+0000", 2024-09-21T10:20:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5399999999999998e-06 2024-09-21T10:20:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.461904+0000", 2024-09-21T10:20:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0000593 2024-09-21T10:20:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:20:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.461914+0000", 2024-09-21T10:20:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7070000000000004e-06 2024-09-21T10:20:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:20:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:66534 8.2 8:4b87b1b1:::benchmark_data_smithi098_226587_object66533:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-21T10:20:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:32.473078+0000", 2024-09-21T10:20:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 107.373343414, 2024-09-21T10:20:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1720339659999999, 2024-09-21T10:20:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:20:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:20:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:20:19.966 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:20:19.966 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:20:19.966 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66534 2024-09-21T10:20:19.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.966 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:20:19.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:20:19.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.473078+0000", 2024-09-21T10:20:19.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:20:19.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.473078+0000", 2024-09-21T10:20:19.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:20:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.473080+0000", 2024-09-21T10:20:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9300000000000002e-06 2024-09-21T10:20:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:20:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.473116+0000", 2024-09-21T10:20:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5840000000000002e-05 2024-09-21T10:20:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:20:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.473129+0000", 2024-09-21T10:20:19.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3285000000000001e-05 2024-09-21T10:20:19.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:20:19.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.473138+0000", 2024-09-21T10:20:19.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3589999999999997e-06 2024-09-21T10:20:19.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.480235+0000", 2024-09-21T10:20:19.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070962280000000004 2024-09-21T10:20:19.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.480242+0000", 2024-09-21T10:20:19.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1019999999999999e-06 2024-09-21T10:20:19.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.470441+0000", 2024-09-21T10:20:19.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.990198837 2024-09-21T10:20:19.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.470586+0000", 2024-09-21T10:20:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014546599999999999 2024-09-21T10:20:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.631513+0000", 2024-09-21T10:20:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1609273769999999 2024-09-21T10:20:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:20:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.631704+0000", 2024-09-21T10:20:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000190335 2024-09-21T10:20:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:20:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.631878+0000", 2024-09-21T10:20:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017390299999999999 2024-09-21T10:20:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.637866+0000", 2024-09-21T10:20:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0059887270000000001 2024-09-21T10:20:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:20:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.641355+0000", 2024-09-21T10:20:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034886909999999999 2024-09-21T10:20:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.641661+0000", 2024-09-21T10:20:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000305686 2024-09-21T10:20:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:20:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.641852+0000", 2024-09-21T10:20:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000191027 2024-09-21T10:20:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:20:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.645112+0000", 2024-09-21T10:20:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032601729999999999 2024-09-21T10:20:19.974 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.974 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:20:19.974 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.974 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:66541 8.2 8:4a411c2e:::benchmark_data_smithi098_226587_object66540:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-21T10:20:19.974 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:32.508561+0000", 2024-09-21T10:20:19.974 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 107.337860685, 2024-09-21T10:20:19.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1401821930000002, 2024-09-21T10:20:19.974 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:20:19.974 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:20:19.974 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:20:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:20:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:20:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66541 2024-09-21T10:20:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:20:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:20:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.508561+0000", 2024-09-21T10:20:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:20:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.508561+0000", 2024-09-21T10:20:19.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:20:19.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.508563+0000", 2024-09-21T10:20:19.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8029999999999999e-06 2024-09-21T10:20:19.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:20:19.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.508599+0000", 2024-09-21T10:20:19.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6341999999999999e-05 2024-09-21T10:20:19.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:20:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.508610+0000", 2024-09-21T10:20:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1636e-05 2024-09-21T10:20:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:20:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.508618+0000", 2024-09-21T10:20:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3429999999999998e-06 2024-09-21T10:20:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.509182+0000", 2024-09-21T10:20:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00056403200000000001 2024-09-21T10:20:19.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.509192+0000", 2024-09-21T10:20:19.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0623e-05 2024-09-21T10:20:19.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.470614+0000", 2024-09-21T10:20:19.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.96142103000000001 2024-09-21T10:20:19.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.470621+0000", 2024-09-21T10:20:19.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9540000000000004e-06 2024-09-21T10:20:19.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.634388+0000", 2024-09-21T10:20:19.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1637665779999999 2024-09-21T10:20:19.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:20:19.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.634564+0000", 2024-09-21T10:20:19.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000175501 2024-09-21T10:20:19.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:20:19.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.634748+0000", 2024-09-21T10:20:19.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018397899999999999 2024-09-21T10:20:19.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.637996+0000", 2024-09-21T10:20:19.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032480439999999998 2024-09-21T10:20:19.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:20:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.645271+0000", 2024-09-21T10:20:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0072753760000000001 2024-09-21T10:20:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.645512+0000", 2024-09-21T10:20:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00024140399999999999 2024-09-21T10:20:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:20:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.645581+0000", 2024-09-21T10:20:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8282999999999997e-05 2024-09-21T10:20:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:20:19.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.648743+0000", 2024-09-21T10:20:19.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003162265 2024-09-21T10:20:19.982 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.982 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:20:19.982 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.982 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:66548 8.8 8.6ed96238 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-21T10:20:19.982 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:32.518774+0000", 2024-09-21T10:20:19.982 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 107.327647271, 2024-09-21T10:20:19.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.94317137299999998, 2024-09-21T10:20:19.982 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:20:19.982 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:20:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-21T10:20:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:20:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:20:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:20:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66548 2024-09-21T10:20:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:20:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:20:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.518774+0000", 2024-09-21T10:20:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:20:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.518774+0000", 2024-09-21T10:20:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:20:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.518775+0000", 2024-09-21T10:20:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8599999999999996e-07 2024-09-21T10:20:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:20:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.518794+0000", 2024-09-21T10:20:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9134999999999999e-05 2024-09-21T10:20:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:20:19.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.518806+0000", 2024-09-21T10:20:19.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1972e-05 2024-09-21T10:20:19.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:20:19.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.518811+0000", 2024-09-21T10:20:19.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2249999999999999e-06 2024-09-21T10:20:19.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.518874+0000", 2024-09-21T10:20:19.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2360999999999995e-05 2024-09-21T10:20:19.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.518879+0000", 2024-09-21T10:20:19.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.011e-06 2024-09-21T10:20:19.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.461938+0000", 2024-09-21T10:20:19.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.94305926600000001 2024-09-21T10:20:19.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:20:19.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.461945+0000", 2024-09-21T10:20:19.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4170000000000003e-06 2024-09-21T10:20:19.987 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.987 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:20:19.987 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.987 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:66553 8.8 8.329526d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-21T10:20:19.987 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:32.523208+0000", 2024-09-21T10:20:19.987 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 107.323213319, 2024-09-21T10:20:19.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.93876435599999997, 2024-09-21T10:20:19.988 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:20:19.988 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:20:19.988 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-21T10:20:19.988 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:20:19.988 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:20:19.988 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:20:19.988 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66553 2024-09-21T10:20:19.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.988 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:20:19.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:20:19.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.523208+0000", 2024-09-21T10:20:19.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:20:19.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.523208+0000", 2024-09-21T10:20:19.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:20:19.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.523209+0000", 2024-09-21T10:20:19.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3790000000000001e-06 2024-09-21T10:20:19.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:20:19.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.523227+0000", 2024-09-21T10:20:19.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7717999999999999e-05 2024-09-21T10:20:19.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:20:19.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.523235+0000", 2024-09-21T10:20:19.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6930000000000007e-06 2024-09-21T10:20:19.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:20:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.523239+0000", 2024-09-21T10:20:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4839999999999998e-06 2024-09-21T10:20:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.523295+0000", 2024-09-21T10:20:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5575999999999998e-05 2024-09-21T10:20:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:20:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.523304+0000", 2024-09-21T10:20:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7390000000000005e-06 2024-09-21T10:20:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.461965+0000", 2024-09-21T10:20:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.93866182200000003 2024-09-21T10:20:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:20:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.461972+0000", 2024-09-21T10:20:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9450000000000004e-06 2024-09-21T10:20:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:20:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:95750 8.c 8:3bd51e92:::benchmark_data_smithi098_226587_object95749:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-21T10:20:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:19:14.919667+0000", 2024-09-21T10:20:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 64.926754277000001, 2024-09-21T10:20:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.31776388300000002, 2024-09-21T10:20:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:20:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:20:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:20:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:20:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:20:19.994 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 95750 2024-09-21T10:20:19.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.994 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:20:19.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:20:19.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.919667+0000", 2024-09-21T10:20:19.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:20:19.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.919667+0000", 2024-09-21T10:20:19.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:19.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:20:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.919669+0000", 2024-09-21T10:20:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.415e-06 2024-09-21T10:20:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:20:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.919688+0000", 2024-09-21T10:20:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9741e-05 2024-09-21T10:20:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:20:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.919698+0000", 2024-09-21T10:20:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9140000000000003e-06 2024-09-21T10:20:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:20:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.919740+0000", 2024-09-21T10:20:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2076000000000002e-05 2024-09-21T10:20:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.919900+0000", 2024-09-21T10:20:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015946299999999999 2024-09-21T10:20:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:20:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.920077+0000", 2024-09-21T10:20:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017700600000000001 2024-09-21T10:20:19.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,7", 2024-09-21T10:20:19.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.920288+0000", 2024-09-21T10:20:19.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00021087499999999999 2024-09-21T10:20:19.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.216770+0000", 2024-09-21T10:20:19.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.29648236 2024-09-21T10:20:19.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:19.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.217100+0000", 2024-09-21T10:20:19.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00033039300000000001 2024-09-21T10:20:19.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:20:19.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.233469+0000", 2024-09-21T10:20:19.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016368671000000001 2024-09-21T10:20:19.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:20:19.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.233626+0000", 2024-09-21T10:20:19.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000157324 2024-09-21T10:20:19.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:19.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:19.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:20:19.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.237431+0000", 2024-09-21T10:20:19.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038046450000000002 2024-09-21T10:20:19.999 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.999 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:20:19.999 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:19.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:20.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:20.000 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:95757 8.6 8:6a63278c:::benchmark_data_smithi098_226587_object95756:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-21T10:20:20.000 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:19:14.931480+0000", 2024-09-21T10:20:20.000 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 64.914941640999999, 2024-09-21T10:20:20.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.30778484299999997, 2024-09-21T10:20:20.000 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:20:20.000 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:20:20.000 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:20.000 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:20:20.000 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:20:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:20:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 95757 2024-09-21T10:20:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:20:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:20:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.931480+0000", 2024-09-21T10:20:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:20:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.931480+0000", 2024-09-21T10:20:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:20:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:20:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.931481+0000", 2024-09-21T10:20:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4160000000000001e-06 2024-09-21T10:20:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:20:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.931508+0000", 2024-09-21T10:20:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6457000000000001e-05 2024-09-21T10:20:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:20:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.931516+0000", 2024-09-21T10:20:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5269999999999998e-06 2024-09-21T10:20:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:20.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:20.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:20:20.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.931522+0000", 2024-09-21T10:20:20.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4480000000000002e-06 2024-09-21T10:20:20.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:20.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:20.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:20:20.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.216781+0000", 2024-09-21T10:20:20.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.285259186 2024-09-21T10:20:20.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:20.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:20.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:20:20.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.216950+0000", 2024-09-21T10:20:20.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016970900000000001 2024-09-21T10:20:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:20:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.217100+0000", 2024-09-21T10:20:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000149171 2024-09-21T10:20:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.226029+0000", 2024-09-21T10:20:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0089289339999999995 2024-09-21T10:20:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.226549+0000", 2024-09-21T10:20:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00052087499999999996 2024-09-21T10:20:20.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:20.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:20.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:20:20.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.239188+0000", 2024-09-21T10:20:20.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012638361000000001 2024-09-21T10:20:20.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:20.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:20.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:20:20.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.239235+0000", 2024-09-21T10:20:20.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7095000000000002e-05 2024-09-21T10:20:20.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:20:20.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:20:20.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:20:20.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.239265+0000", 2024-09-21T10:20:20.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9663999999999999e-05 2024-09-21T10:20:20.006 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:20.006 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:20:20.006 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:20.006 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:20:20.006 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:20:20.006 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:20:20.012 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:20.048 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:20.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:20.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:20.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:20.185 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:20.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:20.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:20.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:20.326 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:20.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:20.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:20.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:20.459 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:20.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:20.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:20.561 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:20.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:20 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:20:20.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:20 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:20:20.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:20 smithi098 ceph-mon[97580]: osdmap e288: 8 total, 8 up, 6 in 2024-09-21T10:20:20.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:20 smithi098 ceph-mon[97580]: pgmap v920: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 949 op/s; 19443/187743 objects degraded (10.356%); 5536/187743 objects misplaced (2.949%); 1.4 MiB/s, 21 objects/s recovering 2024-09-21T10:20:20.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:20 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:20:20.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:20 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:20:20.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:20 smithi098 ceph-mon[100914]: osdmap e288: 8 total, 8 up, 6 in 2024-09-21T10:20:20.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:20 smithi098 ceph-mon[100914]: pgmap v920: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 949 op/s; 19443/187743 objects degraded (10.356%); 5536/187743 objects misplaced (2.949%); 1.4 MiB/s, 21 objects/s recovering 2024-09-21T10:20:20.617 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:20.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:20.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:20.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:20 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:20:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:20 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:20:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:20 smithi196 ceph-mon[84784]: osdmap e288: 8 total, 8 up, 6 in 2024-09-21T10:20:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:20 smithi196 ceph-mon[84784]: pgmap v920: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 949 op/s; 19443/187743 objects degraded (10.356%); 5536/187743 objects misplaced (2.949%); 1.4 MiB/s, 21 objects/s recovering 2024-09-21T10:20:20.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:20:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:20:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:20:20.719 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:20.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:20.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:20.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:20.750 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:20:20.750 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:20:20.789 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:20:20.789 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:20:20.827 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:20:20.827 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:20:20.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:20.860 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:20:20.860 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:20:20.893 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:20:20.893 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:20:20.894 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:20.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:20.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:20.926 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:20:20.926 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:20:20.927 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:20:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:20:20] "GET /metrics HTTP/1.1" 200 36546 "" "Prometheus/2.51.0" 2024-09-21T10:20:20.957 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:20:20.958 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:20:20.990 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:20:20.990 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:20:20.997 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:21.037 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:21.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:21.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:21.139 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:21.169 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:21.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:21.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:21.271 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:21.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:21.304 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:21.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:21.405 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:21.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:21.441 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:21.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:21.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:21.574 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:21.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:21.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:21.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:21 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2468269466' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:20:21.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:21 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/187743 objects degraded (10.356%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:21.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:21 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2468269466' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:20:21.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:21 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/187743 objects degraded (10.356%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:21.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:21 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2468269466' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:20:21.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:21 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/187743 objects degraded (10.356%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:21.676 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:21.712 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:21.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:21.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:21.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:21.848 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:21.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:21.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:21.904 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:20:21.950 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:21.990 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:21.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:21.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:22.092 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:22.121 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:22.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:22.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:22.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:22.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:22.271 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:22.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:22.373 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:22.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:22.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:22.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:22.511 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:22.548 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:22.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:22.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:22.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:22 smithi196 ceph-mon[84784]: pgmap v921: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 673 op/s; 19443/186201 objects degraded (10.442%); 5376/186201 objects misplaced (2.887%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:20:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:22.650 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:22.676 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:22.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:22.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:22.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:22 smithi098 ceph-mon[97580]: pgmap v921: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 673 op/s; 19443/186201 objects degraded (10.442%); 5376/186201 objects misplaced (2.887%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:20:22.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:22.756 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:22 smithi098 ceph-mon[100914]: pgmap v921: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 673 op/s; 19443/186201 objects degraded (10.442%); 5376/186201 objects misplaced (2.887%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:20:22.756 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:22.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:22.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:22.818 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:22.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:22.919 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:22.948 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:22.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:22.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:23.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:23.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:23.088 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:23.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:23.189 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:23.225 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:23.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:23.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:23.326 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:23.367 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:23.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:23.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:23.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:23.500 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:23.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:23.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:23.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:23.641 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:23.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:23.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:23.743 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:23.778 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:23.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:23.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:23.781 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:23.880 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:23.916 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:23.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:23.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:24.017 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:24.049 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:24.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:24.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:24.151 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:24.202 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:24.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:24.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:24.303 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:24.359 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:24.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:24.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:24.461 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:24.470 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:20:24.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:24.502 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:24.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:24.571 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:24.603 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:24.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:24.674 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:24.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:24.774 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:24.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:24.803 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:24.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:24.904 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:24.952 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:24.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:24.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:25.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:25.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:24 smithi098 ceph-mon[97580]: pgmap v922: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 868 op/s; 19443/177828 objects degraded (10.934%); 5376/177828 objects misplaced (3.023%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:20:25.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:24 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:20:25.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:24 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:20:25.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:24 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:20:25.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:24 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:20:25.100 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:24 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:20:25.100 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:24 smithi098 ceph-mon[100914]: pgmap v922: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 868 op/s; 19443/177828 objects degraded (10.934%); 5376/177828 objects misplaced (3.023%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:20:25.100 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:24 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:20:25.100 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:24 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:20:25.100 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:24 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:20:25.100 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:24 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:20:25.100 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:24 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:20:25.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:25.108 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:25.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:25.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:20:25.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:24 smithi196 ceph-mon[84784]: pgmap v922: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 868 op/s; 19443/177828 objects degraded (10.934%); 5376/177828 objects misplaced (3.023%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:20:25.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:24 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:20:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:24 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:20:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:24 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:20:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:24 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:20:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:24 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:20:25.209 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:25.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:25.251 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:25.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:25.352 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:25.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:25.401 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:25.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:25.502 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:25.568 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:25.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:25.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:25.670 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:25.701 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:25.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:25.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:25.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:25.833 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:25.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:25.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:25.935 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:25.966 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:25.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:25.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:26.067 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:26.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:26.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:26.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:26.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:25 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/177828 objects degraded (10.934%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:26.139 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:25 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/177828 objects degraded (10.934%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:26.140 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:25 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/177828 objects degraded (10.934%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:26.202 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:26.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:26.241 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:26.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:26.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:26.378 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:26.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:26.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:26.480 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:26.508 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:26.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:26.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:26.610 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:26.657 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:26.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:26.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:26.759 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:26.792 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:26.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:26.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:26.893 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:26.931 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:26.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:26.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:27.033 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:27.065 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:27.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:27.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:27.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:26 smithi196 ceph-mon[84784]: pgmap v923: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 825 op/s; 19443/173919 objects degraded (11.179%); 5174/173919 objects misplaced (2.975%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T10:20:27.167 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:27.206 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:27.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:27.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:27.308 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:27.334 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:27.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:27.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:27.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:26 smithi098 ceph-mon[97580]: pgmap v923: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 825 op/s; 19443/173919 objects degraded (11.179%); 5174/173919 objects misplaced (2.975%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T10:20:27.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:26 smithi098 ceph-mon[100914]: pgmap v923: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 825 op/s; 19443/173919 objects degraded (11.179%); 5174/173919 objects misplaced (2.975%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T10:20:27.435 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:27.471 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:27.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:27.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:27.572 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:27.597 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:27.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:27.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:27.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:27.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:27.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:27.833 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:27.864 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:27.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:27.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:27.966 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:28.001 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:28.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:28.103 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:28.123 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:28.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:28.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:28.225 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:28.257 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:28.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:28.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:28.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:28.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:28.394 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:28.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:28.494 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:28.533 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:28.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:28.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:28.635 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:28.666 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:28.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:28.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:28.767 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:28.789 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:28.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:28.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:28.890 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:28.918 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:28.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:28.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:29.020 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:29.050 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:29.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:29.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:29.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:28 smithi196 ceph-mon[84784]: pgmap v924: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.4 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 825 op/s; 19443/169281 objects degraded (11.486%); 5174/169281 objects misplaced (3.056%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:20:29.152 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:29.187 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:29.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:29.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:29.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:29.310 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:29.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:29.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:29.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:28 smithi098 ceph-mon[97580]: pgmap v924: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.4 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 825 op/s; 19443/169281 objects degraded (11.486%); 5174/169281 objects misplaced (3.056%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:20:29.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:28 smithi098 ceph-mon[100914]: pgmap v924: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.4 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 825 op/s; 19443/169281 objects degraded (11.486%); 5174/169281 objects misplaced (3.056%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:20:29.412 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:29.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:29.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:29.546 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:29.581 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:29.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:29.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:29.683 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:29.704 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:29.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:29.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:29.806 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:29.838 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:29.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:29.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:29.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:29.960 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:29.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:29.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:30.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:30.097 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:30.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:30.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:30.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:30.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:30.238 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:30.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:30.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:30.375 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:30.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:30.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:30.476 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:30.507 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:30.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:30.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:30.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:30.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:20:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:20:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:20:30.643 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:30.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:30.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:30.739 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:30.745 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:30.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:30.795 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:30.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:30.896 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:30.917 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:30.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:30.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:30.946 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:20:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:20:30] "GET /metrics HTTP/1.1" 200 36550 "" "Prometheus/2.51.0" 2024-09-21T10:20:30.946 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:30 smithi098 ceph-mon[97580]: pgmap v925: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.3 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 811 op/s; 19443/163161 objects degraded (11.916%); 5174/163161 objects misplaced (3.171%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:20:30.947 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:30 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/749596939' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:20:30.947 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:30 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:20:30.947 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:30 smithi098 ceph-mon[100914]: pgmap v925: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.3 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 811 op/s; 19443/163161 objects degraded (11.916%); 5174/163161 objects misplaced (3.171%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:20:30.947 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:30 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/749596939' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:20:30.947 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:30 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:20:30.947 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:30 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/163161 objects degraded (11.916%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:30.955 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T10:20:31.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:31.059 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:31.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:31.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:31.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:30 smithi196 ceph-mon[84784]: pgmap v925: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.3 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 811 op/s; 19443/163161 objects degraded (11.916%); 5174/163161 objects misplaced (3.171%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:20:31.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:30 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/749596939' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:20:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:30 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:20:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:30 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/163161 objects degraded (11.916%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:31.161 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:31.192 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:31.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:31.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:31.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:31.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:31.326 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:31.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:31.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:30 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/163161 objects degraded (11.916%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:31.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:31.447 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:31.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:31.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:31.549 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:31.586 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:31.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:31.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:31.687 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:31.719 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:31.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:31.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:31.821 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:31.843 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:31.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:31.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:31.945 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:31.966 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:31.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:31.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:32.068 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:32.100 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:32.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:32.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:32.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:32.236 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:32.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:32.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:32.338 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:32.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:31 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:20:32.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:31 smithi098 ceph-mon[97580]: osdmap e289: 8 total, 8 up, 6 in 2024-09-21T10:20:32.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:31 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:20:32.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:31 smithi098 ceph-mon[100914]: osdmap e289: 8 total, 8 up, 6 in 2024-09-21T10:20:32.359 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:32.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:32.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:32.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:31 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:20:32.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:31 smithi196 ceph-mon[84784]: osdmap e289: 8 total, 8 up, 6 in 2024-09-21T10:20:32.461 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:32.497 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:32.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:32.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:32.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:32.629 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:32.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:32.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:32.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:32.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:32.770 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:32.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:32.871 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:32.914 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:32.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:32.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:33.016 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:33.040 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:33.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:33.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:33.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:33.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:33.178 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:33.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:33.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:33.296 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:32 smithi098 ceph-mon[100914]: pgmap v927: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.3 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 820 op/s; 19443/161565 objects degraded (12.034%); 4969/161565 objects misplaced (3.076%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:20:33.299 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:32 smithi098 ceph-mon[97580]: pgmap v927: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.3 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 820 op/s; 19443/161565 objects degraded (12.034%); 4969/161565 objects misplaced (3.076%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:20:33.323 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:33.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:33.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:33.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:32 smithi196 ceph-mon[84784]: pgmap v927: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.3 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 820 op/s; 19443/161565 objects degraded (12.034%); 4969/161565 objects misplaced (3.076%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:20:33.425 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:33.452 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:33.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:33.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:33.554 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:33.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:33.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:33.693 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:33.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:33.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:33.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:33.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:33.857 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:33.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:33.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:33.958 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:33.991 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:33.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:33.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:34.093 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:34.129 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:34.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:34.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:34.231 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:34.263 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:34.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:34.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:34.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:34.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:34.408 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:34.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:34.427 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:20:34.427 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:20:34.509 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:34.539 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:34.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:34.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:34.641 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:34.675 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:34.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:34.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:34.777 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:34.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:34.813 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:34.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:34.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:34.947 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:34.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:34.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:35.048 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:35.083 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:35.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:35.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:35.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:34 smithi098 ceph-mon[97580]: pgmap v928: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 812 op/s; 19443/153453 objects degraded (12.670%); 4969/153453 objects misplaced (3.238%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:20:35.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:34 smithi098 ceph-mon[100914]: pgmap v928: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 812 op/s; 19443/153453 objects degraded (12.670%); 4969/153453 objects misplaced (3.238%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:20:35.185 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:35.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:35.215 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:35.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:35.301 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T10:20:35.315 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:35.350 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:35.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:35.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:35.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:34 smithi196 ceph-mon[84784]: pgmap v928: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 812 op/s; 19443/153453 objects degraded (12.670%); 4969/153453 objects misplaced (3.238%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:20:35.452 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:35.491 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:35.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:35.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:35.592 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:35.623 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:35.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:35.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:35.725 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:35.764 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:35.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:35.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:35.866 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:35.894 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:35.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:35.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:35.920 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:20:35.995 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:36.022 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:36.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:36.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:36.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:36.165 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:36.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:36.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:36.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:35 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/153453 objects degraded (12.670%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:36.247 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:35 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/153453 objects degraded (12.670%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:36.266 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:36.292 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:36.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:36.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:36.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:35 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/153453 objects degraded (12.670%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:36.394 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:36.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:36.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:36.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:36.578 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:36.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:36.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:36.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:36.721 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:36.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:36.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:36.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:36.859 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:36.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:36.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:36.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:37.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:37.003 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:37.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:37.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:37.141 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:37.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:37.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:37.242 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:37.284 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:37.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:37.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:37.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:36 smithi098 ceph-mon[100914]: pgmap v929: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 795 op/s; 19443/150057 objects degraded (12.957%); 4722/150057 objects misplaced (3.147%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T10:20:37.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:36 smithi098 ceph-mon[97580]: pgmap v929: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 795 op/s; 19443/150057 objects degraded (12.957%); 4722/150057 objects misplaced (3.147%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T10:20:37.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:37.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:36 smithi196 ceph-mon[84784]: pgmap v929: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 795 op/s; 19443/150057 objects degraded (12.957%); 4722/150057 objects misplaced (3.147%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T10:20:37.415 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:37.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:37.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:37.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:37.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:37.554 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:37.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:37.655 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:37.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:37.699 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:37.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:37.800 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:37.827 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:37.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:37.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:37.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:37.962 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:37.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:37.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:38.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:38.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:38.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:38.210 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:38.242 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:38.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:38.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:38.343 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:38.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:38.371 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:38.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:38.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:38.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:38.473 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:38.509 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:38.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:38.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:38.610 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:38.682 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:38.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:38.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:38.784 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:38.797 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:38.835 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:38.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:38.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:38.937 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:38.958 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:38.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:38.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:39.060 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:39.081 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:39.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:39.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:39.183 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:39.213 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:39.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:39.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:39.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:38 smithi196 ceph-mon[84784]: pgmap v930: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 789 op/s; 19443/145578 objects degraded (13.356%); 4722/145578 objects misplaced (3.244%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T10:20:39.316 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:39.337 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:39.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:39.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:39.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:38 smithi098 ceph-mon[97580]: pgmap v930: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 789 op/s; 19443/145578 objects degraded (13.356%); 4722/145578 objects misplaced (3.244%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T10:20:39.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:38 smithi098 ceph-mon[100914]: pgmap v930: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 789 op/s; 19443/145578 objects degraded (13.356%); 4722/145578 objects misplaced (3.244%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T10:20:39.439 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:39.472 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:39.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:39.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:39.574 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:39.599 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:39.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:39.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:39.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:39.743 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:39.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:39.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:39.844 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:39.886 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:39.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:39.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:39.989 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:40.016 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:40.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:40.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:40.118 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:40.147 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:40.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:40.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:40.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:40.290 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:40.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:40.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:40.391 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:40.420 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:40.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:40.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:40.490 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:20:40.490 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:20:40.490 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:20:40.490 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:20:40.490 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:20:40.490 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:20:40.491 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:20:40.491 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:20:40.491 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:20:40.491 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:20:40.491 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:20:40.491 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:20:40.497 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:20:40.497 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:20:40.502 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:40.522 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:40.525 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:20:40.525 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:20:40.550 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:40.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:40.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:40.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:20:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:20:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:20:40.652 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:40.683 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:40.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:40.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:40.785 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:40.814 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:40.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:40.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:40.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:40.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:40.960 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:40.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:40.996 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:20:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:20:40] "GET /metrics HTTP/1.1" 200 36547 "" "Prometheus/2.51.0" 2024-09-21T10:20:41.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:41.092 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:41.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:41.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:41.194 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:41.227 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:41.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:41.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:41.329 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:41.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:40 smithi098 ceph-mon[97580]: pgmap v931: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 799 op/s; 19443/139161 objects degraded (13.972%); 4722/139161 objects misplaced (3.393%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T10:20:41.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:40 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1375844598' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:20:41.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:40 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/139161 objects degraded (13.972%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:41.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:40 smithi098 ceph-mon[100914]: pgmap v931: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 799 op/s; 19443/139161 objects degraded (13.972%); 4722/139161 objects misplaced (3.393%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T10:20:41.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:40 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1375844598' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:20:41.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:40 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/139161 objects degraded (13.972%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:41.359 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:41.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:41.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:40 smithi196 ceph-mon[84784]: pgmap v931: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 799 op/s; 19443/139161 objects degraded (13.972%); 4722/139161 objects misplaced (3.393%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T10:20:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:40 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1375844598' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:20:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:40 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/139161 objects degraded (13.972%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:41.461 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:41.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:41.491 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:41.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:41.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:41.625 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:41.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:41.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:41.727 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:41.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:41.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:41.861 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:41.899 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:41.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:41.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:42.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:42.014 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:20:42.040 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:42.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:42.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:42.141 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:42.172 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:42.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:42.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:42.271 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:42 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/135726230' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:20:42.271 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:42 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:20:42.271 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:42 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/135726230' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:20:42.271 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:42 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:20:42.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:42.307 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:42.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:42.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:42.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:42 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/135726230' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:20:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:42 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:20:42.409 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:42.433 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:42.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:42.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:42.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:42.578 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:42.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:42.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:42.680 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:42.717 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:42.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:42.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:42.821 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:42.857 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:42.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:42.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:42.959 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:42.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:42.993 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:42.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:43.093 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:43.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:43.128 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:43.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:43.229 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:43.259 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:43.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:43.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:43.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:43 smithi098 ceph-mon[97580]: pgmap v932: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 762 op/s; 19443/137640 objects degraded (14.126%); 4476/137640 objects misplaced (3.252%); 2.9 MiB/s, 47 objects/s recovering 2024-09-21T10:20:43.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:43 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:20:43.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:43 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:20:43.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:43 smithi098 ceph-mon[97580]: osdmap e290: 8 total, 8 up, 6 in 2024-09-21T10:20:43.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:43 smithi098 ceph-mon[100914]: pgmap v932: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 762 op/s; 19443/137640 objects degraded (14.126%); 4476/137640 objects misplaced (3.252%); 2.9 MiB/s, 47 objects/s recovering 2024-09-21T10:20:43.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:43 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:20:43.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:43 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:20:43.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:43 smithi098 ceph-mon[100914]: osdmap e290: 8 total, 8 up, 6 in 2024-09-21T10:20:43.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:43.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:43 smithi196 ceph-mon[84784]: pgmap v932: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 762 op/s; 19443/137640 objects degraded (14.126%); 4476/137640 objects misplaced (3.252%); 2.9 MiB/s, 47 objects/s recovering 2024-09-21T10:20:43.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:43 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:20:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:43 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:20:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:43 smithi196 ceph-mon[84784]: osdmap e290: 8 total, 8 up, 6 in 2024-09-21T10:20:43.392 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:20:43.399 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:43.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:43.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:43.500 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:43.536 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:43.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:43.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:43.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:43.670 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:43.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:43.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:43.772 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:43.800 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:43.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:43.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:43.901 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:43.949 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:43.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:43.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:44.050 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:44.081 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:44.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:44.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:44.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:44.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:44.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:44.324 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:44.356 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:44.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:44.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:44.458 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:44.485 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:44.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:44.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:44.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:44.620 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:44.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:44.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:44.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:44.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:44.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:44.855 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:44.887 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:44.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:44.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:44.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:45.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:45.036 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:45.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:45.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:45 smithi098 ceph-mon[97580]: pgmap v934: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 802 op/s; 19443/129375 objects degraded (15.028%); 4476/129375 objects misplaced (3.460%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T10:20:45.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:45 smithi098 ceph-mon[100914]: pgmap v934: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 802 op/s; 19443/129375 objects degraded (15.028%); 4476/129375 objects misplaced (3.460%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T10:20:45.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:45.165 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:45.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:45.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:45.267 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:45.293 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:45.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:45.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:45.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:45 smithi196 ceph-mon[84784]: pgmap v934: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 802 op/s; 19443/129375 objects degraded (15.028%); 4476/129375 objects misplaced (3.460%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T10:20:45.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:45.404 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:20:45.404 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:20:45.404 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-21T10:20:45.404 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-21T10:20:45.404 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:20:45.423 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:45.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:45.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:45.524 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:45.546 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:45.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:45.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:45.648 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:45.669 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:45.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:45.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:45.771 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:45.793 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:45.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:45.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:45.895 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:45.915 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:45.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:45.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:46.017 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:46.038 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:46.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:46.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:46.139 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:46.168 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:46.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:46.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:46.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:46 smithi098 ceph-mon[97580]: osdmap e291: 8 total, 8 up, 6 in 2024-09-21T10:20:46.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:46 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19443/129375 objects degraded (15.028%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:46.260 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:46 smithi098 ceph-mon[100914]: osdmap e291: 8 total, 8 up, 6 in 2024-09-21T10:20:46.260 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:46 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19443/129375 objects degraded (15.028%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:46.269 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:46.305 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:46.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:46.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:46.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:46 smithi196 ceph-mon[84784]: osdmap e291: 8 total, 8 up, 6 in 2024-09-21T10:20:46.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:46 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19443/129375 objects degraded (15.028%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:46.407 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:46.438 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:46.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:46.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:46.540 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:46.579 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:46.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:46.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:46.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:46.709 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:46.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:46.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:46.753 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:20:46.811 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:46.840 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:20:46.844 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:46.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:46.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:46.945 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:46.971 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:46.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:46.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:47.072 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:47.100 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:47.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:47.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:47.202 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:47.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:47.237 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:47.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:47.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:47 smithi098 ceph-mon[97580]: pgmap v936: 89 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 830 op/s; 19443/125625 objects degraded (15.477%); 4219/125625 objects misplaced (3.358%); 3.9 MiB/s, 62 objects/s recovering 2024-09-21T10:20:47.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:47 smithi098 ceph-mon[97580]: osdmap e292: 8 total, 8 up, 6 in 2024-09-21T10:20:47.330 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:47 smithi098 ceph-mon[100914]: pgmap v936: 89 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 830 op/s; 19443/125625 objects degraded (15.477%); 4219/125625 objects misplaced (3.358%); 3.9 MiB/s, 62 objects/s recovering 2024-09-21T10:20:47.330 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:47 smithi098 ceph-mon[100914]: osdmap e292: 8 total, 8 up, 6 in 2024-09-21T10:20:47.338 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:47.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:47.369 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:47.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:47.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:47 smithi196 ceph-mon[84784]: pgmap v936: 89 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 830 op/s; 19443/125625 objects degraded (15.477%); 4219/125625 objects misplaced (3.358%); 3.9 MiB/s, 62 objects/s recovering 2024-09-21T10:20:47.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:47 smithi196 ceph-mon[84784]: osdmap e292: 8 total, 8 up, 6 in 2024-09-21T10:20:47.470 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:47.523 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:47.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:47.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:47.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:47.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:47.647 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:47.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:47.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:47.784 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:47.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:47.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:47.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:47.924 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:47.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:47.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:48.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T10:20:48.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:48.070 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:48.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:48.171 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:48.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:48.207 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:48.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:48.308 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:48.343 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:48.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:48.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:48.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:48 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3894044292' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:20:48.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:48 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3894044292' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:20:48.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:48 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3894044292' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:20:48.445 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:48.485 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:48.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:48.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:48.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:48.620 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:48.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:48.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:48.722 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:48.756 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:48.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:48.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:48.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:48.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:48.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:48.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:49.026 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:49.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:49.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:49.128 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:49.155 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:49.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:49.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:49.257 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:49.294 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:49.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:49.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:49.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:49 smithi098 ceph-mon[97580]: pgmap v938: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.5 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 888 op/s; 19255/121647 objects degraded (15.829%); 4236/121647 objects misplaced (3.482%); 4.5 MiB/s, 71 objects/s recovering 2024-09-21T10:20:49.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:49 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3894044292' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:20:49.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:49 smithi098 ceph-mon[97580]: osdmap e293: 8 total, 8 up, 6 in 2024-09-21T10:20:49.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:49 smithi098 ceph-mon[100914]: pgmap v938: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.5 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 888 op/s; 19255/121647 objects degraded (15.829%); 4236/121647 objects misplaced (3.482%); 4.5 MiB/s, 71 objects/s recovering 2024-09-21T10:20:49.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:49 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3894044292' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:20:49.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:49 smithi098 ceph-mon[100914]: osdmap e293: 8 total, 8 up, 6 in 2024-09-21T10:20:49.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:49 smithi196 ceph-mon[84784]: pgmap v938: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.5 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 888 op/s; 19255/121647 objects degraded (15.829%); 4236/121647 objects misplaced (3.482%); 4.5 MiB/s, 71 objects/s recovering 2024-09-21T10:20:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:49 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3894044292' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:20:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:49 smithi196 ceph-mon[84784]: osdmap e293: 8 total, 8 up, 6 in 2024-09-21T10:20:49.396 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:49.427 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:49.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:49.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:49.529 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:49.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:49.567 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:49.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:49.668 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:49.698 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:49.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:49.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:49.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:49.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:49.839 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:49.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:49.940 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:49.945 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T10:20:49.945 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:20:49.958 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:49.966 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:49.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:49.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:50.068 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:50.094 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:50.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:50.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:50.195 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:50.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:50.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:50.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:50.395 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:50.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:50.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:50.497 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:50.528 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:50.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:50.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:50.630 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:50.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:20:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:20:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:20:50.668 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:50.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:50.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:50.769 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:50.801 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:50.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:50.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:50.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:50.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:50.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:51.045 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:51.065 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:20:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:20:50] "GET /metrics HTTP/1.1" 200 36547 "" "Prometheus/2.51.0" 2024-09-21T10:20:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:51.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:51.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:51.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:51.221 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:51.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:51.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:51.322 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:51.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:51 smithi098 ceph-mon[97580]: pgmap v940: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.4 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 720 op/s; 19255/116394 objects degraded (16.543%); 4236/116394 objects misplaced (3.639%); 4.5 MiB/s, 71 objects/s recovering 2024-09-21T10:20:51.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:51 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19255/116394 objects degraded (16.543%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:51.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:51 smithi098 ceph-mon[100914]: pgmap v940: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.4 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 720 op/s; 19255/116394 objects degraded (16.543%); 4236/116394 objects misplaced (3.639%); 4.5 MiB/s, 71 objects/s recovering 2024-09-21T10:20:51.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:51 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19255/116394 objects degraded (16.543%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:51.364 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:51.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:51.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:51.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:51 smithi196 ceph-mon[84784]: pgmap v940: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.4 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 720 op/s; 19255/116394 objects degraded (16.543%); 4236/116394 objects misplaced (3.639%); 4.5 MiB/s, 71 objects/s recovering 2024-09-21T10:20:51.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:51 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19255/116394 objects degraded (16.543%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:51.466 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:51.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:51.502 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:51.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:51.603 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:51.625 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:51.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:51.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:51.727 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:51.759 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:51.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:51.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:51.861 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:51.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:51.895 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:51.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:51.996 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:52.030 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:52.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:52.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:52.132 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:52.165 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:52.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:52.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:52.266 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:52.298 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:52.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:52.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:52.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:52 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/485478496' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:20:52.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:52 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/485478496' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:20:52.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:52 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/485478496' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:20:52.400 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:52.442 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:52.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:52.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:52.501 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:20:52.501 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:20:52.502 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:20:52.502 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:20:52.502 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:20:52.502 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:20:52.502 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:20:52.502 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:20:52.502 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:20:52.502 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:20:52.502 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:20:52.502 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:20:52.502 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:20:52.502 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:20:52.503 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:20:52.503 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:20:52.543 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:52.580 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:52.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:52.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:52.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:52.721 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:52.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:52.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:52.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:52.844 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:52.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:52.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:52.945 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:52.983 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:52.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:52.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:53.085 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:53.120 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:53.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:53.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:53.221 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:53.242 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:53.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:53.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:53.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:53.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:53 smithi098 ceph-mon[97580]: pgmap v941: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 552 op/s; 19255/115044 objects degraded (16.737%); 4236/115044 objects misplaced (3.682%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:20:53.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:53.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:53 smithi098 ceph-mon[100914]: pgmap v941: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 552 op/s; 19255/115044 objects degraded (16.737%); 4236/115044 objects misplaced (3.682%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:20:53.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:53.377 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:53.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:53.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:53.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:53 smithi196 ceph-mon[84784]: pgmap v941: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 552 op/s; 19255/115044 objects degraded (16.737%); 4236/115044 objects misplaced (3.682%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:20:53.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:53.479 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:53.519 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:53.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:53.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:53.621 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:53.652 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:53.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:53.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:53.754 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:53.783 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:53.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:53.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:53.885 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:53.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:53.925 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:53.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:54.026 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:54.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:54.058 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:54.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:54.160 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:54.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:54.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:54.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:54.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:54.328 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:54.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:54.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:54.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:54.470 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:54.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:54.571 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:54.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:54.605 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:54.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:54.706 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:54.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:54.730 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:54.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:54.831 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:54.858 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:54.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:54.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:54.959 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:54.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:54.991 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:54.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:55.092 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:55.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:55.127 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:55.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:55.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:55.266 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:55.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:55.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:55.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:55 smithi098 ceph-mon[97580]: pgmap v942: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 799 op/s; 19076/106437 objects degraded (17.922%); 4219/106437 objects misplaced (3.964%); 1.5 MiB/s, 45 objects/s recovering 2024-09-21T10:20:55.332 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:55 smithi098 ceph-mon[100914]: pgmap v942: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 799 op/s; 19076/106437 objects degraded (17.922%); 4219/106437 objects misplaced (3.964%); 1.5 MiB/s, 45 objects/s recovering 2024-09-21T10:20:55.368 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:55.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:55 smithi196 ceph-mon[84784]: pgmap v942: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 799 op/s; 19076/106437 objects degraded (17.922%); 4219/106437 objects misplaced (3.964%); 1.5 MiB/s, 45 objects/s recovering 2024-09-21T10:20:55.406 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:55.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:55.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:55.507 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:55.537 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:55.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:55.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:55.638 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:55.674 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:55.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:55.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:55.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:55.817 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:55.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:55.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:55.919 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:55.953 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:55.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:55.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:56.055 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:56.086 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:56.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:56.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:56.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:56.227 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:56.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:56.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:56.241 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:56.328 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:56.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:56 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 19076/106437 objects degraded (17.922%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:56.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:56 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 19076/106437 objects degraded (17.922%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:56.366 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:56.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:56.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:56.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:56 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 19076/106437 objects degraded (17.922%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:20:56.468 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:56.500 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:56.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:56.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:56.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:56.622 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:56.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:56.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:56.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:56.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:56.761 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:56.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:56.862 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:56.895 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:56.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:56.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:56.997 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:57.027 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:57.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:57.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:57.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:57.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:57.208 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:57.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:57.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:57.340 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:57.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:57.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:57.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:57 smithi098 ceph-mon[97580]: pgmap v943: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 818 op/s; 19076/102597 objects degraded (18.593%); 4219/102597 objects misplaced (4.112%); 1.3 MiB/s, 39 objects/s recovering 2024-09-21T10:20:57.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:57 smithi098 ceph-mon[100914]: pgmap v943: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 818 op/s; 19076/102597 objects degraded (18.593%); 4219/102597 objects misplaced (4.112%); 1.3 MiB/s, 39 objects/s recovering 2024-09-21T10:20:57.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:57 smithi196 ceph-mon[84784]: pgmap v943: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 818 op/s; 19076/102597 objects degraded (18.593%); 4219/102597 objects misplaced (4.112%); 1.3 MiB/s, 39 objects/s recovering 2024-09-21T10:20:57.441 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:57.470 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:57.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:57.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:57.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:57.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:57.618 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:57.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:57.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:57.753 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:57.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:57.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:57.797 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:20:57.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:57.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:57.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:58.006 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:58.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:58.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:58.147 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:58.184 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:58.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:58.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:58.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:58.319 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:58.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:58.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:58.421 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:58.465 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:58.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:58.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:58.509 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:20:58.510 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:20:58.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:58.608 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:58.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:58.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:58.709 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:58.742 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:58.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:58.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:58.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:58.871 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:58.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:58.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:58.972 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:59.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:59.017 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:59.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:59.118 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:59.156 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:59.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:59.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:59.258 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:59.267 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:59.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:59.299 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:20:59.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:59.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:20:59 smithi098 ceph-mon[97580]: pgmap v944: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 791 op/s; 18854/97905 objects degraded (19.257%); 4219/97905 objects misplaced (4.309%); 109 KiB/s, 41 objects/s recovering 2024-09-21T10:20:59.338 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:20:59 smithi098 ceph-mon[100914]: pgmap v944: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 791 op/s; 18854/97905 objects degraded (19.257%); 4219/97905 objects misplaced (4.309%); 109 KiB/s, 41 objects/s recovering 2024-09-21T10:20:59.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:20:59 smithi196 ceph-mon[84784]: pgmap v944: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 791 op/s; 18854/97905 objects degraded (19.257%); 4219/97905 objects misplaced (4.309%); 109 KiB/s, 41 objects/s recovering 2024-09-21T10:20:59.401 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:59.445 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:59.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:59.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:59.546 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:59.584 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:59.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:59.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:59.685 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:59.719 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:59.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:59.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:59.821 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:59.854 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:59.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:59.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:59.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:59.995 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:20:59.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:59.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:00.096 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:00.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:00.136 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:00.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:00.237 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:00.278 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:00.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:00.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:00.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:00.422 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:00.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:00.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:00.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:00.565 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:00.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:00.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:00.620 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:21:00.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:21:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:21:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:21:00.668 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:00.700 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:00.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:00.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:00.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:00.833 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:00.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:00.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:00.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:21:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:21:00] "GET /metrics HTTP/1.1" 200 36545 "" "Prometheus/2.51.0" 2024-09-21T10:21:00.934 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:00.963 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:00.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:00.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:01.064 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:01.098 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:01.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:01.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:01.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:01.245 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:01.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:01.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:01.312 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:01 smithi098 ceph-mon[97580]: pgmap v945: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 799 op/s; 18854/91860 objects degraded (20.525%); 4219/91860 objects misplaced (4.593%); 96 KiB/s, 36 objects/s recovering 2024-09-21T10:21:01.312 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:01 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 18854/91860 objects degraded (20.525%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:01.312 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:01 smithi098 ceph-mon[100914]: pgmap v945: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 799 op/s; 18854/91860 objects degraded (20.525%); 4219/91860 objects misplaced (4.593%); 96 KiB/s, 36 objects/s recovering 2024-09-21T10:21:01.312 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:01 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 18854/91860 objects degraded (20.525%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:01.346 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:01.373 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:01.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:01.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:01.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:01 smithi196 ceph-mon[84784]: pgmap v945: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 799 op/s; 18854/91860 objects degraded (20.525%); 4219/91860 objects misplaced (4.593%); 96 KiB/s, 36 objects/s recovering 2024-09-21T10:21:01.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:01 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 18854/91860 objects degraded (20.525%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:01.475 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:01.508 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:01.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:01.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:01.610 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:01.648 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:01.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:01.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:01.750 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:01.776 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:01.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:01.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:01.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:01.921 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:01.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:01.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:02.022 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:02.067 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:02.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:02.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:02.168 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:02.205 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:02.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:02.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:02.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:02.348 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:02.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:02.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:02.450 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:02.482 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:02.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:02.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:02.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:02.627 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:02.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:02.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:02.728 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:02.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:02.768 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:02.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:02.868 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:02.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:02.913 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:02.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:03.014 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:03.044 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:21:03.045 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:21:03.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:03.048 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:03.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:03.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:03.185 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:03.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:03.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:03.259 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:03 smithi098 ceph-mon[100914]: pgmap v946: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.8 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 725 op/s; 18854/90279 objects degraded (20.884%); 4219/90279 objects misplaced (4.673%); 91 KiB/s, 34 objects/s recovering 2024-09-21T10:21:03.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:03 smithi098 ceph-mon[97580]: pgmap v946: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.8 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 725 op/s; 18854/90279 objects degraded (20.884%); 4219/90279 objects misplaced (4.673%); 91 KiB/s, 34 objects/s recovering 2024-09-21T10:21:03.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:03.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:03.332 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:03.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:03.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:03 smithi196 ceph-mon[84784]: pgmap v946: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.8 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 725 op/s; 18854/90279 objects degraded (20.884%); 4219/90279 objects misplaced (4.673%); 91 KiB/s, 34 objects/s recovering 2024-09-21T10:21:03.433 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:03.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:03.464 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:03.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:03.566 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:03.597 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:03.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:03.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:03.699 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:03.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:03.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:03.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:03.754 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":946,"stamp":"2024-09-21T10:21:01.415853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1975064549,"num_objects":30093,"num_object_clones":0,"num_object_copies":90279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18854,"num_objects_misplaced":4219,"num_objects_unfound":0,"num_objects_dirty":30093,"num_whiteouts":0,"num_read":6289,"num_read_kb":13765,"num_write":178327,"num_write_kb":6648940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5961,"num_bytes_recovered":359606592,"num_keys_recovered":30,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109003,"ondisk_log_size":109003,"up":267,"acting":262,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":10436288,"kb_used_data":5836688,"kb_used_omap":28,"kb_used_meta":4599395,"kb_avail":552034624,"statfs":{"total":575970213888,"available":565283454976,"internally_reserved":0,"allocated":5976768512,"data_stored":5967277178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29289,"internal_metadata":4709780887},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-570490880,"num_objects":-8705,"num_object_clones":0,"num_object_copies":-26115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-401,"num_objects_misplaced":-17,"num_objects_unfound":0,"num_objects_dirty":-8705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8705,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":418,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004155"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":108,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.076965+0000","last_change":"2024-09-21T10:18:33.471373+0000","last_active":"2024-09-21T10:20:48.076965+0000","last_peered":"2024-09-21T10:20:48.076965+0000","last_clean":"2024-09-21T10:20:48.076965+0000","last_became_active":"2024-09-21T10:18:33.471052+0000","last_became_peered":"2024-09-21T10:18:33.471052+0000","last_unstale":"2024-09-21T10:20:48.076965+0000","last_undegraded":"2024-09-21T10:20:48.076965+0000","last_fullsized":"2024-09-21T10:20:48.076965+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:42:19.491129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":307,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.095619+0000","last_change":"2024-09-21T10:08:00.406360+0000","last_active":"2024-09-21T10:20:46.095619+0000","last_peered":"2024-09-21T10:20:46.095619+0000","last_clean":"2024-09-21T10:20:46.095619+0000","last_became_active":"2024-09-21T10:08:00.380042+0000","last_became_peered":"2024-09-21T10:08:00.380042+0000","last_unstale":"2024-09-21T10:20:46.095619+0000","last_undegraded":"2024-09-21T10:20:46.095619+0000","last_fullsized":"2024-09-21T10:20:46.095619+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:49:46.544051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":223,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.095741+0000","last_change":"2024-09-21T10:18:33.465357+0000","last_active":"2024-09-21T10:20:46.095741+0000","last_peered":"2024-09-21T10:20:46.095741+0000","last_clean":"2024-09-21T10:20:46.095741+0000","last_became_active":"2024-09-21T10:18:33.464952+0000","last_became_peered":"2024-09-21T10:18:33.464952+0000","last_unstale":"2024-09-21T10:20:46.095741+0000","last_undegraded":"2024-09-21T10:20:46.095741+0000","last_fullsized":"2024-09-21T10:20:46.095741+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:15:41.090866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":99,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.095476+0000","last_change":"2024-09-21T10:18:33.475291+0000","last_active":"2024-09-21T10:20:46.095476+0000","last_peered":"2024-09-21T10:20:46.095476+0000","last_clean":"2024-09-21T10:20:46.095476+0000","last_became_active":"2024-09-21T10:18:33.474971+0000","last_became_peered":"2024-09-21T10:18:33.474971+0000","last_unstale":"2024-09-21T10:20:46.095476+0000","last_undegraded":"2024-09-21T10:20:46.095476+0000","last_fullsized":"2024-09-21T10:20:46.095476+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:39:40.179668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":461,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.070503+0000","last_change":"2024-09-21T10:18:33.475179+0000","last_active":"2024-09-21T10:20:48.070503+0000","last_peered":"2024-09-21T10:20:48.070503+0000","last_clean":"2024-09-21T10:20:48.070503+0000","last_became_active":"2024-09-21T10:18:33.474914+0000","last_became_peered":"2024-09-21T10:18:33.474914+0000","last_unstale":"2024-09-21T10:20:48.070503+0000","last_undegraded":"2024-09-21T10:20:48.070503+0000","last_fullsized":"2024-09-21T10:20:48.070503+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:15:32.186986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":173,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.044050+0000","last_change":"2024-09-21T10:08:00.385671+0000","last_active":"2024-09-21T10:20:46.044050+0000","last_peered":"2024-09-21T10:20:46.044050+0000","last_clean":"2024-09-21T10:20:46.044050+0000","last_became_active":"2024-09-21T10:08:00.385368+0000","last_became_peered":"2024-09-21T10:08:00.385368+0000","last_unstale":"2024-09-21T10:20:46.044050+0000","last_undegraded":"2024-09-21T10:20:46.044050+0000","last_fullsized":"2024-09-21T10:20:46.044050+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:21.799149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":108,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.084893+0000","last_change":"2024-09-21T10:18:33.481890+0000","last_active":"2024-09-21T10:20:48.084893+0000","last_peered":"2024-09-21T10:20:48.084893+0000","last_clean":"2024-09-21T10:20:48.084893+0000","last_became_active":"2024-09-21T10:18:33.480639+0000","last_became_peered":"2024-09-21T10:18:33.480639+0000","last_unstale":"2024-09-21T10:20:48.084893+0000","last_undegraded":"2024-09-21T10:20:48.084893+0000","last_fullsized":"2024-09-21T10:20:48.084893+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:44.797540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":265,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.045221+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:20:46.045221+0000","last_peered":"2024-09-21T10:20:46.045221+0000","last_clean":"2024-09-21T10:20:46.045221+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:20:46.045221+0000","last_undegraded":"2024-09-21T10:20:46.045221+0000","last_fullsized":"2024-09-21T10:20:46.045221+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"293'11203","reported_seq":11271,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:59.532655+0000","last_change":"2024-09-21T10:17:44.113987+0000","last_active":"2024-09-21T10:20:59.532655+0000","last_peered":"2024-09-21T10:20:59.532655+0000","last_clean":"2024-09-21T10:20:59.532655+0000","last_became_active":"2024-09-21T10:17:44.113680+0000","last_became_peered":"2024-09-21T10:17:44.113680+0000","last_unstale":"2024-09-21T10:20:59.532655+0000","last_undegraded":"2024-09-21T10:20:59.532655+0000","last_fullsized":"2024-09-21T10:20:59.532655+0000","mapping_epoch":265,"log_start":"272'2300","ondisk_log_start":"272'2300","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":8903,"log_dups_size":0,"ondisk_log_size":8903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:38.165184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116457472,"num_objects":1777,"num_object_clones":0,"num_object_copies":5331,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11203,"num_write_kb":415360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":300,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.096007+0000","last_change":"2024-09-21T10:08:00.392499+0000","last_active":"2024-09-21T10:20:46.096007+0000","last_peered":"2024-09-21T10:20:46.096007+0000","last_clean":"2024-09-21T10:20:46.096007+0000","last_became_active":"2024-09-21T10:08:00.379165+0000","last_became_peered":"2024-09-21T10:08:00.379165+0000","last_unstale":"2024-09-21T10:20:46.096007+0000","last_undegraded":"2024-09-21T10:20:46.096007+0000","last_fullsized":"2024-09-21T10:20:46.096007+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:18.933144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":156,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.095242+0000","last_change":"2024-09-21T10:07:43.794677+0000","last_active":"2024-09-21T10:20:46.095242+0000","last_peered":"2024-09-21T10:20:46.095242+0000","last_clean":"2024-09-21T10:20:46.095242+0000","last_became_active":"2024-09-21T10:07:43.794348+0000","last_became_peered":"2024-09-21T10:07:43.794348+0000","last_unstale":"2024-09-21T10:20:46.095242+0000","last_undegraded":"2024-09-21T10:20:46.095242+0000","last_fullsized":"2024-09-21T10:20:46.095242+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:11.159014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"293'11414","reported_seq":11450,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:58.495423+0000","last_change":"2024-09-21T10:17:43.799118+0000","last_active":"2024-09-21T10:20:58.495423+0000","last_peered":"2024-09-21T10:20:58.495423+0000","last_clean":"2024-09-21T10:20:58.495423+0000","last_became_active":"2024-09-21T10:17:43.798738+0000","last_became_peered":"2024-09-21T10:17:43.798738+0000","last_unstale":"2024-09-21T10:20:58.495423+0000","last_undegraded":"2024-09-21T10:20:58.495423+0000","last_fullsized":"2024-09-21T10:20:58.495423+0000","mapping_epoch":265,"log_start":"278'5100","ondisk_log_start":"278'5100","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":6314,"log_dups_size":0,"ondisk_log_size":6314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:31:31.743422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125304832,"num_objects":1912,"num_object_clones":0,"num_object_copies":5736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11414,"num_write_kb":426432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":301,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.097417+0000","last_change":"2024-09-21T10:08:00.412449+0000","last_active":"2024-09-21T10:20:46.097417+0000","last_peered":"2024-09-21T10:20:46.097417+0000","last_clean":"2024-09-21T10:20:46.097417+0000","last_became_active":"2024-09-21T10:08:00.386530+0000","last_became_peered":"2024-09-21T10:08:00.386530+0000","last_unstale":"2024-09-21T10:20:46.097417+0000","last_undegraded":"2024-09-21T10:20:46.097417+0000","last_fullsized":"2024-09-21T10:20:46.097417+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:12:50.581176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":387,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.076770+0000","last_change":"2024-09-21T10:18:33.479057+0000","last_active":"2024-09-21T10:20:48.076770+0000","last_peered":"2024-09-21T10:20:48.076770+0000","last_clean":"2024-09-21T10:20:48.076770+0000","last_became_active":"2024-09-21T10:18:33.478752+0000","last_became_peered":"2024-09-21T10:18:33.478752+0000","last_unstale":"2024-09-21T10:20:48.076770+0000","last_undegraded":"2024-09-21T10:20:48.076770+0000","last_fullsized":"2024-09-21T10:20:48.076770+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:30.068585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"293'10980","reported_seq":11048,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:57.996514+0000","last_change":"2024-09-21T10:17:43.799119+0000","last_active":"2024-09-21T10:20:57.996514+0000","last_peered":"2024-09-21T10:20:57.996514+0000","last_clean":"2024-09-21T10:20:57.996514+0000","last_became_active":"2024-09-21T10:17:43.798511+0000","last_became_peered":"2024-09-21T10:17:43.798511+0000","last_unstale":"2024-09-21T10:20:57.996514+0000","last_undegraded":"2024-09-21T10:20:57.996514+0000","last_fullsized":"2024-09-21T10:20:57.996514+0000","mapping_epoch":265,"log_start":"273'4100","ondisk_log_start":"273'4100","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":6880,"log_dups_size":0,"ondisk_log_size":6880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:00.554890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122552320,"num_objects":1870,"num_object_clones":0,"num_object_copies":5610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10980,"num_write_kb":411200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":241,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.095867+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:20:46.095867+0000","last_peered":"2024-09-21T10:20:46.095867+0000","last_clean":"2024-09-21T10:20:46.095867+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:20:46.095867+0000","last_undegraded":"2024-09-21T10:20:46.095867+0000","last_fullsized":"2024-09-21T10:20:46.095867+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":256,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.043940+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:20:46.043940+0000","last_peered":"2024-09-21T10:20:46.043940+0000","last_clean":"2024-09-21T10:20:46.043940+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:20:46.043940+0000","last_undegraded":"2024-09-21T10:20:46.043940+0000","last_fullsized":"2024-09-21T10:20:46.043940+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"293'11202","reported_seq":11271,"reported_epoch":293,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:20:56.756171+0000","last_change":"2024-09-21T10:18:34.691817+0000","last_active":"2024-09-21T10:20:56.756171+0000","last_peered":"2024-09-21T10:20:56.756171+0000","last_clean":"2024-09-21T10:18:32.440045+0000","last_became_active":"2024-09-21T10:18:34.676050+0000","last_became_peered":"2024-09-21T10:18:34.676050+0000","last_unstale":"2024-09-21T10:20:56.756171+0000","last_undegraded":"2024-09-21T10:18:34.519870+0000","last_fullsized":"2024-09-21T10:18:34.459225+0000","mapping_epoch":275,"log_start":"279'5700","ondisk_log_start":"279'5700","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":5502,"log_dups_size":0,"ondisk_log_size":5502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126615552,"num_objects":1932,"num_object_clones":0,"num_object_copies":5796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11202,"num_write_kb":420288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6],"avail_no_missing":["1","3","6"],"object_location_counts":[{"shards":"1,6","objects":-2343}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":91,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.044231+0000","last_change":"2024-09-21T10:18:33.483457+0000","last_active":"2024-09-21T10:20:46.044231+0000","last_peered":"2024-09-21T10:20:46.044231+0000","last_clean":"2024-09-21T10:20:46.044231+0000","last_became_active":"2024-09-21T10:18:33.483140+0000","last_became_peered":"2024-09-21T10:18:33.483140+0000","last_unstale":"2024-09-21T10:20:46.044231+0000","last_undegraded":"2024-09-21T10:20:46.044231+0000","last_fullsized":"2024-09-21T10:20:46.044231+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:37:39.302566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":483,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.069965+0000","last_change":"2024-09-21T10:18:33.483011+0000","last_active":"2024-09-21T10:20:48.069965+0000","last_peered":"2024-09-21T10:20:48.069965+0000","last_clean":"2024-09-21T10:20:48.069965+0000","last_became_active":"2024-09-21T10:18:33.482712+0000","last_became_peered":"2024-09-21T10:18:33.482712+0000","last_unstale":"2024-09-21T10:20:48.069965+0000","last_undegraded":"2024-09-21T10:20:48.069965+0000","last_fullsized":"2024-09-21T10:20:48.069965+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:05:11.037442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"293'10745","reported_seq":11996,"reported_epoch":293,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T10:20:56.783395+0000","last_change":"2024-09-21T10:20:44.375354+0000","last_active":"2024-09-21T10:20:56.783395+0000","last_peered":"2024-09-21T10:20:56.783395+0000","last_clean":"2024-09-21T10:18:32.459541+0000","last_became_active":"2024-09-21T10:18:41.519168+0000","last_became_peered":"2024-09-21T10:18:41.519168+0000","last_unstale":"2024-09-21T10:20:56.783395+0000","last_undegraded":"2024-09-21T10:18:41.384300+0000","last_fullsized":"2024-09-21T10:18:41.373751+0000","mapping_epoch":275,"log_start":"279'5200","ondisk_log_start":"279'5200","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":5545,"log_dups_size":0,"ondisk_log_size":5545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123797504,"num_objects":1889,"num_object_clones":0,"num_object_copies":5667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3421,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10745,"num_write_kb":404288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":590,"num_bytes_recovered":12386304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5],"avail_no_missing":["1","3","5"],"object_location_counts":[{"shards":"1,5","objects":-1532}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":242,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.081219+0000","last_change":"2024-09-21T10:18:33.488794+0000","last_active":"2024-09-21T10:20:48.081219+0000","last_peered":"2024-09-21T10:20:48.081219+0000","last_clean":"2024-09-21T10:20:48.081219+0000","last_became_active":"2024-09-21T10:18:33.479048+0000","last_became_peered":"2024-09-21T10:18:33.479048+0000","last_unstale":"2024-09-21T10:20:48.081219+0000","last_undegraded":"2024-09-21T10:20:48.081219+0000","last_fullsized":"2024-09-21T10:20:48.081219+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:03.387788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":110,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.044146+0000","last_change":"2024-09-21T10:18:33.475977+0000","last_active":"2024-09-21T10:20:46.044146+0000","last_peered":"2024-09-21T10:20:46.044146+0000","last_clean":"2024-09-21T10:20:46.044146+0000","last_became_active":"2024-09-21T10:18:33.475590+0000","last_became_peered":"2024-09-21T10:18:33.475590+0000","last_unstale":"2024-09-21T10:20:46.044146+0000","last_undegraded":"2024-09-21T10:20:46.044146+0000","last_fullsized":"2024-09-21T10:20:46.044146+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:54:57.304453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1","version":"293'11017","reported_seq":11085,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:56.721947+0000","last_change":"2024-09-21T10:17:43.803150+0000","last_active":"2024-09-21T10:20:56.721947+0000","last_peered":"2024-09-21T10:20:56.721947+0000","last_clean":"2024-09-21T10:20:56.721947+0000","last_became_active":"2024-09-21T10:17:43.801977+0000","last_became_peered":"2024-09-21T10:17:43.801977+0000","last_unstale":"2024-09-21T10:20:56.721947+0000","last_undegraded":"2024-09-21T10:20:56.721947+0000","last_fullsized":"2024-09-21T10:20:56.721947+0000","mapping_epoch":265,"log_start":"279'5500","ondisk_log_start":"279'5500","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":5517,"log_dups_size":0,"ondisk_log_size":5517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:58:28.580896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123535360,"num_objects":1885,"num_object_clones":0,"num_object_copies":5655,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11017,"num_write_kb":412864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":297,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.044053+0000","last_change":"2024-09-21T10:18:33.473849+0000","last_active":"2024-09-21T10:20:46.044053+0000","last_peered":"2024-09-21T10:20:46.044053+0000","last_clean":"2024-09-21T10:20:46.044053+0000","last_became_active":"2024-09-21T10:18:33.473037+0000","last_became_peered":"2024-09-21T10:18:33.473037+0000","last_unstale":"2024-09-21T10:20:46.044053+0000","last_undegraded":"2024-09-21T10:20:46.044053+0000","last_fullsized":"2024-09-21T10:20:46.044053+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:03:16.322667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":388,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.083610+0000","last_change":"2024-09-21T10:08:00.387420+0000","last_active":"2024-09-21T10:20:48.083610+0000","last_peered":"2024-09-21T10:20:48.083610+0000","last_clean":"2024-09-21T10:20:48.083610+0000","last_became_active":"2024-09-21T10:08:00.386843+0000","last_became_peered":"2024-09-21T10:08:00.386843+0000","last_unstale":"2024-09-21T10:20:48.083610+0000","last_undegraded":"2024-09-21T10:20:48.083610+0000","last_fullsized":"2024-09-21T10:20:48.083610+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:51:00.654836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"293'10991","reported_seq":12210,"reported_epoch":293,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:20:57.971609+0000","last_change":"2024-09-21T10:18:41.616297+0000","last_active":"2024-09-21T10:20:57.971609+0000","last_peered":"2024-09-21T10:20:57.971609+0000","last_clean":"2024-09-21T10:18:32.444959+0000","last_became_active":"2024-09-21T10:18:34.618845+0000","last_became_peered":"2024-09-21T10:18:34.618845+0000","last_unstale":"2024-09-21T10:20:57.971609+0000","last_undegraded":"2024-09-21T10:18:41.613207+0000","last_fullsized":"2024-09-21T10:18:34.462595+0000","mapping_epoch":275,"log_start":"276'4100","ondisk_log_start":"276'4100","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":6891,"log_dups_size":0,"ondisk_log_size":6891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125632512,"num_objects":1917,"num_object_clones":0,"num_object_copies":5751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10991,"num_write_kb":413056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":573,"num_bytes_recovered":37552128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,5","objects":-1601}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":372,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.082849+0000","last_change":"2024-09-21T10:08:00.382496+0000","last_active":"2024-09-21T10:20:48.082849+0000","last_peered":"2024-09-21T10:20:48.082849+0000","last_clean":"2024-09-21T10:20:48.082849+0000","last_became_active":"2024-09-21T10:08:00.382180+0000","last_became_peered":"2024-09-21T10:08:00.382180+0000","last_unstale":"2024-09-21T10:20:48.082849+0000","last_undegraded":"2024-09-21T10:20:48.082849+0000","last_fullsized":"2024-09-21T10:20:48.082849+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:16.074736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":226,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.044141+0000","last_change":"2024-09-21T10:08:38.906801+0000","last_active":"2024-09-21T10:20:46.044141+0000","last_peered":"2024-09-21T10:20:46.044141+0000","last_clean":"2024-09-21T10:20:46.044141+0000","last_became_active":"2024-09-21T10:08:38.906487+0000","last_became_peered":"2024-09-21T10:08:38.906487+0000","last_unstale":"2024-09-21T10:20:46.044141+0000","last_undegraded":"2024-09-21T10:20:46.044141+0000","last_fullsized":"2024-09-21T10:20:46.044141+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:31.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"293'11055","reported_seq":11123,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:56.705616+0000","last_change":"2024-09-21T10:17:44.115373+0000","last_active":"2024-09-21T10:20:56.705616+0000","last_peered":"2024-09-21T10:20:56.705616+0000","last_clean":"2024-09-21T10:20:56.705616+0000","last_became_active":"2024-09-21T10:17:44.115083+0000","last_became_peered":"2024-09-21T10:17:44.115083+0000","last_unstale":"2024-09-21T10:20:56.705616+0000","last_undegraded":"2024-09-21T10:20:56.705616+0000","last_fullsized":"2024-09-21T10:20:56.705616+0000","mapping_epoch":265,"log_start":"279'5600","ondisk_log_start":"279'5600","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":5455,"log_dups_size":0,"ondisk_log_size":5455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:18.021725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128385024,"num_objects":1959,"num_object_clones":0,"num_object_copies":5877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11055,"num_write_kb":416448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":109,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.080931+0000","last_change":"2024-09-21T10:18:33.484568+0000","last_active":"2024-09-21T10:20:48.080931+0000","last_peered":"2024-09-21T10:20:48.080931+0000","last_clean":"2024-09-21T10:20:48.080931+0000","last_became_active":"2024-09-21T10:18:33.483975+0000","last_became_peered":"2024-09-21T10:18:33.483975+0000","last_unstale":"2024-09-21T10:20:48.080931+0000","last_undegraded":"2024-09-21T10:20:48.080931+0000","last_fullsized":"2024-09-21T10:20:48.080931+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:13.276318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":471,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.088778+0000","last_change":"2024-09-21T10:18:33.474464+0000","last_active":"2024-09-21T10:20:48.088778+0000","last_peered":"2024-09-21T10:20:48.088778+0000","last_clean":"2024-09-21T10:20:48.088778+0000","last_became_active":"2024-09-21T10:18:33.474128+0000","last_became_peered":"2024-09-21T10:18:33.474128+0000","last_unstale":"2024-09-21T10:20:48.088778+0000","last_undegraded":"2024-09-21T10:20:48.088778+0000","last_fullsized":"2024-09-21T10:20:48.088778+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:32:49.409409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"293'10982","reported_seq":11018,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:58.526879+0000","last_change":"2024-09-21T10:17:44.113304+0000","last_active":"2024-09-21T10:20:58.526879+0000","last_peered":"2024-09-21T10:20:58.526879+0000","last_clean":"2024-09-21T10:20:58.526879+0000","last_became_active":"2024-09-21T10:17:44.112949+0000","last_became_peered":"2024-09-21T10:17:44.112949+0000","last_unstale":"2024-09-21T10:20:58.526879+0000","last_undegraded":"2024-09-21T10:20:58.526879+0000","last_fullsized":"2024-09-21T10:20:58.526879+0000","mapping_epoch":265,"log_start":"278'4700","ondisk_log_start":"278'4700","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":6282,"log_dups_size":0,"ondisk_log_size":6282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:20.864228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119275520,"num_objects":1820,"num_object_clones":0,"num_object_copies":5460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10982,"num_write_kb":409664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":374,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.095323+0000","last_change":"2024-09-21T09:59:54.265556+0000","last_active":"2024-09-21T10:20:46.095323+0000","last_peered":"2024-09-21T10:20:46.095323+0000","last_clean":"2024-09-21T10:20:46.095323+0000","last_became_active":"2024-09-21T09:59:54.265033+0000","last_became_peered":"2024-09-21T09:59:54.265033+0000","last_unstale":"2024-09-21T10:20:46.095323+0000","last_undegraded":"2024-09-21T10:20:46.095323+0000","last_fullsized":"2024-09-21T10:20:46.095323+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T09:48:34.850872+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:40.449756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":752,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.085137+0000","last_change":"2024-09-21T10:07:43.790490+0000","last_active":"2024-09-21T10:20:48.085137+0000","last_peered":"2024-09-21T10:20:48.085137+0000","last_clean":"2024-09-21T10:20:48.085137+0000","last_became_active":"2024-09-21T10:07:43.789986+0000","last_became_peered":"2024-09-21T10:07:43.789986+0000","last_unstale":"2024-09-21T10:20:48.085137+0000","last_undegraded":"2024-09-21T10:20:48.085137+0000","last_fullsized":"2024-09-21T10:20:48.085137+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:08.740356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":481,"num_read_kb":1580,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":44,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.043841+0000","last_change":"2024-09-21T10:18:33.473676+0000","last_active":"2024-09-21T10:20:46.043841+0000","last_peered":"2024-09-21T10:20:46.043841+0000","last_clean":"2024-09-21T10:20:46.043841+0000","last_became_active":"2024-09-21T10:18:33.472593+0000","last_became_peered":"2024-09-21T10:18:33.472593+0000","last_unstale":"2024-09-21T10:20:46.043841+0000","last_undegraded":"2024-09-21T10:20:46.043841+0000","last_fullsized":"2024-09-21T10:20:46.043841+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:07:55.816740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":269,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.044319+0000","last_change":"2024-09-21T10:08:00.386135+0000","last_active":"2024-09-21T10:20:46.044319+0000","last_peered":"2024-09-21T10:20:46.044319+0000","last_clean":"2024-09-21T10:20:46.044319+0000","last_became_active":"2024-09-21T10:08:00.385796+0000","last_became_peered":"2024-09-21T10:08:00.385796+0000","last_unstale":"2024-09-21T10:20:46.044319+0000","last_undegraded":"2024-09-21T10:20:46.044319+0000","last_fullsized":"2024-09-21T10:20:46.044319+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:27.054699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"293'11164","reported_seq":11232,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:56.759391+0000","last_change":"2024-09-21T10:17:44.114644+0000","last_active":"2024-09-21T10:20:56.759391+0000","last_peered":"2024-09-21T10:20:56.759391+0000","last_clean":"2024-09-21T10:20:56.759391+0000","last_became_active":"2024-09-21T10:17:44.114361+0000","last_became_peered":"2024-09-21T10:17:44.114361+0000","last_unstale":"2024-09-21T10:20:56.759391+0000","last_undegraded":"2024-09-21T10:20:56.759391+0000","last_fullsized":"2024-09-21T10:20:56.759391+0000","mapping_epoch":265,"log_start":"279'5700","ondisk_log_start":"279'5700","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":5464,"log_dups_size":0,"ondisk_log_size":5464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:36:37.772952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125435904,"num_objects":1914,"num_object_clones":0,"num_object_copies":5742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11164,"num_write_kb":418496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":637,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.095286+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:20:46.095286+0000","last_peered":"2024-09-21T10:20:46.095286+0000","last_clean":"2024-09-21T10:20:46.095286+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:20:46.095286+0000","last_undegraded":"2024-09-21T10:20:46.095286+0000","last_fullsized":"2024-09-21T10:20:46.095286+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":484,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.071194+0000","last_change":"2024-09-21T10:18:33.471355+0000","last_active":"2024-09-21T10:20:48.071194+0000","last_peered":"2024-09-21T10:20:48.071194+0000","last_clean":"2024-09-21T10:20:48.071194+0000","last_became_active":"2024-09-21T10:18:33.470962+0000","last_became_peered":"2024-09-21T10:18:33.470962+0000","last_unstale":"2024-09-21T10:20:48.071194+0000","last_undegraded":"2024-09-21T10:20:48.071194+0000","last_fullsized":"2024-09-21T10:20:48.071194+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:53:57.516543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":466,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.070705+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:20:48.070705+0000","last_peered":"2024-09-21T10:20:48.070705+0000","last_clean":"2024-09-21T10:20:48.070705+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:20:48.070705+0000","last_undegraded":"2024-09-21T10:20:48.070705+0000","last_fullsized":"2024-09-21T10:20:48.070705+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"293'10997","reported_seq":12318,"reported_epoch":293,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:20:56.797580+0000","last_change":"2024-09-21T10:18:41.561934+0000","last_active":"2024-09-21T10:20:56.797580+0000","last_peered":"2024-09-21T10:20:56.797580+0000","last_clean":"2024-09-21T10:18:32.435113+0000","last_became_active":"2024-09-21T10:18:34.651959+0000","last_became_peered":"2024-09-21T10:18:34.651959+0000","last_unstale":"2024-09-21T10:20:56.797580+0000","last_undegraded":"2024-09-21T10:18:41.559442+0000","last_fullsized":"2024-09-21T10:18:34.459559+0000","mapping_epoch":275,"log_start":"279'5500","ondisk_log_start":"279'5500","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":5497,"log_dups_size":0,"ondisk_log_size":5497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127598592,"num_objects":1947,"num_object_clones":0,"num_object_copies":5841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3504,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10997,"num_write_kb":414208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":624,"num_bytes_recovered":40894464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2","objects":-1557}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":1240,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.082958+0000","last_change":"2024-09-21T10:18:36.495308+0000","last_active":"2024-09-21T10:20:48.082958+0000","last_peered":"2024-09-21T10:20:48.082958+0000","last_clean":"2024-09-21T10:20:48.082958+0000","last_became_active":"2024-09-21T10:18:36.494662+0000","last_became_peered":"2024-09-21T10:18:36.494662+0000","last_unstale":"2024-09-21T10:20:48.082958+0000","last_undegraded":"2024-09-21T10:20:48.082958+0000","last_fullsized":"2024-09-21T10:20:48.082958+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:49.218676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":338,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.080974+0000","last_change":"2024-09-21T10:08:18.470748+0000","last_active":"2024-09-21T10:20:48.080974+0000","last_peered":"2024-09-21T10:20:48.080974+0000","last_clean":"2024-09-21T10:20:48.080974+0000","last_became_active":"2024-09-21T10:08:18.470227+0000","last_became_peered":"2024-09-21T10:08:18.470227+0000","last_unstale":"2024-09-21T10:20:48.080974+0000","last_undegraded":"2024-09-21T10:20:48.080974+0000","last_fullsized":"2024-09-21T10:20:48.080974+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:16.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":460,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.102265+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:20:48.102265+0000","last_peered":"2024-09-21T10:20:48.102265+0000","last_clean":"2024-09-21T10:20:48.102265+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:20:48.102265+0000","last_undegraded":"2024-09-21T10:20:48.102265+0000","last_fullsized":"2024-09-21T10:20:48.102265+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"293'11234","reported_seq":11309,"reported_epoch":293,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-21T10:20:58.403570+0000","last_change":"2024-09-21T10:18:34.720485+0000","last_active":"2024-09-21T10:20:58.403570+0000","last_peered":"2024-09-21T10:20:58.403570+0000","last_clean":"2024-09-21T10:18:32.459708+0000","last_became_active":"2024-09-21T10:18:34.691759+0000","last_became_peered":"2024-09-21T10:18:34.691759+0000","last_unstale":"2024-09-21T10:20:58.403570+0000","last_undegraded":"2024-09-21T10:20:58.403570+0000","last_fullsized":"2024-09-21T10:20:58.403570+0000","mapping_epoch":275,"log_start":"270'1200","ondisk_log_start":"270'1200","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122945536,"num_objects":1876,"num_object_clones":0,"num_object_copies":5628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4219,"num_objects_unfound":0,"num_objects_dirty":1876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11234,"num_write_kb":419520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[3,5,7],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":-2343}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":937,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.077829+0000","last_change":"2024-09-21T10:07:43.794767+0000","last_active":"2024-09-21T10:20:48.077829+0000","last_peered":"2024-09-21T10:20:48.077829+0000","last_clean":"2024-09-21T10:20:48.077829+0000","last_became_active":"2024-09-21T10:07:43.794062+0000","last_became_peered":"2024-09-21T10:07:43.794062+0000","last_unstale":"2024-09-21T10:20:48.077829+0000","last_undegraded":"2024-09-21T10:20:48.077829+0000","last_fullsized":"2024-09-21T10:20:48.077829+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:51.977955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":363,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.069652+0000","last_change":"2024-09-21T10:08:00.389176+0000","last_active":"2024-09-21T10:20:48.069652+0000","last_peered":"2024-09-21T10:20:48.069652+0000","last_clean":"2024-09-21T10:20:48.069652+0000","last_became_active":"2024-09-21T10:08:00.383854+0000","last_became_peered":"2024-09-21T10:08:00.383854+0000","last_unstale":"2024-09-21T10:20:48.069652+0000","last_undegraded":"2024-09-21T10:20:48.069652+0000","last_fullsized":"2024-09-21T10:20:48.069652+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:46.952216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":384,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.083386+0000","last_change":"2024-09-21T10:18:33.465815+0000","last_active":"2024-09-21T10:20:48.083386+0000","last_peered":"2024-09-21T10:20:48.083386+0000","last_clean":"2024-09-21T10:20:48.083386+0000","last_became_active":"2024-09-21T10:18:33.465459+0000","last_became_peered":"2024-09-21T10:18:33.465459+0000","last_unstale":"2024-09-21T10:20:48.083386+0000","last_undegraded":"2024-09-21T10:20:48.083386+0000","last_fullsized":"2024-09-21T10:20:48.083386+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:22:08.199800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"293'11015","reported_seq":11083,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:57.969101+0000","last_change":"2024-09-21T10:17:43.800429+0000","last_active":"2024-09-21T10:20:57.969101+0000","last_peered":"2024-09-21T10:20:57.969101+0000","last_clean":"2024-09-21T10:20:57.969101+0000","last_became_active":"2024-09-21T10:17:43.799643+0000","last_became_peered":"2024-09-21T10:17:43.799643+0000","last_unstale":"2024-09-21T10:20:57.969101+0000","last_undegraded":"2024-09-21T10:20:57.969101+0000","last_fullsized":"2024-09-21T10:20:57.969101+0000","mapping_epoch":265,"log_start":"273'4100","ondisk_log_start":"273'4100","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":6915,"log_dups_size":0,"ondisk_log_size":6915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:38:08.368261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116588544,"num_objects":1779,"num_object_clones":0,"num_object_copies":5337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11015,"num_write_kb":409408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":127,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.083524+0000","last_change":"2024-09-21T10:18:33.467659+0000","last_active":"2024-09-21T10:20:48.083524+0000","last_peered":"2024-09-21T10:20:48.083524+0000","last_clean":"2024-09-21T10:20:48.083524+0000","last_became_active":"2024-09-21T10:18:33.467369+0000","last_became_peered":"2024-09-21T10:18:33.467369+0000","last_unstale":"2024-09-21T10:20:48.083524+0000","last_undegraded":"2024-09-21T10:20:48.083524+0000","last_fullsized":"2024-09-21T10:20:48.083524+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:16:41.951755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":1044,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.076914+0000","last_change":"2024-09-21T10:08:18.468995+0000","last_active":"2024-09-21T10:20:48.076914+0000","last_peered":"2024-09-21T10:20:48.076914+0000","last_clean":"2024-09-21T10:20:48.076914+0000","last_became_active":"2024-09-21T10:08:18.468243+0000","last_became_peered":"2024-09-21T10:08:18.468243+0000","last_unstale":"2024-09-21T10:20:48.076914+0000","last_undegraded":"2024-09-21T10:20:48.076914+0000","last_fullsized":"2024-09-21T10:20:48.076914+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:45.520968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":363,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.078035+0000","last_change":"2024-09-21T10:08:18.468595+0000","last_active":"2024-09-21T10:20:48.078035+0000","last_peered":"2024-09-21T10:20:48.078035+0000","last_clean":"2024-09-21T10:20:48.078035+0000","last_became_active":"2024-09-21T10:08:18.468296+0000","last_became_peered":"2024-09-21T10:08:18.468296+0000","last_unstale":"2024-09-21T10:20:48.078035+0000","last_undegraded":"2024-09-21T10:20:48.078035+0000","last_fullsized":"2024-09-21T10:20:48.078035+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:42.748130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.f","version":"293'11007","reported_seq":19269,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:59.527384+0000","last_change":"2024-09-21T10:20:46.081136+0000","last_active":"2024-09-21T10:20:59.527384+0000","last_peered":"2024-09-21T10:20:59.527384+0000","last_clean":"2024-09-21T10:20:59.527384+0000","last_became_active":"2024-09-21T10:20:46.046374+0000","last_became_peered":"2024-09-21T10:20:46.046374+0000","last_unstale":"2024-09-21T10:20:59.527384+0000","last_undegraded":"2024-09-21T10:20:59.527384+0000","last_fullsized":"2024-09-21T10:20:59.527384+0000","mapping_epoch":291,"log_start":"272'2100","ondisk_log_start":"272'2100","created":265,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":8907,"log_dups_size":0,"ondisk_log_size":8907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:30.571749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118292480,"num_objects":1805,"num_object_clones":0,"num_object_copies":5415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11007,"num_write_kb":409984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4091,"num_bytes_recovered":268107776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":339,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.076817+0000","last_change":"2024-09-21T10:08:18.469907+0000","last_active":"2024-09-21T10:20:48.076817+0000","last_peered":"2024-09-21T10:20:48.076817+0000","last_clean":"2024-09-21T10:20:48.076817+0000","last_became_active":"2024-09-21T10:08:18.469443+0000","last_became_peered":"2024-09-21T10:08:18.469443+0000","last_unstale":"2024-09-21T10:20:48.076817+0000","last_undegraded":"2024-09-21T10:20:48.076817+0000","last_fullsized":"2024-09-21T10:20:48.076817+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:15.645803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":687,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.095946+0000","last_change":"2024-09-21T10:08:00.419641+0000","last_active":"2024-09-21T10:20:46.095946+0000","last_peered":"2024-09-21T10:20:46.095946+0000","last_clean":"2024-09-21T10:20:46.095946+0000","last_became_active":"2024-09-21T10:08:00.387050+0000","last_became_peered":"2024-09-21T10:08:00.387050+0000","last_unstale":"2024-09-21T10:20:46.095946+0000","last_undegraded":"2024-09-21T10:20:46.095946+0000","last_fullsized":"2024-09-21T10:20:46.095946+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:00:32.881619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":445,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.069896+0000","last_change":"2024-09-21T09:59:54.267798+0000","last_active":"2024-09-21T10:20:48.069896+0000","last_peered":"2024-09-21T10:20:48.069896+0000","last_clean":"2024-09-21T10:20:48.069896+0000","last_became_active":"2024-09-21T09:59:54.265615+0000","last_became_peered":"2024-09-21T09:59:54.265615+0000","last_unstale":"2024-09-21T10:20:48.069896+0000","last_undegraded":"2024-09-21T10:20:48.069896+0000","last_fullsized":"2024-09-21T10:20:48.069896+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:30.363872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"293'11109","reported_seq":11180,"reported_epoch":293,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:20:57.969448+0000","last_change":"2024-09-21T10:18:34.660620+0000","last_active":"2024-09-21T10:20:57.969448+0000","last_peered":"2024-09-21T10:20:57.969448+0000","last_clean":"2024-09-21T10:18:32.448320+0000","last_became_active":"2024-09-21T10:18:34.639763+0000","last_became_peered":"2024-09-21T10:18:34.639763+0000","last_unstale":"2024-09-21T10:20:57.969448+0000","last_undegraded":"2024-09-21T10:18:34.495695+0000","last_fullsized":"2024-09-21T10:18:34.464258+0000","mapping_epoch":275,"log_start":"278'4200","ondisk_log_start":"278'4200","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":6909,"log_dups_size":0,"ondisk_log_size":6909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126943232,"num_objects":1937,"num_object_clones":0,"num_object_copies":5811,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11109,"num_write_kb":417472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,7","objects":-2199}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":56,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.095930+0000","last_change":"2024-09-21T10:18:33.489678+0000","last_active":"2024-09-21T10:20:46.095930+0000","last_peered":"2024-09-21T10:20:46.095930+0000","last_clean":"2024-09-21T10:20:46.095930+0000","last_became_active":"2024-09-21T10:18:33.474818+0000","last_became_peered":"2024-09-21T10:18:33.474818+0000","last_unstale":"2024-09-21T10:20:46.095930+0000","last_undegraded":"2024-09-21T10:20:46.095930+0000","last_fullsized":"2024-09-21T10:20:46.095930+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:49.828007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":735,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.077778+0000","last_change":"2024-09-21T10:18:36.503584+0000","last_active":"2024-09-21T10:20:48.077778+0000","last_peered":"2024-09-21T10:20:48.077778+0000","last_clean":"2024-09-21T10:20:48.077778+0000","last_became_active":"2024-09-21T10:18:36.502990+0000","last_became_peered":"2024-09-21T10:18:36.502990+0000","last_unstale":"2024-09-21T10:20:48.077778+0000","last_undegraded":"2024-09-21T10:20:48.077778+0000","last_fullsized":"2024-09-21T10:20:48.077778+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:11.648271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":85,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.095167+0000","last_change":"2024-09-21T10:18:33.479008+0000","last_active":"2024-09-21T10:20:46.095167+0000","last_peered":"2024-09-21T10:20:46.095167+0000","last_clean":"2024-09-21T10:20:46.095167+0000","last_became_active":"2024-09-21T10:18:33.478740+0000","last_became_peered":"2024-09-21T10:18:33.478740+0000","last_unstale":"2024-09-21T10:20:46.095167+0000","last_undegraded":"2024-09-21T10:20:46.095167+0000","last_fullsized":"2024-09-21T10:20:46.095167+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:16:52.594882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.d","version":"293'11156","reported_seq":11192,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:58.375953+0000","last_change":"2024-09-21T10:17:44.114402+0000","last_active":"2024-09-21T10:20:58.375953+0000","last_peered":"2024-09-21T10:20:58.375953+0000","last_clean":"2024-09-21T10:20:58.375953+0000","last_became_active":"2024-09-21T10:17:44.114051+0000","last_became_peered":"2024-09-21T10:17:44.114051+0000","last_unstale":"2024-09-21T10:20:58.375953+0000","last_undegraded":"2024-09-21T10:20:58.375953+0000","last_fullsized":"2024-09-21T10:20:58.375953+0000","mapping_epoch":265,"log_start":"278'4300","ondisk_log_start":"278'4300","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":6856,"log_dups_size":0,"ondisk_log_size":6856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:48:23.466657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120913944,"num_objects":1846,"num_object_clones":0,"num_object_copies":5538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1846,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11156,"num_write_kb":416001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":492,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.070415+0000","last_change":"2024-09-21T10:08:00.390436+0000","last_active":"2024-09-21T10:20:48.070415+0000","last_peered":"2024-09-21T10:20:48.070415+0000","last_clean":"2024-09-21T10:20:48.070415+0000","last_became_active":"2024-09-21T10:08:00.381835+0000","last_became_peered":"2024-09-21T10:08:00.381835+0000","last_unstale":"2024-09-21T10:20:48.070415+0000","last_undegraded":"2024-09-21T10:20:48.070415+0000","last_fullsized":"2024-09-21T10:20:48.070415+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:01:35.697356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":519,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.044450+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:20:46.044450+0000","last_peered":"2024-09-21T10:20:46.044450+0000","last_clean":"2024-09-21T10:20:46.044450+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:20:46.044450+0000","last_undegraded":"2024-09-21T10:20:46.044450+0000","last_fullsized":"2024-09-21T10:20:46.044450+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":157,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.044208+0000","last_change":"2024-09-21T10:08:00.385774+0000","last_active":"2024-09-21T10:20:46.044208+0000","last_peered":"2024-09-21T10:20:46.044208+0000","last_clean":"2024-09-21T10:20:46.044208+0000","last_became_active":"2024-09-21T10:08:00.385445+0000","last_became_peered":"2024-09-21T10:08:00.385445+0000","last_unstale":"2024-09-21T10:20:46.044208+0000","last_undegraded":"2024-09-21T10:20:46.044208+0000","last_fullsized":"2024-09-21T10:20:46.044208+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:42.442190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":216,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.043937+0000","last_change":"2024-09-21T10:08:00.379661+0000","last_active":"2024-09-21T10:20:46.043937+0000","last_peered":"2024-09-21T10:20:46.043937+0000","last_clean":"2024-09-21T10:20:46.043937+0000","last_became_active":"2024-09-21T10:08:00.379301+0000","last_became_peered":"2024-09-21T10:08:00.379301+0000","last_unstale":"2024-09-21T10:20:46.043937+0000","last_undegraded":"2024-09-21T10:20:46.043937+0000","last_fullsized":"2024-09-21T10:20:46.043937+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:15.649523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":239,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.095755+0000","last_change":"2024-09-21T10:08:00.410731+0000","last_active":"2024-09-21T10:20:46.095755+0000","last_peered":"2024-09-21T10:20:46.095755+0000","last_clean":"2024-09-21T10:20:46.095755+0000","last_became_active":"2024-09-21T10:08:00.380324+0000","last_became_peered":"2024-09-21T10:08:00.380324+0000","last_unstale":"2024-09-21T10:20:46.095755+0000","last_undegraded":"2024-09-21T10:20:46.095755+0000","last_fullsized":"2024-09-21T10:20:46.095755+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:47:34.960221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":338,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.077902+0000","last_change":"2024-09-21T10:08:18.470378+0000","last_active":"2024-09-21T10:20:48.077902+0000","last_peered":"2024-09-21T10:20:48.077902+0000","last_clean":"2024-09-21T10:20:48.077902+0000","last_became_active":"2024-09-21T10:08:18.469482+0000","last_became_peered":"2024-09-21T10:08:18.469482+0000","last_unstale":"2024-09-21T10:20:48.077902+0000","last_undegraded":"2024-09-21T10:20:48.077902+0000","last_fullsized":"2024-09-21T10:20:48.077902+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:35.628724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":324,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.095843+0000","last_change":"2024-09-21T10:18:33.479236+0000","last_active":"2024-09-21T10:20:46.095843+0000","last_peered":"2024-09-21T10:20:46.095843+0000","last_clean":"2024-09-21T10:20:46.095843+0000","last_became_active":"2024-09-21T10:18:33.474256+0000","last_became_peered":"2024-09-21T10:18:33.474256+0000","last_unstale":"2024-09-21T10:20:46.095843+0000","last_undegraded":"2024-09-21T10:20:46.095843+0000","last_fullsized":"2024-09-21T10:20:46.095843+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:19:34.666035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":204,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.097369+0000","last_change":"2024-09-21T10:07:43.791830+0000","last_active":"2024-09-21T10:20:46.097369+0000","last_peered":"2024-09-21T10:20:46.097369+0000","last_clean":"2024-09-21T10:20:46.097369+0000","last_became_active":"2024-09-21T10:07:43.791314+0000","last_became_peered":"2024-09-21T10:07:43.791314+0000","last_unstale":"2024-09-21T10:20:46.097369+0000","last_undegraded":"2024-09-21T10:20:46.097369+0000","last_fullsized":"2024-09-21T10:20:46.097369+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:37.086629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":45,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.043819+0000","last_change":"2024-09-21T10:18:33.464989+0000","last_active":"2024-09-21T10:20:46.043819+0000","last_peered":"2024-09-21T10:20:46.043819+0000","last_clean":"2024-09-21T10:20:46.043819+0000","last_became_active":"2024-09-21T10:18:33.464617+0000","last_became_peered":"2024-09-21T10:18:33.464617+0000","last_unstale":"2024-09-21T10:20:46.043819+0000","last_undegraded":"2024-09-21T10:20:46.043819+0000","last_fullsized":"2024-09-21T10:20:46.043819+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:09.300797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":461,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.070148+0000","last_change":"2024-09-21T10:18:33.472415+0000","last_active":"2024-09-21T10:20:48.070148+0000","last_peered":"2024-09-21T10:20:48.070148+0000","last_clean":"2024-09-21T10:20:48.070148+0000","last_became_active":"2024-09-21T10:18:33.471746+0000","last_became_peered":"2024-09-21T10:18:33.471746+0000","last_unstale":"2024-09-21T10:20:48.070148+0000","last_undegraded":"2024-09-21T10:20:48.070148+0000","last_fullsized":"2024-09-21T10:20:48.070148+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:40:37.099458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":107,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.080584+0000","last_change":"2024-09-21T10:18:33.484721+0000","last_active":"2024-09-21T10:20:48.080584+0000","last_peered":"2024-09-21T10:20:48.080584+0000","last_clean":"2024-09-21T10:20:48.080584+0000","last_became_active":"2024-09-21T10:18:33.483950+0000","last_became_peered":"2024-09-21T10:18:33.483950+0000","last_unstale":"2024-09-21T10:20:48.080584+0000","last_undegraded":"2024-09-21T10:20:48.080584+0000","last_fullsized":"2024-09-21T10:20:48.080584+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:54:52.208378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":256,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.044448+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:20:46.044448+0000","last_peered":"2024-09-21T10:20:46.044448+0000","last_clean":"2024-09-21T10:20:46.044448+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:20:46.044448+0000","last_undegraded":"2024-09-21T10:20:46.044448+0000","last_fullsized":"2024-09-21T10:20:46.044448+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":127,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.075938+0000","last_change":"2024-09-21T10:18:33.479635+0000","last_active":"2024-09-21T10:20:48.075938+0000","last_peered":"2024-09-21T10:20:48.075938+0000","last_clean":"2024-09-21T10:20:48.075938+0000","last_became_active":"2024-09-21T10:18:33.478830+0000","last_became_peered":"2024-09-21T10:18:33.478830+0000","last_unstale":"2024-09-21T10:20:48.075938+0000","last_undegraded":"2024-09-21T10:20:48.075938+0000","last_fullsized":"2024-09-21T10:20:48.075938+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:16.950086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":477,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.070626+0000","last_change":"2024-09-21T10:18:33.480350+0000","last_active":"2024-09-21T10:20:48.070626+0000","last_peered":"2024-09-21T10:20:48.070626+0000","last_clean":"2024-09-21T10:20:48.070626+0000","last_became_active":"2024-09-21T10:18:33.479248+0000","last_became_peered":"2024-09-21T10:18:33.479248+0000","last_unstale":"2024-09-21T10:20:48.070626+0000","last_undegraded":"2024-09-21T10:20:48.070626+0000","last_fullsized":"2024-09-21T10:20:48.070626+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:25:28.780572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":471,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.076860+0000","last_change":"2024-09-21T10:18:33.480341+0000","last_active":"2024-09-21T10:20:48.076860+0000","last_peered":"2024-09-21T10:20:48.076860+0000","last_clean":"2024-09-21T10:20:48.076860+0000","last_became_active":"2024-09-21T10:18:33.479869+0000","last_became_peered":"2024-09-21T10:18:33.479869+0000","last_unstale":"2024-09-21T10:20:48.076860+0000","last_undegraded":"2024-09-21T10:20:48.076860+0000","last_fullsized":"2024-09-21T10:20:48.076860+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:32:42.206910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":231,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.077402+0000","last_change":"2024-09-21T10:18:33.473981+0000","last_active":"2024-09-21T10:20:48.077402+0000","last_peered":"2024-09-21T10:20:48.077402+0000","last_clean":"2024-09-21T10:20:48.077402+0000","last_became_active":"2024-09-21T10:18:33.473619+0000","last_became_peered":"2024-09-21T10:18:33.473619+0000","last_unstale":"2024-09-21T10:20:48.077402+0000","last_undegraded":"2024-09-21T10:20:48.077402+0000","last_fullsized":"2024-09-21T10:20:48.077402+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:44:24.630856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":284,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.078076+0000","last_change":"2024-09-21T10:08:18.471282+0000","last_active":"2024-09-21T10:20:48.078076+0000","last_peered":"2024-09-21T10:20:48.078076+0000","last_clean":"2024-09-21T10:20:48.078076+0000","last_became_active":"2024-09-21T10:08:18.470952+0000","last_became_peered":"2024-09-21T10:08:18.470952+0000","last_unstale":"2024-09-21T10:20:48.078076+0000","last_undegraded":"2024-09-21T10:20:48.078076+0000","last_fullsized":"2024-09-21T10:20:48.078076+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:25.597081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":117,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.083232+0000","last_change":"2024-09-21T10:18:33.480128+0000","last_active":"2024-09-21T10:20:48.083232+0000","last_peered":"2024-09-21T10:20:48.083232+0000","last_clean":"2024-09-21T10:20:48.083232+0000","last_became_active":"2024-09-21T10:18:33.479691+0000","last_became_peered":"2024-09-21T10:18:33.479691+0000","last_unstale":"2024-09-21T10:20:48.083232+0000","last_undegraded":"2024-09-21T10:20:48.083232+0000","last_fullsized":"2024-09-21T10:20:48.083232+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:43:49.311244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":468,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.069815+0000","last_change":"2024-09-21T10:08:00.386104+0000","last_active":"2024-09-21T10:20:48.069815+0000","last_peered":"2024-09-21T10:20:48.069815+0000","last_clean":"2024-09-21T10:20:48.069815+0000","last_became_active":"2024-09-21T10:08:00.381484+0000","last_became_peered":"2024-09-21T10:08:00.381484+0000","last_unstale":"2024-09-21T10:20:48.069815+0000","last_undegraded":"2024-09-21T10:20:48.069815+0000","last_fullsized":"2024-09-21T10:20:48.069815+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:39.190191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":145,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.081488+0000","last_change":"2024-09-21T10:18:33.481322+0000","last_active":"2024-09-21T10:20:48.081488+0000","last_peered":"2024-09-21T10:20:48.081488+0000","last_clean":"2024-09-21T10:20:48.081488+0000","last_became_active":"2024-09-21T10:18:33.480554+0000","last_became_peered":"2024-09-21T10:18:33.480554+0000","last_unstale":"2024-09-21T10:20:48.081488+0000","last_undegraded":"2024-09-21T10:20:48.081488+0000","last_fullsized":"2024-09-21T10:20:48.081488+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:15.498594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":295,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.077121+0000","last_change":"2024-09-21T10:08:18.466882+0000","last_active":"2024-09-21T10:20:48.077121+0000","last_peered":"2024-09-21T10:20:48.077121+0000","last_clean":"2024-09-21T10:20:48.077121+0000","last_became_active":"2024-09-21T10:08:18.466470+0000","last_became_peered":"2024-09-21T10:08:18.466470+0000","last_unstale":"2024-09-21T10:20:48.077121+0000","last_undegraded":"2024-09-21T10:20:48.077121+0000","last_fullsized":"2024-09-21T10:20:48.077121+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:51:31.020051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":145,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.083712+0000","last_change":"2024-09-21T10:18:33.472352+0000","last_active":"2024-09-21T10:20:48.083712+0000","last_peered":"2024-09-21T10:20:48.083712+0000","last_clean":"2024-09-21T10:20:48.083712+0000","last_became_active":"2024-09-21T10:18:33.471312+0000","last_became_peered":"2024-09-21T10:18:33.471312+0000","last_unstale":"2024-09-21T10:20:48.083712+0000","last_undegraded":"2024-09-21T10:20:48.083712+0000","last_fullsized":"2024-09-21T10:20:48.083712+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:32:44.897145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":168,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.044243+0000","last_change":"2024-09-21T10:08:00.394350+0000","last_active":"2024-09-21T10:20:46.044243+0000","last_peered":"2024-09-21T10:20:46.044243+0000","last_clean":"2024-09-21T10:20:46.044243+0000","last_became_active":"2024-09-21T10:08:00.387232+0000","last_became_peered":"2024-09-21T10:08:00.387232+0000","last_unstale":"2024-09-21T10:20:46.044243+0000","last_undegraded":"2024-09-21T10:20:46.044243+0000","last_fullsized":"2024-09-21T10:20:46.044243+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:56.160018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":130,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.082940+0000","last_change":"2024-09-21T10:18:33.474216+0000","last_active":"2024-09-21T10:20:48.082940+0000","last_peered":"2024-09-21T10:20:48.082940+0000","last_clean":"2024-09-21T10:20:48.082940+0000","last_became_active":"2024-09-21T10:18:33.473794+0000","last_became_peered":"2024-09-21T10:18:33.473794+0000","last_unstale":"2024-09-21T10:20:48.082940+0000","last_undegraded":"2024-09-21T10:20:48.082940+0000","last_fullsized":"2024-09-21T10:20:48.082940+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:38.956888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":41,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.044310+0000","last_change":"2024-09-21T10:18:33.483576+0000","last_active":"2024-09-21T10:20:46.044310+0000","last_peered":"2024-09-21T10:20:46.044310+0000","last_clean":"2024-09-21T10:20:46.044310+0000","last_became_active":"2024-09-21T10:18:33.483116+0000","last_became_peered":"2024-09-21T10:18:33.483116+0000","last_unstale":"2024-09-21T10:20:46.044310+0000","last_undegraded":"2024-09-21T10:20:46.044310+0000","last_fullsized":"2024-09-21T10:20:46.044310+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:48.964766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":241,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:20:46.045146+0000","last_change":"2024-09-21T10:18:33.471909+0000","last_active":"2024-09-21T10:20:46.045146+0000","last_peered":"2024-09-21T10:20:46.045146+0000","last_clean":"2024-09-21T10:20:46.045146+0000","last_became_active":"2024-09-21T10:18:33.471614+0000","last_became_peered":"2024-09-21T10:18:33.471614+0000","last_unstale":"2024-09-21T10:20:46.045146+0000","last_undegraded":"2024-09-21T10:20:46.045146+0000","last_fullsized":"2024-09-21T10:20:46.045146+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:30:50.582505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":460,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.070055+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:20:48.070055+0000","last_peered":"2024-09-21T10:20:48.070055+0000","last_clean":"2024-09-21T10:20:48.070055+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:20:48.070055+0000","last_undegraded":"2024-09-21T10:20:48.070055+0000","last_fullsized":"2024-09-21T10:20:48.070055+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":1970274328,"num_objects":30065,"num_object_clones":0,"num_object_copies":90195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18854,"num_objects_misplaced":4219,"num_objects_unfound":0,"num_objects_dirty":30065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177271,"num_write_kb":6634689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5879,"num_bytes_recovered":359006208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7570272256,"data_stored":7570260040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107871,"ondisk_log_size":107871,"up":48,"acting":43,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":10016,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5982,"num_read_kb":13005,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397874,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1747408,"kb_used_data":854344,"kb_used_omap":1,"kb_used_meta":893054,"kb_avail":91997744,"statfs":{"total":95995035648,"available":94205689856,"internally_reserved":0,"allocated":874848256,"data_stored":873252600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":914487640},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594102,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1997772,"kb_used_data":1083972,"kb_used_omap":1,"kb_used_meta":913790,"kb_avail":91747380,"statfs":{"total":95995035648,"available":93949317120,"internally_reserved":0,"allocated":1109987328,"data_stored":1108395741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":935721304},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790332,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1995000,"kb_used_data":1086768,"kb_used_omap":1,"kb_used_meta":908222,"kb_avail":91750152,"statfs":{"total":95995035648,"available":93952155648,"internally_reserved":0,"allocated":1112850432,"data_stored":1111274192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":930019672},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986561,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":433791697123,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860650,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1601156,"kb_used_data":941372,"kb_used_omap":8,"kb_used_meta":659767,"kb_avail":92143996,"statfs":{"total":95995035648,"available":94355451904,"internally_reserved":0,"allocated":963964928,"data_stored":962381424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":675602284},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024175,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1455468,"kb_used_data":882852,"kb_used_omap":7,"kb_used_meta":572600,"kb_avail":92289684,"statfs":{"total":95995035648,"available":94504636416,"internally_reserved":0,"allocated":904040448,"data_stored":902475559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":586342962},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187711,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1639484,"kb_used_data":987380,"kb_used_omap":8,"kb_used_meta":651959,"kb_avail":92105668,"statfs":{"total":95995035648,"available":94316204032,"internally_reserved":0,"allocated":1011077120,"data_stored":1009497662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":667607025},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":983564288,"data_stored":983564288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":882442240,"data_stored":882442240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":930025472,"data_stored":930021400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1743716352,"data_stored":1743716352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1085083648,"data_stored":1085079576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1092354048,"data_stored":1092354048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":853086208,"data_stored":853082136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:21:03.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:21:03.816 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:21:03.816 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:21:03.816 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:21:03.816 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:21:03.816 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:21:03.816 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:21:03.816 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:21:03.816 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:21:03.816 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:21:03.817 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:21:03.817 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:21:03.817 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:21:03.817 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:21:03.817 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:21:03.817 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:21:03.817 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:21:03.827 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:03.861 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:03.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:03.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:03.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:04.011 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:04.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:04.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:04.113 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:04.142 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:04.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:04.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:04.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:04.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:04.275 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:04.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:04.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:04.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:04 smithi196 ceph-mon[84784]: from='client.45938 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:21:04.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:04 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4245977446' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:21:04.423 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:04.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:04.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:04.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:04.567 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:04.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:04.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:04.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:04 smithi098 ceph-mon[97580]: from='client.45938 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:21:04.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:04 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4245977446' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:21:04.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:04 smithi098 ceph-mon[100914]: from='client.45938 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:21:04.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:04 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4245977446' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:21:04.669 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:04.703 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:04.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:04.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:04.805 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:04.844 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:04.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:04.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:04.945 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:04.985 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:04.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:04.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:05.087 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:05.124 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:05.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:05.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:05.226 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:05.252 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:05.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:05.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:05.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:05 smithi098 ceph-mon[97580]: pgmap v947: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.7 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 933 op/s; 18618/81423 objects degraded (22.866%); 4219/81423 objects misplaced (5.182%); 91 KiB/s, 54 objects/s recovering 2024-09-21T10:21:05.283 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:05 smithi098 ceph-mon[100914]: pgmap v947: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.7 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 933 op/s; 18618/81423 objects degraded (22.866%); 4219/81423 objects misplaced (5.182%); 91 KiB/s, 54 objects/s recovering 2024-09-21T10:21:05.300 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:05.304 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:05.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:05.381 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:05.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:05.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:05.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:05 smithi196 ceph-mon[84784]: pgmap v947: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.7 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 933 op/s; 18618/81423 objects degraded (22.866%); 4219/81423 objects misplaced (5.182%); 91 KiB/s, 54 objects/s recovering 2024-09-21T10:21:05.483 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:05.511 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:05.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:05.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:05.612 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:05.645 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:05.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:05.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:05.746 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:05.783 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:05.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:05.885 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:05.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:05.924 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:05.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:06.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:06.045 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:06.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:06.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:06.129 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:21:06.147 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:06.172 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:06.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:06.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:06.274 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:06.306 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:06.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:06.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:06.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:06 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3356707659' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:21:06.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:06 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:21:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:06 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 18618/81423 objects degraded (22.866%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:06.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:06.429 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:06.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:06.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:06.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:06 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3356707659' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:21:06.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:06 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:21:06.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:06 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 18618/81423 objects degraded (22.866%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:06.498 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:06 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3356707659' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:21:06.499 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:06 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:21:06.499 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:06 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 18618/81423 objects degraded (22.866%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:06.531 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:06.565 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:06.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:06.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:06.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:06.697 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:06.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:06.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:06.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:21:06.800 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:06.824 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:06.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:06.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:06.926 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:06.970 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:06.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:06.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:07.071 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:07.113 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:07.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:07.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:07.215 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:07.247 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:07.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:07.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:07.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:07.385 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:07.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:07.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:07.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:07 smithi196 ceph-mon[84784]: pgmap v948: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.6 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 809 op/s; 18618/77289 objects degraded (24.089%); 4219/77289 objects misplaced (5.459%); 0 B/s, 38 objects/s recovering 2024-09-21T10:21:07.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:07 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:21:07.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:07 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:21:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:07 smithi196 ceph-mon[84784]: osdmap e294: 8 total, 8 up, 6 in 2024-09-21T10:21:07.487 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:07.517 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:07 smithi098 ceph-mon[97580]: pgmap v948: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.6 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 809 op/s; 18618/77289 objects degraded (24.089%); 4219/77289 objects misplaced (5.459%); 0 B/s, 38 objects/s recovering 2024-09-21T10:21:07.517 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:07 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:21:07.517 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:07 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:21:07.517 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:07 smithi098 ceph-mon[97580]: osdmap e294: 8 total, 8 up, 6 in 2024-09-21T10:21:07.517 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:07 smithi098 ceph-mon[100914]: pgmap v948: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.6 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 809 op/s; 18618/77289 objects degraded (24.089%); 4219/77289 objects misplaced (5.459%); 0 B/s, 38 objects/s recovering 2024-09-21T10:21:07.517 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:07 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:21:07.518 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:07 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:21:07.518 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:07 smithi098 ceph-mon[100914]: osdmap e294: 8 total, 8 up, 6 in 2024-09-21T10:21:07.528 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:07.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:07.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:07.630 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:07.664 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:07.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:07.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:07.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:07.802 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:07.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:07.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:07.904 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:07.935 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:07.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:07.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:08.037 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:08.079 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:08.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:08.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:08.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:08.222 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:08.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:08.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:08.324 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:08.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:21:08.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:08.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:21:08.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:08.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:08.360 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:08.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:08.461 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:08.482 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:08.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:08.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:08.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:08.617 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:08.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:08.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:08.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:21:08.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:08.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:08.753 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:08.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:08.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:08.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:08.890 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:08.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:08.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:08.992 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:09.029 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:09.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:09.131 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:09.166 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:09.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:09.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:09.268 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:09.302 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:09.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:09.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:09.404 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:09.450 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:09.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:09.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:09.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:09 smithi196 ceph-mon[84784]: pgmap v950: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.5 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 833 op/s; 18373/72882 objects degraded (25.209%); 4219/72882 objects misplaced (5.789%); 0 B/s, 48 objects/s recovering 2024-09-21T10:21:09.552 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:09.585 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:09.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:09.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:09.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:09 smithi098 ceph-mon[97580]: pgmap v950: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.5 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 833 op/s; 18373/72882 objects degraded (25.209%); 4219/72882 objects misplaced (5.789%); 0 B/s, 48 objects/s recovering 2024-09-21T10:21:09.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:09 smithi098 ceph-mon[100914]: pgmap v950: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.5 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 833 op/s; 18373/72882 objects degraded (25.209%); 4219/72882 objects misplaced (5.789%); 0 B/s, 48 objects/s recovering 2024-09-21T10:21:09.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:09.727 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:09.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:09.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:09.829 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:09.856 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:09.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:09.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:09.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:09.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:09.994 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:09.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:10.095 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:10.130 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:10.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:10.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:10.232 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:10.268 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:10.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:10.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:10.370 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:10.398 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:10.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:10.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:10.499 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:10.532 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:10.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:10.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:10.633 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:10.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:21:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:21:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:21:10.672 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:10.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:10.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:10.773 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:10.806 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:10.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:10.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:10.908 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:10.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:10.943 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:10.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:11.044 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:11.082 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:11.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:11.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:11.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:21:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:21:10] "GET /metrics HTTP/1.1" 200 36528 "" "Prometheus/2.51.0" 2024-09-21T10:21:11.183 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:11.223 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:11.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:11.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:11.324 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:11.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:11.361 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:11.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:11.462 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:11.494 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:11.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:11.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:11.595 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:11.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:11 smithi098 ceph-mon[97580]: pgmap v951: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.4 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 842 op/s; 18373/66588 objects degraded (27.592%); 4219/66588 objects misplaced (6.336%); 0 B/s, 48 objects/s recovering 2024-09-21T10:21:11.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:11 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 18373/66588 objects degraded (27.592%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:11.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:11 smithi098 ceph-mon[100914]: pgmap v951: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.4 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 842 op/s; 18373/66588 objects degraded (27.592%); 4219/66588 objects misplaced (6.336%); 0 B/s, 48 objects/s recovering 2024-09-21T10:21:11.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:11 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 18373/66588 objects degraded (27.592%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:11.616 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:11.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:11.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:11.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:11 smithi196 ceph-mon[84784]: pgmap v951: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.4 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 842 op/s; 18373/66588 objects degraded (27.592%); 4219/66588 objects misplaced (6.336%); 0 B/s, 48 objects/s recovering 2024-09-21T10:21:11.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:11 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 18373/66588 objects degraded (27.592%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:11.718 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:11.754 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:11.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:11.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:11.856 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:11.885 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:11.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:11.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:11.986 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:12.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:12.019 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:12.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:12.120 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:12.160 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:12.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:12.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:12.261 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:12.289 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:12.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:12.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:12.391 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:12.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:12.418 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:12.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:12.519 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:12.554 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:12.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:12.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:12.656 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:12.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:12.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:12.800 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:12.827 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:12.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:12.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:12.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:12.966 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:12.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:12.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:13.068 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:13.099 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:13.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:13.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:13.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:13.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:13.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:13.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:13.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:13.373 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:13.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:13.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:13.509 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:13.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:13.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:13.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:13 smithi098 ceph-mon[97580]: pgmap v952: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.3 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 837 op/s; 18373/65151 objects degraded (28.201%); 4219/65151 objects misplaced (6.476%); 0 B/s, 48 objects/s recovering 2024-09-21T10:21:13.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:13 smithi098 ceph-mon[100914]: pgmap v952: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.3 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 837 op/s; 18373/65151 objects degraded (28.201%); 4219/65151 objects misplaced (6.476%); 0 B/s, 48 objects/s recovering 2024-09-21T10:21:13.611 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:13.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:13 smithi196 ceph-mon[84784]: pgmap v952: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.3 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 837 op/s; 18373/65151 objects degraded (28.201%); 4219/65151 objects misplaced (6.476%); 0 B/s, 48 objects/s recovering 2024-09-21T10:21:13.639 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:13.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:13.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:13.741 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:13.774 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:13.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:13.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:13.876 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:13.915 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:13.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:13.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:14.017 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:14.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:14.053 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:14.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:14.154 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:14.190 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:14.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:14.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:14.291 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:14.324 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:14.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:14.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:14.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:14.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:14.464 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:14.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:14.565 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:14.599 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:14.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:14.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:14.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:21:14.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:14.757 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:14.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:14.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:14.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:14.889 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:14.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:14.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:14.991 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:15.029 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:15.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:15.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:15.130 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:15.166 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:15.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:15.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:15.267 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:15.302 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:15.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:15.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:15.404 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:15.434 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:15.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:15.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:15.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:15.576 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:15.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:15.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:15.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:15 smithi098 ceph-mon[97580]: pgmap v953: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.2 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 828 op/s; 18142/56556 objects degraded (32.078%); 4219/56556 objects misplaced (7.460%); 0 B/s, 47 objects/s recovering 2024-09-21T10:21:15.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:15 smithi098 ceph-mon[100914]: pgmap v953: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.2 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 828 op/s; 18142/56556 objects degraded (32.078%); 4219/56556 objects misplaced (7.460%); 0 B/s, 47 objects/s recovering 2024-09-21T10:21:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:15 smithi196 ceph-mon[84784]: pgmap v953: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.2 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 828 op/s; 18142/56556 objects degraded (32.078%); 4219/56556 objects misplaced (7.460%); 0 B/s, 47 objects/s recovering 2024-09-21T10:21:15.678 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:15.718 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:15.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:15.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:15.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:15.850 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:15.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:15.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:15.952 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:15.985 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:15.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:15.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:16.087 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:16.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:16.128 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:16.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:16.229 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:16.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:16.254 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:16.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:16.355 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:16.400 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:16.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:16.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:16.501 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:16.534 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:16.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:16.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:16.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:16 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 18142/56556 objects degraded (32.078%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:16.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:16 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 18142/56556 objects degraded (32.078%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:16.636 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:16.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:16 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 18142/56556 objects degraded (32.078%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:16.668 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:16.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:16.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:16.769 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:16.806 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:16.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:16.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:16.908 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:16.943 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:16.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:16.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:17.044 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:17.075 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:17.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:17.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:17.177 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:17.221 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:17.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:17.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:17.323 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:17.361 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:17.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:17.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:17.462 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:17.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:17.494 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:17.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:17.595 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:17.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:17 smithi098 ceph-mon[97580]: pgmap v954: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.1 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 816 op/s; 18142/52776 objects degraded (34.375%); 4219/52776 objects misplaced (7.994%); 0 B/s, 47 objects/s recovering 2024-09-21T10:21:17.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:17 smithi098 ceph-mon[100914]: pgmap v954: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.1 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 816 op/s; 18142/52776 objects degraded (34.375%); 4219/52776 objects misplaced (7.994%); 0 B/s, 47 objects/s recovering 2024-09-21T10:21:17.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:17.623 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:17.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:17.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:17 smithi196 ceph-mon[84784]: pgmap v954: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.1 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 816 op/s; 18142/52776 objects degraded (34.375%); 4219/52776 objects misplaced (7.994%); 0 B/s, 47 objects/s recovering 2024-09-21T10:21:17.724 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:17.756 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:17.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:17.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:17.858 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:17.896 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:17.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:17.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:17.998 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:18.032 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:18.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:18.133 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:18.169 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:18.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:18.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:18.270 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:18.292 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:18.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:18.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:18.393 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:18.421 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:18.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:18.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:18.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:18.564 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:18.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:18.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:18.665 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:18.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:18.701 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:18.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:18.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:18.846 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:18.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:18.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:18.948 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:18.975 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:18.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:18.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:19.077 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:19.113 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:19.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:19.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:19.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:19.249 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:19.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:19.349 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:19.388 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:19.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:19.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:19.490 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:19.515 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:19.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:19.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:19.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:19 smithi098 ceph-mon[97580]: pgmap v955: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1002 MiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 865 op/s; 17879/47967 objects degraded (37.274%); 4219/47967 objects misplaced (8.796%); 0 B/s, 65 objects/s recovering 2024-09-21T10:21:19.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:19 smithi098 ceph-mon[100914]: pgmap v955: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1002 MiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 865 op/s; 17879/47967 objects degraded (37.274%); 4219/47967 objects misplaced (8.796%); 0 B/s, 65 objects/s recovering 2024-09-21T10:21:19.616 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:19.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:19 smithi196 ceph-mon[84784]: pgmap v955: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1002 MiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 865 op/s; 17879/47967 objects degraded (37.274%); 4219/47967 objects misplaced (8.796%); 0 B/s, 65 objects/s recovering 2024-09-21T10:21:19.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:19.643 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:19.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:19.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:19.784 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:19.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:19.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:19.876 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:19.887 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:19.911 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:19.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:19.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:20.013 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:20.041 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:20.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:20.143 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:20.201 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:20.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:20.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:20.303 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:20.341 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:20.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:20.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:20.443 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:20.449 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:21:20.449 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:21:20.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.450 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:196005 8.4 8:2c3f84c7:::benchmark_data_smithi098_226587_object92335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-21T10:21:20.450 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:21:20.413085+0000", 2024-09-21T10:21:20.450 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.032001941999999999, 2024-09-21T10:21:20.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.032037920999999997, 2024-09-21T10:21:20.450 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:21:20.450 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:21:20.450 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:21:20.450 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:21:20.451 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:21:20.451 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:21:20.451 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 196005 2024-09-21T10:21:20.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.451 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:21:20.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:21:20.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.413085+0000", 2024-09-21T10:21:20.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:21:20.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:21:20.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.413085+0000", 2024-09-21T10:21:20.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:21:20.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:21:20.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.413086+0000", 2024-09-21T10:21:20.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4890000000000001e-06 2024-09-21T10:21:20.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:21:20.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.413091+0000", 2024-09-21T10:21:20.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9699999999999998e-06 2024-09-21T10:21:20.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:21:20.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.413104+0000", 2024-09-21T10:21:20.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2709e-05 2024-09-21T10:21:20.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:21:20.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.413121+0000", 2024-09-21T10:21:20.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7173999999999999e-05 2024-09-21T10:21:20.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:21:20.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.413177+0000", 2024-09-21T10:21:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5427000000000003e-05 2024-09-21T10:21:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:21:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.413274+0000", 2024-09-21T10:21:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7657000000000004e-05 2024-09-21T10:21:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:21:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.413387+0000", 2024-09-21T10:21:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000112583 2024-09-21T10:21:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.423258+0000", 2024-09-21T10:21:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0098713059999999998 2024-09-21T10:21:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:21:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.423273+0000", 2024-09-21T10:21:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4448e-05 2024-09-21T10:21:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:21:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:21:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:21:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:196007 8.4 8:20b43d76:::benchmark_data_smithi098_226587_object92337:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-21T10:21:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:21:20.417878+0000", 2024-09-21T10:21:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.027209028999999999, 2024-09-21T10:21:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027320665000000001, 2024-09-21T10:21:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:21:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:21:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:21:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:21:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:21:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:21:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 196007 2024-09-21T10:21:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:21:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:21:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.417878+0000", 2024-09-21T10:21:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:21:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:21:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.417878+0000", 2024-09-21T10:21:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:21:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:21:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.417879+0000", 2024-09-21T10:21:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-21T10:21:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:21:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.417884+0000", 2024-09-21T10:21:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2989999999999996e-06 2024-09-21T10:21:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:21:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.417900+0000", 2024-09-21T10:21:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5913000000000002e-05 2024-09-21T10:21:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:21:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.417909+0000", 2024-09-21T10:21:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5750000000000001e-06 2024-09-21T10:21:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:21:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.418005+0000", 2024-09-21T10:21:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5750999999999999e-05 2024-09-21T10:21:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:21:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.418116+0000", 2024-09-21T10:21:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011136 2024-09-21T10:21:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:21:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.418221+0000", 2024-09-21T10:21:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105006 2024-09-21T10:21:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:21:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.423277+0000", 2024-09-21T10:21:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050557800000000002 2024-09-21T10:21:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.442282+0000", 2024-09-21T10:21:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019005305 2024-09-21T10:21:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:21:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:21:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:21:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:196021 8.4 8:284c9636:::benchmark_data_smithi098_226587_object92351:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-21T10:21:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:21:20.430497+0000", 2024-09-21T10:21:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.014590047, 2024-09-21T10:21:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01476948, 2024-09-21T10:21:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:21:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:21:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:21:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:21:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:21:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:21:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 196021 2024-09-21T10:21:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:21:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:21:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.430497+0000", 2024-09-21T10:21:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:21:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:21:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.430497+0000", 2024-09-21T10:21:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:21:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:21:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.430499+0000", 2024-09-21T10:21:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8169999999999999e-06 2024-09-21T10:21:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:21:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.430504+0000", 2024-09-21T10:21:20.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4e-06 2024-09-21T10:21:20.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:21:20.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.430521+0000", 2024-09-21T10:21:20.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7085e-05 2024-09-21T10:21:20.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:21:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.430529+0000", 2024-09-21T10:21:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7680000000000008e-06 2024-09-21T10:21:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:21:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.430630+0000", 2024-09-21T10:21:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101277 2024-09-21T10:21:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:21:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.430744+0000", 2024-09-21T10:21:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000113717 2024-09-21T10:21:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:21:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.430861+0000", 2024-09-21T10:21:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011692300000000001 2024-09-21T10:21:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.442332+0000", 2024-09-21T10:21:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011470787 2024-09-21T10:21:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:21:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:21:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:21:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:196022 8.4 8:2615a936:::benchmark_data_smithi098_226587_object92352:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-21T10:21:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:21:20.430581+0000", 2024-09-21T10:21:20.472 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.014505354, 2024-09-21T10:21:20.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014746957999999999, 2024-09-21T10:21:20.472 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:21:20.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:21:20.472 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:21:20.472 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:21:20.472 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:21:20.472 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:21:20.472 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 196022 2024-09-21T10:21:20.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.473 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:21:20.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:21:20.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.430581+0000", 2024-09-21T10:21:20.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:21:20.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:21:20.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.430581+0000", 2024-09-21T10:21:20.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:21:20.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:21:20.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.430582+0000", 2024-09-21T10:21:20.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6000000000000004e-07 2024-09-21T10:21:20.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:21:20.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.430585+0000", 2024-09-21T10:21:20.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7379999999999999e-06 2024-09-21T10:21:20.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:21:20.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.430591+0000", 2024-09-21T10:21:20.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6889999999999998e-06 2024-09-21T10:21:20.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:21:20.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.430596+0000", 2024-09-21T10:21:20.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9180000000000002e-06 2024-09-21T10:21:20.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:21:20.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.437518+0000", 2024-09-21T10:21:20.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0069214389999999997 2024-09-21T10:21:20.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:21:20.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.437650+0000", 2024-09-21T10:21:20.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013244099999999999 2024-09-21T10:21:20.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:21:20.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.437783+0000", 2024-09-21T10:21:20.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013283300000000001 2024-09-21T10:21:20.477 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:21:20.477 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:21:20.477 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:21:20.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.478 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:196028 8.4 8:267e0b2b:::benchmark_data_smithi098_226587_object92358:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-21T10:21:20.478 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:21:20.439438+0000", 2024-09-21T10:21:20.478 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0056484530000000003, 2024-09-21T10:21:20.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0059450780000000003, 2024-09-21T10:21:20.478 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:21:20.478 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:21:20.478 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:21:20.478 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:21:20.478 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:21:20.479 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:21:20.479 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 196028 2024-09-21T10:21:20.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.479 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:21:20.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:21:20.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.439438+0000", 2024-09-21T10:21:20.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:21:20.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:21:20.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.439438+0000", 2024-09-21T10:21:20.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:21:20.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:21:20.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.439440+0000", 2024-09-21T10:21:20.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4300000000000001e-06 2024-09-21T10:21:20.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:21:20.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.439444+0000", 2024-09-21T10:21:20.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5040000000000004e-06 2024-09-21T10:21:20.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:21:20.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.439455+0000", 2024-09-21T10:21:20.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0821e-05 2024-09-21T10:21:20.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:21:20.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.439462+0000", 2024-09-21T10:21:20.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1629999999999999e-06 2024-09-21T10:21:20.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:21:20.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.442433+0000", 2024-09-21T10:21:20.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002971193 2024-09-21T10:21:20.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:21:20.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.442701+0000", 2024-09-21T10:21:20.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00026801800000000002 2024-09-21T10:21:20.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:21:20.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.443040+0000", 2024-09-21T10:21:20.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00033879400000000001 2024-09-21T10:21:20.485 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:21:20.485 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:21:20.485 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:21:20.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.485 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.45794.0:196029 8.d e294/265 8:b4d645e7:::benchmark_data_smithi098_226587_object92359:head v 294'12297, mlcod=294'12295)", 2024-09-21T10:21:20.485 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:21:20.443965+0000", 2024-09-21T10:21:20.486 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.001121971, 2024-09-21T10:21:20.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001473928, 2024-09-21T10:21:20.486 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:21:20.486 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:21:20.486 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:21:20.486 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:21:20.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:21:20.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.443965+0000", 2024-09-21T10:21:20.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:21:20.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:21:20.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.443965+0000", 2024-09-21T10:21:20.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:21:20.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:21:20.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.443966+0000", 2024-09-21T10:21:20.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2399999999999997e-07 2024-09-21T10:21:20.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:21:20.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.443971+0000", 2024-09-21T10:21:20.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8479999999999997e-06 2024-09-21T10:21:20.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:21:20.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.443982+0000", 2024-09-21T10:21:20.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0969e-05 2024-09-21T10:21:20.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:21:20.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.443988+0000", 2024-09-21T10:21:20.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4399999999999996e-06 2024-09-21T10:21:20.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:21:20.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.444045+0000", 2024-09-21T10:21:20.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6977e-05 2024-09-21T10:21:20.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:21:20.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:21:20.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:21:20.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:21:20.444065+0000", 2024-09-21T10:21:20.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0191e-05 2024-09-21T10:21:20.491 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:21:20.492 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:21:20.492 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:21:20.492 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:21:20.492 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-21T10:21:20.492 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 6 2024-09-21T10:21:20.492 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:21:20.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:20.494 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:20.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:20.595 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:20.628 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:20.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:20.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:20.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:21:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:21:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:21:20.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:20.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:20.758 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:20.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:20.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:21:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:21:20] "GET /metrics HTTP/1.1" 200 36528 "" "Prometheus/2.51.0" 2024-09-21T10:21:20.858 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:20.885 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:20.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:20.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:20.987 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:21.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:21.031 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:21.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:21.132 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:21.167 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:21.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:21.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:21.268 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:21.303 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:21.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:21.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:21 smithi098 ceph-mon[97580]: pgmap v956: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 871 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 866 op/s; 17879/41667 objects degraded (42.909%); 4219/41667 objects misplaced (10.126%); 0 B/s, 41 objects/s recovering 2024-09-21T10:21:21.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:21 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 17879/41667 objects degraded (42.909%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:21.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:21 smithi098 ceph-mon[100914]: pgmap v956: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 871 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 866 op/s; 17879/41667 objects degraded (42.909%); 4219/41667 objects misplaced (10.126%); 0 B/s, 41 objects/s recovering 2024-09-21T10:21:21.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:21 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 17879/41667 objects degraded (42.909%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:21.405 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:21.441 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:21.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:21.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:21.543 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:21.579 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:21.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:21.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:21.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:21 smithi196 ceph-mon[84784]: pgmap v956: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 871 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 866 op/s; 17879/41667 objects degraded (42.909%); 4219/41667 objects misplaced (10.126%); 0 B/s, 41 objects/s recovering 2024-09-21T10:21:21.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:21 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 17879/41667 objects degraded (42.909%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:21.662 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:21:21.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:21.750 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:21.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:21.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:21.770 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:21:21.774 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:21:21.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:21.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:21.895 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:21.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:21.996 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:22.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:22.038 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:22.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:22.139 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:22.171 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:22.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:22.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:22.183 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:22.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:22.309 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:22.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:22.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:22.365 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:22 smithi098 ceph-mon[97580]: from='client.45953 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:21:22.365 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:22.365 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:22 smithi098 ceph-mon[100914]: from='client.45953 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:21:22.365 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:22.411 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:22.445 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:22.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:22.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:22.547 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:22.578 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:22.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:22.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:22.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:22 smithi196 ceph-mon[84784]: from='client.45953 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:21:22.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:22.680 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:22.709 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:22.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:22.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:22.811 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:22.849 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:22.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:22.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:22.950 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:22.984 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:22.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:22.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:23.086 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:23.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:23.116 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:23.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:23.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:23.262 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:23.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:23.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:23.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:23.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:23.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:23.428 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:23 smithi098 ceph-mon[97580]: pgmap v957: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 839 MiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 734 op/s; 17879/40143 objects degraded (44.538%); 4219/40143 objects misplaced (10.510%); 0 B/s, 41 objects/s recovering 2024-09-21T10:21:23.429 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:23 smithi098 ceph-mon[100914]: pgmap v957: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 839 MiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 734 op/s; 17879/40143 objects degraded (44.538%); 4219/40143 objects misplaced (10.510%); 0 B/s, 41 objects/s recovering 2024-09-21T10:21:23.502 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:23.538 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:23.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:23.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:23.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:23 smithi196 ceph-mon[84784]: pgmap v957: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 839 MiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 734 op/s; 17879/40143 objects degraded (44.538%); 4219/40143 objects misplaced (10.510%); 0 B/s, 41 objects/s recovering 2024-09-21T10:21:23.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:23.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:23.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:23.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:23.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:23.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:23.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:23.913 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:23.965 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:23.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:23.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:24.067 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:24.108 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:24.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:24.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:24.209 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:24.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:24.250 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:24.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:24.351 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:24.384 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:24.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:24.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:24.486 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:24.519 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:24.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:24.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:24.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:24 smithi098 ceph-mon[97580]: pgmap v958: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 679 MiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 908 op/s; 17617/32445 objects degraded (54.298%); 4219/32445 objects misplaced (13.004%); 0 B/s, 62 objects/s recovering 2024-09-21T10:21:24.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:24 smithi098 ceph-mon[100914]: pgmap v958: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 679 MiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 908 op/s; 17617/32445 objects degraded (54.298%); 4219/32445 objects misplaced (13.004%); 0 B/s, 62 objects/s recovering 2024-09-21T10:21:24.620 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:24.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:24 smithi196 ceph-mon[84784]: pgmap v958: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 679 MiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 908 op/s; 17617/32445 objects degraded (54.298%); 4219/32445 objects misplaced (13.004%); 0 B/s, 62 objects/s recovering 2024-09-21T10:21:24.654 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:24.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:24.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:24.755 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:24.785 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:24.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:24.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:24.887 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:24.934 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:24.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:24.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:25.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:25.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:25.076 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:25.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:25.176 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:25.217 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:25.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:25.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:25.321 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:25.360 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:25.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:25.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:25.462 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:25.497 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:25.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:25.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:25.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:25.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:25.634 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:25.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:25 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:21:25.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:25.775 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:25.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:25.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:25.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:25 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:21:25.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:25 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:21:25.876 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:25.924 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:25.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:25.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:26.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:26.062 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:26.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:26.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:26.163 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:26.206 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:26.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:26.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:26.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:26.339 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:26.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:26.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:26.441 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:26.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:26.471 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:26.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:26.572 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:26.608 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:26.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:26.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:26.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:26 smithi196 ceph-mon[84784]: pgmap v959: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 583 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 797 op/s; 17617/27834 objects degraded (63.293%); 4219/27834 objects misplaced (15.158%); 0 B/s, 43 objects/s recovering 2024-09-21T10:21:26.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:26 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 17617/32445 objects degraded (54.298%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:26.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:26.745 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:26.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:26.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:26.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:26 smithi098 ceph-mon[97580]: pgmap v959: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 583 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 797 op/s; 17617/27834 objects degraded (63.293%); 4219/27834 objects misplaced (15.158%); 0 B/s, 43 objects/s recovering 2024-09-21T10:21:26.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:26 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 17617/32445 objects degraded (54.298%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:26.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:26 smithi098 ceph-mon[100914]: pgmap v959: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 583 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 797 op/s; 17617/27834 objects degraded (63.293%); 4219/27834 objects misplaced (15.158%); 0 B/s, 43 objects/s recovering 2024-09-21T10:21:26.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:26 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 17617/32445 objects degraded (54.298%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:26.847 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:26.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:26.886 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:26.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:26.987 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:27.025 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:27.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:27.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:27.084 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":956,"stamp":"2024-09-21T10:21:19.422237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":913119205,"num_objects":13889,"num_object_clones":0,"num_object_copies":41667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17879,"num_objects_misplaced":4219,"num_objects_unfound":0,"num_objects_dirty":13889,"num_whiteouts":0,"num_read":6289,"num_read_kb":13765,"num_write":194531,"num_write_kb":6648940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6936,"num_bytes_recovered":359606592,"num_keys_recovered":30,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109107,"ondisk_log_size":109107,"up":267,"acting":262,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":7352992,"kb_used_data":2735408,"kb_used_omap":28,"kb_used_meta":4617443,"kb_avail":555117920,"statfs":{"total":575970213888,"available":568440750080,"internally_reserved":0,"allocated":2801057792,"data_stored":2791519226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29289,"internal_metadata":4728262039},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,2,6,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-681902080,"num_objects":-10405,"num_object_clones":0,"num_object_copies":-31215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-494,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10405,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":494,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003523"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":111,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.147609+0000","last_change":"2024-09-21T10:18:33.471373+0000","last_active":"2024-09-21T10:21:06.147609+0000","last_peered":"2024-09-21T10:21:06.147609+0000","last_clean":"2024-09-21T10:21:06.147609+0000","last_became_active":"2024-09-21T10:18:33.471052+0000","last_became_peered":"2024-09-21T10:18:33.471052+0000","last_unstale":"2024-09-21T10:21:06.147609+0000","last_undegraded":"2024-09-21T10:21:06.147609+0000","last_fullsized":"2024-09-21T10:21:06.147609+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:42:19.491129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":308,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.074203+0000","last_change":"2024-09-21T10:08:00.406360+0000","last_active":"2024-09-21T10:20:48.074203+0000","last_peered":"2024-09-21T10:20:48.074203+0000","last_clean":"2024-09-21T10:20:48.074203+0000","last_became_active":"2024-09-21T10:08:00.380042+0000","last_became_peered":"2024-09-21T10:08:00.380042+0000","last_unstale":"2024-09-21T10:20:48.074203+0000","last_undegraded":"2024-09-21T10:20:48.074203+0000","last_fullsized":"2024-09-21T10:20:48.074203+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:49:46.544051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":224,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.076821+0000","last_change":"2024-09-21T10:18:33.465357+0000","last_active":"2024-09-21T10:20:48.076821+0000","last_peered":"2024-09-21T10:20:48.076821+0000","last_clean":"2024-09-21T10:20:48.076821+0000","last_became_active":"2024-09-21T10:18:33.464952+0000","last_became_peered":"2024-09-21T10:18:33.464952+0000","last_unstale":"2024-09-21T10:20:48.076821+0000","last_undegraded":"2024-09-21T10:20:48.076821+0000","last_fullsized":"2024-09-21T10:20:48.076821+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:15:41.090866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":100,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.073813+0000","last_change":"2024-09-21T10:18:33.475291+0000","last_active":"2024-09-21T10:20:48.073813+0000","last_peered":"2024-09-21T10:20:48.073813+0000","last_clean":"2024-09-21T10:20:48.073813+0000","last_became_active":"2024-09-21T10:18:33.474971+0000","last_became_peered":"2024-09-21T10:18:33.474971+0000","last_unstale":"2024-09-21T10:20:48.073813+0000","last_undegraded":"2024-09-21T10:20:48.073813+0000","last_fullsized":"2024-09-21T10:20:48.073813+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:39:40.179668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":463,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.141971+0000","last_change":"2024-09-21T10:18:33.475179+0000","last_active":"2024-09-21T10:21:06.141971+0000","last_peered":"2024-09-21T10:21:06.141971+0000","last_clean":"2024-09-21T10:21:06.141971+0000","last_became_active":"2024-09-21T10:18:33.474914+0000","last_became_peered":"2024-09-21T10:18:33.474914+0000","last_unstale":"2024-09-21T10:21:06.141971+0000","last_undegraded":"2024-09-21T10:21:06.141971+0000","last_fullsized":"2024-09-21T10:21:06.141971+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:15:32.186986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":174,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.072479+0000","last_change":"2024-09-21T10:08:00.385671+0000","last_active":"2024-09-21T10:20:48.072479+0000","last_peered":"2024-09-21T10:20:48.072479+0000","last_clean":"2024-09-21T10:20:48.072479+0000","last_became_active":"2024-09-21T10:08:00.385368+0000","last_became_peered":"2024-09-21T10:08:00.385368+0000","last_unstale":"2024-09-21T10:20:48.072479+0000","last_undegraded":"2024-09-21T10:20:48.072479+0000","last_fullsized":"2024-09-21T10:20:48.072479+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:21.799149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":111,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.160384+0000","last_change":"2024-09-21T10:18:33.481890+0000","last_active":"2024-09-21T10:21:06.160384+0000","last_peered":"2024-09-21T10:21:06.160384+0000","last_clean":"2024-09-21T10:21:06.160384+0000","last_became_active":"2024-09-21T10:18:33.480639+0000","last_became_peered":"2024-09-21T10:18:33.480639+0000","last_unstale":"2024-09-21T10:21:06.160384+0000","last_undegraded":"2024-09-21T10:21:06.160384+0000","last_fullsized":"2024-09-21T10:21:06.160384+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:44.797540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":266,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.074804+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:20:48.074804+0000","last_peered":"2024-09-21T10:20:48.074804+0000","last_clean":"2024-09-21T10:20:48.074804+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:20:48.074804+0000","last_undegraded":"2024-09-21T10:20:48.074804+0000","last_fullsized":"2024-09-21T10:20:48.074804+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"294'11952","reported_seq":12021,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:14.560374+0000","last_change":"2024-09-21T10:17:44.113987+0000","last_active":"2024-09-21T10:21:14.560374+0000","last_peered":"2024-09-21T10:21:14.560374+0000","last_clean":"2024-09-21T10:21:14.560374+0000","last_became_active":"2024-09-21T10:17:44.113680+0000","last_became_peered":"2024-09-21T10:17:44.113680+0000","last_unstale":"2024-09-21T10:21:14.560374+0000","last_undegraded":"2024-09-21T10:21:14.560374+0000","last_fullsized":"2024-09-21T10:21:14.560374+0000","mapping_epoch":265,"log_start":"272'3100","ondisk_log_start":"272'3100","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":8852,"log_dups_size":0,"ondisk_log_size":8852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:38.165184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67371008,"num_objects":1028,"num_object_clones":0,"num_object_copies":3084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11952,"num_write_kb":415360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":301,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.074340+0000","last_change":"2024-09-21T10:08:00.392499+0000","last_active":"2024-09-21T10:20:48.074340+0000","last_peered":"2024-09-21T10:20:48.074340+0000","last_clean":"2024-09-21T10:20:48.074340+0000","last_became_active":"2024-09-21T10:08:00.379165+0000","last_became_peered":"2024-09-21T10:08:00.379165+0000","last_unstale":"2024-09-21T10:20:48.074340+0000","last_undegraded":"2024-09-21T10:20:48.074340+0000","last_fullsized":"2024-09-21T10:20:48.074340+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:18.933144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":157,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.073744+0000","last_change":"2024-09-21T10:07:43.794677+0000","last_active":"2024-09-21T10:20:48.073744+0000","last_peered":"2024-09-21T10:20:48.073744+0000","last_clean":"2024-09-21T10:20:48.073744+0000","last_became_active":"2024-09-21T10:07:43.794348+0000","last_became_peered":"2024-09-21T10:07:43.794348+0000","last_unstale":"2024-09-21T10:20:48.073744+0000","last_undegraded":"2024-09-21T10:20:48.073744+0000","last_fullsized":"2024-09-21T10:20:48.073744+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:11.159014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"294'12543","reported_seq":12580,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:18.567937+0000","last_change":"2024-09-21T10:17:43.799118+0000","last_active":"2024-09-21T10:21:18.567937+0000","last_peered":"2024-09-21T10:21:18.567937+0000","last_clean":"2024-09-21T10:21:18.567937+0000","last_became_active":"2024-09-21T10:17:43.798738+0000","last_became_peered":"2024-09-21T10:17:43.798738+0000","last_unstale":"2024-09-21T10:21:18.567937+0000","last_undegraded":"2024-09-21T10:21:18.567937+0000","last_fullsized":"2024-09-21T10:21:18.567937+0000","mapping_epoch":265,"log_start":"280'6200","ondisk_log_start":"280'6200","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":6343,"log_dups_size":0,"ondisk_log_size":6343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:31:31.743422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51314688,"num_objects":783,"num_object_clones":0,"num_object_copies":2349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12543,"num_write_kb":426432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":302,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.075144+0000","last_change":"2024-09-21T10:08:00.412449+0000","last_active":"2024-09-21T10:20:48.075144+0000","last_peered":"2024-09-21T10:20:48.075144+0000","last_clean":"2024-09-21T10:20:48.075144+0000","last_became_active":"2024-09-21T10:08:00.386530+0000","last_became_peered":"2024-09-21T10:08:00.386530+0000","last_unstale":"2024-09-21T10:20:48.075144+0000","last_undegraded":"2024-09-21T10:20:48.075144+0000","last_fullsized":"2024-09-21T10:20:48.075144+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:12:50.581176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":389,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.146563+0000","last_change":"2024-09-21T10:18:33.479057+0000","last_active":"2024-09-21T10:21:06.146563+0000","last_peered":"2024-09-21T10:21:06.146563+0000","last_clean":"2024-09-21T10:21:06.146563+0000","last_became_active":"2024-09-21T10:18:33.478752+0000","last_became_peered":"2024-09-21T10:18:33.478752+0000","last_unstale":"2024-09-21T10:21:06.146563+0000","last_undegraded":"2024-09-21T10:21:06.146563+0000","last_fullsized":"2024-09-21T10:21:06.146563+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:30.068585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"294'12018","reported_seq":12088,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:17.998170+0000","last_change":"2024-09-21T10:17:43.799119+0000","last_active":"2024-09-21T10:21:17.998170+0000","last_peered":"2024-09-21T10:21:17.998170+0000","last_clean":"2024-09-21T10:21:17.998170+0000","last_became_active":"2024-09-21T10:17:43.798511+0000","last_became_peered":"2024-09-21T10:17:43.798511+0000","last_unstale":"2024-09-21T10:21:17.998170+0000","last_undegraded":"2024-09-21T10:21:17.998170+0000","last_fullsized":"2024-09-21T10:21:17.998170+0000","mapping_epoch":265,"log_start":"278'5100","ondisk_log_start":"278'5100","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":6918,"log_dups_size":0,"ondisk_log_size":6918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:00.554890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":832,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12018,"num_write_kb":411200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":242,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.077080+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:20:48.077080+0000","last_peered":"2024-09-21T10:20:48.077080+0000","last_clean":"2024-09-21T10:20:48.077080+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:20:48.077080+0000","last_undegraded":"2024-09-21T10:20:48.077080+0000","last_fullsized":"2024-09-21T10:20:48.077080+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":257,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.078492+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:20:48.078492+0000","last_peered":"2024-09-21T10:20:48.078492+0000","last_clean":"2024-09-21T10:20:48.078492+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:20:48.078492+0000","last_undegraded":"2024-09-21T10:20:48.078492+0000","last_fullsized":"2024-09-21T10:20:48.078492+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"294'12223","reported_seq":12296,"reported_epoch":294,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:21:16.824448+0000","last_change":"2024-09-21T10:18:34.691817+0000","last_active":"2024-09-21T10:21:16.824448+0000","last_peered":"2024-09-21T10:21:16.824448+0000","last_clean":"2024-09-21T10:18:32.440045+0000","last_became_active":"2024-09-21T10:18:34.676050+0000","last_became_peered":"2024-09-21T10:18:34.676050+0000","last_unstale":"2024-09-21T10:21:16.824448+0000","last_undegraded":"2024-09-21T10:18:34.519870+0000","last_fullsized":"2024-09-21T10:18:34.459225+0000","mapping_epoch":275,"log_start":"280'6700","ondisk_log_start":"280'6700","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":5523,"log_dups_size":0,"ondisk_log_size":5523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59703296,"num_objects":911,"num_object_clones":0,"num_object_copies":2733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12223,"num_write_kb":420288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6],"avail_no_missing":["1","3","6"],"object_location_counts":[{"shards":"1,6","objects":-3364}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":92,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.072730+0000","last_change":"2024-09-21T10:18:33.483457+0000","last_active":"2024-09-21T10:20:48.072730+0000","last_peered":"2024-09-21T10:20:48.072730+0000","last_clean":"2024-09-21T10:20:48.072730+0000","last_became_active":"2024-09-21T10:18:33.483140+0000","last_became_peered":"2024-09-21T10:18:33.483140+0000","last_unstale":"2024-09-21T10:20:48.072730+0000","last_undegraded":"2024-09-21T10:20:48.072730+0000","last_fullsized":"2024-09-21T10:20:48.072730+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:37:39.302566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":485,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.135924+0000","last_change":"2024-09-21T10:18:33.483011+0000","last_active":"2024-09-21T10:21:06.135924+0000","last_peered":"2024-09-21T10:21:06.135924+0000","last_clean":"2024-09-21T10:21:06.135924+0000","last_became_active":"2024-09-21T10:18:33.482712+0000","last_became_peered":"2024-09-21T10:18:33.482712+0000","last_unstale":"2024-09-21T10:21:06.135924+0000","last_undegraded":"2024-09-21T10:21:06.135924+0000","last_fullsized":"2024-09-21T10:21:06.135924+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:05:11.037442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"294'11724","reported_seq":14927,"reported_epoch":294,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T10:21:16.888809+0000","last_change":"2024-09-21T10:20:44.375354+0000","last_active":"2024-09-21T10:21:16.888809+0000","last_peered":"2024-09-21T10:21:16.888809+0000","last_clean":"2024-09-21T10:18:32.459541+0000","last_became_active":"2024-09-21T10:18:41.519168+0000","last_became_peered":"2024-09-21T10:18:41.519168+0000","last_unstale":"2024-09-21T10:21:16.888809+0000","last_undegraded":"2024-09-21T10:18:41.384300+0000","last_fullsized":"2024-09-21T10:18:41.373751+0000","mapping_epoch":275,"log_start":"280'6200","ondisk_log_start":"280'6200","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":5524,"log_dups_size":0,"ondisk_log_size":5524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59637760,"num_objects":910,"num_object_clones":0,"num_object_copies":2730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2446,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11724,"num_write_kb":404288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1565,"num_bytes_recovered":12386304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5],"avail_no_missing":["1","3","5"],"object_location_counts":[{"shards":"1,5","objects":-1536}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":244,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.163011+0000","last_change":"2024-09-21T10:18:33.488794+0000","last_active":"2024-09-21T10:21:06.163011+0000","last_peered":"2024-09-21T10:21:06.163011+0000","last_clean":"2024-09-21T10:21:06.163011+0000","last_became_active":"2024-09-21T10:18:33.479048+0000","last_became_peered":"2024-09-21T10:18:33.479048+0000","last_unstale":"2024-09-21T10:21:06.163011+0000","last_undegraded":"2024-09-21T10:21:06.163011+0000","last_fullsized":"2024-09-21T10:21:06.163011+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:03.387788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":111,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.072666+0000","last_change":"2024-09-21T10:18:33.475977+0000","last_active":"2024-09-21T10:20:48.072666+0000","last_peered":"2024-09-21T10:20:48.072666+0000","last_clean":"2024-09-21T10:20:48.072666+0000","last_became_active":"2024-09-21T10:18:33.475590+0000","last_became_peered":"2024-09-21T10:18:33.475590+0000","last_unstale":"2024-09-21T10:20:48.072666+0000","last_undegraded":"2024-09-21T10:20:48.072666+0000","last_fullsized":"2024-09-21T10:20:48.072666+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:54:57.304453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1","version":"294'12040","reported_seq":12110,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:16.886649+0000","last_change":"2024-09-21T10:17:43.803150+0000","last_active":"2024-09-21T10:21:16.886649+0000","last_peered":"2024-09-21T10:21:16.886649+0000","last_clean":"2024-09-21T10:21:16.886649+0000","last_became_active":"2024-09-21T10:17:43.801977+0000","last_became_peered":"2024-09-21T10:17:43.801977+0000","last_unstale":"2024-09-21T10:21:16.886649+0000","last_undegraded":"2024-09-21T10:21:16.886649+0000","last_fullsized":"2024-09-21T10:21:16.886649+0000","mapping_epoch":265,"log_start":"280'6500","ondisk_log_start":"280'6500","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":5540,"log_dups_size":0,"ondisk_log_size":5540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:58:28.580896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56492032,"num_objects":862,"num_object_clones":0,"num_object_copies":2586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12040,"num_write_kb":412864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":298,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.078610+0000","last_change":"2024-09-21T10:18:33.473849+0000","last_active":"2024-09-21T10:20:48.078610+0000","last_peered":"2024-09-21T10:20:48.078610+0000","last_clean":"2024-09-21T10:20:48.078610+0000","last_became_active":"2024-09-21T10:18:33.473037+0000","last_became_peered":"2024-09-21T10:18:33.473037+0000","last_unstale":"2024-09-21T10:20:48.078610+0000","last_undegraded":"2024-09-21T10:20:48.078610+0000","last_fullsized":"2024-09-21T10:20:48.078610+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:03:16.322667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":390,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.159274+0000","last_change":"2024-09-21T10:08:00.387420+0000","last_active":"2024-09-21T10:21:06.159274+0000","last_peered":"2024-09-21T10:21:06.159274+0000","last_clean":"2024-09-21T10:21:06.159274+0000","last_became_active":"2024-09-21T10:08:00.386843+0000","last_became_peered":"2024-09-21T10:08:00.386843+0000","last_unstale":"2024-09-21T10:21:06.159274+0000","last_undegraded":"2024-09-21T10:21:06.159274+0000","last_fullsized":"2024-09-21T10:21:06.159274+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:51:00.654836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"294'12084","reported_seq":13305,"reported_epoch":294,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:21:18.029247+0000","last_change":"2024-09-21T10:18:41.616297+0000","last_active":"2024-09-21T10:21:18.029247+0000","last_peered":"2024-09-21T10:21:18.029247+0000","last_clean":"2024-09-21T10:18:32.444959+0000","last_became_active":"2024-09-21T10:18:34.618845+0000","last_became_peered":"2024-09-21T10:18:34.618845+0000","last_unstale":"2024-09-21T10:21:18.029247+0000","last_undegraded":"2024-09-21T10:18:41.613207+0000","last_fullsized":"2024-09-21T10:18:34.462595+0000","mapping_epoch":275,"log_start":"279'5200","ondisk_log_start":"279'5200","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":6884,"log_dups_size":0,"ondisk_log_size":6884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54001664,"num_objects":824,"num_object_clones":0,"num_object_copies":2472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12084,"num_write_kb":413056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":573,"num_bytes_recovered":37552128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,5","objects":-2694}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":374,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.144515+0000","last_change":"2024-09-21T10:08:00.382496+0000","last_active":"2024-09-21T10:21:06.144515+0000","last_peered":"2024-09-21T10:21:06.144515+0000","last_clean":"2024-09-21T10:21:06.144515+0000","last_became_active":"2024-09-21T10:08:00.382180+0000","last_became_peered":"2024-09-21T10:08:00.382180+0000","last_unstale":"2024-09-21T10:21:06.144515+0000","last_undegraded":"2024-09-21T10:21:06.144515+0000","last_fullsized":"2024-09-21T10:21:06.144515+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:16.074736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":227,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.072089+0000","last_change":"2024-09-21T10:08:38.906801+0000","last_active":"2024-09-21T10:20:48.072089+0000","last_peered":"2024-09-21T10:20:48.072089+0000","last_clean":"2024-09-21T10:20:48.072089+0000","last_became_active":"2024-09-21T10:08:38.906487+0000","last_became_peered":"2024-09-21T10:08:38.906487+0000","last_unstale":"2024-09-21T10:20:48.072089+0000","last_undegraded":"2024-09-21T10:20:48.072089+0000","last_fullsized":"2024-09-21T10:20:48.072089+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:31.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"294'12139","reported_seq":12209,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:16.855762+0000","last_change":"2024-09-21T10:17:44.115373+0000","last_active":"2024-09-21T10:21:16.855762+0000","last_peered":"2024-09-21T10:21:16.855762+0000","last_clean":"2024-09-21T10:21:16.855762+0000","last_became_active":"2024-09-21T10:17:44.115083+0000","last_became_peered":"2024-09-21T10:17:44.115083+0000","last_unstale":"2024-09-21T10:21:16.855762+0000","last_undegraded":"2024-09-21T10:21:16.855762+0000","last_fullsized":"2024-09-21T10:21:16.855762+0000","mapping_epoch":265,"log_start":"280'6600","ondisk_log_start":"280'6600","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":5539,"log_dups_size":0,"ondisk_log_size":5539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:18.021725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57344000,"num_objects":875,"num_object_clones":0,"num_object_copies":2625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12139,"num_write_kb":416448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":111,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.159826+0000","last_change":"2024-09-21T10:18:33.484568+0000","last_active":"2024-09-21T10:21:06.159826+0000","last_peered":"2024-09-21T10:21:06.159826+0000","last_clean":"2024-09-21T10:21:06.159826+0000","last_became_active":"2024-09-21T10:18:33.483975+0000","last_became_peered":"2024-09-21T10:18:33.483975+0000","last_unstale":"2024-09-21T10:21:06.159826+0000","last_undegraded":"2024-09-21T10:21:06.159826+0000","last_fullsized":"2024-09-21T10:21:06.159826+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:13.276318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":473,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.160430+0000","last_change":"2024-09-21T10:18:33.474464+0000","last_active":"2024-09-21T10:21:06.160430+0000","last_peered":"2024-09-21T10:21:06.160430+0000","last_clean":"2024-09-21T10:21:06.160430+0000","last_became_active":"2024-09-21T10:18:33.474128+0000","last_became_peered":"2024-09-21T10:18:33.474128+0000","last_unstale":"2024-09-21T10:21:06.160430+0000","last_undegraded":"2024-09-21T10:21:06.160430+0000","last_fullsized":"2024-09-21T10:21:06.160430+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:32:49.409409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"294'12007","reported_seq":12044,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:18.574746+0000","last_change":"2024-09-21T10:17:44.113304+0000","last_active":"2024-09-21T10:21:18.574746+0000","last_peered":"2024-09-21T10:21:18.574746+0000","last_clean":"2024-09-21T10:21:18.574746+0000","last_became_active":"2024-09-21T10:17:44.112949+0000","last_became_peered":"2024-09-21T10:17:44.112949+0000","last_unstale":"2024-09-21T10:21:18.574746+0000","last_undegraded":"2024-09-21T10:21:18.574746+0000","last_fullsized":"2024-09-21T10:21:18.574746+0000","mapping_epoch":265,"log_start":"279'5700","ondisk_log_start":"279'5700","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":6307,"log_dups_size":0,"ondisk_log_size":6307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:20.864228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52101120,"num_objects":795,"num_object_clones":0,"num_object_copies":2385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12007,"num_write_kb":409664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":375,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.073923+0000","last_change":"2024-09-21T09:59:54.265556+0000","last_active":"2024-09-21T10:20:48.073923+0000","last_peered":"2024-09-21T10:20:48.073923+0000","last_clean":"2024-09-21T10:20:48.073923+0000","last_became_active":"2024-09-21T09:59:54.265033+0000","last_became_peered":"2024-09-21T09:59:54.265033+0000","last_unstale":"2024-09-21T10:20:48.073923+0000","last_undegraded":"2024-09-21T10:20:48.073923+0000","last_fullsized":"2024-09-21T10:20:48.073923+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T09:48:34.850872+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:40.449756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":754,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.150709+0000","last_change":"2024-09-21T10:07:43.790490+0000","last_active":"2024-09-21T10:21:06.150709+0000","last_peered":"2024-09-21T10:21:06.150709+0000","last_clean":"2024-09-21T10:21:06.150709+0000","last_became_active":"2024-09-21T10:07:43.789986+0000","last_became_peered":"2024-09-21T10:07:43.789986+0000","last_unstale":"2024-09-21T10:21:06.150709+0000","last_undegraded":"2024-09-21T10:21:06.150709+0000","last_fullsized":"2024-09-21T10:21:06.150709+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:08.740356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":481,"num_read_kb":1580,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":45,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.078423+0000","last_change":"2024-09-21T10:18:33.473676+0000","last_active":"2024-09-21T10:20:48.078423+0000","last_peered":"2024-09-21T10:20:48.078423+0000","last_clean":"2024-09-21T10:20:48.078423+0000","last_became_active":"2024-09-21T10:18:33.472593+0000","last_became_peered":"2024-09-21T10:18:33.472593+0000","last_unstale":"2024-09-21T10:20:48.078423+0000","last_undegraded":"2024-09-21T10:20:48.078423+0000","last_fullsized":"2024-09-21T10:20:48.078423+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:07:55.816740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":270,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.072494+0000","last_change":"2024-09-21T10:08:00.386135+0000","last_active":"2024-09-21T10:20:48.072494+0000","last_peered":"2024-09-21T10:20:48.072494+0000","last_clean":"2024-09-21T10:20:48.072494+0000","last_became_active":"2024-09-21T10:08:00.385796+0000","last_became_peered":"2024-09-21T10:08:00.385796+0000","last_unstale":"2024-09-21T10:20:48.072494+0000","last_undegraded":"2024-09-21T10:20:48.072494+0000","last_fullsized":"2024-09-21T10:20:48.072494+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:27.054699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"294'12186","reported_seq":12256,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:16.899237+0000","last_change":"2024-09-21T10:17:44.114644+0000","last_active":"2024-09-21T10:21:16.899237+0000","last_peered":"2024-09-21T10:21:16.899237+0000","last_clean":"2024-09-21T10:21:16.899237+0000","last_became_active":"2024-09-21T10:17:44.114361+0000","last_became_peered":"2024-09-21T10:17:44.114361+0000","last_unstale":"2024-09-21T10:21:16.899237+0000","last_undegraded":"2024-09-21T10:21:16.899237+0000","last_fullsized":"2024-09-21T10:21:16.899237+0000","mapping_epoch":265,"log_start":"280'6700","ondisk_log_start":"280'6700","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":5486,"log_dups_size":0,"ondisk_log_size":5486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:36:37.772952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58458112,"num_objects":892,"num_object_clones":0,"num_object_copies":2676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12186,"num_write_kb":418496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":638,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.073814+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:20:48.073814+0000","last_peered":"2024-09-21T10:20:48.073814+0000","last_clean":"2024-09-21T10:20:48.073814+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:20:48.073814+0000","last_undegraded":"2024-09-21T10:20:48.073814+0000","last_fullsized":"2024-09-21T10:20:48.073814+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":486,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.143622+0000","last_change":"2024-09-21T10:18:33.471355+0000","last_active":"2024-09-21T10:21:06.143622+0000","last_peered":"2024-09-21T10:21:06.143622+0000","last_clean":"2024-09-21T10:21:06.143622+0000","last_became_active":"2024-09-21T10:18:33.470962+0000","last_became_peered":"2024-09-21T10:18:33.470962+0000","last_unstale":"2024-09-21T10:21:06.143622+0000","last_undegraded":"2024-09-21T10:21:06.143622+0000","last_fullsized":"2024-09-21T10:21:06.143622+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:53:57.516543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":468,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.142177+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:21:06.142177+0000","last_peered":"2024-09-21T10:21:06.142177+0000","last_clean":"2024-09-21T10:21:06.142177+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:21:06.142177+0000","last_undegraded":"2024-09-21T10:21:06.142177+0000","last_fullsized":"2024-09-21T10:21:06.142177+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"294'12070","reported_seq":13394,"reported_epoch":294,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:21:16.852186+0000","last_change":"2024-09-21T10:18:41.561934+0000","last_active":"2024-09-21T10:21:16.852186+0000","last_peered":"2024-09-21T10:21:16.852186+0000","last_clean":"2024-09-21T10:18:32.435113+0000","last_became_active":"2024-09-21T10:18:34.651959+0000","last_became_peered":"2024-09-21T10:18:34.651959+0000","last_unstale":"2024-09-21T10:21:16.852186+0000","last_undegraded":"2024-09-21T10:18:41.559442+0000","last_fullsized":"2024-09-21T10:18:34.459559+0000","mapping_epoch":275,"log_start":"280'6600","ondisk_log_start":"280'6600","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":5470,"log_dups_size":0,"ondisk_log_size":5470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57278464,"num_objects":874,"num_object_clones":0,"num_object_copies":2622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3504,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12070,"num_write_kb":414208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":624,"num_bytes_recovered":40894464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2","objects":-2630}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":1242,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.148045+0000","last_change":"2024-09-21T10:18:36.495308+0000","last_active":"2024-09-21T10:21:06.148045+0000","last_peered":"2024-09-21T10:21:06.148045+0000","last_clean":"2024-09-21T10:21:06.148045+0000","last_became_active":"2024-09-21T10:18:36.494662+0000","last_became_peered":"2024-09-21T10:18:36.494662+0000","last_unstale":"2024-09-21T10:21:06.148045+0000","last_undegraded":"2024-09-21T10:21:06.148045+0000","last_fullsized":"2024-09-21T10:21:06.148045+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:49.218676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":340,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.159911+0000","last_change":"2024-09-21T10:08:18.470748+0000","last_active":"2024-09-21T10:21:06.159911+0000","last_peered":"2024-09-21T10:21:06.159911+0000","last_clean":"2024-09-21T10:21:06.159911+0000","last_became_active":"2024-09-21T10:08:18.470227+0000","last_became_peered":"2024-09-21T10:08:18.470227+0000","last_unstale":"2024-09-21T10:21:06.159911+0000","last_undegraded":"2024-09-21T10:21:06.159911+0000","last_fullsized":"2024-09-21T10:21:06.159911+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:16.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":463,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.148469+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:21:06.148469+0000","last_peered":"2024-09-21T10:21:06.148469+0000","last_clean":"2024-09-21T10:21:06.148469+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:21:06.148469+0000","last_undegraded":"2024-09-21T10:21:06.148469+0000","last_fullsized":"2024-09-21T10:21:06.148469+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"294'12283","reported_seq":12360,"reported_epoch":294,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-21T10:21:18.389080+0000","last_change":"2024-09-21T10:18:34.720485+0000","last_active":"2024-09-21T10:21:18.389080+0000","last_peered":"2024-09-21T10:21:18.389080+0000","last_clean":"2024-09-21T10:18:32.459708+0000","last_became_active":"2024-09-21T10:18:34.691759+0000","last_became_peered":"2024-09-21T10:18:34.691759+0000","last_unstale":"2024-09-21T10:21:18.389080+0000","last_undegraded":"2024-09-21T10:21:18.389080+0000","last_fullsized":"2024-09-21T10:21:18.389080+0000","mapping_epoch":275,"log_start":"272'2200","ondisk_log_start":"272'2200","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54198272,"num_objects":827,"num_object_clones":0,"num_object_copies":2481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4219,"num_objects_unfound":0,"num_objects_dirty":827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12283,"num_write_kb":419520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[3,5,7],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":-3392}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":939,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.147276+0000","last_change":"2024-09-21T10:07:43.794767+0000","last_active":"2024-09-21T10:21:06.147276+0000","last_peered":"2024-09-21T10:21:06.147276+0000","last_clean":"2024-09-21T10:21:06.147276+0000","last_became_active":"2024-09-21T10:07:43.794062+0000","last_became_peered":"2024-09-21T10:07:43.794062+0000","last_unstale":"2024-09-21T10:21:06.147276+0000","last_undegraded":"2024-09-21T10:21:06.147276+0000","last_fullsized":"2024-09-21T10:21:06.147276+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:51.977955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":365,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.135639+0000","last_change":"2024-09-21T10:08:00.389176+0000","last_active":"2024-09-21T10:21:06.135639+0000","last_peered":"2024-09-21T10:21:06.135639+0000","last_clean":"2024-09-21T10:21:06.135639+0000","last_became_active":"2024-09-21T10:08:00.383854+0000","last_became_peered":"2024-09-21T10:08:00.383854+0000","last_unstale":"2024-09-21T10:21:06.135639+0000","last_undegraded":"2024-09-21T10:21:06.135639+0000","last_fullsized":"2024-09-21T10:21:06.135639+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:46.952216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":386,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.150558+0000","last_change":"2024-09-21T10:18:33.465815+0000","last_active":"2024-09-21T10:21:06.150558+0000","last_peered":"2024-09-21T10:21:06.150558+0000","last_clean":"2024-09-21T10:21:06.150558+0000","last_became_active":"2024-09-21T10:18:33.465459+0000","last_became_peered":"2024-09-21T10:18:33.465459+0000","last_unstale":"2024-09-21T10:21:06.150558+0000","last_undegraded":"2024-09-21T10:21:06.150558+0000","last_fullsized":"2024-09-21T10:21:06.150558+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:22:08.199800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"294'12036","reported_seq":12106,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:18.042092+0000","last_change":"2024-09-21T10:17:43.800429+0000","last_active":"2024-09-21T10:21:18.042092+0000","last_peered":"2024-09-21T10:21:18.042092+0000","last_clean":"2024-09-21T10:21:18.042092+0000","last_became_active":"2024-09-21T10:17:43.799643+0000","last_became_peered":"2024-09-21T10:17:43.799643+0000","last_unstale":"2024-09-21T10:21:18.042092+0000","last_undegraded":"2024-09-21T10:21:18.042092+0000","last_fullsized":"2024-09-21T10:21:18.042092+0000","mapping_epoch":265,"log_start":"278'5200","ondisk_log_start":"278'5200","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":6836,"log_dups_size":0,"ondisk_log_size":6836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:38:08.368261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49676288,"num_objects":758,"num_object_clones":0,"num_object_copies":2274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12036,"num_write_kb":409408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":129,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.158697+0000","last_change":"2024-09-21T10:18:33.467659+0000","last_active":"2024-09-21T10:21:06.158697+0000","last_peered":"2024-09-21T10:21:06.158697+0000","last_clean":"2024-09-21T10:21:06.158697+0000","last_became_active":"2024-09-21T10:18:33.467369+0000","last_became_peered":"2024-09-21T10:18:33.467369+0000","last_unstale":"2024-09-21T10:21:06.158697+0000","last_undegraded":"2024-09-21T10:21:06.158697+0000","last_fullsized":"2024-09-21T10:21:06.158697+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:16:41.951755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":1046,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.147556+0000","last_change":"2024-09-21T10:08:18.468995+0000","last_active":"2024-09-21T10:21:06.147556+0000","last_peered":"2024-09-21T10:21:06.147556+0000","last_clean":"2024-09-21T10:21:06.147556+0000","last_became_active":"2024-09-21T10:08:18.468243+0000","last_became_peered":"2024-09-21T10:08:18.468243+0000","last_unstale":"2024-09-21T10:21:06.147556+0000","last_undegraded":"2024-09-21T10:21:06.147556+0000","last_fullsized":"2024-09-21T10:21:06.147556+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:45.520968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":365,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.152788+0000","last_change":"2024-09-21T10:08:18.468595+0000","last_active":"2024-09-21T10:21:06.152788+0000","last_peered":"2024-09-21T10:21:06.152788+0000","last_clean":"2024-09-21T10:21:06.152788+0000","last_became_active":"2024-09-21T10:08:18.468296+0000","last_became_peered":"2024-09-21T10:08:18.468296+0000","last_unstale":"2024-09-21T10:21:06.152788+0000","last_undegraded":"2024-09-21T10:21:06.152788+0000","last_fullsized":"2024-09-21T10:21:06.152788+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:42.748130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.f","version":"294'11771","reported_seq":20035,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:14.564260+0000","last_change":"2024-09-21T10:20:46.081136+0000","last_active":"2024-09-21T10:21:14.564260+0000","last_peered":"2024-09-21T10:21:14.564260+0000","last_clean":"2024-09-21T10:21:14.564260+0000","last_became_active":"2024-09-21T10:20:46.046374+0000","last_became_peered":"2024-09-21T10:20:46.046374+0000","last_unstale":"2024-09-21T10:21:14.564260+0000","last_undegraded":"2024-09-21T10:21:14.564260+0000","last_fullsized":"2024-09-21T10:21:14.564260+0000","mapping_epoch":291,"log_start":"272'2900","ondisk_log_start":"272'2900","created":265,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":8871,"log_dups_size":0,"ondisk_log_size":8871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:30.571749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68222976,"num_objects":1041,"num_object_clones":0,"num_object_copies":3123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11771,"num_write_kb":409984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4091,"num_bytes_recovered":268107776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":341,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.147467+0000","last_change":"2024-09-21T10:08:18.469907+0000","last_active":"2024-09-21T10:21:06.147467+0000","last_peered":"2024-09-21T10:21:06.147467+0000","last_clean":"2024-09-21T10:21:06.147467+0000","last_became_active":"2024-09-21T10:08:18.469443+0000","last_became_peered":"2024-09-21T10:08:18.469443+0000","last_unstale":"2024-09-21T10:21:06.147467+0000","last_undegraded":"2024-09-21T10:21:06.147467+0000","last_fullsized":"2024-09-21T10:21:06.147467+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:15.645803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":688,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.077160+0000","last_change":"2024-09-21T10:08:00.419641+0000","last_active":"2024-09-21T10:20:48.077160+0000","last_peered":"2024-09-21T10:20:48.077160+0000","last_clean":"2024-09-21T10:20:48.077160+0000","last_became_active":"2024-09-21T10:08:00.387050+0000","last_became_peered":"2024-09-21T10:08:00.387050+0000","last_unstale":"2024-09-21T10:20:48.077160+0000","last_undegraded":"2024-09-21T10:20:48.077160+0000","last_fullsized":"2024-09-21T10:20:48.077160+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:00:32.881619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":447,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.135854+0000","last_change":"2024-09-21T09:59:54.267798+0000","last_active":"2024-09-21T10:21:06.135854+0000","last_peered":"2024-09-21T10:21:06.135854+0000","last_clean":"2024-09-21T10:21:06.135854+0000","last_became_active":"2024-09-21T09:59:54.265615+0000","last_became_peered":"2024-09-21T09:59:54.265615+0000","last_unstale":"2024-09-21T10:21:06.135854+0000","last_undegraded":"2024-09-21T10:21:06.135854+0000","last_fullsized":"2024-09-21T10:21:06.135854+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:30.363872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"294'12195","reported_seq":12268,"reported_epoch":294,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:21:18.028010+0000","last_change":"2024-09-21T10:18:34.660620+0000","last_active":"2024-09-21T10:21:18.028010+0000","last_peered":"2024-09-21T10:21:18.028010+0000","last_clean":"2024-09-21T10:18:32.448320+0000","last_became_active":"2024-09-21T10:18:34.639763+0000","last_became_peered":"2024-09-21T10:18:34.639763+0000","last_unstale":"2024-09-21T10:21:18.028010+0000","last_undegraded":"2024-09-21T10:18:34.495695+0000","last_fullsized":"2024-09-21T10:18:34.464258+0000","mapping_epoch":275,"log_start":"279'5300","ondisk_log_start":"279'5300","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":6895,"log_dups_size":0,"ondisk_log_size":6895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55771136,"num_objects":851,"num_object_clones":0,"num_object_copies":2553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12195,"num_write_kb":417472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,7","objects":-3285}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":57,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.074254+0000","last_change":"2024-09-21T10:18:33.489678+0000","last_active":"2024-09-21T10:20:48.074254+0000","last_peered":"2024-09-21T10:20:48.074254+0000","last_clean":"2024-09-21T10:20:48.074254+0000","last_became_active":"2024-09-21T10:18:33.474818+0000","last_became_peered":"2024-09-21T10:18:33.474818+0000","last_unstale":"2024-09-21T10:20:48.074254+0000","last_undegraded":"2024-09-21T10:20:48.074254+0000","last_fullsized":"2024-09-21T10:20:48.074254+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:49.828007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":738,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.151457+0000","last_change":"2024-09-21T10:18:36.503584+0000","last_active":"2024-09-21T10:21:06.151457+0000","last_peered":"2024-09-21T10:21:06.151457+0000","last_clean":"2024-09-21T10:21:06.151457+0000","last_became_active":"2024-09-21T10:18:36.502990+0000","last_became_peered":"2024-09-21T10:18:36.502990+0000","last_unstale":"2024-09-21T10:21:06.151457+0000","last_undegraded":"2024-09-21T10:21:06.151457+0000","last_fullsized":"2024-09-21T10:21:06.151457+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:11.648271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":86,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.073598+0000","last_change":"2024-09-21T10:18:33.479008+0000","last_active":"2024-09-21T10:20:48.073598+0000","last_peered":"2024-09-21T10:20:48.073598+0000","last_clean":"2024-09-21T10:20:48.073598+0000","last_became_active":"2024-09-21T10:18:33.478740+0000","last_became_peered":"2024-09-21T10:18:33.478740+0000","last_unstale":"2024-09-21T10:20:48.073598+0000","last_undegraded":"2024-09-21T10:20:48.073598+0000","last_fullsized":"2024-09-21T10:20:48.073598+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:16:52.594882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.d","version":"294'12204","reported_seq":12241,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:18.431249+0000","last_change":"2024-09-21T10:17:44.114402+0000","last_active":"2024-09-21T10:21:18.431249+0000","last_peered":"2024-09-21T10:21:18.431249+0000","last_clean":"2024-09-21T10:21:18.431249+0000","last_became_active":"2024-09-21T10:17:44.114051+0000","last_became_peered":"2024-09-21T10:17:44.114051+0000","last_unstale":"2024-09-21T10:21:18.431249+0000","last_undegraded":"2024-09-21T10:21:18.431249+0000","last_fullsized":"2024-09-21T10:21:18.431249+0000","mapping_epoch":265,"log_start":"279'5300","ondisk_log_start":"279'5300","created":265,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:17:42.786173+0000","last_clean_scrub_stamp":"2024-09-21T10:17:42.786173+0000","objects_scrubbed":0,"log_size":6904,"log_dups_size":0,"ondisk_log_size":6904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:48:23.466657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52232216,"num_objects":798,"num_object_clones":0,"num_object_copies":2394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":798,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12204,"num_write_kb":416001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":494,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.141878+0000","last_change":"2024-09-21T10:08:00.390436+0000","last_active":"2024-09-21T10:21:06.141878+0000","last_peered":"2024-09-21T10:21:06.141878+0000","last_clean":"2024-09-21T10:21:06.141878+0000","last_became_active":"2024-09-21T10:08:00.381835+0000","last_became_peered":"2024-09-21T10:08:00.381835+0000","last_unstale":"2024-09-21T10:21:06.141878+0000","last_undegraded":"2024-09-21T10:21:06.141878+0000","last_fullsized":"2024-09-21T10:21:06.141878+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:01:35.697356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":520,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.079396+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:20:48.079396+0000","last_peered":"2024-09-21T10:20:48.079396+0000","last_clean":"2024-09-21T10:20:48.079396+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:20:48.079396+0000","last_undegraded":"2024-09-21T10:20:48.079396+0000","last_fullsized":"2024-09-21T10:20:48.079396+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":158,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.072412+0000","last_change":"2024-09-21T10:08:00.385774+0000","last_active":"2024-09-21T10:20:48.072412+0000","last_peered":"2024-09-21T10:20:48.072412+0000","last_clean":"2024-09-21T10:20:48.072412+0000","last_became_active":"2024-09-21T10:08:00.385445+0000","last_became_peered":"2024-09-21T10:08:00.385445+0000","last_unstale":"2024-09-21T10:20:48.072412+0000","last_undegraded":"2024-09-21T10:20:48.072412+0000","last_fullsized":"2024-09-21T10:20:48.072412+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:42.442190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":217,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.077244+0000","last_change":"2024-09-21T10:08:00.379661+0000","last_active":"2024-09-21T10:20:48.077244+0000","last_peered":"2024-09-21T10:20:48.077244+0000","last_clean":"2024-09-21T10:20:48.077244+0000","last_became_active":"2024-09-21T10:08:00.379301+0000","last_became_peered":"2024-09-21T10:08:00.379301+0000","last_unstale":"2024-09-21T10:20:48.077244+0000","last_undegraded":"2024-09-21T10:20:48.077244+0000","last_fullsized":"2024-09-21T10:20:48.077244+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:15.649523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":240,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.074108+0000","last_change":"2024-09-21T10:08:00.410731+0000","last_active":"2024-09-21T10:20:48.074108+0000","last_peered":"2024-09-21T10:20:48.074108+0000","last_clean":"2024-09-21T10:20:48.074108+0000","last_became_active":"2024-09-21T10:08:00.380324+0000","last_became_peered":"2024-09-21T10:08:00.380324+0000","last_unstale":"2024-09-21T10:20:48.074108+0000","last_undegraded":"2024-09-21T10:20:48.074108+0000","last_fullsized":"2024-09-21T10:20:48.074108+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:47:34.960221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":340,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.147864+0000","last_change":"2024-09-21T10:08:18.470378+0000","last_active":"2024-09-21T10:21:06.147864+0000","last_peered":"2024-09-21T10:21:06.147864+0000","last_clean":"2024-09-21T10:21:06.147864+0000","last_became_active":"2024-09-21T10:08:18.469482+0000","last_became_peered":"2024-09-21T10:08:18.469482+0000","last_unstale":"2024-09-21T10:21:06.147864+0000","last_undegraded":"2024-09-21T10:21:06.147864+0000","last_fullsized":"2024-09-21T10:21:06.147864+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:35.628724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":325,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.076922+0000","last_change":"2024-09-21T10:18:33.479236+0000","last_active":"2024-09-21T10:20:48.076922+0000","last_peered":"2024-09-21T10:20:48.076922+0000","last_clean":"2024-09-21T10:20:48.076922+0000","last_became_active":"2024-09-21T10:18:33.474256+0000","last_became_peered":"2024-09-21T10:18:33.474256+0000","last_unstale":"2024-09-21T10:20:48.076922+0000","last_undegraded":"2024-09-21T10:20:48.076922+0000","last_fullsized":"2024-09-21T10:20:48.076922+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:19:34.666035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":205,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.075030+0000","last_change":"2024-09-21T10:07:43.791830+0000","last_active":"2024-09-21T10:20:48.075030+0000","last_peered":"2024-09-21T10:20:48.075030+0000","last_clean":"2024-09-21T10:20:48.075030+0000","last_became_active":"2024-09-21T10:07:43.791314+0000","last_became_peered":"2024-09-21T10:07:43.791314+0000","last_unstale":"2024-09-21T10:20:48.075030+0000","last_undegraded":"2024-09-21T10:20:48.075030+0000","last_fullsized":"2024-09-21T10:20:48.075030+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:37.086629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":46,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.072343+0000","last_change":"2024-09-21T10:18:33.464989+0000","last_active":"2024-09-21T10:20:48.072343+0000","last_peered":"2024-09-21T10:20:48.072343+0000","last_clean":"2024-09-21T10:20:48.072343+0000","last_became_active":"2024-09-21T10:18:33.464617+0000","last_became_peered":"2024-09-21T10:18:33.464617+0000","last_unstale":"2024-09-21T10:20:48.072343+0000","last_undegraded":"2024-09-21T10:20:48.072343+0000","last_fullsized":"2024-09-21T10:20:48.072343+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:09.300797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":463,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.147073+0000","last_change":"2024-09-21T10:18:33.472415+0000","last_active":"2024-09-21T10:21:06.147073+0000","last_peered":"2024-09-21T10:21:06.147073+0000","last_clean":"2024-09-21T10:21:06.147073+0000","last_became_active":"2024-09-21T10:18:33.471746+0000","last_became_peered":"2024-09-21T10:18:33.471746+0000","last_unstale":"2024-09-21T10:21:06.147073+0000","last_undegraded":"2024-09-21T10:21:06.147073+0000","last_fullsized":"2024-09-21T10:21:06.147073+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:40:37.099458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":110,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.157708+0000","last_change":"2024-09-21T10:18:33.484721+0000","last_active":"2024-09-21T10:21:06.157708+0000","last_peered":"2024-09-21T10:21:06.157708+0000","last_clean":"2024-09-21T10:21:06.157708+0000","last_became_active":"2024-09-21T10:18:33.483950+0000","last_became_peered":"2024-09-21T10:18:33.483950+0000","last_unstale":"2024-09-21T10:21:06.157708+0000","last_undegraded":"2024-09-21T10:21:06.157708+0000","last_fullsized":"2024-09-21T10:21:06.157708+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:54:52.208378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":257,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.079359+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:20:48.079359+0000","last_peered":"2024-09-21T10:20:48.079359+0000","last_clean":"2024-09-21T10:20:48.079359+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:20:48.079359+0000","last_undegraded":"2024-09-21T10:20:48.079359+0000","last_fullsized":"2024-09-21T10:20:48.079359+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":129,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.135412+0000","last_change":"2024-09-21T10:18:33.479635+0000","last_active":"2024-09-21T10:21:06.135412+0000","last_peered":"2024-09-21T10:21:06.135412+0000","last_clean":"2024-09-21T10:21:06.135412+0000","last_became_active":"2024-09-21T10:18:33.478830+0000","last_became_peered":"2024-09-21T10:18:33.478830+0000","last_unstale":"2024-09-21T10:21:06.135412+0000","last_undegraded":"2024-09-21T10:21:06.135412+0000","last_fullsized":"2024-09-21T10:21:06.135412+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:16.950086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":479,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.142097+0000","last_change":"2024-09-21T10:18:33.480350+0000","last_active":"2024-09-21T10:21:06.142097+0000","last_peered":"2024-09-21T10:21:06.142097+0000","last_clean":"2024-09-21T10:21:06.142097+0000","last_became_active":"2024-09-21T10:18:33.479248+0000","last_became_peered":"2024-09-21T10:18:33.479248+0000","last_unstale":"2024-09-21T10:21:06.142097+0000","last_undegraded":"2024-09-21T10:21:06.142097+0000","last_fullsized":"2024-09-21T10:21:06.142097+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:25:28.780572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":473,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.146641+0000","last_change":"2024-09-21T10:18:33.480341+0000","last_active":"2024-09-21T10:21:06.146641+0000","last_peered":"2024-09-21T10:21:06.146641+0000","last_clean":"2024-09-21T10:21:06.146641+0000","last_became_active":"2024-09-21T10:18:33.479869+0000","last_became_peered":"2024-09-21T10:18:33.479869+0000","last_unstale":"2024-09-21T10:21:06.146641+0000","last_undegraded":"2024-09-21T10:21:06.146641+0000","last_fullsized":"2024-09-21T10:21:06.146641+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:32:42.206910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":234,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.147681+0000","last_change":"2024-09-21T10:18:33.473981+0000","last_active":"2024-09-21T10:21:06.147681+0000","last_peered":"2024-09-21T10:21:06.147681+0000","last_clean":"2024-09-21T10:21:06.147681+0000","last_became_active":"2024-09-21T10:18:33.473619+0000","last_became_peered":"2024-09-21T10:18:33.473619+0000","last_unstale":"2024-09-21T10:21:06.147681+0000","last_undegraded":"2024-09-21T10:21:06.147681+0000","last_fullsized":"2024-09-21T10:21:06.147681+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:44:24.630856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":286,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.152869+0000","last_change":"2024-09-21T10:08:18.471282+0000","last_active":"2024-09-21T10:21:06.152869+0000","last_peered":"2024-09-21T10:21:06.152869+0000","last_clean":"2024-09-21T10:21:06.152869+0000","last_became_active":"2024-09-21T10:08:18.470952+0000","last_became_peered":"2024-09-21T10:08:18.470952+0000","last_unstale":"2024-09-21T10:21:06.152869+0000","last_undegraded":"2024-09-21T10:21:06.152869+0000","last_fullsized":"2024-09-21T10:21:06.152869+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:25.597081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":119,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.150382+0000","last_change":"2024-09-21T10:18:33.480128+0000","last_active":"2024-09-21T10:21:06.150382+0000","last_peered":"2024-09-21T10:21:06.150382+0000","last_clean":"2024-09-21T10:21:06.150382+0000","last_became_active":"2024-09-21T10:18:33.479691+0000","last_became_peered":"2024-09-21T10:18:33.479691+0000","last_unstale":"2024-09-21T10:21:06.150382+0000","last_undegraded":"2024-09-21T10:21:06.150382+0000","last_fullsized":"2024-09-21T10:21:06.150382+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:43:49.311244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":470,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.135776+0000","last_change":"2024-09-21T10:08:00.386104+0000","last_active":"2024-09-21T10:21:06.135776+0000","last_peered":"2024-09-21T10:21:06.135776+0000","last_clean":"2024-09-21T10:21:06.135776+0000","last_became_active":"2024-09-21T10:08:00.381484+0000","last_became_peered":"2024-09-21T10:08:00.381484+0000","last_unstale":"2024-09-21T10:21:06.135776+0000","last_undegraded":"2024-09-21T10:21:06.135776+0000","last_fullsized":"2024-09-21T10:21:06.135776+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:39.190191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":148,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.163447+0000","last_change":"2024-09-21T10:18:33.481322+0000","last_active":"2024-09-21T10:21:06.163447+0000","last_peered":"2024-09-21T10:21:06.163447+0000","last_clean":"2024-09-21T10:21:06.163447+0000","last_became_active":"2024-09-21T10:18:33.480554+0000","last_became_peered":"2024-09-21T10:18:33.480554+0000","last_unstale":"2024-09-21T10:21:06.163447+0000","last_undegraded":"2024-09-21T10:21:06.163447+0000","last_fullsized":"2024-09-21T10:21:06.163447+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:15.498594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":297,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.147785+0000","last_change":"2024-09-21T10:08:18.466882+0000","last_active":"2024-09-21T10:21:06.147785+0000","last_peered":"2024-09-21T10:21:06.147785+0000","last_clean":"2024-09-21T10:21:06.147785+0000","last_became_active":"2024-09-21T10:08:18.466470+0000","last_became_peered":"2024-09-21T10:08:18.466470+0000","last_unstale":"2024-09-21T10:21:06.147785+0000","last_undegraded":"2024-09-21T10:21:06.147785+0000","last_fullsized":"2024-09-21T10:21:06.147785+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:51:31.020051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":148,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.152166+0000","last_change":"2024-09-21T10:18:33.472352+0000","last_active":"2024-09-21T10:21:06.152166+0000","last_peered":"2024-09-21T10:21:06.152166+0000","last_clean":"2024-09-21T10:21:06.152166+0000","last_became_active":"2024-09-21T10:18:33.471312+0000","last_became_peered":"2024-09-21T10:18:33.471312+0000","last_unstale":"2024-09-21T10:21:06.152166+0000","last_undegraded":"2024-09-21T10:21:06.152166+0000","last_fullsized":"2024-09-21T10:21:06.152166+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:32:44.897145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":169,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.072371+0000","last_change":"2024-09-21T10:08:00.394350+0000","last_active":"2024-09-21T10:20:48.072371+0000","last_peered":"2024-09-21T10:20:48.072371+0000","last_clean":"2024-09-21T10:20:48.072371+0000","last_became_active":"2024-09-21T10:08:00.387232+0000","last_became_peered":"2024-09-21T10:08:00.387232+0000","last_unstale":"2024-09-21T10:20:48.072371+0000","last_undegraded":"2024-09-21T10:20:48.072371+0000","last_fullsized":"2024-09-21T10:20:48.072371+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:56.160018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":132,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.147960+0000","last_change":"2024-09-21T10:18:33.474216+0000","last_active":"2024-09-21T10:21:06.147960+0000","last_peered":"2024-09-21T10:21:06.147960+0000","last_clean":"2024-09-21T10:21:06.147960+0000","last_became_active":"2024-09-21T10:18:33.473794+0000","last_became_peered":"2024-09-21T10:18:33.473794+0000","last_unstale":"2024-09-21T10:21:06.147960+0000","last_undegraded":"2024-09-21T10:21:06.147960+0000","last_fullsized":"2024-09-21T10:21:06.147960+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:38.956888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":42,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.072252+0000","last_change":"2024-09-21T10:18:33.483576+0000","last_active":"2024-09-21T10:20:48.072252+0000","last_peered":"2024-09-21T10:20:48.072252+0000","last_clean":"2024-09-21T10:20:48.072252+0000","last_became_active":"2024-09-21T10:18:33.483116+0000","last_became_peered":"2024-09-21T10:18:33.483116+0000","last_unstale":"2024-09-21T10:20:48.072252+0000","last_undegraded":"2024-09-21T10:20:48.072252+0000","last_fullsized":"2024-09-21T10:20:48.072252+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:48.964766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":242,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-21T10:20:48.074638+0000","last_change":"2024-09-21T10:18:33.471909+0000","last_active":"2024-09-21T10:20:48.074638+0000","last_peered":"2024-09-21T10:20:48.074638+0000","last_clean":"2024-09-21T10:20:48.074638+0000","last_became_active":"2024-09-21T10:18:33.471614+0000","last_became_peered":"2024-09-21T10:18:33.471614+0000","last_unstale":"2024-09-21T10:20:48.074638+0000","last_undegraded":"2024-09-21T10:20:48.074638+0000","last_fullsized":"2024-09-21T10:20:48.074638+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:30:50.582505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":462,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-21T10:21:06.146962+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:21:06.146962+0000","last_peered":"2024-09-21T10:21:06.146962+0000","last_clean":"2024-09-21T10:21:06.146962+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:21:06.146962+0000","last_undegraded":"2024-09-21T10:21:06.146962+0000","last_fullsized":"2024-09-21T10:21:06.146962+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":908328984,"num_objects":13861,"num_object_clones":0,"num_object_copies":41583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17879,"num_objects_misplaced":4219,"num_objects_unfound":0,"num_objects_dirty":13861,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193475,"num_write_kb":6634689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6854,"num_bytes_recovered":359006208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4354093056,"data_stored":4354080840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107975,"ondisk_log_size":107975,"up":48,"acting":43,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":10016,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5982,"num_read_kb":13005,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397878,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1283808,"kb_used_data":378264,"kb_used_omap":1,"kb_used_meta":905534,"kb_avail":92461344,"statfs":{"total":95995035648,"available":94680416256,"internally_reserved":0,"allocated":387342336,"data_stored":385738808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":927267160},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594106,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1385500,"kb_used_data":480020,"kb_used_omap":1,"kb_used_meta":905470,"kb_avail":92359652,"statfs":{"total":95995035648,"available":94576283648,"internally_reserved":0,"allocated":491540480,"data_stored":489941021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":927201624},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790336,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1396552,"kb_used_data":488256,"kb_used_omap":1,"kb_used_meta":908222,"kb_avail":92348600,"statfs":{"total":95995035648,"available":94564966400,"internally_reserved":0,"allocated":499974144,"data_stored":498390032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":930019672},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986565,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":433791697127,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860653,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1176532,"kb_used_data":525068,"kb_used_omap":8,"kb_used_meta":651447,"kb_avail":92568620,"statfs":{"total":95995035648,"available":94790266880,"internally_reserved":0,"allocated":537669632,"data_stored":536078256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":667082604},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024179,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":983100,"kb_used_data":418804,"kb_used_omap":7,"kb_used_meta":564280,"kb_avail":92762052,"statfs":{"total":95995035648,"available":94988341248,"internally_reserved":0,"allocated":428855296,"data_stored":427282535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":577823282},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187715,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1127500,"kb_used_data":444996,"kb_used_omap":8,"kb_used_meta":682487,"kb_avail":92617652,"statfs":{"total":95995035648,"available":94840475648,"internally_reserved":0,"allocated":455675904,"data_stored":454088574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":698867697},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":428474368,"data_stored":428474368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":409272320,"data_stored":409272320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":526979072,"data_stored":526975000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1674969088,"data_stored":1674969088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":471666688,"data_stored":471662616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":477298688,"data_stored":477298688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":365432832,"data_stored":365428760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:21:27.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T10:21:27.128 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:27.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:27.157 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:27.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:27.258 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:27.285 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:27.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:27.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:27.386 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:27.428 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:27.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:27.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:27.530 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:27.569 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:27.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:27.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:27.627 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:27.671 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:27.729 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:27.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:27.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:27.831 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:27.875 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:27.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:27.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:27.889 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:21:27.889 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:21:27.920 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:21:27.920 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:21:27.951 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:21:27.952 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:21:27.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:27.987 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:21:27.987 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:21:28.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:28.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:28.016 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:21:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:21:28.050 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:21:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:21:28.087 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:21:28.087 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:21:28.117 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:28.123 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:21:28.123 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:21:28.158 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:28.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:28.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:28.260 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:28.298 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:28.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:28.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:28.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:28.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:28.443 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:28.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:28.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:28.588 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:28.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:28.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:28.690 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:28.730 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:28.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:28.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:28.732 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:28 smithi098 ceph-mon[97580]: pgmap v960: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 485 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 822 op/s; 17325/23154 objects degraded (74.825%); 4219/23154 objects misplaced (18.221%); 0 B/s, 68 objects/s recovering 2024-09-21T10:21:28.732 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:28 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3068145969' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:21:28.732 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:28 smithi098 ceph-mon[100914]: pgmap v960: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 485 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 822 op/s; 17325/23154 objects degraded (74.825%); 4219/23154 objects misplaced (18.221%); 0 B/s, 68 objects/s recovering 2024-09-21T10:21:28.732 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:28 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3068145969' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:21:28.832 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:28.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:28.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:28 smithi196 ceph-mon[84784]: pgmap v960: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 485 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 822 op/s; 17325/23154 objects degraded (74.825%); 4219/23154 objects misplaced (18.221%); 0 B/s, 68 objects/s recovering 2024-09-21T10:21:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:28 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3068145969' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:21:28.984 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:29.012 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:29.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:29.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:29.113 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:29.145 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:29.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:29.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:29.246 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:29.270 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:29.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:29.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:29.371 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:29.392 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:29.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:29.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:29.493 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:29.530 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:29.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:29.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:29.632 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:29.664 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:29.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:29.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:29.765 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:29.822 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:29.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:29.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:29.924 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:29.960 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:29.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:29.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:30.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:30.096 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:30.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:30.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:30.198 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:30.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:30.256 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:30.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:30.357 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:30.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:30.408 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:30.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:30.509 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:30.546 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:30.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:30.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:30.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:21:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:21:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:21:30.648 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:30.678 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:30.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:30.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:30.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:30.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:30.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:30.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:30.914 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:30.946 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:30.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:30.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:31.048 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:31.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:30 smithi098 ceph-mon[97580]: pgmap v961: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 355 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 862 op/s; 17325/16911 objects degraded (102.448%); 4219/16911 objects misplaced (24.948%); 0 B/s, 46 objects/s recovering 2024-09-21T10:21:31.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:21:31.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:30 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:21:31.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:30 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 17325/16911 objects degraded (102.448%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:31.075 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:21:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:21:30] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-09-21T10:21:31.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:30 smithi098 ceph-mon[100914]: pgmap v961: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 355 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 862 op/s; 17325/16911 objects degraded (102.448%); 4219/16911 objects misplaced (24.948%); 0 B/s, 46 objects/s recovering 2024-09-21T10:21:31.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:21:31.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:30 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:21:31.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:30 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 17325/16911 objects degraded (102.448%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:31.087 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:31.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:31.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:31.190 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:31.221 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:31.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:31.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:31.323 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:31.347 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:31.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:31.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:30 smithi196 ceph-mon[84784]: pgmap v961: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 355 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 862 op/s; 17325/16911 objects degraded (102.448%); 4219/16911 objects misplaced (24.948%); 0 B/s, 46 objects/s recovering 2024-09-21T10:21:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:21:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:30 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:21:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:30 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 17325/16911 objects degraded (102.448%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:31.448 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:31.489 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:31.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:31.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:31.590 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:31.621 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:31.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:31.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:31.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:31.757 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:31.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:31.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:31.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:31.903 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:31.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:31.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:32.005 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:32.036 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:32.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:32.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:32.137 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:32.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:32.178 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:32.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:32.279 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:32.327 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:32.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:32.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:32.428 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:32.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:32.470 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:32.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:32.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:32.597 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:32.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:32.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:32.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:32.732 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:32.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:32.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:32.767 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:32.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:32.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:32.892 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:32.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:32.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:33.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:33.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:33.039 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:32 smithi098 ceph-mon[97580]: pgmap v962: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 324 MiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 729 op/s; 17325/15399 objects degraded (112.507%); 4219/15399 objects misplaced (27.398%); 0 B/s, 46 objects/s recovering 2024-09-21T10:21:33.040 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:32 smithi098 ceph-mon[100914]: pgmap v962: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 324 MiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 729 op/s; 17325/15399 objects degraded (112.507%); 4219/15399 objects misplaced (27.398%); 0 B/s, 46 objects/s recovering 2024-09-21T10:21:33.133 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:33.176 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:33.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:33.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:33.278 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:33.311 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:33.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:33.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:33.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:32 smithi196 ceph-mon[84784]: pgmap v962: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 324 MiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 729 op/s; 17325/15399 objects degraded (112.507%); 4219/15399 objects misplaced (27.398%); 0 B/s, 46 objects/s recovering 2024-09-21T10:21:33.413 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:33.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:33.470 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:33.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:33.571 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:33.603 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:33.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:33.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:33.705 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:33.733 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:33.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:33.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:33.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:33.863 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:21:33.877 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:33.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:33.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:33.924 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:21:33.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:33.994 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:33 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3458022974' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:21:33.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:33 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3458022974' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:21:34.024 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:34.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:34.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:34.125 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:34.136 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Removed 103667 objects 2024-09-21T10:21:34.136 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Clean up completed and total clean up time :127.971 2024-09-21T10:21:34.147 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:34.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:34.248 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:34.269 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:34.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:34.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:34.371 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:34.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:33 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3458022974' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:21:34.391 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:34.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:34.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:34.493 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:34.514 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:34.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:34.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:34.615 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:34.623 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-21T10:21:34.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-21T10:21:34.635 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:34.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:34.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:34.737 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:34.758 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:34.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:34.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:34.860 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:34.881 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:34.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:34.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:34.982 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:35.003 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:35.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:35.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:35.104 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:35.125 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:35.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:35.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:35.227 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:35.247 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:35.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:35.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:35.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:34 smithi098 ceph-mon[97580]: pgmap v963: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 166 MiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 897 op/s; 17001/7818 objects degraded (217.460%); 4219/7818 objects misplaced (53.965%); 0 B/s, 73 objects/s recovering 2024-09-21T10:21:35.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:35 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:21:35.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:35 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3458022974' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:21:35.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:35 smithi098 ceph-mon[97580]: osdmap e295: 8 total, 8 up, 6 in 2024-09-21T10:21:35.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:34 smithi098 ceph-mon[100914]: pgmap v963: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 166 MiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 897 op/s; 17001/7818 objects degraded (217.460%); 4219/7818 objects misplaced (53.965%); 0 B/s, 73 objects/s recovering 2024-09-21T10:21:35.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:35 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:21:35.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:35 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3458022974' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:21:35.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:35 smithi098 ceph-mon[100914]: osdmap e295: 8 total, 8 up, 6 in 2024-09-21T10:21:35.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:35.371 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:35.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:35.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:35.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:34 smithi196 ceph-mon[84784]: pgmap v963: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 166 MiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 897 op/s; 17001/7818 objects degraded (217.460%); 4219/7818 objects misplaced (53.965%); 0 B/s, 73 objects/s recovering 2024-09-21T10:21:35.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:34 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:21:35.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:34 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3458022974' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:21:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:34 smithi196 ceph-mon[84784]: osdmap e295: 8 total, 8 up, 6 in 2024-09-21T10:21:35.473 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:35.494 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:35.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:35.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:35.596 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:35.616 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:35.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:35.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:35.718 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:35.738 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:35.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:35.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:35.840 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:35.861 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:35.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:35.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:35.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:35.985 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:35.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:35.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:36.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:36.108 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:36.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:36.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:36.210 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:36.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:36.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:36.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:36.265 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:35 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 17001/7818 objects degraded (217.460%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:36.265 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:35 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 17001/7818 objects degraded (217.460%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:36.332 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:36.353 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:36.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:36.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:36.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:35 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 17001/7818 objects degraded (217.460%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:36.455 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:36.475 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:36.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:36.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:36.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:36.600 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:36.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:36.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:36.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:36.723 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:36.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:36.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:36.825 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:36.846 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:36.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:36.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:36.948 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:36.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:36.971 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:36.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:37.071 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:37.092 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:37.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:37.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:37.194 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:37.215 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:37.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:37.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:37.316 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:37.337 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:37.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:37.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:37.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:36 smithi098 ceph-mon[97580]: pgmap v965: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 68 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 822 op/s; 17001/3153 objects degraded (539.201%); 4219/3153 objects misplaced (133.809%); 0 B/s, 61 objects/s recovering 2024-09-21T10:21:37.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:36 smithi098 ceph-mon[100914]: pgmap v965: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 68 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 822 op/s; 17001/3153 objects degraded (539.201%); 4219/3153 objects misplaced (133.809%); 0 B/s, 61 objects/s recovering 2024-09-21T10:21:37.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:36 smithi196 ceph-mon[84784]: pgmap v965: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 68 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 822 op/s; 17001/3153 objects degraded (539.201%); 4219/3153 objects misplaced (133.809%); 0 B/s, 61 objects/s recovering 2024-09-21T10:21:37.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:37.459 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:37.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:37.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:37.560 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:37.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:37.583 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:37.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:37.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:37.704 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:37.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:37.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:37.806 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:37.828 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:37.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:37.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:37.929 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:37.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:37.951 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:37.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:38.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:38.072 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:38.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:38.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:38.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:38.196 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:38.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:38.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:38.298 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:38.319 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:38.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:38.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:38.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:38.442 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:38.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:38.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:38.544 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:38.568 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:38.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:38.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:38.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:21:38.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:38.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:21:38.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:21:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:38.670 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:38.695 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:38.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:38.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:38.797 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:38.822 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:38.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:38.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:38.924 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:38.949 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:38.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:38.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:39.051 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:39.076 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:39.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:39.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:39.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:39.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:39.203 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:39.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:39.304 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:39.327 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:39.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:39.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:39.429 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:39.456 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:39.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:39.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:39.525 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:39 smithi098 ceph-mon[97580]: pgmap v966: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 27 MiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 732 op/s; 15958/1158 objects degraded (1378.066%); 4219/1158 objects misplaced (364.335%); 0 B/s, 136 objects/s recovering 2024-09-21T10:21:39.525 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:39 smithi098 ceph-mon[97580]: osdmap e296: 8 total, 8 up, 6 in 2024-09-21T10:21:39.525 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:39 smithi098 ceph-mon[100914]: pgmap v966: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 27 MiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 732 op/s; 15958/1158 objects degraded (1378.066%); 4219/1158 objects misplaced (364.335%); 0 B/s, 136 objects/s recovering 2024-09-21T10:21:39.525 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:39 smithi098 ceph-mon[100914]: osdmap e296: 8 total, 8 up, 6 in 2024-09-21T10:21:39.534 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:21:39.534 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:21:39.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:39 smithi196 ceph-mon[84784]: pgmap v966: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 27 MiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 732 op/s; 15958/1158 objects degraded (1378.066%); 4219/1158 objects misplaced (364.335%); 0 B/s, 136 objects/s recovering 2024-09-21T10:21:39.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:39 smithi196 ceph-mon[84784]: osdmap e296: 8 total, 8 up, 6 in 2024-09-21T10:21:39.558 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:39.583 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:39.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:39.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:39.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:39.709 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:39.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:39.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:39.811 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:39.836 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:39.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:39.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:39.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:39.960 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:39.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:39.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:40.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:40.088 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:40.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:40.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:40.190 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:40.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:40.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:40.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:40.338 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:40.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:40.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:40.439 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:40.465 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:40.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:40.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:40.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:40.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:40.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:40 smithi098 ceph-mon[97580]: osdmap e297: 8 total, 8 up, 6 in 2024-09-21T10:21:40.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:40 smithi098 ceph-mon[100914]: osdmap e297: 8 total, 8 up, 6 in 2024-09-21T10:21:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:40 smithi196 ceph-mon[84784]: osdmap e297: 8 total, 8 up, 6 in 2024-09-21T10:21:40.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:21:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:21:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:21:40.691 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:40.717 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:40.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:40.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:40.818 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:40.840 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:40.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:40.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:40.912 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:21:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:21:40] "GET /metrics HTTP/1.1" 200 36536 "" "Prometheus/2.51.0" 2024-09-21T10:21:40.914 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:21:40.941 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:40.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:40.967 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:40.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:41.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:41.093 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:41.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:41.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:41.195 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:41.221 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:41.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:41.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:41.271 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:21:41.322 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:41.348 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:41.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:41.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:41.450 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:41.473 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:41.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:41.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:41.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:41.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:41 smithi098 ceph-mon[100914]: pgmap v969: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 429 op/s; 15958/84 objects degraded (18997.619%); 4185/84 objects misplaced (4982.143%); 0 B/s, 179 objects/s recovering 2024-09-21T10:21:41.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:41 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 15958/84 objects degraded (18997.619%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:41.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:41 smithi098 ceph-mon[97580]: pgmap v969: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 429 op/s; 15958/84 objects degraded (18997.619%); 4185/84 objects misplaced (4982.143%); 0 B/s, 179 objects/s recovering 2024-09-21T10:21:41.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:41 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 15958/84 objects degraded (18997.619%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:41.598 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:41.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:41.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:41.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:41 smithi196 ceph-mon[84784]: pgmap v969: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 429 op/s; 15958/84 objects degraded (18997.619%); 4185/84 objects misplaced (4982.143%); 0 B/s, 179 objects/s recovering 2024-09-21T10:21:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:41 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 15958/84 objects degraded (18997.619%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:41.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:41.723 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:41.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:41.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:41.824 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:41.850 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:41.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:41.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:41.951 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:41.976 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:41.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:41.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:42.078 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:42.100 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:42.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:42.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:42.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:42.223 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:42.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:42.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:42.325 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:42.350 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:42.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:42.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:42.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:42.476 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:42.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:42.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:42.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:42.601 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:42.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:42.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:42.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:42.725 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:42.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:42.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:42.826 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:42.848 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:42.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:42.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:42.950 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:42.972 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:42.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:42.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:43.073 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:43.099 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:43.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:43.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:43.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:43.224 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:43.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:43.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:43.326 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:43.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:43 smithi098 ceph-mon[97580]: pgmap v970: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 205 op/s; 15958/84 objects degraded (18997.619%); 4185/84 objects misplaced (4982.143%); 0 B/s, 143 objects/s recovering 2024-09-21T10:21:43.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:43 smithi098 ceph-mon[100914]: pgmap v970: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 205 op/s; 15958/84 objects degraded (18997.619%); 4185/84 objects misplaced (4982.143%); 0 B/s, 143 objects/s recovering 2024-09-21T10:21:43.350 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:43.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:43.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:43.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:43.474 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:43.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:43.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:43.575 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:43.601 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:43.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:43.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:43.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:43 smithi196 ceph-mon[84784]: pgmap v970: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 205 op/s; 15958/84 objects degraded (18997.619%); 4185/84 objects misplaced (4982.143%); 0 B/s, 143 objects/s recovering 2024-09-21T10:21:43.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:43.725 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:43.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:43.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:43.827 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:43.849 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:43.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:43.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:43.910 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:43.951 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:43.969 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:44.006 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:44.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:44.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:44.109 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:44.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:44.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:44.235 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:44.261 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:44.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:44.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:44.363 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:44.388 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:44.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:44.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:44.490 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:44.512 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:44.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:44.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:44.613 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:44.640 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:44.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:44.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:44.741 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:44.764 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:44.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:44.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:44.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:44.888 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:44.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:44.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:44.990 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:45.016 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:45.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:45.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:45.118 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:45.146 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:45.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:45.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:45.251 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:45.274 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:45.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:45.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:45.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:45 smithi098 ceph-mon[97580]: pgmap v971: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 127 op/s; 15073/84 objects degraded (17944.048%); 4185/84 objects misplaced (4982.143%); 0 B/s, 245 objects/s recovering 2024-09-21T10:21:45.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:45 smithi098 ceph-mon[100914]: pgmap v971: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 127 op/s; 15073/84 objects degraded (17944.048%); 4185/84 objects misplaced (4982.143%); 0 B/s, 245 objects/s recovering 2024-09-21T10:21:45.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:45.398 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:45.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:45.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:45.499 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:45.525 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:45.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:45.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:45.627 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:45.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:45 smithi196 ceph-mon[84784]: pgmap v971: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 127 op/s; 15073/84 objects degraded (17944.048%); 4185/84 objects misplaced (4982.143%); 0 B/s, 245 objects/s recovering 2024-09-21T10:21:45.652 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:45.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:45.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:45.754 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:45.780 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:45.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:45.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:45.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:45.907 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:45.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:45.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:46.008 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:46.031 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:46.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:46.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:46.133 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:46.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:46.159 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:46.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:46.260 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:46.285 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:46.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:46.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:46.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:46.408 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:46.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:46.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:46.510 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:46.533 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:46.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:46.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:46.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:46 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 15073/84 objects degraded (17944.048%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:46.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:46 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 15073/84 objects degraded (17944.048%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:46.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:46.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:46 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 15073/84 objects degraded (17944.048%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:46.656 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:46.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:46.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:46.758 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:46.784 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:46.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:46.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:46.885 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:46.908 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:46.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:46.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:47.010 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:47.035 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:47.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:47.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:47.137 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:47.160 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:47.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:47.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:47.262 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:47.288 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:47.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:47.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:47.390 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:47.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:47.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:47.514 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:47.539 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:47.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:47.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:47.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:47 smithi098 ceph-mon[97580]: pgmap v972: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 44 op/s; 15073/84 objects degraded (17944.048%); 3663/84 objects misplaced (4360.714%); 0 B/s, 180 objects/s recovering 2024-09-21T10:21:47.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:47 smithi098 ceph-mon[100914]: pgmap v972: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 44 op/s; 15073/84 objects degraded (17944.048%); 3663/84 objects misplaced (4360.714%); 0 B/s, 180 objects/s recovering 2024-09-21T10:21:47.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:47 smithi196 ceph-mon[84784]: pgmap v972: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 44 op/s; 15073/84 objects degraded (17944.048%); 3663/84 objects misplaced (4360.714%); 0 B/s, 180 objects/s recovering 2024-09-21T10:21:47.641 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:47.666 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:47.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:47.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:47.768 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:47.793 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:47.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:47.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:47.895 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:47.919 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:47.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:47.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:48.021 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:48.047 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:48.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:48.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:48.149 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:48.174 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:48.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:48.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:48.276 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:48.298 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:48.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:48.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:48.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:48.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:48.426 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:48.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:48.527 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:48.553 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:48.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:48.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:48.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:48.677 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:48.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:48.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:48.778 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:48.804 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:48.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:48.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:48.906 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:48.931 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:48.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:48.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:49.033 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:49.054 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:49.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:49.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:49.156 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:49.181 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:49.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:49.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:49.282 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:49.307 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:49.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:49.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:49.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:49.432 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:49.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:49.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:49.534 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:49.556 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:49.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:49.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:49.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:49 smithi098 ceph-mon[97580]: pgmap v973: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 12 op/s; 14536/84 objects degraded (17304.762%); 3663/84 objects misplaced (4360.714%); 0 B/s, 215 objects/s recovering 2024-09-21T10:21:49.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:49 smithi098 ceph-mon[100914]: pgmap v973: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 12 op/s; 14536/84 objects degraded (17304.762%); 3663/84 objects misplaced (4360.714%); 0 B/s, 215 objects/s recovering 2024-09-21T10:21:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:49 smithi196 ceph-mon[84784]: pgmap v973: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 12 op/s; 14536/84 objects degraded (17304.762%); 3663/84 objects misplaced (4360.714%); 0 B/s, 215 objects/s recovering 2024-09-21T10:21:49.658 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:49.684 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:49.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:49.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:49.785 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:49.808 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:49.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:49.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:49.909 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:49.937 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:49.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:49.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:50.038 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:50.055 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:50.064 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:50.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:50.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:50.166 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:50.192 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:50.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:50.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:50.294 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:50.317 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:50.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:50.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:50.419 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:50.444 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:50.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:50.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:50.546 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:50.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:50.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:50.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:50.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:21:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:21:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:21:50.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:50.699 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:50.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:50.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:50.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:50.824 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:50.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:50.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:50.926 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:50.945 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:21:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:21:50] "GET /metrics HTTP/1.1" 200 36536 "" "Prometheus/2.51.0" 2024-09-21T10:21:50.951 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:50.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:50.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:51.053 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:51.078 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:51.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:51.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:51.180 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:51.203 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:51.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:51.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:51.305 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:51.331 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:51.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:51.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:51.432 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:51.458 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:51.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:51.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:51.559 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:51.584 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:51.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:51.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:51.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:51 smithi098 ceph-mon[97580]: pgmap v974: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 14536/84 objects degraded (17304.762%); 3195/84 objects misplaced (3803.571%); 0 B/s, 236 objects/s recovering 2024-09-21T10:21:51.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:51 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 14536/84 objects degraded (17304.762%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:51.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:51 smithi098 ceph-mon[100914]: pgmap v974: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 14536/84 objects degraded (17304.762%); 3195/84 objects misplaced (3803.571%); 0 B/s, 236 objects/s recovering 2024-09-21T10:21:51.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:51 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 14536/84 objects degraded (17304.762%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:51.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:51 smithi196 ceph-mon[84784]: pgmap v974: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 14536/84 objects degraded (17304.762%); 3195/84 objects misplaced (3803.571%); 0 B/s, 236 objects/s recovering 2024-09-21T10:21:51.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:51 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 14536/84 objects degraded (17304.762%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:51.686 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:51.711 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:51.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:51.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:51.812 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:51.836 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:51.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:51.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:51.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:51.960 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:51.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:51.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:52.061 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:52.087 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:52.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:52.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:52.189 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:52.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:52.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:52.304 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:21:52.304 INFO:teuthology.orchestra.run.smithi098.stdout:{"fsid":"26a15512-77fe-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14536/84 objects degraded (17304.762%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1409,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":297,"num_osds":8,"num_up_osds":8,"osd_up_since":1726912933,"num_in_osds":6,"osd_in_since":1726913912,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":84},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+remapped","count":1}],"num_pgs":89,"num_pools":5,"num_objects":28,"data_bytes":4790221,"bytes_used":4798558208,"bytes_avail":571171655680,"bytes_total":575970213888,"degraded_objects":14536,"degraded_total":84,"degraded_ratio":173.04761904761904,"misplaced_objects":3195,"misplaced_total":84,"misplaced_ratio":38.035714285714285,"recovering_objects_per_sec":236,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2412,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi098.stcwsu","status":"up:active","gid":14640}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.98:8443/","prometheus":"http://172.21.15.98:9283/"}},"servicemap":{"epoch":27,"modified":"2024-09-21T10:17:39.336971+0000","services":{}},"progress_events":{"6de3e7dc-7aed-4a0c-b700-51217030d18b":{"message":"Upgrade to 19.3.0-5092-g62da54b4 (4m)\n [============................] (remaining: 5m)","progress":0.44999998807907104,"add_to_ceph_s":true},"f7c971c5-96fe-4b4a-a1d0-3f1abe257d1d":{"message":"Global Recovery Event (3m)\n [==========================..] (remaining: 10s)","progress":0.94382023811340332,"add_to_ceph_s":true}}} 2024-09-21T10:21:52.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:52.336 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:52.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:52.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:52.438 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:52.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:52.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:52.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:52.584 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:52.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:52.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:52.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:52 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2722920047' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:21:52.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:52.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:52 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2722920047' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:21:52.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:52.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:52 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2722920047' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:21:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:52.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:52.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:52.710 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:52.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:52.811 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:52.836 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:52.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:52.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:52.938 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:52.963 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:52.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:52.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:53.065 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:53.089 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:53.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:53.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:53.191 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:53.214 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:53.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:53.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:53.315 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:53.341 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:53.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:53.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:53.443 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:53.469 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:53.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:53.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:53.571 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:53.594 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:53.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:53.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:53.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:53 smithi098 ceph-mon[97580]: pgmap v975: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 14536/84 objects degraded (17304.762%); 3195/84 objects misplaced (3803.571%); 0 B/s, 200 objects/s recovering 2024-09-21T10:21:53.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:53 smithi098 ceph-mon[100914]: pgmap v975: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 14536/84 objects degraded (17304.762%); 3195/84 objects misplaced (3803.571%); 0 B/s, 200 objects/s recovering 2024-09-21T10:21:53.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:53 smithi196 ceph-mon[84784]: pgmap v975: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 14536/84 objects degraded (17304.762%); 3195/84 objects misplaced (3803.571%); 0 B/s, 200 objects/s recovering 2024-09-21T10:21:53.696 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:53.722 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:53.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:53.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:53.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:53.848 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:53.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:53.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:53.949 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:53.974 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:53.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:53.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:54.076 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:54.100 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:54.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:54.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:54.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:54.225 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:54.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:54.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:54.327 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:54.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:54.354 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:54.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:54.455 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:54.481 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:54.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:54.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:54.582 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:54.607 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:54.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:54.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:54.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:54.731 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:54.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:54.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:54.833 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:54.859 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:54.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:54.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:54.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:54.986 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:54.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:54.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:55.088 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:55.113 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:55.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:55.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:55.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:55.237 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:55.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:55.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:55.338 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:55.364 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:55.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:55.466 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:55.491 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:55.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:55.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:55.592 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:55.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:55 smithi098 ceph-mon[97580]: pgmap v976: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 13933/84 objects degraded (16586.905%); 3195/84 objects misplaced (3803.571%); 0 B/s, 251 objects/s recovering 2024-09-21T10:21:55.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:55 smithi098 ceph-mon[100914]: pgmap v976: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 13933/84 objects degraded (16586.905%); 3195/84 objects misplaced (3803.571%); 0 B/s, 251 objects/s recovering 2024-09-21T10:21:55.617 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:55.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:55.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:55.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:55 smithi196 ceph-mon[84784]: pgmap v976: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 13933/84 objects degraded (16586.905%); 3195/84 objects misplaced (3803.571%); 0 B/s, 251 objects/s recovering 2024-09-21T10:21:55.719 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:55.745 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:55.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:55.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:55.846 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:55.870 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:55.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:55.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:55.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:55.997 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:55.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:55.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:56.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:56.123 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:56.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:56.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:56.225 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:56.247 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:56.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:56.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:56.349 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:56.371 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:56.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:56.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:56.472 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:56.494 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:56.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:56.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:56.596 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:56.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:56 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 13933/84 objects degraded (16586.905%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:56 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 13933/84 objects degraded (16586.905%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:56.622 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:56.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:56.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:56.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:56 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 13933/84 objects degraded (16586.905%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:21:56.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:56.746 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:56.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:56.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:56.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:56.870 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:56.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:56.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:56.972 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:56.994 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:56.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:56.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:57.095 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:57.121 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:57.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:57.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:57.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:57.246 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:57.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:57.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:57.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:57.370 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:57.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:57.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:57.471 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:57.494 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:57.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:57.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:57.596 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:57.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:57 smithi098 ceph-mon[97580]: pgmap v977: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 13933/84 objects degraded (16586.905%); 2687/84 objects misplaced (3198.810%); 0 B/s, 219 objects/s recovering 2024-09-21T10:21:57.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:57 smithi098 ceph-mon[100914]: pgmap v977: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 13933/84 objects degraded (16586.905%); 2687/84 objects misplaced (3198.810%); 0 B/s, 219 objects/s recovering 2024-09-21T10:21:57.617 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:57.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:57.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:57.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:57 smithi196 ceph-mon[84784]: pgmap v977: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 13933/84 objects degraded (16586.905%); 2687/84 objects misplaced (3198.810%); 0 B/s, 219 objects/s recovering 2024-09-21T10:21:57.719 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:57.745 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:57.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:57.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:57.846 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:57.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:57.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:57.996 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:57.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:57.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:58.098 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:58.121 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:58.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:58.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:58.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:58.246 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:58.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:58.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:58.348 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:58.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:58.375 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:58.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:58.476 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:58.501 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:58.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:58.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:58.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:58.624 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:58.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:58.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:58.726 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:58.748 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:58.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:58.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:58.850 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:58.875 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:58.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:58.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:58.977 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:59.003 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:59.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:59.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:59.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:59.128 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:59.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:59.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:59.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:59.253 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:59.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:59.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:59.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:21:59 smithi098 ceph-mon[97580]: pgmap v978: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 13226/84 objects degraded (15745.238%); 2687/84 objects misplaced (3198.810%); 0 B/s, 235 objects/s recovering 2024-09-21T10:21:59.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:21:59 smithi098 ceph-mon[100914]: pgmap v978: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 13226/84 objects degraded (15745.238%); 2687/84 objects misplaced (3198.810%); 0 B/s, 235 objects/s recovering 2024-09-21T10:21:59.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:59.377 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:59.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:59.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:59.478 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:59.502 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:59.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:59.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:59.603 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:59.626 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:59.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:59.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:59.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:21:59 smithi196 ceph-mon[84784]: pgmap v978: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 13226/84 objects degraded (15745.238%); 2687/84 objects misplaced (3198.810%); 0 B/s, 235 objects/s recovering 2024-09-21T10:21:59.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:59.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:59.751 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:21:59.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:59.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:59.875 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:21:59.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:59.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:59.976 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:59.999 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:00.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:00.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:00.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:00.126 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:00.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:00.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:00.227 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:00.250 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:00.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:00.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:00.352 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:00.374 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:00.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:00.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:00.475 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:00.497 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:00.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:00.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:00.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:00.621 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:00.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:00.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:00.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:22:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:22:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:22:00.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:00.746 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:00.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:00.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:00.848 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:00.863 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:22:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:22:00] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-09-21T10:22:00.870 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:00.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:00.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:00.972 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:00.995 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:00.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:00.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:01.096 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:01.121 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:01.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:01.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:01.222 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:01.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:01.247 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:01.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:01.348 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:01.371 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:01.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:01.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:01.433 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-21T10:22:01.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:22:01.473 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:01.497 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:01.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:01.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:01 smithi098 ceph-mon[97580]: pgmap v979: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 13226/84 objects degraded (15745.238%); 2117/84 objects misplaced (2520.238%); 0 B/s, 237 objects/s recovering 2024-09-21T10:22:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:01 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 13226/84 objects degraded (15745.238%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:22:01.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:01 smithi098 ceph-mon[100914]: pgmap v979: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 13226/84 objects degraded (15745.238%); 2117/84 objects misplaced (2520.238%); 0 B/s, 237 objects/s recovering 2024-09-21T10:22:01.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:01 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 13226/84 objects degraded (15745.238%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:22:01.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:01.623 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:01.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:01.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:01.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:01 smithi196 ceph-mon[84784]: pgmap v979: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 13226/84 objects degraded (15745.238%); 2117/84 objects misplaced (2520.238%); 0 B/s, 237 objects/s recovering 2024-09-21T10:22:01.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:01 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 13226/84 objects degraded (15745.238%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:22:01.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:01.748 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:01.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:01.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:01.850 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:01.876 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:01.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:01.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:01.978 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:02.003 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:02.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:02.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:02.105 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:02.130 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:02.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:02.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:02.232 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:02.257 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:02.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:02.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:02.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:02.383 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:02.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:02.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:02.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:02.509 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:02.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:02.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:02.611 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:02.636 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:02.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:02.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:02.738 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:02.764 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:02.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:02.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:02.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:02.888 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:02.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:02.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:02.990 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:03.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:03.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:03.036 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:22:03.036 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:22:03.115 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:03.139 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:03.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:03.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:03.240 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:03.266 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:03.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:03.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:03.368 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:03.390 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:03.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:03.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:03.492 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:03.514 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:03.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:03.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:03.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:03 smithi098 ceph-mon[97580]: pgmap v980: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 13226/84 objects degraded (15745.238%); 2117/84 objects misplaced (2520.238%); 0 B/s, 198 objects/s recovering 2024-09-21T10:22:03.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:03 smithi098 ceph-mon[100914]: pgmap v980: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 13226/84 objects degraded (15745.238%); 2117/84 objects misplaced (2520.238%); 0 B/s, 198 objects/s recovering 2024-09-21T10:22:03.615 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:03.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:03 smithi196 ceph-mon[84784]: pgmap v980: 89 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 13226/84 objects degraded (15745.238%); 2117/84 objects misplaced (2520.238%); 0 B/s, 198 objects/s recovering 2024-09-21T10:22:03.639 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:03.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:03.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:03.741 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:03.766 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:03.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:03.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:03.868 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:03.893 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:03.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:03.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:03.995 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:04.019 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:04.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:04.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:04.120 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:04.144 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:04.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:04.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:04.246 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:04.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:04.271 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:04.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:04.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:04.397 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:04.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:04.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:04.499 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:04.523 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:04.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:04.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:04.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:04.648 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:04.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:04.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:04.750 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:04.774 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:04.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:04.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:04.875 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:04.901 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:04.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:04.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:05.003 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:05.030 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:05.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:05.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:05.133 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:05.158 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:05.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:05.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:05.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:05.282 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:05.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:05.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:05.384 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:05.412 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:05.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:05.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:05.448 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'unique_pool_3' removed 2024-09-21T10:22:05.514 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:05.550 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:05.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:05.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:05.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:05 smithi098 ceph-mon[97580]: pgmap v981: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 12355/84 objects degraded (14708.333%); 2117/84 objects misplaced (2520.238%); 0 B/s, 271 objects/s recovering 2024-09-21T10:22:05.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:05 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1578105444' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:22:05.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:05 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:22:05.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:05 smithi098 ceph-mon[100914]: pgmap v981: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 12355/84 objects degraded (14708.333%); 2117/84 objects misplaced (2520.238%); 0 B/s, 271 objects/s recovering 2024-09-21T10:22:05.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:05 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1578105444' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:22:05.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:05 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:22:05.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:05 smithi196 ceph-mon[84784]: pgmap v981: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 12355/84 objects degraded (14708.333%); 2117/84 objects misplaced (2520.238%); 0 B/s, 271 objects/s recovering 2024-09-21T10:22:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:05 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1578105444' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:22:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:05 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:22:05.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:05.677 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:05.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:05.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:05.779 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:05.800 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:05.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:05.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:05.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:05.923 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:05.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:05.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:06.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:06.046 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:06.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:06.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:06.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:06.169 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:06.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:06.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:06.271 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:06.296 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:06.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:06.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:06.398 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:06.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:06.421 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:06.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:06.522 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:06.548 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:06.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:06.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:06.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:06 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:22:06.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:06 smithi098 ceph-mon[97580]: osdmap e298: 8 total, 8 up, 6 in 2024-09-21T10:22:06.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:06 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 12355/84 objects degraded (14708.333%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:22:06.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:06 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:22:06.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:06 smithi098 ceph-mon[100914]: osdmap e298: 8 total, 8 up, 6 in 2024-09-21T10:22:06.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:06 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 12355/84 objects degraded (14708.333%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:22:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:06 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:22:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:06 smithi196 ceph-mon[84784]: osdmap e298: 8 total, 8 up, 6 in 2024-09-21T10:22:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:06 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 12355/84 objects degraded (14708.333%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:22:06.650 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:06.675 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:06.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:06.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:06.777 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:06.802 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:06.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:06.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:06.904 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:06.931 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:06.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:06.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:07.034 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:07.060 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:07.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:07.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:07.162 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:07.187 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:07.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:07.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:07.289 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:07.314 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:07.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:07.416 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:07.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:07.443 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:07.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:07.545 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:07.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:07.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:07.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:07 smithi098 ceph-mon[97580]: pgmap v983: 73 pgs: 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:07 smithi098 ceph-mon[97580]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12355/84 objects degraded (14708.333%), 4 pgs degraded, 4 pgs undersized) 2024-09-21T10:22:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:07.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:07.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:07 smithi098 ceph-mon[100914]: pgmap v983: 73 pgs: 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:07.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:07 smithi098 ceph-mon[100914]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12355/84 objects degraded (14708.333%), 4 pgs degraded, 4 pgs undersized) 2024-09-21T10:22:07.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:07.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:07.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:07 smithi196 ceph-mon[84784]: pgmap v983: 73 pgs: 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:07 smithi196 ceph-mon[84784]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12355/84 objects degraded (14708.333%), 4 pgs degraded, 4 pgs undersized) 2024-09-21T10:22:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:07.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:07.697 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:07.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:07.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:07.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:07.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:07.824 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:07.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:07.925 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:07.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:07.952 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:07.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:08.053 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:08.079 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:08.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:08.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:08.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:08.206 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:08.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:08.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:08.308 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:08.335 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:08.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:08.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:08.437 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:08.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:08.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:08.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:08.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:08.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:08 smithi098 ceph-mon[97580]: pgmap v984: 73 pgs: 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:08.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:08 smithi098 ceph-mon[100914]: pgmap v984: 73 pgs: 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:08.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:08 smithi196 ceph-mon[84784]: pgmap v984: 73 pgs: 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:08.690 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:08.715 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:08.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:08.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:08.766 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:22:08.767 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:22:08.767 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-21T10:22:08.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_4 16 2024-09-21T10:22:08.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:08.842 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:08.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:08.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:08.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:08.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:08.971 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:08.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:09.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:09.094 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:09.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:09.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:09.195 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:09.216 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:09.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:09.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:09.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:09.331 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:22:09.341 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:09.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:09.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:09.442 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:09.468 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:09.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:09.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:09.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:09.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:09 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2723687223' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:22:09.583 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:09 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2723687223' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:22:09.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:22:09.595 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:09.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:09.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:09.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:09 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2723687223' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:22:09.696 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:09.717 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:09.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:09.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:09.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:09.840 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:09.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:09.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:09.941 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:09.963 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:09.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:09.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:10.029 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:22:10.029 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:22:10.041 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:22:10.042 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:22:10.059 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:22:10.059 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:22:10.065 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:10.076 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:22:10.076 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:22:10.092 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:10.092 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:22:10.093 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:22:10.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:10.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:10.108 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:22:10.108 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:22:10.124 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:22:10.124 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:22:10.140 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:22:10.140 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:22:10.194 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:10.215 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:10.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:10.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:10.316 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:10.343 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:10.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:10.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:10.444 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:10.469 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:10.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:10.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:10.571 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:10.595 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:10.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:10.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:10.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:10 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:22:10.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:10 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2723687223' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:22:10.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:10 smithi098 ceph-mon[100914]: osdmap e299: 8 total, 8 up, 6 in 2024-09-21T10:22:10.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:10 smithi098 ceph-mon[100914]: pgmap v986: 73 pgs: 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:10.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:10 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1170421128' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:10.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:10 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:22:10.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:10 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2723687223' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:22:10.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:10 smithi098 ceph-mon[97580]: osdmap e299: 8 total, 8 up, 6 in 2024-09-21T10:22:10.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:10 smithi098 ceph-mon[97580]: pgmap v986: 73 pgs: 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:10.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:10 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1170421128' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:10.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:10 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:22:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:10 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2723687223' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:22:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:10 smithi196 ceph-mon[84784]: osdmap e299: 8 total, 8 up, 6 in 2024-09-21T10:22:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:10 smithi196 ceph-mon[84784]: pgmap v986: 73 pgs: 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:10 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1170421128' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:10.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:22:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:22:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:22:10.698 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:10.723 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:10.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:10.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:10.825 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:10.850 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:10.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:10.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:10.952 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:10.975 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:10.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:10.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:11.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:11.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:22:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:22:10] "GET /metrics HTTP/1.1" 200 34174 "" "Prometheus/2.51.0" 2024-09-21T10:22:11.098 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:11.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:11.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:11.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:11.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:11.223 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:11.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:11.324 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:11.350 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:11.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:11.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:11.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:11.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:11.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:11.575 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:11.580 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:11.599 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:11.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:11.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:11.701 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:11.722 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:11.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:11.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:11.824 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:11.844 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:11.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:11.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:11.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:11.967 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:11.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:11.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:12.058 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:12.069 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:12.094 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:12.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:12.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:12.196 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:12.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:12.219 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:12.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:12.319 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:12.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:12.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:12.443 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:12.469 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:12.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:12.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:12.571 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:12.593 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:12.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:12.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:12.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:12 smithi098 ceph-mon[97580]: pgmap v987: 73 pgs: 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:12.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:12 smithi098 ceph-mon[100914]: pgmap v987: 73 pgs: 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:12.695 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:12.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:12.722 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:12.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:12.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:12.848 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:12.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:12.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:12 smithi196 ceph-mon[84784]: pgmap v987: 73 pgs: 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:12.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:12.975 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:12.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:12.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:13.077 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:13.102 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:13.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:13.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:13.204 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:13.229 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:13.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:13.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:13.253 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:22:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:13.331 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:13.353 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:13.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:13.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:13.455 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:13.480 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:13.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:13.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:13.582 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:13.607 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:13.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:13.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:13.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:13.731 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:13.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:13.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:13.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:13.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:13 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:13.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:13 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:13.855 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:13.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:13.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:13.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:13 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:13.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:13.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:13.982 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:13.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:14.071 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:22:14.083 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:14.108 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:14.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:14.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:14.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:14.231 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:14.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:14.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:14.333 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:14.354 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:14.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:14.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:14.455 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:14.476 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:14.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:14.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:14.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:14.601 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:14.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:14.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:14.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:14.725 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:14.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:14.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:14.826 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:14.852 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:14.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:14.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:14 smithi196 ceph-mon[84784]: pgmap v988: 73 pgs: 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:14.925 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:14 smithi098 ceph-mon[97580]: pgmap v988: 73 pgs: 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:14.925 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:14 smithi098 ceph-mon[100914]: pgmap v988: 73 pgs: 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:14.954 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:14.975 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:14.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:14.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:15.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:15.098 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:15.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:15.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:15.200 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:15.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:15.221 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:15.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:15.322 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:15.348 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:15.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:15.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:15.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:15.471 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:15.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:15.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:15.572 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:15.599 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:15.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:15.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:15.701 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:15.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:15.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:15.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:15.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:15.852 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:15.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:15.954 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:15.976 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:15.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:15.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:16.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:16.099 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:16.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:16.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:16.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:16.223 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:16.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:16.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:16.324 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:16.346 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:16.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:16.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:16.448 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:16.474 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:16.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:16.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:16.576 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:16.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:16.601 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:16.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:16.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:16.725 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:16.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:16.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:16.826 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:16.847 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:16.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:16.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:16 smithi196 ceph-mon[84784]: pgmap v989: 73 pgs: 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:16.906 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:16 smithi098 ceph-mon[97580]: pgmap v989: 73 pgs: 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:16.906 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:16 smithi098 ceph-mon[100914]: pgmap v989: 73 pgs: 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:16.949 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:16.975 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:16.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:16.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:17.076 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:17.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:17.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:17.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:17.203 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:17.228 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:17.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:17.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:17.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:17.352 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:17.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:17.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:17.453 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:17.475 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:17.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:17.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:17.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:17.598 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:17.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:17.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:17.645 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'unique_pool_4' created 2024-09-21T10:22:17.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:17.721 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:17.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:17.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:17.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:17.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:17.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:17.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:17 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2152168304' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T10:22:17.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:17 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T10:22:17.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:17 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2152168304' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T10:22:17.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:17 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T10:22:17.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:17 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2152168304' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T10:22:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:17 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T10:22:17.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:17.967 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:17.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:17.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:18.069 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:18.095 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:18.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:18.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:18.192 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:22:18.192 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:22:18.192 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-21T10:22:18.192 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:22:18.197 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:18.218 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:18.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:18.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:18.320 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:18.341 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:18.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:18.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:18.442 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:18.468 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:18.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:18.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:18.570 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:18.591 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:18.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:18.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:18.693 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:18.718 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:18.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:18.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:18.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:18.844 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:18.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:18.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:18.893 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:18 smithi098 ceph-mon[97580]: pgmap v990: 73 pgs: 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:18.893 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:18 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-21T10:22:18.893 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:18 smithi098 ceph-mon[97580]: osdmap e300: 8 total, 8 up, 6 in 2024-09-21T10:22:18.894 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:18 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:18.894 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:18 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:18.894 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:18 smithi098 ceph-mon[100914]: pgmap v990: 73 pgs: 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:18.894 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:18 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-21T10:22:18.894 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:18 smithi098 ceph-mon[100914]: osdmap e300: 8 total, 8 up, 6 in 2024-09-21T10:22:18.894 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:18 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:18.894 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:18 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:18.895 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:22:18.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:18.967 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:18.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:18.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:19.069 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:19.095 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:19.096 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:19.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:19.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:18 smithi196 ceph-mon[84784]: pgmap v990: 73 pgs: 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:18 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-21T10:22:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:18 smithi196 ceph-mon[84784]: osdmap e300: 8 total, 8 up, 6 in 2024-09-21T10:22:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:18 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:18 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:19.198 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:19.223 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:19.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:19.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:19.325 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:19.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:19.348 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:19.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:19.449 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:19.474 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:19.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:19.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:19.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-21T10:22:19.575 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:19.600 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:19.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:19.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:19.701 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:19.723 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:19.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:19.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:19.801 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:19 smithi098 ceph-mon[97580]: osdmap e301: 8 total, 8 up, 6 in 2024-09-21T10:22:19.802 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:19 smithi098 ceph-mon[100914]: osdmap e301: 8 total, 8 up, 6 in 2024-09-21T10:22:19.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:19.847 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:19.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:19.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:19.948 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:19.969 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:19.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:19.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:20.071 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:20.092 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:20.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:20.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:20.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:19 smithi196 ceph-mon[84784]: osdmap e301: 8 total, 8 up, 6 in 2024-09-21T10:22:20.194 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:20.215 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:20.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:20.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:20.316 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:20.343 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:20.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:20.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:20.444 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:20.446 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:20.469 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:20.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:20.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:20.487 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:20.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:20.594 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:20.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:20.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:20.628 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:22:20.629 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:22:20.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:22:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:22:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:22:20.696 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:20.718 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:20.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:20.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:20.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:20.842 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:20.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:20.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:20.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:22:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:22:20] "GET /metrics HTTP/1.1" 200 34174 "" "Prometheus/2.51.0" 2024-09-21T10:22:20.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:20 smithi098 ceph-mon[97580]: pgmap v993: 89 pgs: 10 creating+peering, 6 unknown, 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:20.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:20 smithi098 ceph-mon[97580]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:22:20.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:20 smithi098 ceph-mon[97580]: osdmap e302: 8 total, 8 up, 6 in 2024-09-21T10:22:20.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:20 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:22:20.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:20 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:22:20.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:20 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:20.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:20 smithi098 ceph-mon[100914]: pgmap v993: 89 pgs: 10 creating+peering, 6 unknown, 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:20.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:20 smithi098 ceph-mon[100914]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:22:20.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:20 smithi098 ceph-mon[100914]: osdmap e302: 8 total, 8 up, 6 in 2024-09-21T10:22:20.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:20 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:22:20.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:20 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:22:20.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:20 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:20.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:20.965 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:20.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:20.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:21.067 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:21.088 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:21.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:21.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:20 smithi196 ceph-mon[84784]: pgmap v993: 89 pgs: 10 creating+peering, 6 unknown, 73 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:20 smithi196 ceph-mon[84784]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:22:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:20 smithi196 ceph-mon[84784]: osdmap e302: 8 total, 8 up, 6 in 2024-09-21T10:22:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:20 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:22:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:20 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:22:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:20 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:21.189 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:21.210 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:21.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:21.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:21.312 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:21.338 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:21.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:21.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:21.440 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:21.460 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:21.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:21.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:21.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:21.584 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:21.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:21.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:21.586 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":994,"stamp":"2024-09-21T10:22:19.649802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6288,"num_read_kb":13764,"num_write":1056,"num_write_kb":14251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":600384,"num_keys_recovered":30,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1132,"ondisk_log_size":1132,"up":252,"acting":252,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":224,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":4694644,"kb_used_data":36840,"kb_used_omap":28,"kb_used_meta":4657699,"kb_avail":557776268,"statfs":{"total":575970213888,"available":571162898432,"internally_reserved":0,"allocated":37724160,"data_stored":27964145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29289,"internal_metadata":4769484183},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.209336"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":121,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:10.192419+0000","last_change":"2024-09-21T10:18:33.471373+0000","last_active":"2024-09-21T10:22:10.192419+0000","last_peered":"2024-09-21T10:22:10.192419+0000","last_clean":"2024-09-21T10:22:10.192419+0000","last_became_active":"2024-09-21T10:18:33.471052+0000","last_became_peered":"2024-09-21T10:18:33.471052+0000","last_unstale":"2024-09-21T10:22:10.192419+0000","last_undegraded":"2024-09-21T10:22:10.192419+0000","last_fullsized":"2024-09-21T10:22:10.192419+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:42:19.491129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":314,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.331902+0000","last_change":"2024-09-21T10:08:00.406360+0000","last_active":"2024-09-21T10:22:09.331902+0000","last_peered":"2024-09-21T10:22:09.331902+0000","last_clean":"2024-09-21T10:22:09.331902+0000","last_became_active":"2024-09-21T10:08:00.380042+0000","last_became_peered":"2024-09-21T10:08:00.380042+0000","last_unstale":"2024-09-21T10:22:09.331902+0000","last_undegraded":"2024-09-21T10:22:09.331902+0000","last_fullsized":"2024-09-21T10:22:09.331902+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:49:46.544051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":230,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.331950+0000","last_change":"2024-09-21T10:18:33.465357+0000","last_active":"2024-09-21T10:22:09.331950+0000","last_peered":"2024-09-21T10:22:09.331950+0000","last_clean":"2024-09-21T10:22:09.331950+0000","last_became_active":"2024-09-21T10:18:33.464952+0000","last_became_peered":"2024-09-21T10:18:33.464952+0000","last_unstale":"2024-09-21T10:22:09.331950+0000","last_undegraded":"2024-09-21T10:22:09.331950+0000","last_fullsized":"2024-09-21T10:22:09.331950+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:15:41.090866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":106,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.331722+0000","last_change":"2024-09-21T10:18:33.475291+0000","last_active":"2024-09-21T10:22:09.331722+0000","last_peered":"2024-09-21T10:22:09.331722+0000","last_clean":"2024-09-21T10:22:09.331722+0000","last_became_active":"2024-09-21T10:18:33.474971+0000","last_became_peered":"2024-09-21T10:18:33.474971+0000","last_unstale":"2024-09-21T10:22:09.331722+0000","last_undegraded":"2024-09-21T10:22:09.331722+0000","last_fullsized":"2024-09-21T10:22:09.331722+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:39:40.179668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":476,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T10:22:17.648417+0000","last_change":"2024-09-21T10:18:33.475179+0000","last_active":"2024-09-21T10:22:17.648417+0000","last_peered":"2024-09-21T10:22:17.648417+0000","last_clean":"2024-09-21T10:22:17.648417+0000","last_became_active":"2024-09-21T10:18:33.474914+0000","last_became_peered":"2024-09-21T10:18:33.474914+0000","last_unstale":"2024-09-21T10:22:17.648417+0000","last_undegraded":"2024-09-21T10:22:17.648417+0000","last_fullsized":"2024-09-21T10:22:17.648417+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:15:32.186986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":180,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.765010+0000","last_change":"2024-09-21T10:08:00.385671+0000","last_active":"2024-09-21T10:22:09.765010+0000","last_peered":"2024-09-21T10:22:09.765010+0000","last_clean":"2024-09-21T10:22:09.765010+0000","last_became_active":"2024-09-21T10:08:00.385368+0000","last_became_peered":"2024-09-21T10:08:00.385368+0000","last_unstale":"2024-09-21T10:22:09.765010+0000","last_undegraded":"2024-09-21T10:22:09.765010+0000","last_fullsized":"2024-09-21T10:22:09.765010+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:21.799149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":121,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:10.192747+0000","last_change":"2024-09-21T10:18:33.481890+0000","last_active":"2024-09-21T10:22:10.192747+0000","last_peered":"2024-09-21T10:22:10.192747+0000","last_clean":"2024-09-21T10:22:10.192747+0000","last_became_active":"2024-09-21T10:18:33.480639+0000","last_became_peered":"2024-09-21T10:18:33.480639+0000","last_unstale":"2024-09-21T10:22:10.192747+0000","last_undegraded":"2024-09-21T10:22:10.192747+0000","last_fullsized":"2024-09-21T10:22:10.192747+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:44.797540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":272,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.764906+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:22:09.764906+0000","last_peered":"2024-09-21T10:22:09.764906+0000","last_clean":"2024-09-21T10:22:09.764906+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:22:09.764906+0000","last_undegraded":"2024-09-21T10:22:09.764906+0000","last_fullsized":"2024-09-21T10:22:09.764906+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":300,"state":"creating+peering","last_fresh":"2024-09-21T10:22:17.645672+0000","last_change":"2024-09-21T10:22:17.645672+0000","last_active":"2024-09-21T10:22:17.626792+0000","last_peered":"2024-09-21T10:22:17.626792+0000","last_clean":"2024-09-21T10:22:17.626792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:22:17.645672+0000","last_undegraded":"2024-09-21T10:22:17.645672+0000","last_fullsized":"2024-09-21T10:22:17.645672+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":307,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.332280+0000","last_change":"2024-09-21T10:08:00.392499+0000","last_active":"2024-09-21T10:22:09.332280+0000","last_peered":"2024-09-21T10:22:09.332280+0000","last_clean":"2024-09-21T10:22:09.332280+0000","last_became_active":"2024-09-21T10:08:00.379165+0000","last_became_peered":"2024-09-21T10:08:00.379165+0000","last_unstale":"2024-09-21T10:22:09.332280+0000","last_undegraded":"2024-09-21T10:22:09.332280+0000","last_fullsized":"2024-09-21T10:22:09.332280+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:18.933144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":163,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.331512+0000","last_change":"2024-09-21T10:07:43.794677+0000","last_active":"2024-09-21T10:22:09.331512+0000","last_peered":"2024-09-21T10:22:09.331512+0000","last_clean":"2024-09-21T10:22:09.331512+0000","last_became_active":"2024-09-21T10:07:43.794348+0000","last_became_peered":"2024-09-21T10:07:43.794348+0000","last_unstale":"2024-09-21T10:22:09.331512+0000","last_undegraded":"2024-09-21T10:22:09.331512+0000","last_fullsized":"2024-09-21T10:22:09.331512+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:11.159014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T10:22:17.626792+0000","last_change":"2024-09-21T10:22:17.626792+0000","last_active":"2024-09-21T10:22:17.626792+0000","last_peered":"2024-09-21T10:22:17.626792+0000","last_clean":"2024-09-21T10:22:17.626792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:22:17.626792+0000","last_undegraded":"2024-09-21T10:22:17.626792+0000","last_fullsized":"2024-09-21T10:22:17.626792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":308,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.332239+0000","last_change":"2024-09-21T10:08:00.412449+0000","last_active":"2024-09-21T10:22:09.332239+0000","last_peered":"2024-09-21T10:22:09.332239+0000","last_clean":"2024-09-21T10:22:09.332239+0000","last_became_active":"2024-09-21T10:08:00.386530+0000","last_became_peered":"2024-09-21T10:08:00.386530+0000","last_unstale":"2024-09-21T10:22:09.332239+0000","last_undegraded":"2024-09-21T10:22:09.332239+0000","last_fullsized":"2024-09-21T10:22:09.332239+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:12:50.581176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":400,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:10.193422+0000","last_change":"2024-09-21T10:18:33.479057+0000","last_active":"2024-09-21T10:22:10.193422+0000","last_peered":"2024-09-21T10:22:10.193422+0000","last_clean":"2024-09-21T10:22:10.193422+0000","last_became_active":"2024-09-21T10:18:33.478752+0000","last_became_peered":"2024-09-21T10:18:33.478752+0000","last_unstale":"2024-09-21T10:22:10.193422+0000","last_undegraded":"2024-09-21T10:22:10.193422+0000","last_fullsized":"2024-09-21T10:22:10.193422+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:30.068585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-21T10:22:17.626792+0000","last_change":"2024-09-21T10:22:17.643906+0000","last_active":"2024-09-21T10:22:17.626792+0000","last_peered":"2024-09-21T10:22:17.626792+0000","last_clean":"2024-09-21T10:22:17.626792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:22:17.626792+0000","last_undegraded":"2024-09-21T10:22:17.626792+0000","last_fullsized":"2024-09-21T10:22:17.626792+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:33:55.232554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":248,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.332096+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:22:09.332096+0000","last_peered":"2024-09-21T10:22:09.332096+0000","last_clean":"2024-09-21T10:22:09.332096+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:22:09.332096+0000","last_undegraded":"2024-09-21T10:22:09.332096+0000","last_fullsized":"2024-09-21T10:22:09.332096+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":263,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.765011+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:22:09.765011+0000","last_peered":"2024-09-21T10:22:09.765011+0000","last_clean":"2024-09-21T10:22:09.765011+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:22:09.765011+0000","last_undegraded":"2024-09-21T10:22:09.765011+0000","last_fullsized":"2024-09-21T10:22:09.765011+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T10:22:19.107586+0000","last_change":"2024-09-21T10:22:19.106238+0000","last_active":"2024-09-21T10:22:19.107586+0000","last_peered":"2024-09-21T10:22:19.107586+0000","last_clean":"2024-09-21T10:22:19.107586+0000","last_became_active":"2024-09-21T10:22:19.105893+0000","last_became_peered":"2024-09-21T10:22:19.105893+0000","last_unstale":"2024-09-21T10:22:19.107586+0000","last_undegraded":"2024-09-21T10:22:19.107586+0000","last_fullsized":"2024-09-21T10:22:19.107586+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:34:45.976176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":98,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.765228+0000","last_change":"2024-09-21T10:18:33.483457+0000","last_active":"2024-09-21T10:22:09.765228+0000","last_peered":"2024-09-21T10:22:09.765228+0000","last_clean":"2024-09-21T10:22:09.765228+0000","last_became_active":"2024-09-21T10:18:33.483140+0000","last_became_peered":"2024-09-21T10:18:33.483140+0000","last_unstale":"2024-09-21T10:22:09.765228+0000","last_undegraded":"2024-09-21T10:22:09.765228+0000","last_fullsized":"2024-09-21T10:22:09.765228+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:37:39.302566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":497,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T10:22:17.647482+0000","last_change":"2024-09-21T10:18:33.483011+0000","last_active":"2024-09-21T10:22:17.647482+0000","last_peered":"2024-09-21T10:22:17.647482+0000","last_clean":"2024-09-21T10:22:17.647482+0000","last_became_active":"2024-09-21T10:18:33.482712+0000","last_became_peered":"2024-09-21T10:18:33.482712+0000","last_unstale":"2024-09-21T10:22:17.647482+0000","last_undegraded":"2024-09-21T10:22:17.647482+0000","last_fullsized":"2024-09-21T10:22:17.647482+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:05:11.037442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-21T10:22:17.626792+0000","last_change":"2024-09-21T10:22:17.645631+0000","last_active":"2024-09-21T10:22:17.626792+0000","last_peered":"2024-09-21T10:22:17.626792+0000","last_clean":"2024-09-21T10:22:17.626792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:22:17.626792+0000","last_undegraded":"2024-09-21T10:22:17.626792+0000","last_fullsized":"2024-09-21T10:22:17.626792+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:25:40.874704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":254,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:10.192493+0000","last_change":"2024-09-21T10:18:33.488794+0000","last_active":"2024-09-21T10:22:10.192493+0000","last_peered":"2024-09-21T10:22:10.192493+0000","last_clean":"2024-09-21T10:22:10.192493+0000","last_became_active":"2024-09-21T10:18:33.479048+0000","last_became_peered":"2024-09-21T10:18:33.479048+0000","last_unstale":"2024-09-21T10:22:10.192493+0000","last_undegraded":"2024-09-21T10:22:10.192493+0000","last_fullsized":"2024-09-21T10:22:10.192493+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:03.387788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":117,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.764558+0000","last_change":"2024-09-21T10:18:33.475977+0000","last_active":"2024-09-21T10:22:09.764558+0000","last_peered":"2024-09-21T10:22:09.764558+0000","last_clean":"2024-09-21T10:22:09.764558+0000","last_became_active":"2024-09-21T10:18:33.475590+0000","last_became_peered":"2024-09-21T10:18:33.475590+0000","last_unstale":"2024-09-21T10:22:09.764558+0000","last_undegraded":"2024-09-21T10:22:09.764558+0000","last_fullsized":"2024-09-21T10:22:09.764558+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:54:57.304453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T10:22:17.626792+0000","last_change":"2024-09-21T10:22:17.626792+0000","last_active":"2024-09-21T10:22:17.626792+0000","last_peered":"2024-09-21T10:22:17.626792+0000","last_clean":"2024-09-21T10:22:17.626792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:22:17.626792+0000","last_undegraded":"2024-09-21T10:22:17.626792+0000","last_fullsized":"2024-09-21T10:22:17.626792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":304,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.765144+0000","last_change":"2024-09-21T10:18:33.473849+0000","last_active":"2024-09-21T10:22:09.765144+0000","last_peered":"2024-09-21T10:22:09.765144+0000","last_clean":"2024-09-21T10:22:09.765144+0000","last_became_active":"2024-09-21T10:18:33.473037+0000","last_became_peered":"2024-09-21T10:18:33.473037+0000","last_unstale":"2024-09-21T10:22:09.765144+0000","last_undegraded":"2024-09-21T10:22:09.765144+0000","last_fullsized":"2024-09-21T10:22:09.765144+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:03:16.322667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":404,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T10:22:18.646675+0000","last_change":"2024-09-21T10:08:00.387420+0000","last_active":"2024-09-21T10:22:18.646675+0000","last_peered":"2024-09-21T10:22:18.646675+0000","last_clean":"2024-09-21T10:22:18.646675+0000","last_became_active":"2024-09-21T10:08:00.386843+0000","last_became_peered":"2024-09-21T10:08:00.386843+0000","last_unstale":"2024-09-21T10:22:18.646675+0000","last_undegraded":"2024-09-21T10:22:18.646675+0000","last_fullsized":"2024-09-21T10:22:18.646675+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:51:00.654836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-21T10:22:17.626792+0000","last_change":"2024-09-21T10:22:17.643906+0000","last_active":"2024-09-21T10:22:17.626792+0000","last_peered":"2024-09-21T10:22:17.626792+0000","last_clean":"2024-09-21T10:22:17.626792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:22:17.626792+0000","last_undegraded":"2024-09-21T10:22:17.626792+0000","last_fullsized":"2024-09-21T10:22:17.626792+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:54:18.444376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":389,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T10:22:18.646959+0000","last_change":"2024-09-21T10:08:00.382496+0000","last_active":"2024-09-21T10:22:18.646959+0000","last_peered":"2024-09-21T10:22:18.646959+0000","last_clean":"2024-09-21T10:22:18.646959+0000","last_became_active":"2024-09-21T10:08:00.382180+0000","last_became_peered":"2024-09-21T10:08:00.382180+0000","last_unstale":"2024-09-21T10:22:18.646959+0000","last_undegraded":"2024-09-21T10:22:18.646959+0000","last_fullsized":"2024-09-21T10:22:18.646959+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:16.074736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":233,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.765305+0000","last_change":"2024-09-21T10:08:38.906801+0000","last_active":"2024-09-21T10:22:09.765305+0000","last_peered":"2024-09-21T10:22:09.765305+0000","last_clean":"2024-09-21T10:22:09.765305+0000","last_became_active":"2024-09-21T10:08:38.906487+0000","last_became_peered":"2024-09-21T10:08:38.906487+0000","last_unstale":"2024-09-21T10:22:09.765305+0000","last_undegraded":"2024-09-21T10:22:09.765305+0000","last_fullsized":"2024-09-21T10:22:09.765305+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:31.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T10:22:17.626792+0000","last_change":"2024-09-21T10:22:17.626792+0000","last_active":"2024-09-21T10:22:17.626792+0000","last_peered":"2024-09-21T10:22:17.626792+0000","last_clean":"2024-09-21T10:22:17.626792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:22:17.626792+0000","last_undegraded":"2024-09-21T10:22:17.626792+0000","last_fullsized":"2024-09-21T10:22:17.626792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":121,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:10.192325+0000","last_change":"2024-09-21T10:18:33.484568+0000","last_active":"2024-09-21T10:22:10.192325+0000","last_peered":"2024-09-21T10:22:10.192325+0000","last_clean":"2024-09-21T10:22:10.192325+0000","last_became_active":"2024-09-21T10:18:33.483975+0000","last_became_peered":"2024-09-21T10:18:33.483975+0000","last_unstale":"2024-09-21T10:22:10.192325+0000","last_undegraded":"2024-09-21T10:22:10.192325+0000","last_fullsized":"2024-09-21T10:22:10.192325+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:13.276318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":484,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:10.192641+0000","last_change":"2024-09-21T10:18:33.474464+0000","last_active":"2024-09-21T10:22:10.192641+0000","last_peered":"2024-09-21T10:22:10.192641+0000","last_clean":"2024-09-21T10:22:10.192641+0000","last_became_active":"2024-09-21T10:18:33.474128+0000","last_became_peered":"2024-09-21T10:18:33.474128+0000","last_unstale":"2024-09-21T10:22:10.192641+0000","last_undegraded":"2024-09-21T10:22:10.192641+0000","last_fullsized":"2024-09-21T10:22:10.192641+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:32:49.409409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":300,"state":"creating+peering","last_fresh":"2024-09-21T10:22:17.645355+0000","last_change":"2024-09-21T10:22:17.645355+0000","last_active":"2024-09-21T10:22:17.626792+0000","last_peered":"2024-09-21T10:22:17.626792+0000","last_clean":"2024-09-21T10:22:17.626792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:22:17.645355+0000","last_undegraded":"2024-09-21T10:22:17.645355+0000","last_fullsized":"2024-09-21T10:22:17.645355+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":381,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.331684+0000","last_change":"2024-09-21T09:59:54.265556+0000","last_active":"2024-09-21T10:22:09.331684+0000","last_peered":"2024-09-21T10:22:09.331684+0000","last_clean":"2024-09-21T10:22:09.331684+0000","last_became_active":"2024-09-21T09:59:54.265033+0000","last_became_peered":"2024-09-21T09:59:54.265033+0000","last_unstale":"2024-09-21T10:22:09.331684+0000","last_undegraded":"2024-09-21T10:22:09.331684+0000","last_fullsized":"2024-09-21T10:22:09.331684+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T09:48:34.850872+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:40.449756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":769,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T10:22:18.646082+0000","last_change":"2024-09-21T10:07:43.790490+0000","last_active":"2024-09-21T10:22:18.646082+0000","last_peered":"2024-09-21T10:22:18.646082+0000","last_clean":"2024-09-21T10:22:18.646082+0000","last_became_active":"2024-09-21T10:07:43.789986+0000","last_became_peered":"2024-09-21T10:07:43.789986+0000","last_unstale":"2024-09-21T10:22:18.646082+0000","last_undegraded":"2024-09-21T10:22:18.646082+0000","last_fullsized":"2024-09-21T10:22:18.646082+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:08.740356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":481,"num_read_kb":1580,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":51,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.764932+0000","last_change":"2024-09-21T10:18:33.473676+0000","last_active":"2024-09-21T10:22:09.764932+0000","last_peered":"2024-09-21T10:22:09.764932+0000","last_clean":"2024-09-21T10:22:09.764932+0000","last_became_active":"2024-09-21T10:18:33.472593+0000","last_became_peered":"2024-09-21T10:18:33.472593+0000","last_unstale":"2024-09-21T10:22:09.764932+0000","last_undegraded":"2024-09-21T10:22:09.764932+0000","last_fullsized":"2024-09-21T10:22:09.764932+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:07:55.816740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":276,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.765471+0000","last_change":"2024-09-21T10:08:00.386135+0000","last_active":"2024-09-21T10:22:09.765471+0000","last_peered":"2024-09-21T10:22:09.765471+0000","last_clean":"2024-09-21T10:22:09.765471+0000","last_became_active":"2024-09-21T10:08:00.385796+0000","last_became_peered":"2024-09-21T10:08:00.385796+0000","last_unstale":"2024-09-21T10:22:09.765471+0000","last_undegraded":"2024-09-21T10:22:09.765471+0000","last_fullsized":"2024-09-21T10:22:09.765471+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:27.054699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":300,"state":"creating+peering","last_fresh":"2024-09-21T10:22:17.645378+0000","last_change":"2024-09-21T10:22:17.645378+0000","last_active":"2024-09-21T10:22:17.626792+0000","last_peered":"2024-09-21T10:22:17.626792+0000","last_clean":"2024-09-21T10:22:17.626792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:22:17.645378+0000","last_undegraded":"2024-09-21T10:22:17.645378+0000","last_fullsized":"2024-09-21T10:22:17.645378+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":644,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.331897+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:22:09.331897+0000","last_peered":"2024-09-21T10:22:09.331897+0000","last_clean":"2024-09-21T10:22:09.331897+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:22:09.331897+0000","last_undegraded":"2024-09-21T10:22:09.331897+0000","last_fullsized":"2024-09-21T10:22:09.331897+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":498,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T10:22:17.646788+0000","last_change":"2024-09-21T10:18:33.471355+0000","last_active":"2024-09-21T10:22:17.646788+0000","last_peered":"2024-09-21T10:22:17.646788+0000","last_clean":"2024-09-21T10:22:17.646788+0000","last_became_active":"2024-09-21T10:18:33.470962+0000","last_became_peered":"2024-09-21T10:18:33.470962+0000","last_unstale":"2024-09-21T10:22:17.646788+0000","last_undegraded":"2024-09-21T10:22:17.646788+0000","last_fullsized":"2024-09-21T10:22:17.646788+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:53:57.516543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":481,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T10:22:17.648685+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:22:17.648685+0000","last_peered":"2024-09-21T10:22:17.648685+0000","last_clean":"2024-09-21T10:22:17.648685+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:22:17.648685+0000","last_undegraded":"2024-09-21T10:22:17.648685+0000","last_fullsized":"2024-09-21T10:22:17.648685+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T10:22:17.626792+0000","last_change":"2024-09-21T10:22:17.626792+0000","last_active":"2024-09-21T10:22:17.626792+0000","last_peered":"2024-09-21T10:22:17.626792+0000","last_clean":"2024-09-21T10:22:17.626792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:22:17.626792+0000","last_undegraded":"2024-09-21T10:22:17.626792+0000","last_fullsized":"2024-09-21T10:22:17.626792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":1256,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T10:22:18.647135+0000","last_change":"2024-09-21T10:18:36.495308+0000","last_active":"2024-09-21T10:22:18.647135+0000","last_peered":"2024-09-21T10:22:18.647135+0000","last_clean":"2024-09-21T10:22:18.647135+0000","last_became_active":"2024-09-21T10:18:36.494662+0000","last_became_peered":"2024-09-21T10:18:36.494662+0000","last_unstale":"2024-09-21T10:22:18.647135+0000","last_undegraded":"2024-09-21T10:22:18.647135+0000","last_fullsized":"2024-09-21T10:22:18.647135+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:49.218676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":350,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:10.192889+0000","last_change":"2024-09-21T10:08:18.470748+0000","last_active":"2024-09-21T10:22:10.192889+0000","last_peered":"2024-09-21T10:22:10.192889+0000","last_clean":"2024-09-21T10:22:10.192889+0000","last_became_active":"2024-09-21T10:08:18.470227+0000","last_became_peered":"2024-09-21T10:08:18.470227+0000","last_unstale":"2024-09-21T10:22:10.192889+0000","last_undegraded":"2024-09-21T10:22:10.192889+0000","last_fullsized":"2024-09-21T10:22:10.192889+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:16.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":477,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T10:22:18.648201+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:22:18.648201+0000","last_peered":"2024-09-21T10:22:18.648201+0000","last_clean":"2024-09-21T10:22:18.648201+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:22:18.648201+0000","last_undegraded":"2024-09-21T10:22:18.648201+0000","last_fullsized":"2024-09-21T10:22:18.648201+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":300,"state":"creating+peering","last_fresh":"2024-09-21T10:22:17.645364+0000","last_change":"2024-09-21T10:22:17.645364+0000","last_active":"2024-09-21T10:22:17.626792+0000","last_peered":"2024-09-21T10:22:17.626792+0000","last_clean":"2024-09-21T10:22:17.626792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:22:17.645364+0000","last_undegraded":"2024-09-21T10:22:17.645364+0000","last_fullsized":"2024-09-21T10:22:17.645364+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":949,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:10.192862+0000","last_change":"2024-09-21T10:07:43.794767+0000","last_active":"2024-09-21T10:22:10.192862+0000","last_peered":"2024-09-21T10:22:10.192862+0000","last_clean":"2024-09-21T10:22:10.192862+0000","last_became_active":"2024-09-21T10:07:43.794062+0000","last_became_peered":"2024-09-21T10:07:43.794062+0000","last_unstale":"2024-09-21T10:22:10.192862+0000","last_undegraded":"2024-09-21T10:22:10.192862+0000","last_fullsized":"2024-09-21T10:22:10.192862+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:51.977955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":378,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T10:22:17.647044+0000","last_change":"2024-09-21T10:08:00.389176+0000","last_active":"2024-09-21T10:22:17.647044+0000","last_peered":"2024-09-21T10:22:17.647044+0000","last_clean":"2024-09-21T10:22:17.647044+0000","last_became_active":"2024-09-21T10:08:00.383854+0000","last_became_peered":"2024-09-21T10:08:00.383854+0000","last_unstale":"2024-09-21T10:22:17.647044+0000","last_undegraded":"2024-09-21T10:22:17.647044+0000","last_fullsized":"2024-09-21T10:22:17.647044+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:46.952216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":400,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T10:22:18.646451+0000","last_change":"2024-09-21T10:18:33.465815+0000","last_active":"2024-09-21T10:22:18.646451+0000","last_peered":"2024-09-21T10:22:18.646451+0000","last_clean":"2024-09-21T10:22:18.646451+0000","last_became_active":"2024-09-21T10:18:33.465459+0000","last_became_peered":"2024-09-21T10:18:33.465459+0000","last_unstale":"2024-09-21T10:22:18.646451+0000","last_undegraded":"2024-09-21T10:22:18.646451+0000","last_fullsized":"2024-09-21T10:22:18.646451+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:22:08.199800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T10:22:17.626792+0000","last_change":"2024-09-21T10:22:17.626792+0000","last_active":"2024-09-21T10:22:17.626792+0000","last_peered":"2024-09-21T10:22:17.626792+0000","last_clean":"2024-09-21T10:22:17.626792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:22:17.626792+0000","last_undegraded":"2024-09-21T10:22:17.626792+0000","last_fullsized":"2024-09-21T10:22:17.626792+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":143,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T10:22:18.646578+0000","last_change":"2024-09-21T10:18:33.467659+0000","last_active":"2024-09-21T10:22:18.646578+0000","last_peered":"2024-09-21T10:22:18.646578+0000","last_clean":"2024-09-21T10:22:18.646578+0000","last_became_active":"2024-09-21T10:18:33.467369+0000","last_became_peered":"2024-09-21T10:18:33.467369+0000","last_unstale":"2024-09-21T10:22:18.646578+0000","last_undegraded":"2024-09-21T10:22:18.646578+0000","last_fullsized":"2024-09-21T10:22:18.646578+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:16:41.951755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":1056,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:10.193323+0000","last_change":"2024-09-21T10:08:18.468995+0000","last_active":"2024-09-21T10:22:10.193323+0000","last_peered":"2024-09-21T10:22:10.193323+0000","last_clean":"2024-09-21T10:22:10.193323+0000","last_became_active":"2024-09-21T10:08:18.468243+0000","last_became_peered":"2024-09-21T10:08:18.468243+0000","last_unstale":"2024-09-21T10:22:10.193323+0000","last_undegraded":"2024-09-21T10:22:10.193323+0000","last_fullsized":"2024-09-21T10:22:10.193323+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:45.520968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":375,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:10.193008+0000","last_change":"2024-09-21T10:08:18.468595+0000","last_active":"2024-09-21T10:22:10.193008+0000","last_peered":"2024-09-21T10:22:10.193008+0000","last_clean":"2024-09-21T10:22:10.193008+0000","last_became_active":"2024-09-21T10:08:18.468296+0000","last_became_peered":"2024-09-21T10:08:18.468296+0000","last_unstale":"2024-09-21T10:22:10.193008+0000","last_undegraded":"2024-09-21T10:22:10.193008+0000","last_fullsized":"2024-09-21T10:22:10.193008+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:42.748130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-21T10:22:17.626792+0000","last_change":"2024-09-21T10:22:17.643927+0000","last_active":"2024-09-21T10:22:17.626792+0000","last_peered":"2024-09-21T10:22:17.626792+0000","last_clean":"2024-09-21T10:22:17.626792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:22:17.626792+0000","last_undegraded":"2024-09-21T10:22:17.626792+0000","last_fullsized":"2024-09-21T10:22:17.626792+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:54:10.036099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":351,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:10.193241+0000","last_change":"2024-09-21T10:08:18.469907+0000","last_active":"2024-09-21T10:22:10.193241+0000","last_peered":"2024-09-21T10:22:10.193241+0000","last_clean":"2024-09-21T10:22:10.193241+0000","last_became_active":"2024-09-21T10:08:18.469443+0000","last_became_peered":"2024-09-21T10:08:18.469443+0000","last_unstale":"2024-09-21T10:22:10.193241+0000","last_undegraded":"2024-09-21T10:22:10.193241+0000","last_fullsized":"2024-09-21T10:22:10.193241+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:15.645803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":694,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.332131+0000","last_change":"2024-09-21T10:08:00.419641+0000","last_active":"2024-09-21T10:22:09.332131+0000","last_peered":"2024-09-21T10:22:09.332131+0000","last_clean":"2024-09-21T10:22:09.332131+0000","last_became_active":"2024-09-21T10:08:00.387050+0000","last_became_peered":"2024-09-21T10:08:00.387050+0000","last_unstale":"2024-09-21T10:22:09.332131+0000","last_undegraded":"2024-09-21T10:22:09.332131+0000","last_fullsized":"2024-09-21T10:22:09.332131+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:00:32.881619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":459,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T10:22:17.647372+0000","last_change":"2024-09-21T09:59:54.267798+0000","last_active":"2024-09-21T10:22:17.647372+0000","last_peered":"2024-09-21T10:22:17.647372+0000","last_clean":"2024-09-21T10:22:17.647372+0000","last_became_active":"2024-09-21T09:59:54.265615+0000","last_became_peered":"2024-09-21T09:59:54.265615+0000","last_unstale":"2024-09-21T10:22:17.647372+0000","last_undegraded":"2024-09-21T10:22:17.647372+0000","last_fullsized":"2024-09-21T10:22:17.647372+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:30.363872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-21T10:22:17.626792+0000","last_change":"2024-09-21T10:22:17.643827+0000","last_active":"2024-09-21T10:22:17.626792+0000","last_peered":"2024-09-21T10:22:17.626792+0000","last_clean":"2024-09-21T10:22:17.626792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:22:17.626792+0000","last_undegraded":"2024-09-21T10:22:17.626792+0000","last_fullsized":"2024-09-21T10:22:17.626792+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:21.613613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":63,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.332223+0000","last_change":"2024-09-21T10:18:33.489678+0000","last_active":"2024-09-21T10:22:09.332223+0000","last_peered":"2024-09-21T10:22:09.332223+0000","last_clean":"2024-09-21T10:22:09.332223+0000","last_became_active":"2024-09-21T10:18:33.474818+0000","last_became_peered":"2024-09-21T10:18:33.474818+0000","last_unstale":"2024-09-21T10:22:09.332223+0000","last_undegraded":"2024-09-21T10:22:09.332223+0000","last_fullsized":"2024-09-21T10:22:09.332223+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:49.828007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":748,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:10.193584+0000","last_change":"2024-09-21T10:18:36.503584+0000","last_active":"2024-09-21T10:22:10.193584+0000","last_peered":"2024-09-21T10:22:10.193584+0000","last_clean":"2024-09-21T10:22:10.193584+0000","last_became_active":"2024-09-21T10:18:36.502990+0000","last_became_peered":"2024-09-21T10:18:36.502990+0000","last_unstale":"2024-09-21T10:22:10.193584+0000","last_undegraded":"2024-09-21T10:22:10.193584+0000","last_fullsized":"2024-09-21T10:22:10.193584+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:11.648271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":92,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.331406+0000","last_change":"2024-09-21T10:18:33.479008+0000","last_active":"2024-09-21T10:22:09.331406+0000","last_peered":"2024-09-21T10:22:09.331406+0000","last_clean":"2024-09-21T10:22:09.331406+0000","last_became_active":"2024-09-21T10:18:33.478740+0000","last_became_peered":"2024-09-21T10:18:33.478740+0000","last_unstale":"2024-09-21T10:22:09.331406+0000","last_undegraded":"2024-09-21T10:22:09.331406+0000","last_fullsized":"2024-09-21T10:22:09.331406+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:16:52.594882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-21T10:22:17.626792+0000","last_change":"2024-09-21T10:22:17.643925+0000","last_active":"2024-09-21T10:22:17.626792+0000","last_peered":"2024-09-21T10:22:17.626792+0000","last_clean":"2024-09-21T10:22:17.626792+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T10:22:17.626792+0000","last_undegraded":"2024-09-21T10:22:17.626792+0000","last_fullsized":"2024-09-21T10:22:17.626792+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:34.394485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":507,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T10:22:17.648381+0000","last_change":"2024-09-21T10:08:00.390436+0000","last_active":"2024-09-21T10:22:17.648381+0000","last_peered":"2024-09-21T10:22:17.648381+0000","last_clean":"2024-09-21T10:22:17.648381+0000","last_became_active":"2024-09-21T10:08:00.381835+0000","last_became_peered":"2024-09-21T10:08:00.381835+0000","last_unstale":"2024-09-21T10:22:17.648381+0000","last_undegraded":"2024-09-21T10:22:17.648381+0000","last_fullsized":"2024-09-21T10:22:17.648381+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:01:35.697356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":526,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.765454+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:22:09.765454+0000","last_peered":"2024-09-21T10:22:09.765454+0000","last_clean":"2024-09-21T10:22:09.765454+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:22:09.765454+0000","last_undegraded":"2024-09-21T10:22:09.765454+0000","last_fullsized":"2024-09-21T10:22:09.765454+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":164,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.764890+0000","last_change":"2024-09-21T10:08:00.385774+0000","last_active":"2024-09-21T10:22:09.764890+0000","last_peered":"2024-09-21T10:22:09.764890+0000","last_clean":"2024-09-21T10:22:09.764890+0000","last_became_active":"2024-09-21T10:08:00.385445+0000","last_became_peered":"2024-09-21T10:08:00.385445+0000","last_unstale":"2024-09-21T10:22:09.764890+0000","last_undegraded":"2024-09-21T10:22:09.764890+0000","last_fullsized":"2024-09-21T10:22:09.764890+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:42.442190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":223,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.765047+0000","last_change":"2024-09-21T10:08:00.379661+0000","last_active":"2024-09-21T10:22:09.765047+0000","last_peered":"2024-09-21T10:22:09.765047+0000","last_clean":"2024-09-21T10:22:09.765047+0000","last_became_active":"2024-09-21T10:08:00.379301+0000","last_became_peered":"2024-09-21T10:08:00.379301+0000","last_unstale":"2024-09-21T10:22:09.765047+0000","last_undegraded":"2024-09-21T10:22:09.765047+0000","last_fullsized":"2024-09-21T10:22:09.765047+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:15.649523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":246,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.331807+0000","last_change":"2024-09-21T10:08:00.410731+0000","last_active":"2024-09-21T10:22:09.331807+0000","last_peered":"2024-09-21T10:22:09.331807+0000","last_clean":"2024-09-21T10:22:09.331807+0000","last_became_active":"2024-09-21T10:08:00.380324+0000","last_became_peered":"2024-09-21T10:08:00.380324+0000","last_unstale":"2024-09-21T10:22:09.331807+0000","last_undegraded":"2024-09-21T10:22:09.331807+0000","last_fullsized":"2024-09-21T10:22:09.331807+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:47:34.960221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":350,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:10.192864+0000","last_change":"2024-09-21T10:08:18.470378+0000","last_active":"2024-09-21T10:22:10.192864+0000","last_peered":"2024-09-21T10:22:10.192864+0000","last_clean":"2024-09-21T10:22:10.192864+0000","last_became_active":"2024-09-21T10:08:18.469482+0000","last_became_peered":"2024-09-21T10:08:18.469482+0000","last_unstale":"2024-09-21T10:22:10.192864+0000","last_undegraded":"2024-09-21T10:22:10.192864+0000","last_fullsized":"2024-09-21T10:22:10.192864+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:35.628724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":331,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.331988+0000","last_change":"2024-09-21T10:18:33.479236+0000","last_active":"2024-09-21T10:22:09.331988+0000","last_peered":"2024-09-21T10:22:09.331988+0000","last_clean":"2024-09-21T10:22:09.331988+0000","last_became_active":"2024-09-21T10:18:33.474256+0000","last_became_peered":"2024-09-21T10:18:33.474256+0000","last_unstale":"2024-09-21T10:22:09.331988+0000","last_undegraded":"2024-09-21T10:22:09.331988+0000","last_fullsized":"2024-09-21T10:22:09.331988+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:19:34.666035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":211,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.332143+0000","last_change":"2024-09-21T10:07:43.791830+0000","last_active":"2024-09-21T10:22:09.332143+0000","last_peered":"2024-09-21T10:22:09.332143+0000","last_clean":"2024-09-21T10:22:09.332143+0000","last_became_active":"2024-09-21T10:07:43.791314+0000","last_became_peered":"2024-09-21T10:07:43.791314+0000","last_unstale":"2024-09-21T10:22:09.332143+0000","last_undegraded":"2024-09-21T10:22:09.332143+0000","last_fullsized":"2024-09-21T10:22:09.332143+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:37.086629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":52,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.764548+0000","last_change":"2024-09-21T10:18:33.464989+0000","last_active":"2024-09-21T10:22:09.764548+0000","last_peered":"2024-09-21T10:22:09.764548+0000","last_clean":"2024-09-21T10:22:09.764548+0000","last_became_active":"2024-09-21T10:18:33.464617+0000","last_became_peered":"2024-09-21T10:18:33.464617+0000","last_unstale":"2024-09-21T10:22:09.764548+0000","last_undegraded":"2024-09-21T10:22:09.764548+0000","last_fullsized":"2024-09-21T10:22:09.764548+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:09.300797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":476,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T10:22:17.648685+0000","last_change":"2024-09-21T10:18:33.472415+0000","last_active":"2024-09-21T10:22:17.648685+0000","last_peered":"2024-09-21T10:22:17.648685+0000","last_clean":"2024-09-21T10:22:17.648685+0000","last_became_active":"2024-09-21T10:18:33.471746+0000","last_became_peered":"2024-09-21T10:18:33.471746+0000","last_unstale":"2024-09-21T10:22:17.648685+0000","last_undegraded":"2024-09-21T10:22:17.648685+0000","last_fullsized":"2024-09-21T10:22:17.648685+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:40:37.099458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":120,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:10.192600+0000","last_change":"2024-09-21T10:18:33.484721+0000","last_active":"2024-09-21T10:22:10.192600+0000","last_peered":"2024-09-21T10:22:10.192600+0000","last_clean":"2024-09-21T10:22:10.192600+0000","last_became_active":"2024-09-21T10:18:33.483950+0000","last_became_peered":"2024-09-21T10:18:33.483950+0000","last_unstale":"2024-09-21T10:22:10.192600+0000","last_undegraded":"2024-09-21T10:22:10.192600+0000","last_fullsized":"2024-09-21T10:22:10.192600+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:54:52.208378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":263,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.765410+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:22:09.765410+0000","last_peered":"2024-09-21T10:22:09.765410+0000","last_clean":"2024-09-21T10:22:09.765410+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:22:09.765410+0000","last_undegraded":"2024-09-21T10:22:09.765410+0000","last_fullsized":"2024-09-21T10:22:09.765410+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":141,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T10:22:17.646376+0000","last_change":"2024-09-21T10:18:33.479635+0000","last_active":"2024-09-21T10:22:17.646376+0000","last_peered":"2024-09-21T10:22:17.646376+0000","last_clean":"2024-09-21T10:22:17.646376+0000","last_became_active":"2024-09-21T10:18:33.478830+0000","last_became_peered":"2024-09-21T10:18:33.478830+0000","last_unstale":"2024-09-21T10:22:17.646376+0000","last_undegraded":"2024-09-21T10:22:17.646376+0000","last_fullsized":"2024-09-21T10:22:17.646376+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:16.950086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":492,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T10:22:17.648598+0000","last_change":"2024-09-21T10:18:33.480350+0000","last_active":"2024-09-21T10:22:17.648598+0000","last_peered":"2024-09-21T10:22:17.648598+0000","last_clean":"2024-09-21T10:22:17.648598+0000","last_became_active":"2024-09-21T10:18:33.479248+0000","last_became_peered":"2024-09-21T10:18:33.479248+0000","last_unstale":"2024-09-21T10:22:17.648598+0000","last_undegraded":"2024-09-21T10:22:17.648598+0000","last_fullsized":"2024-09-21T10:22:17.648598+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:25:28.780572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":483,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:10.193168+0000","last_change":"2024-09-21T10:18:33.480341+0000","last_active":"2024-09-21T10:22:10.193168+0000","last_peered":"2024-09-21T10:22:10.193168+0000","last_clean":"2024-09-21T10:22:10.193168+0000","last_became_active":"2024-09-21T10:18:33.479869+0000","last_became_peered":"2024-09-21T10:18:33.479869+0000","last_unstale":"2024-09-21T10:22:10.193168+0000","last_undegraded":"2024-09-21T10:22:10.193168+0000","last_fullsized":"2024-09-21T10:22:10.193168+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:32:42.206910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":244,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:10.193317+0000","last_change":"2024-09-21T10:18:33.473981+0000","last_active":"2024-09-21T10:22:10.193317+0000","last_peered":"2024-09-21T10:22:10.193317+0000","last_clean":"2024-09-21T10:22:10.193317+0000","last_became_active":"2024-09-21T10:18:33.473619+0000","last_became_peered":"2024-09-21T10:18:33.473619+0000","last_unstale":"2024-09-21T10:22:10.193317+0000","last_undegraded":"2024-09-21T10:22:10.193317+0000","last_fullsized":"2024-09-21T10:22:10.193317+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:44:24.630856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":296,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:10.193514+0000","last_change":"2024-09-21T10:08:18.471282+0000","last_active":"2024-09-21T10:22:10.193514+0000","last_peered":"2024-09-21T10:22:10.193514+0000","last_clean":"2024-09-21T10:22:10.193514+0000","last_became_active":"2024-09-21T10:08:18.470952+0000","last_became_peered":"2024-09-21T10:08:18.470952+0000","last_unstale":"2024-09-21T10:22:10.193514+0000","last_undegraded":"2024-09-21T10:22:10.193514+0000","last_fullsized":"2024-09-21T10:22:10.193514+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:25.597081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":133,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T10:22:18.646236+0000","last_change":"2024-09-21T10:18:33.480128+0000","last_active":"2024-09-21T10:22:18.646236+0000","last_peered":"2024-09-21T10:22:18.646236+0000","last_clean":"2024-09-21T10:22:18.646236+0000","last_became_active":"2024-09-21T10:18:33.479691+0000","last_became_peered":"2024-09-21T10:18:33.479691+0000","last_unstale":"2024-09-21T10:22:18.646236+0000","last_undegraded":"2024-09-21T10:22:18.646236+0000","last_fullsized":"2024-09-21T10:22:18.646236+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:43:49.311244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":482,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T10:22:17.647270+0000","last_change":"2024-09-21T10:08:00.386104+0000","last_active":"2024-09-21T10:22:17.647270+0000","last_peered":"2024-09-21T10:22:17.647270+0000","last_clean":"2024-09-21T10:22:17.647270+0000","last_became_active":"2024-09-21T10:08:00.381484+0000","last_became_peered":"2024-09-21T10:08:00.381484+0000","last_unstale":"2024-09-21T10:22:17.647270+0000","last_undegraded":"2024-09-21T10:22:17.647270+0000","last_fullsized":"2024-09-21T10:22:17.647270+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:39.190191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":158,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:10.192414+0000","last_change":"2024-09-21T10:18:33.481322+0000","last_active":"2024-09-21T10:22:10.192414+0000","last_peered":"2024-09-21T10:22:10.192414+0000","last_clean":"2024-09-21T10:22:10.192414+0000","last_became_active":"2024-09-21T10:18:33.480554+0000","last_became_peered":"2024-09-21T10:18:33.480554+0000","last_unstale":"2024-09-21T10:22:10.192414+0000","last_undegraded":"2024-09-21T10:22:10.192414+0000","last_fullsized":"2024-09-21T10:22:10.192414+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:15.498594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":307,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:10.193458+0000","last_change":"2024-09-21T10:08:18.466882+0000","last_active":"2024-09-21T10:22:10.193458+0000","last_peered":"2024-09-21T10:22:10.193458+0000","last_clean":"2024-09-21T10:22:10.193458+0000","last_became_active":"2024-09-21T10:08:18.466470+0000","last_became_peered":"2024-09-21T10:08:18.466470+0000","last_unstale":"2024-09-21T10:22:10.193458+0000","last_undegraded":"2024-09-21T10:22:10.193458+0000","last_fullsized":"2024-09-21T10:22:10.193458+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:51:31.020051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":160,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T10:22:17.647279+0000","last_change":"2024-09-21T10:18:33.472352+0000","last_active":"2024-09-21T10:22:17.647279+0000","last_peered":"2024-09-21T10:22:17.647279+0000","last_clean":"2024-09-21T10:22:17.647279+0000","last_became_active":"2024-09-21T10:18:33.471312+0000","last_became_peered":"2024-09-21T10:18:33.471312+0000","last_unstale":"2024-09-21T10:22:17.647279+0000","last_undegraded":"2024-09-21T10:22:17.647279+0000","last_fullsized":"2024-09-21T10:22:17.647279+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:32:44.897145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":175,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.765404+0000","last_change":"2024-09-21T10:08:00.394350+0000","last_active":"2024-09-21T10:22:09.765404+0000","last_peered":"2024-09-21T10:22:09.765404+0000","last_clean":"2024-09-21T10:22:09.765404+0000","last_became_active":"2024-09-21T10:08:00.387232+0000","last_became_peered":"2024-09-21T10:08:00.387232+0000","last_unstale":"2024-09-21T10:22:09.765404+0000","last_undegraded":"2024-09-21T10:22:09.765404+0000","last_fullsized":"2024-09-21T10:22:09.765404+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:56.160018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":146,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T10:22:18.647048+0000","last_change":"2024-09-21T10:18:33.474216+0000","last_active":"2024-09-21T10:22:18.647048+0000","last_peered":"2024-09-21T10:22:18.647048+0000","last_clean":"2024-09-21T10:22:18.647048+0000","last_became_active":"2024-09-21T10:18:33.473794+0000","last_became_peered":"2024-09-21T10:18:33.473794+0000","last_unstale":"2024-09-21T10:22:18.647048+0000","last_undegraded":"2024-09-21T10:22:18.647048+0000","last_fullsized":"2024-09-21T10:22:18.647048+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:38.956888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":48,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.765323+0000","last_change":"2024-09-21T10:18:33.483576+0000","last_active":"2024-09-21T10:22:09.765323+0000","last_peered":"2024-09-21T10:22:09.765323+0000","last_clean":"2024-09-21T10:22:09.765323+0000","last_became_active":"2024-09-21T10:18:33.483116+0000","last_became_peered":"2024-09-21T10:18:33.483116+0000","last_unstale":"2024-09-21T10:22:09.765323+0000","last_undegraded":"2024-09-21T10:22:09.765323+0000","last_fullsized":"2024-09-21T10:22:09.765323+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:48.964766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":248,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.764680+0000","last_change":"2024-09-21T10:18:33.471909+0000","last_active":"2024-09-21T10:22:09.764680+0000","last_peered":"2024-09-21T10:22:09.764680+0000","last_clean":"2024-09-21T10:22:09.764680+0000","last_became_active":"2024-09-21T10:18:33.471614+0000","last_became_peered":"2024-09-21T10:18:33.471614+0000","last_unstale":"2024-09-21T10:22:09.764680+0000","last_undegraded":"2024-09-21T10:22:09.764680+0000","last_fullsized":"2024-09-21T10:22:09.764680+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:30:50.582505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":474,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T10:22:17.648580+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:22:17.648580+0000","last_peered":"2024-09-21T10:22:17.648580+0000","last_clean":"2024-09-21T10:22:17.648580+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:22:17.648580+0000","last_undegraded":"2024-09-21T10:22:17.648580+0000","last_fullsized":"2024-09-21T10:22:17.648580+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":4},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":10016,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5982,"num_read_kb":13005,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397890,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":905720,"kb_used_data":8492,"kb_used_omap":1,"kb_used_meta":897214,"kb_avail":92839432,"statfs":{"total":95995035648,"available":95067578368,"internally_reserved":0,"allocated":8695808,"data_stored":7056864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918747480},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594118,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":905720,"kb_used_data":4396,"kb_used_omap":1,"kb_used_meta":901310,"kb_avail":92839432,"statfs":{"total":95995035648,"available":95067578368,"internally_reserved":0,"allocated":4501504,"data_stored":2862557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922941784},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790348,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":911968,"kb_used_data":7892,"kb_used_omap":1,"kb_used_meta":904062,"kb_avail":92833184,"statfs":{"total":95995035648,"available":95061180416,"internally_reserved":0,"allocated":8081408,"data_stored":6461880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":925759832},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986577,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":433791697139,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860666,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":689916,"kb_used_data":7916,"kb_used_omap":8,"kb_used_meta":681975,"kb_avail":93055236,"statfs":{"total":95995035648,"available":95288561664,"internally_reserved":0,"allocated":8105984,"data_stored":6475265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":698343276},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024191,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":594444,"kb_used_data":3776,"kb_used_omap":7,"kb_used_meta":590648,"kb_avail":93150708,"statfs":{"total":95995035648,"available":95386324992,"internally_reserved":0,"allocated":3866624,"data_stored":2257970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":604824114},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187727,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":686876,"kb_used_data":4368,"kb_used_omap":8,"kb_used_meta":682487,"kb_avail":93058276,"statfs":{"total":95995035648,"available":95291674624,"internally_reserved":0,"allocated":4472832,"data_stored":2849609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":698867697},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:22:21.686 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:21.712 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:21.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:21.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:21.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:21.835 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:21.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:21.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:21.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:21 smithi098 ceph-mon[97580]: from='client.45998 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:22:21.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:21 smithi098 ceph-mon[100914]: from='client.45998 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:22:21.937 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:21.958 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:21.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:21.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:22.060 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:22.085 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:22.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:22.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:22.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:21 smithi196 ceph-mon[84784]: from='client.45998 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:22:22.187 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:22.212 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:22.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:22.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:22.314 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:22.339 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:22.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:22.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:22.440 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:22.462 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:22.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:22.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:22.611 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:22.636 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:22.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:22.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:22.739 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:22.761 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:22.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:22.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:22.763 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:22.863 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:22.888 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:22.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:22.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:22.990 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:23.015 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:23.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:23.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:23.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:23.139 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:23.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:23.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:23.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:23.264 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:23.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:23.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:23.366 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:23.392 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:23.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:23.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:23.494 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:23.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:23.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:23.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:23 smithi098 ceph-mon[97580]: pgmap v995: 89 pgs: 10 creating+peering, 5 unknown, 74 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:23.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:23 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:23.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:23 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:23.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:23 smithi098 ceph-mon[100914]: pgmap v995: 89 pgs: 10 creating+peering, 5 unknown, 74 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:23.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:23 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:23.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:23 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:23.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:23.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:23 smithi196 ceph-mon[84784]: pgmap v995: 89 pgs: 10 creating+peering, 5 unknown, 74 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:23 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:23 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:23.640 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:23.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:23.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:23.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:23.763 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:23.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:23.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:23.865 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:23.891 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:23.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:23.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:23.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:24.014 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:24.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:24.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:24.116 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:24.141 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:24.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:24.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:24.243 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:24.267 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:24.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:24.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:24.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:24.394 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:24.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:24.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:24.496 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:24.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:24.521 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:24.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:24.622 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:24.628 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:22:24.645 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:24.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:24.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:24.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:24.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:24.773 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:24.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:24.777 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:22:24.777 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:22:24.794 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:22:24.794 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:22:24.811 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:22:24.811 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:22:24.827 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:22:24.827 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:22:24.844 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:22:24.845 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:22:24.860 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:22:24.860 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:22:24.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:24.875 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:22:24.876 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:22:24.891 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:22:24.891 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:22:24.900 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:24.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:24.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:25.001 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:25.022 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:25.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:25.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:25.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:25.145 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:25.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:25.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:25.239 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:22:25.247 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:25.273 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:25.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:25.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:25.375 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:25.397 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:25.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:25.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:25.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:25.520 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:25.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:25.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:25.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:25 smithi098 ceph-mon[97580]: pgmap v996: 89 pgs: 6 creating+peering, 83 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:25.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:25 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1760400328' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:25.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:25 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/473286277' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:22:25.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:25 smithi098 ceph-mon[100914]: pgmap v996: 89 pgs: 6 creating+peering, 83 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:25.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:25 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1760400328' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:25.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:25 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/473286277' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:22:25.621 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:25.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:25 smithi196 ceph-mon[84784]: pgmap v996: 89 pgs: 6 creating+peering, 83 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:25.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:25 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1760400328' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:25 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/473286277' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:22:25.646 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:25.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:25.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:25.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:25.770 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:25.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:25.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:25.871 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:25.893 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:25.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:25.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:25.995 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:26.016 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:26.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:26.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:26.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:26.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:26.139 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:26.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:26.240 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:26.261 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:26.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:26.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:26.363 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:26.390 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:26.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:26.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:26.492 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:26.512 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:26.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:26.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:26.517 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:26 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:22:26.518 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:26 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/473286277' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:22:26.518 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:26 smithi098 ceph-mon[97580]: osdmap e303: 8 total, 8 up, 6 in 2024-09-21T10:22:26.518 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:26 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:22:26.518 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:26 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/473286277' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:22:26.518 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:26 smithi098 ceph-mon[100914]: osdmap e303: 8 total, 8 up, 6 in 2024-09-21T10:22:26.614 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:26.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:26 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:22:26.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:26 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/473286277' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:22:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:26 smithi196 ceph-mon[84784]: osdmap e303: 8 total, 8 up, 6 in 2024-09-21T10:22:26.639 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:26.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:26.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:26.741 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:26.766 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:26.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:26.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:26.868 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:26.893 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:26.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:26.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:26.995 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:27.017 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:27.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:27.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:27.118 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:27.140 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:27.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:27.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:27.242 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:27.264 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:27.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:27.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:27.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:27.386 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:27.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:27.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:27.487 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:27.509 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:27.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:27.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:27.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:27 smithi098 ceph-mon[97580]: pgmap v998: 89 pgs: 89 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:27.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:27 smithi098 ceph-mon[100914]: pgmap v998: 89 pgs: 89 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:27.611 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:27.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:27.639 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:27.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:27 smithi196 ceph-mon[84784]: pgmap v998: 89 pgs: 89 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:27.740 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:27.766 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:27.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:27.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:27.856 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:27.868 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:27.892 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:27.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:27.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:27.906 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:22:27.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:28.019 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:28.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:28.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:28.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:28.142 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:28.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:28.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:28.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:28.264 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:28.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:28.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:28.366 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:28.375 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:22:28.387 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:28.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:28.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:28.489 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:28.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:28.515 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:28.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:28.616 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:28.638 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:28.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:28.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:28.739 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:28.764 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:28.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:28.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:28.866 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:28.891 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:28.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:28.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:28.992 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:29.007 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:22:29.007 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:22:29.017 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:29.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:29.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:29.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:29.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:29.142 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:29.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:29.244 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:29.265 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:29.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:29.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:29.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:29 smithi098 ceph-mon[97580]: pgmap v999: 89 pgs: 89 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:29.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:29 smithi098 ceph-mon[100914]: pgmap v999: 89 pgs: 89 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:29.367 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:29.388 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:29.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:29.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:29.490 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:29.516 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:29.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:29.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:29.618 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:29.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:29 smithi196 ceph-mon[84784]: pgmap v999: 89 pgs: 89 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:29.643 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:29.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:29.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:29.745 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:29.770 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:29.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:29.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:29.872 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:29.898 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:29.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:29.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:30.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:30.024 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:30.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:30.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:30.126 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:30.150 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:30.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:30.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:30.221 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:30.248 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:22:30.252 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:30.259 INFO:teuthology.orchestra.run.smithi098.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-21T10:22:30.282 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:30.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:30.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:30.384 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:30.410 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:30.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:30.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:30.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:30.532 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:30.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:30.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:30.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:30 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/977567510' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:22:30.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:30 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/977567510' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:22:30.635 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:30.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:30 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/977567510' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:22:30.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:22:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:22:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:22:30.660 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:30.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:30.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:30.762 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:30.787 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:30.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:30.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:30.889 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:30.913 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:30.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:30.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:31.015 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:31.040 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:31.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:31.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:31.098 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:22:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:22:30] "GET /metrics HTTP/1.1" 200 36468 "" "Prometheus/2.51.0" 2024-09-21T10:22:31.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:31.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:31.169 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:31.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:31.270 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:31.294 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:31.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:31.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:31.396 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:31.421 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:31.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:31.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:31.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:31.548 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:31.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:31.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:31.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:31 smithi098 ceph-mon[100914]: pgmap v1000: 89 pgs: 89 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:31.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:31 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/977567510' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:22:31.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:31 smithi098 ceph-mon[100914]: osdmap e304: 8 total, 8 up, 6 in 2024-09-21T10:22:31.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:31 smithi098 ceph-mon[97580]: pgmap v1000: 89 pgs: 89 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:31.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:31 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/977567510' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:22:31.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:31 smithi098 ceph-mon[97580]: osdmap e304: 8 total, 8 up, 6 in 2024-09-21T10:22:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:31 smithi196 ceph-mon[84784]: pgmap v1000: 89 pgs: 89 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:31 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/977567510' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:22:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:31 smithi196 ceph-mon[84784]: osdmap e304: 8 total, 8 up, 6 in 2024-09-21T10:22:31.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:31.676 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:31.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:31.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:31.778 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:31.803 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:31.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:31.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:31.905 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:31.930 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:31.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:31.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:32.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:32.056 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:32.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:32.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:32.158 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:32.181 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:32.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:32.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:32.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:32.306 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:32.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:32.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:32.386 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:32 smithi098 ceph-mon[97580]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:22:32.387 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:32 smithi098 ceph-mon[100914]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:22:32.408 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:32.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:32.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:32.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:32.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:32.559 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:32.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:32.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:32.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:32 smithi196 ceph-mon[84784]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:22:32.661 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:32.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:32.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:32.785 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:32.811 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:32.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:32.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:32.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:32.928 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:32.938 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:32.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:32.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:33.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:33.063 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:33.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:33.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:33.162 DEBUG:teuthology.orchestra.run.smithi098:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-21T10:22:33.165 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:22:33.165 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:22:33.165 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:33.186 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:33.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:33.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:33.211 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:hints = 1 2024-09-21T10:22:33.211 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:22:33.211 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Object prefix: benchmark_data_smithi098_254407 2024-09-21T10:22:33.212 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:22:33.212 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:22:33.289 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:33.323 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:33.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:33.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:33.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:33.466 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:33.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:33.568 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:33.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:33 smithi098 ceph-mon[97580]: pgmap v1002: 89 pgs: 89 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:33.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:33 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:33.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:33 smithi098 ceph-mon[100914]: pgmap v1002: 89 pgs: 89 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:33.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:33 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:33.601 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:33.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:33.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:33.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:33 smithi196 ceph-mon[84784]: pgmap v1002: 89 pgs: 89 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:33 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:33.703 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:33.734 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:33.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:33.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:33.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:33.870 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:33.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:33.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:33.972 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:34.009 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:34.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:34.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:34.111 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:34.153 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:34.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:34.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:34.211 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 1 16 4090 4074 254.611 254.625 0.0041854 0.00391392 2024-09-21T10:22:34.255 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:34.287 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:34.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:34.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:34.388 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:34.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:34.423 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:34.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:34.524 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:34.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:34.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:34.662 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:34.694 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:34.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:34.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:34.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:34.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:34.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:34.932 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:34.963 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:34.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:34.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:35.065 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:35.101 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:35.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:35.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:35.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:35.211 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 2 16 7934 7918 247.42 240.25 0.00502226 0.0040325 2024-09-21T10:22:35.238 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:35.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:35.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:35.340 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:35.373 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:35.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:35.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:35.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:35.507 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:35.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:35.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:35.593 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:35 smithi098 ceph-mon[97580]: pgmap v1003: 89 pgs: 89 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:35.593 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:35 smithi098 ceph-mon[100914]: pgmap v1003: 89 pgs: 89 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:35.609 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:35.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:35 smithi196 ceph-mon[84784]: pgmap v1003: 89 pgs: 89 active+clean; 4.6 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail 2024-09-21T10:22:35.644 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:35.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:35.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:35.746 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:35.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:35.781 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:35.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:35.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:35.917 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:35.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:35.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:36.019 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:36.047 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:36.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:36.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:36.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:36.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:36.185 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:36.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:36.211 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 3 16 11168 11152 232.316 202.125 0.0029478 0.00429594 2024-09-21T10:22:36.286 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:36.319 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:36.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:36.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:36.421 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:36.451 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:36.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:36.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:36.485 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:36 smithi098 ceph-mon[97580]: pgmap v1004: 89 pgs: 89 active+clean; 54 MiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 5.0 MiB/s wr, 79 op/s 2024-09-21T10:22:36.486 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:36 smithi098 ceph-mon[100914]: pgmap v1004: 89 pgs: 89 active+clean; 54 MiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 5.0 MiB/s wr, 79 op/s 2024-09-21T10:22:36.553 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:36.558 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:36.581 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:36.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:36.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:36.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:36 smithi196 ceph-mon[84784]: pgmap v1004: 89 pgs: 89 active+clean; 54 MiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 5.0 MiB/s wr, 79 op/s 2024-09-21T10:22:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:22:36.670 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:22:36.683 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:36.714 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:36.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:36.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:36.816 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:36.850 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:36.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:36.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:36.952 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:36.985 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:36.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:36.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:37.087 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:37.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:37.131 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:37.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:37.211 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 4 16 14096 14080 219.981 183 0.00430859 0.00453919 2024-09-21T10:22:37.232 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:37.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:37.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:37.346 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:22:37.375 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:37.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:37.422 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:37.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:37.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:37.559 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:37.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:37.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:37 smithi098 ceph-mon[97580]: from='client.55357 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:22:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:37 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3008612698' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:22:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:37 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:22:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:37.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:37.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:37 smithi098 ceph-mon[100914]: from='client.55357 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:22:37.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:37 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3008612698' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:22:37.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:37 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:22:37.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:37.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:37.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:37 smithi196 ceph-mon[84784]: from='client.55357 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:22:37.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:37 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3008612698' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:22:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:37 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:22:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:22:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:37.661 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:37.694 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:37.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:37.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:37.796 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:37.829 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:37.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:37.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:37.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:37.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:37.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:38.053 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:38.091 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:38.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:38.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:38.193 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:38.211 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 5 16 16496 16480 205.981 150 0.00453793 0.00484927 2024-09-21T10:22:38.224 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:38.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:38.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:38.326 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:38.356 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:38.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:38.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:38.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:38.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:38.493 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:38.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:38.594 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:38.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:38 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:22:38.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:38 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:22:38.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:38 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:22:38.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:38 smithi098 ceph-mon[97580]: osdmap e305: 8 total, 8 up, 6 in 2024-09-21T10:22:38.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:38 smithi098 ceph-mon[97580]: pgmap v1006: 89 pgs: 89 active+clean; 317 MiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 39 MiB/s wr, 625 op/s 2024-09-21T10:22:38.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [7, 2]} : dispatch 2024-09-21T10:22:38.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:38 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:22:38.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:38 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:22:38.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:38 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:22:38.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:38 smithi098 ceph-mon[100914]: osdmap e305: 8 total, 8 up, 6 in 2024-09-21T10:22:38.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:38 smithi098 ceph-mon[100914]: pgmap v1006: 89 pgs: 89 active+clean; 317 MiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 39 MiB/s wr, 625 op/s 2024-09-21T10:22:38.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [7, 2]} : dispatch 2024-09-21T10:22:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:38.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:38.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:38 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:22:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:38 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:22:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:38 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:22:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:38 smithi196 ceph-mon[84784]: osdmap e305: 8 total, 8 up, 6 in 2024-09-21T10:22:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:38 smithi196 ceph-mon[84784]: pgmap v1006: 89 pgs: 89 active+clean; 317 MiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 39 MiB/s wr, 625 op/s 2024-09-21T10:22:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [7, 2]} : dispatch 2024-09-21T10:22:38.733 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:38.764 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:38.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:38.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:38.866 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:38.896 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:38.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:38.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:38.998 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:39.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:39.034 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:39.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:39.134 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:39.166 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:39.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:39.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:39.211 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 6 16 19335 19319 201.221 177.438 0.00537282 0.004964 2024-09-21T10:22:39.268 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:39.301 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:39.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:39.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:39.403 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:39.429 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:39.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:39.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:39.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:39.566 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:39.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:39.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:39.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:39 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [7, 2]}]': finished 2024-09-21T10:22:39.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:39 smithi196 ceph-mon[84784]: osdmap e306: 8 total, 8 up, 6 in 2024-09-21T10:22:39.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:39.701 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:39.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:39.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:39.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:39.838 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:39.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:39.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:39.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:39 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [7, 2]}]': finished 2024-09-21T10:22:39.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:39 smithi098 ceph-mon[97580]: osdmap e306: 8 total, 8 up, 6 in 2024-09-21T10:22:39.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:39 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [7, 2]}]': finished 2024-09-21T10:22:39.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:39 smithi098 ceph-mon[100914]: osdmap e306: 8 total, 8 up, 6 in 2024-09-21T10:22:39.940 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:39.975 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:39.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:39.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:40.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:40.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:40.117 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:40.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:40.211 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 7 16 22189 22173 197.955 178.375 0.00603425 0.00504676 2024-09-21T10:22:40.218 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:40.252 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:40.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:40.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:40.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:40.387 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:40.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:40.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:40.488 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:40.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:40 smithi098 ceph-mon[97580]: osdmap e307: 8 total, 8 up, 6 in 2024-09-21T10:22:40.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:40 smithi098 ceph-mon[97580]: pgmap v1009: 89 pgs: 89 active+clean; 947 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 157 MiB/s wr, 2.51k op/s 2024-09-21T10:22:40.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:40 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2523664425' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:40.514 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:40 smithi098 ceph-mon[100914]: osdmap e307: 8 total, 8 up, 6 in 2024-09-21T10:22:40.514 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:40 smithi098 ceph-mon[100914]: pgmap v1009: 89 pgs: 89 active+clean; 947 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 157 MiB/s wr, 2.51k op/s 2024-09-21T10:22:40.514 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:40 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2523664425' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:40.515 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1004,"stamp":"2024-09-21T10:22:35.445334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":57022413,"num_objects":825,"num_object_clones":0,"num_object_copies":2475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":825,"num_whiteouts":0,"num_read":6288,"num_read_kb":13764,"num_write":1853,"num_write_kb":65259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":600384,"num_keys_recovered":30,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1929,"ondisk_log_size":1929,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":4789588,"kb_used_data":123476,"kb_used_omap":28,"kb_used_meta":4666019,"kb_avail":557681324,"statfs":{"total":575970213888,"available":571065675776,"internally_reserved":0,"allocated":126439424,"data_stored":116585258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29289,"internal_metadata":4778003863},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":52232192,"num_objects":797,"num_object_clones":0,"num_object_copies":2391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":797,"num_write_kb":51008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002486"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":131,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569791+0000","last_change":"2024-09-21T10:18:33.471373+0000","last_active":"2024-09-21T10:22:30.569791+0000","last_peered":"2024-09-21T10:22:30.569791+0000","last_clean":"2024-09-21T10:22:30.569791+0000","last_became_active":"2024-09-21T10:18:33.471052+0000","last_became_peered":"2024-09-21T10:18:33.471052+0000","last_unstale":"2024-09-21T10:22:30.569791+0000","last_undegraded":"2024-09-21T10:22:30.569791+0000","last_fullsized":"2024-09-21T10:22:30.569791+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:42:19.491129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":318,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.944203+0000","last_change":"2024-09-21T10:08:00.406360+0000","last_active":"2024-09-21T10:22:25.944203+0000","last_peered":"2024-09-21T10:22:25.944203+0000","last_clean":"2024-09-21T10:22:25.944203+0000","last_became_active":"2024-09-21T10:08:00.380042+0000","last_became_peered":"2024-09-21T10:08:00.380042+0000","last_unstale":"2024-09-21T10:22:25.944203+0000","last_undegraded":"2024-09-21T10:22:25.944203+0000","last_fullsized":"2024-09-21T10:22:25.944203+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:49:46.544051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":234,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.944473+0000","last_change":"2024-09-21T10:18:33.465357+0000","last_active":"2024-09-21T10:22:25.944473+0000","last_peered":"2024-09-21T10:22:25.944473+0000","last_clean":"2024-09-21T10:22:25.944473+0000","last_became_active":"2024-09-21T10:18:33.464952+0000","last_became_peered":"2024-09-21T10:18:33.464952+0000","last_unstale":"2024-09-21T10:22:25.944473+0000","last_undegraded":"2024-09-21T10:22:25.944473+0000","last_fullsized":"2024-09-21T10:22:25.944473+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:15:41.090866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":110,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.944045+0000","last_change":"2024-09-21T10:18:33.475291+0000","last_active":"2024-09-21T10:22:25.944045+0000","last_peered":"2024-09-21T10:22:25.944045+0000","last_clean":"2024-09-21T10:22:25.944045+0000","last_became_active":"2024-09-21T10:18:33.474971+0000","last_became_peered":"2024-09-21T10:18:33.474971+0000","last_unstale":"2024-09-21T10:22:25.944045+0000","last_undegraded":"2024-09-21T10:22:25.944045+0000","last_fullsized":"2024-09-21T10:22:25.944045+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:39:40.179668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":484,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569330+0000","last_change":"2024-09-21T10:18:33.475179+0000","last_active":"2024-09-21T10:22:30.569330+0000","last_peered":"2024-09-21T10:22:30.569330+0000","last_clean":"2024-09-21T10:22:30.569330+0000","last_became_active":"2024-09-21T10:18:33.474914+0000","last_became_peered":"2024-09-21T10:18:33.474914+0000","last_unstale":"2024-09-21T10:22:30.569330+0000","last_undegraded":"2024-09-21T10:22:30.569330+0000","last_fullsized":"2024-09-21T10:22:30.569330+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:15:32.186986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":184,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.945289+0000","last_change":"2024-09-21T10:08:00.385671+0000","last_active":"2024-09-21T10:22:25.945289+0000","last_peered":"2024-09-21T10:22:25.945289+0000","last_clean":"2024-09-21T10:22:25.945289+0000","last_became_active":"2024-09-21T10:08:00.385368+0000","last_became_peered":"2024-09-21T10:08:00.385368+0000","last_unstale":"2024-09-21T10:22:25.945289+0000","last_undegraded":"2024-09-21T10:22:25.945289+0000","last_fullsized":"2024-09-21T10:22:25.945289+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:21.799149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":131,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569319+0000","last_change":"2024-09-21T10:18:33.481890+0000","last_active":"2024-09-21T10:22:30.569319+0000","last_peered":"2024-09-21T10:22:30.569319+0000","last_clean":"2024-09-21T10:22:30.569319+0000","last_became_active":"2024-09-21T10:18:33.480639+0000","last_became_peered":"2024-09-21T10:18:33.480639+0000","last_unstale":"2024-09-21T10:22:30.569319+0000","last_undegraded":"2024-09-21T10:22:30.569319+0000","last_fullsized":"2024-09-21T10:22:30.569319+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:44.797540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":276,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.944902+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:22:25.944902+0000","last_peered":"2024-09-21T10:22:25.944902+0000","last_clean":"2024-09-21T10:22:25.944902+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:22:25.944902+0000","last_undegraded":"2024-09-21T10:22:25.944902+0000","last_fullsized":"2024-09-21T10:22:25.944902+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":19,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.568880+0000","last_change":"2024-09-21T10:22:19.108552+0000","last_active":"2024-09-21T10:22:30.568880+0000","last_peered":"2024-09-21T10:22:30.568880+0000","last_clean":"2024-09-21T10:22:30.568880+0000","last_became_active":"2024-09-21T10:22:19.107377+0000","last_became_peered":"2024-09-21T10:22:19.107377+0000","last_unstale":"2024-09-21T10:22:30.568880+0000","last_undegraded":"2024-09-21T10:22:30.568880+0000","last_fullsized":"2024-09-21T10:22:30.568880+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:39.289538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":311,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.944285+0000","last_change":"2024-09-21T10:08:00.392499+0000","last_active":"2024-09-21T10:22:25.944285+0000","last_peered":"2024-09-21T10:22:25.944285+0000","last_clean":"2024-09-21T10:22:25.944285+0000","last_became_active":"2024-09-21T10:08:00.379165+0000","last_became_peered":"2024-09-21T10:08:00.379165+0000","last_unstale":"2024-09-21T10:22:25.944285+0000","last_undegraded":"2024-09-21T10:22:25.944285+0000","last_fullsized":"2024-09-21T10:22:25.944285+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:18.933144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":167,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.944448+0000","last_change":"2024-09-21T10:07:43.794677+0000","last_active":"2024-09-21T10:22:25.944448+0000","last_peered":"2024-09-21T10:22:25.944448+0000","last_clean":"2024-09-21T10:22:25.944448+0000","last_became_active":"2024-09-21T10:07:43.794348+0000","last_became_peered":"2024-09-21T10:07:43.794348+0000","last_unstale":"2024-09-21T10:22:25.944448+0000","last_undegraded":"2024-09-21T10:22:25.944448+0000","last_fullsized":"2024-09-21T10:22:25.944448+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:11.159014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":19,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569233+0000","last_change":"2024-09-21T10:22:18.651045+0000","last_active":"2024-09-21T10:22:30.569233+0000","last_peered":"2024-09-21T10:22:30.569233+0000","last_clean":"2024-09-21T10:22:30.569233+0000","last_became_active":"2024-09-21T10:22:18.650806+0000","last_became_peered":"2024-09-21T10:22:18.650806+0000","last_unstale":"2024-09-21T10:22:30.569233+0000","last_undegraded":"2024-09-21T10:22:30.569233+0000","last_fullsized":"2024-09-21T10:22:30.569233+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:45:21.716041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":312,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.944857+0000","last_change":"2024-09-21T10:08:00.412449+0000","last_active":"2024-09-21T10:22:25.944857+0000","last_peered":"2024-09-21T10:22:25.944857+0000","last_clean":"2024-09-21T10:22:25.944857+0000","last_became_active":"2024-09-21T10:08:00.386530+0000","last_became_peered":"2024-09-21T10:08:00.386530+0000","last_unstale":"2024-09-21T10:22:25.944857+0000","last_undegraded":"2024-09-21T10:22:25.944857+0000","last_fullsized":"2024-09-21T10:22:25.944857+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:12:50.581176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":410,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569252+0000","last_change":"2024-09-21T10:18:33.479057+0000","last_active":"2024-09-21T10:22:30.569252+0000","last_peered":"2024-09-21T10:22:30.569252+0000","last_clean":"2024-09-21T10:22:30.569252+0000","last_became_active":"2024-09-21T10:18:33.478752+0000","last_became_peered":"2024-09-21T10:18:33.478752+0000","last_unstale":"2024-09-21T10:22:30.569252+0000","last_undegraded":"2024-09-21T10:22:30.569252+0000","last_fullsized":"2024-09-21T10:22:30.569252+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:30.068585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"304'83","reported_seq":95,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:33.504152+0000","last_change":"2024-09-21T10:22:19.107933+0000","last_active":"2024-09-21T10:22:33.504152+0000","last_peered":"2024-09-21T10:22:33.504152+0000","last_clean":"2024-09-21T10:22:33.504152+0000","last_became_active":"2024-09-21T10:22:19.107577+0000","last_became_peered":"2024-09-21T10:22:19.107577+0000","last_unstale":"2024-09-21T10:22:33.504152+0000","last_undegraded":"2024-09-21T10:22:33.504152+0000","last_fullsized":"2024-09-21T10:22:33.504152+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:39:34.494252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5439488,"num_objects":83,"num_object_clones":0,"num_object_copies":249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83,"num_write_kb":5312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":252,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.944636+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:22:25.944636+0000","last_peered":"2024-09-21T10:22:25.944636+0000","last_clean":"2024-09-21T10:22:25.944636+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:22:25.944636+0000","last_undegraded":"2024-09-21T10:22:25.944636+0000","last_fullsized":"2024-09-21T10:22:25.944636+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":267,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.945239+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:22:25.945239+0000","last_peered":"2024-09-21T10:22:25.945239+0000","last_clean":"2024-09-21T10:22:25.945239+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:22:25.945239+0000","last_undegraded":"2024-09-21T10:22:25.945239+0000","last_fullsized":"2024-09-21T10:22:25.945239+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"304'338","reported_seq":357,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:34.610265+0000","last_change":"2024-09-21T10:22:19.106238+0000","last_active":"2024-09-21T10:22:34.610265+0000","last_peered":"2024-09-21T10:22:34.610265+0000","last_clean":"2024-09-21T10:22:34.610265+0000","last_became_active":"2024-09-21T10:22:19.105893+0000","last_became_peered":"2024-09-21T10:22:19.105893+0000","last_unstale":"2024-09-21T10:22:34.610265+0000","last_undegraded":"2024-09-21T10:22:34.610265+0000","last_fullsized":"2024-09-21T10:22:34.610265+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:36:41.147519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22151168,"num_objects":338,"num_object_clones":0,"num_object_copies":1014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":338,"num_write_kb":21632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":102,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.945650+0000","last_change":"2024-09-21T10:18:33.483457+0000","last_active":"2024-09-21T10:22:25.945650+0000","last_peered":"2024-09-21T10:22:25.945650+0000","last_clean":"2024-09-21T10:22:25.945650+0000","last_became_active":"2024-09-21T10:18:33.483140+0000","last_became_peered":"2024-09-21T10:18:33.483140+0000","last_unstale":"2024-09-21T10:22:25.945650+0000","last_undegraded":"2024-09-21T10:22:25.945650+0000","last_fullsized":"2024-09-21T10:22:25.945650+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:37:39.302566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":506,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569216+0000","last_change":"2024-09-21T10:18:33.483011+0000","last_active":"2024-09-21T10:22:30.569216+0000","last_peered":"2024-09-21T10:22:30.569216+0000","last_clean":"2024-09-21T10:22:30.569216+0000","last_became_active":"2024-09-21T10:18:33.482712+0000","last_became_peered":"2024-09-21T10:18:33.482712+0000","last_unstale":"2024-09-21T10:22:30.569216+0000","last_undegraded":"2024-09-21T10:22:30.569216+0000","last_fullsized":"2024-09-21T10:22:30.569216+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:05:11.037442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"304'106","reported_seq":118,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:33.617082+0000","last_change":"2024-09-21T10:22:18.648413+0000","last_active":"2024-09-21T10:22:33.617082+0000","last_peered":"2024-09-21T10:22:33.617082+0000","last_clean":"2024-09-21T10:22:33.617082+0000","last_became_active":"2024-09-21T10:22:18.648152+0000","last_became_peered":"2024-09-21T10:22:18.648152+0000","last_unstale":"2024-09-21T10:22:33.617082+0000","last_undegraded":"2024-09-21T10:22:33.617082+0000","last_fullsized":"2024-09-21T10:22:33.617082+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:44.431340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6946816,"num_objects":106,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":6784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":264,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.568642+0000","last_change":"2024-09-21T10:18:33.488794+0000","last_active":"2024-09-21T10:22:30.568642+0000","last_peered":"2024-09-21T10:22:30.568642+0000","last_clean":"2024-09-21T10:22:30.568642+0000","last_became_active":"2024-09-21T10:18:33.479048+0000","last_became_peered":"2024-09-21T10:18:33.479048+0000","last_unstale":"2024-09-21T10:22:30.568642+0000","last_undegraded":"2024-09-21T10:22:30.568642+0000","last_fullsized":"2024-09-21T10:22:30.568642+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:03.387788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":121,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.945639+0000","last_change":"2024-09-21T10:18:33.475977+0000","last_active":"2024-09-21T10:22:25.945639+0000","last_peered":"2024-09-21T10:22:25.945639+0000","last_clean":"2024-09-21T10:22:25.945639+0000","last_became_active":"2024-09-21T10:18:33.475590+0000","last_became_peered":"2024-09-21T10:18:33.475590+0000","last_unstale":"2024-09-21T10:22:25.945639+0000","last_undegraded":"2024-09-21T10:22:25.945639+0000","last_fullsized":"2024-09-21T10:22:25.945639+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:54:57.304453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":19,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.568486+0000","last_change":"2024-09-21T10:22:18.649410+0000","last_active":"2024-09-21T10:22:30.568486+0000","last_peered":"2024-09-21T10:22:30.568486+0000","last_clean":"2024-09-21T10:22:30.568486+0000","last_became_active":"2024-09-21T10:22:18.648626+0000","last_became_peered":"2024-09-21T10:22:18.648626+0000","last_unstale":"2024-09-21T10:22:30.568486+0000","last_undegraded":"2024-09-21T10:22:30.568486+0000","last_fullsized":"2024-09-21T10:22:30.568486+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:42:50.134303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":308,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.945303+0000","last_change":"2024-09-21T10:18:33.473849+0000","last_active":"2024-09-21T10:22:25.945303+0000","last_peered":"2024-09-21T10:22:25.945303+0000","last_clean":"2024-09-21T10:22:25.945303+0000","last_became_active":"2024-09-21T10:18:33.473037+0000","last_became_peered":"2024-09-21T10:18:33.473037+0000","last_unstale":"2024-09-21T10:22:25.945303+0000","last_undegraded":"2024-09-21T10:22:25.945303+0000","last_fullsized":"2024-09-21T10:22:25.945303+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:03:16.322667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":410,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.570156+0000","last_change":"2024-09-21T10:08:00.387420+0000","last_active":"2024-09-21T10:22:30.570156+0000","last_peered":"2024-09-21T10:22:30.570156+0000","last_clean":"2024-09-21T10:22:30.570156+0000","last_became_active":"2024-09-21T10:08:00.386843+0000","last_became_peered":"2024-09-21T10:08:00.386843+0000","last_unstale":"2024-09-21T10:22:30.570156+0000","last_undegraded":"2024-09-21T10:22:30.570156+0000","last_fullsized":"2024-09-21T10:22:30.570156+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:51:00.654836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3","version":"304'61","reported_seq":73,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:33.502368+0000","last_change":"2024-09-21T10:22:19.106259+0000","last_active":"2024-09-21T10:22:33.502368+0000","last_peered":"2024-09-21T10:22:33.502368+0000","last_clean":"2024-09-21T10:22:33.502368+0000","last_became_active":"2024-09-21T10:22:19.105931+0000","last_became_peered":"2024-09-21T10:22:19.105931+0000","last_unstale":"2024-09-21T10:22:33.502368+0000","last_undegraded":"2024-09-21T10:22:33.502368+0000","last_fullsized":"2024-09-21T10:22:33.502368+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:38:07.993075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3997696,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":3904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":395,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569488+0000","last_change":"2024-09-21T10:08:00.382496+0000","last_active":"2024-09-21T10:22:30.569488+0000","last_peered":"2024-09-21T10:22:30.569488+0000","last_clean":"2024-09-21T10:22:30.569488+0000","last_became_active":"2024-09-21T10:08:00.382180+0000","last_became_peered":"2024-09-21T10:08:00.382180+0000","last_unstale":"2024-09-21T10:22:30.569488+0000","last_undegraded":"2024-09-21T10:22:30.569488+0000","last_fullsized":"2024-09-21T10:22:30.569488+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:16.074736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":237,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.945802+0000","last_change":"2024-09-21T10:08:38.906801+0000","last_active":"2024-09-21T10:22:25.945802+0000","last_peered":"2024-09-21T10:22:25.945802+0000","last_clean":"2024-09-21T10:22:25.945802+0000","last_became_active":"2024-09-21T10:08:38.906487+0000","last_became_peered":"2024-09-21T10:08:38.906487+0000","last_unstale":"2024-09-21T10:22:25.945802+0000","last_undegraded":"2024-09-21T10:22:25.945802+0000","last_fullsized":"2024-09-21T10:22:25.945802+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:31.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":19,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.568688+0000","last_change":"2024-09-21T10:22:18.649478+0000","last_active":"2024-09-21T10:22:30.568688+0000","last_peered":"2024-09-21T10:22:30.568688+0000","last_clean":"2024-09-21T10:22:30.568688+0000","last_became_active":"2024-09-21T10:22:18.649167+0000","last_became_peered":"2024-09-21T10:22:18.649167+0000","last_unstale":"2024-09-21T10:22:30.568688+0000","last_undegraded":"2024-09-21T10:22:30.568688+0000","last_fullsized":"2024-09-21T10:22:30.568688+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:17.191131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":131,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.568985+0000","last_change":"2024-09-21T10:18:33.484568+0000","last_active":"2024-09-21T10:22:30.568985+0000","last_peered":"2024-09-21T10:22:30.568985+0000","last_clean":"2024-09-21T10:22:30.568985+0000","last_became_active":"2024-09-21T10:18:33.483975+0000","last_became_peered":"2024-09-21T10:18:33.483975+0000","last_unstale":"2024-09-21T10:22:30.568985+0000","last_undegraded":"2024-09-21T10:22:30.568985+0000","last_fullsized":"2024-09-21T10:22:30.568985+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:13.276318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":494,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569405+0000","last_change":"2024-09-21T10:18:33.474464+0000","last_active":"2024-09-21T10:22:30.569405+0000","last_peered":"2024-09-21T10:22:30.569405+0000","last_clean":"2024-09-21T10:22:30.569405+0000","last_became_active":"2024-09-21T10:18:33.474128+0000","last_became_peered":"2024-09-21T10:18:33.474128+0000","last_unstale":"2024-09-21T10:22:30.569405+0000","last_undegraded":"2024-09-21T10:22:30.569405+0000","last_fullsized":"2024-09-21T10:22:30.569405+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:32:49.409409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":19,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569675+0000","last_change":"2024-09-21T10:22:19.107370+0000","last_active":"2024-09-21T10:22:30.569675+0000","last_peered":"2024-09-21T10:22:30.569675+0000","last_clean":"2024-09-21T10:22:30.569675+0000","last_became_active":"2024-09-21T10:22:19.106979+0000","last_became_peered":"2024-09-21T10:22:19.106979+0000","last_unstale":"2024-09-21T10:22:30.569675+0000","last_undegraded":"2024-09-21T10:22:30.569675+0000","last_fullsized":"2024-09-21T10:22:30.569675+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:10.360218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":385,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.944684+0000","last_change":"2024-09-21T09:59:54.265556+0000","last_active":"2024-09-21T10:22:25.944684+0000","last_peered":"2024-09-21T10:22:25.944684+0000","last_clean":"2024-09-21T10:22:25.944684+0000","last_became_active":"2024-09-21T09:59:54.265033+0000","last_became_peered":"2024-09-21T09:59:54.265033+0000","last_unstale":"2024-09-21T10:22:25.944684+0000","last_undegraded":"2024-09-21T10:22:25.944684+0000","last_fullsized":"2024-09-21T10:22:25.944684+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T09:48:34.850872+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:40.449756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":775,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569861+0000","last_change":"2024-09-21T10:07:43.790490+0000","last_active":"2024-09-21T10:22:30.569861+0000","last_peered":"2024-09-21T10:22:30.569861+0000","last_clean":"2024-09-21T10:22:30.569861+0000","last_became_active":"2024-09-21T10:07:43.789986+0000","last_became_peered":"2024-09-21T10:07:43.789986+0000","last_unstale":"2024-09-21T10:22:30.569861+0000","last_undegraded":"2024-09-21T10:22:30.569861+0000","last_fullsized":"2024-09-21T10:22:30.569861+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:08.740356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":481,"num_read_kb":1580,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":55,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.945140+0000","last_change":"2024-09-21T10:18:33.473676+0000","last_active":"2024-09-21T10:22:25.945140+0000","last_peered":"2024-09-21T10:22:25.945140+0000","last_clean":"2024-09-21T10:22:25.945140+0000","last_became_active":"2024-09-21T10:18:33.472593+0000","last_became_peered":"2024-09-21T10:18:33.472593+0000","last_unstale":"2024-09-21T10:22:25.945140+0000","last_undegraded":"2024-09-21T10:22:25.945140+0000","last_fullsized":"2024-09-21T10:22:25.945140+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:07:55.816740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":280,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.945907+0000","last_change":"2024-09-21T10:08:00.386135+0000","last_active":"2024-09-21T10:22:25.945907+0000","last_peered":"2024-09-21T10:22:25.945907+0000","last_clean":"2024-09-21T10:22:25.945907+0000","last_became_active":"2024-09-21T10:08:00.385796+0000","last_became_peered":"2024-09-21T10:08:00.385796+0000","last_unstale":"2024-09-21T10:22:25.945907+0000","last_undegraded":"2024-09-21T10:22:25.945907+0000","last_fullsized":"2024-09-21T10:22:25.945907+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:27.054699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":19,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569247+0000","last_change":"2024-09-21T10:22:19.108211+0000","last_active":"2024-09-21T10:22:30.569247+0000","last_peered":"2024-09-21T10:22:30.569247+0000","last_clean":"2024-09-21T10:22:30.569247+0000","last_became_active":"2024-09-21T10:22:19.107919+0000","last_became_peered":"2024-09-21T10:22:19.107919+0000","last_unstale":"2024-09-21T10:22:30.569247+0000","last_undegraded":"2024-09-21T10:22:30.569247+0000","last_fullsized":"2024-09-21T10:22:30.569247+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:20.144889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":648,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.944579+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:22:25.944579+0000","last_peered":"2024-09-21T10:22:25.944579+0000","last_clean":"2024-09-21T10:22:25.944579+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:22:25.944579+0000","last_undegraded":"2024-09-21T10:22:25.944579+0000","last_fullsized":"2024-09-21T10:22:25.944579+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":506,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.568705+0000","last_change":"2024-09-21T10:18:33.471355+0000","last_active":"2024-09-21T10:22:30.568705+0000","last_peered":"2024-09-21T10:22:30.568705+0000","last_clean":"2024-09-21T10:22:30.568705+0000","last_became_active":"2024-09-21T10:18:33.470962+0000","last_became_peered":"2024-09-21T10:18:33.470962+0000","last_unstale":"2024-09-21T10:22:30.568705+0000","last_undegraded":"2024-09-21T10:22:30.568705+0000","last_fullsized":"2024-09-21T10:22:30.568705+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:53:57.516543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":489,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569557+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:22:30.569557+0000","last_peered":"2024-09-21T10:22:30.569557+0000","last_clean":"2024-09-21T10:22:30.569557+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:22:30.569557+0000","last_undegraded":"2024-09-21T10:22:30.569557+0000","last_fullsized":"2024-09-21T10:22:30.569557+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":19,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569434+0000","last_change":"2024-09-21T10:22:19.108927+0000","last_active":"2024-09-21T10:22:30.569434+0000","last_peered":"2024-09-21T10:22:30.569434+0000","last_clean":"2024-09-21T10:22:30.569434+0000","last_became_active":"2024-09-21T10:22:19.108649+0000","last_became_peered":"2024-09-21T10:22:19.108649+0000","last_unstale":"2024-09-21T10:22:30.569434+0000","last_undegraded":"2024-09-21T10:22:30.569434+0000","last_fullsized":"2024-09-21T10:22:30.569434+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:02:10.256858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":1263,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569608+0000","last_change":"2024-09-21T10:18:36.495308+0000","last_active":"2024-09-21T10:22:30.569608+0000","last_peered":"2024-09-21T10:22:30.569608+0000","last_clean":"2024-09-21T10:22:30.569608+0000","last_became_active":"2024-09-21T10:18:36.494662+0000","last_became_peered":"2024-09-21T10:18:36.494662+0000","last_unstale":"2024-09-21T10:22:30.569608+0000","last_undegraded":"2024-09-21T10:22:30.569608+0000","last_fullsized":"2024-09-21T10:22:30.569608+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:49.218676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":361,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569079+0000","last_change":"2024-09-21T10:08:18.470748+0000","last_active":"2024-09-21T10:22:30.569079+0000","last_peered":"2024-09-21T10:22:30.569079+0000","last_clean":"2024-09-21T10:22:30.569079+0000","last_became_active":"2024-09-21T10:08:18.470227+0000","last_became_peered":"2024-09-21T10:08:18.470227+0000","last_unstale":"2024-09-21T10:22:30.569079+0000","last_undegraded":"2024-09-21T10:22:30.569079+0000","last_fullsized":"2024-09-21T10:22:30.569079+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:16.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":483,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.570423+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:22:30.570423+0000","last_peered":"2024-09-21T10:22:30.570423+0000","last_clean":"2024-09-21T10:22:30.570423+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:22:30.570423+0000","last_undegraded":"2024-09-21T10:22:30.570423+0000","last_fullsized":"2024-09-21T10:22:30.570423+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":19,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.568559+0000","last_change":"2024-09-21T10:22:19.106312+0000","last_active":"2024-09-21T10:22:30.568559+0000","last_peered":"2024-09-21T10:22:30.568559+0000","last_clean":"2024-09-21T10:22:30.568559+0000","last_became_active":"2024-09-21T10:22:19.106002+0000","last_became_peered":"2024-09-21T10:22:19.106002+0000","last_unstale":"2024-09-21T10:22:30.568559+0000","last_undegraded":"2024-09-21T10:22:30.568559+0000","last_fullsized":"2024-09-21T10:22:30.568559+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:24.317806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":960,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569943+0000","last_change":"2024-09-21T10:07:43.794767+0000","last_active":"2024-09-21T10:22:30.569943+0000","last_peered":"2024-09-21T10:22:30.569943+0000","last_clean":"2024-09-21T10:22:30.569943+0000","last_became_active":"2024-09-21T10:07:43.794062+0000","last_became_peered":"2024-09-21T10:07:43.794062+0000","last_unstale":"2024-09-21T10:22:30.569943+0000","last_undegraded":"2024-09-21T10:22:30.569943+0000","last_fullsized":"2024-09-21T10:22:30.569943+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:51.977955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":386,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.568713+0000","last_change":"2024-09-21T10:08:00.389176+0000","last_active":"2024-09-21T10:22:30.568713+0000","last_peered":"2024-09-21T10:22:30.568713+0000","last_clean":"2024-09-21T10:22:30.568713+0000","last_became_active":"2024-09-21T10:08:00.383854+0000","last_became_peered":"2024-09-21T10:08:00.383854+0000","last_unstale":"2024-09-21T10:22:30.568713+0000","last_undegraded":"2024-09-21T10:22:30.568713+0000","last_fullsized":"2024-09-21T10:22:30.568713+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:46.952216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":407,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.570113+0000","last_change":"2024-09-21T10:18:33.465815+0000","last_active":"2024-09-21T10:22:30.570113+0000","last_peered":"2024-09-21T10:22:30.570113+0000","last_clean":"2024-09-21T10:22:30.570113+0000","last_became_active":"2024-09-21T10:18:33.465459+0000","last_became_peered":"2024-09-21T10:18:33.465459+0000","last_unstale":"2024-09-21T10:22:30.570113+0000","last_undegraded":"2024-09-21T10:22:30.570113+0000","last_fullsized":"2024-09-21T10:22:30.570113+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:22:08.199800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":19,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569345+0000","last_change":"2024-09-21T10:22:19.107714+0000","last_active":"2024-09-21T10:22:30.569345+0000","last_peered":"2024-09-21T10:22:30.569345+0000","last_clean":"2024-09-21T10:22:30.569345+0000","last_became_active":"2024-09-21T10:22:19.107130+0000","last_became_peered":"2024-09-21T10:22:19.107130+0000","last_unstale":"2024-09-21T10:22:30.569345+0000","last_undegraded":"2024-09-21T10:22:30.569345+0000","last_fullsized":"2024-09-21T10:22:30.569345+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:50:53.512892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":149,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.570058+0000","last_change":"2024-09-21T10:18:33.467659+0000","last_active":"2024-09-21T10:22:30.570058+0000","last_peered":"2024-09-21T10:22:30.570058+0000","last_clean":"2024-09-21T10:22:30.570058+0000","last_became_active":"2024-09-21T10:18:33.467369+0000","last_became_peered":"2024-09-21T10:18:33.467369+0000","last_unstale":"2024-09-21T10:22:30.570058+0000","last_undegraded":"2024-09-21T10:22:30.570058+0000","last_fullsized":"2024-09-21T10:22:30.570058+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:16:41.951755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":1066,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569704+0000","last_change":"2024-09-21T10:08:18.468995+0000","last_active":"2024-09-21T10:22:30.569704+0000","last_peered":"2024-09-21T10:22:30.569704+0000","last_clean":"2024-09-21T10:22:30.569704+0000","last_became_active":"2024-09-21T10:08:18.468243+0000","last_became_peered":"2024-09-21T10:08:18.468243+0000","last_unstale":"2024-09-21T10:22:30.569704+0000","last_undegraded":"2024-09-21T10:22:30.569704+0000","last_fullsized":"2024-09-21T10:22:30.569704+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:45.520968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":385,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.570126+0000","last_change":"2024-09-21T10:08:18.468595+0000","last_active":"2024-09-21T10:22:30.570126+0000","last_peered":"2024-09-21T10:22:30.570126+0000","last_clean":"2024-09-21T10:22:30.570126+0000","last_became_active":"2024-09-21T10:08:18.468296+0000","last_became_peered":"2024-09-21T10:08:18.468296+0000","last_unstale":"2024-09-21T10:22:30.570126+0000","last_undegraded":"2024-09-21T10:22:30.570126+0000","last_fullsized":"2024-09-21T10:22:30.570126+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:42.748130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"304'67","reported_seq":79,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:33.487073+0000","last_change":"2024-09-21T10:22:18.649813+0000","last_active":"2024-09-21T10:22:33.487073+0000","last_peered":"2024-09-21T10:22:33.487073+0000","last_clean":"2024-09-21T10:22:33.487073+0000","last_became_active":"2024-09-21T10:22:18.649480+0000","last_became_peered":"2024-09-21T10:22:18.649480+0000","last_unstale":"2024-09-21T10:22:33.487073+0000","last_undegraded":"2024-09-21T10:22:33.487073+0000","last_fullsized":"2024-09-21T10:22:33.487073+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:36:28.426420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4390912,"num_objects":67,"num_object_clones":0,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67,"num_write_kb":4288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":361,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569507+0000","last_change":"2024-09-21T10:08:18.469907+0000","last_active":"2024-09-21T10:22:30.569507+0000","last_peered":"2024-09-21T10:22:30.569507+0000","last_clean":"2024-09-21T10:22:30.569507+0000","last_became_active":"2024-09-21T10:08:18.469443+0000","last_became_peered":"2024-09-21T10:08:18.469443+0000","last_unstale":"2024-09-21T10:22:30.569507+0000","last_undegraded":"2024-09-21T10:22:30.569507+0000","last_fullsized":"2024-09-21T10:22:30.569507+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:15.645803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":698,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.944731+0000","last_change":"2024-09-21T10:08:00.419641+0000","last_active":"2024-09-21T10:22:25.944731+0000","last_peered":"2024-09-21T10:22:25.944731+0000","last_clean":"2024-09-21T10:22:25.944731+0000","last_became_active":"2024-09-21T10:08:00.387050+0000","last_became_peered":"2024-09-21T10:08:00.387050+0000","last_unstale":"2024-09-21T10:22:25.944731+0000","last_undegraded":"2024-09-21T10:22:25.944731+0000","last_fullsized":"2024-09-21T10:22:25.944731+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:00:32.881619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":467,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.568934+0000","last_change":"2024-09-21T09:59:54.267798+0000","last_active":"2024-09-21T10:22:30.568934+0000","last_peered":"2024-09-21T10:22:30.568934+0000","last_clean":"2024-09-21T10:22:30.568934+0000","last_became_active":"2024-09-21T09:59:54.265615+0000","last_became_peered":"2024-09-21T09:59:54.265615+0000","last_unstale":"2024-09-21T10:22:30.568934+0000","last_undegraded":"2024-09-21T10:22:30.568934+0000","last_fullsized":"2024-09-21T10:22:30.568934+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:30.363872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"304'68","reported_seq":80,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:33.492762+0000","last_change":"2024-09-21T10:22:19.106548+0000","last_active":"2024-09-21T10:22:33.492762+0000","last_peered":"2024-09-21T10:22:33.492762+0000","last_clean":"2024-09-21T10:22:33.492762+0000","last_became_active":"2024-09-21T10:22:19.105920+0000","last_became_peered":"2024-09-21T10:22:19.105920+0000","last_unstale":"2024-09-21T10:22:33.492762+0000","last_undegraded":"2024-09-21T10:22:33.492762+0000","last_fullsized":"2024-09-21T10:22:33.492762+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:30.951700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4456448,"num_objects":68,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":4352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":67,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.944239+0000","last_change":"2024-09-21T10:18:33.489678+0000","last_active":"2024-09-21T10:22:25.944239+0000","last_peered":"2024-09-21T10:22:25.944239+0000","last_clean":"2024-09-21T10:22:25.944239+0000","last_became_active":"2024-09-21T10:18:33.474818+0000","last_became_peered":"2024-09-21T10:18:33.474818+0000","last_unstale":"2024-09-21T10:22:25.944239+0000","last_undegraded":"2024-09-21T10:22:25.944239+0000","last_fullsized":"2024-09-21T10:22:25.944239+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:49.828007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":758,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569519+0000","last_change":"2024-09-21T10:18:36.503584+0000","last_active":"2024-09-21T10:22:30.569519+0000","last_peered":"2024-09-21T10:22:30.569519+0000","last_clean":"2024-09-21T10:22:30.569519+0000","last_became_active":"2024-09-21T10:18:36.502990+0000","last_became_peered":"2024-09-21T10:18:36.502990+0000","last_unstale":"2024-09-21T10:22:30.569519+0000","last_undegraded":"2024-09-21T10:22:30.569519+0000","last_fullsized":"2024-09-21T10:22:30.569519+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:11.648271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":96,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.944359+0000","last_change":"2024-09-21T10:18:33.479008+0000","last_active":"2024-09-21T10:22:25.944359+0000","last_peered":"2024-09-21T10:22:25.944359+0000","last_clean":"2024-09-21T10:22:25.944359+0000","last_became_active":"2024-09-21T10:18:33.478740+0000","last_became_peered":"2024-09-21T10:18:33.478740+0000","last_unstale":"2024-09-21T10:22:25.944359+0000","last_undegraded":"2024-09-21T10:22:25.944359+0000","last_fullsized":"2024-09-21T10:22:25.944359+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:16:52.594882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"304'74","reported_seq":85,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:33.501558+0000","last_change":"2024-09-21T10:22:18.649356+0000","last_active":"2024-09-21T10:22:33.501558+0000","last_peered":"2024-09-21T10:22:33.501558+0000","last_clean":"2024-09-21T10:22:33.501558+0000","last_became_active":"2024-09-21T10:22:18.649035+0000","last_became_peered":"2024-09-21T10:22:18.649035+0000","last_unstale":"2024-09-21T10:22:33.501558+0000","last_undegraded":"2024-09-21T10:22:33.501558+0000","last_fullsized":"2024-09-21T10:22:33.501558+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:17:20.665367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4849664,"num_objects":74,"num_object_clones":0,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74,"num_write_kb":4736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":515,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569098+0000","last_change":"2024-09-21T10:08:00.390436+0000","last_active":"2024-09-21T10:22:30.569098+0000","last_peered":"2024-09-21T10:22:30.569098+0000","last_clean":"2024-09-21T10:22:30.569098+0000","last_became_active":"2024-09-21T10:08:00.381835+0000","last_became_peered":"2024-09-21T10:08:00.381835+0000","last_unstale":"2024-09-21T10:22:30.569098+0000","last_undegraded":"2024-09-21T10:22:30.569098+0000","last_fullsized":"2024-09-21T10:22:30.569098+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:01:35.697356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":530,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.945947+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:22:25.945947+0000","last_peered":"2024-09-21T10:22:25.945947+0000","last_clean":"2024-09-21T10:22:25.945947+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:22:25.945947+0000","last_undegraded":"2024-09-21T10:22:25.945947+0000","last_fullsized":"2024-09-21T10:22:25.945947+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":168,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.945203+0000","last_change":"2024-09-21T10:08:00.385774+0000","last_active":"2024-09-21T10:22:25.945203+0000","last_peered":"2024-09-21T10:22:25.945203+0000","last_clean":"2024-09-21T10:22:25.945203+0000","last_became_active":"2024-09-21T10:08:00.385445+0000","last_became_peered":"2024-09-21T10:08:00.385445+0000","last_unstale":"2024-09-21T10:22:25.945203+0000","last_undegraded":"2024-09-21T10:22:25.945203+0000","last_fullsized":"2024-09-21T10:22:25.945203+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:42.442190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":227,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.945430+0000","last_change":"2024-09-21T10:08:00.379661+0000","last_active":"2024-09-21T10:22:25.945430+0000","last_peered":"2024-09-21T10:22:25.945430+0000","last_clean":"2024-09-21T10:22:25.945430+0000","last_became_active":"2024-09-21T10:08:00.379301+0000","last_became_peered":"2024-09-21T10:08:00.379301+0000","last_unstale":"2024-09-21T10:22:25.945430+0000","last_undegraded":"2024-09-21T10:22:25.945430+0000","last_fullsized":"2024-09-21T10:22:25.945430+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:15.649523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":250,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.944343+0000","last_change":"2024-09-21T10:08:00.410731+0000","last_active":"2024-09-21T10:22:25.944343+0000","last_peered":"2024-09-21T10:22:25.944343+0000","last_clean":"2024-09-21T10:22:25.944343+0000","last_became_active":"2024-09-21T10:08:00.380324+0000","last_became_peered":"2024-09-21T10:08:00.380324+0000","last_unstale":"2024-09-21T10:22:25.944343+0000","last_undegraded":"2024-09-21T10:22:25.944343+0000","last_fullsized":"2024-09-21T10:22:25.944343+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:47:34.960221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":361,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.570026+0000","last_change":"2024-09-21T10:08:18.470378+0000","last_active":"2024-09-21T10:22:30.570026+0000","last_peered":"2024-09-21T10:22:30.570026+0000","last_clean":"2024-09-21T10:22:30.570026+0000","last_became_active":"2024-09-21T10:08:18.469482+0000","last_became_peered":"2024-09-21T10:08:18.469482+0000","last_unstale":"2024-09-21T10:22:30.570026+0000","last_undegraded":"2024-09-21T10:22:30.570026+0000","last_fullsized":"2024-09-21T10:22:30.570026+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:35.628724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":335,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.944726+0000","last_change":"2024-09-21T10:18:33.479236+0000","last_active":"2024-09-21T10:22:25.944726+0000","last_peered":"2024-09-21T10:22:25.944726+0000","last_clean":"2024-09-21T10:22:25.944726+0000","last_became_active":"2024-09-21T10:18:33.474256+0000","last_became_peered":"2024-09-21T10:18:33.474256+0000","last_unstale":"2024-09-21T10:22:25.944726+0000","last_undegraded":"2024-09-21T10:22:25.944726+0000","last_fullsized":"2024-09-21T10:22:25.944726+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:19:34.666035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":215,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.944713+0000","last_change":"2024-09-21T10:07:43.791830+0000","last_active":"2024-09-21T10:22:25.944713+0000","last_peered":"2024-09-21T10:22:25.944713+0000","last_clean":"2024-09-21T10:22:25.944713+0000","last_became_active":"2024-09-21T10:07:43.791314+0000","last_became_peered":"2024-09-21T10:07:43.791314+0000","last_unstale":"2024-09-21T10:22:25.944713+0000","last_undegraded":"2024-09-21T10:22:25.944713+0000","last_fullsized":"2024-09-21T10:22:25.944713+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:37.086629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":56,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.945088+0000","last_change":"2024-09-21T10:18:33.464989+0000","last_active":"2024-09-21T10:22:25.945088+0000","last_peered":"2024-09-21T10:22:25.945088+0000","last_clean":"2024-09-21T10:22:25.945088+0000","last_became_active":"2024-09-21T10:18:33.464617+0000","last_became_peered":"2024-09-21T10:18:33.464617+0000","last_unstale":"2024-09-21T10:22:25.945088+0000","last_undegraded":"2024-09-21T10:22:25.945088+0000","last_fullsized":"2024-09-21T10:22:25.945088+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:09.300797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":484,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569543+0000","last_change":"2024-09-21T10:18:33.472415+0000","last_active":"2024-09-21T10:22:30.569543+0000","last_peered":"2024-09-21T10:22:30.569543+0000","last_clean":"2024-09-21T10:22:30.569543+0000","last_became_active":"2024-09-21T10:18:33.471746+0000","last_became_peered":"2024-09-21T10:18:33.471746+0000","last_unstale":"2024-09-21T10:22:30.569543+0000","last_undegraded":"2024-09-21T10:22:30.569543+0000","last_fullsized":"2024-09-21T10:22:30.569543+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:40:37.099458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":130,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.568710+0000","last_change":"2024-09-21T10:18:33.484721+0000","last_active":"2024-09-21T10:22:30.568710+0000","last_peered":"2024-09-21T10:22:30.568710+0000","last_clean":"2024-09-21T10:22:30.568710+0000","last_became_active":"2024-09-21T10:18:33.483950+0000","last_became_peered":"2024-09-21T10:18:33.483950+0000","last_unstale":"2024-09-21T10:22:30.568710+0000","last_undegraded":"2024-09-21T10:22:30.568710+0000","last_fullsized":"2024-09-21T10:22:30.568710+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:54:52.208378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":267,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.945914+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:22:25.945914+0000","last_peered":"2024-09-21T10:22:25.945914+0000","last_clean":"2024-09-21T10:22:25.945914+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:22:25.945914+0000","last_undegraded":"2024-09-21T10:22:25.945914+0000","last_fullsized":"2024-09-21T10:22:25.945914+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":149,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569672+0000","last_change":"2024-09-21T10:18:33.479635+0000","last_active":"2024-09-21T10:22:30.569672+0000","last_peered":"2024-09-21T10:22:30.569672+0000","last_clean":"2024-09-21T10:22:30.569672+0000","last_became_active":"2024-09-21T10:18:33.478830+0000","last_became_peered":"2024-09-21T10:18:33.478830+0000","last_unstale":"2024-09-21T10:22:30.569672+0000","last_undegraded":"2024-09-21T10:22:30.569672+0000","last_fullsized":"2024-09-21T10:22:30.569672+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:16.950086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":500,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569476+0000","last_change":"2024-09-21T10:18:33.480350+0000","last_active":"2024-09-21T10:22:30.569476+0000","last_peered":"2024-09-21T10:22:30.569476+0000","last_clean":"2024-09-21T10:22:30.569476+0000","last_became_active":"2024-09-21T10:18:33.479248+0000","last_became_peered":"2024-09-21T10:18:33.479248+0000","last_unstale":"2024-09-21T10:22:30.569476+0000","last_undegraded":"2024-09-21T10:22:30.569476+0000","last_fullsized":"2024-09-21T10:22:30.569476+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:25:28.780572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":493,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569347+0000","last_change":"2024-09-21T10:18:33.480341+0000","last_active":"2024-09-21T10:22:30.569347+0000","last_peered":"2024-09-21T10:22:30.569347+0000","last_clean":"2024-09-21T10:22:30.569347+0000","last_became_active":"2024-09-21T10:18:33.479869+0000","last_became_peered":"2024-09-21T10:18:33.479869+0000","last_unstale":"2024-09-21T10:22:30.569347+0000","last_undegraded":"2024-09-21T10:22:30.569347+0000","last_fullsized":"2024-09-21T10:22:30.569347+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:32:42.206910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":254,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569057+0000","last_change":"2024-09-21T10:18:33.473981+0000","last_active":"2024-09-21T10:22:30.569057+0000","last_peered":"2024-09-21T10:22:30.569057+0000","last_clean":"2024-09-21T10:22:30.569057+0000","last_became_active":"2024-09-21T10:18:33.473619+0000","last_became_peered":"2024-09-21T10:18:33.473619+0000","last_unstale":"2024-09-21T10:22:30.569057+0000","last_undegraded":"2024-09-21T10:22:30.569057+0000","last_fullsized":"2024-09-21T10:22:30.569057+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:44:24.630856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":306,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.570208+0000","last_change":"2024-09-21T10:08:18.471282+0000","last_active":"2024-09-21T10:22:30.570208+0000","last_peered":"2024-09-21T10:22:30.570208+0000","last_clean":"2024-09-21T10:22:30.570208+0000","last_became_active":"2024-09-21T10:08:18.470952+0000","last_became_peered":"2024-09-21T10:08:18.470952+0000","last_unstale":"2024-09-21T10:22:30.570208+0000","last_undegraded":"2024-09-21T10:22:30.570208+0000","last_fullsized":"2024-09-21T10:22:30.570208+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:25.597081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":139,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569900+0000","last_change":"2024-09-21T10:18:33.480128+0000","last_active":"2024-09-21T10:22:30.569900+0000","last_peered":"2024-09-21T10:22:30.569900+0000","last_clean":"2024-09-21T10:22:30.569900+0000","last_became_active":"2024-09-21T10:18:33.479691+0000","last_became_peered":"2024-09-21T10:18:33.479691+0000","last_unstale":"2024-09-21T10:22:30.569900+0000","last_undegraded":"2024-09-21T10:22:30.569900+0000","last_fullsized":"2024-09-21T10:22:30.569900+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:43:49.311244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":491,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.568850+0000","last_change":"2024-09-21T10:08:00.386104+0000","last_active":"2024-09-21T10:22:30.568850+0000","last_peered":"2024-09-21T10:22:30.568850+0000","last_clean":"2024-09-21T10:22:30.568850+0000","last_became_active":"2024-09-21T10:08:00.381484+0000","last_became_peered":"2024-09-21T10:08:00.381484+0000","last_unstale":"2024-09-21T10:22:30.568850+0000","last_undegraded":"2024-09-21T10:22:30.568850+0000","last_fullsized":"2024-09-21T10:22:30.568850+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:39.190191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":168,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.568919+0000","last_change":"2024-09-21T10:18:33.481322+0000","last_active":"2024-09-21T10:22:30.568919+0000","last_peered":"2024-09-21T10:22:30.568919+0000","last_clean":"2024-09-21T10:22:30.568919+0000","last_became_active":"2024-09-21T10:18:33.480554+0000","last_became_peered":"2024-09-21T10:18:33.480554+0000","last_unstale":"2024-09-21T10:22:30.568919+0000","last_undegraded":"2024-09-21T10:22:30.568919+0000","last_fullsized":"2024-09-21T10:22:30.568919+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:15.498594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":317,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.570143+0000","last_change":"2024-09-21T10:08:18.466882+0000","last_active":"2024-09-21T10:22:30.570143+0000","last_peered":"2024-09-21T10:22:30.570143+0000","last_clean":"2024-09-21T10:22:30.570143+0000","last_became_active":"2024-09-21T10:08:18.466470+0000","last_became_peered":"2024-09-21T10:08:18.466470+0000","last_unstale":"2024-09-21T10:22:30.570143+0000","last_undegraded":"2024-09-21T10:22:30.570143+0000","last_fullsized":"2024-09-21T10:22:30.570143+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:51:31.020051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":168,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.568836+0000","last_change":"2024-09-21T10:18:33.472352+0000","last_active":"2024-09-21T10:22:30.568836+0000","last_peered":"2024-09-21T10:22:30.568836+0000","last_clean":"2024-09-21T10:22:30.568836+0000","last_became_active":"2024-09-21T10:18:33.471312+0000","last_became_peered":"2024-09-21T10:18:33.471312+0000","last_unstale":"2024-09-21T10:22:30.568836+0000","last_undegraded":"2024-09-21T10:22:30.568836+0000","last_fullsized":"2024-09-21T10:22:30.568836+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:32:44.897145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":179,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.945824+0000","last_change":"2024-09-21T10:08:00.394350+0000","last_active":"2024-09-21T10:22:25.945824+0000","last_peered":"2024-09-21T10:22:25.945824+0000","last_clean":"2024-09-21T10:22:25.945824+0000","last_became_active":"2024-09-21T10:08:00.387232+0000","last_became_peered":"2024-09-21T10:08:00.387232+0000","last_unstale":"2024-09-21T10:22:25.945824+0000","last_undegraded":"2024-09-21T10:22:25.945824+0000","last_fullsized":"2024-09-21T10:22:25.945824+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:56.160018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":152,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569519+0000","last_change":"2024-09-21T10:18:33.474216+0000","last_active":"2024-09-21T10:22:30.569519+0000","last_peered":"2024-09-21T10:22:30.569519+0000","last_clean":"2024-09-21T10:22:30.569519+0000","last_became_active":"2024-09-21T10:18:33.473794+0000","last_became_peered":"2024-09-21T10:18:33.473794+0000","last_unstale":"2024-09-21T10:22:30.569519+0000","last_undegraded":"2024-09-21T10:22:30.569519+0000","last_fullsized":"2024-09-21T10:22:30.569519+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:38.956888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":52,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.943643+0000","last_change":"2024-09-21T10:18:33.483576+0000","last_active":"2024-09-21T10:22:25.943643+0000","last_peered":"2024-09-21T10:22:25.943643+0000","last_clean":"2024-09-21T10:22:25.943643+0000","last_became_active":"2024-09-21T10:18:33.483116+0000","last_became_peered":"2024-09-21T10:18:33.483116+0000","last_unstale":"2024-09-21T10:22:25.943643+0000","last_undegraded":"2024-09-21T10:22:25.943643+0000","last_fullsized":"2024-09-21T10:22:25.943643+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:48.964766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":252,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-21T10:22:25.943574+0000","last_change":"2024-09-21T10:18:33.471909+0000","last_active":"2024-09-21T10:22:25.943574+0000","last_peered":"2024-09-21T10:22:25.943574+0000","last_clean":"2024-09-21T10:22:25.943574+0000","last_became_active":"2024-09-21T10:18:33.471614+0000","last_became_peered":"2024-09-21T10:18:33.471614+0000","last_unstale":"2024-09-21T10:22:25.943574+0000","last_undegraded":"2024-09-21T10:22:25.943574+0000","last_fullsized":"2024-09-21T10:22:25.943574+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:30:50.582505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":482,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-21T10:22:30.569437+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:22:30.569437+0000","last_peered":"2024-09-21T10:22:30.569437+0000","last_clean":"2024-09-21T10:22:30.569437+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:22:30.569437+0000","last_undegraded":"2024-09-21T10:22:30.569437+0000","last_fullsized":"2024-09-21T10:22:30.569437+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":52232192,"num_objects":797,"num_object_clones":0,"num_object_copies":2391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":797,"num_write_kb":51008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":323682304,"data_stored":323682304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":797,"ondisk_log_size":797,"up":48,"acting":48,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":10016,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5982,"num_read_kb":13005,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397893,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":905768,"kb_used_data":8540,"kb_used_omap":1,"kb_used_meta":897214,"kb_avail":92839384,"statfs":{"total":95995035648,"available":95067529216,"internally_reserved":0,"allocated":8744960,"data_stored":7090868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918747480},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594121,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":942820,"kb_used_data":37340,"kb_used_omap":1,"kb_used_meta":905470,"kb_avail":92802332,"statfs":{"total":95995035648,"available":95029587968,"internally_reserved":0,"allocated":38236160,"data_stored":36582065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":927201624},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790351,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":912016,"kb_used_data":7940,"kb_used_omap":1,"kb_used_meta":904062,"kb_avail":92833136,"statfs":{"total":95995035648,"available":95061131264,"internally_reserved":0,"allocated":8130560,"data_stored":6495884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":925759832},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986580,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":433791697142,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860669,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":747544,"kb_used_data":61392,"kb_used_omap":8,"kb_used_meta":686135,"kb_avail":92997608,"statfs":{"total":95995035648,"available":95229550592,"internally_reserved":0,"allocated":62865408,"data_stored":61223468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":702603116},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024194,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":594504,"kb_used_data":3836,"kb_used_omap":7,"kb_used_meta":590648,"kb_avail":93150648,"statfs":{"total":95995035648,"available":95386263552,"internally_reserved":0,"allocated":3928064,"data_stored":2300667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":604824114},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187730,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":686936,"kb_used_data":4428,"kb_used_omap":8,"kb_used_meta":682487,"kb_avail":93058216,"statfs":{"total":95995035648,"available":95291613184,"internally_reserved":0,"allocated":4534272,"data_stored":2892306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":698867697},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140640256,"data_stored":140640256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":93847552,"data_stored":93847552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36110336,"data_stored":36110336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53084160,"data_stored":53084160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:22:40.519 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T10:22:40.520 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:22:40.520 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-21T10:22:40.520 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:22:40.520 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:22:40.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-21T10:22:40.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:40.522 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:40.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:40.578 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:22:40.579 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:22:40.579 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:22:40.579 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:22:40.579 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:22:40.579 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:22:40.579 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:22:40.579 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:22:40.579 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:22:40.579 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:22:40.579 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:22:40.580 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:22:40.580 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:22:40.580 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:22:40.580 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:22:40.580 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:22:40.594 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:40 smithi196 ceph-mon[84784]: osdmap e307: 8 total, 8 up, 6 in 2024-09-21T10:22:40.595 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:40 smithi196 ceph-mon[84784]: pgmap v1009: 89 pgs: 89 active+clean; 947 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 157 MiB/s wr, 2.51k op/s 2024-09-21T10:22:40.595 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:40 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2523664425' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:40.595 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:22:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:22:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:22:40.622 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:40.656 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:40.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:40.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:40.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:40.792 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:40.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:40.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:40.802 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:22:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:22:40] "GET /metrics HTTP/1.1" 200 36521 "" "Prometheus/2.51.0" 2024-09-21T10:22:40.894 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:40.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:40.928 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:40.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:40.980 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:22:41.029 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:41.065 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:41.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:41.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:41.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:41.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:41.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:41.211 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 8 16 24838 24822 193.904 165.562 0.0041628 0.0051515 2024-09-21T10:22:41.308 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:41.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:41.346 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:41.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:41.447 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:41.467 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:41.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:41.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:41.569 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:41.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:41.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:41.706 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:41.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:41.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:41.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:41.843 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:41.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:41.880 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:41.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:41.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:42.018 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:42.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:42.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:42.120 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:42.153 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:42.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:42.211 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 9 16 27026 27010 187.552 136.75 0.00296481 0.0052953 2024-09-21T10:22:42.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:42.275 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:42.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:42.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:42.377 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:42.413 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:42.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:42.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:42.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:42.548 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:42.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:42.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:42.650 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:42.683 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:42.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:42.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:42.784 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:42.821 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:42.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:42.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:42.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:42 smithi098 ceph-mon[97580]: pgmap v1010: 89 pgs: 89 active+clean; 1008 MiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-21T10:22:42.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:42 smithi098 ceph-mon[100914]: pgmap v1010: 89 pgs: 89 active+clean; 1008 MiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-21T10:22:42.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:42 smithi196 ceph-mon[84784]: pgmap v1010: 89 pgs: 89 active+clean; 1008 MiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-21T10:22:42.923 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:42.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:42.959 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:42.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:43.060 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:43.089 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:43.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:43.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:43.190 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:43.211 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 10 16 29014 28998 181.221 124.25 0.00227334 0.00546095 2024-09-21T10:22:43.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:43.212 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:43.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:43.313 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:43.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:43.348 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:43.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:43.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:43.475 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:43.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:43.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:43.577 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:43.597 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:43.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:43.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:43.687 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:22:43.687 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-21T10:22:43.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-21T10:22:43.687 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:22:43.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.687 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:66530 8.8 8:1703476e:::benchmark_data_smithi098_226587_object66529:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e275)", 2024-09-21T10:22:43.687 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:33.462736+0000", 2024-09-21T10:22:43.687 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 250.22164067099999, 2024-09-21T10:22:43.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.26090346, 2024-09-21T10:22:43.688 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:22:43.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:22:43.688 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-21T10:22:43.688 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:22:43.688 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:22:43.688 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:22:43.688 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66530 2024-09-21T10:22:43.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.688 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:22:43.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:22:43.689 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.462736+0000", 2024-09-21T10:22:43.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:22:43.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:22:43.689 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.462736+0000", 2024-09-21T10:22:43.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:22:43.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:22:43.689 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.462737+0000", 2024-09-21T10:22:43.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5409999999999999e-06 2024-09-21T10:22:43.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:22:43.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.462761+0000", 2024-09-21T10:22:43.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4034000000000001e-05 2024-09-21T10:22:43.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:22:43.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.462779+0000", 2024-09-21T10:22:43.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7659000000000002e-05 2024-09-21T10:22:43.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:22:43.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.462788+0000", 2024-09-21T10:22:43.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6690000000000001e-06 2024-09-21T10:22:43.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:22:43.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.463013+0000", 2024-09-21T10:22:43.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022451700000000001 2024-09-21T10:22:43.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:22:43.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.463022+0000", 2024-09-21T10:22:43.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8200000000000003e-06 2024-09-21T10:22:43.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:22:43.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.701909+0000", 2024-09-21T10:22:43.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2388876689999999 2024-09-21T10:22:43.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for degraded object", 2024-09-21T10:22:43.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.703356+0000", 2024-09-21T10:22:43.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0014468079999999999 2024-09-21T10:22:43.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:22:43.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.723531+0000", 2024-09-21T10:22:43.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020174847999999999 2024-09-21T10:22:43.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:22:43.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.723639+0000", 2024-09-21T10:22:43.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107895 2024-09-21T10:22:43.692 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:22:43.693 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:22:43.693 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:22:43.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.693 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:66548 8.8 8:1c469b76:::benchmark_data_smithi098_226587_object66547:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e275)", 2024-09-21T10:22:43.693 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:33.462803+0000", 2024-09-21T10:22:43.693 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 250.22157329800001, 2024-09-21T10:22:43.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.263875914, 2024-09-21T10:22:43.693 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:22:43.693 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:22:43.693 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:43.693 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:22:43.693 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:22:43.693 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:22:43.693 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66548 2024-09-21T10:22:43.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.694 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:22:43.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.694 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:22:43.694 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.462803+0000", 2024-09-21T10:22:43.694 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:22:43.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.694 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:22:43.694 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.462803+0000", 2024-09-21T10:22:43.694 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:22:43.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.694 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:22:43.694 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.462803+0000", 2024-09-21T10:22:43.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.37e-07 2024-09-21T10:22:43.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.695 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:22:43.695 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.462825+0000", 2024-09-21T10:22:43.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1464e-05 2024-09-21T10:22:43.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.695 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:22:43.695 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.462832+0000", 2024-09-21T10:22:43.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.576e-06 2024-09-21T10:22:43.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.695 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:22:43.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.462838+0000", 2024-09-21T10:22:43.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2989999999999996e-06 2024-09-21T10:22:43.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.696 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:22:43.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.463046+0000", 2024-09-21T10:22:43.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020775899999999999 2024-09-21T10:22:43.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.696 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:22:43.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.463052+0000", 2024-09-21T10:22:43.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2620000000000004e-06 2024-09-21T10:22:43.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.697 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:22:43.697 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.706996+0000", 2024-09-21T10:22:43.697 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2439441170000001 2024-09-21T10:22:43.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.697 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:22:43.697 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.707213+0000", 2024-09-21T10:22:43.697 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000217438 2024-09-21T10:22:43.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.697 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,5,7", 2024-09-21T10:22:43.698 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.707338+0000", 2024-09-21T10:22:43.698 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012501 2024-09-21T10:22:43.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.698 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:43.698 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.723681+0000", 2024-09-21T10:22:43.698 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016343127999999998 2024-09-21T10:22:43.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.698 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:43.698 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.723737+0000", 2024-09-21T10:22:43.698 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5699999999999999e-05 2024-09-21T10:22:43.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:22:43.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.723754+0000", 2024-09-21T10:22:43.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6835000000000001e-05 2024-09-21T10:22:43.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:43.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.723885+0000", 2024-09-21T10:22:43.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013049200000000001 2024-09-21T10:22:43.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:22:43.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.723938+0000", 2024-09-21T10:22:43.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3177999999999998e-05 2024-09-21T10:22:43.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.700 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:22:43.700 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.726679+0000", 2024-09-21T10:22:43.700 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027411190000000002 2024-09-21T10:22:43.700 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:22:43.700 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:22:43.700 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:22:43.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.701 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:66553 8.8 8:1b64a94c:::benchmark_data_smithi098_226587_object66552:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e275)", 2024-09-21T10:22:43.701 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:33.462846+0000", 2024-09-21T10:22:43.701 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 250.22152972200001, 2024-09-21T10:22:43.701 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2975810720000001, 2024-09-21T10:22:43.701 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:22:43.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:22:43.701 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:43.701 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:22:43.701 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:22:43.701 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:22:43.702 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66553 2024-09-21T10:22:43.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.702 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:22:43.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.702 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:22:43.702 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.462846+0000", 2024-09-21T10:22:43.702 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:22:43.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.702 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:22:43.702 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.462846+0000", 2024-09-21T10:22:43.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:22:43.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:22:43.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.462847+0000", 2024-09-21T10:22:43.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5199999999999997e-07 2024-09-21T10:22:43.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:22:43.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.462868+0000", 2024-09-21T10:22:43.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1030999999999999e-05 2024-09-21T10:22:43.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.704 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:22:43.704 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.462876+0000", 2024-09-21T10:22:43.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9310000000000008e-06 2024-09-21T10:22:43.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.704 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:22:43.704 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.462882+0000", 2024-09-21T10:22:43.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9649999999999996e-06 2024-09-21T10:22:43.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.705 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:22:43.705 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.463075+0000", 2024-09-21T10:22:43.705 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019348799999999999 2024-09-21T10:22:43.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.705 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:22:43.705 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.463082+0000", 2024-09-21T10:22:43.705 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1770000000000003e-06 2024-09-21T10:22:43.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.705 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:22:43.705 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.726746+0000", 2024-09-21T10:22:43.705 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.263664425 2024-09-21T10:22:43.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.706 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:22:43.706 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.726922+0000", 2024-09-21T10:22:43.706 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000176427 2024-09-21T10:22:43.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.706 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,5,7", 2024-09-21T10:22:43.706 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.727093+0000", 2024-09-21T10:22:43.706 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017092600000000001 2024-09-21T10:22:43.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.707 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:43.707 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.734577+0000", 2024-09-21T10:22:43.707 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0074840640000000003 2024-09-21T10:22:43.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.707 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:43.707 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.734605+0000", 2024-09-21T10:22:43.707 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7752000000000001e-05 2024-09-21T10:22:43.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.708 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:22:43.708 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.740046+0000", 2024-09-21T10:22:43.708 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054408599999999996 2024-09-21T10:22:43.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.708 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:43.708 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.757140+0000", 2024-09-21T10:22:43.708 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017094030999999999 2024-09-21T10:22:43.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.708 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:22:43.708 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.757210+0000", 2024-09-21T10:22:43.708 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9769000000000002e-05 2024-09-21T10:22:43.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.709 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:22:43.709 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.760428+0000", 2024-09-21T10:22:43.709 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032176739999999998 2024-09-21T10:22:43.709 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:22:43.709 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:22:43.709 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:22:43.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.709 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:95762 8.8 8:1c9c65fc:::benchmark_data_smithi098_226587_object95761:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-21T10:22:43.709 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:19:14.934697+0000", 2024-09-21T10:22:43.709 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 208.74967903199999, 2024-09-21T10:22:43.709 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.29361763699999999, 2024-09-21T10:22:43.709 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:22:43.709 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:22:43.709 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:43.710 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:22:43.710 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:22:43.710 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:22:43.710 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 95762 2024-09-21T10:22:43.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.710 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:22:43.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.710 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:22:43.710 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.934697+0000", 2024-09-21T10:22:43.710 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:22:43.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.710 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:22:43.710 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.934697+0000", 2024-09-21T10:22:43.711 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:22:43.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.711 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:22:43.711 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.934699+0000", 2024-09-21T10:22:43.711 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0490000000000002e-06 2024-09-21T10:22:43.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.711 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:22:43.711 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.934721+0000", 2024-09-21T10:22:43.711 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1304000000000001e-05 2024-09-21T10:22:43.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.711 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:22:43.711 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.934738+0000", 2024-09-21T10:22:43.711 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7719000000000001e-05 2024-09-21T10:22:43.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.712 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:22:43.712 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.934746+0000", 2024-09-21T10:22:43.712 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.588e-06 2024-09-21T10:22:43.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.712 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:22:43.712 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.934805+0000", 2024-09-21T10:22:43.712 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8712000000000003e-05 2024-09-21T10:22:43.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.712 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:22:43.712 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.934997+0000", 2024-09-21T10:22:43.712 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019199700000000001 2024-09-21T10:22:43.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.713 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,5,7", 2024-09-21T10:22:43.713 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.935201+0000", 2024-09-21T10:22:43.713 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000204835 2024-09-21T10:22:43.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.713 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:22:43.713 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.938392+0000", 2024-09-21T10:22:43.713 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003190146 2024-09-21T10:22:43.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.713 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:43.713 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.939186+0000", 2024-09-21T10:22:43.713 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00079440299999999995 2024-09-21T10:22:43.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.714 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:43.714 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.939542+0000", 2024-09-21T10:22:43.714 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00035647599999999999 2024-09-21T10:22:43.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.714 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:43.714 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.224421+0000", 2024-09-21T10:22:43.714 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.28487878100000003 2024-09-21T10:22:43.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.714 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:22:43.715 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.224758+0000", 2024-09-21T10:22:43.715 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00033709000000000001 2024-09-21T10:22:43.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:22:43.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:22:43.715 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:22:43.715 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.228315+0000", 2024-09-21T10:22:43.715 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035565369999999998 2024-09-21T10:22:43.715 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:22:43.715 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:22:43.715 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:22:43.715 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:22:43.715 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:22:43.715 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:22:43.716 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:43.736 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:43.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:43.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:43.838 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:43.871 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:43.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:43.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:43.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:44.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:44.007 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:44.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:44.108 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:44.129 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:44.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:44.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:44.211 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:22:44.212 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 11 15 30698 30683 174.319 105.312 0.00769078 0.00573178 2024-09-21T10:22:44.231 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:44.262 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:44.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:44.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:44.364 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:44.391 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:44.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:44.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:44.493 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:44.510 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:44 smithi098 ceph-mon[97580]: pgmap v1011: 89 pgs: 89 active+clean; 1.4 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 187 MiB/s wr, 3.00k op/s 2024-09-21T10:22:44.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:44.531 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:44.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:44.618 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:22:44.632 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:44.670 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:22:44.684 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:44.700 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:44.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:44.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:44.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:44.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:44 smithi098 ceph-mon[100914]: pgmap v1011: 89 pgs: 89 active+clean; 1.4 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 187 MiB/s wr, 3.00k op/s 2024-09-21T10:22:44.843 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:44.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:44.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:44 smithi196 ceph-mon[84784]: pgmap v1011: 89 pgs: 89 active+clean; 1.4 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 187 MiB/s wr, 3.00k op/s 2024-09-21T10:22:44.945 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:44.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:44.985 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:44.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:45.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:45.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:45.123 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:45.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:45.212 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 12 16 32643 32627 169.917 121.5 0.00926792 0.00587854 2024-09-21T10:22:45.220 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:22:45.224 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:45.257 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:45.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:45.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:45.359 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:45.396 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:45.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:45.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:45.498 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:45.534 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:45.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:45.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:45.561 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:45.636 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:45.673 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:45.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:45.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:45.774 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:45.811 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:45.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:45.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:45.913 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:45.947 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:45.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:45.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:46.048 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:46.084 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:46.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:46.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:46.186 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:46.212 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 13 16 34924 34908 167.812 142.562 0.00302848 0.00595304 2024-09-21T10:22:46.221 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:46.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:46.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:46.322 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:46.360 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:46.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:46.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:46.462 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:46.496 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:46.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:46.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:46.598 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:46.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:46 smithi098 ceph-mon[97580]: pgmap v1012: 89 pgs: 89 active+clean; 1.7 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 184 MiB/s wr, 2.95k op/s 2024-09-21T10:22:46.616 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:46 smithi098 ceph-mon[100914]: pgmap v1012: 89 pgs: 89 active+clean; 1.7 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 184 MiB/s wr, 2.95k op/s 2024-09-21T10:22:46.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:46.638 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:46.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:46.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:46 smithi196 ceph-mon[84784]: pgmap v1012: 89 pgs: 89 active+clean; 1.7 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 184 MiB/s wr, 2.95k op/s 2024-09-21T10:22:46.739 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:46.771 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:46.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:46.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:46.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:46.905 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:46.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:46.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:47.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:47.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:47.043 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:47.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:47.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:47.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:47.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:47.212 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 14 16 36864 36848 164.485 121.25 0.00726876 0.00607467 2024-09-21T10:22:47.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:47.313 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:47.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:47.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:47.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:47.441 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:47.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:47.543 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:47.574 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:47.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:47.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:47.588 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:22:47.588 INFO:teuthology.orchestra.run.smithi196.stdout: "success": "" 2024-09-21T10:22:47.588 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:22:47.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:47.709 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:47.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:47.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:47.811 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:47.850 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:47.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:47.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:47.952 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:47.987 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:47.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:47.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:48.089 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:48.123 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:48.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:48.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:48.212 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 15 16 38776 38760 161.485 119.5 0.00653686 0.00618785 2024-09-21T10:22:48.224 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:48.249 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:48.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:48.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:48.351 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:48.386 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:48.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:48.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:48.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:48.522 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:48.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:48.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:48.624 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:48.658 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:48.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:48.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:48.760 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:48.800 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:48.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:48.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:48.833 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:48 smithi196 ceph-mon[84784]: pgmap v1013: 89 pgs: 89 active+clean; 1.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-09-21T10:22:48.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:48 smithi098 ceph-mon[97580]: pgmap v1013: 89 pgs: 89 active+clean; 1.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-09-21T10:22:48.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:48 smithi098 ceph-mon[100914]: pgmap v1013: 89 pgs: 89 active+clean; 1.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-09-21T10:22:48.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:48.924 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:48.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:48.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:48.930 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:22:48.930 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:22:48.930 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-21T10:22:48.930 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:22:49.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:49.061 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:49.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:49.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:49.163 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:49.195 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:49.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:49.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:49.227 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 16 15 40375 40360 157.494 100 0.00453883 0.00634496 2024-09-21T10:22:49.298 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:49.333 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:49.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:49.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:49.435 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:49.470 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:49.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:49.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:49.572 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:49.603 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:49.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:49.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:49.697 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:22:49.705 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:49.737 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:49.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:49.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:49.838 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:49.870 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:49.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:49.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:49.971 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:50.007 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:50.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:50.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:50.108 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:50.111 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:22:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:50.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:50.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:50.227 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 17 16 42084 42068 154.511 106.75 0.100808 0.00644482 2024-09-21T10:22:50.241 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:50.276 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:50.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:50.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:50.377 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:50.401 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:50.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:50.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:50.502 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:50.518 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:22:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:22:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:22:50.536 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:50.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:50.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:50.638 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:50.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:50.672 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:50.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:50.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:50.812 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:50.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:50.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:50.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:50 smithi098 ceph-mon[97580]: pgmap v1014: 89 pgs: 89 active+clean; 2.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 140 MiB/s wr, 2.25k op/s 2024-09-21T10:22:50.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:22:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:22:50] "GET /metrics HTTP/1.1" 200 36521 "" "Prometheus/2.51.0" 2024-09-21T10:22:50.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:50 smithi098 ceph-mon[100914]: pgmap v1014: 89 pgs: 89 active+clean; 2.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 140 MiB/s wr, 2.25k op/s 2024-09-21T10:22:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:50 smithi196 ceph-mon[84784]: pgmap v1014: 89 pgs: 89 active+clean; 2.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 140 MiB/s wr, 2.25k op/s 2024-09-21T10:22:50.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:50.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:50.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:50.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:51.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:51.079 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:51.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:51.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:51.180 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:51.215 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:51.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:51.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:51.227 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 18 16 43788 43772 151.846 106.5 0.00702369 0.00658111 2024-09-21T10:22:51.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:51.344 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:51.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:51.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:51.445 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:51.482 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:51.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:51.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:22:51.584 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:51.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:51.616 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:51.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:51.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:51 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2752475284' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:22:51.717 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:51.756 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:51.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:51.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:51.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:51 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2752475284' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:22:51.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:51 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2752475284' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:22:51.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:51.902 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:51.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:51.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:51.981 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:22:51.981 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:22:51.981 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-21T10:22:51.982 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-21T10:22:51.982 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:22:52.004 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:52.042 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:52.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:52.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:52.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:52.179 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:52.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:52.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:52.227 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 19 16 45527 45511 149.576 108.688 0.00366812 0.00667933 2024-09-21T10:22:52.282 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:52.315 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:52.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:52.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:52.418 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:52.438 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:52.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:52.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:52.539 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:52.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:52.575 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:52.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:52.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:52.698 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:22:52.708 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:52.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:52.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:52.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:52.844 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:52.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:52.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:52.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:52 smithi098 ceph-mon[97580]: pgmap v1015: 89 pgs: 89 active+clean; 2.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-09-21T10:22:52.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:52 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:22:52.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:52 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2752475284' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:22:52.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:52 smithi098 ceph-mon[97580]: osdmap e308: 8 total, 8 up, 6 in 2024-09-21T10:22:52.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:52.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:52 smithi098 ceph-mon[100914]: pgmap v1015: 89 pgs: 89 active+clean; 2.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-09-21T10:22:52.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:52 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:22:52.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:52 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2752475284' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:22:52.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:52 smithi098 ceph-mon[100914]: osdmap e308: 8 total, 8 up, 6 in 2024-09-21T10:22:52.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:52.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:52 smithi196 ceph-mon[84784]: pgmap v1015: 89 pgs: 89 active+clean; 2.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-09-21T10:22:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:52 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:22:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:52 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2752475284' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:22:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:52 smithi196 ceph-mon[84784]: osdmap e308: 8 total, 8 up, 6 in 2024-09-21T10:22:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:52.945 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:52.981 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:52.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:52.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:53.076 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:22:53.083 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:53.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:53.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:53.211 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:53.227 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:22:53.226486+0000 min lat: 0.00136025 max lat: 0.142779 avg lat: 0.00679056 2024-09-21T10:22:53.264 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:22:53.264 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 20 16 47146 47130 147.158 101.188 0.00581615 0.00679056 2024-09-21T10:22:53.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:53.265 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:53.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:53.290 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7] out_osds: [4, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:22:53.290 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:22:53.290 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-21T10:22:53.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:22:53.366 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:53.389 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:53.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:53.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:53.490 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:53.511 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:53.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:53.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:53.613 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:53.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:53.647 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:53.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:53.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:53.782 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:53.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:53.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:53.884 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:53.917 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:53.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:53.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:54.019 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:54.055 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:54.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:54.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:54.156 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:54.193 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:54.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:54.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:54.227 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 21 16 48601 48585 144.482 90.9375 0.00551186 0.00691634 2024-09-21T10:22:54.295 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:54.326 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:54.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:54.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:54.428 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:54.460 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:54.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:54.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:54.561 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:54.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:54.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:54 smithi196 ceph-mon[84784]: pgmap v1017: 89 pgs: 89 active+clean; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 123 MiB/s wr, 1.97k op/s 2024-09-21T10:22:54.696 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:54.730 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:54.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:54.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:54.832 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:54.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:54 smithi098 ceph-mon[97580]: pgmap v1017: 89 pgs: 89 active+clean; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 123 MiB/s wr, 1.97k op/s 2024-09-21T10:22:54.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:54 smithi098 ceph-mon[100914]: pgmap v1017: 89 pgs: 89 active+clean; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 123 MiB/s wr, 1.97k op/s 2024-09-21T10:22:54.868 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:54.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:54.970 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:55.003 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:55.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:55.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:55.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:55.138 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:55.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:55.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:55.228 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 22 16 50274 50258 142.668 104.562 0.00779711 0.00700323 2024-09-21T10:22:55.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:55.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:55.282 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:55.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:55.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:55.403 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:55.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:55.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:55.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:55.532 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:55.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:55.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:55.634 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:55.677 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:55.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:55.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:55.765 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:22:55.765 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-21T10:22:55.765 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-21T10:22:55.765 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [] 2024-09-21T10:22:55.765 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:22:55.778 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:55.825 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:55.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:55.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:55.927 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:55.957 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:55.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:55.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:56.059 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:56.094 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:56.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:56.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:56.196 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:56.227 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 23 16 51823 51807 140.675 96.8125 0.00282271 0.00710292 2024-09-21T10:22:56.228 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:56.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:56.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:56.330 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:56.351 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:56.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:56.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:56.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:56.484 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:56.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:56.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:56.586 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:56.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:56.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:56.709 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:56.735 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:56.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:56.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:56.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:56.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:56 smithi098 ceph-mon[97580]: pgmap v1018: 89 pgs: 89 active+clean; 2.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-09-21T10:22:56.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:56 smithi098 ceph-mon[100914]: pgmap v1018: 89 pgs: 89 active+clean; 2.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-09-21T10:22:56.871 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:56.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:56.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:56.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:56 smithi196 ceph-mon[84784]: pgmap v1018: 89 pgs: 89 active+clean; 2.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-09-21T10:22:56.974 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:56.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:56.996 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:56.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:57.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:57.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:57.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:57.219 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:57.228 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 24 16 52825 52809 137.423 62.625 0.0893889 0.00725391 2024-09-21T10:22:57.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:57.251 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:57.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:57.352 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:57.381 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:57.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:57.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:57.483 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:57.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:57.524 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:57.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:57.625 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:57.669 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:57.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:57.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:57.771 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:57.804 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:57.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:57.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:57.906 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:57.941 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:57.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:57.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:58.043 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:58.075 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:58.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:58.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:58.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:58.210 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:58.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:58.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:58.228 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 25 16 54493 54477 136.096 104.25 0.00796284 0.00734278 2024-09-21T10:22:58.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:58.353 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:58.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:58.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:58.455 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:58.482 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:58.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:58.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:58.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:58.620 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:58.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:58.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:58.722 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:58.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:58.756 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:58.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:58.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:58 smithi098 ceph-mon[97580]: pgmap v1019: 89 pgs: 89 active+clean; 3.0 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-09-21T10:22:58.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:58 smithi098 ceph-mon[100914]: pgmap v1019: 89 pgs: 89 active+clean; 3.0 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-09-21T10:22:58.857 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:58 smithi196 ceph-mon[84784]: pgmap v1019: 89 pgs: 89 active+clean; 3.0 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-09-21T10:22:58.895 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:58.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:58.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:58.997 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:59.030 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:59.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:59.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:59.131 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:59.172 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:59.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:59.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:59.233 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 26 16 56219 56203 134.988 107.875 0.0117224 0.00740247 2024-09-21T10:22:59.233 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:59.247 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:22:59.274 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:59.326 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:22:59.326 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:22:59.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:59.327 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:59.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:59.361 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:22:59.361 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:22:59.393 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:22:59.394 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:22:59.428 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:22:59.429 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:22:59.429 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:59.459 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:22:59.459 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:22:59.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:59.485 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:59.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:59.490 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:22:59.490 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:22:59.521 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:22:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:22:59.551 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:22:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:22:59.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:22:59 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3960949237' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:59.585 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:59.623 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:59.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:59.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:59.640 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:22:59.726 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:59.764 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:22:59.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:59.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:59.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:22:59 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3960949237' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:59.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:22:59 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3960949237' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:59.866 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:59.897 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:22:59.905 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:22:59.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:59.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:00.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:00.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:00.054 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:00.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:00.155 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:00.188 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:00.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:00.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:00.233 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 27 16 57684 57668 133.381 91.5625 0.00616464 0.00749191 2024-09-21T10:23:00.290 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:00.322 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:00.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:00.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:00.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:23:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:23:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:23:00.424 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:00.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:00.462 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:00.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:00.563 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:00.598 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:00.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:00.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:00.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:00 smithi098 ceph-mon[97580]: pgmap v1020: 89 pgs: 89 active+clean; 3.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-21T10:23:00.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:00 smithi098 ceph-mon[100914]: pgmap v1020: 89 pgs: 89 active+clean; 3.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-21T10:23:00.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:00.737 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:00.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:00.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:00.823 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:00 smithi196 ceph-mon[84784]: pgmap v1020: 89 pgs: 89 active+clean; 3.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-21T10:23:00.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:00.872 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:00.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:00.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:00.974 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:01.023 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:01.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:01.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:01.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:23:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:23:00] "GET /metrics HTTP/1.1" 200 36533 "" "Prometheus/2.51.0" 2024-09-21T10:23:01.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:01.160 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:01.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:01.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:01.233 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 28 16 59309 59293 132.245 101.562 0.00556853 0.0075573 2024-09-21T10:23:01.261 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:01.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:01.302 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:01.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:01.403 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:01.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:01.438 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:01.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:01.539 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:01.571 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:01.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:01.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:01.623 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:23:01.624 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:23:01.673 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:01.711 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:01.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:01.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:01.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:01.851 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:01.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:01.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:01.953 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:01.988 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:01.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:01.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:02.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:02.122 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:02.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:02.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:02.224 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:02.233 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 29 16 60883 60867 131.077 98.375 0.0115267 0.00762392 2024-09-21T10:23:02.263 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:02.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:02.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:02.364 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:02.384 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:23:02.400 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:02.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:02.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:02.471 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:02.503 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:02.534 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:02.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:02.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:02.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:02 smithi098 ceph-mon[97580]: pgmap v1021: 89 pgs: 89 active+clean; 3.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-21T10:23:02.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:02 smithi098 ceph-mon[100914]: pgmap v1021: 89 pgs: 89 active+clean; 3.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-21T10:23:02.636 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:02 smithi196 ceph-mon[84784]: pgmap v1021: 89 pgs: 89 active+clean; 3.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-21T10:23:02.695 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:02.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:02.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:02.702 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:23:02.703 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:23:02.781 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:23:02.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:02.836 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:02.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:02.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:02.937 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:02.970 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:02.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:02.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:03.071 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:03.103 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:03.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:03.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:03.205 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:03.233 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 30 16 62188 62172 129.428 81.5625 0.00655193 0.00772156 2024-09-21T10:23:03.240 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:03.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:03.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:03.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:03.379 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:03.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:03.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:03.481 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:03.502 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:03.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:03.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:03.604 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:03.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:03.643 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:03.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:03.744 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:03.754 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:23:03.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:03.788 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:03.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:03.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:03 smithi098 ceph-mon[100914]: from='client.46055 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:23:03.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:03 smithi098 ceph-mon[97580]: from='client.46055 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:23:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:03 smithi196 ceph-mon[84784]: from='client.46055 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:23:03.889 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:03.920 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:03.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:03.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:04.022 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:04.064 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:04.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:04.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:04.166 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:04.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:04.205 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:04.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:04.233 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 31 16 63678 63662 128.257 93.125 0.00854402 0.00779174 2024-09-21T10:23:04.306 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:04.340 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:04.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:04.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:04.442 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:04.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:04.480 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:04.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:04.581 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:04.588 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1021,"stamp":"2024-09-21T10:23:01.455270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3556120525,"num_objects":54217,"num_object_clones":0,"num_object_copies":162651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54217,"num_whiteouts":0,"num_read":6288,"num_read_kb":13764,"num_write":55245,"num_write_kb":3482347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":600384,"num_keys_recovered":30,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55321,"ondisk_log_size":55321,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":15172068,"kb_used_data":10378228,"kb_used_omap":28,"kb_used_meta":4793763,"kb_avail":547298844,"statfs":{"total":575970213888,"available":560434016256,"internally_reserved":0,"allocated":10627305472,"data_stored":10617357154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29289,"internal_metadata":4908813719},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,8,5,6,7,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1039269888,"num_objects":15858,"num_object_clones":0,"num_object_copies":47574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15858,"num_write_kb":1014912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004897"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":139,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.548778+0000","last_change":"2024-09-21T10:18:33.471373+0000","last_active":"2024-09-21T10:22:51.548778+0000","last_peered":"2024-09-21T10:22:51.548778+0000","last_clean":"2024-09-21T10:22:51.548778+0000","last_became_active":"2024-09-21T10:18:33.471052+0000","last_became_peered":"2024-09-21T10:18:33.471052+0000","last_unstale":"2024-09-21T10:22:51.548778+0000","last_undegraded":"2024-09-21T10:22:51.548778+0000","last_fullsized":"2024-09-21T10:22:51.548778+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:42:19.491129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":322,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.375777+0000","last_change":"2024-09-21T10:08:00.406360+0000","last_active":"2024-09-21T10:22:39.375777+0000","last_peered":"2024-09-21T10:22:39.375777+0000","last_clean":"2024-09-21T10:22:39.375777+0000","last_became_active":"2024-09-21T10:08:00.380042+0000","last_became_peered":"2024-09-21T10:08:00.380042+0000","last_unstale":"2024-09-21T10:22:39.375777+0000","last_undegraded":"2024-09-21T10:22:39.375777+0000","last_fullsized":"2024-09-21T10:22:39.375777+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:49:46.544051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":238,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.374702+0000","last_change":"2024-09-21T10:18:33.465357+0000","last_active":"2024-09-21T10:22:39.374702+0000","last_peered":"2024-09-21T10:22:39.374702+0000","last_clean":"2024-09-21T10:22:39.374702+0000","last_became_active":"2024-09-21T10:18:33.464952+0000","last_became_peered":"2024-09-21T10:18:33.464952+0000","last_unstale":"2024-09-21T10:22:39.374702+0000","last_undegraded":"2024-09-21T10:22:39.374702+0000","last_fullsized":"2024-09-21T10:22:39.374702+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:15:41.090866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":114,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.373931+0000","last_change":"2024-09-21T10:18:33.475291+0000","last_active":"2024-09-21T10:22:39.373931+0000","last_peered":"2024-09-21T10:22:39.373931+0000","last_clean":"2024-09-21T10:22:39.373931+0000","last_became_active":"2024-09-21T10:18:33.474971+0000","last_became_peered":"2024-09-21T10:18:33.474971+0000","last_unstale":"2024-09-21T10:22:39.373931+0000","last_undegraded":"2024-09-21T10:22:39.373931+0000","last_fullsized":"2024-09-21T10:22:39.373931+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:39:40.179668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":492,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.548631+0000","last_change":"2024-09-21T10:18:33.475179+0000","last_active":"2024-09-21T10:22:51.548631+0000","last_peered":"2024-09-21T10:22:51.548631+0000","last_clean":"2024-09-21T10:22:51.548631+0000","last_became_active":"2024-09-21T10:18:33.474914+0000","last_became_peered":"2024-09-21T10:18:33.474914+0000","last_unstale":"2024-09-21T10:22:51.548631+0000","last_undegraded":"2024-09-21T10:22:51.548631+0000","last_fullsized":"2024-09-21T10:22:51.548631+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:15:32.186986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":188,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.378949+0000","last_change":"2024-09-21T10:08:00.385671+0000","last_active":"2024-09-21T10:22:39.378949+0000","last_peered":"2024-09-21T10:22:39.378949+0000","last_clean":"2024-09-21T10:22:39.378949+0000","last_became_active":"2024-09-21T10:08:00.385368+0000","last_became_peered":"2024-09-21T10:08:00.385368+0000","last_unstale":"2024-09-21T10:22:39.378949+0000","last_undegraded":"2024-09-21T10:22:39.378949+0000","last_fullsized":"2024-09-21T10:22:39.378949+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:21.799149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":139,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.548734+0000","last_change":"2024-09-21T10:18:33.481890+0000","last_active":"2024-09-21T10:22:51.548734+0000","last_peered":"2024-09-21T10:22:51.548734+0000","last_clean":"2024-09-21T10:22:51.548734+0000","last_became_active":"2024-09-21T10:18:33.480639+0000","last_became_peered":"2024-09-21T10:18:33.480639+0000","last_unstale":"2024-09-21T10:22:51.548734+0000","last_undegraded":"2024-09-21T10:22:51.548734+0000","last_fullsized":"2024-09-21T10:22:51.548734+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:49:44.797540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":280,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.378558+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:22:39.378558+0000","last_peered":"2024-09-21T10:22:39.378558+0000","last_clean":"2024-09-21T10:22:39.378558+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:22:39.378558+0000","last_undegraded":"2024-09-21T10:22:39.378558+0000","last_fullsized":"2024-09-21T10:22:39.378558+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"308'3477","reported_seq":3504,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:58.082851+0000","last_change":"2024-09-21T10:22:19.108552+0000","last_active":"2024-09-21T10:22:58.082851+0000","last_peered":"2024-09-21T10:22:58.082851+0000","last_clean":"2024-09-21T10:22:58.082851+0000","last_became_active":"2024-09-21T10:22:19.107377+0000","last_became_peered":"2024-09-21T10:22:19.107377+0000","last_unstale":"2024-09-21T10:22:58.082851+0000","last_undegraded":"2024-09-21T10:22:58.082851+0000","last_fullsized":"2024-09-21T10:22:58.082851+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":3477,"log_dups_size":0,"ondisk_log_size":3477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:39.289538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227868672,"num_objects":3477,"num_object_clones":0,"num_object_copies":10431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3477,"num_write_kb":222528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":315,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.377162+0000","last_change":"2024-09-21T10:08:00.392499+0000","last_active":"2024-09-21T10:22:39.377162+0000","last_peered":"2024-09-21T10:22:39.377162+0000","last_clean":"2024-09-21T10:22:39.377162+0000","last_became_active":"2024-09-21T10:08:00.379165+0000","last_became_peered":"2024-09-21T10:08:00.379165+0000","last_unstale":"2024-09-21T10:22:39.377162+0000","last_undegraded":"2024-09-21T10:22:39.377162+0000","last_fullsized":"2024-09-21T10:22:39.377162+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:18.933144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":171,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.373786+0000","last_change":"2024-09-21T10:07:43.794677+0000","last_active":"2024-09-21T10:22:39.373786+0000","last_peered":"2024-09-21T10:22:39.373786+0000","last_clean":"2024-09-21T10:22:39.373786+0000","last_became_active":"2024-09-21T10:07:43.794348+0000","last_became_peered":"2024-09-21T10:07:43.794348+0000","last_unstale":"2024-09-21T10:22:39.373786+0000","last_undegraded":"2024-09-21T10:22:39.373786+0000","last_fullsized":"2024-09-21T10:22:39.373786+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:11.159014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"308'3243","reported_seq":3270,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:56.913189+0000","last_change":"2024-09-21T10:22:18.651045+0000","last_active":"2024-09-21T10:22:56.913189+0000","last_peered":"2024-09-21T10:22:56.913189+0000","last_clean":"2024-09-21T10:22:56.913189+0000","last_became_active":"2024-09-21T10:22:18.650806+0000","last_became_peered":"2024-09-21T10:22:18.650806+0000","last_unstale":"2024-09-21T10:22:56.913189+0000","last_undegraded":"2024-09-21T10:22:56.913189+0000","last_fullsized":"2024-09-21T10:22:56.913189+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":3243,"log_dups_size":0,"ondisk_log_size":3243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:45:21.716041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212533248,"num_objects":3243,"num_object_clones":0,"num_object_copies":9729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3243,"num_write_kb":207552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":316,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.376992+0000","last_change":"2024-09-21T10:08:00.412449+0000","last_active":"2024-09-21T10:22:39.376992+0000","last_peered":"2024-09-21T10:22:39.376992+0000","last_clean":"2024-09-21T10:22:39.376992+0000","last_became_active":"2024-09-21T10:08:00.386530+0000","last_became_peered":"2024-09-21T10:08:00.386530+0000","last_unstale":"2024-09-21T10:22:39.376992+0000","last_undegraded":"2024-09-21T10:22:39.376992+0000","last_fullsized":"2024-09-21T10:22:39.376992+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:12:50.581176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":418,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.546082+0000","last_change":"2024-09-21T10:18:33.479057+0000","last_active":"2024-09-21T10:22:51.546082+0000","last_peered":"2024-09-21T10:22:51.546082+0000","last_clean":"2024-09-21T10:22:51.546082+0000","last_became_active":"2024-09-21T10:18:33.478752+0000","last_became_peered":"2024-09-21T10:18:33.478752+0000","last_unstale":"2024-09-21T10:22:51.546082+0000","last_undegraded":"2024-09-21T10:22:51.546082+0000","last_fullsized":"2024-09-21T10:22:51.546082+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:30.068585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"308'3480","reported_seq":3496,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:58.550630+0000","last_change":"2024-09-21T10:22:19.107933+0000","last_active":"2024-09-21T10:22:58.550630+0000","last_peered":"2024-09-21T10:22:58.550630+0000","last_clean":"2024-09-21T10:22:58.550630+0000","last_became_active":"2024-09-21T10:22:19.107577+0000","last_became_peered":"2024-09-21T10:22:19.107577+0000","last_unstale":"2024-09-21T10:22:58.550630+0000","last_undegraded":"2024-09-21T10:22:58.550630+0000","last_fullsized":"2024-09-21T10:22:58.550630+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":3480,"log_dups_size":0,"ondisk_log_size":3480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:39:34.494252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228065280,"num_objects":3480,"num_object_clones":0,"num_object_copies":10440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3480,"num_write_kb":222720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":256,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.375708+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:22:39.375708+0000","last_peered":"2024-09-21T10:22:39.375708+0000","last_clean":"2024-09-21T10:22:39.375708+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:22:39.375708+0000","last_undegraded":"2024-09-21T10:22:39.375708+0000","last_fullsized":"2024-09-21T10:22:39.375708+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":271,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.378218+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:22:39.378218+0000","last_peered":"2024-09-21T10:22:39.378218+0000","last_clean":"2024-09-21T10:22:39.378218+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:22:39.378218+0000","last_undegraded":"2024-09-21T10:22:39.378218+0000","last_fullsized":"2024-09-21T10:22:39.378218+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"308'3623","reported_seq":3649,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:59.624856+0000","last_change":"2024-09-21T10:22:19.106238+0000","last_active":"2024-09-21T10:22:59.624856+0000","last_peered":"2024-09-21T10:22:59.624856+0000","last_clean":"2024-09-21T10:22:59.624856+0000","last_became_active":"2024-09-21T10:22:19.105893+0000","last_became_peered":"2024-09-21T10:22:19.105893+0000","last_unstale":"2024-09-21T10:22:59.624856+0000","last_undegraded":"2024-09-21T10:22:59.624856+0000","last_fullsized":"2024-09-21T10:22:59.624856+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":3623,"log_dups_size":0,"ondisk_log_size":3623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:36:41.147519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237436928,"num_objects":3623,"num_object_clones":0,"num_object_copies":10869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3623,"num_write_kb":231872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":106,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.378936+0000","last_change":"2024-09-21T10:18:33.483457+0000","last_active":"2024-09-21T10:22:39.378936+0000","last_peered":"2024-09-21T10:22:39.378936+0000","last_clean":"2024-09-21T10:22:39.378936+0000","last_became_active":"2024-09-21T10:18:33.483140+0000","last_became_peered":"2024-09-21T10:18:33.483140+0000","last_unstale":"2024-09-21T10:22:39.378936+0000","last_undegraded":"2024-09-21T10:22:39.378936+0000","last_fullsized":"2024-09-21T10:22:39.378936+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:37:39.302566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":514,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.547913+0000","last_change":"2024-09-21T10:18:33.483011+0000","last_active":"2024-09-21T10:22:51.547913+0000","last_peered":"2024-09-21T10:22:51.547913+0000","last_clean":"2024-09-21T10:22:51.547913+0000","last_became_active":"2024-09-21T10:18:33.482712+0000","last_became_peered":"2024-09-21T10:18:33.482712+0000","last_unstale":"2024-09-21T10:22:51.547913+0000","last_undegraded":"2024-09-21T10:22:51.547913+0000","last_fullsized":"2024-09-21T10:22:51.547913+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:05:11.037442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"308'3490","reported_seq":3506,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:58.599961+0000","last_change":"2024-09-21T10:22:18.648413+0000","last_active":"2024-09-21T10:22:58.599961+0000","last_peered":"2024-09-21T10:22:58.599961+0000","last_clean":"2024-09-21T10:22:58.599961+0000","last_became_active":"2024-09-21T10:22:18.648152+0000","last_became_peered":"2024-09-21T10:22:18.648152+0000","last_unstale":"2024-09-21T10:22:58.599961+0000","last_undegraded":"2024-09-21T10:22:58.599961+0000","last_fullsized":"2024-09-21T10:22:58.599961+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":3490,"log_dups_size":0,"ondisk_log_size":3490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:44.431340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228720640,"num_objects":3490,"num_object_clones":0,"num_object_copies":10470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3490,"num_write_kb":223360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":272,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.549054+0000","last_change":"2024-09-21T10:18:33.488794+0000","last_active":"2024-09-21T10:22:51.549054+0000","last_peered":"2024-09-21T10:22:51.549054+0000","last_clean":"2024-09-21T10:22:51.549054+0000","last_became_active":"2024-09-21T10:18:33.479048+0000","last_became_peered":"2024-09-21T10:18:33.479048+0000","last_unstale":"2024-09-21T10:22:51.549054+0000","last_undegraded":"2024-09-21T10:22:51.549054+0000","last_fullsized":"2024-09-21T10:22:51.549054+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:03.387788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":125,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.378852+0000","last_change":"2024-09-21T10:18:33.475977+0000","last_active":"2024-09-21T10:22:39.378852+0000","last_peered":"2024-09-21T10:22:39.378852+0000","last_clean":"2024-09-21T10:22:39.378852+0000","last_became_active":"2024-09-21T10:18:33.475590+0000","last_became_peered":"2024-09-21T10:18:33.475590+0000","last_unstale":"2024-09-21T10:22:39.378852+0000","last_undegraded":"2024-09-21T10:22:39.378852+0000","last_fullsized":"2024-09-21T10:22:39.378852+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:54:57.304453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"308'3303","reported_seq":3330,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:56.941346+0000","last_change":"2024-09-21T10:22:18.649410+0000","last_active":"2024-09-21T10:22:56.941346+0000","last_peered":"2024-09-21T10:22:56.941346+0000","last_clean":"2024-09-21T10:22:56.941346+0000","last_became_active":"2024-09-21T10:22:18.648626+0000","last_became_peered":"2024-09-21T10:22:18.648626+0000","last_unstale":"2024-09-21T10:22:56.941346+0000","last_undegraded":"2024-09-21T10:22:56.941346+0000","last_fullsized":"2024-09-21T10:22:56.941346+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":3303,"log_dups_size":0,"ondisk_log_size":3303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:42:50.134303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216465408,"num_objects":3303,"num_object_clones":0,"num_object_copies":9909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3303,"num_write_kb":211392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":312,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.378334+0000","last_change":"2024-09-21T10:18:33.473849+0000","last_active":"2024-09-21T10:22:39.378334+0000","last_peered":"2024-09-21T10:22:39.378334+0000","last_clean":"2024-09-21T10:22:39.378334+0000","last_became_active":"2024-09-21T10:18:33.473037+0000","last_became_peered":"2024-09-21T10:18:33.473037+0000","last_unstale":"2024-09-21T10:22:39.378334+0000","last_undegraded":"2024-09-21T10:22:39.378334+0000","last_fullsized":"2024-09-21T10:22:39.378334+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:03:16.322667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":419,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.555645+0000","last_change":"2024-09-21T10:08:00.387420+0000","last_active":"2024-09-21T10:22:51.555645+0000","last_peered":"2024-09-21T10:22:51.555645+0000","last_clean":"2024-09-21T10:22:51.555645+0000","last_became_active":"2024-09-21T10:08:00.386843+0000","last_became_peered":"2024-09-21T10:08:00.386843+0000","last_unstale":"2024-09-21T10:22:51.555645+0000","last_undegraded":"2024-09-21T10:22:51.555645+0000","last_fullsized":"2024-09-21T10:22:51.555645+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:51:00.654836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3","version":"308'3465","reported_seq":3481,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:58.557721+0000","last_change":"2024-09-21T10:22:19.106259+0000","last_active":"2024-09-21T10:22:58.557721+0000","last_peered":"2024-09-21T10:22:58.557721+0000","last_clean":"2024-09-21T10:22:58.557721+0000","last_became_active":"2024-09-21T10:22:19.105931+0000","last_became_peered":"2024-09-21T10:22:19.105931+0000","last_unstale":"2024-09-21T10:22:58.557721+0000","last_undegraded":"2024-09-21T10:22:58.557721+0000","last_fullsized":"2024-09-21T10:22:58.557721+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":3465,"log_dups_size":0,"ondisk_log_size":3465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:38:07.993075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227082240,"num_objects":3465,"num_object_clones":0,"num_object_copies":10395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3465,"num_write_kb":221760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":403,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.550874+0000","last_change":"2024-09-21T10:08:00.382496+0000","last_active":"2024-09-21T10:22:51.550874+0000","last_peered":"2024-09-21T10:22:51.550874+0000","last_clean":"2024-09-21T10:22:51.550874+0000","last_became_active":"2024-09-21T10:08:00.382180+0000","last_became_peered":"2024-09-21T10:08:00.382180+0000","last_unstale":"2024-09-21T10:22:51.550874+0000","last_undegraded":"2024-09-21T10:22:51.550874+0000","last_fullsized":"2024-09-21T10:22:51.550874+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:16.074736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":241,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.379102+0000","last_change":"2024-09-21T10:08:38.906801+0000","last_active":"2024-09-21T10:22:39.379102+0000","last_peered":"2024-09-21T10:22:39.379102+0000","last_clean":"2024-09-21T10:22:39.379102+0000","last_became_active":"2024-09-21T10:08:38.906487+0000","last_became_peered":"2024-09-21T10:08:38.906487+0000","last_unstale":"2024-09-21T10:22:39.379102+0000","last_undegraded":"2024-09-21T10:22:39.379102+0000","last_fullsized":"2024-09-21T10:22:39.379102+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:31.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"308'3342","reported_seq":3369,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:56.910275+0000","last_change":"2024-09-21T10:22:18.649478+0000","last_active":"2024-09-21T10:22:56.910275+0000","last_peered":"2024-09-21T10:22:56.910275+0000","last_clean":"2024-09-21T10:22:56.910275+0000","last_became_active":"2024-09-21T10:22:18.649167+0000","last_became_peered":"2024-09-21T10:22:18.649167+0000","last_unstale":"2024-09-21T10:22:56.910275+0000","last_undegraded":"2024-09-21T10:22:56.910275+0000","last_fullsized":"2024-09-21T10:22:56.910275+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":3342,"log_dups_size":0,"ondisk_log_size":3342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:17.191131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219021312,"num_objects":3342,"num_object_clones":0,"num_object_copies":10026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3342,"num_write_kb":213888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":139,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.548923+0000","last_change":"2024-09-21T10:18:33.484568+0000","last_active":"2024-09-21T10:22:51.548923+0000","last_peered":"2024-09-21T10:22:51.548923+0000","last_clean":"2024-09-21T10:22:51.548923+0000","last_became_active":"2024-09-21T10:18:33.483975+0000","last_became_peered":"2024-09-21T10:18:33.483975+0000","last_unstale":"2024-09-21T10:22:51.548923+0000","last_undegraded":"2024-09-21T10:22:51.548923+0000","last_fullsized":"2024-09-21T10:22:51.548923+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:13.276318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":502,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.548825+0000","last_change":"2024-09-21T10:18:33.474464+0000","last_active":"2024-09-21T10:22:51.548825+0000","last_peered":"2024-09-21T10:22:51.548825+0000","last_clean":"2024-09-21T10:22:51.548825+0000","last_became_active":"2024-09-21T10:18:33.474128+0000","last_became_peered":"2024-09-21T10:18:33.474128+0000","last_unstale":"2024-09-21T10:22:51.548825+0000","last_undegraded":"2024-09-21T10:22:51.548825+0000","last_fullsized":"2024-09-21T10:22:51.548825+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:32:49.409409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"308'3320","reported_seq":3347,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:58.086165+0000","last_change":"2024-09-21T10:22:19.107370+0000","last_active":"2024-09-21T10:22:58.086165+0000","last_peered":"2024-09-21T10:22:58.086165+0000","last_clean":"2024-09-21T10:22:58.086165+0000","last_became_active":"2024-09-21T10:22:19.106979+0000","last_became_peered":"2024-09-21T10:22:19.106979+0000","last_unstale":"2024-09-21T10:22:58.086165+0000","last_undegraded":"2024-09-21T10:22:58.086165+0000","last_fullsized":"2024-09-21T10:22:58.086165+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":3320,"log_dups_size":0,"ondisk_log_size":3320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:10.360218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217579520,"num_objects":3320,"num_object_clones":0,"num_object_copies":9960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3320,"num_write_kb":212480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":389,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.373921+0000","last_change":"2024-09-21T09:59:54.265556+0000","last_active":"2024-09-21T10:22:39.373921+0000","last_peered":"2024-09-21T10:22:39.373921+0000","last_clean":"2024-09-21T10:22:39.373921+0000","last_became_active":"2024-09-21T09:59:54.265033+0000","last_became_peered":"2024-09-21T09:59:54.265033+0000","last_unstale":"2024-09-21T10:22:39.373921+0000","last_undegraded":"2024-09-21T10:22:39.373921+0000","last_fullsized":"2024-09-21T10:22:39.373921+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T09:48:34.850872+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:40.449756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":783,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.551162+0000","last_change":"2024-09-21T10:07:43.790490+0000","last_active":"2024-09-21T10:22:51.551162+0000","last_peered":"2024-09-21T10:22:51.551162+0000","last_clean":"2024-09-21T10:22:51.551162+0000","last_became_active":"2024-09-21T10:07:43.789986+0000","last_became_peered":"2024-09-21T10:07:43.789986+0000","last_unstale":"2024-09-21T10:22:51.551162+0000","last_undegraded":"2024-09-21T10:22:51.551162+0000","last_fullsized":"2024-09-21T10:22:51.551162+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:08.740356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":481,"num_read_kb":1580,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":59,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.378115+0000","last_change":"2024-09-21T10:18:33.473676+0000","last_active":"2024-09-21T10:22:39.378115+0000","last_peered":"2024-09-21T10:22:39.378115+0000","last_clean":"2024-09-21T10:22:39.378115+0000","last_became_active":"2024-09-21T10:18:33.472593+0000","last_became_peered":"2024-09-21T10:18:33.472593+0000","last_unstale":"2024-09-21T10:22:39.378115+0000","last_undegraded":"2024-09-21T10:22:39.378115+0000","last_fullsized":"2024-09-21T10:22:39.378115+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:07:55.816740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":284,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.379040+0000","last_change":"2024-09-21T10:08:00.386135+0000","last_active":"2024-09-21T10:22:39.379040+0000","last_peered":"2024-09-21T10:22:39.379040+0000","last_clean":"2024-09-21T10:22:39.379040+0000","last_became_active":"2024-09-21T10:08:00.385796+0000","last_became_peered":"2024-09-21T10:08:00.385796+0000","last_unstale":"2024-09-21T10:22:39.379040+0000","last_undegraded":"2024-09-21T10:22:39.379040+0000","last_fullsized":"2024-09-21T10:22:39.379040+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:27.054699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"308'3353","reported_seq":3380,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:58.110188+0000","last_change":"2024-09-21T10:22:19.108211+0000","last_active":"2024-09-21T10:22:58.110188+0000","last_peered":"2024-09-21T10:22:58.110188+0000","last_clean":"2024-09-21T10:22:58.110188+0000","last_became_active":"2024-09-21T10:22:19.107919+0000","last_became_peered":"2024-09-21T10:22:19.107919+0000","last_unstale":"2024-09-21T10:22:58.110188+0000","last_undegraded":"2024-09-21T10:22:58.110188+0000","last_fullsized":"2024-09-21T10:22:58.110188+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":3353,"log_dups_size":0,"ondisk_log_size":3353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:20.144889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219742208,"num_objects":3353,"num_object_clones":0,"num_object_copies":10059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3353,"num_write_kb":214592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":652,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.373825+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:22:39.373825+0000","last_peered":"2024-09-21T10:22:39.373825+0000","last_clean":"2024-09-21T10:22:39.373825+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:22:39.373825+0000","last_undegraded":"2024-09-21T10:22:39.373825+0000","last_fullsized":"2024-09-21T10:22:39.373825+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":515,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.549134+0000","last_change":"2024-09-21T10:18:33.471355+0000","last_active":"2024-09-21T10:22:51.549134+0000","last_peered":"2024-09-21T10:22:51.549134+0000","last_clean":"2024-09-21T10:22:51.549134+0000","last_became_active":"2024-09-21T10:18:33.470962+0000","last_became_peered":"2024-09-21T10:18:33.470962+0000","last_unstale":"2024-09-21T10:22:51.549134+0000","last_undegraded":"2024-09-21T10:22:51.549134+0000","last_fullsized":"2024-09-21T10:22:51.549134+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:53:57.516543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":497,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.548859+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:22:51.548859+0000","last_peered":"2024-09-21T10:22:51.548859+0000","last_clean":"2024-09-21T10:22:51.548859+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:22:51.548859+0000","last_undegraded":"2024-09-21T10:22:51.548859+0000","last_fullsized":"2024-09-21T10:22:51.548859+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"308'3265","reported_seq":3292,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:56.910201+0000","last_change":"2024-09-21T10:22:19.108927+0000","last_active":"2024-09-21T10:22:56.910201+0000","last_peered":"2024-09-21T10:22:56.910201+0000","last_clean":"2024-09-21T10:22:56.910201+0000","last_became_active":"2024-09-21T10:22:19.108649+0000","last_became_peered":"2024-09-21T10:22:19.108649+0000","last_unstale":"2024-09-21T10:22:56.910201+0000","last_undegraded":"2024-09-21T10:22:56.910201+0000","last_fullsized":"2024-09-21T10:22:56.910201+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":3265,"log_dups_size":0,"ondisk_log_size":3265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:02:10.256858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213975040,"num_objects":3265,"num_object_clones":0,"num_object_copies":9795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3265,"num_write_kb":208960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":1271,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.551068+0000","last_change":"2024-09-21T10:18:36.495308+0000","last_active":"2024-09-21T10:22:51.551068+0000","last_peered":"2024-09-21T10:22:51.551068+0000","last_clean":"2024-09-21T10:22:51.551068+0000","last_became_active":"2024-09-21T10:18:36.494662+0000","last_became_peered":"2024-09-21T10:18:36.494662+0000","last_unstale":"2024-09-21T10:22:51.551068+0000","last_undegraded":"2024-09-21T10:22:51.551068+0000","last_fullsized":"2024-09-21T10:22:51.551068+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:49.218676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":369,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.549008+0000","last_change":"2024-09-21T10:08:18.470748+0000","last_active":"2024-09-21T10:22:51.549008+0000","last_peered":"2024-09-21T10:22:51.549008+0000","last_clean":"2024-09-21T10:22:51.549008+0000","last_became_active":"2024-09-21T10:08:18.470227+0000","last_became_peered":"2024-09-21T10:08:18.470227+0000","last_unstale":"2024-09-21T10:22:51.549008+0000","last_undegraded":"2024-09-21T10:22:51.549008+0000","last_fullsized":"2024-09-21T10:22:51.549008+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:16.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":491,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.555949+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:22:51.555949+0000","last_peered":"2024-09-21T10:22:51.555949+0000","last_clean":"2024-09-21T10:22:51.555949+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:22:51.555949+0000","last_undegraded":"2024-09-21T10:22:51.555949+0000","last_fullsized":"2024-09-21T10:22:51.555949+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.9","version":"308'3313","reported_seq":3340,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:58.103024+0000","last_change":"2024-09-21T10:22:19.106312+0000","last_active":"2024-09-21T10:22:58.103024+0000","last_peered":"2024-09-21T10:22:58.103024+0000","last_clean":"2024-09-21T10:22:58.103024+0000","last_became_active":"2024-09-21T10:22:19.106002+0000","last_became_peered":"2024-09-21T10:22:19.106002+0000","last_unstale":"2024-09-21T10:22:58.103024+0000","last_undegraded":"2024-09-21T10:22:58.103024+0000","last_fullsized":"2024-09-21T10:22:58.103024+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":3313,"log_dups_size":0,"ondisk_log_size":3313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:24.317806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217120768,"num_objects":3313,"num_object_clones":0,"num_object_copies":9939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3313,"num_write_kb":212032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":968,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.549303+0000","last_change":"2024-09-21T10:07:43.794767+0000","last_active":"2024-09-21T10:22:51.549303+0000","last_peered":"2024-09-21T10:22:51.549303+0000","last_clean":"2024-09-21T10:22:51.549303+0000","last_became_active":"2024-09-21T10:07:43.794062+0000","last_became_peered":"2024-09-21T10:07:43.794062+0000","last_unstale":"2024-09-21T10:22:51.549303+0000","last_undegraded":"2024-09-21T10:22:51.549303+0000","last_fullsized":"2024-09-21T10:22:51.549303+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:51.977955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":394,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.547356+0000","last_change":"2024-09-21T10:08:00.389176+0000","last_active":"2024-09-21T10:22:51.547356+0000","last_peered":"2024-09-21T10:22:51.547356+0000","last_clean":"2024-09-21T10:22:51.547356+0000","last_became_active":"2024-09-21T10:08:00.383854+0000","last_became_peered":"2024-09-21T10:08:00.383854+0000","last_unstale":"2024-09-21T10:22:51.547356+0000","last_undegraded":"2024-09-21T10:22:51.547356+0000","last_fullsized":"2024-09-21T10:22:51.547356+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:46.952216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":415,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.552463+0000","last_change":"2024-09-21T10:18:33.465815+0000","last_active":"2024-09-21T10:22:51.552463+0000","last_peered":"2024-09-21T10:22:51.552463+0000","last_clean":"2024-09-21T10:22:51.552463+0000","last_became_active":"2024-09-21T10:18:33.465459+0000","last_became_peered":"2024-09-21T10:18:33.465459+0000","last_unstale":"2024-09-21T10:22:51.552463+0000","last_undegraded":"2024-09-21T10:22:51.552463+0000","last_fullsized":"2024-09-21T10:22:51.552463+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:22:08.199800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.f","version":"308'3283","reported_seq":3310,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:56.930549+0000","last_change":"2024-09-21T10:22:19.107714+0000","last_active":"2024-09-21T10:22:56.930549+0000","last_peered":"2024-09-21T10:22:56.930549+0000","last_clean":"2024-09-21T10:22:56.930549+0000","last_became_active":"2024-09-21T10:22:19.107130+0000","last_became_peered":"2024-09-21T10:22:19.107130+0000","last_unstale":"2024-09-21T10:22:56.930549+0000","last_undegraded":"2024-09-21T10:22:56.930549+0000","last_fullsized":"2024-09-21T10:22:56.930549+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":3283,"log_dups_size":0,"ondisk_log_size":3283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:50:53.512892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215154688,"num_objects":3283,"num_object_clones":0,"num_object_copies":9849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3283,"num_write_kb":210112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":158,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.555520+0000","last_change":"2024-09-21T10:18:33.467659+0000","last_active":"2024-09-21T10:22:51.555520+0000","last_peered":"2024-09-21T10:22:51.555520+0000","last_clean":"2024-09-21T10:22:51.555520+0000","last_became_active":"2024-09-21T10:18:33.467369+0000","last_became_peered":"2024-09-21T10:18:33.467369+0000","last_unstale":"2024-09-21T10:22:51.555520+0000","last_undegraded":"2024-09-21T10:22:51.555520+0000","last_fullsized":"2024-09-21T10:22:51.555520+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:16:41.951755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":1075,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.548724+0000","last_change":"2024-09-21T10:08:18.468995+0000","last_active":"2024-09-21T10:22:51.548724+0000","last_peered":"2024-09-21T10:22:51.548724+0000","last_clean":"2024-09-21T10:22:51.548724+0000","last_became_active":"2024-09-21T10:08:18.468243+0000","last_became_peered":"2024-09-21T10:08:18.468243+0000","last_unstale":"2024-09-21T10:22:51.548724+0000","last_undegraded":"2024-09-21T10:22:51.548724+0000","last_fullsized":"2024-09-21T10:22:51.548724+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:45.520968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":394,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.551598+0000","last_change":"2024-09-21T10:08:18.468595+0000","last_active":"2024-09-21T10:22:51.551598+0000","last_peered":"2024-09-21T10:22:51.551598+0000","last_clean":"2024-09-21T10:22:51.551598+0000","last_became_active":"2024-09-21T10:08:18.468296+0000","last_became_peered":"2024-09-21T10:08:18.468296+0000","last_unstale":"2024-09-21T10:22:51.551598+0000","last_undegraded":"2024-09-21T10:22:51.551598+0000","last_fullsized":"2024-09-21T10:22:51.551598+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:42.748130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"308'3353","reported_seq":3369,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:58.585832+0000","last_change":"2024-09-21T10:22:18.649813+0000","last_active":"2024-09-21T10:22:58.585832+0000","last_peered":"2024-09-21T10:22:58.585832+0000","last_clean":"2024-09-21T10:22:58.585832+0000","last_became_active":"2024-09-21T10:22:18.649480+0000","last_became_peered":"2024-09-21T10:22:18.649480+0000","last_unstale":"2024-09-21T10:22:58.585832+0000","last_undegraded":"2024-09-21T10:22:58.585832+0000","last_fullsized":"2024-09-21T10:22:58.585832+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":3353,"log_dups_size":0,"ondisk_log_size":3353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:36:28.426420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219742208,"num_objects":3353,"num_object_clones":0,"num_object_copies":10059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3353,"num_write_kb":214592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":370,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.548631+0000","last_change":"2024-09-21T10:08:18.469907+0000","last_active":"2024-09-21T10:22:51.548631+0000","last_peered":"2024-09-21T10:22:51.548631+0000","last_clean":"2024-09-21T10:22:51.548631+0000","last_became_active":"2024-09-21T10:08:18.469443+0000","last_became_peered":"2024-09-21T10:08:18.469443+0000","last_unstale":"2024-09-21T10:22:51.548631+0000","last_undegraded":"2024-09-21T10:22:51.548631+0000","last_fullsized":"2024-09-21T10:22:51.548631+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:15.645803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":702,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.375781+0000","last_change":"2024-09-21T10:08:00.419641+0000","last_active":"2024-09-21T10:22:39.375781+0000","last_peered":"2024-09-21T10:22:39.375781+0000","last_clean":"2024-09-21T10:22:39.375781+0000","last_became_active":"2024-09-21T10:08:00.387050+0000","last_became_peered":"2024-09-21T10:08:00.387050+0000","last_unstale":"2024-09-21T10:22:39.375781+0000","last_undegraded":"2024-09-21T10:22:39.375781+0000","last_fullsized":"2024-09-21T10:22:39.375781+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:00:32.881619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":476,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.547624+0000","last_change":"2024-09-21T09:59:54.267798+0000","last_active":"2024-09-21T10:22:51.547624+0000","last_peered":"2024-09-21T10:22:51.547624+0000","last_clean":"2024-09-21T10:22:51.547624+0000","last_became_active":"2024-09-21T09:59:54.265615+0000","last_became_peered":"2024-09-21T09:59:54.265615+0000","last_unstale":"2024-09-21T10:22:51.547624+0000","last_undegraded":"2024-09-21T10:22:51.547624+0000","last_fullsized":"2024-09-21T10:22:51.547624+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:30.363872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"308'3432","reported_seq":3448,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:58.586105+0000","last_change":"2024-09-21T10:22:19.106548+0000","last_active":"2024-09-21T10:22:58.586105+0000","last_peered":"2024-09-21T10:22:58.586105+0000","last_clean":"2024-09-21T10:22:58.586105+0000","last_became_active":"2024-09-21T10:22:19.105920+0000","last_became_peered":"2024-09-21T10:22:19.105920+0000","last_unstale":"2024-09-21T10:22:58.586105+0000","last_undegraded":"2024-09-21T10:22:58.586105+0000","last_fullsized":"2024-09-21T10:22:58.586105+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":3432,"log_dups_size":0,"ondisk_log_size":3432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:30.951700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224919552,"num_objects":3432,"num_object_clones":0,"num_object_copies":10296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3432,"num_write_kb":219648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":71,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.377070+0000","last_change":"2024-09-21T10:18:33.489678+0000","last_active":"2024-09-21T10:22:39.377070+0000","last_peered":"2024-09-21T10:22:39.377070+0000","last_clean":"2024-09-21T10:22:39.377070+0000","last_became_active":"2024-09-21T10:18:33.474818+0000","last_became_peered":"2024-09-21T10:18:33.474818+0000","last_unstale":"2024-09-21T10:22:39.377070+0000","last_undegraded":"2024-09-21T10:22:39.377070+0000","last_fullsized":"2024-09-21T10:22:39.377070+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:49.828007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":766,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.551266+0000","last_change":"2024-09-21T10:18:36.503584+0000","last_active":"2024-09-21T10:22:51.551266+0000","last_peered":"2024-09-21T10:22:51.551266+0000","last_clean":"2024-09-21T10:22:51.551266+0000","last_became_active":"2024-09-21T10:18:36.502990+0000","last_became_peered":"2024-09-21T10:18:36.502990+0000","last_unstale":"2024-09-21T10:22:51.551266+0000","last_undegraded":"2024-09-21T10:22:51.551266+0000","last_fullsized":"2024-09-21T10:22:51.551266+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:11.648271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":100,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.373702+0000","last_change":"2024-09-21T10:18:33.479008+0000","last_active":"2024-09-21T10:22:39.373702+0000","last_peered":"2024-09-21T10:22:39.373702+0000","last_clean":"2024-09-21T10:22:39.373702+0000","last_became_active":"2024-09-21T10:18:33.478740+0000","last_became_peered":"2024-09-21T10:18:33.478740+0000","last_unstale":"2024-09-21T10:22:39.373702+0000","last_undegraded":"2024-09-21T10:22:39.373702+0000","last_fullsized":"2024-09-21T10:22:39.373702+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:16:52.594882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"308'3447","reported_seq":3463,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:58.584132+0000","last_change":"2024-09-21T10:22:18.649356+0000","last_active":"2024-09-21T10:22:58.584132+0000","last_peered":"2024-09-21T10:22:58.584132+0000","last_clean":"2024-09-21T10:22:58.584132+0000","last_became_active":"2024-09-21T10:22:18.649035+0000","last_became_peered":"2024-09-21T10:22:18.649035+0000","last_unstale":"2024-09-21T10:22:58.584132+0000","last_undegraded":"2024-09-21T10:22:58.584132+0000","last_fullsized":"2024-09-21T10:22:58.584132+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":3447,"log_dups_size":0,"ondisk_log_size":3447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:17:20.665367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225902592,"num_objects":3447,"num_object_clones":0,"num_object_copies":10341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3447,"num_write_kb":220608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":523,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.548527+0000","last_change":"2024-09-21T10:08:00.390436+0000","last_active":"2024-09-21T10:22:51.548527+0000","last_peered":"2024-09-21T10:22:51.548527+0000","last_clean":"2024-09-21T10:22:51.548527+0000","last_became_active":"2024-09-21T10:08:00.381835+0000","last_became_peered":"2024-09-21T10:08:00.381835+0000","last_unstale":"2024-09-21T10:22:51.548527+0000","last_undegraded":"2024-09-21T10:22:51.548527+0000","last_fullsized":"2024-09-21T10:22:51.548527+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:01:35.697356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":534,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.379210+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:22:39.379210+0000","last_peered":"2024-09-21T10:22:39.379210+0000","last_clean":"2024-09-21T10:22:39.379210+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:22:39.379210+0000","last_undegraded":"2024-09-21T10:22:39.379210+0000","last_fullsized":"2024-09-21T10:22:39.379210+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":172,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.378950+0000","last_change":"2024-09-21T10:08:00.385774+0000","last_active":"2024-09-21T10:22:39.378950+0000","last_peered":"2024-09-21T10:22:39.378950+0000","last_clean":"2024-09-21T10:22:39.378950+0000","last_became_active":"2024-09-21T10:08:00.385445+0000","last_became_peered":"2024-09-21T10:08:00.385445+0000","last_unstale":"2024-09-21T10:22:39.378950+0000","last_undegraded":"2024-09-21T10:22:39.378950+0000","last_fullsized":"2024-09-21T10:22:39.378950+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:42.442190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":231,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.378338+0000","last_change":"2024-09-21T10:08:00.379661+0000","last_active":"2024-09-21T10:22:39.378338+0000","last_peered":"2024-09-21T10:22:39.378338+0000","last_clean":"2024-09-21T10:22:39.378338+0000","last_became_active":"2024-09-21T10:08:00.379301+0000","last_became_peered":"2024-09-21T10:08:00.379301+0000","last_unstale":"2024-09-21T10:22:39.378338+0000","last_undegraded":"2024-09-21T10:22:39.378338+0000","last_fullsized":"2024-09-21T10:22:39.378338+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:15.649523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":254,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.374113+0000","last_change":"2024-09-21T10:08:00.410731+0000","last_active":"2024-09-21T10:22:39.374113+0000","last_peered":"2024-09-21T10:22:39.374113+0000","last_clean":"2024-09-21T10:22:39.374113+0000","last_became_active":"2024-09-21T10:08:00.380324+0000","last_became_peered":"2024-09-21T10:08:00.380324+0000","last_unstale":"2024-09-21T10:22:39.374113+0000","last_undegraded":"2024-09-21T10:22:39.374113+0000","last_fullsized":"2024-09-21T10:22:39.374113+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:47:34.960221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":369,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.551515+0000","last_change":"2024-09-21T10:08:18.470378+0000","last_active":"2024-09-21T10:22:51.551515+0000","last_peered":"2024-09-21T10:22:51.551515+0000","last_clean":"2024-09-21T10:22:51.551515+0000","last_became_active":"2024-09-21T10:08:18.469482+0000","last_became_peered":"2024-09-21T10:08:18.469482+0000","last_unstale":"2024-09-21T10:22:51.551515+0000","last_undegraded":"2024-09-21T10:22:51.551515+0000","last_fullsized":"2024-09-21T10:22:51.551515+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:35.628724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":339,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.374645+0000","last_change":"2024-09-21T10:18:33.479236+0000","last_active":"2024-09-21T10:22:39.374645+0000","last_peered":"2024-09-21T10:22:39.374645+0000","last_clean":"2024-09-21T10:22:39.374645+0000","last_became_active":"2024-09-21T10:18:33.474256+0000","last_became_peered":"2024-09-21T10:18:33.474256+0000","last_unstale":"2024-09-21T10:22:39.374645+0000","last_undegraded":"2024-09-21T10:22:39.374645+0000","last_fullsized":"2024-09-21T10:22:39.374645+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:19:34.666035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.552346+0000","last_change":"2024-09-21T10:22:39.385291+0000","last_active":"2024-09-21T10:22:51.552346+0000","last_peered":"2024-09-21T10:22:51.552346+0000","last_clean":"2024-09-21T10:22:51.552346+0000","last_became_active":"2024-09-21T10:22:39.384875+0000","last_became_peered":"2024-09-21T10:22:39.384875+0000","last_unstale":"2024-09-21T10:22:51.552346+0000","last_undegraded":"2024-09-21T10:22:51.552346+0000","last_fullsized":"2024-09-21T10:22:51.552346+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:33:30.610497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":60,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.378852+0000","last_change":"2024-09-21T10:18:33.464989+0000","last_active":"2024-09-21T10:22:39.378852+0000","last_peered":"2024-09-21T10:22:39.378852+0000","last_clean":"2024-09-21T10:22:39.378852+0000","last_became_active":"2024-09-21T10:18:33.464617+0000","last_became_peered":"2024-09-21T10:18:33.464617+0000","last_unstale":"2024-09-21T10:22:39.378852+0000","last_undegraded":"2024-09-21T10:22:39.378852+0000","last_fullsized":"2024-09-21T10:22:39.378852+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:09.300797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":492,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.547862+0000","last_change":"2024-09-21T10:18:33.472415+0000","last_active":"2024-09-21T10:22:51.547862+0000","last_peered":"2024-09-21T10:22:51.547862+0000","last_clean":"2024-09-21T10:22:51.547862+0000","last_became_active":"2024-09-21T10:18:33.471746+0000","last_became_peered":"2024-09-21T10:18:33.471746+0000","last_unstale":"2024-09-21T10:22:51.547862+0000","last_undegraded":"2024-09-21T10:22:51.547862+0000","last_fullsized":"2024-09-21T10:22:51.547862+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:40:37.099458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":138,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.548657+0000","last_change":"2024-09-21T10:18:33.484721+0000","last_active":"2024-09-21T10:22:51.548657+0000","last_peered":"2024-09-21T10:22:51.548657+0000","last_clean":"2024-09-21T10:22:51.548657+0000","last_became_active":"2024-09-21T10:18:33.483950+0000","last_became_peered":"2024-09-21T10:18:33.483950+0000","last_unstale":"2024-09-21T10:22:51.548657+0000","last_undegraded":"2024-09-21T10:22:51.548657+0000","last_fullsized":"2024-09-21T10:22:51.548657+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:54:52.208378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":271,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.379181+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:22:39.379181+0000","last_peered":"2024-09-21T10:22:39.379181+0000","last_clean":"2024-09-21T10:22:39.379181+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:22:39.379181+0000","last_undegraded":"2024-09-21T10:22:39.379181+0000","last_fullsized":"2024-09-21T10:22:39.379181+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":157,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.547978+0000","last_change":"2024-09-21T10:18:33.479635+0000","last_active":"2024-09-21T10:22:51.547978+0000","last_peered":"2024-09-21T10:22:51.547978+0000","last_clean":"2024-09-21T10:22:51.547978+0000","last_became_active":"2024-09-21T10:18:33.478830+0000","last_became_peered":"2024-09-21T10:18:33.478830+0000","last_unstale":"2024-09-21T10:22:51.547978+0000","last_undegraded":"2024-09-21T10:22:51.547978+0000","last_fullsized":"2024-09-21T10:22:51.547978+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:16.950086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":508,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.548777+0000","last_change":"2024-09-21T10:18:33.480350+0000","last_active":"2024-09-21T10:22:51.548777+0000","last_peered":"2024-09-21T10:22:51.548777+0000","last_clean":"2024-09-21T10:22:51.548777+0000","last_became_active":"2024-09-21T10:18:33.479248+0000","last_became_peered":"2024-09-21T10:18:33.479248+0000","last_unstale":"2024-09-21T10:22:51.548777+0000","last_undegraded":"2024-09-21T10:22:51.548777+0000","last_fullsized":"2024-09-21T10:22:51.548777+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:25:28.780572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":502,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.546260+0000","last_change":"2024-09-21T10:18:33.480341+0000","last_active":"2024-09-21T10:22:51.546260+0000","last_peered":"2024-09-21T10:22:51.546260+0000","last_clean":"2024-09-21T10:22:51.546260+0000","last_became_active":"2024-09-21T10:18:33.479869+0000","last_became_peered":"2024-09-21T10:18:33.479869+0000","last_unstale":"2024-09-21T10:22:51.546260+0000","last_undegraded":"2024-09-21T10:22:51.546260+0000","last_fullsized":"2024-09-21T10:22:51.546260+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:32:42.206910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":262,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.548057+0000","last_change":"2024-09-21T10:18:33.473981+0000","last_active":"2024-09-21T10:22:51.548057+0000","last_peered":"2024-09-21T10:22:51.548057+0000","last_clean":"2024-09-21T10:22:51.548057+0000","last_became_active":"2024-09-21T10:18:33.473619+0000","last_became_peered":"2024-09-21T10:18:33.473619+0000","last_unstale":"2024-09-21T10:22:51.548057+0000","last_undegraded":"2024-09-21T10:22:51.548057+0000","last_fullsized":"2024-09-21T10:22:51.548057+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:44:24.630856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":314,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.551635+0000","last_change":"2024-09-21T10:08:18.471282+0000","last_active":"2024-09-21T10:22:51.551635+0000","last_peered":"2024-09-21T10:22:51.551635+0000","last_clean":"2024-09-21T10:22:51.551635+0000","last_became_active":"2024-09-21T10:08:18.470952+0000","last_became_peered":"2024-09-21T10:08:18.470952+0000","last_unstale":"2024-09-21T10:22:51.551635+0000","last_undegraded":"2024-09-21T10:22:51.551635+0000","last_fullsized":"2024-09-21T10:22:51.551635+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:25.597081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":147,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.552290+0000","last_change":"2024-09-21T10:18:33.480128+0000","last_active":"2024-09-21T10:22:51.552290+0000","last_peered":"2024-09-21T10:22:51.552290+0000","last_clean":"2024-09-21T10:22:51.552290+0000","last_became_active":"2024-09-21T10:18:33.479691+0000","last_became_peered":"2024-09-21T10:18:33.479691+0000","last_unstale":"2024-09-21T10:22:51.552290+0000","last_undegraded":"2024-09-21T10:22:51.552290+0000","last_fullsized":"2024-09-21T10:22:51.552290+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:43:49.311244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":499,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.547526+0000","last_change":"2024-09-21T10:08:00.386104+0000","last_active":"2024-09-21T10:22:51.547526+0000","last_peered":"2024-09-21T10:22:51.547526+0000","last_clean":"2024-09-21T10:22:51.547526+0000","last_became_active":"2024-09-21T10:08:00.381484+0000","last_became_peered":"2024-09-21T10:08:00.381484+0000","last_unstale":"2024-09-21T10:22:51.547526+0000","last_undegraded":"2024-09-21T10:22:51.547526+0000","last_fullsized":"2024-09-21T10:22:51.547526+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:39.190191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":176,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.549348+0000","last_change":"2024-09-21T10:18:33.481322+0000","last_active":"2024-09-21T10:22:51.549348+0000","last_peered":"2024-09-21T10:22:51.549348+0000","last_clean":"2024-09-21T10:22:51.549348+0000","last_became_active":"2024-09-21T10:18:33.480554+0000","last_became_peered":"2024-09-21T10:18:33.480554+0000","last_unstale":"2024-09-21T10:22:51.549348+0000","last_undegraded":"2024-09-21T10:22:51.549348+0000","last_fullsized":"2024-09-21T10:22:51.549348+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:15.498594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":325,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.548895+0000","last_change":"2024-09-21T10:08:18.466882+0000","last_active":"2024-09-21T10:22:51.548895+0000","last_peered":"2024-09-21T10:22:51.548895+0000","last_clean":"2024-09-21T10:22:51.548895+0000","last_became_active":"2024-09-21T10:08:18.466470+0000","last_became_peered":"2024-09-21T10:08:18.466470+0000","last_unstale":"2024-09-21T10:22:51.548895+0000","last_undegraded":"2024-09-21T10:22:51.548895+0000","last_fullsized":"2024-09-21T10:22:51.548895+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:51:31.020051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":176,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.547567+0000","last_change":"2024-09-21T10:18:33.472352+0000","last_active":"2024-09-21T10:22:51.547567+0000","last_peered":"2024-09-21T10:22:51.547567+0000","last_clean":"2024-09-21T10:22:51.547567+0000","last_became_active":"2024-09-21T10:18:33.471312+0000","last_became_peered":"2024-09-21T10:18:33.471312+0000","last_unstale":"2024-09-21T10:22:51.547567+0000","last_undegraded":"2024-09-21T10:22:51.547567+0000","last_fullsized":"2024-09-21T10:22:51.547567+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:32:44.897145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":183,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.378900+0000","last_change":"2024-09-21T10:08:00.394350+0000","last_active":"2024-09-21T10:22:39.378900+0000","last_peered":"2024-09-21T10:22:39.378900+0000","last_clean":"2024-09-21T10:22:39.378900+0000","last_became_active":"2024-09-21T10:08:00.387232+0000","last_became_peered":"2024-09-21T10:08:00.387232+0000","last_unstale":"2024-09-21T10:22:39.378900+0000","last_undegraded":"2024-09-21T10:22:39.378900+0000","last_fullsized":"2024-09-21T10:22:39.378900+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:56.160018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":160,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.550983+0000","last_change":"2024-09-21T10:18:33.474216+0000","last_active":"2024-09-21T10:22:51.550983+0000","last_peered":"2024-09-21T10:22:51.550983+0000","last_clean":"2024-09-21T10:22:51.550983+0000","last_became_active":"2024-09-21T10:18:33.473794+0000","last_became_peered":"2024-09-21T10:18:33.473794+0000","last_unstale":"2024-09-21T10:22:51.550983+0000","last_undegraded":"2024-09-21T10:22:51.550983+0000","last_fullsized":"2024-09-21T10:22:51.550983+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:38.956888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":56,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.379116+0000","last_change":"2024-09-21T10:18:33.483576+0000","last_active":"2024-09-21T10:22:39.379116+0000","last_peered":"2024-09-21T10:22:39.379116+0000","last_clean":"2024-09-21T10:22:39.379116+0000","last_became_active":"2024-09-21T10:18:33.483116+0000","last_became_peered":"2024-09-21T10:18:33.483116+0000","last_unstale":"2024-09-21T10:22:39.379116+0000","last_undegraded":"2024-09-21T10:22:39.379116+0000","last_fullsized":"2024-09-21T10:22:39.379116+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:48.964766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":256,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T10:22:39.377965+0000","last_change":"2024-09-21T10:18:33.471909+0000","last_active":"2024-09-21T10:22:39.377965+0000","last_peered":"2024-09-21T10:22:39.377965+0000","last_clean":"2024-09-21T10:22:39.377965+0000","last_became_active":"2024-09-21T10:18:33.471614+0000","last_became_peered":"2024-09-21T10:18:33.471614+0000","last_unstale":"2024-09-21T10:22:39.377965+0000","last_undegraded":"2024-09-21T10:22:39.377965+0000","last_fullsized":"2024-09-21T10:22:39.377965+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:30:50.582505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":491,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-09-21T10:22:51.547763+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:22:51.547763+0000","last_peered":"2024-09-21T10:22:51.547763+0000","last_clean":"2024-09-21T10:22:51.547763+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:22:51.547763+0000","last_undegraded":"2024-09-21T10:22:51.547763+0000","last_fullsized":"2024-09-21T10:22:51.547763+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":3551330304,"num_objects":54189,"num_object_clones":0,"num_object_copies":162567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54189,"num_write_kb":3468096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10779754496,"data_stored":10779754496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54189,"ondisk_log_size":54189,"up":48,"acting":48,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":10016,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5982,"num_read_kb":13005,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397898,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3270292,"kb_used_data":2368908,"kb_used_omap":1,"kb_used_meta":901374,"kb_avail":90474860,"statfs":{"total":95995035648,"available":92646256640,"internally_reserved":0,"allocated":2425761792,"data_stored":2424092008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":923007320},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594126,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2007828,"kb_used_data":1106508,"kb_used_omap":1,"kb_used_meta":901310,"kb_avail":91737324,"statfs":{"total":95995035648,"available":93939019776,"internally_reserved":0,"allocated":1133064192,"data_stored":1131394405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922941784},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790356,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3094652,"kb_used_data":2190580,"kb_used_omap":1,"kb_used_meta":904062,"kb_avail":90650500,"statfs":{"total":95995035648,"available":92826112000,"internally_reserved":0,"allocated":2243153920,"data_stored":2241503552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":925759832},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986585,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":433791697147,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860674,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2078984,"kb_used_data":1365056,"kb_used_omap":8,"kb_used_meta":713911,"kb_avail":91666168,"statfs":{"total":95995035648,"available":93866156032,"internally_reserved":0,"allocated":1397817344,"data_stored":1396159712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":731045740},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024199,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2496564,"kb_used_data":1878124,"kb_used_omap":7,"kb_used_meta":618424,"kb_avail":91248588,"statfs":{"total":95995035648,"available":93438554112,"internally_reserved":0,"allocated":1923198976,"data_stored":1921555887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":633266738},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187735,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2223748,"kb_used_data":1469052,"kb_used_omap":8,"kb_used_meta":754679,"kb_avail":91521404,"statfs":{"total":95995035648,"available":93717917696,"internally_reserved":0,"allocated":1504309248,"data_stored":1502651590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":772792305},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1542586368,"data_stored":1542586368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1938948096,"data_stored":1938948096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1411252224,"data_stored":1411252224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2274492416,"data_stored":2274492416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1129447424,"data_stored":1129447424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2483027968,"data_stored":2483027968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:23:04.592 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T10:23:04.593 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pg_num 11 2024-09-21T10:23:04.612 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:04.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:04.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:04.714 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:04.763 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:04.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:04.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:04.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:04 smithi098 ceph-mon[97580]: pgmap v1022: 89 pgs: 89 active+clean; 3.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-21T10:23:04.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:04 smithi098 ceph-mon[100914]: pgmap v1022: 89 pgs: 89 active+clean; 3.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-21T10:23:04.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:04.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:04 smithi196 ceph-mon[84784]: pgmap v1022: 89 pgs: 89 active+clean; 3.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-21T10:23:04.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:04.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:05.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:05.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:05.051 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:05.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:05.152 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:05.182 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:05.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:05.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:05.206 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:05.235 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 32 16 65122 65106 127.067 90.25 0.011108 0.00786471 2024-09-21T10:23:05.283 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:05.317 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:05.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:05.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:05.418 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:05.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:05.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:05.553 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:05.590 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:05.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:05.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:05.606 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:23:05.633 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:23:05.633 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:23:05.672 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:05 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1867469092' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:05.672 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:05 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:05.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:05 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1867469092' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:05.673 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:05 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:05.692 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:05.717 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:05.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:05.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:05.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:05.855 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:05.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:05.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:05.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:05 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1867469092' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:05 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:05.957 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:05.991 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:05.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:05.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:06.093 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:06.125 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:06.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:06.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:06.227 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:06.234 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 33 16 66478 66462 125.785 84.75 0.0121241 0.00794581 2024-09-21T10:23:06.264 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:06.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:06.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:06.366 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:06.401 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:06.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:06.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:06.503 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:06.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:06.544 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:06.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:06.645 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:06.681 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:06.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:06.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:06.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:06 smithi098 ceph-mon[97580]: pgmap v1023: 89 pgs: 89 active+clean; 3.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:23:06.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:06 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:06.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:06 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:23:06.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:06 smithi098 ceph-mon[97580]: osdmap e309: 8 total, 8 up, 6 in 2024-09-21T10:23:06.750 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:06 smithi098 ceph-mon[100914]: pgmap v1023: 89 pgs: 89 active+clean; 3.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:23:06.750 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:06 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:06.750 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:06 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:23:06.750 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:06 smithi098 ceph-mon[100914]: osdmap e309: 8 total, 8 up, 6 in 2024-09-21T10:23:06.758 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:06 smithi196 ceph-mon[84784]: pgmap v1023: 89 pgs: 89 active+clean; 3.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:23:06.759 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:06 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:06.759 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:06 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:23:06.759 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:06 smithi196 ceph-mon[84784]: osdmap e309: 8 total, 8 up, 6 in 2024-09-21T10:23:06.784 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:06.805 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:23:06.813 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:06.820 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:06.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:06.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:06.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:06.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:06.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:07.054 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:07.090 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:07.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:07.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:07.192 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:07.234 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 34 16 67980 67964 124.847 93.875 0.0190131 0.00800448 2024-09-21T10:23:07.241 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:07.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:07.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:07.253 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:23:07.343 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:07.379 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:07.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:07.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:07.481 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:07.521 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:07.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:07.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:07.623 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:07.656 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:07.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:07.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:07.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:07.758 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:07.790 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:07.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:07.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:07.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:07.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:07.892 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:07.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:07.938 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:07.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:08.039 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:08.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:08.073 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:08.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:08.173 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:08.204 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:08.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:08.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:08.234 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 35 16 69429 69413 123.868 90.5625 0.0159455 0.00806859 2024-09-21T10:23:08.239 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:23:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:23:08.277 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:23:08.278 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:23:08.306 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:08.313 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:23:08.313 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:23:08.342 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:08.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:08.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:08.343 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:23:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:23:08.373 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:23:08.373 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:23:08.401 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:23:08.402 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:23:08.435 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:23:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:23:08.443 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:08.471 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:23:08.471 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:23:08.477 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:08.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:08.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:08.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:08.619 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:08.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:08.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:08.721 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:08.752 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:08.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:08.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:08.788 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:23:08.822 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:08 smithi196 ceph-mon[84784]: pgmap v1025: 89 pgs: 89 active+clean; 3.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-21T10:23:08.822 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:08 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3787353495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:08.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:08 smithi098 ceph-mon[97580]: pgmap v1025: 89 pgs: 89 active+clean; 3.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-21T10:23:08.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:08 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3787353495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:08.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:08 smithi098 ceph-mon[100914]: pgmap v1025: 89 pgs: 89 active+clean; 3.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-21T10:23:08.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:08 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3787353495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:08.854 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:08.886 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:08.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:08.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:08.988 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:09.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:09.035 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:09.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:09.136 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:09.163 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:23:09.163 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-21T10:23:09.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-21T10:23:09.163 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:23:09.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.163 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:109079 8.c e282/275 8:315a71ff:::benchmark_data_smithi098_226587_object5409:head v 282'6833, mlcod=0'0)", 2024-09-21T10:23:09.163 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:19:32.808374+0000", 2024-09-21T10:23:09.163 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 216.350138472, 2024-09-21T10:23:09.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.053664874000000001, 2024-09-21T10:23:09.163 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:09.163 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:09.163 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:09.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:09.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:32.808374+0000", 2024-09-21T10:23:09.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:09.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:32.808374+0000", 2024-09-21T10:23:09.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:09.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:32.808373+0000", 2024-09-21T10:23:09.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:23:09.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:09.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:32.808381+0000", 2024-09-21T10:23:09.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5700000000000004e-06 2024-09-21T10:23:09.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:09.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:32.808395+0000", 2024-09-21T10:23:09.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3981e-05 2024-09-21T10:23:09.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:09.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:32.808401+0000", 2024-09-21T10:23:09.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9610000000000002e-06 2024-09-21T10:23:09.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:09.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:32.825788+0000", 2024-09-21T10:23:09.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017387052 2024-09-21T10:23:09.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:09.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:32.825809+0000", 2024-09-21T10:23:09.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0755e-05 2024-09-21T10:23:09.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:09.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:32.861881+0000", 2024-09-21T10:23:09.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.036072601000000003 2024-09-21T10:23:09.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.168 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:09.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:19:32.862039+0000", 2024-09-21T10:23:09.168 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000157949 2024-09-21T10:23:09.168 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.168 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:09.168 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.168 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.168 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:138665 8.c e286/275 8:3a586884:::benchmark_data_smithi098_226587_object34995:head v 286'8687, mlcod=0'0)", 2024-09-21T10:23:09.168 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:20:09.587897+0000", 2024-09-21T10:23:09.168 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 179.57061573999999, 2024-09-21T10:23:09.168 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.049674033999999999, 2024-09-21T10:23:09.169 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:09.169 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:09.169 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:09.169 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.169 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:09.169 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:09.587897+0000", 2024-09-21T10:23:09.169 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.169 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.169 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:09.169 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:09.587897+0000", 2024-09-21T10:23:09.169 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.169 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.170 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:09.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:09.587896+0000", 2024-09-21T10:23:09.170 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:23:09.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.170 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.170 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:09.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:09.587905+0000", 2024-09-21T10:23:09.170 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3389999999999999e-06 2024-09-21T10:23:09.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.170 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.170 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:09.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:09.587924+0000", 2024-09-21T10:23:09.171 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8788999999999999e-05 2024-09-21T10:23:09.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.171 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:09.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:09.587932+0000", 2024-09-21T10:23:09.171 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9370000000000004e-06 2024-09-21T10:23:09.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.171 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:09.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:09.598688+0000", 2024-09-21T10:23:09.171 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010756051000000001 2024-09-21T10:23:09.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.171 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:09.172 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:09.598714+0000", 2024-09-21T10:23:09.172 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5947000000000001e-05 2024-09-21T10:23:09.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.172 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.172 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:09.172 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:09.637431+0000", 2024-09-21T10:23:09.172 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.038717440999999998 2024-09-21T10:23:09.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.172 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.172 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:09.172 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:09.637571+0000", 2024-09-21T10:23:09.172 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000139689 2024-09-21T10:23:09.172 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.173 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:09.173 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.173 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.173 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:147283 8.4 e288/265 8:2b23d1e1:::benchmark_data_smithi098_226587_object43613:head v 288'9232, mlcod=288'9228)", 2024-09-21T10:23:09.174 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:20:20.225413+0000", 2024-09-21T10:23:09.174 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 168.933100142, 2024-09-21T10:23:09.174 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.049720124999999997, 2024-09-21T10:23:09.174 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:09.174 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:09.174 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:09.174 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.175 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:09.175 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:20.225413+0000", 2024-09-21T10:23:09.175 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.175 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.175 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:09.175 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:20.225413+0000", 2024-09-21T10:23:09.175 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.175 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.175 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:09.175 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:20.225411+0000", 2024-09-21T10:23:09.176 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:23:09.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.176 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.176 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:09.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:20.225420+0000", 2024-09-21T10:23:09.176 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2879999999999993e-06 2024-09-21T10:23:09.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.176 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.176 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:09.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:20.225435+0000", 2024-09-21T10:23:09.176 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4866e-05 2024-09-21T10:23:09.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.176 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.176 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:09.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:20.225441+0000", 2024-09-21T10:23:09.177 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2709999999999998e-06 2024-09-21T10:23:09.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.177 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.177 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:09.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:20.237927+0000", 2024-09-21T10:23:09.177 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012486383 2024-09-21T10:23:09.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.177 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.177 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:09.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:20.237947+0000", 2024-09-21T10:23:09.177 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9279e-05 2024-09-21T10:23:09.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.177 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.178 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:09.178 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:20.275055+0000", 2024-09-21T10:23:09.178 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037108602999999997 2024-09-21T10:23:09.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.178 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.178 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:09.178 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:20.275133+0000", 2024-09-21T10:23:09.178 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7552999999999994e-05 2024-09-21T10:23:09.178 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.178 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:09.178 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.179 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.179 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:166965 8.4 e290/265 8:26eb4cdf:::benchmark_data_smithi098_226587_object63295:head v 290'10438, mlcod=290'10435)", 2024-09-21T10:23:09.179 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:20:44.556562+0000", 2024-09-21T10:23:09.179 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 144.601951042, 2024-09-21T10:23:09.179 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.052534816999999998, 2024-09-21T10:23:09.179 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:09.179 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:09.180 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:09.180 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.180 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:09.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:44.556562+0000", 2024-09-21T10:23:09.180 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.180 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.180 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:09.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:44.556562+0000", 2024-09-21T10:23:09.180 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.181 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.181 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:09.181 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:44.556561+0000", 2024-09-21T10:23:09.181 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:23:09.181 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.181 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.181 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:09.181 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:44.556565+0000", 2024-09-21T10:23:09.181 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0049999999999998e-06 2024-09-21T10:23:09.181 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.181 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.182 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:09.182 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:44.556574+0000", 2024-09-21T10:23:09.182 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.507e-06 2024-09-21T10:23:09.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.182 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.182 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:09.182 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:44.556578+0000", 2024-09-21T10:23:09.182 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1629999999999998e-06 2024-09-21T10:23:09.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.182 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.182 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:09.182 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:44.562960+0000", 2024-09-21T10:23:09.182 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0063821529999999998 2024-09-21T10:23:09.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.183 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.183 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:09.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:44.562982+0000", 2024-09-21T10:23:09.183 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1471000000000002e-05 2024-09-21T10:23:09.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.183 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.183 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:09.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:44.608947+0000", 2024-09-21T10:23:09.183 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.045965182 2024-09-21T10:23:09.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.184 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.184 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:09.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:20:44.609096+0000", 2024-09-21T10:23:09.184 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014973800000000001 2024-09-21T10:23:09.184 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.184 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:09.184 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.184 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.184 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:185399 8.c e294/275 8:3300e304:::benchmark_data_smithi098_226587_object81729:head v 294'11645, mlcod=0'0)", 2024-09-21T10:23:09.184 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:21:07.607516+0000", 2024-09-21T10:23:09.184 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 121.550996345, 2024-09-21T10:23:09.184 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.063349242, 2024-09-21T10:23:09.184 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:09.185 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:09.185 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:09.185 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.185 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:09.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.607516+0000", 2024-09-21T10:23:09.185 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.185 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.185 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:09.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.607516+0000", 2024-09-21T10:23:09.185 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.185 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.185 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:09.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.607516+0000", 2024-09-21T10:23:09.186 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:23:09.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.186 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.186 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:09.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.607525+0000", 2024-09-21T10:23:09.186 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0129999999999999e-06 2024-09-21T10:23:09.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.186 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.186 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:09.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.607534+0000", 2024-09-21T10:23:09.186 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1740000000000006e-06 2024-09-21T10:23:09.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.187 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.187 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:09.187 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.607539+0000", 2024-09-21T10:23:09.187 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1800000000000004e-06 2024-09-21T10:23:09.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.187 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.187 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:09.187 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.615066+0000", 2024-09-21T10:23:09.187 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0075273600000000003 2024-09-21T10:23:09.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.187 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.187 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:09.187 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.615176+0000", 2024-09-21T10:23:09.187 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000110191 2024-09-21T10:23:09.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.188 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.188 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:09.188 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.670836+0000", 2024-09-21T10:23:09.188 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.055659226999999999 2024-09-21T10:23:09.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.188 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.188 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:09.188 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.670866+0000", 2024-09-21T10:23:09.188 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9940000000000001e-05 2024-09-21T10:23:09.188 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.188 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:09.188 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.189 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.189 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:185404 8.c e294/275 8:353e6f7f:::benchmark_data_smithi098_226587_object81734:head v 294'11646, mlcod=0'0)", 2024-09-21T10:23:09.189 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:21:07.611704+0000", 2024-09-21T10:23:09.189 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 121.54680863199999, 2024-09-21T10:23:09.189 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.059192305000000001, 2024-09-21T10:23:09.189 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:09.189 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:09.189 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:09.189 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.189 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:09.189 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.611704+0000", 2024-09-21T10:23:09.189 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.189 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.189 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:09.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.611704+0000", 2024-09-21T10:23:09.190 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.190 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.190 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:09.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.611703+0000", 2024-09-21T10:23:09.190 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:23:09.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.190 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.190 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:09.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.611714+0000", 2024-09-21T10:23:09.190 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1647999999999999e-05 2024-09-21T10:23:09.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.191 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.191 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:09.191 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.611731+0000", 2024-09-21T10:23:09.191 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6668e-05 2024-09-21T10:23:09.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.191 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.191 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:09.191 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.611738+0000", 2024-09-21T10:23:09.191 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7449999999999998e-06 2024-09-21T10:23:09.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.191 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.191 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:09.191 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.623882+0000", 2024-09-21T10:23:09.192 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012143713 2024-09-21T10:23:09.192 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.192 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.192 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:09.192 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.623903+0000", 2024-09-21T10:23:09.192 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1693000000000001e-05 2024-09-21T10:23:09.192 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.192 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.192 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:09.192 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.670869+0000", 2024-09-21T10:23:09.192 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.046965453999999997 2024-09-21T10:23:09.192 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.192 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.192 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:09.192 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.670896+0000", 2024-09-21T10:23:09.193 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7712000000000002e-05 2024-09-21T10:23:09.193 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.193 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:09.193 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.193 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.193 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:185405 8.c e294/275 8:37c7085b:::benchmark_data_smithi098_226587_object81735:head v 294'11647, mlcod=0'0)", 2024-09-21T10:23:09.193 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:21:07.616350+0000", 2024-09-21T10:23:09.193 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 121.542162541, 2024-09-21T10:23:09.193 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.054578033999999997, 2024-09-21T10:23:09.193 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:09.193 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:09.193 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:09.194 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.194 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:09.194 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.616350+0000", 2024-09-21T10:23:09.194 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.194 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.194 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:09.194 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.616350+0000", 2024-09-21T10:23:09.194 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.194 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.194 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:09.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.616349+0000", 2024-09-21T10:23:09.195 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:23:09.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.195 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.195 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:09.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.616360+0000", 2024-09-21T10:23:09.195 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0905e-05 2024-09-21T10:23:09.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.195 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.195 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:09.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.616371+0000", 2024-09-21T10:23:09.195 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.066e-05 2024-09-21T10:23:09.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.195 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.196 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:09.196 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.616376+0000", 2024-09-21T10:23:09.196 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5720000000000002e-06 2024-09-21T10:23:09.196 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.196 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.196 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:09.196 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.633653+0000", 2024-09-21T10:23:09.196 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017276771 2024-09-21T10:23:09.196 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.196 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.196 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:09.196 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.633745+0000", 2024-09-21T10:23:09.196 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2311000000000004e-05 2024-09-21T10:23:09.196 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.196 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.197 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:09.197 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.670916+0000", 2024-09-21T10:23:09.197 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037170933000000003 2024-09-21T10:23:09.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.197 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.197 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:09.197 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.670928+0000", 2024-09-21T10:23:09.197 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1992e-05 2024-09-21T10:23:09.197 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.197 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:09.197 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.197 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.197 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:185410 8.4 e294/265 8:2ddd92e0:::benchmark_data_smithi098_226587_object81740:head v 294'11605, mlcod=294'11601)", 2024-09-21T10:23:09.198 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:21:07.617882+0000", 2024-09-21T10:23:09.198 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 121.540630967, 2024-09-21T10:23:09.198 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.053033223999999997, 2024-09-21T10:23:09.198 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:09.198 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:09.198 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:09.198 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.198 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:09.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.617882+0000", 2024-09-21T10:23:09.198 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.199 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.199 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:09.199 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.617882+0000", 2024-09-21T10:23:09.199 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.199 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.199 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.199 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:09.199 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.617881+0000", 2024-09-21T10:23:09.199 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:23:09.199 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.199 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.199 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:09.199 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.617889+0000", 2024-09-21T10:23:09.199 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6100000000000006e-06 2024-09-21T10:23:09.200 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.200 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.200 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:09.200 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.617903+0000", 2024-09-21T10:23:09.200 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3628000000000001e-05 2024-09-21T10:23:09.200 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.200 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.200 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:09.200 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.617909+0000", 2024-09-21T10:23:09.200 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3990000000000001e-06 2024-09-21T10:23:09.200 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.200 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.200 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:09.201 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.629410+0000", 2024-09-21T10:23:09.201 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011500112999999999 2024-09-21T10:23:09.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.201 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.201 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:09.201 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.629422+0000", 2024-09-21T10:23:09.201 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2751999999999999e-05 2024-09-21T10:23:09.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.201 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.201 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:09.201 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.670899+0000", 2024-09-21T10:23:09.201 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.041477133999999999 2024-09-21T10:23:09.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.201 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.202 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:09.202 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.670915+0000", 2024-09-21T10:23:09.202 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5503e-05 2024-09-21T10:23:09.202 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.202 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:09.202 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.202 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.202 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:185418 8.c e294/275 8:31ee3db5:::benchmark_data_smithi098_226587_object81748:head v 294'11648, mlcod=0'0)", 2024-09-21T10:23:09.202 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:21:07.620220+0000", 2024-09-21T10:23:09.202 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 121.53829290199999, 2024-09-21T10:23:09.202 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.050720714, 2024-09-21T10:23:09.202 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:09.203 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:09.203 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:09.203 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.203 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:09.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.620220+0000", 2024-09-21T10:23:09.203 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.203 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.203 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:09.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.620220+0000", 2024-09-21T10:23:09.203 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.203 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.203 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:09.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.620219+0000", 2024-09-21T10:23:09.203 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:23:09.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.204 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.204 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:09.204 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.620226+0000", 2024-09-21T10:23:09.204 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3559999999999998e-06 2024-09-21T10:23:09.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.204 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.204 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:09.204 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.620234+0000", 2024-09-21T10:23:09.204 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8869999999999992e-06 2024-09-21T10:23:09.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.204 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.204 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:09.204 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.620238+0000", 2024-09-21T10:23:09.204 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2429999999999999e-06 2024-09-21T10:23:09.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.205 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.205 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:09.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.642169+0000", 2024-09-21T10:23:09.205 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.021931731999999999 2024-09-21T10:23:09.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.205 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.205 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:09.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.642209+0000", 2024-09-21T10:23:09.205 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9567999999999998e-05 2024-09-21T10:23:09.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.205 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.205 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:09.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.670930+0000", 2024-09-21T10:23:09.205 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.028720514999999999 2024-09-21T10:23:09.206 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.206 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.206 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:09.206 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.670941+0000", 2024-09-21T10:23:09.206 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0954999999999999e-05 2024-09-21T10:23:09.206 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.206 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:09.206 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.206 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.206 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.206 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:185579 8.c e294/275 8:3093f11d:::benchmark_data_smithi098_226587_object81909:head v 294'11661, mlcod=0'0)", 2024-09-21T10:23:09.206 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:21:07.805554+0000", 2024-09-21T10:23:09.206 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 121.352958626, 2024-09-21T10:23:09.207 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.059237999, 2024-09-21T10:23:09.207 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:09.207 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:09.207 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:09.207 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.207 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:09.207 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.805554+0000", 2024-09-21T10:23:09.207 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.207 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.207 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.207 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:09.207 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.805554+0000", 2024-09-21T10:23:09.207 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.207 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.208 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.208 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:09.208 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.805553+0000", 2024-09-21T10:23:09.208 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:23:09.208 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.208 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.208 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:09.208 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.805560+0000", 2024-09-21T10:23:09.208 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7410000000000003e-06 2024-09-21T10:23:09.208 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.208 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.208 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:09.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:09.210 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:09.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:09.210 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.805574+0000", 2024-09-21T10:23:09.211 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3704e-05 2024-09-21T10:23:09.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.211 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.211 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:09.211 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.805579+0000", 2024-09-21T10:23:09.211 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7710000000000003e-06 2024-09-21T10:23:09.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.211 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.211 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:09.211 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.821730+0000", 2024-09-21T10:23:09.211 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016150889000000002 2024-09-21T10:23:09.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.212 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.212 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:09.212 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.821775+0000", 2024-09-21T10:23:09.212 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4144000000000003e-05 2024-09-21T10:23:09.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.212 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.212 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:09.212 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.864641+0000", 2024-09-21T10:23:09.212 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.042866948000000002 2024-09-21T10:23:09.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.212 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.212 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:09.212 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:07.864792+0000", 2024-09-21T10:23:09.213 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015061299999999999 2024-09-21T10:23:09.213 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.213 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:09.213 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.213 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.213 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:201699 8.c e294/275 8:338f6930:::benchmark_data_smithi098_226587_object98029:head v 294'12694, mlcod=0'0)", 2024-09-21T10:23:09.213 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:21:27.296003+0000", 2024-09-21T10:23:09.213 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 101.86250936, 2024-09-21T10:23:09.213 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.050231401000000002, 2024-09-21T10:23:09.213 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:09.213 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:09.213 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:09.213 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.213 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:09.213 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.296003+0000", 2024-09-21T10:23:09.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.214 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.214 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:09.214 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.296003+0000", 2024-09-21T10:23:09.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.214 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.214 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:09.214 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.296002+0000", 2024-09-21T10:23:09.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:23:09.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.214 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.214 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:09.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.296011+0000", 2024-09-21T10:23:09.215 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2870000000000004e-06 2024-09-21T10:23:09.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:09.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.296024+0000", 2024-09-21T10:23:09.215 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3271999999999999e-05 2024-09-21T10:23:09.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:09.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.296030+0000", 2024-09-21T10:23:09.215 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2609999999999999e-06 2024-09-21T10:23:09.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.216 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:09.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.296110+0000", 2024-09-21T10:23:09.216 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9631999999999999e-05 2024-09-21T10:23:09.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.216 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:09.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.296131+0000", 2024-09-21T10:23:09.216 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1565000000000001e-05 2024-09-21T10:23:09.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.216 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:09.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.345888+0000", 2024-09-21T10:23:09.216 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.049756793000000001 2024-09-21T10:23:09.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.217 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.217 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:09.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.346235+0000", 2024-09-21T10:23:09.217 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000346499 2024-09-21T10:23:09.217 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.217 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:09.217 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.217 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.217 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:201707 8.c e294/275 8:3183fb90:::benchmark_data_smithi098_226587_object98037:head v 294'12695, mlcod=0'0)", 2024-09-21T10:23:09.217 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:21:27.305200+0000", 2024-09-21T10:23:09.217 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 101.85331237, 2024-09-21T10:23:09.218 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.050247869000000001, 2024-09-21T10:23:09.218 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:09.218 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:09.218 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:09.218 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.218 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:09.218 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.305200+0000", 2024-09-21T10:23:09.218 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.218 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.218 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.218 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:09.219 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.305200+0000", 2024-09-21T10:23:09.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.219 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.219 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:09.219 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.305199+0000", 2024-09-21T10:23:09.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:23:09.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.219 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.219 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:09.219 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.305211+0000", 2024-09-21T10:23:09.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1423000000000001e-05 2024-09-21T10:23:09.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:09.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.305224+0000", 2024-09-21T10:23:09.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2816999999999999e-05 2024-09-21T10:23:09.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:09.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.305231+0000", 2024-09-21T10:23:09.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.34e-06 2024-09-21T10:23:09.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:09.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.309858+0000", 2024-09-21T10:23:09.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046271350000000001 2024-09-21T10:23:09.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.221 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:09.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.309892+0000", 2024-09-21T10:23:09.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3401000000000001e-05 2024-09-21T10:23:09.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.221 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:09.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.355408+0000", 2024-09-21T10:23:09.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.045516155000000003 2024-09-21T10:23:09.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:09.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.355448+0000", 2024-09-21T10:23:09.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0472000000000001e-05 2024-09-21T10:23:09.222 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.222 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:09.222 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.222 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:201720 8.c e294/275 8:3416b68e:::benchmark_data_smithi098_226587_object98050:head v 294'12698, mlcod=0'0)", 2024-09-21T10:23:09.222 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:21:27.321016+0000", 2024-09-21T10:23:09.222 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 101.837496857, 2024-09-21T10:23:09.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.050638584, 2024-09-21T10:23:09.223 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:09.223 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:09.223 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:09.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:09.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.321016+0000", 2024-09-21T10:23:09.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:09.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.321016+0000", 2024-09-21T10:23:09.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:09.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.321015+0000", 2024-09-21T10:23:09.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:23:09.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:09.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.321025+0000", 2024-09-21T10:23:09.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0227000000000001e-05 2024-09-21T10:23:09.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:09.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.321040+0000", 2024-09-21T10:23:09.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4651000000000001e-05 2024-09-21T10:23:09.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:09.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.321046+0000", 2024-09-21T10:23:09.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.891e-06 2024-09-21T10:23:09.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:09.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.338117+0000", 2024-09-21T10:23:09.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017071567999999999 2024-09-21T10:23:09.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.226 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:09.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.338146+0000", 2024-09-21T10:23:09.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9130000000000001e-05 2024-09-21T10:23:09.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.226 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:09.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.371608+0000", 2024-09-21T10:23:09.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.033461642999999999 2024-09-21T10:23:09.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.227 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:09.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.371654+0000", 2024-09-21T10:23:09.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6292000000000003e-05 2024-09-21T10:23:09.227 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.227 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:09.227 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.227 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.45794.0:201727 8.c e294/275 8:30dcedc2:::benchmark_data_smithi098_226587_object98057:head v 294'12699, mlcod=0'0)", 2024-09-21T10:23:09.227 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:21:27.326804+0000", 2024-09-21T10:23:09.227 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 101.83170823, 2024-09-21T10:23:09.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.049319406000000003, 2024-09-21T10:23:09.227 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:09.228 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:09.228 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:09.228 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.228 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:09.228 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.326804+0000", 2024-09-21T10:23:09.228 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.228 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.228 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:09.228 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.326804+0000", 2024-09-21T10:23:09.228 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.228 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.228 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:09.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.326804+0000", 2024-09-21T10:23:09.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:23:09.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:09.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.326811+0000", 2024-09-21T10:23:09.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7510000000000002e-06 2024-09-21T10:23:09.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:09.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.326821+0000", 2024-09-21T10:23:09.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0302000000000001e-05 2024-09-21T10:23:09.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:09.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.326825+0000", 2024-09-21T10:23:09.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2050000000000004e-06 2024-09-21T10:23:09.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:09.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.346261+0000", 2024-09-21T10:23:09.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019435279 2024-09-21T10:23:09.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:09.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.346282+0000", 2024-09-21T10:23:09.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1086000000000001e-05 2024-09-21T10:23:09.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:09.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.376070+0000", 2024-09-21T10:23:09.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.029788775999999999 2024-09-21T10:23:09.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:09.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:21:27.376124+0000", 2024-09-21T10:23:09.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3489000000000002e-05 2024-09-21T10:23:09.231 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.231 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:09.232 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.232 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46025.0:29872 9.7 e307/300 9:e2963160:::benchmark_data_smithi098_254407_object29871:head v 307'1879, mlcod=307'1877)", 2024-09-21T10:23:09.232 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:22:43.709026+0000", 2024-09-21T10:23:09.232 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.449487102999999, 2024-09-21T10:23:09.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.118138437, 2024-09-21T10:23:09.232 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:09.232 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:09.232 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:09.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:09.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709026+0000", 2024-09-21T10:23:09.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:09.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709026+0000", 2024-09-21T10:23:09.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:09.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709025+0000", 2024-09-21T10:23:09.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:23:09.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:09.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709047+0000", 2024-09-21T10:23:09.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2416e-05 2024-09-21T10:23:09.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.234 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:09.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709059+0000", 2024-09-21T10:23:09.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1548e-05 2024-09-21T10:23:09.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.234 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:09.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709064+0000", 2024-09-21T10:23:09.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5539999999999997e-06 2024-09-21T10:23:09.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.234 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:09.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709101+0000", 2024-09-21T10:23:09.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6871e-05 2024-09-21T10:23:09.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:09.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709120+0000", 2024-09-21T10:23:09.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8379000000000001e-05 2024-09-21T10:23:09.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:09.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.710335+0000", 2024-09-21T10:23:09.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001215631 2024-09-21T10:23:09.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:09.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.827164+0000", 2024-09-21T10:23:09.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.116828895 2024-09-21T10:23:09.236 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.236 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:09.236 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.236 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46025.0:29871 9.9 e307/300 9:90b07bd6:::benchmark_data_smithi098_254407_object29870:head v 307'1822, mlcod=307'1820)", 2024-09-21T10:23:09.236 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:22:43.709126+0000", 2024-09-21T10:23:09.236 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.449386493999999, 2024-09-21T10:23:09.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.11807324299999999, 2024-09-21T10:23:09.236 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:09.236 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:09.237 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:09.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.237 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:09.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709126+0000", 2024-09-21T10:23:09.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.237 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:09.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709126+0000", 2024-09-21T10:23:09.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.237 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:09.238 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709126+0000", 2024-09-21T10:23:09.238 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:23:09.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.238 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.238 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:09.238 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709243+0000", 2024-09-21T10:23:09.238 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011697900000000001 2024-09-21T10:23:09.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.238 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.238 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:09.238 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709252+0000", 2024-09-21T10:23:09.238 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4790000000000002e-06 2024-09-21T10:23:09.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:09.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709257+0000", 2024-09-21T10:23:09.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9119999999999997e-06 2024-09-21T10:23:09.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:09.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709297+0000", 2024-09-21T10:23:09.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9909e-05 2024-09-21T10:23:09.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:09.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709314+0000", 2024-09-21T10:23:09.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7289e-05 2024-09-21T10:23:09.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.240 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.240 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:09.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.827144+0000", 2024-09-21T10:23:09.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.11783031300000001 2024-09-21T10:23:09.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.240 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.240 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:09.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.827199+0000", 2024-09-21T10:23:09.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5043999999999997e-05 2024-09-21T10:23:09.240 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.240 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:09.240 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.241 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.241 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46025.0:29876 9.5 e307/300 9:a939da45:::benchmark_data_smithi098_254407_object29875:head v 307'1955, mlcod=307'1953)", 2024-09-21T10:23:09.241 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:22:43.709790+0000", 2024-09-21T10:23:09.241 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.448722677999999, 2024-09-21T10:23:09.241 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.11742913100000001, 2024-09-21T10:23:09.241 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:09.241 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:09.241 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:09.241 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.241 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:09.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709790+0000", 2024-09-21T10:23:09.241 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:09.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709790+0000", 2024-09-21T10:23:09.242 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:09.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709789+0000", 2024-09-21T10:23:09.242 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:23:09.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:09.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709857+0000", 2024-09-21T10:23:09.242 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8091999999999998e-05 2024-09-21T10:23:09.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.243 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:09.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709905+0000", 2024-09-21T10:23:09.243 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7822999999999999e-05 2024-09-21T10:23:09.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.243 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:09.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709911+0000", 2024-09-21T10:23:09.243 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2230000000000004e-06 2024-09-21T10:23:09.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.243 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:09.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.710063+0000", 2024-09-21T10:23:09.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000151745 2024-09-21T10:23:09.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.244 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:09.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.710082+0000", 2024-09-21T10:23:09.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.895e-05 2024-09-21T10:23:09.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.244 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:09.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.827153+0000", 2024-09-21T10:23:09.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.117071469 2024-09-21T10:23:09.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.244 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:09.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.827219+0000", 2024-09-21T10:23:09.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5710000000000006e-05 2024-09-21T10:23:09.245 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.245 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:09.245 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.245 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46025.0:29878 9.8 e307/300 9:122cf424:::benchmark_data_smithi098_254407_object29877:head v 307'1845, mlcod=307'1843)", 2024-09-21T10:23:09.245 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:22:43.709925+0000", 2024-09-21T10:23:09.245 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.448587352000001, 2024-09-21T10:23:09.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.12002425, 2024-09-21T10:23:09.245 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:09.245 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:09.245 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:09.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:09.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709925+0000", 2024-09-21T10:23:09.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:09.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709925+0000", 2024-09-21T10:23:09.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:09.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709925+0000", 2024-09-21T10:23:09.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:23:09.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:09.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.709991+0000", 2024-09-21T10:23:09.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6209000000000003e-05 2024-09-21T10:23:09.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:09.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.710000+0000", 2024-09-21T10:23:09.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6989999999999992e-06 2024-09-21T10:23:09.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:09.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.710021+0000", 2024-09-21T10:23:09.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0937e-05 2024-09-21T10:23:09.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.248 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:09.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.827174+0000", 2024-09-21T10:23:09.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.11715334400000001 2024-09-21T10:23:09.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.248 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:09.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.827201+0000", 2024-09-21T10:23:09.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6500999999999999e-05 2024-09-21T10:23:09.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.248 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:09.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.829906+0000", 2024-09-21T10:23:09.249 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027056469999999998 2024-09-21T10:23:09.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.249 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.249 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:09.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.829950+0000", 2024-09-21T10:23:09.249 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.333e-05 2024-09-21T10:23:09.249 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.249 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:09.249 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.249 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.249 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46025.0:29880 9.9 e307/300 9:9d0584e7:::benchmark_data_smithi098_254407_object29879:head v 307'1823, mlcod=307'1820)", 2024-09-21T10:23:09.249 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:22:43.711433+0000", 2024-09-21T10:23:09.249 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.447079623, 2024-09-21T10:23:09.250 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.119483019, 2024-09-21T10:23:09.250 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:09.250 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:09.250 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:09.250 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.250 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:09.250 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.711433+0000", 2024-09-21T10:23:09.250 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.250 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.250 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.250 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:09.250 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.711433+0000", 2024-09-21T10:23:09.250 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:09.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.711432+0000", 2024-09-21T10:23:09.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:23:09.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:09.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.711475+0000", 2024-09-21T10:23:09.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2675000000000001e-05 2024-09-21T10:23:09.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:09.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.827122+0000", 2024-09-21T10:23:09.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.115647099 2024-09-21T10:23:09.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:09.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.827141+0000", 2024-09-21T10:23:09.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8947e-05 2024-09-21T10:23:09.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:09.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.827256+0000", 2024-09-21T10:23:09.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011491499999999999 2024-09-21T10:23:09.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:09.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.827276+0000", 2024-09-21T10:23:09.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0492999999999998e-05 2024-09-21T10:23:09.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:09.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.830886+0000", 2024-09-21T10:23:09.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036091669999999999 2024-09-21T10:23:09.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:09.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.830916+0000", 2024-09-21T10:23:09.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0559000000000003e-05 2024-09-21T10:23:09.253 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.253 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:09.254 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.254 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.254 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46025.0:29883 9.f e307/300 9:fc5926c3:::benchmark_data_smithi098_254407_object29882:head v 307'1904, mlcod=307'1902)", 2024-09-21T10:23:09.254 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 36 16 70812 70796 122.828 86.4375 0.0215153 0.00813619 2024-09-21T10:23:09.255 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:22:43.713786+0000", 2024-09-21T10:23:09.255 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.444727136000001, 2024-09-21T10:23:09.255 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.116774264, 2024-09-21T10:23:09.255 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:09.255 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:09.255 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:09.255 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.255 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:09.255 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.713786+0000", 2024-09-21T10:23:09.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:09.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.713786+0000", 2024-09-21T10:23:09.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:09.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:09.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.713785+0000", 2024-09-21T10:23:09.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:23:09.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:09.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.713838+0000", 2024-09-21T10:23:09.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3384999999999998e-05 2024-09-21T10:23:09.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:09.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.827145+0000", 2024-09-21T10:23:09.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.1133067 2024-09-21T10:23:09.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:09.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.827154+0000", 2024-09-21T10:23:09.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8720000000000008e-06 2024-09-21T10:23:09.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:09.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.827192+0000", 2024-09-21T10:23:09.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7880000000000003e-05 2024-09-21T10:23:09.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:09.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.827208+0000", 2024-09-21T10:23:09.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6578e-05 2024-09-21T10:23:09.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.259 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:09.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.830516+0000", 2024-09-21T10:23:09.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0033076949999999998 2024-09-21T10:23:09.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:09.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:09.259 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:09.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:43.830560+0000", 2024-09-21T10:23:09.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3993999999999999e-05 2024-09-21T10:23:09.259 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.259 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:09.260 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.260 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:09.260 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:09.260 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:23:09.311 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:09.348 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:09.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:09.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:09.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:09.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:09.483 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:09.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:09.584 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:09.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:09.620 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:09.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:09.721 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:09.754 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:09.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:09.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:09.856 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:09.877 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:09.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:09.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:09.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:10.011 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:10.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:10.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:10.112 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:10.138 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:10.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:10.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:10.234 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 37 16 71972 71956 121.469 72.5 0.0139396 0.00822745 2024-09-21T10:23:10.240 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:10.277 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:10.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:10.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:10.379 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:10.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:23:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:23:10.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:23:10.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:10.417 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:10.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:10.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:10.540 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:10.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:10.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:10.640 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:10 smithi098 ceph-mon[97580]: pgmap v1026: 89 pgs: 89 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 93 MiB/s wr, 1.50k op/s 2024-09-21T10:23:10.640 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:10 smithi098 ceph-mon[100914]: pgmap v1026: 89 pgs: 89 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 93 MiB/s wr, 1.50k op/s 2024-09-21T10:23:10.642 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:10.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:10.675 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:10.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:10.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:10.808 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:10.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:10.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:10 smithi196 ceph-mon[84784]: pgmap v1026: 89 pgs: 89 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 93 MiB/s wr, 1.50k op/s 2024-09-21T10:23:10.910 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:10.948 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:10.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:10.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:11.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:11.082 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:11.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:11.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:11.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:23:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:23:10] "GET /metrics HTTP/1.1" 200 36535 "" "Prometheus/2.51.0" 2024-09-21T10:23:11.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:11.222 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:11.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:11.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:11.234 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 38 16 73091 73075 120.113 69.9375 0.00901743 0.0083213 2024-09-21T10:23:11.324 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:11.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:11.368 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:11.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:11.469 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:11.499 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:11.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:11.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:11.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:23:11.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:11.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:11.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:11.636 INFO:teuthology.orchestra.run.smithi098.stderr:set pool 1 pg_num to 11 2024-09-21T10:23:11.737 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:11.753 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:23:11.775 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:11.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:11.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:11.877 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:11.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:11 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3008538333' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T10:23:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:11 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T10:23:11.930 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:11.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:11.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:11.968 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:23:12.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:12.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:12.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:12.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:12.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:11 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3008538333' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T10:23:12.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:11 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T10:23:12.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:11 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3008538333' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T10:23:12.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:11 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T10:23:12.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:12.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:12.217 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:12.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:12.235 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 39 16 74433 74417 119.184 83.875 0.0077428 0.00838578 2024-09-21T10:23:12.317 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:12.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:12.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:12.454 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:12.478 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:12.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:12.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:12.580 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:12.616 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:12.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:12.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:12.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:12.757 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:12.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:12.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:12.859 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:12 smithi196 ceph-mon[84784]: pgmap v1027: 89 pgs: 89 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-21T10:23:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:12 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T10:23:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:12 smithi196 ceph-mon[84784]: osdmap e310: 8 total, 8 up, 6 in 2024-09-21T10:23:12.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:12.895 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:12.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:12.901 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:12 smithi098 ceph-mon[97580]: pgmap v1027: 89 pgs: 89 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-21T10:23:12.901 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:12 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T10:23:12.901 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:12 smithi098 ceph-mon[97580]: osdmap e310: 8 total, 8 up, 6 in 2024-09-21T10:23:12.902 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:12 smithi098 ceph-mon[100914]: pgmap v1027: 89 pgs: 89 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-21T10:23:12.902 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:12 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T10:23:12.902 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:12 smithi098 ceph-mon[100914]: osdmap e310: 8 total, 8 up, 6 in 2024-09-21T10:23:12.995 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:13.030 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:13.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:13.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:13.132 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:13.170 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:13.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:13.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:13.235 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:23:13.233828+0000 min lat: 0.00136025 max lat: 0.157931 avg lat: 0.00844366 2024-09-21T10:23:13.235 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:23:13.235 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 40 16 75816 75800 118.366 86.4375 0.0133528 0.00844366 2024-09-21T10:23:13.272 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:13.316 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:13.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:13.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:13.418 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:13.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:13.454 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:13.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:13.555 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:13.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:13.603 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:13.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:13.704 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:13.736 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:13.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:13.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:13.755 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:13 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T10:23:13.838 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:13.879 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:13.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:13.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:13.906 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:23:13.906 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:23:13.906 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.906 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:76627 9.1 9:80079d4f:::benchmark_data_smithi098_254407_object76626:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e311)", 2024-09-21T10:23:13.906 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:23:13.886741+0000", 2024-09-21T10:23:13.906 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.015008267, 2024-09-21T10:23:13.907 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015040321000000001, 2024-09-21T10:23:13.907 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:13.907 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:23:13.907 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:23:13.907 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:23:13.907 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:23:13.907 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 76627 2024-09-21T10:23:13.907 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.907 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:13.907 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.907 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:13.907 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.886741+0000", 2024-09-21T10:23:13.908 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:13.908 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.908 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.908 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:13.908 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.886741+0000", 2024-09-21T10:23:13.908 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:13.908 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.908 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.908 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:13.909 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.886741+0000", 2024-09-21T10:23:13.909 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:23:13.909 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.909 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.909 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:13.909 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.886870+0000", 2024-09-21T10:23:13.909 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012926300000000001 2024-09-21T10:23:13.909 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.909 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.909 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:13.910 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.886889+0000", 2024-09-21T10:23:13.910 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9334000000000002e-05 2024-09-21T10:23:13.910 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.910 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.910 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:13.910 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.886894+0000", 2024-09-21T10:23:13.910 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9180000000000002e-06 2024-09-21T10:23:13.910 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.910 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.910 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:13.910 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.891748+0000", 2024-09-21T10:23:13.910 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048534720000000002 2024-09-21T10:23:13.910 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.911 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.911 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:13.911 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.891955+0000", 2024-09-21T10:23:13.911 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000206877 2024-09-21T10:23:13.911 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.911 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.911 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:23:13.911 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.892101+0000", 2024-09-21T10:23:13.911 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000146549 2024-09-21T10:23:13.911 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.911 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.911 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:23:13.911 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.894296+0000", 2024-09-21T10:23:13.912 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021945340000000002 2024-09-21T10:23:13.912 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.912 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.912 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:13.912 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.894631+0000", 2024-09-21T10:23:13.912 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00033574599999999999 2024-09-21T10:23:13.912 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:13.912 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:13.912 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:13.912 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.912 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.912 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:76637 9.1 9:87985042:::benchmark_data_smithi098_254407_object76636:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e311)", 2024-09-21T10:23:13.912 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:23:13.895269+0000", 2024-09-21T10:23:13.913 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.006480558, 2024-09-21T10:23:13.913 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0066009459999999999, 2024-09-21T10:23:13.913 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:13.913 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:23:13.913 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:23:13.913 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:23:13.913 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:23:13.913 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 76637 2024-09-21T10:23:13.913 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.913 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:13.913 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.913 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:13.913 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.895269+0000", 2024-09-21T10:23:13.913 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:13.914 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.914 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.914 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:13.914 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.895269+0000", 2024-09-21T10:23:13.914 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:13.914 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.914 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.914 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:13.914 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.895268+0000", 2024-09-21T10:23:13.914 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:23:13.914 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.914 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:13.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.895326+0000", 2024-09-21T10:23:13.915 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8347999999999998e-05 2024-09-21T10:23:13.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:13.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.895337+0000", 2024-09-21T10:23:13.915 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1029999999999999e-05 2024-09-21T10:23:13.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:13.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.895342+0000", 2024-09-21T10:23:13.915 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6539999999999998e-06 2024-09-21T10:23:13.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.916 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.916 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:13.916 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.895385+0000", 2024-09-21T10:23:13.916 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2756000000000002e-05 2024-09-21T10:23:13.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.916 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.916 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:13.916 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.895498+0000", 2024-09-21T10:23:13.916 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011322700000000001 2024-09-21T10:23:13.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.916 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.916 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:23:13.917 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.895645+0000", 2024-09-21T10:23:13.917 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014673600000000001 2024-09-21T10:23:13.917 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.917 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.917 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:23:13.917 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.898707+0000", 2024-09-21T10:23:13.917 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003062614 2024-09-21T10:23:13.917 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.917 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.917 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:13.917 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.900944+0000", 2024-09-21T10:23:13.917 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022366909999999999 2024-09-21T10:23:13.917 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:13.917 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:13.918 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:13.918 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.918 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.918 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46025.0:76633 9.0 e311/300 9:03640e8c:::benchmark_data_smithi098_254407_object76632:head v 311'4825, mlcod=311'4823)", 2024-09-21T10:23:13.918 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:23:13.901043+0000", 2024-09-21T10:23:13.918 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.00070625699999999996, 2024-09-21T10:23:13.918 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00091062900000000004, 2024-09-21T10:23:13.918 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:13.918 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:23:13.918 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:13.918 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.918 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:13.918 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.901043+0000", 2024-09-21T10:23:13.919 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:13.919 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.919 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.919 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:13.919 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.901043+0000", 2024-09-21T10:23:13.919 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:13.919 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.919 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.919 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:13.919 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.901043+0000", 2024-09-21T10:23:13.919 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:23:13.919 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.920 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.920 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:13.920 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.901076+0000", 2024-09-21T10:23:13.920 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3000999999999998e-05 2024-09-21T10:23:13.920 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.920 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.920 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:13.920 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.901092+0000", 2024-09-21T10:23:13.920 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6413999999999999e-05 2024-09-21T10:23:13.920 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.920 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.920 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:13.920 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.901098+0000", 2024-09-21T10:23:13.921 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7660000000000004e-06 2024-09-21T10:23:13.921 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:13.921 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.901151+0000", 2024-09-21T10:23:13.921 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3742999999999997e-05 2024-09-21T10:23:13.921 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:13.921 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.901165+0000", 2024-09-21T10:23:13.921 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3396e-05 2024-09-21T10:23:13.921 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:13.921 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:13.922 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:13.922 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.922 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46025.0:76642 9.4 e311/300 9:24c03845:::benchmark_data_smithi098_254407_object76641:head v 311'4769, mlcod=311'4767)", 2024-09-21T10:23:13.922 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:23:13.901693+0000", 2024-09-21T10:23:13.922 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 5.6478999999999999e-05, 2024-09-21T10:23:13.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00031426400000000003, 2024-09-21T10:23:13.922 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:13.922 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:23:13.922 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:13.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.922 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:13.922 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.901693+0000", 2024-09-21T10:23:13.923 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:13.923 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.923 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:13.923 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.901693+0000", 2024-09-21T10:23:13.923 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:13.923 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.923 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:13.923 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.901692+0000", 2024-09-21T10:23:13.923 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:23:13.923 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.923 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:13.924 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.901760+0000", 2024-09-21T10:23:13.924 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7886999999999996e-05 2024-09-21T10:23:13.924 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.924 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.924 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:13.924 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.901775+0000", 2024-09-21T10:23:13.924 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4972e-05 2024-09-21T10:23:13.924 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.924 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.924 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:13.924 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.901781+0000", 2024-09-21T10:23:13.924 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4099999999999999e-06 2024-09-21T10:23:13.924 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.924 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.925 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:13.925 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.901841+0000", 2024-09-21T10:23:13.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9867999999999997e-05 2024-09-21T10:23:13.925 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:13.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:13.925 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:13.925 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:23:13.901862+0000", 2024-09-21T10:23:13.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1537999999999999e-05 2024-09-21T10:23:13.925 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:13.925 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:13.925 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:13.925 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:13.926 INFO:teuthology.orchestra.run.smithi196.stdout: ], 2024-09-21T10:23:13.926 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 4 2024-09-21T10:23:13.926 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:23:13.926 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:13 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T10:23:13.926 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:13 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T10:23:13.981 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:14.017 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:14.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:14.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:14.119 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:14.157 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:14.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:14.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:14.235 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 41 16 77092 77076 117.424 79.75 0.00534057 0.00851155 2024-09-21T10:23:14.259 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:14.301 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:14.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:14.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:14.403 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:14.437 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:14.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:14.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:14.538 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:14.563 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:14.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:14.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:14.664 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:14.668 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:23:14.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:14 smithi098 ceph-mon[97580]: pgmap v1029: 89 pgs: 89 active+clean; 4.4 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-21T10:23:14.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:14 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T10:23:14.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:14 smithi098 ceph-mon[97580]: osdmap e311: 8 total, 8 up, 6 in 2024-09-21T10:23:14.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:14 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/151448843' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:23:14.683 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:14 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:23:14.683 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:14 smithi098 ceph-mon[100914]: pgmap v1029: 89 pgs: 89 active+clean; 4.4 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-21T10:23:14.683 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:14 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T10:23:14.683 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:14 smithi098 ceph-mon[100914]: osdmap e311: 8 total, 8 up, 6 in 2024-09-21T10:23:14.683 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:14 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/151448843' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:23:14.683 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:14 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:23:14.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:14.702 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:14.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:14.803 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:14.840 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:14.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:14.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:14.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:14.984 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:14.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:14.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:15.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:14 smithi196 ceph-mon[84784]: pgmap v1029: 89 pgs: 89 active+clean; 4.4 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-21T10:23:15.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:14 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T10:23:15.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:14 smithi196 ceph-mon[84784]: osdmap e311: 8 total, 8 up, 6 in 2024-09-21T10:23:15.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:14 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/151448843' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:23:15.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:14 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:23:15.087 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:15.128 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:15.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:15.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:15.230 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:15.235 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 42 16 78326 78310 116.465 77.125 0.00900889 0.00858123 2024-09-21T10:23:15.271 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:15.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:15.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:15.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:23:15.373 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:15.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:15.424 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:15.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:15.525 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:15.579 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:15.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:15.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:15.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:15.731 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:15.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:15.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:15.791 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:23:15.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:15 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:15.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:15 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:23:15.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:15 smithi098 ceph-mon[100914]: osdmap e312: 8 total, 8 up, 6 in 2024-09-21T10:23:15.813 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:15 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:15.813 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:15 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:23:15.813 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:15 smithi098 ceph-mon[97580]: osdmap e312: 8 total, 8 up, 6 in 2024-09-21T10:23:15.833 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:15.875 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:15.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:15.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:15.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:15 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:15.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:15 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:23:15.929 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:15 smithi196 ceph-mon[84784]: osdmap e312: 8 total, 8 up, 6 in 2024-09-21T10:23:15.977 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:16.020 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:16.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:16.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:16.121 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:16.161 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:16.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:16.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:16.235 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 43 16 79601 79585 115.61 79.6875 0.00497485 0.00864507 2024-09-21T10:23:16.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:16.293 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:16.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:16.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:16.395 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:16.427 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:16.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:16.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:16.529 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:16.567 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:16.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:16.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:16.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:16.705 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:16.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:16.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:16.808 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:16.843 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:16.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:16.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:16.845 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:23:16.845 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:23:16.845 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:23:16.845 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:23:16.845 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:23:16.845 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:23:16.846 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:23:16.846 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:23:16.846 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:23:16.846 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:23:16.846 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:23:16.846 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:23:16.846 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:23:16.846 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:23:16.847 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:23:16.847 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:23:16.945 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:16.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:16.977 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:16.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:16.992 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:16 smithi196 ceph-mon[84784]: pgmap v1032: 99 pgs: 10 unknown, 89 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T10:23:16.992 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:16 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1823252818' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:17.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:17.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:16 smithi098 ceph-mon[97580]: pgmap v1032: 99 pgs: 10 unknown, 89 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T10:23:17.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:16 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1823252818' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:17.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:16 smithi098 ceph-mon[100914]: pgmap v1032: 99 pgs: 10 unknown, 89 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T10:23:17.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:16 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1823252818' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:17.213 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:17.235 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 44 16 80824 80808 114.72 76.4375 0.00820453 0.00871175 2024-09-21T10:23:17.248 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:17.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:17.349 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:17.389 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:17.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:17.491 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:17.530 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:17.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:17.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:23:17.631 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:17.669 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:17.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:17.728 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:23:17.728 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:23:17.771 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:17.806 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7] out_osds: [4, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:23:17.806 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:23:17.806 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-21T10:23:17.807 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-21T10:23:17.814 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:17.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:17.916 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:17.952 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:17.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:18.053 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:18.098 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:18.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:18.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:18.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:18.222 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:18.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:18.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:18.235 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 45 16 82035 82019 113.853 75.6875 0.00340622 0.00877021 2024-09-21T10:23:18.324 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:18.345 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:18.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:18.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:18.447 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:18.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:18.480 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:18.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:18.581 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:18.618 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:18.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:18.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:18.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:18.748 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:18.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:18.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:18.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:18.881 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:18.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:18.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:18.983 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:19.014 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:19.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:19.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:18 smithi098 ceph-mon[97580]: pgmap v1033: 99 pgs: 10 unknown, 89 active+clean; 4.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T10:23:19.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:18 smithi098 ceph-mon[100914]: pgmap v1033: 99 pgs: 10 unknown, 89 active+clean; 4.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T10:23:19.116 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:19.138 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:18 smithi196 ceph-mon[84784]: pgmap v1033: 99 pgs: 10 unknown, 89 active+clean; 4.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T10:23:19.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:19.235 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 46 16 82996 82980 112.684 60.0625 0.0125645 0.00886961 2024-09-21T10:23:19.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:19.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.278 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:19.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:19.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:19.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.414 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:19.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:19.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:19.543 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:19.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:19.646 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:19.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.681 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:19.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:19.782 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:19.819 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:19.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:19.840 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:19 smithi196 ceph-mon[84784]: 1.0 scrub starts 2024-09-21T10:23:19.840 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:19 smithi196 ceph-mon[84784]: 1.0 scrub ok 2024-09-21T10:23:19.841 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:19 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T10:23:19.883 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:23:19.922 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:19.980 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:19.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:20.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:20.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:19 smithi098 ceph-mon[97580]: 1.0 scrub starts 2024-09-21T10:23:20.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:19 smithi098 ceph-mon[97580]: 1.0 scrub ok 2024-09-21T10:23:20.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:19 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T10:23:20.086 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:19 smithi098 ceph-mon[100914]: 1.0 scrub starts 2024-09-21T10:23:20.086 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:19 smithi098 ceph-mon[100914]: 1.0 scrub ok 2024-09-21T10:23:20.086 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:19 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T10:23:20.126 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:20.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:20.226 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:20.228 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:20.235 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:20.235 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 47 16 84004 83988 111.627 63 0.0124544 0.00895318 2024-09-21T10:23:20.281 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:20.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:20.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:20.399 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:23:20.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.412 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:20.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:20.513 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:20.525 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:23:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:23:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:23:20.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.546 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:20.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:20.647 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:20.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.683 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:20.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:20.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:20 smithi196 ceph-mon[84784]: pgmap v1034: 99 pgs: 99 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:23:20.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:20 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-21T10:23:20.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:20 smithi196 ceph-mon[84784]: osdmap e313: 8 total, 8 up, 6 in 2024-09-21T10:23:20.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:20 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:23:20.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:20.815 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:20.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:20.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:20.949 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:20.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:21.051 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:21.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:21.085 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:21.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:21.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:23:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:23:20] "GET /metrics HTTP/1.1" 200 36535 "" "Prometheus/2.51.0" 2024-09-21T10:23:21.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:20 smithi098 ceph-mon[100914]: pgmap v1034: 99 pgs: 99 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:23:21.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:20 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-21T10:23:21.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:20 smithi098 ceph-mon[100914]: osdmap e313: 8 total, 8 up, 6 in 2024-09-21T10:23:21.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:20 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:23:21.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:20 smithi098 ceph-mon[97580]: pgmap v1034: 99 pgs: 99 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:23:21.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:20 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-21T10:23:21.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:20 smithi098 ceph-mon[97580]: osdmap e313: 8 total, 8 up, 6 in 2024-09-21T10:23:21.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:20 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:23:21.186 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:21.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:21.226 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:21.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:21.235 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 48 16 85137 85121 110.777 70.8125 0.00940139 0.00902257 2024-09-21T10:23:21.326 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:21.362 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:21.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:21.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:21.464 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:21.499 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:21.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:21.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:21.600 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:21.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:21.635 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:21.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:21.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:21.760 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:21.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:21.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:21.862 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:21.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:21.884 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:21.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:21.953 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:21 smithi196 ceph-mon[84784]: osdmap e314: 8 total, 8 up, 6 in 2024-09-21T10:23:21.984 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:22.025 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:22.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:22.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:21 smithi098 ceph-mon[97580]: osdmap e314: 8 total, 8 up, 6 in 2024-09-21T10:23:22.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:21 smithi098 ceph-mon[100914]: osdmap e314: 8 total, 8 up, 6 in 2024-09-21T10:23:22.127 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:22.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:23:22.161 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:22.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:22.236 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 49 16 86212 86196 109.888 67.1875 0.00829085 0.00909567 2024-09-21T10:23:22.263 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:22.305 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:22.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:22.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:22.453 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:22.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:22.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:22.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.588 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:22.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:22.689 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:22.728 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:22.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:22.830 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:22.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:22 smithi098 ceph-mon[97580]: 1.5 scrub starts 2024-09-21T10:23:22.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:22 smithi098 ceph-mon[97580]: 1.5 scrub ok 2024-09-21T10:23:22.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:22 smithi098 ceph-mon[97580]: 1.1 scrub starts 2024-09-21T10:23:22.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:22 smithi098 ceph-mon[97580]: 1.1 scrub ok 2024-09-21T10:23:22.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:22 smithi098 ceph-mon[97580]: pgmap v1037: 99 pgs: 99 active+clean; 5.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 57 MiB/s wr, 910 op/s 2024-09-21T10:23:22.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:22 smithi098 ceph-mon[97580]: 1.4 scrub starts 2024-09-21T10:23:22.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:22 smithi098 ceph-mon[97580]: 1.4 scrub ok 2024-09-21T10:23:22.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:22.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:22 smithi098 ceph-mon[100914]: 1.5 scrub starts 2024-09-21T10:23:22.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:22 smithi098 ceph-mon[100914]: 1.5 scrub ok 2024-09-21T10:23:22.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:22 smithi098 ceph-mon[100914]: 1.1 scrub starts 2024-09-21T10:23:22.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:22 smithi098 ceph-mon[100914]: 1.1 scrub ok 2024-09-21T10:23:22.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:22 smithi098 ceph-mon[100914]: pgmap v1037: 99 pgs: 99 active+clean; 5.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 57 MiB/s wr, 910 op/s 2024-09-21T10:23:22.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:22 smithi098 ceph-mon[100914]: 1.4 scrub starts 2024-09-21T10:23:22.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:22 smithi098 ceph-mon[100914]: 1.4 scrub ok 2024-09-21T10:23:22.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:22.859 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:22.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:22.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:22.994 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:22.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:23.049 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:22 smithi196 ceph-mon[84784]: 1.5 scrub starts 2024-09-21T10:23:23.049 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:22 smithi196 ceph-mon[84784]: 1.5 scrub ok 2024-09-21T10:23:23.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:22 smithi196 ceph-mon[84784]: 1.1 scrub starts 2024-09-21T10:23:23.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:22 smithi196 ceph-mon[84784]: 1.1 scrub ok 2024-09-21T10:23:23.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:22 smithi196 ceph-mon[84784]: pgmap v1037: 99 pgs: 99 active+clean; 5.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 57 MiB/s wr, 910 op/s 2024-09-21T10:23:23.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:22 smithi196 ceph-mon[84784]: 1.4 scrub starts 2024-09-21T10:23:23.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:22 smithi196 ceph-mon[84784]: 1.4 scrub ok 2024-09-21T10:23:23.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:23.097 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:23.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:23.139 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:23.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:23.236 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 50 16 87322 87306 109.078 69.375 0.0125824 0.00916282 2024-09-21T10:23:23.240 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:23.275 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:23.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:23.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:23.377 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:23.405 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:23.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:23.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:23.506 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:23.537 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:23.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:23.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:23.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:23.641 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:23:23.641 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:23:23.680 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:23.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:23.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:23.782 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:23.814 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:23.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:23.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:23 smithi196 ceph-mon[84784]: 1.3 scrub starts 2024-09-21T10:23:23.993 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:23 smithi196 ceph-mon[84784]: 1.3 scrub ok 2024-09-21T10:23:23.993 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:23 smithi196 ceph-mon[84784]: 1.9 scrub starts 2024-09-21T10:23:23.993 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:23 smithi196 ceph-mon[84784]: 1.9 scrub ok 2024-09-21T10:23:23.994 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:23.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:23 smithi098 ceph-mon[97580]: 1.3 scrub starts 2024-09-21T10:23:23.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:23 smithi098 ceph-mon[97580]: 1.3 scrub ok 2024-09-21T10:23:23.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:23 smithi098 ceph-mon[97580]: 1.9 scrub starts 2024-09-21T10:23:23.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:23 smithi098 ceph-mon[97580]: 1.9 scrub ok 2024-09-21T10:23:23.995 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:23 smithi098 ceph-mon[100914]: 1.3 scrub starts 2024-09-21T10:23:23.996 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:23 smithi098 ceph-mon[100914]: 1.3 scrub ok 2024-09-21T10:23:23.996 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:23 smithi098 ceph-mon[100914]: 1.9 scrub starts 2024-09-21T10:23:23.996 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:23 smithi098 ceph-mon[100914]: 1.9 scrub ok 2024-09-21T10:23:24.026 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:24.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:24.128 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:24.153 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:24.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:24.236 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 51 16 88486 88470 108.365 72.75 0.0130463 0.00922296 2024-09-21T10:23:24.254 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:24.293 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:24.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:24.395 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:24.419 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:24.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:24.521 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:24.555 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:24.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:24.657 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:24.692 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:24.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:24.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:24.835 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:24.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:24 smithi098 ceph-mon[97580]: pgmap v1038: 99 pgs: 99 active+clean; 5.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-21T10:23:24.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:24 smithi098 ceph-mon[100914]: pgmap v1038: 99 pgs: 99 active+clean; 5.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-21T10:23:24.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:24.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.975 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:24.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:25.076 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:25.113 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:25.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:25.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:25.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:24 smithi196 ceph-mon[84784]: pgmap v1038: 99 pgs: 99 active+clean; 5.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-21T10:23:25.216 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:25.236 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 52 16 89701 89685 107.742 75.9375 0.00918505 0.00927682 2024-09-21T10:23:25.247 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:25.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:25.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:25.349 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:25.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:25.384 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:25.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:25.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:25.525 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:25.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:25.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:25.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:25.667 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:25.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:25.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:25.769 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:25.791 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:23:25.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:25.823 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:25.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:25.912 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:25 smithi098 ceph-mon[97580]: 1.6 scrub starts 2024-09-21T10:23:25.912 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:25 smithi098 ceph-mon[97580]: 1.6 scrub ok 2024-09-21T10:23:25.912 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:25 smithi098 ceph-mon[100914]: 1.6 scrub starts 2024-09-21T10:23:25.912 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:25 smithi098 ceph-mon[100914]: 1.6 scrub ok 2024-09-21T10:23:25.924 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:25.984 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:25.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:25.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:26.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:26.113 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:26.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:26.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:25 smithi196 ceph-mon[84784]: 1.6 scrub starts 2024-09-21T10:23:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:25 smithi196 ceph-mon[84784]: 1.6 scrub ok 2024-09-21T10:23:26.215 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:26.236 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 53 16 90822 90806 107.031 70.0625 0.0128525 0.00933778 2024-09-21T10:23:26.257 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:26.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:26.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:26.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.404 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:26.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:26.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:26.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.552 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:26.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:26.653 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:26.679 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:23:26.682 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:26.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:26.784 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:26.820 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:26.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:26.922 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:26.954 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:26.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:27.056 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:26 smithi098 ceph-mon[97580]: pgmap v1039: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-09-21T10:23:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:26 smithi098 ceph-mon[97580]: 1.2 scrub starts 2024-09-21T10:23:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:26 smithi098 ceph-mon[97580]: 1.2 scrub ok 2024-09-21T10:23:27.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:26 smithi098 ceph-mon[100914]: pgmap v1039: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-09-21T10:23:27.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:26 smithi098 ceph-mon[100914]: 1.2 scrub starts 2024-09-21T10:23:27.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:26 smithi098 ceph-mon[100914]: 1.2 scrub ok 2024-09-21T10:23:27.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:27.113 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:27.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:26 smithi196 ceph-mon[84784]: pgmap v1039: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-09-21T10:23:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:26 smithi196 ceph-mon[84784]: 1.2 scrub starts 2024-09-21T10:23:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:26 smithi196 ceph-mon[84784]: 1.2 scrub ok 2024-09-21T10:23:27.214 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:27.236 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 54 16 92015 91999 106.43 74.5625 0.01616 0.00939085 2024-09-21T10:23:27.255 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:27.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:27.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:27.357 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:27.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:27.388 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:27.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:27.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:27.535 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:27.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:27.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:27.637 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:27.678 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:27.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:27.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:27.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:27.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:27.817 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:27.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:27.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:27 smithi098 ceph-mon[97580]: 1.a scrub starts 2024-09-21T10:23:27.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:27 smithi098 ceph-mon[97580]: 1.a scrub ok 2024-09-21T10:23:27.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:27 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:27.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:27 smithi098 ceph-mon[100914]: 1.a scrub starts 2024-09-21T10:23:27.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:27 smithi098 ceph-mon[100914]: 1.a scrub ok 2024-09-21T10:23:27.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:27 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:27.918 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:27.955 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:27.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:27.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:28.056 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:28.091 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:28.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:28.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:27 smithi196 ceph-mon[84784]: 1.a scrub starts 2024-09-21T10:23:28.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:27 smithi196 ceph-mon[84784]: 1.a scrub ok 2024-09-21T10:23:28.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:27 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:28.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:28.228 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:28.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:28.236 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 55 16 93217 93201 105.861 75.125 0.0102715 0.00944148 2024-09-21T10:23:28.330 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:28.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.373 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:28.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:28.417 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:28.475 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:28.523 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:28.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:28.624 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:28.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.661 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:28.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:28.744 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:23:28.762 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:28.802 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:28.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:28.879 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:28 smithi098 ceph-mon[97580]: 1.8 scrub starts 2024-09-21T10:23:28.879 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:28 smithi098 ceph-mon[97580]: 1.8 scrub ok 2024-09-21T10:23:28.879 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:28 smithi098 ceph-mon[97580]: pgmap v1040: 99 pgs: 99 active+clean; 5.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 58 MiB/s wr, 927 op/s 2024-09-21T10:23:28.879 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:28 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2462078028' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:28.879 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:28 smithi098 ceph-mon[100914]: 1.8 scrub starts 2024-09-21T10:23:28.880 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:28 smithi098 ceph-mon[100914]: 1.8 scrub ok 2024-09-21T10:23:28.880 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:28 smithi098 ceph-mon[100914]: pgmap v1040: 99 pgs: 99 active+clean; 5.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 58 MiB/s wr, 927 op/s 2024-09-21T10:23:28.880 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:28 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2462078028' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:28 smithi196 ceph-mon[84784]: 1.8 scrub starts 2024-09-21T10:23:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:28 smithi196 ceph-mon[84784]: 1.8 scrub ok 2024-09-21T10:23:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:28 smithi196 ceph-mon[84784]: pgmap v1040: 99 pgs: 99 active+clean; 5.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 58 MiB/s wr, 927 op/s 2024-09-21T10:23:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:28 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2462078028' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:28.905 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:28.941 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:28.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:29.042 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:29.071 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:29.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:29.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:29.172 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:29.216 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:29.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:29.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:29.236 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 56 16 94334 94318 105.217 69.8125 0.0154892 0.00949974 2024-09-21T10:23:29.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:29.361 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:29.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:29.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:29.463 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:29.524 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:29.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:29.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:29.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:29.660 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:29.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:29.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:29.762 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:29.797 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:29.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:29.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:29.830 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:29 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:23:29.830 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:29 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:23:29.830 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:29 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2462078028' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:23:29.830 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:29 smithi196 ceph-mon[84784]: osdmap e315: 8 total, 8 up, 6 in 2024-09-21T10:23:29.837 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:23:29.837 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:23:29.899 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:29.932 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:29.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:29.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:30.034 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:30.076 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:30.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:30.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:30.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:29 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:23:30.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:29 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:23:30.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:29 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2462078028' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:23:30.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:29 smithi098 ceph-mon[97580]: osdmap e315: 8 total, 8 up, 6 in 2024-09-21T10:23:30.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:29 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:23:30.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:29 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:23:30.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:29 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2462078028' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:23:30.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:29 smithi098 ceph-mon[100914]: osdmap e315: 8 total, 8 up, 6 in 2024-09-21T10:23:30.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:30.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:30.217 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:30.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:30.240 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 57 16 95520 95504 104.671 74.125 0.0129626 0.00954838 2024-09-21T10:23:30.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:30.354 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:30.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:30.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:30.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:23:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:23:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:23:30.456 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:30.491 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:30.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:30.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:30.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:23:30.592 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:30.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:30.637 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:30.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:30.738 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:30.770 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:23:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:23:30] "GET /metrics HTTP/1.1" 200 36532 "" "Prometheus/2.51.0" 2024-09-21T10:23:30.774 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:30.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:30.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:30.875 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:30.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:30.946 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:30.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:31.047 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:31.096 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:31.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:30 smithi098 ceph-mon[100914]: pgmap v1042: 99 pgs: 99 active+clean; 5.7 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 77 MiB/s wr, 1.22k op/s 2024-09-21T10:23:31.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:30 smithi098 ceph-mon[97580]: pgmap v1042: 99 pgs: 99 active+clean; 5.7 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 77 MiB/s wr, 1.22k op/s 2024-09-21T10:23:31.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:31.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:31.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:30 smithi196 ceph-mon[84784]: pgmap v1042: 99 pgs: 99 active+clean; 5.7 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 77 MiB/s wr, 1.22k op/s 2024-09-21T10:23:31.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:31.237 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:31.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:31.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:31.239 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 58 16 96653 96637 104.083 70.8125 0.0149619 0.00960272 2024-09-21T10:23:31.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:31.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:31.391 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:31.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:31.402 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:23:31.492 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:31.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:31.533 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:31.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:31.634 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:31.665 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:31.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:31.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:31.766 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:31.803 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:31 smithi196 ceph-mon[84784]: 1.7 scrub starts 2024-09-21T10:23:31.803 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:31 smithi196 ceph-mon[84784]: 1.7 scrub ok 2024-09-21T10:23:31.803 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:31 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:31.803 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:31 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:31.817 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-21T10:23:31.821 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:31.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:31.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:31.922 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:31.961 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:31.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:31.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:32.062 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:32.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:31 smithi098 ceph-mon[97580]: 1.7 scrub starts 2024-09-21T10:23:32.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:31 smithi098 ceph-mon[97580]: 1.7 scrub ok 2024-09-21T10:23:32.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:31 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:32.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:31 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:32.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:31 smithi098 ceph-mon[100914]: 1.7 scrub starts 2024-09-21T10:23:32.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:31 smithi098 ceph-mon[100914]: 1.7 scrub ok 2024-09-21T10:23:32.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:31 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:32.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:31 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:32.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:32.103 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:32.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:32.204 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:32.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:32.238 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:32.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:32.239 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 59 16 97749 97733 103.48 68.5 0.0135221 0.00965876 2024-09-21T10:23:32.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:32.375 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:32.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:32.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:32.477 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:32.514 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:32.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:32.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:32.616 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:32.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:32.659 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:32.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:32.760 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:32.797 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:32.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:32.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:32.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:32.919 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:32.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:32.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:33.021 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:33.042 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:33.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:33.046 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:32 smithi098 ceph-mon[97580]: pgmap v1043: 99 pgs: 99 active+clean; 5.7 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-21T10:23:33.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:32 smithi098 ceph-mon[100914]: pgmap v1043: 99 pgs: 99 active+clean; 5.7 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-21T10:23:33.092 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:32 smithi196 ceph-mon[84784]: pgmap v1043: 99 pgs: 99 active+clean; 5.7 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-21T10:23:33.144 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:33.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:23:33.214 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:33.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:33.239 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:23:33.238721+0000 min lat: 0.00136025 max lat: 0.247537 avg lat: 0.00972844 2024-09-21T10:23:33.240 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:23:33.240 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 60 16 98696 98680 102.742 59.1875 0.0158514 0.00972844 2024-09-21T10:23:33.316 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:33.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.369 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:33.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:33.470 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:33.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:33.598 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:33.638 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:33.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:33.740 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:33.787 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:33.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:33.800 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:23:33.800 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:23:33.889 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:33.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.938 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:33.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:34.039 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:34.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:34.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:34.173 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:34.207 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:34.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:34.240 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 61 16 99786 99770 102.175 68.125 0.0225703 0.00978274 2024-09-21T10:23:34.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:34.342 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:34.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:34.444 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:34.485 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:34.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:34.521 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:23:34.587 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:34.621 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:34.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:34.723 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:34.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.760 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:34.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:34.861 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:34.913 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:34.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:34.988 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:23:35.015 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:35.041 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:35.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:35.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:35.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:34 smithi098 ceph-mon[97580]: pgmap v1044: 99 pgs: 99 active+clean; 5.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 72 MiB/s wr, 1.14k op/s 2024-09-21T10:23:35.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:34 smithi098 ceph-mon[100914]: pgmap v1044: 99 pgs: 99 active+clean; 5.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 72 MiB/s wr, 1.14k op/s 2024-09-21T10:23:35.120 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:34 smithi196 ceph-mon[84784]: pgmap v1044: 99 pgs: 99 active+clean; 5.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 72 MiB/s wr, 1.14k op/s 2024-09-21T10:23:35.143 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:35.165 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:35.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:35.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:35.240 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 62 16 100690 100674 101.438 56.5 0.0361348 0.00985188 2024-09-21T10:23:35.267 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:35.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:35.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:35.351 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:23:35.352 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:23:35.352 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:23:35.352 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:23:35.352 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:23:35.352 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:23:35.352 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:23:35.352 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:23:35.365 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:23:35.366 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:23:35.396 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:23:35.396 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:23:35.405 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:35.431 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:23:35.431 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:23:35.470 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:23:35.471 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:23:35.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:35.490 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:35.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:35.591 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:35.622 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:35.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:35.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:35.724 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:35.763 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:35.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:35.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:35.856 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:35.864 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:35.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:35.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:36.011 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:36.045 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:36.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:36.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:36.065 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:35 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3670938309' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:36.065 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:35 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3670938309' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:36.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:35 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3670938309' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:36.147 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:36.182 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:36.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:36.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:36.240 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 63 16 101802 101786 100.931 69.5 0.0167705 0.00990292 2024-09-21T10:23:36.283 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:36.330 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:36.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:36.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:36.432 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:36.476 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:36.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:36.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:36.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:36.624 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:36.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:36.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:36.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:36.759 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:36.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:36.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:36.861 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:36.890 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:36.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:36.924 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:23:36.925 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-21T10:23:36.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-21T10:23:36.925 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:23:36.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.925 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:55406 7.11 7:8973fea9:::benchmark_data_smithi098_197834_object55405:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-21T10:23:36.925 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:13:43.756897+0000", 2024-09-21T10:23:36.925 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 593.16186217500001, 2024-09-21T10:23:36.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2483234169999999, 2024-09-21T10:23:36.925 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:36.925 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:36.925 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:23:36.926 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:23:36.926 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:23:36.926 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 55406 2024-09-21T10:23:36.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.926 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:36.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.926 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:36.926 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.756897+0000", 2024-09-21T10:23:36.926 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:36.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.927 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:36.927 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.756897+0000", 2024-09-21T10:23:36.927 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:36.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.927 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.927 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:36.927 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.756895+0000", 2024-09-21T10:23:36.927 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:23:36.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.927 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.927 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:36.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.757047+0000", 2024-09-21T10:23:36.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000151359 2024-09-21T10:23:36.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:36.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.757060+0000", 2024-09-21T10:23:36.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3158e-05 2024-09-21T10:23:36.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:36.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.757064+0000", 2024-09-21T10:23:36.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7520000000000004e-06 2024-09-21T10:23:36.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.757111+0000", 2024-09-21T10:23:36.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6539000000000003e-05 2024-09-21T10:23:36.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:36.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.757121+0000", 2024-09-21T10:23:36.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0447e-05 2024-09-21T10:23:36.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.930 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.930 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677439+0000", 2024-09-21T10:23:36.930 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.92031749100000004 2024-09-21T10:23:36.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.930 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.930 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:36.930 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677449+0000", 2024-09-21T10:23:36.930 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0059e-05 2024-09-21T10:23:36.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.930 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.930 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.930 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:45.996507+0000", 2024-09-21T10:23:36.931 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3190579410000001 2024-09-21T10:23:36.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.931 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.931 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:36.931 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:45.996650+0000", 2024-09-21T10:23:36.931 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014312799999999999 2024-09-21T10:23:36.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.931 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.931 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:23:36.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:45.996835+0000", 2024-09-21T10:23:36.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00018520099999999999 2024-09-21T10:23:36.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:23:36.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.000187+0000", 2024-09-21T10:23:36.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0033520080000000001 2024-09-21T10:23:36.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:36.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.000974+0000", 2024-09-21T10:23:36.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00078717999999999995 2024-09-21T10:23:36.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:36.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.004097+0000", 2024-09-21T10:23:36.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0031225979999999999 2024-09-21T10:23:36.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:36.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.004130+0000", 2024-09-21T10:23:36.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3303999999999998e-05 2024-09-21T10:23:36.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:36.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.005220+0000", 2024-09-21T10:23:36.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010896840000000001 2024-09-21T10:23:36.936 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:36.936 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:36.936 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:36.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.936 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:55408 7.11 7:898dbbe2:::benchmark_data_smithi098_197834_object55407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-21T10:23:36.937 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:13:43.757912+0000", 2024-09-21T10:23:36.937 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 593.16084683600002, 2024-09-21T10:23:36.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2481479279999999, 2024-09-21T10:23:36.937 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:36.937 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:36.937 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:23:36.937 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:23:36.937 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:23:36.937 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 55408 2024-09-21T10:23:36.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.937 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:36.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:36.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.757912+0000", 2024-09-21T10:23:36.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:36.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:36.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.757912+0000", 2024-09-21T10:23:36.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:36.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.939 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:36.939 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.757911+0000", 2024-09-21T10:23:36.939 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:23:36.939 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.939 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.939 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:36.939 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.758019+0000", 2024-09-21T10:23:36.939 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107135 2024-09-21T10:23:36.939 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.939 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.939 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:36.940 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.758025+0000", 2024-09-21T10:23:36.940 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9639999999999997e-06 2024-09-21T10:23:36.940 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.940 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.940 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:36.940 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.758029+0000", 2024-09-21T10:23:36.940 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5279999999999999e-06 2024-09-21T10:23:36.940 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.940 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.940 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.940 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.758096+0000", 2024-09-21T10:23:36.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6988000000000003e-05 2024-09-21T10:23:36.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:36.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.758102+0000", 2024-09-21T10:23:36.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3029999999999996e-06 2024-09-21T10:23:36.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677615+0000", 2024-09-21T10:23:36.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.91951281200000001 2024-09-21T10:23:36.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:36.942 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677623+0000", 2024-09-21T10:23:36.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3410000000000003e-06 2024-09-21T10:23:36.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.942 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:45.998401+0000", 2024-09-21T10:23:36.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3207773190000001 2024-09-21T10:23:36.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.943 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.943 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:36.943 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:45.998492+0000", 2024-09-21T10:23:36.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1407000000000001e-05 2024-09-21T10:23:36.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.943 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.943 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:23:36.943 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:45.998589+0000", 2024-09-21T10:23:36.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6668000000000003e-05 2024-09-21T10:23:36.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:23:36.944 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.000927+0000", 2024-09-21T10:23:36.944 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023384780000000002 2024-09-21T10:23:36.944 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:36.944 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.001022+0000", 2024-09-21T10:23:36.944 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5004e-05 2024-09-21T10:23:36.944 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.945 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:36.945 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.005274+0000", 2024-09-21T10:23:36.945 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042519480000000002 2024-09-21T10:23:36.945 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.945 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.945 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:36.945 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.005313+0000", 2024-09-21T10:23:36.945 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8371999999999998e-05 2024-09-21T10:23:36.945 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.945 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.945 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:36.946 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.006060+0000", 2024-09-21T10:23:36.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00074726199999999995 2024-09-21T10:23:36.946 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:36.946 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:36.946 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:36.946 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.946 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.946 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:55450 7.11 7:8a8f55f9:::benchmark_data_smithi098_197834_object55449:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-21T10:23:36.946 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:13:43.786722+0000", 2024-09-21T10:23:36.946 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 593.13203672300006, 2024-09-21T10:23:36.947 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2212804739999998, 2024-09-21T10:23:36.947 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:36.947 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:36.947 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:23:36.947 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:23:36.947 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:23:36.947 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 55450 2024-09-21T10:23:36.947 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.947 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:36.947 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:36.948 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.786722+0000", 2024-09-21T10:23:36.948 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:36.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:36.948 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.786722+0000", 2024-09-21T10:23:36.948 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:36.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:36.948 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.786721+0000", 2024-09-21T10:23:36.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:23:36.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.949 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:36.949 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.786835+0000", 2024-09-21T10:23:36.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000113991 2024-09-21T10:23:36.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.949 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:36.949 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.786845+0000", 2024-09-21T10:23:36.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9280000000000005e-06 2024-09-21T10:23:36.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.950 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:36.950 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.786851+0000", 2024-09-21T10:23:36.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0900000000000001e-06 2024-09-21T10:23:36.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.950 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.950 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.786908+0000", 2024-09-21T10:23:36.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7497999999999997e-05 2024-09-21T10:23:36.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:36.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.786916+0000", 2024-09-21T10:23:36.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0290000000000005e-06 2024-09-21T10:23:36.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677650+0000", 2024-09-21T10:23:36.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.89073383100000003 2024-09-21T10:23:36.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:36.952 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677655+0000", 2024-09-21T10:23:36.952 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5210000000000004e-06 2024-09-21T10:23:36.952 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.952 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.952 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.952 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:45.999264+0000", 2024-09-21T10:23:36.952 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3216095270000001 2024-09-21T10:23:36.952 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.952 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.952 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:36.952 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:45.999386+0000", 2024-09-21T10:23:36.952 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000121192 2024-09-21T10:23:36.953 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.953 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:23:36.953 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:45.999537+0000", 2024-09-21T10:23:36.953 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015125199999999999 2024-09-21T10:23:36.953 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.953 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:23:36.953 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.001032+0000", 2024-09-21T10:23:36.953 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014955820000000001 2024-09-21T10:23:36.953 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.953 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:36.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.004067+0000", 2024-09-21T10:23:36.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030350500000000001 2024-09-21T10:23:36.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:36.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.007158+0000", 2024-09-21T10:23:36.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030909790000000002 2024-09-21T10:23:36.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:36.955 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.007199+0000", 2024-09-21T10:23:36.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0232999999999999e-05 2024-09-21T10:23:36.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.955 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:36.955 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.008003+0000", 2024-09-21T10:23:36.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00080395800000000002 2024-09-21T10:23:36.955 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:36.955 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:36.955 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:36.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.956 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.956 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:55472 7.11 7:8b981b06:::benchmark_data_smithi098_197834_object55471:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-21T10:23:36.956 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:13:43.797560+0000", 2024-09-21T10:23:36.956 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 593.12119861199994, 2024-09-21T10:23:36.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2112651310000002, 2024-09-21T10:23:36.956 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:36.956 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:36.956 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:23:36.956 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:23:36.956 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:23:36.956 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 55472 2024-09-21T10:23:36.956 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.957 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:36.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.957 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:36.957 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.797560+0000", 2024-09-21T10:23:36.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:36.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.957 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:36.957 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.797560+0000", 2024-09-21T10:23:36.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:36.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.957 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:36.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.797559+0000", 2024-09-21T10:23:36.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:23:36.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:36.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.797597+0000", 2024-09-21T10:23:36.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8365000000000003e-05 2024-09-21T10:23:36.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:36.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.797612+0000", 2024-09-21T10:23:36.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4567e-05 2024-09-21T10:23:36.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.959 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:36.959 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.797622+0000", 2024-09-21T10:23:36.959 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0456e-05 2024-09-21T10:23:36.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.959 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.959 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.797672+0000", 2024-09-21T10:23:36.959 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9434999999999997e-05 2024-09-21T10:23:36.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.959 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:36.960 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.797682+0000", 2024-09-21T10:23:36.960 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0318e-05 2024-09-21T10:23:36.960 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.961 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.961 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677680+0000", 2024-09-21T10:23:36.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.87999812499999996 2024-09-21T10:23:36.961 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.961 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:36.961 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677687+0000", 2024-09-21T10:23:36.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9820000000000002e-06 2024-09-21T10:23:36.961 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.961 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.000218+0000", 2024-09-21T10:23:36.962 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.322531254 2024-09-21T10:23:36.962 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.962 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.962 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:36.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.000307+0000", 2024-09-21T10:23:36.962 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8635999999999996e-05 2024-09-21T10:23:36.962 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.962 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.962 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:23:36.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.000406+0000", 2024-09-21T10:23:36.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8945999999999998e-05 2024-09-21T10:23:36.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.963 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:23:36.963 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.002466+0000", 2024-09-21T10:23:36.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020606090000000001 2024-09-21T10:23:36.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.963 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:36.963 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.007127+0000", 2024-09-21T10:23:36.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046607339999999997 2024-09-21T10:23:36.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.964 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.964 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:36.964 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.008062+0000", 2024-09-21T10:23:36.964 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00093477600000000001 2024-09-21T10:23:36.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.964 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.964 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:36.964 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.008115+0000", 2024-09-21T10:23:36.964 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2685000000000001e-05 2024-09-21T10:23:36.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.964 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.964 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:36.965 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.008825+0000", 2024-09-21T10:23:36.965 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00071077399999999995 2024-09-21T10:23:36.965 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:36.965 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:36.965 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:36.965 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.965 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.965 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:55518 7.11 7:8fe9d082:::benchmark_data_smithi098_197834_object55517:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-21T10:23:36.965 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:13:43.832082+0000", 2024-09-21T10:23:36.965 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 593.08667727900001, 2024-09-21T10:23:36.965 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1816013430000001, 2024-09-21T10:23:36.965 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:36.966 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:36.966 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:23:36.966 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:23:36.966 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:23:36.966 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 55518 2024-09-21T10:23:36.966 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.966 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:36.966 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.966 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:36.966 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.832082+0000", 2024-09-21T10:23:36.966 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:36.967 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.967 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:36.967 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.832082+0000", 2024-09-21T10:23:36.967 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:36.967 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.967 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:36.967 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.832080+0000", 2024-09-21T10:23:36.967 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:23:36.967 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.968 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.968 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:36.968 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.832227+0000", 2024-09-21T10:23:36.968 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014749699999999999 2024-09-21T10:23:36.968 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.968 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.968 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:36.968 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.832240+0000", 2024-09-21T10:23:36.968 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2454e-05 2024-09-21T10:23:36.968 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.968 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.969 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:36.969 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.832246+0000", 2024-09-21T10:23:36.969 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0240000000000003e-06 2024-09-21T10:23:36.969 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.969 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.969 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.969 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.832296+0000", 2024-09-21T10:23:36.969 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0318000000000002e-05 2024-09-21T10:23:36.969 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.969 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.969 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:36.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.832306+0000", 2024-09-21T10:23:36.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0164e-05 2024-09-21T10:23:36.970 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677737+0000", 2024-09-21T10:23:36.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.84543046899999996 2024-09-21T10:23:36.970 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:36.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677743+0000", 2024-09-21T10:23:36.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5470000000000002e-06 2024-09-21T10:23:36.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.971 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.971 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.001047+0000", 2024-09-21T10:23:36.971 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3233035319999999 2024-09-21T10:23:36.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.971 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:36.971 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.001122+0000", 2024-09-21T10:23:36.971 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4595e-05 2024-09-21T10:23:36.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.971 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:23:36.972 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.001216+0000", 2024-09-21T10:23:36.972 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4889000000000006e-05 2024-09-21T10:23:36.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.972 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:23:36.972 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.004025+0000", 2024-09-21T10:23:36.972 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002808166 2024-09-21T10:23:36.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.972 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:36.972 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.008948+0000", 2024-09-21T10:23:36.972 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049232160000000002 2024-09-21T10:23:36.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.973 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.973 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:36.973 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.012478+0000", 2024-09-21T10:23:36.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0035298109999999999 2024-09-21T10:23:36.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.973 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.973 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:36.973 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.012519+0000", 2024-09-21T10:23:36.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1593e-05 2024-09-21T10:23:36.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.973 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.974 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:36.974 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.013683+0000", 2024-09-21T10:23:36.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001163662 2024-09-21T10:23:36.974 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:36.974 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:36.974 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:36.974 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.974 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:55535 7.11 7:8c5840ea:::benchmark_data_smithi098_197834_object55534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-21T10:23:36.974 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:13:43.842295+0000", 2024-09-21T10:23:36.974 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 593.07646408599999, 2024-09-21T10:23:36.975 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1724636749999999, 2024-09-21T10:23:36.975 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:36.975 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:36.975 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:23:36.975 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:23:36.975 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:23:36.975 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 55535 2024-09-21T10:23:36.975 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.975 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:36.975 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.975 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:36.975 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.842295+0000", 2024-09-21T10:23:36.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:36.976 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.976 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:36.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.842295+0000", 2024-09-21T10:23:36.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:36.976 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.976 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:36.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.842294+0000", 2024-09-21T10:23:36.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:23:36.976 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.976 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:36.977 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.842420+0000", 2024-09-21T10:23:36.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012685100000000001 2024-09-21T10:23:36.977 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.977 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:36.977 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.842432+0000", 2024-09-21T10:23:36.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1100999999999999e-05 2024-09-21T10:23:36.977 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.977 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:36.977 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.842437+0000", 2024-09-21T10:23:36.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2970000000000003e-06 2024-09-21T10:23:36.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.978 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.978 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.978 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.842477+0000", 2024-09-21T10:23:36.978 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0611999999999997e-05 2024-09-21T10:23:36.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.978 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.978 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:36.978 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.842487+0000", 2024-09-21T10:23:36.978 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1090000000000004e-06 2024-09-21T10:23:36.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.978 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677764+0000", 2024-09-21T10:23:36.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83527696799999995 2024-09-21T10:23:36.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:36.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677767+0000", 2024-09-21T10:23:36.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.461e-06 2024-09-21T10:23:36.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.980 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.001729+0000", 2024-09-21T10:23:36.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.323962077 2024-09-21T10:23:36.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.980 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:36.980 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.001848+0000", 2024-09-21T10:23:36.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011905600000000001 2024-09-21T10:23:36.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.980 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:23:36.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.001954+0000", 2024-09-21T10:23:36.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105565 2024-09-21T10:23:36.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:23:36.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.005235+0000", 2024-09-21T10:23:36.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003281651 2024-09-21T10:23:36.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:36.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.008996+0000", 2024-09-21T10:23:36.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0037602930000000001 2024-09-21T10:23:36.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:36.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.013729+0000", 2024-09-21T10:23:36.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047330059999999997 2024-09-21T10:23:36.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:36.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.013782+0000", 2024-09-21T10:23:36.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3090000000000002e-05 2024-09-21T10:23:36.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.983 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.983 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:36.983 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.014758+0000", 2024-09-21T10:23:36.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00097667900000000005 2024-09-21T10:23:36.983 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:36.983 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:36.983 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:36.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.983 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.983 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:55537 7.11 7:8b80ba5f:::benchmark_data_smithi098_197834_object55536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-21T10:23:36.983 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:13:43.844111+0000", 2024-09-21T10:23:36.984 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 593.07464772900005, 2024-09-21T10:23:36.984 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.171681317, 2024-09-21T10:23:36.984 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:36.984 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:36.984 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:23:36.984 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:23:36.984 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:23:36.984 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 55537 2024-09-21T10:23:36.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.984 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:36.984 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:36.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844111+0000", 2024-09-21T10:23:36.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:36.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:36.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844111+0000", 2024-09-21T10:23:36.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:36.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:36.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844110+0000", 2024-09-21T10:23:36.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:23:36.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:36.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844326+0000", 2024-09-21T10:23:36.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00021639700000000001 2024-09-21T10:23:36.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:36.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844335+0000", 2024-09-21T10:23:36.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4470000000000006e-06 2024-09-21T10:23:36.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:36.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844340+0000", 2024-09-21T10:23:36.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7999999999999998e-06 2024-09-21T10:23:36.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844381+0000", 2024-09-21T10:23:36.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0890000000000003e-05 2024-09-21T10:23:36.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:36.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844389+0000", 2024-09-21T10:23:36.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4400000000000005e-06 2024-09-21T10:23:36.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677794+0000", 2024-09-21T10:23:36.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83340498200000002 2024-09-21T10:23:36.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:36.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677801+0000", 2024-09-21T10:23:36.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6959999999999999e-06 2024-09-21T10:23:36.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.002504+0000", 2024-09-21T10:23:36.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3247034090000001 2024-09-21T10:23:36.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:36.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.002600+0000", 2024-09-21T10:23:36.990 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6107999999999995e-05 2024-09-21T10:23:36.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.990 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:23:36.990 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.002697+0000", 2024-09-21T10:23:36.990 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6932e-05 2024-09-21T10:23:36.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.990 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:23:36.990 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.005242+0000", 2024-09-21T10:23:36.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025451430000000001 2024-09-21T10:23:36.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.991 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.991 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:36.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.014826+0000", 2024-09-21T10:23:36.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0095836979999999999 2024-09-21T10:23:36.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.991 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.991 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:36.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.014908+0000", 2024-09-21T10:23:36.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2025000000000002e-05 2024-09-21T10:23:36.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.992 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:36.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.014961+0000", 2024-09-21T10:23:36.992 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2889000000000001e-05 2024-09-21T10:23:36.992 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.992 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:36.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.015792+0000", 2024-09-21T10:23:36.992 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00083145400000000003 2024-09-21T10:23:36.992 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:36.993 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:36.993 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:36.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.993 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.993 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:55539 7.11 7:8eebc2e8:::benchmark_data_smithi098_197834_object55538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-21T10:23:36.993 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:13:43.844573+0000", 2024-09-21T10:23:36.993 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 593.07418550800003, 2024-09-21T10:23:36.993 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1757300549999998, 2024-09-21T10:23:36.993 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:36.993 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:36.993 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:23:36.993 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:23:36.994 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:23:36.994 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 55539 2024-09-21T10:23:36.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.994 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:36.994 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.994 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:36.994 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844573+0000", 2024-09-21T10:23:36.994 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:36.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.994 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.994 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:36.994 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844573+0000", 2024-09-21T10:23:36.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:36.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.995 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:36.995 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844573+0000", 2024-09-21T10:23:36.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:23:36.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.995 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:36.995 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844653+0000", 2024-09-21T10:23:36.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0142999999999995e-05 2024-09-21T10:23:36.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.996 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.996 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:36.996 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844664+0000", 2024-09-21T10:23:36.996 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1620999999999999e-05 2024-09-21T10:23:36.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.996 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.996 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:36.996 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844670+0000", 2024-09-21T10:23:36.996 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7570000000000001e-06 2024-09-21T10:23:36.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.996 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.996 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.997 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844710+0000", 2024-09-21T10:23:36.997 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9780999999999997e-05 2024-09-21T10:23:36.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.997 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.997 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:36.997 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.844718+0000", 2024-09-21T10:23:36.997 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0420000000000003e-06 2024-09-21T10:23:36.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.997 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.997 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.997 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677821+0000", 2024-09-21T10:23:36.997 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83310285299999998 2024-09-21T10:23:36.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.998 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:36.998 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677824+0000", 2024-09-21T10:23:36.998 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2959999999999999e-06 2024-09-21T10:23:36.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.998 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:36.998 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.003258+0000", 2024-09-21T10:23:36.998 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.325434038 2024-09-21T10:23:36.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:36.999 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.003396+0000", 2024-09-21T10:23:36.999 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013825799999999999 2024-09-21T10:23:36.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:23:36.999 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.003552+0000", 2024-09-21T10:23:36.999 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015607699999999999 2024-09-21T10:23:36.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:36.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:36.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:23:37.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.007088+0000", 2024-09-21T10:23:37.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0035353870000000001 2024-09-21T10:23:37.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.000 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.000 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.015837+0000", 2024-09-21T10:23:37.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.008748881 2024-09-21T10:23:37.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.000 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.000 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.019617+0000", 2024-09-21T10:23:37.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0037801419999999998 2024-09-21T10:23:37.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.001 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:37.001 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.019673+0000", 2024-09-21T10:23:37.001 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6329000000000002e-05 2024-09-21T10:23:37.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.001 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:37.001 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.020303+0000", 2024-09-21T10:23:37.001 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00063017499999999996 2024-09-21T10:23:37.001 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.001 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:37.001 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.002 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.002 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:55586 7.11 7:8ac9c414:::benchmark_data_smithi098_197834_object55585:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-21T10:23:37.002 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:13:43.877450+0000", 2024-09-21T10:23:37.002 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 593.04130844300005, 2024-09-21T10:23:37.002 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1437025329999999, 2024-09-21T10:23:37.002 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:37.002 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:37.002 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:23:37.002 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:23:37.002 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:23:37.002 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 55586 2024-09-21T10:23:37.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.003 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:37.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:37.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.877450+0000", 2024-09-21T10:23:37.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:37.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.877450+0000", 2024-09-21T10:23:37.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.004 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.004 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:37.004 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.877449+0000", 2024-09-21T10:23:37.004 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:23:37.004 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.004 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.004 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:37.004 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.877505+0000", 2024-09-21T10:23:37.004 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5727999999999999e-05 2024-09-21T10:23:37.004 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.004 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.004 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:37.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.877515+0000", 2024-09-21T10:23:37.005 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9180000000000006e-06 2024-09-21T10:23:37.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.005 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.005 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:37.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.877520+0000", 2024-09-21T10:23:37.005 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3650000000000003e-06 2024-09-21T10:23:37.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.005 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.005 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.877558+0000", 2024-09-21T10:23:37.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8013999999999999e-05 2024-09-21T10:23:37.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.006 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:37.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:43.877567+0000", 2024-09-21T10:23:37.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9039999999999998e-06 2024-09-21T10:23:37.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.006 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677836+0000", 2024-09-21T10:23:37.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80026863000000004 2024-09-21T10:23:37.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.007 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:37.007 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:44.677839+0000", 2024-09-21T10:23:37.007 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.044e-06 2024-09-21T10:23:37.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.007 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.007 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.006105+0000", 2024-09-21T10:23:37.007 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3282661060000001 2024-09-21T10:23:37.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.008 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:37.008 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.006239+0000", 2024-09-21T10:23:37.008 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013374799999999999 2024-09-21T10:23:37.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.008 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.008 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:23:37.008 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.006343+0000", 2024-09-21T10:23:37.008 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010435800000000001 2024-09-21T10:23:37.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.008 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.008 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:23:37.008 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.008909+0000", 2024-09-21T10:23:37.009 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002565459 2024-09-21T10:23:37.009 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.009 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.009 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:37.010 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:36 smithi098 ceph-mon[97580]: pgmap v1045: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T10:23:37.010 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:36 smithi098 ceph-mon[100914]: pgmap v1045: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T10:23:37.011 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.011 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.015898+0000", 2024-09-21T10:23:37.012 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0069897850000000001 2024-09-21T10:23:37.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.012 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.012 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.020402+0000", 2024-09-21T10:23:37.012 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045034910000000001 2024-09-21T10:23:37.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.012 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.012 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:37.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.020446+0000", 2024-09-21T10:23:37.012 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3784e-05 2024-09-21T10:23:37.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.013 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.013 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:37.013 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:13:46.021153+0000", 2024-09-21T10:23:37.013 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00070732399999999995 2024-09-21T10:23:37.013 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.013 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:37.013 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.013 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.013 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:126295 7.15 7:a83ab893:::benchmark_data_smithi098_197834_object9778:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-21T10:23:37.014 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:14:53.662164+0000", 2024-09-21T10:23:37.014 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 523.25659499300002, 2024-09-21T10:23:37.014 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.316171792, 2024-09-21T10:23:37.014 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:37.014 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:23:37.014 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:23:37.014 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:23:37.014 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:23:37.014 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 126295 2024-09-21T10:23:37.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.014 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:37.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.015 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:37.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.662164+0000", 2024-09-21T10:23:37.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.015 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:37.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.662164+0000", 2024-09-21T10:23:37.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.016 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:37.016 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.662163+0000", 2024-09-21T10:23:37.016 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:23:37.016 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.016 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.016 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:37.016 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.662166+0000", 2024-09-21T10:23:37.016 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.791e-06 2024-09-21T10:23:37.016 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.016 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.016 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:37.016 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.662172+0000", 2024-09-21T10:23:37.017 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.105e-06 2024-09-21T10:23:37.017 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.017 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.017 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:37.017 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.662176+0000", 2024-09-21T10:23:37.017 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0840000000000002e-06 2024-09-21T10:23:37.017 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.017 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.017 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.017 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.663493+0000", 2024-09-21T10:23:37.017 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0013171050000000001 2024-09-21T10:23:37.018 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.018 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.018 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:37.018 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.663622+0000", 2024-09-21T10:23:37.018 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012832499999999999 2024-09-21T10:23:37.018 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.018 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.018 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,2", 2024-09-21T10:23:37.018 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.663722+0000", 2024-09-21T10:23:37.018 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9994000000000005e-05 2024-09-21T10:23:37.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.019 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.019 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:23:37.019 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.665921+0000", 2024-09-21T10:23:37.019 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021989739999999998 2024-09-21T10:23:37.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.019 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.019 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.019 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.674333+0000", 2024-09-21T10:23:37.019 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0084118720000000008 2024-09-21T10:23:37.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.019 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.020 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.020 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.685186+0000", 2024-09-21T10:23:37.020 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010853115 2024-09-21T10:23:37.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.020 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.020 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:37.020 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.685196+0000", 2024-09-21T10:23:37.020 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0404000000000001e-05 2024-09-21T10:23:37.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.020 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.020 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.020 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687220+0000", 2024-09-21T10:23:37.021 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0020234910000001 2024-09-21T10:23:37.021 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.021 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.021 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:37.021 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687230+0000", 2024-09-21T10:23:37.021 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0789e-05 2024-09-21T10:23:37.021 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.021 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.021 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.021 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.972981+0000", 2024-09-21T10:23:37.021 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2857503000000001 2024-09-21T10:23:37.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.022 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.022 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for missing object", 2024-09-21T10:23:37.022 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.973091+0000", 2024-09-21T10:23:37.022 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000110454 2024-09-21T10:23:37.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.022 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.022 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.022 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.978279+0000", 2024-09-21T10:23:37.022 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051879990000000004 2024-09-21T10:23:37.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.022 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.023 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:37.023 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.978336+0000", 2024-09-21T10:23:37.023 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6446000000000001e-05 2024-09-21T10:23:37.023 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.023 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:37.023 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.023 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.023 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.023 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:126344 7.15 7:af8722a8:::benchmark_data_smithi098_197834_object9827:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-09-21T10:23:37.023 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:14:53.717412+0000", 2024-09-21T10:23:37.023 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 523.20134662800001, 2024-09-21T10:23:37.024 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2618246119999998, 2024-09-21T10:23:37.024 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:37.024 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:37.024 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:23:37.024 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:23:37.024 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:23:37.024 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 126344 2024-09-21T10:23:37.024 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.024 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:37.024 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.024 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:37.024 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717412+0000", 2024-09-21T10:23:37.025 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.025 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.025 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.025 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:37.025 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717412+0000", 2024-09-21T10:23:37.025 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.025 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.025 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.025 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:37.025 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717411+0000", 2024-09-21T10:23:37.025 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:23:37.026 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.026 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.026 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:37.026 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717417+0000", 2024-09-21T10:23:37.026 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9109999999999998e-06 2024-09-21T10:23:37.026 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.026 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.026 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:37.026 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717428+0000", 2024-09-21T10:23:37.026 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.102e-05 2024-09-21T10:23:37.026 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.026 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.027 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:37.027 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717432+0000", 2024-09-21T10:23:37.027 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3869999999999997e-06 2024-09-21T10:23:37.027 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.027 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.027 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.027 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717479+0000", 2024-09-21T10:23:37.027 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6888000000000002e-05 2024-09-21T10:23:37.027 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.027 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.027 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:37.027 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717488+0000", 2024-09-21T10:23:37.028 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1619999999999997e-06 2024-09-21T10:23:37.028 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.028 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.028 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.028 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687267+0000", 2024-09-21T10:23:37.028 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.969778794 2024-09-21T10:23:37.028 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.028 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.028 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:37.028 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687272+0000", 2024-09-21T10:23:37.028 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2220000000000001e-06 2024-09-21T10:23:37.028 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.029 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.029 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.029 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.974032+0000", 2024-09-21T10:23:37.029 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.28675984 2024-09-21T10:23:37.029 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.029 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.029 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:37.029 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.974120+0000", 2024-09-21T10:23:37.029 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8318000000000004e-05 2024-09-21T10:23:37.029 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.029 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.029 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:23:37.029 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.974212+0000", 2024-09-21T10:23:37.030 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2009000000000006e-05 2024-09-21T10:23:37.030 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.030 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.030 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.030 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.978367+0000", 2024-09-21T10:23:37.030 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041543960000000003 2024-09-21T10:23:37.030 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.030 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.030 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:23:37.030 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.978435+0000", 2024-09-21T10:23:37.030 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8034000000000006e-05 2024-09-21T10:23:37.030 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.030 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.031 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.031 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.978471+0000", 2024-09-21T10:23:37.031 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6238000000000001e-05 2024-09-21T10:23:37.031 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.031 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.031 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:37.031 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.978501+0000", 2024-09-21T10:23:37.031 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9850999999999999e-05 2024-09-21T10:23:37.031 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.031 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.031 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:37.031 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.979237+0000", 2024-09-21T10:23:37.031 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00073583399999999999 2024-09-21T10:23:37.032 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.032 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:37.032 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.032 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.032 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.032 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:126345 7.15 7:a9aca293:::benchmark_data_smithi098_197834_object9828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-09-21T10:23:37.032 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:14:53.717858+0000", 2024-09-21T10:23:37.032 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 523.20090127100002, 2024-09-21T10:23:37.032 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2720590949999999, 2024-09-21T10:23:37.032 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:37.032 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:37.032 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:23:37.033 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:23:37.033 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:23:37.033 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 126345 2024-09-21T10:23:37.033 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.033 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:37.033 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.033 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:37.033 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717858+0000", 2024-09-21T10:23:37.033 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.033 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.033 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.033 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:37.033 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717858+0000", 2024-09-21T10:23:37.033 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.034 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.034 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.034 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:37.034 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717857+0000", 2024-09-21T10:23:37.034 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:23:37.034 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.034 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.034 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:37.034 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717861+0000", 2024-09-21T10:23:37.034 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8680000000000001e-06 2024-09-21T10:23:37.034 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.035 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.035 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:37.035 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717879+0000", 2024-09-21T10:23:37.035 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7989e-05 2024-09-21T10:23:37.035 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.035 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.035 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:37.035 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717884+0000", 2024-09-21T10:23:37.035 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6010000000000002e-06 2024-09-21T10:23:37.035 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.035 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.035 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.035 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717920+0000", 2024-09-21T10:23:37.036 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5914000000000002e-05 2024-09-21T10:23:37.036 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.036 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.036 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:37.036 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.717928+0000", 2024-09-21T10:23:37.036 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5700000000000004e-06 2024-09-21T10:23:37.036 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.036 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.036 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.036 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687305+0000", 2024-09-21T10:23:37.036 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.969376825 2024-09-21T10:23:37.036 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.036 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.037 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:37.037 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687310+0000", 2024-09-21T10:23:37.037 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9019999999999998e-06 2024-09-21T10:23:37.037 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.037 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.037 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.037 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.975661+0000", 2024-09-21T10:23:37.037 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2883512429999999 2024-09-21T10:23:37.037 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.037 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.037 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:37.037 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.975770+0000", 2024-09-21T10:23:37.037 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109239 2024-09-21T10:23:37.037 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.038 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.038 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:23:37.038 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.975847+0000", 2024-09-21T10:23:37.038 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7354999999999994e-05 2024-09-21T10:23:37.038 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.038 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.038 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:23:37.038 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.978443+0000", 2024-09-21T10:23:37.038 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025957739999999999 2024-09-21T10:23:37.038 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.038 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.038 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.039 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.979284+0000", 2024-09-21T10:23:37.039 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000840459 2024-09-21T10:23:37.039 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.039 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.039 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.039 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.988637+0000", 2024-09-21T10:23:37.039 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0093530850000000006 2024-09-21T10:23:37.039 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.039 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.039 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:37.039 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.988669+0000", 2024-09-21T10:23:37.039 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2756999999999998e-05 2024-09-21T10:23:37.040 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.040 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.040 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:37.040 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.989917+0000", 2024-09-21T10:23:37.040 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012471909999999999 2024-09-21T10:23:37.040 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.040 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:37.040 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.040 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.040 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.040 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:126354 7.15 7:a9f8f369:::benchmark_data_smithi098_197834_object9837:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-09-21T10:23:37.040 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:14:53.724439+0000", 2024-09-21T10:23:37.041 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 523.19431990800001, 2024-09-21T10:23:37.041 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2705321569999999, 2024-09-21T10:23:37.041 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:37.041 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:37.041 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:23:37.041 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:23:37.041 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:23:37.041 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 126354 2024-09-21T10:23:37.041 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.041 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:37.041 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.042 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:37.042 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.724439+0000", 2024-09-21T10:23:37.042 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.042 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.042 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.042 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:37.042 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.724439+0000", 2024-09-21T10:23:37.042 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.042 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.042 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.042 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:37.042 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.724438+0000", 2024-09-21T10:23:37.042 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:23:37.043 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.043 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.043 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:37.043 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.724443+0000", 2024-09-21T10:23:37.043 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5470000000000004e-06 2024-09-21T10:23:37.043 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.043 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.043 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:37.043 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.724453+0000", 2024-09-21T10:23:37.043 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8430000000000005e-06 2024-09-21T10:23:37.043 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.043 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.044 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:37.044 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.724459+0000", 2024-09-21T10:23:37.044 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5720000000000002e-06 2024-09-21T10:23:37.044 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.044 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.044 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.044 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.724486+0000", 2024-09-21T10:23:37.044 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7474000000000002e-05 2024-09-21T10:23:37.044 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.044 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.044 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:37.045 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.724491+0000", 2024-09-21T10:23:37.045 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4160000000000003e-06 2024-09-21T10:23:37.045 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.045 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.045 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.045 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687339+0000", 2024-09-21T10:23:37.045 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.96284762499999998 2024-09-21T10:23:37.045 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.045 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.045 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:37.045 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687344+0000", 2024-09-21T10:23:37.046 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3159999999999996e-06 2024-09-21T10:23:37.046 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.046 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.046 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.046 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.979308+0000", 2024-09-21T10:23:37.046 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.291963355 2024-09-21T10:23:37.046 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.046 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.046 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:37.046 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.979390+0000", 2024-09-21T10:23:37.046 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1928000000000006e-05 2024-09-21T10:23:37.047 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.047 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.047 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:23:37.047 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.979466+0000", 2024-09-21T10:23:37.047 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6395000000000003e-05 2024-09-21T10:23:37.047 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.047 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.047 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:23:37.047 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.983649+0000", 2024-09-21T10:23:37.047 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041824210000000004 2024-09-21T10:23:37.047 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.047 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.047 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.048 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.988605+0000", 2024-09-21T10:23:37.048 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004956556 2024-09-21T10:23:37.048 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.048 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.048 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.048 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.994039+0000", 2024-09-21T10:23:37.048 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0054335190000000004 2024-09-21T10:23:37.048 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.048 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.048 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:37.048 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.994071+0000", 2024-09-21T10:23:37.048 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2288e-05 2024-09-21T10:23:37.049 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.049 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.049 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:37.049 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.994971+0000", 2024-09-21T10:23:37.049 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00090020300000000003 2024-09-21T10:23:37.049 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.049 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:37.049 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.049 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.049 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.049 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:126362 7.15 7:ab907e32:::benchmark_data_smithi098_197834_object9845:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-09-21T10:23:37.049 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:14:53.731370+0000", 2024-09-21T10:23:37.049 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 523.18738865800003, 2024-09-21T10:23:37.050 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2648341489999999, 2024-09-21T10:23:37.050 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:37.050 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:37.050 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:23:37.050 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:23:37.050 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:23:37.050 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 126362 2024-09-21T10:23:37.050 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.050 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:37.050 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.051 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:37.051 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.731370+0000", 2024-09-21T10:23:37.051 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.051 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.051 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.051 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:37.051 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.731370+0000", 2024-09-21T10:23:37.051 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.051 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.051 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.051 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:37.052 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.731368+0000", 2024-09-21T10:23:37.052 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:23:37.052 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.052 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.052 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:37.052 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.731376+0000", 2024-09-21T10:23:37.052 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4479999999999997e-06 2024-09-21T10:23:37.052 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.052 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.052 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:37.052 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.731388+0000", 2024-09-21T10:23:37.052 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1684e-05 2024-09-21T10:23:37.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.053 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.053 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:37.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.731392+0000", 2024-09-21T10:23:37.053 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6949999999999999e-06 2024-09-21T10:23:37.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.053 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.053 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.731429+0000", 2024-09-21T10:23:37.053 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6405000000000001e-05 2024-09-21T10:23:37.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.054 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.054 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:37.054 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.731439+0000", 2024-09-21T10:23:37.054 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8230000000000006e-06 2024-09-21T10:23:37.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.054 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.054 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.054 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687364+0000", 2024-09-21T10:23:37.054 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.95592560199999999 2024-09-21T10:23:37.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.054 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.055 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:37.055 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687368+0000", 2024-09-21T10:23:37.055 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5370000000000002e-06 2024-09-21T10:23:37.055 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.055 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.055 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.055 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.980449+0000", 2024-09-21T10:23:37.055 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2930808410000001 2024-09-21T10:23:37.055 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.055 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.055 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:37.056 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.980563+0000", 2024-09-21T10:23:37.056 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011435800000000001 2024-09-21T10:23:37.056 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.056 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.056 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:23:37.056 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.980648+0000", 2024-09-21T10:23:37.056 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4917e-05 2024-09-21T10:23:37.056 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.056 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.056 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:23:37.057 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.983656+0000", 2024-09-21T10:23:37.057 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030083100000000002 2024-09-21T10:23:37.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.057 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.057 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.057 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.989968+0000", 2024-09-21T10:23:37.057 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006311924 2024-09-21T10:23:37.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.057 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.057 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.058 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.995064+0000", 2024-09-21T10:23:37.058 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0050958599999999998 2024-09-21T10:23:37.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.058 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.058 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:37.058 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.995114+0000", 2024-09-21T10:23:37.058 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9855000000000003e-05 2024-09-21T10:23:37.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.058 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.058 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:37.058 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.996204+0000", 2024-09-21T10:23:37.058 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010905579999999999 2024-09-21T10:23:37.059 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.059 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:37.059 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.059 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.059 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:126420 7.15 7:aea4d8a7:::benchmark_data_smithi098_197834_object9903:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-09-21T10:23:37.059 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:14:53.782086+0000", 2024-09-21T10:23:37.059 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 523.13667267799997, 2024-09-21T10:23:37.059 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.216431933, 2024-09-21T10:23:37.059 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:37.059 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:37.059 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:23:37.060 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:23:37.060 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:23:37.060 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 126420 2024-09-21T10:23:37.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.060 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:37.060 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.060 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:37.060 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.782086+0000", 2024-09-21T10:23:37.060 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.060 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.061 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:37.061 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.782086+0000", 2024-09-21T10:23:37.061 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.061 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.061 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.061 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:37.061 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.782084+0000", 2024-09-21T10:23:37.061 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:23:37.061 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.061 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.061 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:37.062 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.782091+0000", 2024-09-21T10:23:37.062 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7889999999999997e-06 2024-09-21T10:23:37.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.062 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.062 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:37.062 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.782105+0000", 2024-09-21T10:23:37.062 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3903000000000001e-05 2024-09-21T10:23:37.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.062 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.062 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:37.062 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.782111+0000", 2024-09-21T10:23:37.063 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0140000000000004e-06 2024-09-21T10:23:37.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.063 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.063 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.063 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.782159+0000", 2024-09-21T10:23:37.063 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8111000000000001e-05 2024-09-21T10:23:37.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.063 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.063 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:37.063 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.782169+0000", 2024-09-21T10:23:37.063 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.448e-06 2024-09-21T10:23:37.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.064 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.064 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687385+0000", 2024-09-21T10:23:37.064 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.90521664599999996 2024-09-21T10:23:37.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.064 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.064 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:37.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687389+0000", 2024-09-21T10:23:37.064 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4599999999999999e-06 2024-09-21T10:23:37.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.064 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.065 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.065 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.981530+0000", 2024-09-21T10:23:37.065 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.294141193 2024-09-21T10:23:37.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.065 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.065 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:37.065 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.981627+0000", 2024-09-21T10:23:37.065 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6613999999999994e-05 2024-09-21T10:23:37.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.065 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.065 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:23:37.065 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.981706+0000", 2024-09-21T10:23:37.065 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9658000000000002e-05 2024-09-21T10:23:37.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.066 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.066 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:23:37.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.985669+0000", 2024-09-21T10:23:37.066 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039624200000000004 2024-09-21T10:23:37.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.066 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.066 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.990000+0000", 2024-09-21T10:23:37.066 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043309350000000002 2024-09-21T10:23:37.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.067 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.067 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.067 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.997553+0000", 2024-09-21T10:23:37.067 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0075531210000000003 2024-09-21T10:23:37.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.067 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.067 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:37.067 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.997585+0000", 2024-09-21T10:23:37.067 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2069000000000001e-05 2024-09-21T10:23:37.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.067 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.068 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:37.068 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.998518+0000", 2024-09-21T10:23:37.068 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00093336299999999999 2024-09-21T10:23:37.068 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.068 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:37.068 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.068 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.068 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:126426 7.15 7:aff43bec:::benchmark_data_smithi098_197834_object9909:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-09-21T10:23:37.068 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:14:53.790525+0000", 2024-09-21T10:23:37.068 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 523.12823427399996, 2024-09-21T10:23:37.069 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2095328049999998, 2024-09-21T10:23:37.069 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:37.069 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:37.069 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:23:37.069 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:23:37.069 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:23:37.069 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 126426 2024-09-21T10:23:37.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.069 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:37.069 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.069 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:37.069 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.790525+0000", 2024-09-21T10:23:37.070 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.070 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.070 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:37.070 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.790525+0000", 2024-09-21T10:23:37.070 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.070 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.070 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:37.070 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.790523+0000", 2024-09-21T10:23:37.070 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:23:37.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.071 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.071 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:37.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.790529+0000", 2024-09-21T10:23:37.071 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6329999999999999e-06 2024-09-21T10:23:37.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.071 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.071 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:37.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.790542+0000", 2024-09-21T10:23:37.071 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2525e-05 2024-09-21T10:23:37.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.072 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.072 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:37.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.790548+0000", 2024-09-21T10:23:37.072 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7049999999999996e-06 2024-09-21T10:23:37.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.072 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.072 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.790600+0000", 2024-09-21T10:23:37.072 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2506999999999997e-05 2024-09-21T10:23:37.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.072 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.072 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:37.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.790610+0000", 2024-09-21T10:23:37.073 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8570000000000007e-06 2024-09-21T10:23:37.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.073 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.073 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687405+0000", 2024-09-21T10:23:37.073 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.89679551000000002 2024-09-21T10:23:37.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.073 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.073 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:37.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687409+0000", 2024-09-21T10:23:37.073 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.258e-06 2024-09-21T10:23:37.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.074 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.982677+0000", 2024-09-21T10:23:37.074 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.295268587 2024-09-21T10:23:37.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.074 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:37.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.982762+0000", 2024-09-21T10:23:37.074 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4469999999999996e-05 2024-09-21T10:23:37.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.075 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:23:37.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.982864+0000", 2024-09-21T10:23:37.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000101946 2024-09-21T10:23:37.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.075 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.075 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:23:37.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.986623+0000", 2024-09-21T10:23:37.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003759255 2024-09-21T10:23:37.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.075 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.075 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.991755+0000", 2024-09-21T10:23:37.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051318320000000002 2024-09-21T10:23:37.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.076 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.076 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.998586+0000", 2024-09-21T10:23:37.076 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0068313940000000002 2024-09-21T10:23:37.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.076 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.076 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:37.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.998633+0000", 2024-09-21T10:23:37.076 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6925000000000002e-05 2024-09-21T10:23:37.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.077 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.077 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:37.077 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:56.000057+0000", 2024-09-21T10:23:37.077 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014242549999999999 2024-09-21T10:23:37.077 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.077 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:37.077 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.077 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.077 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:126444 7.15 7:a889100f:::benchmark_data_smithi098_197834_object9927:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-09-21T10:23:37.077 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:14:53.808084+0000", 2024-09-21T10:23:37.077 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 523.11067466999998, 2024-09-21T10:23:37.078 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1976205769999999, 2024-09-21T10:23:37.078 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:37.078 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:37.078 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:23:37.078 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:23:37.078 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:23:37.078 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 126444 2024-09-21T10:23:37.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.079 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:37.079 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.079 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:37.079 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.808084+0000", 2024-09-21T10:23:37.079 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.079 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.079 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:37.079 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.808084+0000", 2024-09-21T10:23:37.079 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.079 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.080 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:37.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.808082+0000", 2024-09-21T10:23:37.080 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:23:37.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.080 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.080 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:37.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.808091+0000", 2024-09-21T10:23:37.080 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1559999999999995e-06 2024-09-21T10:23:37.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.080 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.080 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:37.081 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.808110+0000", 2024-09-21T10:23:37.081 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9683000000000001e-05 2024-09-21T10:23:37.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.081 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.081 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:37.081 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.808123+0000", 2024-09-21T10:23:37.081 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2422999999999999e-05 2024-09-21T10:23:37.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.081 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.081 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.808228+0000", 2024-09-21T10:23:37.082 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105097 2024-09-21T10:23:37.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.082 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.082 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:37.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.808234+0000", 2024-09-21T10:23:37.082 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8619999999999999e-06 2024-09-21T10:23:37.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.082 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.082 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687425+0000", 2024-09-21T10:23:37.083 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.87919187700000001 2024-09-21T10:23:37.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.083 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.083 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:37.083 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687429+0000", 2024-09-21T10:23:37.083 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.36e-06 2024-09-21T10:23:37.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.083 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.083 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.083 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.983691+0000", 2024-09-21T10:23:37.083 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2962624059999999 2024-09-21T10:23:37.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.084 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.084 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:37.084 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.983786+0000", 2024-09-21T10:23:37.084 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5231999999999999e-05 2024-09-21T10:23:37.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.084 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.084 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:23:37.084 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.983890+0000", 2024-09-21T10:23:37.084 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010302300000000001 2024-09-21T10:23:37.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.084 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.084 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:23:37.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.986626+0000", 2024-09-21T10:23:37.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027369120000000002 2024-09-21T10:23:37.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.085 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.085 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.991788+0000", 2024-09-21T10:23:37.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051614230000000001 2024-09-21T10:23:37.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.085 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.085 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:56.003715+0000", 2024-09-21T10:23:37.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011927589000000001 2024-09-21T10:23:37.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.086 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:37.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:56.003773+0000", 2024-09-21T10:23:37.086 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7102000000000003e-05 2024-09-21T10:23:37.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.086 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:37.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:56.005705+0000", 2024-09-21T10:23:37.086 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001932198 2024-09-21T10:23:37.086 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.087 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:37.087 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.087 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.087 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:126522 7.15 7:a82e3069:::benchmark_data_smithi098_197834_object10005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-09-21T10:23:37.087 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:14:53.885910+0000", 2024-09-21T10:23:37.087 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 523.03284891700002, 2024-09-21T10:23:37.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1211003129999999, 2024-09-21T10:23:37.087 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:37.087 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:37.087 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:23:37.088 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:23:37.088 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:23:37.088 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 126522 2024-09-21T10:23:37.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.088 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:37.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:37.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.885910+0000", 2024-09-21T10:23:37.088 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:37.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.885910+0000", 2024-09-21T10:23:37.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.089 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.089 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:37.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.885909+0000", 2024-09-21T10:23:37.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:23:37.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.089 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.089 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:37.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.885914+0000", 2024-09-21T10:23:37.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.853e-06 2024-09-21T10:23:37.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.090 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.090 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:37.090 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.885923+0000", 2024-09-21T10:23:37.090 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4690000000000003e-06 2024-09-21T10:23:37.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.090 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.090 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:37.090 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.885927+0000", 2024-09-21T10:23:37.090 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1489999999999996e-06 2024-09-21T10:23:37.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.091 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.091 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.091 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.885970+0000", 2024-09-21T10:23:37.091 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3260000000000003e-05 2024-09-21T10:23:37.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.091 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.091 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:37.091 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.885978+0000", 2024-09-21T10:23:37.091 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1400000000000002e-06 2024-09-21T10:23:37.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.091 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.091 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687445+0000", 2024-09-21T10:23:37.092 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80146784000000004 2024-09-21T10:23:37.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.092 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.092 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:37.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687449+0000", 2024-09-21T10:23:37.092 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.264e-06 2024-09-21T10:23:37.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.092 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.092 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.984716+0000", 2024-09-21T10:23:37.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2972670179999999 2024-09-21T10:23:37.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:37.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.984800+0000", 2024-09-21T10:23:37.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4655999999999997e-05 2024-09-21T10:23:37.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.095 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:23:37.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.984900+0000", 2024-09-21T10:23:37.095 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9747999999999994e-05 2024-09-21T10:23:37.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.095 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.095 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:23:37.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.987585+0000", 2024-09-21T10:23:37.095 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026850649999999999 2024-09-21T10:23:37.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.095 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.991822+0000", 2024-09-21T10:23:37.096 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004237088 2024-09-21T10:23:37.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.096 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:56.005774+0000", 2024-09-21T10:23:37.096 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013951882000000001 2024-09-21T10:23:37.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.096 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:37.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:56.005836+0000", 2024-09-21T10:23:37.097 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1477999999999997e-05 2024-09-21T10:23:37.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.097 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:37.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:56.007010+0000", 2024-09-21T10:23:37.097 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001174551 2024-09-21T10:23:37.097 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.097 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:37.097 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.097 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:126540 7.15 7:a9cf6911:::benchmark_data_smithi098_197834_object10023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-09-21T10:23:37.098 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:14:53.908019+0000", 2024-09-21T10:23:37.098 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 523.01073973799998, 2024-09-21T10:23:37.098 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1021127829999999, 2024-09-21T10:23:37.098 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:37.098 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:37.098 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:23:37.098 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:23:37.098 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:23:37.098 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 126540 2024-09-21T10:23:37.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.098 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:37.099 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.099 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:37.099 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.908019+0000", 2024-09-21T10:23:37.099 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.099 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.099 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:37.099 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.908019+0000", 2024-09-21T10:23:37.099 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.099 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.099 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.908019+0000", 2024-09-21T10:23:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:23:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.908022+0000", 2024-09-21T10:23:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1109999999999999e-06 2024-09-21T10:23:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.908028+0000", 2024-09-21T10:23:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3300000000000004e-06 2024-09-21T10:23:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.908032+0000", 2024-09-21T10:23:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2350000000000001e-06 2024-09-21T10:23:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.908073+0000", 2024-09-21T10:23:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1137999999999998e-05 2024-09-21T10:23:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:53.908080+0000", 2024-09-21T10:23:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6309999999999997e-06 2024-09-21T10:23:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687465+0000", 2024-09-21T10:23:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.77938554400000004 2024-09-21T10:23:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:23:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:54.687469+0000", 2024-09-21T10:23:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3100000000000001e-06 2024-09-21T10:23:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.985687+0000", 2024-09-21T10:23:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2982183919999999 2024-09-21T10:23:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.985771+0000", 2024-09-21T10:23:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4031999999999998e-05 2024-09-21T10:23:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:23:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.985858+0000", 2024-09-21T10:23:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6815000000000001e-05 2024-09-21T10:23:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:23:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.988566+0000", 2024-09-21T10:23:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027079109999999999 2024-09-21T10:23:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:55.991852+0000", 2024-09-21T10:23:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003286527 2024-09-21T10:23:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:56.008824+0000", 2024-09-21T10:23:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016971198 2024-09-21T10:23:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:56.008888+0000", 2024-09-21T10:23:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3833999999999999e-05 2024-09-21T10:23:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:14:56.010132+0000", 2024-09-21T10:23:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012443560000000001 2024-09-21T10:23:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:52171 9.c 9:360aee28:::benchmark_data_smithi098_254407_object52170:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-21T10:23:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:22:56.544104+0000", 2024-09-21T10:23:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 40.374654980999999, 2024-09-21T10:23:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.157414996, 2024-09-21T10:23:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:23:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:23:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:23:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:23:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 52171 2024-09-21T10:23:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:23:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:23:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.544104+0000", 2024-09-21T10:23:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:23:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.544104+0000", 2024-09-21T10:23:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:23:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:23:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.544098+0000", 2024-09-21T10:23:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999943 2024-09-21T10:23:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:23:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.544136+0000", 2024-09-21T10:23:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8164000000000003e-05 2024-09-21T10:23:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:23:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.544146+0000", 2024-09-21T10:23:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5119999999999997e-06 2024-09-21T10:23:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:23:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.544150+0000", 2024-09-21T10:23:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2880000000000003e-06 2024-09-21T10:23:37.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.111 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:23:37.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.544190+0000", 2024-09-21T10:23:37.111 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0119e-05 2024-09-21T10:23:37.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.111 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:23:37.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.544310+0000", 2024-09-21T10:23:37.111 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119418 2024-09-21T10:23:37.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.112 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:23:37.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.544445+0000", 2024-09-21T10:23:37.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013546200000000001 2024-09-21T10:23:37.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.112 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:23:37.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.547672+0000", 2024-09-21T10:23:37.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032270559999999998 2024-09-21T10:23:37.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.112 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.547861+0000", 2024-09-21T10:23:37.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00018934900000000001 2024-09-21T10:23:37.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.113 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:37.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.701428+0000", 2024-09-21T10:23:37.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.153566906 2024-09-21T10:23:37.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.113 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:23:37.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.701475+0000", 2024-09-21T10:23:37.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6854999999999998e-05 2024-09-21T10:23:37.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:23:37.114 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:23:37.114 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:23:37.114 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.701519+0000", 2024-09-21T10:23:37.114 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3587999999999997e-05 2024-09-21T10:23:37.114 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.114 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:37.114 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.114 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:23:37.114 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:23:37.114 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:23:37.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:36 smithi196 ceph-mon[84784]: pgmap v1045: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T10:23:37.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.116 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:37.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:37.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:37.240 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 64 16 102924 102908 100.45 70.125 0.00951893 0.00995006 2024-09-21T10:23:37.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.255 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:37.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:37.356 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:37.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.390 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:37.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:37.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:37.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.526 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:37.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:37.627 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:37.666 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:37.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:37.768 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:37.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:37.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:37.919 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:37.925 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:37.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:38.027 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:38.065 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:38.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:38.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:38.167 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:38.195 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:38.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:38.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:38.240 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 65 16 104013 103997 99.9515 68.0625 0.0089363 0.0100004 2024-09-21T10:23:38.296 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:38.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:38.335 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:38.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:38.436 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:38.471 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:38.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:38.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:38.573 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:38.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:38.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:38.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:38.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:38.614 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:38.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:38.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:38.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:38.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:38.716 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:38.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:38.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:38.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:38.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:38.878 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:38.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:38.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:38.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:39.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:39.016 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:39.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:39.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:39.147 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:39.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:39.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:39.240 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 66 16 105072 105056 99.4401 66.1875 0.00792027 0.0100519 2024-09-21T10:23:39.250 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:39.284 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:39.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:39.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:39.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:39.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:39 smithi196 ceph-mon[84784]: pgmap v1046: 99 pgs: 99 active+clean; 6.1 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 71 MiB/s wr, 1.12k op/s 2024-09-21T10:23:39.419 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:39.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:39.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:39.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:39.556 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:39.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:39.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:39.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:39 smithi098 ceph-mon[97580]: pgmap v1046: 99 pgs: 99 active+clean; 6.1 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 71 MiB/s wr, 1.12k op/s 2024-09-21T10:23:39.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:39 smithi098 ceph-mon[100914]: pgmap v1046: 99 pgs: 99 active+clean; 6.1 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 71 MiB/s wr, 1.12k op/s 2024-09-21T10:23:39.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:39.691 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:39.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:39.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:39.793 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:39.833 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:39.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:39.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:39.935 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:39.950 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:23:39.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:39.973 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:39.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:40.074 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:40.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:40.096 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:40.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:40.198 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:40.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:40.234 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:40.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:40.240 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 67 16 106057 106041 98.8749 61.5625 0.014542 0.0101088 2024-09-21T10:23:40.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:40.365 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:40.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:40.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:40.467 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:40.511 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:40.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:40.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:40.524 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:23:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:23:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:23:40.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:23:40.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:40.648 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:40.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:40.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:40.750 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:40.784 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:40.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:40.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:40.886 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:40.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:40.929 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:40.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:41.030 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:41.063 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:41.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:41.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:41.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:23:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:23:40] "GET /metrics HTTP/1.1" 200 36520 "" "Prometheus/2.51.0" 2024-09-21T10:23:41.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:41.203 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:41.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:41.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:41.240 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 68 16 107149 107133 98.4246 68.25 0.016615 0.0101557 2024-09-21T10:23:41.305 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:41.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:41.347 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:41.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:41.448 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:41.464 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:41 smithi098 ceph-mon[97580]: pgmap v1047: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 64 MiB/s wr, 1.01k op/s 2024-09-21T10:23:41.464 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:41 smithi098 ceph-mon[100914]: pgmap v1047: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 64 MiB/s wr, 1.01k op/s 2024-09-21T10:23:41.491 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:41.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:41.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:41.593 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:41.620 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:41.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:41.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:41.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:41 smithi196 ceph-mon[84784]: pgmap v1047: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 64 MiB/s wr, 1.01k op/s 2024-09-21T10:23:41.722 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:41.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:41.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:41.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:41.859 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:41.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:41.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:41.998 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:42.036 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:42.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:42.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:42.173 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:42.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:42.240 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 69 16 108305 108289 98.0453 72.25 0.0145635 0.010195 2024-09-21T10:23:42.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:42.306 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:42.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:42.408 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:42.444 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:42.546 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:42.579 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:42.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:42.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:42.717 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:42.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:42.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:42.859 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:42.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:42.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:42.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.997 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:42.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:43.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:43.138 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:43.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:43.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:43.240 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:43.241 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 70 16 109490 109474 97.7027 74.0625 0.00975902 0.0102309 2024-09-21T10:23:43.260 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:23:43.276 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:43.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:43.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:43.338 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-21T10:23:43.378 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:43.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:43.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:43.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:43.557 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:43.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:43.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:43.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:43 smithi098 ceph-mon[97580]: pgmap v1048: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 58 MiB/s wr, 932 op/s 2024-09-21T10:23:43.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:43 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2174930410' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:23:43.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:43 smithi098 ceph-mon[100914]: pgmap v1048: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 58 MiB/s wr, 932 op/s 2024-09-21T10:23:43.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:43 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2174930410' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:23:43.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:43 smithi196 ceph-mon[84784]: pgmap v1048: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 58 MiB/s wr, 932 op/s 2024-09-21T10:23:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:43 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2174930410' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:23:43.659 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:43.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:43.703 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:43.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:43.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:43.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:43.841 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:43.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:43.942 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:43.980 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:43.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:43.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:44.082 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:44.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:44.120 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:44.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:44.221 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:44.243 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 71 16 110677 110661 97.3688 74.1875 0.0124552 0.0102653 2024-09-21T10:23:44.252 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:44.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:44.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:44.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:44 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:44.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:44 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2174930410' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:23:44.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:44 smithi098 ceph-mon[97580]: osdmap e316: 8 total, 8 up, 6 in 2024-09-21T10:23:44.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:44 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:44.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:44 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2174930410' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:23:44.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:44 smithi098 ceph-mon[100914]: osdmap e316: 8 total, 8 up, 6 in 2024-09-21T10:23:44.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:44.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:44.389 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:44.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:44.490 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:44.519 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:44.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:44.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:44.604 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:44.621 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:44.628 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:44.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:44 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:44 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2174930410' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:23:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:44 smithi196 ceph-mon[84784]: osdmap e316: 8 total, 8 up, 6 in 2024-09-21T10:23:44.684 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:44.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:44.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:44.786 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:44.820 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:44.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:44.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:44.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:44.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:44.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:45.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:45.133 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:45.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:45.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:45.235 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:45.243 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 72 16 111805 111789 96.9957 70.5 0.0143956 0.0103052 2024-09-21T10:23:45.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:45 smithi098 ceph-mon[97580]: pgmap v1050: 99 pgs: 99 active+clean; 6.5 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-21T10:23:45.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:45 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:45.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:45 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:45.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:45.290 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:45.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:45.390 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:45.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:45.428 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:45.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:45.529 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:45.564 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:45.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:45.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:45.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:45 smithi098 ceph-mon[100914]: pgmap v1050: 99 pgs: 99 active+clean; 6.5 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-21T10:23:45.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:45 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:45.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:45 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:45 smithi196 ceph-mon[84784]: pgmap v1050: 99 pgs: 99 active+clean; 6.5 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-21T10:23:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:45 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:45 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:45.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:45.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:45.701 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:45.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:45.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:45.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:45.840 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:45.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:45.940 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:45.972 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:45.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:45.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:46.074 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:46.111 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:46.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:46.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:46.140 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:23:46.213 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:46.244 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 73 16 113005 112989 96.6928 75 0.0133442 0.0103371 2024-09-21T10:23:46.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:46.256 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:46.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:46.357 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:46.396 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:46.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:46.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:46.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:46.536 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:46.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:46.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:46.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:23:46.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:23:46.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:23:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:23:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:46.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:23:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:23:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:46.672 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:46.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:46.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:46.744 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:46.774 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:46.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:46.814 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:46.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:46.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:46.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:46.950 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:46.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:47.051 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:47.087 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:47.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:47.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:47.189 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:47.220 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:47.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:47.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:47.245 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 74 16 114172 114156 96.3706 72.9375 0.0381913 0.0103719 2024-09-21T10:23:47.322 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:47.371 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:47.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:47.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:47.473 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:47.480 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:47 smithi098 ceph-mon[97580]: pgmap v1051: 99 pgs: 99 active+clean; 6.7 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-21T10:23:47.481 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:47 smithi098 ceph-mon[100914]: pgmap v1051: 99 pgs: 99 active+clean; 6.7 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-21T10:23:47.484 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:47.507 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:47.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:47.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:47.609 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:47 smithi196 ceph-mon[84784]: pgmap v1051: 99 pgs: 99 active+clean; 6.7 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-21T10:23:47.651 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:47.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:47.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:47.753 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:47.788 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:47.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:47.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:47.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:47.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:47.935 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:47.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:48.024 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:23:48.025 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:23:48.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.025 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46025.0:115026 9.5 9:a398cedf:::benchmark_data_smithi098_254407_object115025:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T10:23:48.025 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:23:48.002310+0000", 2024-09-21T10:23:48.025 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.018418026000000001, 2024-09-21T10:23:48.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018447443000000001, 2024-09-21T10:23:48.025 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:23:48.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:23:48.025 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:23:48.025 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:23:48.025 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46025", 2024-09-21T10:23:48.025 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:23:48.025 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 115026 2024-09-21T10:23:48.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.026 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:23:48.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:23:48.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.002310+0000", 2024-09-21T10:23:48.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:23:48.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:23:48.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.002310+0000", 2024-09-21T10:23:48.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:23:48.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:23:48.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.002312+0000", 2024-09-21T10:23:48.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.598e-06 2024-09-21T10:23:48.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:23:48.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.002334+0000", 2024-09-21T10:23:48.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2548000000000002e-05 2024-09-21T10:23:48.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:23:48.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.002345+0000", 2024-09-21T10:23:48.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0478e-05 2024-09-21T10:23:48.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:23:48.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.002352+0000", 2024-09-21T10:23:48.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4359999999999996e-06 2024-09-21T10:23:48.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:23:48.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.003083+0000", 2024-09-21T10:23:48.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00073069300000000003 2024-09-21T10:23:48.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:23:48.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.003277+0000", 2024-09-21T10:23:48.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000194472 2024-09-21T10:23:48.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,7", 2024-09-21T10:23:48.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.004153+0000", 2024-09-21T10:23:48.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00087613899999999995 2024-09-21T10:23:48.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:23:48.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.008822+0000", 2024-09-21T10:23:48.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046685479999999998 2024-09-21T10:23:48.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:48.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.009016+0000", 2024-09-21T10:23:48.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019378700000000001 2024-09-21T10:23:48.030 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:23:48.030 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:23:48.030 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:23:48.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.030 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46025.0:115036 9.8 9:1081bd2b:::benchmark_data_smithi098_254407_object115035:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T10:23:48.030 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:23:48.010004+0000", 2024-09-21T10:23:48.030 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.010724083000000001, 2024-09-21T10:23:48.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010830879999999999, 2024-09-21T10:23:48.030 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:23:48.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:23:48.030 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:23:48.031 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:23:48.031 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46025", 2024-09-21T10:23:48.031 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:23:48.031 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 115036 2024-09-21T10:23:48.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.031 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:23:48.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:23:48.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.010004+0000", 2024-09-21T10:23:48.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:23:48.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:23:48.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.010004+0000", 2024-09-21T10:23:48.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:23:48.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:23:48.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.010005+0000", 2024-09-21T10:23:48.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.105e-06 2024-09-21T10:23:48.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:23:48.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.010023+0000", 2024-09-21T10:23:48.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8277000000000001e-05 2024-09-21T10:23:48.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:23:48.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.010032+0000", 2024-09-21T10:23:48.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3420000000000008e-06 2024-09-21T10:23:48.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:23:48.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.010036+0000", 2024-09-21T10:23:48.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6569999999999996e-06 2024-09-21T10:23:48.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:23:48.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.010088+0000", 2024-09-21T10:23:48.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1235e-05 2024-09-21T10:23:48.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:23:48.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.010254+0000", 2024-09-21T10:23:48.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016631099999999999 2024-09-21T10:23:48.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,7", 2024-09-21T10:23:48.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.010399+0000", 2024-09-21T10:23:48.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000145532 2024-09-21T10:23:48.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:48.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.014228+0000", 2024-09-21T10:23:48.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003828101 2024-09-21T10:23:48.034 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:23:48.035 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:23:48.035 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:23:48.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.035 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46025.0:115039 9.5 9:a9efa105:::benchmark_data_smithi098_254407_object115038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T10:23:48.035 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:23:48.013841+0000", 2024-09-21T10:23:48.035 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0068869689999999997, 2024-09-21T10:23:48.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070687049999999998, 2024-09-21T10:23:48.035 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:23:48.035 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:23:48.035 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:23:48.035 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:23:48.035 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46025", 2024-09-21T10:23:48.035 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:23:48.036 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 115039 2024-09-21T10:23:48.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.036 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:23:48.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:23:48.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.013841+0000", 2024-09-21T10:23:48.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:23:48.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:23:48.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.013841+0000", 2024-09-21T10:23:48.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:23:48.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:23:48.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.013843+0000", 2024-09-21T10:23:48.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5719999999999999e-06 2024-09-21T10:23:48.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:23:48.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.013892+0000", 2024-09-21T10:23:48.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9589999999999998e-05 2024-09-21T10:23:48.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:23:48.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.013905+0000", 2024-09-21T10:23:48.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2252999999999999e-05 2024-09-21T10:23:48.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:23:48.038 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.013912+0000", 2024-09-21T10:23:48.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1339999999999998e-06 2024-09-21T10:23:48.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:23:48.038 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:48.039 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.014110+0000", 2024-09-21T10:23:48.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019858699999999999 2024-09-21T10:23:48.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.039 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:23:48.039 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.014253+0000", 2024-09-21T10:23:48.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014308100000000001 2024-09-21T10:23:48.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,7", 2024-09-21T10:23:48.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.014474+0000", 2024-09-21T10:23:48.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000220526 2024-09-21T10:23:48.040 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:23:48.040 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:23:48.040 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:23:48.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.040 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46025.0:115042 9.8 9:1b68e0f3:::benchmark_data_smithi098_254407_object115041:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T10:23:48.040 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:23:48.013933+0000", 2024-09-21T10:23:48.040 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0067953500000000003, 2024-09-21T10:23:48.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070438990000000002, 2024-09-21T10:23:48.040 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:23:48.040 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:23:48.040 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:23:48.041 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:23:48.041 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46025", 2024-09-21T10:23:48.041 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:23:48.041 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 115042 2024-09-21T10:23:48.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.041 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:23:48.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:23:48.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.013933+0000", 2024-09-21T10:23:48.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:23:48.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:23:48.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.013933+0000", 2024-09-21T10:23:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:23:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:23:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.013933+0000", 2024-09-21T10:23:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1200000000000003e-07 2024-09-21T10:23:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:23:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.013986+0000", 2024-09-21T10:23:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3003000000000001e-05 2024-09-21T10:23:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:23:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.013995+0000", 2024-09-21T10:23:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6510000000000007e-06 2024-09-21T10:23:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:23:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.014001+0000", 2024-09-21T10:23:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9200000000000001e-06 2024-09-21T10:23:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:23:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.014261+0000", 2024-09-21T10:23:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00026039800000000001 2024-09-21T10:23:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:23:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.014454+0000", 2024-09-21T10:23:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019311900000000001 2024-09-21T10:23:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,7", 2024-09-21T10:23:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.014605+0000", 2024-09-21T10:23:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015064599999999999 2024-09-21T10:23:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:23:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:23:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:23:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.46025.0:115036 9.8 e316/300)", 2024-09-21T10:23:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:23:48.014612+0000", 2024-09-21T10:23:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0061164460000000002, 2024-09-21T10:23:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0064239969999999999, 2024-09-21T10:23:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:23:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:23:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-21T10:23:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:23:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:23:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.014612+0000", 2024-09-21T10:23:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:23:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:23:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.014612+0000", 2024-09-21T10:23:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:23:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:23:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.014612+0000", 2024-09-21T10:23:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0100000000000004e-07 2024-09-21T10:23:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:23:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.014616+0000", 2024-09-21T10:23:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.72e-06 2024-09-21T10:23:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:23:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.014625+0000", 2024-09-21T10:23:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8950000000000004e-06 2024-09-21T10:23:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:23:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.014631+0000", 2024-09-21T10:23:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8329999999999999e-06 2024-09-21T10:23:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:23:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:23:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:23:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.46025.0:115042 9.8 e316/300)", 2024-09-21T10:23:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:23:48.018288+0000", 2024-09-21T10:23:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.002440055, 2024-09-21T10:23:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002782531, 2024-09-21T10:23:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:23:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:23:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-21T10:23:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:23:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:23:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.018288+0000", 2024-09-21T10:23:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:23:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:23:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.018288+0000", 2024-09-21T10:23:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:23:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:23:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.018289+0000", 2024-09-21T10:23:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0070000000000001e-06 2024-09-21T10:23:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:23:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.018295+0000", 2024-09-21T10:23:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7180000000000001e-06 2024-09-21T10:23:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:23:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.018306+0000", 2024-09-21T10:23:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1272e-05 2024-09-21T10:23:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:23:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.018313+0000", 2024-09-21T10:23:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1790000000000002e-06 2024-09-21T10:23:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:23:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:23:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:23:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.46025.0:115026 9.5 e316/300 ondisk, result = 0)", 2024-09-21T10:23:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:23:48.018329+0000", 2024-09-21T10:23:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0023990080000000002, 2024-09-21T10:23:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027855530000000001, 2024-09-21T10:23:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:23:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:23:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:23:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:23:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:23:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.018329+0000", 2024-09-21T10:23:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:23:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:23:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.018329+0000", 2024-09-21T10:23:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:23:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:23:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.018329+0000", 2024-09-21T10:23:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1399999999999998e-07 2024-09-21T10:23:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:23:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.018332+0000", 2024-09-21T10:23:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3760000000000002e-06 2024-09-21T10:23:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:23:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.018338+0000", 2024-09-21T10:23:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6440000000000003e-06 2024-09-21T10:23:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:23:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.018343+0000", 2024-09-21T10:23:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.544e-06 2024-09-21T10:23:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:23:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.021067+0000", 2024-09-21T10:23:48.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027241140000000001 2024-09-21T10:23:48.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.054 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:23:48.054 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.021075+0000", 2024-09-21T10:23:48.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.977e-06 2024-09-21T10:23:48.054 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:23:48.054 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:23:48.054 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:23:48.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.054 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.46025.0:115039 9.5 e316/300)", 2024-09-21T10:23:48.054 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:23:48.018351+0000", 2024-09-21T10:23:48.054 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.002377171, 2024-09-21T10:23:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028133569999999998, 2024-09-21T10:23:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:23:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:23:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-21T10:23:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:23:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:23:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.018351+0000", 2024-09-21T10:23:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:23:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:23:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.018351+0000", 2024-09-21T10:23:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:23:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:23:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.018351+0000", 2024-09-21T10:23:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1199999999999999e-07 2024-09-21T10:23:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:23:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.018353+0000", 2024-09-21T10:23:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0940000000000002e-06 2024-09-21T10:23:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:23:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.018359+0000", 2024-09-21T10:23:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8980000000000001e-06 2024-09-21T10:23:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:23:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:23:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:23:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:23:48.018364+0000", 2024-09-21T10:23:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1209999999999998e-06 2024-09-21T10:23:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:23:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:23:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:23:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:23:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-21T10:23:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 8 2024-09-21T10:23:48.057 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:23:48.089 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:48.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:48.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:48.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.234 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:48.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:48.245 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 75 16 115308 115292 96.0324 71 0.0109193 0.0104086 2024-09-21T10:23:48.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:48.367 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:48.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:48.468 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:48.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.505 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:48.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:48.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:48.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.642 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:48.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:48.743 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:48.787 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:48.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:48.888 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:48.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.927 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:48.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:49.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:49.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:49.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:49.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:49.202 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:49.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:49.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:49.245 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 76 16 116518 116502 95.7639 75.625 0.0148912 0.0104373 2024-09-21T10:23:49.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:49.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:49.343 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:49.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:49.355 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:23:49.355 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:23:49.394 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:23:49.394 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:23:49.427 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:23:49.427 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:23:49.427 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:49 smithi098 ceph-mon[97580]: pgmap v1052: 99 pgs: 99 active+clean; 6.8 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-21T10:23:49.428 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:49 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1312344762' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:49.428 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:49 smithi098 ceph-mon[100914]: pgmap v1052: 99 pgs: 99 active+clean; 6.8 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-21T10:23:49.428 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:49 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1312344762' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:49.444 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:49.457 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:23:49.457 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:23:49.481 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:49.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:49.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:49.486 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:23:49.486 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:23:49.520 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:23:49.520 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:23:49.552 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:23:49.553 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:23:49.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:49.586 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:23:49.586 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:23:49.624 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:49.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:49.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:49.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:49 smithi196 ceph-mon[84784]: pgmap v1052: 99 pgs: 99 active+clean; 6.8 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-21T10:23:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:49 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1312344762' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:49.726 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:49.766 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:49.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:49.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:49.867 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:49.900 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:49.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:49.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:50.002 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:50.044 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:50.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:50.146 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:50.174 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:50.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:50.245 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 77 16 117668 117652 95.4537 71.875 0.0118891 0.0104712 2024-09-21T10:23:50.276 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:50.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.311 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:50.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:50.412 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:50.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:50.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:50.584 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:50.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:50.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:23:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:23:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:23:50.686 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:50.724 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:50.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:50.826 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:50.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.861 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:50.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:50.962 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:50.991 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:50.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:51.093 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:51.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:23:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:23:50] "GET /metrics HTTP/1.1" 200 36520 "" "Prometheus/2.51.0" 2024-09-21T10:23:51.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:51.129 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:51.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:51.230 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:51.245 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 78 16 118878 118862 95.1994 75.625 0.0169135 0.0104996 2024-09-21T10:23:51.271 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:51.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:51.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:51.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:51.405 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:23:51.411 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:51.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:51.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:51.513 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:51.551 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:51.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:51.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:51.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:51 smithi098 ceph-mon[97580]: pgmap v1053: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-21T10:23:51.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:51 smithi098 ceph-mon[100914]: pgmap v1053: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-21T10:23:51.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:51 smithi196 ceph-mon[84784]: pgmap v1053: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-21T10:23:51.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:51.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:51.691 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:51.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:51.792 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:51.830 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:51.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:51.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:51.931 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:51.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:51.964 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:51.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:52.065 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:52.097 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:52.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:52.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:52.199 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:52.233 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:52.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:52.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:52.246 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 79 16 120034 120018 94.909 72.25 0.0191932 0.0105315 2024-09-21T10:23:52.335 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:52.366 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:52.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:52.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:52.468 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:52.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:52.503 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:52.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:52.605 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:52.641 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:52.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:52.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:52.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:52.777 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:52.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:52.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:52.879 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:52.915 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:52.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:52.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:53.016 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:53.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.056 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:53.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:53.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:53.186 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:53.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:53.219 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:53.246 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:23:53.244698+0000 min lat: 0.00136025 max lat: 0.247537 avg lat: 0.0105591 2024-09-21T10:23:53.246 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:23:53.246 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 80 16 121240 121224 94.6648 75.375 0.0085194 0.0105591 2024-09-21T10:23:53.287 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:53.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.326 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:53.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:53.427 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:53.470 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:53.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:53.572 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:53.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:53 smithi098 ceph-mon[97580]: pgmap v1054: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-21T10:23:53.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:53.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:53.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:53 smithi098 ceph-mon[100914]: pgmap v1054: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-21T10:23:53.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:53.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:53.608 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:53.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:53.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:53 smithi196 ceph-mon[84784]: pgmap v1054: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-21T10:23:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:23:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:53.710 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:53.731 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-21T10:23:53.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:53.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:53.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:53.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.883 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:53.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:53.891 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:23:53.984 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:54.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:54.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:54.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:54.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:54.174 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:54.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:54.248 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 81 16 122379 122363 94.3717 71.1875 0.0147617 0.0105914 2024-09-21T10:23:54.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:54.306 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:54.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:54.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:54.407 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:54.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:54.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:54.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:54.536 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:54.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:54.571 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:54.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:54.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:54.709 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:54.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:54.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:54.811 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:54.844 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:54.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:54.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:54.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:54.981 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:54.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:54.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:55.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:55.119 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:55.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:55.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:55.221 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:55.246 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:55.249 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 82 16 123527 123511 94.0952 71.75 0.00842508 0.010623 2024-09-21T10:23:55.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:55.258 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:55.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:55.258 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:55 smithi098 ceph-mon[100914]: pgmap v1055: 99 pgs: 99 active+clean; 7.2 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-09-21T10:23:55.259 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:55 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2518561580' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:55.269 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:23:55.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:55.403 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:55.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:55.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:55.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:55.536 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:55.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:55.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:55.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:55 smithi098 ceph-mon[97580]: pgmap v1055: 99 pgs: 99 active+clean; 7.2 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-09-21T10:23:55.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:55 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2518561580' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:55.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:55 smithi196 ceph-mon[84784]: pgmap v1055: 99 pgs: 99 active+clean; 7.2 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-09-21T10:23:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:55 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2518561580' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:55.675 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:55.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:55.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:55.777 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:55.808 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:55.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:55.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:55.909 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:55.944 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:55.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:55.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:56.045 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:56.081 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:56.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:56.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:56.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:56.204 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:56.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:56.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:56.249 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 83 16 124631 124615 93.7929 69 0.014465 0.010657 2024-09-21T10:23:56.305 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:56.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:56.338 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:56.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:56.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:56 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:56.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:56 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2518561580' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:23:56.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:56 smithi098 ceph-mon[97580]: osdmap e317: 8 total, 8 up, 6 in 2024-09-21T10:23:56.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:56 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:56.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:56 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2518561580' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:23:56.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:56 smithi098 ceph-mon[100914]: osdmap e317: 8 total, 8 up, 6 in 2024-09-21T10:23:56.440 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:56.472 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:56.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:56.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:56.573 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:56.609 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:56.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:56.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:56 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:56 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2518561580' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:23:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:56 smithi196 ceph-mon[84784]: osdmap e317: 8 total, 8 up, 6 in 2024-09-21T10:23:56.711 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:56.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:56.747 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:56.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:56.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:56.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:56.883 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:56.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:56.984 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:57.017 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:57.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:57.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:57.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.154 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:57.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:57.249 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 84 16 125730 125714 93.4941 68.6875 0.00926943 0.0106915 2024-09-21T10:23:57.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:57.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:57.392 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:57.425 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:57.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:57.527 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:57.569 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:57.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:57.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:57 smithi098 ceph-mon[97580]: pgmap v1057: 99 pgs: 99 active+clean; 7.4 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-21T10:23:57.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:57 smithi098 ceph-mon[100914]: pgmap v1057: 99 pgs: 99 active+clean; 7.4 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-21T10:23:57.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:57 smithi196 ceph-mon[84784]: pgmap v1057: 99 pgs: 99 active+clean; 7.4 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-21T10:23:57.671 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:57.707 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:57.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:57.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:57.846 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:57.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:57.948 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:57.980 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:57.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:58.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:58.120 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:58.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:58.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:58.221 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:58.249 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 85 16 126919 126903 93.2684 74.3125 0.0160355 0.0107173 2024-09-21T10:23:58.252 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:58.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:58.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:58.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:58.386 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:58.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:58.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:58.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:58.522 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:58.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:58.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:58.624 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:58.656 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:58.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:58.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:58.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:58.791 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:58.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:58.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:58.893 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:58.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:58.929 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:58.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:59.030 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:59.067 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:59.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:59.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:59.168 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:59.193 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:59.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:59.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:59.249 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 86 16 128086 128070 93.032 72.9375 0.00479455 0.0107445 2024-09-21T10:23:59.295 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:59.333 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:59.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:59.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:59.435 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:59.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:59.472 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:59.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:59.573 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:59.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:23:59 smithi098 ceph-mon[97580]: pgmap v1058: 99 pgs: 99 active+clean; 7.5 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T10:23:59.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:23:59 smithi098 ceph-mon[100914]: pgmap v1058: 99 pgs: 99 active+clean; 7.5 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T10:23:59.608 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:59.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:59.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:59.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:23:59 smithi196 ceph-mon[84784]: pgmap v1058: 99 pgs: 99 active+clean; 7.5 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T10:23:59.710 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:59.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:59.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:23:59.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:59.844 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:59.880 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:23:59.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:59.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:59.982 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:00.021 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:00.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:00.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:00.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:00.153 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:00.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:00.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:00.250 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 87 16 129248 129232 92.7975 72.625 0.0116375 0.0107717 2024-09-21T10:24:00.255 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:00.293 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:00.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:00.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:00.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:00.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:00.426 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:00.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:00.527 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:00.560 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:00.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:00.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:00.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:24:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:24:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:24:00.662 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:00.703 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:00.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:00.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:00.805 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:00.836 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:00.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:00.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:00.937 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:00.972 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:00.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:00.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:01.042 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:24:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:24:00] "GET /metrics HTTP/1.1" 200 36525 "" "Prometheus/2.51.0" 2024-09-21T10:24:01.074 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:01.126 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:01.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:01.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:01.228 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:01.231 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:24:01.250 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 88 16 130456 130440 92.6008 75.5 0.0124021 0.0107945 2024-09-21T10:24:01.262 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:01.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:01.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:01.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:01 smithi098 ceph-mon[97580]: pgmap v1059: 99 pgs: 99 active+clean; 7.7 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T10:24:01.343 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:01 smithi098 ceph-mon[100914]: pgmap v1059: 99 pgs: 99 active+clean; 7.7 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T10:24:01.363 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:01.395 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:01.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:01.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:01.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:01.547 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:01.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:01.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:01.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:01 smithi196 ceph-mon[84784]: pgmap v1059: 99 pgs: 99 active+clean; 7.7 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T10:24:01.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:01.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:01.687 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:01.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:01.788 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:01.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:01.828 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:01.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:01.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:01.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:01.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:02.070 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:02.112 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:02.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:02.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:02.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:02.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:02.248 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:02.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:02.250 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 89 16 131583 131567 92.3517 70.4375 0.0166295 0.0108236 2024-09-21T10:24:02.350 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:02.391 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:02.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:02.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:02.492 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:02.525 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:02.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:02.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:02.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:02.661 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:02.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:02.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:02.762 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:02.797 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:02.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:02.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:02.899 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:02.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:02.934 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:02.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:03.035 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:03.070 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:03.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:03.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:03.172 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:03.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:03.208 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:03.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:03.250 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Total time run: 90.0036 2024-09-21T10:24:03.250 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Total writes made: 132728 2024-09-21T10:24:03.251 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Write size: 65536 2024-09-21T10:24:03.251 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Object size: 65536 2024-09-21T10:24:03.251 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Bandwidth (MB/sec): 92.1685 2024-09-21T10:24:03.251 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Stddev Bandwidth: 37.9463 2024-09-21T10:24:03.251 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Max bandwidth (MB/sec): 254.625 2024-09-21T10:24:03.251 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Min bandwidth (MB/sec): 56.5 2024-09-21T10:24:03.251 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Average IOPS: 1474 2024-09-21T10:24:03.251 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Stddev IOPS: 607.141 2024-09-21T10:24:03.251 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Max IOPS: 4074 2024-09-21T10:24:03.251 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Min IOPS: 904 2024-09-21T10:24:03.251 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Average Latency(s): 0.0108459 2024-09-21T10:24:03.251 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Stddev Latency(s): 0.00956165 2024-09-21T10:24:03.251 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Max latency(s): 0.247537 2024-09-21T10:24:03.251 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Min latency(s): 0.00136025 2024-09-21T10:24:03.257 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:24:03.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:03.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:03.344 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:03.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:03.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:03 smithi098 ceph-mon[97580]: pgmap v1060: 99 pgs: 99 active+clean; 7.7 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T10:24:03.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:03 smithi098 ceph-mon[100914]: pgmap v1060: 99 pgs: 99 active+clean; 7.7 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T10:24:03.445 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:03.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:03.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:03.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:03.616 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:03.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:03.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:03.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:03 smithi196 ceph-mon[84784]: pgmap v1060: 99 pgs: 99 active+clean; 7.7 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T10:24:03.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:03.752 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:03.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:03.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:03.853 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:03.892 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:03.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:03.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:03.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:04.023 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:04.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:04.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:04.125 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:04.166 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:04.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:04.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:04.268 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:04.297 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:04.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:04.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:04.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:04.436 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:04.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:04.537 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:04.569 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:04.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:04.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:04.671 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:04.703 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:04.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:04.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:04.804 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:04.841 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:04.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:04.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:04.943 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:04.972 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:04.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:04.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:05.073 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:05.116 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:05.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:05.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:05.217 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:05.254 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:05.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:05.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:05.355 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:05.388 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:05.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:05.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:05.490 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:05.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:05.526 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:05.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:05.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:05 smithi098 ceph-mon[97580]: pgmap v1061: 99 pgs: 99 active+clean; 7.9 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-21T10:24:05.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:05 smithi098 ceph-mon[100914]: pgmap v1061: 99 pgs: 99 active+clean; 7.9 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-21T10:24:05.627 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:05 smithi196 ceph-mon[84784]: pgmap v1061: 99 pgs: 99 active+clean; 7.9 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-21T10:24:05.668 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:05.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:05.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:05.770 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:05.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:05.806 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:05.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:05.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:05.946 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:05.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:05.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:06.048 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:06.083 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:06.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:06.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:06.185 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:06.227 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:06.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:06.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:06.330 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:06.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:06.370 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:06.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:06.435 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-21T10:24:06.471 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:06.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:06.515 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:06.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:06.616 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:06.674 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:06.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:06.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:06.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:06.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:06.806 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:06.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:06.908 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:06.949 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:06.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:06.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:07.050 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:07.086 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:07.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:07.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:07.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:07.232 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:07.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:07.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:07.303 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:24:07.303 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:24:07.303 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-21T10:24:07.303 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-21T10:24:07.303 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:24:07.334 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:07.365 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:07.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:07.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:07.380 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:24:07.380 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:24:07.380 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:24:07.380 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:24:07.380 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:24:07.380 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:24:07.380 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:24:07.380 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:24:07.380 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:24:07.381 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:24:07.391 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:24:07.391 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:24:07.426 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:24:07.426 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:24:07.450 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:24:07.450 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:24:07.467 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:07.514 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:07.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:07.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:07 smithi098 ceph-mon[97580]: pgmap v1062: 99 pgs: 99 active+clean; 8.1 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 100 B/s rd, 69 MiB/s wr, 1.13k op/s 2024-09-21T10:24:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:07 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3966344414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:07.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:07 smithi098 ceph-mon[100914]: pgmap v1062: 99 pgs: 99 active+clean; 8.1 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 100 B/s rd, 69 MiB/s wr, 1.13k op/s 2024-09-21T10:24:07.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:07 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3966344414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:07.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:07.615 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:07 smithi196 ceph-mon[84784]: pgmap v1062: 99 pgs: 99 active+clean; 8.1 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 100 B/s rd, 69 MiB/s wr, 1.13k op/s 2024-09-21T10:24:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:07 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3966344414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:07.650 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:07.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:07.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:07.752 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:07.788 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:07.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:07.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:07.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:07.924 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:07.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:07.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:08.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:08.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:08.061 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:08.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:08.126 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:24:08.162 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:08.193 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:08.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:08.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:08.295 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:08.302 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:08.321 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:08.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:08.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:08.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:08.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:08.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:08.563 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:08.594 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:08.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:08.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:08.696 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:08.734 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:08.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:08.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:08.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:08.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:08.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:08.970 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:09.003 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:09.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:09.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:09.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:09.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:09.151 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:09.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:09.251 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:09.288 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:09.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:09.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:09.390 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:09.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:09.426 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:09.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:09.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:09 smithi098 ceph-mon[97580]: pgmap v1063: 99 pgs: 99 active+clean; 8.0 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 85 B/s rd, 61 MiB/s wr, 1.12k op/s 2024-09-21T10:24:09.524 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:09 smithi098 ceph-mon[100914]: pgmap v1063: 99 pgs: 99 active+clean; 8.0 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 85 B/s rd, 61 MiB/s wr, 1.12k op/s 2024-09-21T10:24:09.527 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:09.560 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:09.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:09.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:09.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:09 smithi196 ceph-mon[84784]: pgmap v1063: 99 pgs: 99 active+clean; 8.0 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 85 B/s rd, 61 MiB/s wr, 1.12k op/s 2024-09-21T10:24:09.662 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:09.697 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:09.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:09.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:09.799 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:09.834 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:09.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:09.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:09.842 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:09.937 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:09.968 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:09.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:09.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:10.069 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:10.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:10.108 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:10.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:10.209 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:10.246 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:10.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:10.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:10.347 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:10.383 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:10.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:10.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:10.485 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:10.521 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:10.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:10.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:10.623 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:10.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:24:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:24:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:24:10.658 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:10.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:10.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:10.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:10.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:10.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:10.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:24:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:24:10] "GET /metrics HTTP/1.1" 200 36525 "" "Prometheus/2.51.0" 2024-09-21T10:24:10.897 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:10.934 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:10.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:10.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:10.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:24:11.036 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:11.074 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:11.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:11.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:11.176 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:11.207 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:11.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:11.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:11.308 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:11.349 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:11.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:11.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:11.451 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:11.477 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:11.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:11.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:11.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:11.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:11 smithi098 ceph-mon[97580]: pgmap v1064: 99 pgs: 99 active+clean; 7.8 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 85 B/s rd, 52 MiB/s wr, 1.29k op/s 2024-09-21T10:24:11.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:11 smithi098 ceph-mon[100914]: pgmap v1064: 99 pgs: 99 active+clean; 7.8 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 85 B/s rd, 52 MiB/s wr, 1.29k op/s 2024-09-21T10:24:11.614 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:11.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:11.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:11.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:11 smithi196 ceph-mon[84784]: pgmap v1064: 99 pgs: 99 active+clean; 7.8 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 85 B/s rd, 52 MiB/s wr, 1.29k op/s 2024-09-21T10:24:11.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:11.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:11.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:11.849 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:11.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:11.881 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:11.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:11.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:12.020 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:12.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:12.122 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:12.154 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:12.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:12.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:12.292 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:12.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:12.394 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:12.437 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:12.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:12.539 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:12.579 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:12.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:12.680 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:12.707 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:12.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:12.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:12.843 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:12.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:12.945 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:12.980 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:12.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:13.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:13.114 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:13.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:13.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:13.141 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:13.216 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:13.247 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:13.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:13.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:13.324 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:24:13.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:13.381 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:13.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:13.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:13.466 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:13 smithi098 ceph-mon[97580]: pgmap v1065: 99 pgs: 99 active+clean; 7.7 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 85 B/s rd, 33 MiB/s wr, 1.02k op/s 2024-09-21T10:24:13.466 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:13 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3657407306' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:24:13.466 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:13 smithi098 ceph-mon[100914]: pgmap v1065: 99 pgs: 99 active+clean; 7.7 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 85 B/s rd, 33 MiB/s wr, 1.02k op/s 2024-09-21T10:24:13.466 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:13 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3657407306' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:24:13.483 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:13.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:13.519 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:13.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:13.620 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:13.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:13 smithi196 ceph-mon[84784]: pgmap v1065: 99 pgs: 99 active+clean; 7.7 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 85 B/s rd, 33 MiB/s wr, 1.02k op/s 2024-09-21T10:24:13.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:13 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3657407306' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:24:13.651 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:13.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:13.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:13.754 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:13.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:13.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:13.888 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:13.926 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:13.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:13.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:14.027 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:14.065 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:14.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:14.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:14.167 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:14.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:14.207 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:14.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:14.308 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:14.342 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:14.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:14.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:14.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:14.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:14.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:14.576 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:14.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:14 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:14.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:14 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3657407306' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:24:14.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:14 smithi098 ceph-mon[97580]: osdmap e318: 8 total, 8 up, 6 in 2024-09-21T10:24:14.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:14 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:14.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:14 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3657407306' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:24:14.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:14 smithi098 ceph-mon[100914]: osdmap e318: 8 total, 8 up, 6 in 2024-09-21T10:24:14.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:14.618 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:14.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:14.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:14 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:14 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3657407306' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:24:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:14 smithi196 ceph-mon[84784]: osdmap e318: 8 total, 8 up, 6 in 2024-09-21T10:24:14.719 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:14.753 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:14.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:14.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:14.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:14.885 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:14.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:14.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:14.987 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:15.025 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:15.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:15.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:15.126 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:15.159 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:15.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:15.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:15.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:15.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:15.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:15.400 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:15.437 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:15.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:15.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:15.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:15.576 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:15 smithi098 ceph-mon[97580]: pgmap v1067: 99 pgs: 99 active+clean; 7.5 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 102 B/s rd, 17 MiB/s wr, 1.22k op/s 2024-09-21T10:24:15.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:15 smithi098 ceph-mon[100914]: pgmap v1067: 99 pgs: 99 active+clean; 7.5 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 102 B/s rd, 17 MiB/s wr, 1.22k op/s 2024-09-21T10:24:15.578 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:15.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:15.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:15.619 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-21T10:24:15.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:15 smithi196 ceph-mon[84784]: pgmap v1067: 99 pgs: 99 active+clean; 7.5 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 102 B/s rd, 17 MiB/s wr, 1.22k op/s 2024-09-21T10:24:15.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:15.728 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:15.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:15.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:15.830 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:15.864 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:15.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:15.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:15.965 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:16.002 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:16.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:16.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:16.104 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:16.142 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:16.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:16.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:16.243 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:16.279 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:16.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:16.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:16.381 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:16.419 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:16.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:16.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:16.521 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:16.557 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:16.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:16.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:16.659 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:16.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:16.701 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:16.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:16.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:16.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:16.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:16.935 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:16.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:16.970 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:16.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:17.071 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:17.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:17.106 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:17.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:17.207 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:17.239 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:17.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:17.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:17.341 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:17.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:17.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:17.476 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:17.519 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:17.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:17.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:17.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:17 smithi098 ceph-mon[97580]: pgmap v1068: 99 pgs: 99 active+clean; 7.4 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 2.8 MiB/s wr, 1.24k op/s 2024-09-21T10:24:17.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:17 smithi098 ceph-mon[100914]: pgmap v1068: 99 pgs: 99 active+clean; 7.4 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 2.8 MiB/s wr, 1.24k op/s 2024-09-21T10:24:17.621 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:17.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:17 smithi196 ceph-mon[84784]: pgmap v1068: 99 pgs: 99 active+clean; 7.4 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 2.8 MiB/s wr, 1.24k op/s 2024-09-21T10:24:17.657 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:17.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:17.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:17.759 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:17.791 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:17.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:17.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:17.892 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:17.933 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:17.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:17.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:18.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:18.073 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:18.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:18.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:18.175 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:18.207 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:18.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:18.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:18.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:18.350 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:18.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:18.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:18.452 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:18.489 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:18.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:18.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:18.590 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:18.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:18.622 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:18.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:18.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:18.759 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:18.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:18.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:18.861 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:18.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:18.899 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:18.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:18.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:24:19.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:19.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:19.039 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:19.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:19.140 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:19.181 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:19.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:19.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:19.283 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:19.320 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:19.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:19.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:19.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:19.462 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:19.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:19.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:19.566 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:19.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:19 smithi098 ceph-mon[97580]: pgmap v1069: 99 pgs: 99 active+clean; 7.2 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 0 B/s wr, 1.23k op/s 2024-09-21T10:24:19.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:19 smithi098 ceph-mon[100914]: pgmap v1069: 99 pgs: 99 active+clean; 7.2 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 0 B/s wr, 1.23k op/s 2024-09-21T10:24:19.602 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:19.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:19.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:19.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:19 smithi196 ceph-mon[84784]: pgmap v1069: 99 pgs: 99 active+clean; 7.2 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 0 B/s wr, 1.23k op/s 2024-09-21T10:24:19.704 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:19.733 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:19.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:19.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:19.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:19.874 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:19.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:19.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:19.976 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:20.018 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:20.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:20.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:20.148 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:20.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:20.250 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:20.283 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:20.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:20.384 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:20.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.422 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:20.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:20.522 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:20.544 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:20.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:20.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:20 smithi196 ceph-mon[84784]: pgmap v1070: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 0 B/s wr, 1.24k op/s 2024-09-21T10:24:20.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:24:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:24:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:24:20.646 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:20.661 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:20 smithi098 ceph-mon[97580]: pgmap v1070: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 0 B/s wr, 1.24k op/s 2024-09-21T10:24:20.661 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:20 smithi098 ceph-mon[100914]: pgmap v1070: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 0 B/s wr, 1.24k op/s 2024-09-21T10:24:20.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.677 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:20.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:20.779 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:20.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.819 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:20.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:20.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:20.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.959 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:20.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:21.060 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:21.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:24:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:24:20] "GET /metrics HTTP/1.1" 200 36525 "" "Prometheus/2.51.0" 2024-09-21T10:24:21.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:21.100 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:21.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:21.201 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:21.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:21.234 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:21.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:21.335 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:21.371 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:21.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:21.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:21.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:21.496 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:21.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:21.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:21.598 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:21.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:21.643 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:21.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:21.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:21.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:21.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:21.885 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:21.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:21.919 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:21.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:22.020 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:22.053 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:22.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:22.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:22.154 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:22.195 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:22.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:22.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:22.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:22.328 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:22.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:22.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:22.431 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:22.466 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:22.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:22.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:22.478 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:24:22.478 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-21T10:24:22.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-21T10:24:22.478 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:24:22.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.479 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:126668 7.16 7:68e12100:::benchmark_data_smithi098_197834_object10151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:24:22.479 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:14:56.015607+0000", 2024-09-21T10:24:22.479 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 566.06448345700005, 2024-09-21T10:24:22.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8170184630000001, 2024-09-21T10:24:22.479 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:24:22.479 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:24:22.479 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:22.479 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:24:22.479 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:24:22.479 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:24:22.479 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 126668 2024-09-21T10:24:22.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.480 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:24:22.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:24:22.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.015607+0000", 2024-09-21T10:24:22.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:24:22.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.015607+0000", 2024-09-21T10:24:22.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:24:22.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.015609+0000", 2024-09-21T10:24:22.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.533e-06 2024-09-21T10:24:22.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:24:22.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.015614+0000", 2024-09-21T10:24:22.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2140000000000004e-06 2024-09-21T10:24:22.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:24:22.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.015624+0000", 2024-09-21T10:24:22.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.815e-06 2024-09-21T10:24:22.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:24:22.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.015630+0000", 2024-09-21T10:24:22.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1369999999999999e-06 2024-09-21T10:24:22.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.017016+0000", 2024-09-21T10:24:22.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001386013 2024-09-21T10:24:22.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.017025+0000", 2024-09-21T10:24:22.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1409999999999994e-06 2024-09-21T10:24:22.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.699261+0000", 2024-09-21T10:24:22.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.68223615599999998 2024-09-21T10:24:22.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.699273+0000", 2024-09-21T10:24:22.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1814000000000001e-05 2024-09-21T10:24:22.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.825506+0000", 2024-09-21T10:24:22.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.126233045 2024-09-21T10:24:22.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:24:22.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.825671+0000", 2024-09-21T10:24:22.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016523300000000001 2024-09-21T10:24:22.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:24:22.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.825793+0000", 2024-09-21T10:24:22.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012155 2024-09-21T10:24:22.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.830734+0000", 2024-09-21T10:24:22.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0049411480000000002 2024-09-21T10:24:22.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.830771+0000", 2024-09-21T10:24:22.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6936e-05 2024-09-21T10:24:22.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:24:22.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.830783+0000", 2024-09-21T10:24:22.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2553e-05 2024-09-21T10:24:22.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:24:22.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.830821+0000", 2024-09-21T10:24:22.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7963999999999998e-05 2024-09-21T10:24:22.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:24:22.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.832626+0000", 2024-09-21T10:24:22.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001804211 2024-09-21T10:24:22.486 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.486 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:24:22.486 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.486 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:126698 7.16 7:6d3aa5f6:::benchmark_data_smithi098_197834_object10181:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:24:22.487 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:14:56.033118+0000", 2024-09-21T10:24:22.487 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 566.04697303299997, 2024-09-21T10:24:22.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.801116078, 2024-09-21T10:24:22.487 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:24:22.487 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:24:22.487 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:22.487 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:24:22.487 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:24:22.487 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:24:22.487 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 126698 2024-09-21T10:24:22.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.487 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:24:22.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:24:22.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.033118+0000", 2024-09-21T10:24:22.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:24:22.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.033118+0000", 2024-09-21T10:24:22.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:24:22.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.033119+0000", 2024-09-21T10:24:22.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2890000000000001e-06 2024-09-21T10:24:22.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:24:22.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.033123+0000", 2024-09-21T10:24:22.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6410000000000001e-06 2024-09-21T10:24:22.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:24:22.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.033135+0000", 2024-09-21T10:24:22.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1348999999999999e-05 2024-09-21T10:24:22.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:24:22.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.033142+0000", 2024-09-21T10:24:22.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2019999999999998e-06 2024-09-21T10:24:22.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.033216+0000", 2024-09-21T10:24:22.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3530999999999998e-05 2024-09-21T10:24:22.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.033225+0000", 2024-09-21T10:24:22.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0799999999999995e-06 2024-09-21T10:24:22.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.699403+0000", 2024-09-21T10:24:22.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.66617875199999999 2024-09-21T10:24:22.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.699413+0000", 2024-09-21T10:24:22.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9110000000000005e-06 2024-09-21T10:24:22.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.828983+0000", 2024-09-21T10:24:22.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1295696719999999 2024-09-21T10:24:22.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:24:22.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.829080+0000", 2024-09-21T10:24:22.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6639999999999996e-05 2024-09-21T10:24:22.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:24:22.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.829178+0000", 2024-09-21T10:24:22.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8247999999999999e-05 2024-09-21T10:24:22.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.832692+0000", 2024-09-21T10:24:22.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035141450000000002 2024-09-21T10:24:22.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.832758+0000", 2024-09-21T10:24:22.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5729999999999996e-05 2024-09-21T10:24:22.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:24:22.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.832774+0000", 2024-09-21T10:24:22.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6767000000000001e-05 2024-09-21T10:24:22.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:24:22.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.832827+0000", 2024-09-21T10:24:22.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2122e-05 2024-09-21T10:24:22.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:24:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.834234+0000", 2024-09-21T10:24:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0014069989999999999 2024-09-21T10:24:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:24:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:126813 7.16 7:6a5b5ef2:::benchmark_data_smithi098_197834_object10296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:24:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:14:56.141981+0000", 2024-09-21T10:24:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 565.93810969900005, 2024-09-21T10:24:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.783015048, 2024-09-21T10:24:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:24:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:24:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:24:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:24:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:24:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 126813 2024-09-21T10:24:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:24:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:24:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.141981+0000", 2024-09-21T10:24:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:24:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.141981+0000", 2024-09-21T10:24:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:24:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.141983+0000", 2024-09-21T10:24:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.933e-06 2024-09-21T10:24:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:24:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.141988+0000", 2024-09-21T10:24:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5559999999999998e-06 2024-09-21T10:24:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:24:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.142009+0000", 2024-09-21T10:24:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0928999999999999e-05 2024-09-21T10:24:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:24:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.142018+0000", 2024-09-21T10:24:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2150000000000007e-06 2024-09-21T10:24:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.145154+0000", 2024-09-21T10:24:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031358969999999999 2024-09-21T10:24:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.145164+0000", 2024-09-21T10:24:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9289999999999993e-06 2024-09-21T10:24:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.699455+0000", 2024-09-21T10:24:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.55429031399999995 2024-09-21T10:24:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.699463+0000", 2024-09-21T10:24:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6349999999999995e-06 2024-09-21T10:24:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.834271+0000", 2024-09-21T10:24:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1348073839999999 2024-09-21T10:24:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:24:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.834381+0000", 2024-09-21T10:24:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000110729 2024-09-21T10:24:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:24:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.834478+0000", 2024-09-21T10:24:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6676999999999996e-05 2024-09-21T10:24:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.840122+0000", 2024-09-21T10:24:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056437630000000004 2024-09-21T10:24:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:24:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.840204+0000", 2024-09-21T10:24:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2526000000000002e-05 2024-09-21T10:24:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.922789+0000", 2024-09-21T10:24:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.082584802999999998 2024-09-21T10:24:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:24:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.922856+0000", 2024-09-21T10:24:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6940000000000006e-05 2024-09-21T10:24:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:24:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.924996+0000", 2024-09-21T10:24:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0021398179999999999 2024-09-21T10:24:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:24:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:126858 7.16 7:6a6893ad:::benchmark_data_smithi098_197834_object10341:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:24:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:14:56.182825+0000", 2024-09-21T10:24:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 565.897266006, 2024-09-21T10:24:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.746662014, 2024-09-21T10:24:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:24:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:24:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:24:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:24:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:24:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 126858 2024-09-21T10:24:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:24:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:24:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182825+0000", 2024-09-21T10:24:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:24:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182825+0000", 2024-09-21T10:24:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:24:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182826+0000", 2024-09-21T10:24:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0950000000000001e-06 2024-09-21T10:24:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:24:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182830+0000", 2024-09-21T10:24:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6600000000000003e-06 2024-09-21T10:24:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:24:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182840+0000", 2024-09-21T10:24:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0182e-05 2024-09-21T10:24:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:24:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182847+0000", 2024-09-21T10:24:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6760000000000001e-06 2024-09-21T10:24:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182960+0000", 2024-09-21T10:24:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000112698 2024-09-21T10:24:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182969+0000", 2024-09-21T10:24:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8410000000000006e-06 2024-09-21T10:24:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.699504+0000", 2024-09-21T10:24:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.516535623 2024-09-21T10:24:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.699510+0000", 2024-09-21T10:24:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5709999999999997e-06 2024-09-21T10:24:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.836487+0000", 2024-09-21T10:24:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.136976695 2024-09-21T10:24:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:24:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.836645+0000", 2024-09-21T10:24:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000158247 2024-09-21T10:24:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:24:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.836788+0000", 2024-09-21T10:24:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014280099999999999 2024-09-21T10:24:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.840186+0000", 2024-09-21T10:24:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003398098 2024-09-21T10:24:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:24:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.840212+0000", 2024-09-21T10:24:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5944000000000001e-05 2024-09-21T10:24:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.925064+0000", 2024-09-21T10:24:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.084852219000000006 2024-09-21T10:24:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:24:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.925132+0000", 2024-09-21T10:24:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7625000000000004e-05 2024-09-21T10:24:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:24:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.929487+0000", 2024-09-21T10:24:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043550389999999998 2024-09-21T10:24:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:24:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:126859 7.16 7:6e5030ca:::benchmark_data_smithi098_197834_object10342:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:24:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:14:56.182867+0000", 2024-09-21T10:24:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 565.89722392199997, 2024-09-21T10:24:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7560969719999999, 2024-09-21T10:24:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:24:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:24:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:24:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:24:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:24:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 126859 2024-09-21T10:24:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:24:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:24:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182867+0000", 2024-09-21T10:24:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:24:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182867+0000", 2024-09-21T10:24:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:24:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182867+0000", 2024-09-21T10:24:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.63e-07 2024-09-21T10:24:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:24:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182869+0000", 2024-09-21T10:24:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9080000000000002e-06 2024-09-21T10:24:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:24:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182875+0000", 2024-09-21T10:24:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0900000000000001e-06 2024-09-21T10:24:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:24:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182880+0000", 2024-09-21T10:24:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8879999999999996e-06 2024-09-21T10:24:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182991+0000", 2024-09-21T10:24:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000110759 2024-09-21T10:24:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.182997+0000", 2024-09-21T10:24:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0719999999999996e-06 2024-09-21T10:24:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.699575+0000", 2024-09-21T10:24:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.51657803499999999 2024-09-21T10:24:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:56.699579+0000", 2024-09-21T10:24:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7859999999999998e-06 2024-09-21T10:24:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.838504+0000", 2024-09-21T10:24:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1389254849999999 2024-09-21T10:24:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:24:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.838616+0000", 2024-09-21T10:24:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000111607 2024-09-21T10:24:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:24:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.838700+0000", 2024-09-21T10:24:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4171000000000004e-05 2024-09-21T10:24:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.840240+0000", 2024-09-21T10:24:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001540144 2024-09-21T10:24:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:24:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.841268+0000", 2024-09-21T10:24:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010283899999999999 2024-09-21T10:24:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.936801+0000", 2024-09-21T10:24:22.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.095532814999999993 2024-09-21T10:24:22.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:24:22.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.936855+0000", 2024-09-21T10:24:22.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3912000000000002e-05 2024-09-21T10:24:22.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:24:22.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:14:57.938964+0000", 2024-09-21T10:24:22.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0021084469999999998 2024-09-21T10:24:22.517 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.517 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:24:22.517 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.517 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:183398 7.6 7:677fe744:::benchmark_data_smithi098_197834_object66881:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-09-21T10:24:22.517 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:16:03.629420+0000", 2024-09-21T10:24:22.517 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 498.45067057099999, 2024-09-21T10:24:22.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.423753906, 2024-09-21T10:24:22.517 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:24:22.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:24:22.517 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:22.517 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:24:22.518 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:24:22.518 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:24:22.518 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 183398 2024-09-21T10:24:22.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.518 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:24:22.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:24:22.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.629420+0000", 2024-09-21T10:24:22.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:24:22.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.629420+0000", 2024-09-21T10:24:22.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:24:22.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.629420+0000", 2024-09-21T10:24:22.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7e-07 2024-09-21T10:24:22.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:24:22.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.629423+0000", 2024-09-21T10:24:22.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1229999999999998e-06 2024-09-21T10:24:22.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:24:22.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.629429+0000", 2024-09-21T10:24:22.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0909999999999998e-06 2024-09-21T10:24:22.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:24:22.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.629434+0000", 2024-09-21T10:24:22.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8409999999999999e-06 2024-09-21T10:24:22.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.631093+0000", 2024-09-21T10:24:22.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016591920000000001 2024-09-21T10:24:22.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-09-21T10:24:22.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.631098+0000", 2024-09-21T10:24:22.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7230000000000001e-06 2024-09-21T10:24:22.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.662137+0000", 2024-09-21T10:24:22.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.031038204999999999 2024-09-21T10:24:22.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.662145+0000", 2024-09-21T10:24:22.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7700000000000007e-06 2024-09-21T10:24:22.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628715+0000", 2024-09-21T10:24:22.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.96656951999999996 2024-09-21T10:24:22.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628725+0000", 2024-09-21T10:24:22.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6139999999999998e-06 2024-09-21T10:24:22.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.039822+0000", 2024-09-21T10:24:22.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4110971880000001 2024-09-21T10:24:22.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:24:22.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.039986+0000", 2024-09-21T10:24:22.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016436000000000001 2024-09-21T10:24:22.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:24:22.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.040543+0000", 2024-09-21T10:24:22.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00055650899999999995 2024-09-21T10:24:22.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.044937+0000", 2024-09-21T10:24:22.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043948149999999998 2024-09-21T10:24:22.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:24:22.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.048484+0000", 2024-09-21T10:24:22.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035470599999999999 2024-09-21T10:24:22.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.051634+0000", 2024-09-21T10:24:22.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031494790000000002 2024-09-21T10:24:22.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:24:22.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.051697+0000", 2024-09-21T10:24:22.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3421000000000001e-05 2024-09-21T10:24:22.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:24:22.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.053174+0000", 2024-09-21T10:24:22.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001476525 2024-09-21T10:24:22.525 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.525 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:24:22.525 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.525 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:183404 7.6 7:60ad48eb:::benchmark_data_smithi098_197834_object66887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-09-21T10:24:22.526 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:16:03.654829+0000", 2024-09-21T10:24:22.526 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 498.42526191100001, 2024-09-21T10:24:22.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4193103439999999, 2024-09-21T10:24:22.526 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:24:22.526 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:24:22.526 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:22.526 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:24:22.526 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:24:22.526 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:24:22.526 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 183404 2024-09-21T10:24:22.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.526 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:24:22.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:24:22.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.654829+0000", 2024-09-21T10:24:22.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:24:22.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.654829+0000", 2024-09-21T10:24:22.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:24:22.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.654830+0000", 2024-09-21T10:24:22.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.53e-06 2024-09-21T10:24:22.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:24:22.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.654835+0000", 2024-09-21T10:24:22.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9919999999999998e-06 2024-09-21T10:24:22.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:24:22.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.654847+0000", 2024-09-21T10:24:22.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1412e-05 2024-09-21T10:24:22.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:24:22.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.654853+0000", 2024-09-21T10:24:22.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9079999999999998e-06 2024-09-21T10:24:22.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.662168+0000", 2024-09-21T10:24:22.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.007314643 2024-09-21T10:24:22.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.662175+0000", 2024-09-21T10:24:22.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7109999999999998e-06 2024-09-21T10:24:22.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628748+0000", 2024-09-21T10:24:22.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.96657321299999999 2024-09-21T10:24:22.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628756+0000", 2024-09-21T10:24:22.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6520000000000006e-06 2024-09-21T10:24:22.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.045075+0000", 2024-09-21T10:24:22.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4163190370000001 2024-09-21T10:24:22.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:24:22.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.045262+0000", 2024-09-21T10:24:22.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018714900000000001 2024-09-21T10:24:22.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:24:22.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.045819+0000", 2024-09-21T10:24:22.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00055741799999999995 2024-09-21T10:24:22.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.053244+0000", 2024-09-21T10:24:22.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0074251730000000002 2024-09-21T10:24:22.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:24:22.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.053318+0000", 2024-09-21T10:24:22.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3857999999999997e-05 2024-09-21T10:24:22.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.072458+0000", 2024-09-21T10:24:22.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019139417999999998 2024-09-21T10:24:22.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:24:22.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.072519+0000", 2024-09-21T10:24:22.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1099000000000006e-05 2024-09-21T10:24:22.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:24:22.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.074139+0000", 2024-09-21T10:24:22.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016201309999999999 2024-09-21T10:24:22.535 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.535 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:24:22.535 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.535 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:183421 7.6 7:631fb813:::benchmark_data_smithi098_197834_object66904:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-09-21T10:24:22.535 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:16:03.673007+0000", 2024-09-21T10:24:22.535 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 498.40708330299998, 2024-09-21T10:24:22.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4134710660000001, 2024-09-21T10:24:22.535 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:24:22.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:24:22.536 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:22.536 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:24:22.536 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:24:22.536 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:24:22.536 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 183421 2024-09-21T10:24:22.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.536 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:24:22.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:24:22.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.673007+0000", 2024-09-21T10:24:22.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:24:22.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.673007+0000", 2024-09-21T10:24:22.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:24:22.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.673008+0000", 2024-09-21T10:24:22.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3900000000000005e-07 2024-09-21T10:24:22.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:24:22.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.673010+0000", 2024-09-21T10:24:22.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1519999999999999e-06 2024-09-21T10:24:22.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:24:22.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.673016+0000", 2024-09-21T10:24:22.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2899999999999999e-06 2024-09-21T10:24:22.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:24:22.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.673021+0000", 2024-09-21T10:24:22.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4870000000000004e-06 2024-09-21T10:24:22.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.673092+0000", 2024-09-21T10:24:22.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1606000000000005e-05 2024-09-21T10:24:22.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.673100+0000", 2024-09-21T10:24:22.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9840000000000001e-06 2024-09-21T10:24:22.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628779+0000", 2024-09-21T10:24:22.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.95567843699999999 2024-09-21T10:24:22.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628786+0000", 2024-09-21T10:24:22.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1119999999999998e-06 2024-09-21T10:24:22.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.048533+0000", 2024-09-21T10:24:22.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.419747353 2024-09-21T10:24:22.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:24:22.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.048699+0000", 2024-09-21T10:24:22.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000165923 2024-09-21T10:24:22.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:24:22.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.049206+0000", 2024-09-21T10:24:22.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00050717899999999996 2024-09-21T10:24:22.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.053304+0000", 2024-09-21T10:24:22.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040978960000000002 2024-09-21T10:24:22.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:24:22.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.053325+0000", 2024-09-21T10:24:22.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0358999999999999e-05 2024-09-21T10:24:22.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.084799+0000", 2024-09-21T10:24:22.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.031474087999999997 2024-09-21T10:24:22.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:24:22.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.084928+0000", 2024-09-21T10:24:22.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012917699999999999 2024-09-21T10:24:22.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:24:22.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.086478+0000", 2024-09-21T10:24:22.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001550484 2024-09-21T10:24:22.542 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.542 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:24:22.543 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.543 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:183441 7.6 7:60d4bafe:::benchmark_data_smithi098_197834_object66924:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-09-21T10:24:22.543 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:16:03.704441+0000", 2024-09-21T10:24:22.543 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 498.375649608, 2024-09-21T10:24:22.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3832941619999999, 2024-09-21T10:24:22.543 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:24:22.543 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:24:22.543 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:22.543 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:24:22.543 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:24:22.543 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:24:22.543 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 183441 2024-09-21T10:24:22.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.544 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:24:22.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:24:22.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.704441+0000", 2024-09-21T10:24:22.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:24:22.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.704441+0000", 2024-09-21T10:24:22.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:24:22.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.704442+0000", 2024-09-21T10:24:22.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5900000000000005e-07 2024-09-21T10:24:22.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:24:22.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.704446+0000", 2024-09-21T10:24:22.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2629999999999997e-06 2024-09-21T10:24:22.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:24:22.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.704458+0000", 2024-09-21T10:24:22.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1457999999999999e-05 2024-09-21T10:24:22.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:24:22.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.704464+0000", 2024-09-21T10:24:22.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8600000000000004e-06 2024-09-21T10:24:22.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.705054+0000", 2024-09-21T10:24:22.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00058917300000000002 2024-09-21T10:24:22.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.705063+0000", 2024-09-21T10:24:22.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2699999999999993e-06 2024-09-21T10:24:22.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.547 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628813+0000", 2024-09-21T10:24:22.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.92374978600000002 2024-09-21T10:24:22.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.547 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628820+0000", 2024-09-21T10:24:22.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9519999999999997e-06 2024-09-21T10:24:22.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.547 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.053354+0000", 2024-09-21T10:24:22.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.424534186 2024-09-21T10:24:22.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:24:22.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.053483+0000", 2024-09-21T10:24:22.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012914799999999999 2024-09-21T10:24:22.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:24:22.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.054026+0000", 2024-09-21T10:24:22.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00054301499999999999 2024-09-21T10:24:22.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.060726+0000", 2024-09-21T10:24:22.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0066995049999999997 2024-09-21T10:24:22.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:24:22.549 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.060740+0000", 2024-09-21T10:24:22.549 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4569000000000001e-05 2024-09-21T10:24:22.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.549 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.086532+0000", 2024-09-21T10:24:22.549 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.025792356999999998 2024-09-21T10:24:22.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:24:22.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.086588+0000", 2024-09-21T10:24:22.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5507000000000002e-05 2024-09-21T10:24:22.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.550 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:24:22.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.087735+0000", 2024-09-21T10:24:22.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011471540000000001 2024-09-21T10:24:22.550 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.550 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:24:22.550 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.550 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:183484 7.6 7:670a48b1:::benchmark_data_smithi098_197834_object66967:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-09-21T10:24:22.550 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:16:03.754909+0000", 2024-09-21T10:24:22.551 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 498.325181181, 2024-09-21T10:24:22.551 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.333857107, 2024-09-21T10:24:22.551 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:24:22.551 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:24:22.551 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:22.551 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:24:22.551 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:24:22.551 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:24:22.551 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 183484 2024-09-21T10:24:22.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.551 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:24:22.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:24:22.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.754909+0000", 2024-09-21T10:24:22.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.552 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:24:22.552 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.754909+0000", 2024-09-21T10:24:22.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.552 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:24:22.552 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.754911+0000", 2024-09-21T10:24:22.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5400000000000001e-06 2024-09-21T10:24:22.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:24:22.553 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.754917+0000", 2024-09-21T10:24:22.553 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7400000000000001e-06 2024-09-21T10:24:22.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:24:22.553 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.754933+0000", 2024-09-21T10:24:22.553 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6416999999999999e-05 2024-09-21T10:24:22.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:24:22.553 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.754941+0000", 2024-09-21T10:24:22.553 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9719999999999992e-06 2024-09-21T10:24:22.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.755487+0000", 2024-09-21T10:24:22.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000545474 2024-09-21T10:24:22.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.755496+0000", 2024-09-21T10:24:22.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1130000000000007e-06 2024-09-21T10:24:22.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628840+0000", 2024-09-21T10:24:22.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.87334427299999995 2024-09-21T10:24:22.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628844+0000", 2024-09-21T10:24:22.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0199999999999996e-06 2024-09-21T10:24:22.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.057341+0000", 2024-09-21T10:24:22.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.428497514 2024-09-21T10:24:22.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:24:22.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.057502+0000", 2024-09-21T10:24:22.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016084699999999999 2024-09-21T10:24:22.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.556 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:24:22.556 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.058030+0000", 2024-09-21T10:24:22.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00052808200000000003 2024-09-21T10:24:22.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.556 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.556 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.072370+0000", 2024-09-21T10:24:22.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014339849 2024-09-21T10:24:22.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.556 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:24:22.557 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.074251+0000", 2024-09-21T10:24:22.557 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018803139999999999 2024-09-21T10:24:22.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.557 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.557 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.087785+0000", 2024-09-21T10:24:22.557 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013534934 2024-09-21T10:24:22.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.557 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:24:22.557 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.087833+0000", 2024-09-21T10:24:22.557 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7547999999999999e-05 2024-09-21T10:24:22.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.558 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:24:22.558 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.088766+0000", 2024-09-21T10:24:22.558 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00093347000000000005 2024-09-21T10:24:22.558 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.558 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:24:22.558 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.558 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:183536 7.6 7:60ea16cd:::benchmark_data_smithi098_197834_object67019:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-09-21T10:24:22.558 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:16:03.824423+0000", 2024-09-21T10:24:22.558 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 498.25566748, 2024-09-21T10:24:22.558 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.265316457, 2024-09-21T10:24:22.558 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:24:22.558 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:24:22.559 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:22.559 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:24:22.559 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:24:22.559 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:24:22.559 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 183536 2024-09-21T10:24:22.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.559 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:24:22.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:24:22.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.824423+0000", 2024-09-21T10:24:22.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:24:22.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.824423+0000", 2024-09-21T10:24:22.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:24:22.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.824425+0000", 2024-09-21T10:24:22.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.72e-06 2024-09-21T10:24:22.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:24:22.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.824430+0000", 2024-09-21T10:24:22.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3959999999999997e-06 2024-09-21T10:24:22.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.561 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:24:22.561 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.824445+0000", 2024-09-21T10:24:22.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4331999999999999e-05 2024-09-21T10:24:22.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.561 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:24:22.561 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.824453+0000", 2024-09-21T10:24:22.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0269999999999995e-06 2024-09-21T10:24:22.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.561 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.561 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.824517+0000", 2024-09-21T10:24:22.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4252999999999996e-05 2024-09-21T10:24:22.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.562 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.562 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.824527+0000", 2024-09-21T10:24:22.562 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0684e-05 2024-09-21T10:24:22.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.562 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.562 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628856+0000", 2024-09-21T10:24:22.562 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.80432857999999996 2024-09-21T10:24:22.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.562 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.562 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628860+0000", 2024-09-21T10:24:22.563 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4589999999999998e-06 2024-09-21T10:24:22.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.563 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.563 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.060765+0000", 2024-09-21T10:24:22.563 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.431905904 2024-09-21T10:24:22.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.563 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:24:22.563 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.060929+0000", 2024-09-21T10:24:22.563 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016366699999999999 2024-09-21T10:24:22.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.563 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:24:22.564 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.061444+0000", 2024-09-21T10:24:22.564 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00051481699999999999 2024-09-21T10:24:22.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.564 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.564 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.072415+0000", 2024-09-21T10:24:22.564 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010971553 2024-09-21T10:24:22.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.564 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:24:22.564 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.074259+0000", 2024-09-21T10:24:22.564 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001843807 2024-09-21T10:24:22.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.565 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.565 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.088812+0000", 2024-09-21T10:24:22.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014552235 2024-09-21T10:24:22.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.565 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:24:22.565 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.088859+0000", 2024-09-21T10:24:22.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7005000000000001e-05 2024-09-21T10:24:22.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.565 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:24:22.565 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.089740+0000", 2024-09-21T10:24:22.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088101800000000001 2024-09-21T10:24:22.566 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.566 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:24:22.566 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.566 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.55000.0:183551 7.6 7:6005932d:::benchmark_data_smithi098_197834_object67034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-09-21T10:24:22.566 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:16:03.864376+0000", 2024-09-21T10:24:22.566 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 498.21571481199999, 2024-09-21T10:24:22.566 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.226529432, 2024-09-21T10:24:22.566 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:24:22.566 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:24:22.566 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:22.566 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:24:22.566 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.55000", 2024-09-21T10:24:22.567 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:24:22.567 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 183551 2024-09-21T10:24:22.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.567 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:24:22.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.567 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:24:22.567 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.864376+0000", 2024-09-21T10:24:22.567 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.567 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:24:22.567 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.864376+0000", 2024-09-21T10:24:22.567 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.568 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:24:22.568 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.864378+0000", 2024-09-21T10:24:22.568 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.469e-06 2024-09-21T10:24:22.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.568 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:24:22.568 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.864388+0000", 2024-09-21T10:24:22.568 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2990000000000002e-06 2024-09-21T10:24:22.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.568 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:24:22.568 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.864404+0000", 2024-09-21T10:24:22.568 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6912e-05 2024-09-21T10:24:22.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.569 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:24:22.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.864416+0000", 2024-09-21T10:24:22.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1532000000000001e-05 2024-09-21T10:24:22.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.569 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.864509+0000", 2024-09-21T10:24:22.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3158000000000005e-05 2024-09-21T10:24:22.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.569 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:03.864521+0000", 2024-09-21T10:24:22.570 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1640999999999999e-05 2024-09-21T10:24:22.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.570 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.570 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628925+0000", 2024-09-21T10:24:22.570 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.76440443999999996 2024-09-21T10:24:22.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.570 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.570 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:04.628933+0000", 2024-09-21T10:24:22.570 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9619999999999993e-06 2024-09-21T10:24:22.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.570 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.571 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.064395+0000", 2024-09-21T10:24:22.571 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4354620650000001 2024-09-21T10:24:22.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.571 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:24:22.571 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.064538+0000", 2024-09-21T10:24:22.571 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014270799999999999 2024-09-21T10:24:22.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.571 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:24:22.571 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.065083+0000", 2024-09-21T10:24:22.571 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00054525600000000004 2024-09-21T10:24:22.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.572 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.572 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.074235+0000", 2024-09-21T10:24:22.572 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0091516800000000006 2024-09-21T10:24:22.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.572 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.572 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:24:22.572 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.074265+0000", 2024-09-21T10:24:22.572 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0219999999999999e-05 2024-09-21T10:24:22.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.572 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.572 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.572 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.089842+0000", 2024-09-21T10:24:22.572 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015576865000000001 2024-09-21T10:24:22.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.573 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:24:22.573 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.089937+0000", 2024-09-21T10:24:22.573 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5376000000000003e-05 2024-09-21T10:24:22.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.573 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:24:22.573 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:16:06.090905+0000", 2024-09-21T10:24:22.573 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00096784900000000001 2024-09-21T10:24:22.573 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.573 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:24:22.573 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.574 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:66523 8.2 8:405f6d1b:::benchmark_data_smithi098_226587_object66522:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e273)", 2024-09-21T10:24:22.574 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:32.447947+0000", 2024-09-21T10:24:22.574 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 349.63214312100001, 2024-09-21T10:24:22.574 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1806429519999999, 2024-09-21T10:24:22.574 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:24:22.574 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:24:22.574 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-21T10:24:22.574 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:24:22.574 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:24:22.574 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:24:22.574 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66523 2024-09-21T10:24:22.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.574 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:24:22.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.575 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:24:22.575 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.447947+0000", 2024-09-21T10:24:22.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.575 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:24:22.575 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.447947+0000", 2024-09-21T10:24:22.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.575 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:24:22.575 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.447949+0000", 2024-09-21T10:24:22.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.471e-06 2024-09-21T10:24:22.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.576 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:24:22.576 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.447973+0000", 2024-09-21T10:24:22.576 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4209000000000001e-05 2024-09-21T10:24:22.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.576 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:24:22.576 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.447998+0000", 2024-09-21T10:24:22.576 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5290999999999999e-05 2024-09-21T10:24:22.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.576 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:24:22.576 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.448006+0000", 2024-09-21T10:24:22.577 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.729e-06 2024-09-21T10:24:22.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.577 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.577 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.448058+0000", 2024-09-21T10:24:22.577 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2318999999999998e-05 2024-09-21T10:24:22.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.577 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:24:22.577 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.448181+0000", 2024-09-21T10:24:22.577 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000122908 2024-09-21T10:24:22.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.577 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:24:22.578 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.448276+0000", 2024-09-21T10:24:22.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4541000000000002e-05 2024-09-21T10:24:22.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.578 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:24:22.578 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.449994+0000", 2024-09-21T10:24:22.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001718352 2024-09-21T10:24:22.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.578 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.578 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.450401+0000", 2024-09-21T10:24:22.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00040696500000000002 2024-09-21T10:24:22.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.579 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.579 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.480193+0000", 2024-09-21T10:24:22.579 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029792268 2024-09-21T10:24:22.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.579 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.579 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.480205+0000", 2024-09-21T10:24:22.579 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1619e-05 2024-09-21T10:24:22.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.579 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.579 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.470348+0000", 2024-09-21T10:24:22.579 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.99014276800000001 2024-09-21T10:24:22.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.580 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.580 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.470363+0000", 2024-09-21T10:24:22.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4776999999999999e-05 2024-09-21T10:24:22.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.580 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.580 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.628376+0000", 2024-09-21T10:24:22.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1580132670000001 2024-09-21T10:24:22.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.580 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:24:22.580 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.628590+0000", 2024-09-21T10:24:22.581 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00021446799999999999 2024-09-21T10:24:22.581 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.581 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:24:22.581 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.581 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:66530 8.8 8.76e2c0e8 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-21T10:24:22.581 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:32.460424+0000", 2024-09-21T10:24:22.581 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 349.61966673699999, 2024-09-21T10:24:22.581 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.001489912, 2024-09-21T10:24:22.581 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:24:22.581 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:24:22.581 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-21T10:24:22.581 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:24:22.582 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:24:22.582 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:24:22.582 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66530 2024-09-21T10:24:22.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.582 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:24:22.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:24:22.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.460424+0000", 2024-09-21T10:24:22.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:24:22.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.460424+0000", 2024-09-21T10:24:22.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:24:22.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.460424+0000", 2024-09-21T10:24:22.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9399999999999995e-07 2024-09-21T10:24:22.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:24:22.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.460446+0000", 2024-09-21T10:24:22.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1838999999999999e-05 2024-09-21T10:24:22.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:24:22.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.460454+0000", 2024-09-21T10:24:22.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6830000000000008e-06 2024-09-21T10:24:22.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.584 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:24:22.584 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.460460+0000", 2024-09-21T10:24:22.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2190000000000001e-06 2024-09-21T10:24:22.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.584 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.584 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.461837+0000", 2024-09-21T10:24:22.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013771300000000001 2024-09-21T10:24:22.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.584 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.585 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.461845+0000", 2024-09-21T10:24:22.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5399999999999998e-06 2024-09-21T10:24:22.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.585 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.585 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.461904+0000", 2024-09-21T10:24:22.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0000593 2024-09-21T10:24:22.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.585 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:24:22.585 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.461914+0000", 2024-09-21T10:24:22.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7070000000000004e-06 2024-09-21T10:24:22.585 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.585 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:24:22.586 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.586 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:66534 8.2 8:4b87b1b1:::benchmark_data_smithi098_226587_object66533:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-21T10:24:22.586 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:32.473078+0000", 2024-09-21T10:24:22.586 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 349.60701258099999, 2024-09-21T10:24:22.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1720339659999999, 2024-09-21T10:24:22.586 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:24:22.586 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:24:22.586 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:22.586 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:24:22.586 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:24:22.586 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:24:22.586 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66534 2024-09-21T10:24:22.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.587 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:24:22.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.587 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:24:22.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.473078+0000", 2024-09-21T10:24:22.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.587 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:24:22.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.473078+0000", 2024-09-21T10:24:22.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.587 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:24:22.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.473080+0000", 2024-09-21T10:24:22.588 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9300000000000002e-06 2024-09-21T10:24:22.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.588 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:24:22.588 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.473116+0000", 2024-09-21T10:24:22.588 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5840000000000002e-05 2024-09-21T10:24:22.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.588 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:24:22.588 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.473129+0000", 2024-09-21T10:24:22.588 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3285000000000001e-05 2024-09-21T10:24:22.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.588 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:24:22.589 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.473138+0000", 2024-09-21T10:24:22.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3589999999999997e-06 2024-09-21T10:24:22.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.589 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.589 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.480235+0000", 2024-09-21T10:24:22.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070962280000000004 2024-09-21T10:24:22.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.589 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.589 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.480242+0000", 2024-09-21T10:24:22.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1019999999999999e-06 2024-09-21T10:24:22.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.470441+0000", 2024-09-21T10:24:22.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.990198837 2024-09-21T10:24:22.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.470586+0000", 2024-09-21T10:24:22.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014546599999999999 2024-09-21T10:24:22.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.631513+0000", 2024-09-21T10:24:22.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1609273769999999 2024-09-21T10:24:22.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:24:22.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.631704+0000", 2024-09-21T10:24:22.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000190335 2024-09-21T10:24:22.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:24:22.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.631878+0000", 2024-09-21T10:24:22.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017390299999999999 2024-09-21T10:24:22.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.637866+0000", 2024-09-21T10:24:22.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0059887270000000001 2024-09-21T10:24:22.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.592 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:24:22.592 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.641355+0000", 2024-09-21T10:24:22.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034886909999999999 2024-09-21T10:24:22.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.592 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.592 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.641661+0000", 2024-09-21T10:24:22.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000305686 2024-09-21T10:24:22.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.592 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:24:22.592 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.641852+0000", 2024-09-21T10:24:22.593 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000191027 2024-09-21T10:24:22.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.593 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:24:22.593 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.645112+0000", 2024-09-21T10:24:22.593 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032601729999999999 2024-09-21T10:24:22.593 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.593 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:24:22.593 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.593 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:66541 8.2 8:4a411c2e:::benchmark_data_smithi098_226587_object66540:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-21T10:24:22.593 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:32.508561+0000", 2024-09-21T10:24:22.594 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 349.57152985200003, 2024-09-21T10:24:22.594 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1401821930000002, 2024-09-21T10:24:22.594 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:24:22.594 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:24:22.594 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:22.594 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:24:22.594 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:24:22.594 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:24:22.594 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66541 2024-09-21T10:24:22.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.594 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:24:22.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.594 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:24:22.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.508561+0000", 2024-09-21T10:24:22.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:24:22.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.508561+0000", 2024-09-21T10:24:22.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:24:22.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.508563+0000", 2024-09-21T10:24:22.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8029999999999999e-06 2024-09-21T10:24:22.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.596 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:24:22.596 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.508599+0000", 2024-09-21T10:24:22.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6341999999999999e-05 2024-09-21T10:24:22.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.596 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:24:22.596 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.508610+0000", 2024-09-21T10:24:22.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1636e-05 2024-09-21T10:24:22.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.596 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:24:22.596 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.508618+0000", 2024-09-21T10:24:22.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3429999999999998e-06 2024-09-21T10:24:22.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.597 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.597 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.509182+0000", 2024-09-21T10:24:22.597 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00056403200000000001 2024-09-21T10:24:22.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.597 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.597 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.509192+0000", 2024-09-21T10:24:22.597 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0623e-05 2024-09-21T10:24:22.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.597 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.597 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.470614+0000", 2024-09-21T10:24:22.597 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.96142103000000001 2024-09-21T10:24:22.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.598 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.598 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.470621+0000", 2024-09-21T10:24:22.598 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9540000000000004e-06 2024-09-21T10:24:22.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.598 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.598 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.634388+0000", 2024-09-21T10:24:22.598 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1637665779999999 2024-09-21T10:24:22.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.598 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:24:22.598 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.634564+0000", 2024-09-21T10:24:22.599 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000175501 2024-09-21T10:24:22.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:24:22.599 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.634748+0000", 2024-09-21T10:24:22.599 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018397899999999999 2024-09-21T10:24:22.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.599 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.637996+0000", 2024-09-21T10:24:22.599 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032480439999999998 2024-09-21T10:24:22.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:24:22.600 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.645271+0000", 2024-09-21T10:24:22.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0072753760000000001 2024-09-21T10:24:22.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.600 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.600 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.645512+0000", 2024-09-21T10:24:22.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00024140399999999999 2024-09-21T10:24:22.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.600 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:24:22.600 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.645581+0000", 2024-09-21T10:24:22.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8282999999999997e-05 2024-09-21T10:24:22.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.601 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:24:22.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:34.648743+0000", 2024-09-21T10:24:22.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003162265 2024-09-21T10:24:22.601 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.601 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:24:22.601 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.601 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:66548 8.8 8.6ed96238 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-21T10:24:22.601 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:32.518774+0000", 2024-09-21T10:24:22.601 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 349.56131643800001, 2024-09-21T10:24:22.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.94317137299999998, 2024-09-21T10:24:22.601 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:24:22.601 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:24:22.601 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-21T10:24:22.602 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:24:22.602 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:24:22.602 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:24:22.602 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66548 2024-09-21T10:24:22.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.602 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:24:22.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.602 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:24:22.602 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.518774+0000", 2024-09-21T10:24:22.602 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.602 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:24:22.602 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.518774+0000", 2024-09-21T10:24:22.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.603 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:24:22.603 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.518775+0000", 2024-09-21T10:24:22.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8599999999999996e-07 2024-09-21T10:24:22.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.603 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:24:22.603 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.518794+0000", 2024-09-21T10:24:22.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9134999999999999e-05 2024-09-21T10:24:22.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.604 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:24:22.604 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.518806+0000", 2024-09-21T10:24:22.604 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1972e-05 2024-09-21T10:24:22.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.604 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:24:22.604 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.518811+0000", 2024-09-21T10:24:22.604 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2249999999999999e-06 2024-09-21T10:24:22.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.604 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.604 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.518874+0000", 2024-09-21T10:24:22.604 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2360999999999995e-05 2024-09-21T10:24:22.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.518879+0000", 2024-09-21T10:24:22.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.011e-06 2024-09-21T10:24:22.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.461938+0000", 2024-09-21T10:24:22.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.94305926600000001 2024-09-21T10:24:22.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:24:22.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.461945+0000", 2024-09-21T10:24:22.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4170000000000003e-06 2024-09-21T10:24:22.605 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.606 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:24:22.606 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.606 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:66553 8.8 8.329526d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-21T10:24:22.606 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:18:32.523208+0000", 2024-09-21T10:24:22.606 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 349.55688248600001, 2024-09-21T10:24:22.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.93876435599999997, 2024-09-21T10:24:22.606 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:24:22.606 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:24:22.606 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-21T10:24:22.606 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:24:22.606 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:24:22.607 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:24:22.607 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66553 2024-09-21T10:24:22.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.607 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:24:22.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.607 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:24:22.607 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.523208+0000", 2024-09-21T10:24:22.607 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.607 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:24:22.607 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.523208+0000", 2024-09-21T10:24:22.607 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.608 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:24:22.608 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.523209+0000", 2024-09-21T10:24:22.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3790000000000001e-06 2024-09-21T10:24:22.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.608 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:24:22.608 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.523227+0000", 2024-09-21T10:24:22.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7717999999999999e-05 2024-09-21T10:24:22.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.608 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:24:22.608 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.523235+0000", 2024-09-21T10:24:22.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6930000000000007e-06 2024-09-21T10:24:22.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:24:22.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.523239+0000", 2024-09-21T10:24:22.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4839999999999998e-06 2024-09-21T10:24:22.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.523295+0000", 2024-09-21T10:24:22.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5575999999999998e-05 2024-09-21T10:24:22.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:24:22.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:32.523304+0000", 2024-09-21T10:24:22.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7390000000000005e-06 2024-09-21T10:24:22.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.610 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.610 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.461965+0000", 2024-09-21T10:24:22.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.93866182200000003 2024-09-21T10:24:22.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.610 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:24:22.610 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:18:33.461972+0000", 2024-09-21T10:24:22.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9450000000000004e-06 2024-09-21T10:24:22.610 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.610 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:24:22.610 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.611 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:95750 8.c 8:3bd51e92:::benchmark_data_smithi098_226587_object95749:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-21T10:24:22.611 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:19:14.919667+0000", 2024-09-21T10:24:22.611 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 307.160423444, 2024-09-21T10:24:22.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.31776388300000002, 2024-09-21T10:24:22.611 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:24:22.611 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:24:22.611 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:22.611 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:24:22.611 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:24:22.611 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:24:22.611 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 95750 2024-09-21T10:24:22.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.612 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:24:22.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:24:22.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.919667+0000", 2024-09-21T10:24:22.612 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:24:22.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.919667+0000", 2024-09-21T10:24:22.612 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:24:22.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.919669+0000", 2024-09-21T10:24:22.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.415e-06 2024-09-21T10:24:22.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.613 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:24:22.613 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.919688+0000", 2024-09-21T10:24:22.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9741e-05 2024-09-21T10:24:22.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.613 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:24:22.613 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.919698+0000", 2024-09-21T10:24:22.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9140000000000003e-06 2024-09-21T10:24:22.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.613 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:24:22.614 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.919740+0000", 2024-09-21T10:24:22.614 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2076000000000002e-05 2024-09-21T10:24:22.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.614 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.614 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.919900+0000", 2024-09-21T10:24:22.614 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015946299999999999 2024-09-21T10:24:22.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.614 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:24:22.614 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.920077+0000", 2024-09-21T10:24:22.614 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017700600000000001 2024-09-21T10:24:22.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.614 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5,7", 2024-09-21T10:24:22.615 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.920288+0000", 2024-09-21T10:24:22.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00021087499999999999 2024-09-21T10:24:22.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.615 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.615 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.216770+0000", 2024-09-21T10:24:22.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.29648236 2024-09-21T10:24:22.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.615 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.615 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.217100+0000", 2024-09-21T10:24:22.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00033039300000000001 2024-09-21T10:24:22.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.616 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:24:22.616 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.233469+0000", 2024-09-21T10:24:22.616 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016368671000000001 2024-09-21T10:24:22.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.616 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:24:22.616 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.233626+0000", 2024-09-21T10:24:22.616 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000157324 2024-09-21T10:24:22.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.616 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:24:22.616 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.237431+0000", 2024-09-21T10:24:22.616 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038046450000000002 2024-09-21T10:24:22.616 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.616 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:24:22.617 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.617 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:95757 8.6 8:6a63278c:::benchmark_data_smithi098_226587_object95756:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-21T10:24:22.617 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:19:14.931480+0000", 2024-09-21T10:24:22.617 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 307.148610808, 2024-09-21T10:24:22.617 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.30778484299999997, 2024-09-21T10:24:22.617 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:24:22.617 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:24:22.617 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:22.617 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:24:22.617 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:24:22.617 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:24:22.617 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 95757 2024-09-21T10:24:22.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.618 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:24:22.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.618 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:24:22.618 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.931480+0000", 2024-09-21T10:24:22.618 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.618 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:24:22.618 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.931480+0000", 2024-09-21T10:24:22.618 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:24:22.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:24:22.619 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.931481+0000", 2024-09-21T10:24:22.619 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4160000000000001e-06 2024-09-21T10:24:22.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:24:22.619 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.931508+0000", 2024-09-21T10:24:22.619 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6457000000000001e-05 2024-09-21T10:24:22.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:24:22.619 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.931516+0000", 2024-09-21T10:24:22.619 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5269999999999998e-06 2024-09-21T10:24:22.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.620 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:24:22.620 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:14.931522+0000", 2024-09-21T10:24:22.620 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4480000000000002e-06 2024-09-21T10:24:22.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.620 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:24:22.620 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.216781+0000", 2024-09-21T10:24:22.620 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.285259186 2024-09-21T10:24:22.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.620 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:24:22.620 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.216950+0000", 2024-09-21T10:24:22.620 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016970900000000001 2024-09-21T10:24:22.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.621 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:24:22.621 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.217100+0000", 2024-09-21T10:24:22.621 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000149171 2024-09-21T10:24:22.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.621 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.621 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.226029+0000", 2024-09-21T10:24:22.621 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0089289339999999995 2024-09-21T10:24:22.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.621 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:22.621 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.226549+0000", 2024-09-21T10:24:22.621 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00052087499999999996 2024-09-21T10:24:22.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.622 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:24:22.622 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.239188+0000", 2024-09-21T10:24:22.622 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012638361000000001 2024-09-21T10:24:22.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.622 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:24:22.622 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.239235+0000", 2024-09-21T10:24:22.622 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7095000000000002e-05 2024-09-21T10:24:22.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:24:22.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:24:22.622 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:24:22.622 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:19:15.239265+0000", 2024-09-21T10:24:22.622 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9663999999999999e-05 2024-09-21T10:24:22.623 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.623 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:24:22.623 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.623 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:24:22.623 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:24:22.623 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:24:22.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:22.654 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:22.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:22.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:22.756 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:22.790 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:22.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:22.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:22 smithi098 ceph-mon[97580]: pgmap v1071: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 1.24k op/s 2024-09-21T10:24:22.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:22.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:22 smithi098 ceph-mon[100914]: pgmap v1071: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 1.24k op/s 2024-09-21T10:24:22.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:22 smithi196 ceph-mon[84784]: pgmap v1071: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 1.24k op/s 2024-09-21T10:24:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:22.892 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:22.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:22.926 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:22.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:23.027 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:23.063 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:23.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:23.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:23.165 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:23.202 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:23.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:23.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:23.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:23.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:23.339 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:23.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:23.440 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:23.476 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:23.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:23.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:23.578 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:23.611 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:23.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:23.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:23.616 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:23.624 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:24:23.629 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:23.713 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:23.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:23.756 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:23.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:23.857 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:23.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:23.900 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:23.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:24.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:24.039 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:24.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:24.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:24.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:24.197 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:24.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:24.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:24.299 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:24.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:24.333 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:24.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:24.434 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:24.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:24.468 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:24.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:24.569 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:24.604 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:24.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:24.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:24.706 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:24.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:24.739 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:24.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:24.840 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:24 smithi098 ceph-mon[97580]: pgmap v1072: 99 pgs: 99 active+clean; 6.8 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 1.21k op/s 2024-09-21T10:24:24.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:24 smithi098 ceph-mon[100914]: pgmap v1072: 99 pgs: 99 active+clean; 6.8 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 1.21k op/s 2024-09-21T10:24:24.877 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:24.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:24.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:24 smithi196 ceph-mon[84784]: pgmap v1072: 99 pgs: 99 active+clean; 6.8 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 1.21k op/s 2024-09-21T10:24:24.979 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:25.017 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:25.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:25.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:25.119 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:25.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:25.155 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:25.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:25.207 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:24:25.208 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:24:25.243 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:24:25.244 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:24:25.256 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:25.277 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:24:25.277 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:24:25.288 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:25.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:25.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:25.303 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:24:25.303 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:24:25.335 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:24:25.374 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:24:25.374 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:24:25.375 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:24:25.390 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:25.395 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:24:25.395 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:24:25.423 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:24:25.423 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:24:25.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:25.441 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:25.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:25.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:25.578 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:25.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:25.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:25.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:25.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:25.717 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:25.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:25.818 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:25.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:25 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3408774474' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:25.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:25 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3408774474' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:25.854 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:25.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:25.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:25.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:25 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3408774474' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:25.956 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:25.986 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:25.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:26.088 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:26.120 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:26.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:26.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:26.221 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:26.255 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:26.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:26.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:26.357 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:26.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:26.391 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:26.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:26.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:26.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:26.525 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:26.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:26.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:26.655 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:26.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:26.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:26.757 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:26.790 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:26.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:26.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:26.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:26 smithi098 ceph-mon[97580]: pgmap v1073: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 1.24k op/s 2024-09-21T10:24:26.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:26 smithi098 ceph-mon[100914]: pgmap v1073: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 1.24k op/s 2024-09-21T10:24:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:26 smithi196 ceph-mon[84784]: pgmap v1073: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 1.24k op/s 2024-09-21T10:24:26.892 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:26.930 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:26.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:26.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:27.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:27.062 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:27.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:27.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:27.164 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:27.193 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:27.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:27.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:27.294 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:27.333 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:27.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:27.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:27.435 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:27.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:27.466 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:27.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:27.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:27.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:27.600 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:27.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:27.701 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:27.737 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:27.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:27.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:27.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:27.875 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:27.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:27.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:27.976 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:28.014 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:28.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:28.116 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:28.149 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:28.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:28.250 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:28.287 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:28.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:28.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:24:28.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:28.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.439 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:28.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:28.540 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:28.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.569 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:28.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:28.669 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:28.718 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:28.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:28.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:28.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:28 smithi098 ceph-mon[100914]: pgmap v1074: 99 pgs: 99 active+clean; 6.5 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 1.17k op/s 2024-09-21T10:24:28.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:28 smithi098 ceph-mon[97580]: pgmap v1074: 99 pgs: 99 active+clean; 6.5 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 1.17k op/s 2024-09-21T10:24:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:28.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:28 smithi196 ceph-mon[84784]: pgmap v1074: 99 pgs: 99 active+clean; 6.5 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 1.17k op/s 2024-09-21T10:24:28.955 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:28.965 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:28.984 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:28.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:29.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:29.114 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:29.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:29.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:29.216 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:29.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:29.251 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:29.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:29.352 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:29.389 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:29.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:29.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:29.490 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:29.525 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:29.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:29.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:29.587 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:24:29.627 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:29.659 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:29.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:29.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:29.761 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:29.799 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:29.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:29.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:29.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:29 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2523583694' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:29.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:29 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:29.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:29 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2523583694' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:29.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:29 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:29.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:29 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2523583694' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:29 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:29.901 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:29.940 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:29.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:29.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:30.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:30.081 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:30.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:30.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:30.182 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:30.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:30.214 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:30.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:30.315 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:30.350 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:30.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:30.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:30.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:30.482 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:30.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:30.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:30.564 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:30.575 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:24:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:24:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:24:30.583 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:30.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:30.629 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:30.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:30.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:30.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:30.765 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:30.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:30.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:24:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:24:30] "GET /metrics HTTP/1.1" 200 36524 "" "Prometheus/2.51.0" 2024-09-21T10:24:30.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:30 smithi098 ceph-mon[97580]: pgmap v1075: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 1.34k op/s 2024-09-21T10:24:30.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:30 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:24:30.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:30 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:24:30.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:30 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:24:30.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:30 smithi098 ceph-mon[97580]: osdmap e319: 8 total, 8 up, 6 in 2024-09-21T10:24:30.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:30 smithi098 ceph-mon[100914]: pgmap v1075: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 1.34k op/s 2024-09-21T10:24:30.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:30 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:24:30.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:30 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:24:30.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:30 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:24:30.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:30 smithi098 ceph-mon[100914]: osdmap e319: 8 total, 8 up, 6 in 2024-09-21T10:24:30.866 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:30 smithi196 ceph-mon[84784]: pgmap v1075: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 1.34k op/s 2024-09-21T10:24:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:30 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:24:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:30 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:24:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:30 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:24:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:30 smithi196 ceph-mon[84784]: osdmap e319: 8 total, 8 up, 6 in 2024-09-21T10:24:30.905 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:30.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:30.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:31.007 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:31.041 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:31.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:31.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:31.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:31.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:31.184 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:31.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:31.285 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:31.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:31.324 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:31.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:31.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:31.462 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:31.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:31.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:31.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:31.611 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:31.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:31.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:31.713 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:31.744 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:31.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:31.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:31.846 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:31.876 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:31.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:31.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:31.978 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:32.018 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:32.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:32.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:32.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:32.151 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:32.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:32.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:32.253 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:32.286 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:32.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:32.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:32.388 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:32.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:32.424 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:32.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:32.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:32.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:24:32.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:32.560 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:32.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:32.661 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:32.722 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:32.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:32.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:32.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:32.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:32 smithi098 ceph-mon[97580]: pgmap v1077: 99 pgs: 99 active+clean; 6.2 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 1.22k op/s 2024-09-21T10:24:32.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:32 smithi098 ceph-mon[100914]: pgmap v1077: 99 pgs: 99 active+clean; 6.2 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 1.22k op/s 2024-09-21T10:24:32.881 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:32.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:32.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:32.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:32 smithi196 ceph-mon[84784]: pgmap v1077: 99 pgs: 99 active+clean; 6.2 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 1.22k op/s 2024-09-21T10:24:32.983 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:33.021 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:33.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:33.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:33.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:33.161 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:33.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:33.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:33.253 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:24:33.253 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:24:33.263 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:33.300 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:33.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:33.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:33.402 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:33.439 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:33.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:33.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:33.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:33.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:33.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:33.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:33.714 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:33.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:33.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:33.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:33.848 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:33.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:33.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:33.942 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7] out_osds: [4, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:24:33.942 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:24:33.942 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-21T10:24:33.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd in 4 2024-09-21T10:24:33.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:33.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:33.982 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:33.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:34.083 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:34.120 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:34.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:34.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:34.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.270 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:34.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:34.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:34.408 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:34.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:34.510 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:34.547 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:34.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:34.649 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:34.686 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:34.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:34.788 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:34.820 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:34.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:34.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:34 smithi098 ceph-mon[97580]: pgmap v1078: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 1.23k op/s 2024-09-21T10:24:34.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:34 smithi098 ceph-mon[100914]: pgmap v1078: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 1.23k op/s 2024-09-21T10:24:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:34 smithi196 ceph-mon[84784]: pgmap v1078: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 1.23k op/s 2024-09-21T10:24:34.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:34.958 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:34.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:35.060 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:35.096 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:35.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:35.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:35.198 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:35.232 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:35.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:35.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:35.334 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:35.368 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:35.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:35.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:35.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:24:35.470 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:35.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:35.508 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:35.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:35.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:35.646 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:35.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:35.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:35.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:35.782 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:35.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:35.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:35.884 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:35.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:35.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:36.021 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:36.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:36.056 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:36.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:36.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:36.190 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:36.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:36.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:36.291 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:36.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:36.324 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:36.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:36.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:36.455 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:36.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:36.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:36.557 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:36.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:36.597 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:36.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:36.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:36 smithi098 ceph-mon[97580]: pgmap v1079: 99 pgs: 99 active+clean; 5.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1.24k op/s 2024-09-21T10:24:36.653 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:36 smithi098 ceph-mon[100914]: pgmap v1079: 99 pgs: 99 active+clean; 5.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1.24k op/s 2024-09-21T10:24:36.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:36.739 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:36.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:36.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:36.840 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:36.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:36.879 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:36.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:36.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:36 smithi196 ceph-mon[84784]: pgmap v1079: 99 pgs: 99 active+clean; 5.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1.24k op/s 2024-09-21T10:24:36.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:37.015 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:37.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:37.117 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:37.149 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:37.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:37.184 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:24:37.184 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:24:37.184 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:24:37.184 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:24:37.184 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:24:37.184 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:24:37.185 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:24:37.185 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:24:37.185 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:24:37.185 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:24:37.185 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:24:37.185 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:24:37.185 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:24:37.185 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:24:37.209 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:24:37.210 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:24:37.251 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:37.290 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:37.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:37.392 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:37.429 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:37.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:37.531 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:37.563 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:37.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:37.665 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:37.700 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:37.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:37.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:37.834 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:37.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:37.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:37 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3229013401' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:37.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:24:37.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:37.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:37 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3229013401' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:37.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:24:37.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:37.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:37 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3229013401' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:37.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:24:37.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:37.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:37.974 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:37.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:38.076 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:38.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:38.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:38.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:38.130 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:38.145 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:38.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:38.242 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:38.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:38.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:38.343 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:38.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:38.375 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:38.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:38.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:38.518 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:38.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:38.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:38.620 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:38.654 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:38.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:38.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:38.756 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:38.790 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:38.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:38.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:38.892 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:38.923 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:38.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:38.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:39.025 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:39.058 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:39.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:39.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:39.160 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:39.191 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:39.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:39.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:39.242 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:39.292 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:39.330 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:39.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:39.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:39.431 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:39.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:39.471 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:39.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:39.571 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:39.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:39 smithi098 ceph-mon[97580]: pgmap v1080: 99 pgs: 99 active+clean; 5.7 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1.24k op/s 2024-09-21T10:24:39.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:39 smithi098 ceph-mon[100914]: pgmap v1080: 99 pgs: 99 active+clean; 5.7 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1.24k op/s 2024-09-21T10:24:39.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:39.615 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:39.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:39.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:39 smithi196 ceph-mon[84784]: pgmap v1080: 99 pgs: 99 active+clean; 5.7 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1.24k op/s 2024-09-21T10:24:39.716 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:39.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:39.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:39.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:39.851 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:39.885 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:39.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:39.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:39.987 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:40.022 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:40.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:40.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:40.112 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:24:40.125 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:40.160 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:40.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:40.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:40.262 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:40.293 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:40.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:40.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:40.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:40.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:40.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:40.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:40.577 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:40.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:40.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:40.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:24:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:24:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:24:40.678 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:40.700 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:24:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:24:40] "GET /metrics HTTP/1.1" 200 36526 "" "Prometheus/2.51.0" 2024-09-21T10:24:40.716 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:40.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:40.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:40.818 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:40.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:40.852 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:40.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:40.953 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:40.984 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:40.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:40.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:41.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:41.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:41.118 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:41.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:41.219 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:41.247 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:41.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:41.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:41.349 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:41.387 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:41.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:41.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:41.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:41.523 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:41.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:41.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:41.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:41 smithi098 ceph-mon[97580]: pgmap v1081: 99 pgs: 99 active+clean; 5.5 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 1.22k op/s 2024-09-21T10:24:41.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:41 smithi098 ceph-mon[100914]: pgmap v1081: 99 pgs: 99 active+clean; 5.5 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 1.22k op/s 2024-09-21T10:24:41.625 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:41.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:41 smithi196 ceph-mon[84784]: pgmap v1081: 99 pgs: 99 active+clean; 5.5 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 1.22k op/s 2024-09-21T10:24:41.665 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:41.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:41.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:41.767 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:41.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:41.801 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:41.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:41.903 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:41.931 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:41.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:41.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:42.033 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:42.069 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:42.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:42.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:42.170 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:42.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:42.208 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:42.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:42.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:42.342 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:42.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:42.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:42.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:42.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:42.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:42.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:42.624 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:42.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:42.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:42.726 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:42.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:42.764 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:42.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:42.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:42 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a[97576]: 2024-09-21T10:24:42.480+0000 7f65561f9640 -1 mon.a@0(leader).osd e319 definitely_dead 0 2024-09-21T10:24:42.793 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:42.865 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:42.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:42.903 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:42.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:43.004 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:43.037 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:43.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:43.139 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:43.174 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:43.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:43.270 INFO:teuthology.orchestra.run.smithi098.stderr:marked in osd.4. 2024-09-21T10:24:43.276 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:43.311 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:43.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:43.413 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:43.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.435 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:43.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:43.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:43.558 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:43.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:43.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:43 smithi098 ceph-mon[97580]: pgmap v1082: 99 pgs: 99 active+clean; 5.5 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T10:24:43.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:43 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3273633919' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:24:43.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:43 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:24:43.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:43 smithi098 ceph-mon[100914]: pgmap v1082: 99 pgs: 99 active+clean; 5.5 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T10:24:43.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:43 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3273633919' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:24:43.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:43 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:24:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:43 smithi196 ceph-mon[84784]: pgmap v1082: 99 pgs: 99 active+clean; 5.5 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T10:24:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:43 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3273633919' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:24:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:43 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:24:43.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:43.682 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:43.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:43.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:43.804 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:43.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:43.906 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:43.928 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:43.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:44.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:44.052 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:44.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:44.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:44.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:44.175 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:44.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:44.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:44.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:44.306 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:44.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:44.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:44.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:44.430 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:44.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:44.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:44.532 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:44.550 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:44 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T10:24:44.550 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:44 smithi098 ceph-mon[97580]: osdmap e320: 8 total, 8 up, 7 in 2024-09-21T10:24:44.550 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:44 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T10:24:44.551 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:44 smithi098 ceph-mon[100914]: osdmap e320: 8 total, 8 up, 7 in 2024-09-21T10:24:44.553 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:44.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:44.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:44.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:44 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T10:24:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:44 smithi196 ceph-mon[84784]: osdmap e320: 8 total, 8 up, 7 in 2024-09-21T10:24:44.655 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:44.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:44.682 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:44.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:44.783 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:44.807 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:44.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:44.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:44.908 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:44.932 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:44.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:44.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:45.034 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:45.059 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:45.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:45.160 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:45.182 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-21T10:24:45.183 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:45.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:45.284 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:45.316 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:45.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:45.377 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:24:45.377 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:24:45.377 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-21T10:24:45.377 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-21T10:24:45.377 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:24:45.418 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:45.447 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:45.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:45.548 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:45.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.595 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:45.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:45.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:45 smithi098 ceph-mon[97580]: pgmap v1084: 99 pgs: 23 peering, 76 active+clean; 5.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.22k op/s 2024-09-21T10:24:45.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:45 smithi098 ceph-mon[97580]: osdmap e321: 8 total, 8 up, 7 in 2024-09-21T10:24:45.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:45 smithi098 ceph-mon[100914]: pgmap v1084: 99 pgs: 23 peering, 76 active+clean; 5.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.22k op/s 2024-09-21T10:24:45.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:45 smithi098 ceph-mon[100914]: osdmap e321: 8 total, 8 up, 7 in 2024-09-21T10:24:45.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:45 smithi196 ceph-mon[84784]: pgmap v1084: 99 pgs: 23 peering, 76 active+clean; 5.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.22k op/s 2024-09-21T10:24:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:45 smithi196 ceph-mon[84784]: osdmap e321: 8 total, 8 up, 7 in 2024-09-21T10:24:45.696 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:45.731 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:45.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:45.832 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:45.860 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:45.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:45.962 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:45.991 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:45.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:46.093 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:46.107 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:24:46.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:46.144 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:46.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:46.245 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:46.288 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:46.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:46.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:46.390 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:46.424 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:46.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:46.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:46.526 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:46.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:46.557 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:46.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:46.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:46 smithi098 ceph-mon[97580]: osdmap e322: 8 total, 8 up, 7 in 2024-09-21T10:24:46.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:46 smithi098 ceph-mon[97580]: 9.c starting backfill to osd.4 from (0'0,0'0] MAX to 319'11392 2024-09-21T10:24:46.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:46 smithi098 ceph-mon[97580]: 9.4 starting backfill to osd.4 from (0'0,0'0] MAX to 319'11286 2024-09-21T10:24:46.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:46 smithi098 ceph-mon[97580]: 9.1 starting backfill to osd.4 from (0'0,0'0] MAX to 319'11474 2024-09-21T10:24:46.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:46 smithi098 ceph-mon[97580]: 9.e starting backfill to osd.4 from (0'0,0'0] MAX to 319'11274 2024-09-21T10:24:46.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:46 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:24:46.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:46 smithi098 ceph-mon[97580]: osdmap e323: 8 total, 8 up, 7 in 2024-09-21T10:24:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:46 smithi098 ceph-mon[100914]: osdmap e322: 8 total, 8 up, 7 in 2024-09-21T10:24:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:46 smithi098 ceph-mon[100914]: 9.c starting backfill to osd.4 from (0'0,0'0] MAX to 319'11392 2024-09-21T10:24:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:46 smithi098 ceph-mon[100914]: 9.4 starting backfill to osd.4 from (0'0,0'0] MAX to 319'11286 2024-09-21T10:24:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:46 smithi098 ceph-mon[100914]: 9.1 starting backfill to osd.4 from (0'0,0'0] MAX to 319'11474 2024-09-21T10:24:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:46 smithi098 ceph-mon[100914]: 9.e starting backfill to osd.4 from (0'0,0'0] MAX to 319'11274 2024-09-21T10:24:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:46 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:24:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:46 smithi098 ceph-mon[100914]: osdmap e323: 8 total, 8 up, 7 in 2024-09-21T10:24:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:46 smithi196 ceph-mon[84784]: osdmap e322: 8 total, 8 up, 7 in 2024-09-21T10:24:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:46 smithi196 ceph-mon[84784]: 9.c starting backfill to osd.4 from (0'0,0'0] MAX to 319'11392 2024-09-21T10:24:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:46 smithi196 ceph-mon[84784]: 9.4 starting backfill to osd.4 from (0'0,0'0] MAX to 319'11286 2024-09-21T10:24:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:46 smithi196 ceph-mon[84784]: 9.1 starting backfill to osd.4 from (0'0,0'0] MAX to 319'11474 2024-09-21T10:24:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:46 smithi196 ceph-mon[84784]: 9.e starting backfill to osd.4 from (0'0,0'0] MAX to 319'11274 2024-09-21T10:24:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:46 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:24:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:46 smithi196 ceph-mon[84784]: osdmap e323: 8 total, 8 up, 7 in 2024-09-21T10:24:46.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:46.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:46.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:46.809 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:46.862 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:46.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:46.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:46.964 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:47.005 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:47.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:47.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:47.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:47.134 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:47.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:47.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:47.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:47.262 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:47.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:47.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:47.364 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:47.427 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:47 smithi098 ceph-mon[97580]: pgmap v1087: 99 pgs: 34 peering, 65 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 987 op/s 2024-09-21T10:24:47.427 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:47 smithi098 ceph-mon[97580]: osdmap e324: 8 total, 8 up, 7 in 2024-09-21T10:24:47.428 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:47 smithi098 ceph-mon[100914]: pgmap v1087: 99 pgs: 34 peering, 65 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 987 op/s 2024-09-21T10:24:47.428 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:47 smithi098 ceph-mon[100914]: osdmap e324: 8 total, 8 up, 7 in 2024-09-21T10:24:47.432 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:47.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:47.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:47.534 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:47.572 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:47.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:47.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:47.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:47 smithi196 ceph-mon[84784]: pgmap v1087: 99 pgs: 34 peering, 65 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 987 op/s 2024-09-21T10:24:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:47 smithi196 ceph-mon[84784]: osdmap e324: 8 total, 8 up, 7 in 2024-09-21T10:24:47.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:47.702 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:47.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:47.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:47.803 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:47.834 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:47.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:47.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:47.936 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:47.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:47.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:48.069 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:48.104 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:48.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:48.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:48.188 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:48.206 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:48.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:48.271 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:48.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:48.342 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:24:48.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:48.405 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:48.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:48.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:48.507 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:48.566 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:48.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:48.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:48.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:48 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2713210610' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:24:48.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:48 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:24:48.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:48 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2713210610' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:24:48.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:48 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:24:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:48 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2713210610' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:24:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:48 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:24:48.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:48.698 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:48.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:48.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:48.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:48.843 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:48.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:48.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:48.945 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:48.996 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:48.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:48.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:49.097 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:49.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.132 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:49.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:49.233 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:49.274 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:49.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:49.375 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:49.416 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:49 smithi098 ceph-mon[97580]: pgmap v1090: 99 pgs: 1 active+remapped+backfill_wait, 34 peering, 64 active+clean; 5.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 771 op/s; 5083/250278 objects misplaced (2.031%); 16 KiB/s, 2 objects/s recovering 2024-09-21T10:24:49.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:49 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:49.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:49 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:24:49.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:49 smithi098 ceph-mon[97580]: osdmap e325: 8 total, 8 up, 7 in 2024-09-21T10:24:49.417 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:49 smithi098 ceph-mon[100914]: pgmap v1090: 99 pgs: 1 active+remapped+backfill_wait, 34 peering, 64 active+clean; 5.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 771 op/s; 5083/250278 objects misplaced (2.031%); 16 KiB/s, 2 objects/s recovering 2024-09-21T10:24:49.417 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:49 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:49.417 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:49 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:24:49.417 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:49 smithi098 ceph-mon[100914]: osdmap e325: 8 total, 8 up, 7 in 2024-09-21T10:24:49.423 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:49.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:49.524 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:49.575 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:49.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:49 smithi196 ceph-mon[84784]: pgmap v1090: 99 pgs: 1 active+remapped+backfill_wait, 34 peering, 64 active+clean; 5.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 771 op/s; 5083/250278 objects misplaced (2.031%); 16 KiB/s, 2 objects/s recovering 2024-09-21T10:24:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:49 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:49 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:24:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:49 smithi196 ceph-mon[84784]: osdmap e325: 8 total, 8 up, 7 in 2024-09-21T10:24:49.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:49.689 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:24:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:24:49.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.717 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:49.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:49.725 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:24:49.725 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:24:49.752 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:24:49.752 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:24:49.783 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:24:49.783 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:24:49.809 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:24:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:24:49.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:49.844 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:24:49.845 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:24:49.857 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:49.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:49.872 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:24:49.872 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:24:49.906 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:24:49.907 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:24:49.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:49.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.992 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:49.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:50.093 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:50.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:50.128 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:50.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:50.183 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 4] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:24:50.184 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T10:24:50.229 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:50.270 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:50.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:50.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:50.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:50.410 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:50.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:50.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:50.512 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:50.547 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:50.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:50.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:50.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:50 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4040460825' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:50.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:50 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4040460825' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:50 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4040460825' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:50.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:24:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:24:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:24:50.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:50.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:50.682 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:50.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:50.783 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:50.821 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:50.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:50.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:50.923 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:50.953 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:50.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:50.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:51.055 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:51.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:51.091 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:51.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:51.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:24:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:24:50] "GET /metrics HTTP/1.1" 200 36526 "" "Prometheus/2.51.0" 2024-09-21T10:24:51.160 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:24:51.193 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:51.227 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:51.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:51.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:51.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:51.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:51.352 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:51.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:51.453 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:51.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:51.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:51.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:51.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:51 smithi098 ceph-mon[97580]: pgmap v1092: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 4.5 KiB/s wr, 788 op/s; 19814/243408 objects misplaced (8.140%); 6.0 MiB/s, 2 keys/s, 104 objects/s recovering 2024-09-21T10:24:51.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:51 smithi098 ceph-mon[100914]: pgmap v1092: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 4.5 KiB/s wr, 788 op/s; 19814/243408 objects misplaced (8.140%); 6.0 MiB/s, 2 keys/s, 104 objects/s recovering 2024-09-21T10:24:51.605 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:51.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:51.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:51.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:51 smithi196 ceph-mon[84784]: pgmap v1092: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 4.5 KiB/s wr, 788 op/s; 19814/243408 objects misplaced (8.140%); 6.0 MiB/s, 2 keys/s, 104 objects/s recovering 2024-09-21T10:24:51.708 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:51.747 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:51.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:51.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:51.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:51.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:51.877 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:51.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:51.978 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:52.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:52.007 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:52.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:52.107 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:52.143 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:52.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:52.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:52.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:52.284 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:52.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:52.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:52.385 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:52.426 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:52.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:52.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:52.527 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:52.568 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:52.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:52.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:52.670 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:52.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:52.705 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:52.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:52.806 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:52.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:52.851 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:52.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:52.951 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:53.021 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:53.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:53.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:53.123 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:53.153 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:53.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:53.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:53.255 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:53.291 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:53.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:53.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:53.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:53 smithi098 ceph-mon[97580]: pgmap v1093: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 3.2 KiB/s wr, 602 op/s; 19814/242526 objects misplaced (8.170%); 4.2 MiB/s, 1 keys/s, 73 objects/s recovering 2024-09-21T10:24:53.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:24:53.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:53.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:53 smithi098 ceph-mon[100914]: pgmap v1093: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 3.2 KiB/s wr, 602 op/s; 19814/242526 objects misplaced (8.170%); 4.2 MiB/s, 1 keys/s, 73 objects/s recovering 2024-09-21T10:24:53.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:24:53.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:53.393 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:53.434 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:53.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:53.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:53.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:53.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:53.566 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:53.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:53.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:53 smithi196 ceph-mon[84784]: pgmap v1093: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 3.2 KiB/s wr, 602 op/s; 19814/242526 objects misplaced (8.170%); 4.2 MiB/s, 1 keys/s, 73 objects/s recovering 2024-09-21T10:24:53.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:24:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:53.667 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:53.695 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:24:53.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:53.720 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:53.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:53.821 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:53.863 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:53.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:53.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:53.929 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:53.961 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:24:53.961 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:24:53.965 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:54.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:54.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:54.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:54.163 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:54.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:54.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:54.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:54.296 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:54.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:54.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:54.398 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:54.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:54.443 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:54.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:54.544 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:54.582 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:54.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:54.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:54.684 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:54.725 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:54.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:54.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:54.827 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:54.858 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:54.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:54.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:54.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:55.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:55.019 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:55.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:55.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:55.153 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:55.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:55.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:55.255 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:55.294 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:55.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:55.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:55.396 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:55 smithi098 ceph-mon[97580]: pgmap v1094: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 2.5 KiB/s wr, 860 op/s; 19232/233544 objects misplaced (8.235%); 3.3 MiB/s, 1 keys/s, 57 objects/s recovering 2024-09-21T10:24:55.396 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:55 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:24:55.396 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:55 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:24:55.397 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:55 smithi098 ceph-mon[100914]: pgmap v1094: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 2.5 KiB/s wr, 860 op/s; 19232/233544 objects misplaced (8.235%); 3.3 MiB/s, 1 keys/s, 57 objects/s recovering 2024-09-21T10:24:55.397 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:55 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:24:55.397 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:55 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:24:55.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:55.429 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:55.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:55.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:55.531 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:55.562 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:55.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:55.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:55.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:55 smithi196 ceph-mon[84784]: pgmap v1094: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 2.5 KiB/s wr, 860 op/s; 19232/233544 objects misplaced (8.235%); 3.3 MiB/s, 1 keys/s, 57 objects/s recovering 2024-09-21T10:24:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:55 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:24:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:55 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:24:55.664 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:55.693 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:55.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:55.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:55.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:55.826 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:55.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:55.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:55.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:55.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:24:56.012 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:56.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:56.114 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:56.159 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:56.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:56.261 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:56.298 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:56.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:56.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:56.439 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:56.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:56.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:56.572 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:56.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:56.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:56.711 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:56.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:56.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:56.841 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:56.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:56.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:56.969 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:56.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:57.071 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:57.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:57.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:57.199 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:57.232 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:57.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:57.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:57.334 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:57.367 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:57.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:57.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:57.468 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:57.502 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:57.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:57.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:57.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:57 smithi098 ceph-mon[97580]: pgmap v1095: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 2.2 KiB/s wr, 964 op/s; 18074/228270 objects misplaced (7.918%); 7.5 MiB/s, 1 keys/s, 123 objects/s recovering 2024-09-21T10:24:57.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:57 smithi098 ceph-mon[100914]: pgmap v1095: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 2.2 KiB/s wr, 964 op/s; 18074/228270 objects misplaced (7.918%); 7.5 MiB/s, 1 keys/s, 123 objects/s recovering 2024-09-21T10:24:57.604 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:57.637 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:57.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:57.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:57 smithi196 ceph-mon[84784]: pgmap v1095: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 2.2 KiB/s wr, 964 op/s; 18074/228270 objects misplaced (7.918%); 7.5 MiB/s, 1 keys/s, 123 objects/s recovering 2024-09-21T10:24:57.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:57.774 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:57.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:57.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:57.875 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:57.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:57.907 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:57.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:58.008 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:58.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:58.140 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:58.168 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:58.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:58.270 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:58.299 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:58.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:58.401 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:58.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.443 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:58.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:58.543 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:58.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.586 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:58.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:58.687 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:58.718 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:58.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:58.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:58.860 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:58.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:58.962 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:58.983 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:58.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:59.086 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:59.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:59.128 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:59.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:59.229 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:59.266 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:59.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:59.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:59.368 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:59.394 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:59.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:59.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:59.496 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:59.525 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:59.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:59.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:59.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:24:59 smithi098 ceph-mon[97580]: pgmap v1096: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.9 KiB/s wr, 886 op/s; 17744/223788 objects misplaced (7.929%); 6.5 MiB/s, 0 keys/s, 106 objects/s recovering 2024-09-21T10:24:59.538 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:24:59 smithi098 ceph-mon[100914]: pgmap v1096: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.9 KiB/s wr, 886 op/s; 17744/223788 objects misplaced (7.929%); 6.5 MiB/s, 0 keys/s, 106 objects/s recovering 2024-09-21T10:24:59.626 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:59.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:24:59 smithi196 ceph-mon[84784]: pgmap v1096: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.9 KiB/s wr, 886 op/s; 17744/223788 objects misplaced (7.929%); 6.5 MiB/s, 0 keys/s, 106 objects/s recovering 2024-09-21T10:24:59.666 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:24:59.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:59.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:59.768 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:59.800 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:59.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:59.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:59.901 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:59.950 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:24:59.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:59.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:00.052 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:00.084 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:00.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:00.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:00.186 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:00.224 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:00.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:00.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:00.260 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:25:00.325 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:00.364 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:00.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:00.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:00.465 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:00.494 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:00.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:00.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:00.596 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:00.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:00 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2771036035' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:00.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:00 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2771036035' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:00.636 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:00.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:00.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:00.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:00 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2771036035' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:00.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:25:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:25:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:25:00.737 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:00.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:00.775 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:00.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:00.876 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:00.915 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:00.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:00.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:01.017 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:01.052 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:01.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:01.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:25:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:25:00] "GET /metrics HTTP/1.1" 200 36552 "" "Prometheus/2.51.0" 2024-09-21T10:25:01.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:01.192 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:01.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:01.281 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T10:25:01.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:01.331 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:01.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:01.433 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:01.465 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:01.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:01.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:01.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:01 smithi098 ceph-mon[97580]: pgmap v1097: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s wr, 1.06k op/s; 16166/214713 objects misplaced (7.529%); 10 MiB/s, 0 keys/s, 161 objects/s recovering 2024-09-21T10:25:01.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:01 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:01.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:01 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2771036035' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:25:01.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:01 smithi098 ceph-mon[97580]: osdmap e326: 8 total, 8 up, 7 in 2024-09-21T10:25:01.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:01 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3763493974' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:25:01.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:01 smithi098 ceph-mon[100914]: pgmap v1097: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s wr, 1.06k op/s; 16166/214713 objects misplaced (7.529%); 10 MiB/s, 0 keys/s, 161 objects/s recovering 2024-09-21T10:25:01.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:01 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:01.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:01 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2771036035' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:25:01.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:01 smithi098 ceph-mon[100914]: osdmap e326: 8 total, 8 up, 7 in 2024-09-21T10:25:01.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:01 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3763493974' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:25:01.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:01.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:01.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:01 smithi196 ceph-mon[84784]: pgmap v1097: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s wr, 1.06k op/s; 16166/214713 objects misplaced (7.529%); 10 MiB/s, 0 keys/s, 161 objects/s recovering 2024-09-21T10:25:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:01 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:01 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2771036035' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:25:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:01 smithi196 ceph-mon[84784]: osdmap e326: 8 total, 8 up, 7 in 2024-09-21T10:25:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:01 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3763493974' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:25:01.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:01.729 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:01.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:01.831 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:01.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.862 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:01.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:01.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:01.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:02.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:02.142 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:02.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:02.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:02.244 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:02.274 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:02.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:02.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:02.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:02.404 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:02.426 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:02.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:02.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:02.507 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:02 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3763493974' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:25:02.508 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:02 smithi098 ceph-mon[97580]: osdmap e327: 8 total, 8 up, 7 in 2024-09-21T10:25:02.508 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:02 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3763493974' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:25:02.508 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:02 smithi098 ceph-mon[100914]: osdmap e327: 8 total, 8 up, 7 in 2024-09-21T10:25:02.527 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:02.563 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:02.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:02.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:02.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:02 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3763493974' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:25:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:02 smithi196 ceph-mon[84784]: osdmap e327: 8 total, 8 up, 7 in 2024-09-21T10:25:02.666 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:02.703 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:02.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:02.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:02.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:02.833 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:02.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:02.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:02.934 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:02.979 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:02.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:02.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:03.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T10:25:03.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:03.111 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:03.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:03.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:03.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:03.258 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:03.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:03.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:03.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:03.405 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:03.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:03.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:03.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:03 smithi098 ceph-mon[97580]: pgmap v1100: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 821 op/s; 16166/213822 objects misplaced (7.560%); 11 MiB/s, 171 objects/s recovering 2024-09-21T10:25:03.506 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:03 smithi098 ceph-mon[100914]: pgmap v1100: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 821 op/s; 16166/213822 objects misplaced (7.560%); 11 MiB/s, 171 objects/s recovering 2024-09-21T10:25:03.507 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:03.541 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:03.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:03.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:03.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:03 smithi196 ceph-mon[84784]: pgmap v1100: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 821 op/s; 16166/213822 objects misplaced (7.560%); 11 MiB/s, 171 objects/s recovering 2024-09-21T10:25:03.643 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:03.688 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:03.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:03.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:03.790 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:03.827 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:03.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:03.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:03.929 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:03.968 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:03.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:03.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:04.054 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:04.069 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:04.121 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:04.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:04.147 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:25:04.223 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:04.267 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:04.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:04.368 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:04.407 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:04.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:04.509 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:04.545 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:04.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:04.646 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:04.674 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:04.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:04.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:04.812 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:04.813 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:04.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:04.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:04.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.952 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:04.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:05.053 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:05.099 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:05.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:05.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:05.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:05.226 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:05.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:05.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:05.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:05.362 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:05.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:05.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:05.464 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:05.504 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:05.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:05.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:05.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:05 smithi098 ceph-mon[97580]: pgmap v1101: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 973 op/s; 15593/204897 objects misplaced (7.610%); 5.8 MiB/s, 92 objects/s recovering 2024-09-21T10:25:05.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:05 smithi098 ceph-mon[100914]: pgmap v1101: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 973 op/s; 15593/204897 objects misplaced (7.610%); 5.8 MiB/s, 92 objects/s recovering 2024-09-21T10:25:05.606 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:05.636 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:05.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:05.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:05.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:05 smithi196 ceph-mon[84784]: pgmap v1101: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 973 op/s; 15593/204897 objects misplaced (7.610%); 5.8 MiB/s, 92 objects/s recovering 2024-09-21T10:25:05.738 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:05.774 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:05.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:05.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:05.875 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:05.914 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:05.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:05.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:06.016 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:06.052 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:06.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:06.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:06.154 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:06.187 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:06.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:06.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:06.289 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:06.323 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:06.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:06.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:06.424 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:06.462 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:06.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:06.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:06.564 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:06.601 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:06.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:06.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:06.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:06.734 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:06.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:06.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:06.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:06.866 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:06.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:06.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:06.995 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:06.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:06.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:07.097 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:07.128 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:07.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:07.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:07.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:07.267 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:07.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:07.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:07.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:07.406 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:07.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:07.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:07.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:07.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:07.548 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:07.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:07 smithi098 ceph-mon[97580]: pgmap v1102: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 14419/199545 objects misplaced (7.226%); 11 MiB/s, 180 objects/s recovering 2024-09-21T10:25:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:07.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:07 smithi098 ceph-mon[100914]: pgmap v1102: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 14419/199545 objects misplaced (7.226%); 11 MiB/s, 180 objects/s recovering 2024-09-21T10:25:07.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:07.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:07 smithi196 ceph-mon[84784]: pgmap v1102: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 14419/199545 objects misplaced (7.226%); 11 MiB/s, 180 objects/s recovering 2024-09-21T10:25:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:07.649 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:07.686 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:07.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:07.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:07.788 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:07.824 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:07.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:07.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:07.926 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:07.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:07.962 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:07.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:08.063 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:08.103 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:08.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:08.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:08.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:08.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:08.237 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:08.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:08.338 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:08.367 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:08.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:08.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:08.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:08.513 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:08.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:08.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:08.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:08.635 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:08.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:08.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:08.737 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:08.770 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:08.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:08.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:08.872 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:08.914 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:08.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:08.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:09.016 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:09.047 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:09.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:09.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:09.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:09.170 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:09.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:09.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:09.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:09.305 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:09.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:09.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:09.407 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:09.439 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:09.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:09.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:09.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:09.570 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:09.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:09.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:09.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:09 smithi098 ceph-mon[97580]: pgmap v1103: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 828 op/s; 14124/194826 objects misplaced (7.250%); 5.5 MiB/s, 88 objects/s recovering 2024-09-21T10:25:09.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:09 smithi098 ceph-mon[100914]: pgmap v1103: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 828 op/s; 14124/194826 objects misplaced (7.250%); 5.5 MiB/s, 88 objects/s recovering 2024-09-21T10:25:09.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:09 smithi196 ceph-mon[84784]: pgmap v1103: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 4.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 828 op/s; 14124/194826 objects misplaced (7.250%); 5.5 MiB/s, 88 objects/s recovering 2024-09-21T10:25:09.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:09.706 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:09.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:09.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:09.808 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:09.846 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:09.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:09.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:09.948 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:09.974 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:09.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:09.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:10.075 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:10.119 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:10.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:10.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:10.220 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:10.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:10.258 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:10.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:10.359 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:10.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:10.386 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:10.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:10.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:25:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:25:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:25:10.487 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:10.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:10.536 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:10.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:10.637 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:10.665 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:10.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:10.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:10.768 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:10.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:10.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:10.806 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:25:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:25:10] "GET /metrics HTTP/1.1" 200 36548 "" "Prometheus/2.51.0" 2024-09-21T10:25:10.899 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:10.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:10.939 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:10.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:11.041 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:11.083 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:11.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:11.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:11.185 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:11.210 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:11.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:11.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:11.312 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:11.346 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:11.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:11 smithi098 ceph-mon[97580]: pgmap v1104: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1k op/s; 12694/186075 objects misplaced (6.822%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:25:11.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:11 smithi098 ceph-mon[100914]: pgmap v1104: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1k op/s; 12694/186075 objects misplaced (6.822%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:25:11.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:11.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:11.448 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:11.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:11.479 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:11.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:11.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:11.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:11.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:11.617 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:11 smithi196 ceph-mon[84784]: pgmap v1104: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1k op/s; 12694/186075 objects misplaced (6.822%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:25:11.712 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:11.758 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:11.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:11.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:11.861 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:11.894 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:11.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:11.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:11.996 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:12.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:12.037 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:12.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:12.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:12.198 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:12.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:12.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:12.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:12.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:12.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:12.440 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:12.473 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:12.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:12.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:12.575 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:12.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:12.621 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:12.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:12.722 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:12.765 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:12.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:12.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:12.867 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:12.915 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:12.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:12.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:13.017 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:13.055 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:13.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:13.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:13.157 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:13.191 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:13.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:13.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:13.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:13.327 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:13.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:13.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:13.429 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:13.468 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:13.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:13.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:13.500 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:25:13.500 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:25:13.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:13 smithi098 ceph-mon[97580]: pgmap v1105: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 930 op/s; 12694/185280 objects misplaced (6.851%); 8.9 MiB/s, 142 objects/s recovering 2024-09-21T10:25:13.501 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:13 smithi098 ceph-mon[100914]: pgmap v1105: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 930 op/s; 12694/185280 objects misplaced (6.851%); 8.9 MiB/s, 142 objects/s recovering 2024-09-21T10:25:13.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:13.606 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:13.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:13.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:13.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:13 smithi196 ceph-mon[84784]: pgmap v1105: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 930 op/s; 12694/185280 objects misplaced (6.851%); 8.9 MiB/s, 142 objects/s recovering 2024-09-21T10:25:13.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:13.743 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:13.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:13.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:13.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:13.871 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:13.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:13.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:13.973 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:14.008 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:14.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:14.110 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:14.148 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:14.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:14.249 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:14.287 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:14.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:14.335 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T10:25:14.389 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:14.428 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:14.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:14.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:14.576 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:14.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:14.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:14 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3558446762' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:25:14.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:14 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:25:14.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:14 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3558446762' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:25:14.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:14 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:25:14.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:14 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3558446762' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:25:14.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:14 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:25:14.679 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:14.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.716 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:14.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:14.781 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:25:14.781 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:25:14.808 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:25:14.808 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:25:14.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:14.837 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:25:14.837 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:25:14.860 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:25:14.860 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:25:14.863 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:14.895 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:25:14.895 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:25:14.922 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:25:14.923 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:25:14.963 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:25:14.963 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:25:14.965 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:14.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.997 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:14.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:15.001 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:25:15.001 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:25:15.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:15.132 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:15.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:15.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:15.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:15.270 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:15.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:15.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:15.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:15.406 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:15.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:15.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:15.508 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:15.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:15.547 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:15.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:15.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:15 smithi098 ceph-mon[97580]: pgmap v1106: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 3.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 12175/177369 objects misplaced (6.864%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T10:25:15.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:15 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:25:15.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:15 smithi098 ceph-mon[97580]: osdmap e328: 8 total, 8 up, 7 in 2024-09-21T10:25:15.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:15 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/605123710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:15.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:15 smithi098 ceph-mon[100914]: pgmap v1106: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 3.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 12175/177369 objects misplaced (6.864%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T10:25:15.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:15 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:25:15.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:15 smithi098 ceph-mon[100914]: osdmap e328: 8 total, 8 up, 7 in 2024-09-21T10:25:15.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:15 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/605123710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:15.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:15 smithi196 ceph-mon[84784]: pgmap v1106: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 3.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 12175/177369 objects misplaced (6.864%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T10:25:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:15 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:25:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:15 smithi196 ceph-mon[84784]: osdmap e328: 8 total, 8 up, 7 in 2024-09-21T10:25:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:15 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/605123710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:15.648 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:15.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T10:25:15.683 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:15.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:15.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:15.785 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:15.817 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:15.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:15.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:15.918 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:15.956 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:15.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:15.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:16.058 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:16.086 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:16.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:16.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:16.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.217 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:16.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:16.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:16.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.357 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:16.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:16.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:16.487 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:16.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:16.588 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:16.623 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:16.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:16.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:16.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.760 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:16.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:16.860 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:16.907 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:16.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:17.008 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:17.044 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:17.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:17.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:17.145 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:17.184 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:17.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:17.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:17.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:17.315 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:17.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:17.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:17.417 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:17.461 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:17.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:17.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:17.563 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:17.597 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:17.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:17 smithi098 ceph-mon[97580]: pgmap v1108: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 909 op/s; 11187/172239 objects misplaced (6.495%); 8.7 MiB/s, 139 objects/s recovering 2024-09-21T10:25:17.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:17 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1072812036' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:25:17.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:17 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:25:17.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:17 smithi098 ceph-mon[100914]: pgmap v1108: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 909 op/s; 11187/172239 objects misplaced (6.495%); 8.7 MiB/s, 139 objects/s recovering 2024-09-21T10:25:17.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:17 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1072812036' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:25:17.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:17 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:25:17.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:17.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:17.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:17 smithi196 ceph-mon[84784]: pgmap v1108: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 909 op/s; 11187/172239 objects misplaced (6.495%); 8.7 MiB/s, 139 objects/s recovering 2024-09-21T10:25:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:17 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1072812036' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:25:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:17 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:25:17.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:25:17.702 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:17.737 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:17.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:17.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:17.758 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:25:17.838 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:17.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:17.876 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:17.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:17.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:18.004 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:18.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:18.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:18.138 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:18.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:18.240 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:18.273 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:18.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:18.375 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:18.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.408 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:18.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:18.509 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:18.537 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:18.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:18.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:18 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:18 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:25:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:18 smithi196 ceph-mon[84784]: osdmap e329: 8 total, 8 up, 7 in 2024-09-21T10:25:18.661 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:18.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:18.761 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:18 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:18.761 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:18 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:25:18.761 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:18 smithi098 ceph-mon[97580]: osdmap e329: 8 total, 8 up, 7 in 2024-09-21T10:25:18.761 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:18 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:18.761 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:18 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:25:18.761 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:18 smithi098 ceph-mon[100914]: osdmap e329: 8 total, 8 up, 7 in 2024-09-21T10:25:18.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:18.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:18.886 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:18.909 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:18.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:19.011 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:19.033 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:19.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:19.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:19.123 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:25:19.134 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:19.187 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:19.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:19.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:19.207 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:19.246 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:19.290 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:19.312 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:19.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:19.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:19.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:19.437 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:19.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:19.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:19.510 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:19 smithi098 ceph-mon[97580]: pgmap v1110: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 759 op/s; 10902/167832 objects misplaced (6.496%); 5.0 MiB/s, 79 objects/s recovering 2024-09-21T10:25:19.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:19 smithi098 ceph-mon[97580]: osdmap e330: 8 total, 8 up, 7 in 2024-09-21T10:25:19.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:19 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:25:19.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:19 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:25:19.511 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:19 smithi098 ceph-mon[100914]: pgmap v1110: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 759 op/s; 10902/167832 objects misplaced (6.496%); 5.0 MiB/s, 79 objects/s recovering 2024-09-21T10:25:19.511 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:19 smithi098 ceph-mon[100914]: osdmap e330: 8 total, 8 up, 7 in 2024-09-21T10:25:19.511 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:19 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:25:19.511 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:19 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:25:19.539 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:19.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:19.585 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:19.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:19.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:19 smithi196 ceph-mon[84784]: pgmap v1110: 99 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 95 active+clean; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 759 op/s; 10902/167832 objects misplaced (6.496%); 5.0 MiB/s, 79 objects/s recovering 2024-09-21T10:25:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:19 smithi196 ceph-mon[84784]: osdmap e330: 8 total, 8 up, 7 in 2024-09-21T10:25:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:19 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:25:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:19 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:25:19.685 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:19.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:19.723 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:19.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:19.824 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:19.854 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:19.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:19.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:19.956 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:19.985 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:19.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:19.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:20.087 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:20.126 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:20.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:20.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:20.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:20.257 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:20.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:20.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:20.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:20.399 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:20.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:20.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:20.500 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:20.529 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:20.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:20.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:20.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:25:20.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:20 smithi098 ceph-mon[97580]: osdmap e331: 8 total, 8 up, 7 in 2024-09-21T10:25:20.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:20 smithi098 ceph-mon[97580]: pgmap v1113: 99 pgs: 1 active+remapped+backfilling, 1 peering, 2 active+remapped+backfill_wait, 95 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 890 op/s; 9530/159300 objects misplaced (5.982%); 9.5 MiB/s, 152 objects/s recovering 2024-09-21T10:25:20.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:20 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:25:20.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:20 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:25:20.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:20 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:25:20.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:20 smithi098 ceph-mon[100914]: osdmap e331: 8 total, 8 up, 7 in 2024-09-21T10:25:20.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:20 smithi098 ceph-mon[100914]: pgmap v1113: 99 pgs: 1 active+remapped+backfilling, 1 peering, 2 active+remapped+backfill_wait, 95 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 890 op/s; 9530/159300 objects misplaced (5.982%); 9.5 MiB/s, 152 objects/s recovering 2024-09-21T10:25:20.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:20 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:25:20.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:20 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:25:20.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:20 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:25:20.631 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:20.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:20 smithi196 ceph-mon[84784]: osdmap e331: 8 total, 8 up, 7 in 2024-09-21T10:25:20.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:20 smithi196 ceph-mon[84784]: pgmap v1113: 99 pgs: 1 active+remapped+backfilling, 1 peering, 2 active+remapped+backfill_wait, 95 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 890 op/s; 9530/159300 objects misplaced (5.982%); 9.5 MiB/s, 152 objects/s recovering 2024-09-21T10:25:20.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:20 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:25:20.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:20 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:25:20.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:20 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:25:20.642 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:25:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:25:20.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:25:20.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:20.670 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:20.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:20.771 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:20.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:20.819 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:20.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:20.920 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:20.950 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:20.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:20.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:21.052 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:21.086 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:21.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:21.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:21.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:25:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:25:20] "GET /metrics HTTP/1.1" 200 36548 "" "Prometheus/2.51.0" 2024-09-21T10:25:21.187 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:21.227 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:21.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:21.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:21.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:21.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:21.378 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:21.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:21.479 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:21.515 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:21.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:21.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:21.617 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:21.655 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:21.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:21.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:21.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:21.787 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:21.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:21.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:21.889 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:21.915 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:21.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:21.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:22.017 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:22.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:22.060 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:22.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:22.161 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:22.203 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:22.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:22.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:22.306 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:22.340 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:22.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:22.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:22.442 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:22.488 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:22.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:22.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:22.590 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:22.618 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:22.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:22.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:22.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:22.776 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:22.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:22.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:22.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:22 smithi098 ceph-mon[97580]: pgmap v1114: 99 pgs: 1 active+remapped+backfilling, 1 peering, 2 active+remapped+backfill_wait, 95 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 9530/158580 objects misplaced (6.010%); 8.2 MiB/s, 131 objects/s recovering 2024-09-21T10:25:22.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:22.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:22 smithi098 ceph-mon[100914]: pgmap v1114: 99 pgs: 1 active+remapped+backfilling, 1 peering, 2 active+remapped+backfill_wait, 95 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 9530/158580 objects misplaced (6.010%); 8.2 MiB/s, 131 objects/s recovering 2024-09-21T10:25:22.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:22.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:22 smithi196 ceph-mon[84784]: pgmap v1114: 99 pgs: 1 active+remapped+backfilling, 1 peering, 2 active+remapped+backfill_wait, 95 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 9530/158580 objects misplaced (6.010%); 8.2 MiB/s, 131 objects/s recovering 2024-09-21T10:25:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:22.916 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:22.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:22.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:23.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:23.047 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:23.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:23.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:23.180 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:23.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:23.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:23.282 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:23.318 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:23.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:23.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:23.419 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:23.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:23.457 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:23.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:23.462 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:25:23.462 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:25:23.462 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-21T10:25:23.462 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-21T10:25:23.462 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:25:23.558 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:23.599 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:23.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:23.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:23.698 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:23.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:23.758 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:23.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:23.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:23.860 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:23.891 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:23.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:23.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:23.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:24.034 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:24.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:24.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:24.136 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:24.177 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:24.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:24.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:24.279 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:24.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:24.324 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:24.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:24.425 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:24.462 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:24.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:24.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:24.564 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:24.583 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T10:25:24.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:24.609 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:24.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:24.651 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:25:24.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:24.744 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:24.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:24.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:24.845 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:24.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:24 smithi098 ceph-mon[97580]: pgmap v1115: 99 pgs: 1 active+remapped+backfilling, 1 peering, 2 active+remapped+backfill_wait, 95 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 893 op/s; 9054/151344 objects misplaced (5.982%); 8.0 MiB/s, 128 objects/s recovering 2024-09-21T10:25:24.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:24 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2232054225' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:25:24.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:24 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:25:24.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:24 smithi098 ceph-mon[100914]: pgmap v1115: 99 pgs: 1 active+remapped+backfilling, 1 peering, 2 active+remapped+backfill_wait, 95 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 893 op/s; 9054/151344 objects misplaced (5.982%); 8.0 MiB/s, 128 objects/s recovering 2024-09-21T10:25:24.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:24 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2232054225' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:25:24.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:24 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:25:24.851 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:24.885 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:24.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:24.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:24 smithi196 ceph-mon[84784]: pgmap v1115: 99 pgs: 1 active+remapped+backfilling, 1 peering, 2 active+remapped+backfill_wait, 95 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 893 op/s; 9054/151344 objects misplaced (5.982%); 8.0 MiB/s, 128 objects/s recovering 2024-09-21T10:25:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:24 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2232054225' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:25:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:24 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:25:24.987 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:25.017 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:25.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:25.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:25.119 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:25.159 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:25.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:25.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:25.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:25.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:25.290 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:25.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:25.391 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:25.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T10:25:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:25.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:25.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:25.529 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:25.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:25.569 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:25.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:25.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:25 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:25:25.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:25 smithi098 ceph-mon[97580]: osdmap e332: 8 total, 8 up, 7 in 2024-09-21T10:25:25.660 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:25 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:25:25.660 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:25 smithi098 ceph-mon[100914]: osdmap e332: 8 total, 8 up, 7 in 2024-09-21T10:25:25.671 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:25.700 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:25.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:25.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:25.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:25.843 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:25.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:25.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:25.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:25 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:25:25.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:25 smithi196 ceph-mon[84784]: osdmap e332: 8 total, 8 up, 7 in 2024-09-21T10:25:25.946 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:25.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:25.992 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:25.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:26.093 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:26.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:26.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:26.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:26.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:26.260 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:26.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:26.361 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:26.403 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:26.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:26.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:26.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:26.531 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:26.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:26.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:26.634 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:26.675 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:26.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:26.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:26.776 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:26.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:26.818 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:26.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:26.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:26 smithi098 ceph-mon[100914]: pgmap v1117: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 96 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 992 op/s; 7890/146559 objects misplaced (5.383%); 16 MiB/s, 248 objects/s recovering 2024-09-21T10:25:26.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:26 smithi098 ceph-mon[97580]: pgmap v1117: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 96 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 992 op/s; 7890/146559 objects misplaced (5.383%); 16 MiB/s, 248 objects/s recovering 2024-09-21T10:25:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:26 smithi196 ceph-mon[84784]: pgmap v1117: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 96 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 992 op/s; 7890/146559 objects misplaced (5.383%); 16 MiB/s, 248 objects/s recovering 2024-09-21T10:25:26.919 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:26.954 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:26.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:26.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:27.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:27.083 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:27.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:27.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:27.185 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:27.225 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:27.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:27.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:27.326 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:27.354 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:27.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:27.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:27.455 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:27.503 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:27.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:27.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:27.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:27.634 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:27.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:27.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:27.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:27.772 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:27.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:27.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:27.874 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:27.903 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:27.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:27.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:28.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:28.039 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:28.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:28.140 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:28.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:28.270 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:28.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.300 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:28.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:28.401 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:28.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.443 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:28.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:28.544 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:28.574 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:28.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:28.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:28.700 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:28.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:28.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:28.849 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:28.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:28.859 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:28 smithi196 ceph-mon[84784]: pgmap v1118: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 96 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 693 op/s; 7622/142362 objects misplaced (5.354%); 7.6 MiB/s, 120 objects/s recovering 2024-09-21T10:25:28.951 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:28.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:28 smithi098 ceph-mon[100914]: pgmap v1118: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 96 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 693 op/s; 7622/142362 objects misplaced (5.354%); 7.6 MiB/s, 120 objects/s recovering 2024-09-21T10:25:28.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:28 smithi098 ceph-mon[97580]: pgmap v1118: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 96 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 693 op/s; 7622/142362 objects misplaced (5.354%); 7.6 MiB/s, 120 objects/s recovering 2024-09-21T10:25:28.987 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:28.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:29.089 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:29.123 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:29.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:29.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:29.225 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:29.266 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:29.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:29.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:29.367 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:29.406 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:29.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:29.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:29.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:29.547 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:29.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:29.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:29.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:29.676 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:29.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:29.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:29.778 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:29.813 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:29.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:29.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:29.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:29.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:29.944 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:29.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:30.045 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:30.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:30.078 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:30.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:30.179 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:30.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:30.221 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:30.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:30.322 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:30.361 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:30.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:30.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:30.463 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:30.485 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:30.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:30.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:30.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:30.607 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:25:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:25:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:25:30.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:30 smithi196 ceph-mon[84784]: pgmap v1119: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 96 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 857 op/s; 6523/133557 objects misplaced (4.884%); 10 MiB/s, 165 objects/s recovering 2024-09-21T10:25:30.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:30 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2563476909' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:30.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:30 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:30.618 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:30.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:30.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:30.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:30.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:30.755 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:30.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:30.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:30.887 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:30.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:30.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:30.989 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:31.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:31.034 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:31.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:31.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:30 smithi098 ceph-mon[97580]: pgmap v1119: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 96 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 857 op/s; 6523/133557 objects misplaced (4.884%); 10 MiB/s, 165 objects/s recovering 2024-09-21T10:25:31.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:30 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2563476909' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:31.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:30 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:31.098 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:25:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:25:30] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-21T10:25:31.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:30 smithi098 ceph-mon[100914]: pgmap v1119: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 96 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 857 op/s; 6523/133557 objects misplaced (4.884%); 10 MiB/s, 165 objects/s recovering 2024-09-21T10:25:31.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:30 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2563476909' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:31.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:30 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:31.135 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:31.171 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:31.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:31.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:31.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:31.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:31.303 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:31.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:31.404 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:31.441 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:31.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:31.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:31.543 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:31.573 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:31.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:31.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:31.675 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:31.719 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:31.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:31.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:31.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:31.849 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:31.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:31.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:31.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:31 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:25:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:31 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:25:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:31 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:25:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:31 smithi196 ceph-mon[84784]: osdmap e333: 8 total, 8 up, 7 in 2024-09-21T10:25:31.951 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:31.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:31 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:25:31.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:31 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:25:31.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:31 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:25:31.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:31 smithi098 ceph-mon[97580]: osdmap e333: 8 total, 8 up, 7 in 2024-09-21T10:25:31.965 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:31 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:25:31.965 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:31 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:25:31.965 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:31 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:25:31.965 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:31 smithi098 ceph-mon[100914]: osdmap e333: 8 total, 8 up, 7 in 2024-09-21T10:25:31.983 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:31.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:31.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:32.085 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:32.126 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:32.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:32.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:32.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:32.261 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:32.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:32.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:32.363 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:32.398 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:32.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:32.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:32.500 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:32.542 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:32.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:32.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:32.645 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:32.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:32.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:32.764 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:25:32.766 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:32 smithi098 ceph-mon[97580]: pgmap v1121: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 96 active+clean; 2.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 779 op/s; 6523/132630 objects misplaced (4.918%); 13 MiB/s, 207 objects/s recovering 2024-09-21T10:25:32.766 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:32 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3563565601' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:32.767 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:32 smithi098 ceph-mon[100914]: pgmap v1121: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 96 active+clean; 2.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 779 op/s; 6523/132630 objects misplaced (4.918%); 13 MiB/s, 207 objects/s recovering 2024-09-21T10:25:32.767 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:32 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3563565601' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:32.787 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:32.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:32.828 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:32.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:32.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:32 smithi196 ceph-mon[84784]: pgmap v1121: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 96 active+clean; 2.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 779 op/s; 6523/132630 objects misplaced (4.918%); 13 MiB/s, 207 objects/s recovering 2024-09-21T10:25:32.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:32 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3563565601' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:32.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:32.959 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:32.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:32.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:33.061 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:33.069 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:25:33.069 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:25:33.069 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:25:33.069 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:25:33.069 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:25:33.070 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:25:33.070 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:25:33.070 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:25:33.070 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:25:33.070 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:25:33.070 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:25:33.070 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:25:33.070 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:25:33.071 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:25:33.071 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:25:33.071 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:25:33.098 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:33.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:33.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:33.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:33.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:33.231 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:33.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:33.332 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:33.375 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:33.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:33.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:33.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:33.505 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:33.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:33.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:33.607 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:33.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:33.651 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:33.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:33.752 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:33.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:33.789 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:33.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:33.890 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:33.926 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:33.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:33.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:34.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:34.075 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:34.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:34.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:34.205 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:34.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:34.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:34.337 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:34.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:34.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:34.474 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:34.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:34.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:34.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:34.707 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:34.735 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:34.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:34.838 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:34.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:34 smithi098 ceph-mon[97580]: pgmap v1122: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 96 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 869 op/s; 5991/124104 objects misplaced (4.827%); 4.7 MiB/s, 75 objects/s recovering 2024-09-21T10:25:34.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:34 smithi098 ceph-mon[100914]: pgmap v1122: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 96 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 869 op/s; 5991/124104 objects misplaced (4.827%); 4.7 MiB/s, 75 objects/s recovering 2024-09-21T10:25:34.866 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:34.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:34 smithi196 ceph-mon[84784]: pgmap v1122: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 96 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 869 op/s; 5991/124104 objects misplaced (4.827%); 4.7 MiB/s, 75 objects/s recovering 2024-09-21T10:25:34.969 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:34.997 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:34.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:35.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:35.137 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:35.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:35.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:35.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:35.281 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:35.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:35.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:35.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:35.415 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:35.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:35.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:35.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:35.544 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:35.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:35.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:35.646 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:35.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:35.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:35.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:35.818 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:35.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:35.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:25:35.920 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:35.951 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:35.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:35.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:36.053 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:36.073 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:36.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:36.175 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:36.201 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:36.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:36.302 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:36.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:36.426 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:36.448 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:36.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:36.549 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:36.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:36.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:36.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:36.694 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:36.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:36.795 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:36.818 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:36.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:36.913 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:25:36.920 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:36.996 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:36.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:37.038 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:36 smithi098 ceph-mon[97580]: pgmap v1123: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 96 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 919 op/s; 5269/118953 objects misplaced (4.429%); 8.3 MiB/s, 132 objects/s recovering 2024-09-21T10:25:37.038 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:36 smithi098 ceph-mon[97580]: osdmap e334: 8 total, 8 up, 7 in 2024-09-21T10:25:37.038 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:36 smithi098 ceph-mon[100914]: pgmap v1123: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 96 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 919 op/s; 5269/118953 objects misplaced (4.429%); 8.3 MiB/s, 132 objects/s recovering 2024-09-21T10:25:37.039 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:36 smithi098 ceph-mon[100914]: osdmap e334: 8 total, 8 up, 7 in 2024-09-21T10:25:37.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:37.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:36 smithi196 ceph-mon[84784]: pgmap v1123: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 96 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 919 op/s; 5269/118953 objects misplaced (4.429%); 8.3 MiB/s, 132 objects/s recovering 2024-09-21T10:25:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:36 smithi196 ceph-mon[84784]: osdmap e334: 8 total, 8 up, 7 in 2024-09-21T10:25:37.151 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:37.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:37.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:37.254 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:37.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:37.355 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:37.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:37.457 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:37.493 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:37.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:37.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:37.595 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:37.601 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:25:37.601 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:25:37.646 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:37.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:37.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:37.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:37.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:37.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:37.890 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:37.919 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:37.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:37.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:37.989 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:37 smithi098 ceph-mon[97580]: osdmap e335: 8 total, 8 up, 7 in 2024-09-21T10:25:37.989 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:25:37.989 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:37.989 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:37 smithi098 ceph-mon[100914]: osdmap e335: 8 total, 8 up, 7 in 2024-09-21T10:25:37.989 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:25:37.989 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:38.021 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:38.056 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:38.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:38.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:37 smithi196 ceph-mon[84784]: osdmap e335: 8 total, 8 up, 7 in 2024-09-21T10:25:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:25:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:38.158 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:38.192 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:38.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:38.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:38.328 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:38.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:38.429 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:38.463 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:38.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:38.476 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:38.481 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:25:38.498 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:38.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:38.601 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:38.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:38.702 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:38.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:38.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:38.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:38.847 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T10:25:38.897 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:38.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:38.998 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:39.028 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:39.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:39.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:38 smithi098 ceph-mon[97580]: pgmap v1126: 99 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 96 active+clean; 2.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 840 op/s; 4504/115254 objects misplaced (3.908%); 11 MiB/s, 170 objects/s recovering 2024-09-21T10:25:39.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:38 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1913995069' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:25:39.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.9", "id": [4, 0]} : dispatch 2024-09-21T10:25:39.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]} : dispatch 2024-09-21T10:25:39.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]} : dispatch 2024-09-21T10:25:39.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:38 smithi098 ceph-mon[100914]: pgmap v1126: 99 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 96 active+clean; 2.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 840 op/s; 4504/115254 objects misplaced (3.908%); 11 MiB/s, 170 objects/s recovering 2024-09-21T10:25:39.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:38 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1913995069' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:25:39.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.9", "id": [4, 0]} : dispatch 2024-09-21T10:25:39.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]} : dispatch 2024-09-21T10:25:39.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]} : dispatch 2024-09-21T10:25:39.129 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:39.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:38 smithi196 ceph-mon[84784]: pgmap v1126: 99 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 96 active+clean; 2.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 840 op/s; 4504/115254 objects misplaced (3.908%); 11 MiB/s, 170 objects/s recovering 2024-09-21T10:25:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:38 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1913995069' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:25:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.9", "id": [4, 0]} : dispatch 2024-09-21T10:25:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]} : dispatch 2024-09-21T10:25:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]} : dispatch 2024-09-21T10:25:39.169 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:39.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:39.270 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:39.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.308 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:39.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:39.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:39.443 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:39.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:39.545 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:39.574 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:39.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:39.675 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:39.705 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:39.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:39.806 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:39.833 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:39.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:39.936 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:39.971 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:39.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:40.073 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:40.099 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:40.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:40.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:40.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:39 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1913995069' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:25:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:39 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.9", "id": [4, 0]}]': finished 2024-09-21T10:25:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:39 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]}]': finished 2024-09-21T10:25:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:39 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]}]': finished 2024-09-21T10:25:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:39 smithi196 ceph-mon[84784]: osdmap e336: 8 total, 8 up, 7 in 2024-09-21T10:25:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:39 smithi196 ceph-mon[84784]: osdmap e337: 8 total, 8 up, 7 in 2024-09-21T10:25:40.199 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:39 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1913995069' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:25:40.199 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:39 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.9", "id": [4, 0]}]': finished 2024-09-21T10:25:40.199 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:39 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]}]': finished 2024-09-21T10:25:40.199 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:39 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]}]': finished 2024-09-21T10:25:40.199 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:39 smithi098 ceph-mon[97580]: osdmap e336: 8 total, 8 up, 7 in 2024-09-21T10:25:40.200 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:39 smithi098 ceph-mon[97580]: osdmap e337: 8 total, 8 up, 7 in 2024-09-21T10:25:40.200 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:39 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1913995069' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:25:40.200 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:39 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.9", "id": [4, 0]}]': finished 2024-09-21T10:25:40.200 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:39 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 5]}]': finished 2024-09-21T10:25:40.200 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:39 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]}]': finished 2024-09-21T10:25:40.200 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:39 smithi098 ceph-mon[100914]: osdmap e336: 8 total, 8 up, 7 in 2024-09-21T10:25:40.200 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:39 smithi098 ceph-mon[100914]: osdmap e337: 8 total, 8 up, 7 in 2024-09-21T10:25:40.201 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:40.242 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:40.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:40.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:40.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:40.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:40.383 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:40.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:40.484 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:40.520 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:40.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:40.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:40.622 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:40.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:25:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:25:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:25:40.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:40.656 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:40.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:40.756 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:40.768 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T10:25:40.804 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:40.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:40.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:40.906 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:40.938 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:40.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:40.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:40.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:40 smithi098 ceph-mon[97580]: pgmap v1128: 99 pgs: 1 peering, 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 96 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 892 op/s; 3984/108024 objects misplaced (3.688%); 15 MiB/s, 243 objects/s recovering 2024-09-21T10:25:40.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:40 smithi098 ceph-mon[97580]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:25:40.992 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:25:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:25:40] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-21T10:25:40.993 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:40 smithi098 ceph-mon[100914]: pgmap v1128: 99 pgs: 1 peering, 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 96 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 892 op/s; 3984/108024 objects misplaced (3.688%); 15 MiB/s, 243 objects/s recovering 2024-09-21T10:25:40.993 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:40 smithi098 ceph-mon[100914]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:25:41.039 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:41.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:41.070 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:41.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:41.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:40 smithi196 ceph-mon[84784]: pgmap v1128: 99 pgs: 1 peering, 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 96 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 892 op/s; 3984/108024 objects misplaced (3.688%); 15 MiB/s, 243 objects/s recovering 2024-09-21T10:25:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:40 smithi196 ceph-mon[84784]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:25:41.170 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:41.202 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:41.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:41.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:41.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:41.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:41.333 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:41.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:41.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:41.479 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:41.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:41.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:41.581 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:41.602 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:41.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:41.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:41.638 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 10:25:41 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=cleanup t=2024-09-21T10:25:41.385938706Z level=info msg="Completed cleanup jobs" duration=31.452687ms 2024-09-21T10:25:41.704 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:41.732 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:41.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:41.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:41.834 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:41.878 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:41.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:41.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:41.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:42.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:42.013 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:42.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:42.114 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:42.155 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:42.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:42.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:42.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:42.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:42.288 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:42.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:42.390 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:42.424 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:42.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:42.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:42.526 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:42.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:42.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:42.664 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:42.704 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:42.710 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:42.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:42.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:42.812 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:42.852 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:42.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:42.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:25:42.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:42.999 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:43.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:43.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:42 smithi098 ceph-mon[97580]: pgmap v1130: 99 pgs: 1 peering, 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 96 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 681 op/s; 3984/107331 objects misplaced (3.712%); 8.9 MiB/s, 142 objects/s recovering 2024-09-21T10:25:43.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:42 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3837132501' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:25:43.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:42 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:25:43.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:42 smithi098 ceph-mon[100914]: pgmap v1130: 99 pgs: 1 peering, 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 96 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 681 op/s; 3984/107331 objects misplaced (3.712%); 8.9 MiB/s, 142 objects/s recovering 2024-09-21T10:25:43.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:42 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3837132501' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:25:43.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:42 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:25:43.101 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:42 smithi196 ceph-mon[84784]: pgmap v1130: 99 pgs: 1 peering, 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 96 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 681 op/s; 3984/107331 objects misplaced (3.712%); 8.9 MiB/s, 142 objects/s recovering 2024-09-21T10:25:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:42 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3837132501' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:25:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:42 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:25:43.139 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:43.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:43.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:43.275 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:43.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:43.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:43.405 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:43.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:43.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:43.533 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:43.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:43.634 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:43.672 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:43.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:43.775 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:43.821 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:43.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:43.923 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:43.956 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:43.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:44.058 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:44.098 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:44.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:44.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:44.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:43 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:43 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:25:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:43 smithi196 ceph-mon[84784]: osdmap e338: 8 total, 8 up, 7 in 2024-09-21T10:25:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:43 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1513631680' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:44.199 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:44.203 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:25:44.203 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:25:44.203 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:25:44.203 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:25:44.203 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:25:44.203 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:25:44.204 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:25:44.204 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:25:44.204 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:25:44.204 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:25:44.204 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:25:44.204 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:25:44.204 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:25:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:25:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:25:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:25:44.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:43 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:44.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:43 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:25:44.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:43 smithi098 ceph-mon[97580]: osdmap e338: 8 total, 8 up, 7 in 2024-09-21T10:25:44.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:43 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1513631680' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:44.206 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:43 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:44.206 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:43 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:25:44.206 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:43 smithi098 ceph-mon[100914]: osdmap e338: 8 total, 8 up, 7 in 2024-09-21T10:25:44.206 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:43 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1513631680' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:44.244 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:44.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:44.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:44.346 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:44.379 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:44.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:44.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:44.481 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:44.514 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:44.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:44.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:44.616 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:44.644 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:44.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:44.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:44.746 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:44.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:44.779 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:44.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:44.880 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:44.916 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:44.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:44.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:45.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:45.046 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:45.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:45.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:45.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:45.176 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:45.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:45.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:45.192 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:44 smithi098 ceph-mon[97580]: pgmap v1132: 99 pgs: 1 active+remapped+backfilling, 1 peering, 1 active+remapped+backfill_wait, 96 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 902 op/s; 2696/99006 objects misplaced (2.723%); 12 MiB/s, 186 objects/s recovering 2024-09-21T10:25:45.193 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:44 smithi098 ceph-mon[100914]: pgmap v1132: 99 pgs: 1 active+remapped+backfilling, 1 peering, 1 active+remapped+backfill_wait, 96 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 902 op/s; 2696/99006 objects misplaced (2.723%); 12 MiB/s, 186 objects/s recovering 2024-09-21T10:25:45.277 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:45.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:45.316 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:45.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:45.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:44 smithi196 ceph-mon[84784]: pgmap v1132: 99 pgs: 1 active+remapped+backfilling, 1 peering, 1 active+remapped+backfill_wait, 96 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 902 op/s; 2696/99006 objects misplaced (2.723%); 12 MiB/s, 186 objects/s recovering 2024-09-21T10:25:45.418 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:45.447 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:45.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:45.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:45.549 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:45.584 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:45.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:45.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:45.686 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:45.725 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:45.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:45.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:45.827 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:45.855 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:45.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:45.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:45.958 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:45.994 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:45.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:45.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:46.051 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:46.097 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:46.134 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:46.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:46.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:46.145 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:45 smithi098 ceph-mon[100914]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:25:46.145 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:45 smithi098 ceph-mon[97580]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:25:46.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:46.273 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:46.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:46.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:46.375 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:46.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:45 smithi196 ceph-mon[84784]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:25:46.412 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:46.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:46.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:46.514 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:46.556 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:46.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:46.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:46.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:46.687 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:46.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:46.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:46.789 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:46.834 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:46.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:46.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:46.936 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:46.968 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:46.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:46.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:46.986 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:25:47.070 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:47.108 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:47.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:47.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:47.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:47.242 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:47.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:47.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:47.310 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:46 smithi098 ceph-mon[97580]: pgmap v1133: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 97 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 851 op/s; 2696/93807 objects misplaced (2.874%); 10 MiB/s, 167 objects/s recovering 2024-09-21T10:25:47.311 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:46 smithi098 ceph-mon[100914]: pgmap v1133: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 97 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 851 op/s; 2696/93807 objects misplaced (2.874%); 10 MiB/s, 167 objects/s recovering 2024-09-21T10:25:47.344 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:47.388 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:47.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:46 smithi196 ceph-mon[84784]: pgmap v1133: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 97 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 851 op/s; 2696/93807 objects misplaced (2.874%); 10 MiB/s, 167 objects/s recovering 2024-09-21T10:25:47.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:47.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:47.490 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:47.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:47.520 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:47.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:47.621 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:47.658 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:47.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:47.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:47.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:47.793 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:47.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:47.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:47.895 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:47.933 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:47.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:47.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:48.035 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:48.069 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:48.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:48.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:48.171 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:48.215 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:48.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:48.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:48.317 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:48.356 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:48.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:48.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:48.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:48.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:48.502 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:48.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:48.587 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:25:48.604 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:48.614 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:25:48.614 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:25:48.614 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-21T10:25:48.615 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:25:48.623 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:48.633 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:48.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:48.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:48.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:48.771 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:48.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:48.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:48.872 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:48.912 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:48.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:48.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:49.014 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:49.049 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:49.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:49.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:49.151 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:49.179 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:49.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:49.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:49.281 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:49.316 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:49.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:49.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:49.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:48 smithi098 ceph-mon[97580]: pgmap v1134: 99 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 97 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 780 op/s; 2499/89286 objects misplaced (2.799%); 6.5 MiB/s, 104 objects/s recovering 2024-09-21T10:25:49.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:48 smithi098 ceph-mon[100914]: pgmap v1134: 99 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 97 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 780 op/s; 2499/89286 objects misplaced (2.799%); 6.5 MiB/s, 104 objects/s recovering 2024-09-21T10:25:49.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:48 smithi196 ceph-mon[84784]: pgmap v1134: 99 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 97 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 780 op/s; 2499/89286 objects misplaced (2.799%); 6.5 MiB/s, 104 objects/s recovering 2024-09-21T10:25:49.419 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:49.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:49.456 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:49.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:49.557 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:49.591 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:49.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:49.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:49.693 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:49.735 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:49.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:49.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:49.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:49.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:49.870 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:49.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:49.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:50.014 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:50.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:50.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:50.116 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:50.145 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:50.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:50.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:50.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:50.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:50.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:50.404 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:50.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:50.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:50.505 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:50.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:25:50.542 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:50.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:50.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:50.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:25:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:25:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:25:50.644 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:50.681 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:50.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:50.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:50.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:50.818 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:50.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:50.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:50.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:25:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:25:50] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-21T10:25:50.920 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:50.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:50.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:50.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:51.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:51.089 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:51.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:51.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:51.191 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:51.197 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:50 smithi098 ceph-mon[97580]: pgmap v1135: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 97 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 928 op/s; 1657/80397 objects misplaced (2.061%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:25:51.197 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:50 smithi098 ceph-mon[100914]: pgmap v1135: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 97 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 928 op/s; 1657/80397 objects misplaced (2.061%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:25:51.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:51.233 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:51.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:51.334 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:51.362 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:51.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:51.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:51.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:50 smithi196 ceph-mon[84784]: pgmap v1135: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 97 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 928 op/s; 1657/80397 objects misplaced (2.061%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:25:51.464 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:51.502 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:51.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:51.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:51.603 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:51.647 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:51.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:51.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:51.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:51.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:51.798 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:51.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:51.898 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:51.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:51.934 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:51.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:52.036 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:52.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:52.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:52.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:52.173 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:52.210 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:52.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:52.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:52.236 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:52.311 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:52.353 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:52.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:52.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:52.455 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:52.492 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:52.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:52.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:52.594 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:52.626 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:52.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:52.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:52.727 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:52.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:52.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:52.859 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:52.892 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:52.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:52.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:52.989 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T10:25:52.994 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:53.013 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:52 smithi098 ceph-mon[97580]: pgmap v1136: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 97 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 927 op/s; 1657/79479 objects misplaced (2.085%); 9.5 MiB/s, 152 objects/s recovering 2024-09-21T10:25:53.013 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:52 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3819402557' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:25:53.013 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:52 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:25:53.013 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:25:53.013 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:53.013 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:52 smithi098 ceph-mon[100914]: pgmap v1136: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 97 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 927 op/s; 1657/79479 objects misplaced (2.085%); 9.5 MiB/s, 152 objects/s recovering 2024-09-21T10:25:53.013 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:52 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3819402557' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:25:53.013 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:52 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:25:53.013 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:25:53.014 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:53.031 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:53.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:53.133 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:53.163 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:53.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:53.286 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:53.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:53.388 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:53.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:52 smithi196 ceph-mon[84784]: pgmap v1136: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 97 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 927 op/s; 1657/79479 objects misplaced (2.085%); 9.5 MiB/s, 152 objects/s recovering 2024-09-21T10:25:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:52 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3819402557' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:25:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:52 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:25:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:25:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:53.411 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:53.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:53.512 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:53.534 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:53.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:53.636 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:53.658 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:53.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:53.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:53.781 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:53.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:53.884 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:53.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.907 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:53.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:54.008 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:54.030 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:54.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:54.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:53 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:25:54.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:53 smithi098 ceph-mon[97580]: osdmap e339: 8 total, 8 up, 7 in 2024-09-21T10:25:54.088 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:53 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:25:54.088 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:53 smithi098 ceph-mon[100914]: osdmap e339: 8 total, 8 up, 7 in 2024-09-21T10:25:54.132 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:54.157 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:54.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:54.259 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:54.281 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:54.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:54.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:54.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:53 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:25:54.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:53 smithi196 ceph-mon[84784]: osdmap e339: 8 total, 8 up, 7 in 2024-09-21T10:25:54.409 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:54.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:54.510 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:54.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.539 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:54.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:54.640 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:54.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.678 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:54.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:54.779 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:54.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:54.909 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:54.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.932 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:54.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:55.033 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:55.068 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:55.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:55.170 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:55.201 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:55.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:55.303 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:55.327 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:55.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:55.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:54 smithi098 ceph-mon[97580]: pgmap v1138: 99 pgs: 1 peering, 1 active+remapped+backfill_wait, 97 active+clean; 1.4 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 947 op/s; 884/70578 objects misplaced (1.253%); 7.8 MiB/s, 124 objects/s recovering 2024-09-21T10:25:55.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:54 smithi098 ceph-mon[97580]: osdmap e340: 8 total, 8 up, 7 in 2024-09-21T10:25:55.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:54 smithi098 ceph-mon[100914]: pgmap v1138: 99 pgs: 1 peering, 1 active+remapped+backfill_wait, 97 active+clean; 1.4 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 947 op/s; 884/70578 objects misplaced (1.253%); 7.8 MiB/s, 124 objects/s recovering 2024-09-21T10:25:55.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:54 smithi098 ceph-mon[100914]: osdmap e340: 8 total, 8 up, 7 in 2024-09-21T10:25:55.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:54 smithi196 ceph-mon[84784]: pgmap v1138: 99 pgs: 1 peering, 1 active+remapped+backfill_wait, 97 active+clean; 1.4 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 947 op/s; 884/70578 objects misplaced (1.253%); 7.8 MiB/s, 124 objects/s recovering 2024-09-21T10:25:55.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:54 smithi196 ceph-mon[84784]: osdmap e340: 8 total, 8 up, 7 in 2024-09-21T10:25:55.429 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:55.450 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:55.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:55.551 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:55.573 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:55.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:55.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:55.695 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:55.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:55.797 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:55.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.821 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:55.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:55.922 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:55.943 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:55.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:56.045 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:56.070 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:56.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:56.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:56.172 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:56.203 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:56.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:56.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:56.305 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:56.344 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:56.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:56.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:56.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:56 smithi098 ceph-mon[97580]: osdmap e341: 8 total, 8 up, 7 in 2024-09-21T10:25:56.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:56 smithi098 ceph-mon[100914]: osdmap e341: 8 total, 8 up, 7 in 2024-09-21T10:25:56.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:56 smithi196 ceph-mon[84784]: osdmap e341: 8 total, 8 up, 7 in 2024-09-21T10:25:56.447 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:56.479 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:56.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:56.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:56.581 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:56.601 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:56.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:56.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:56.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:56.738 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:56.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:56.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:56.840 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:56.879 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:56.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:56.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:56.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:57.019 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:57.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:57.070 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T10:25:57.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:57.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:57.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:57.267 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:57.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:57.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:57 smithi098 ceph-mon[97580]: pgmap v1141: 99 pgs: 1 peering, 1 active+remapped+backfill_wait, 97 active+clean; 1.3 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 787 op/s; 884/66219 objects misplaced (1.335%); 5.8 MiB/s, 92 objects/s recovering 2024-09-21T10:25:57.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:57 smithi098 ceph-mon[97580]: osdmap e342: 8 total, 8 up, 7 in 2024-09-21T10:25:57.329 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:57 smithi098 ceph-mon[100914]: pgmap v1141: 99 pgs: 1 peering, 1 active+remapped+backfill_wait, 97 active+clean; 1.3 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 787 op/s; 884/66219 objects misplaced (1.335%); 5.8 MiB/s, 92 objects/s recovering 2024-09-21T10:25:57.329 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:57 smithi098 ceph-mon[100914]: osdmap e342: 8 total, 8 up, 7 in 2024-09-21T10:25:57.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:57 smithi196 ceph-mon[84784]: pgmap v1141: 99 pgs: 1 peering, 1 active+remapped+backfill_wait, 97 active+clean; 1.3 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 787 op/s; 884/66219 objects misplaced (1.335%); 5.8 MiB/s, 92 objects/s recovering 2024-09-21T10:25:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:57 smithi196 ceph-mon[84784]: osdmap e342: 8 total, 8 up, 7 in 2024-09-21T10:25:57.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:57.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.439 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:57.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:57.540 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:57.589 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:57.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:57.690 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:57.733 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:57.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:57.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:57.860 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:57.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:57.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.008 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:58.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:58.109 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:58.143 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:58.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:58.245 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:58.280 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:58.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:58.382 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:58.421 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:58.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:58.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:58.559 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:58.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:58.661 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:58.697 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:58.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:58.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:58.832 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:58.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:58.904 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:58.934 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:58.969 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:58.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:59.039 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:25:59 smithi098 ceph-mon[97580]: pgmap v1143: 99 pgs: 1 peering, 98 active+clean; 1.3 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 849 op/s; 19 MiB/s, 310 objects/s recovering 2024-09-21T10:25:59.039 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:25:59 smithi098 ceph-mon[100914]: pgmap v1143: 99 pgs: 1 peering, 98 active+clean; 1.3 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 849 op/s; 19 MiB/s, 310 objects/s recovering 2024-09-21T10:25:59.039 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:25:59.040 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:25:59.040 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-21T10:25:59.040 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-21T10:25:59.040 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:25:59.070 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:59.106 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:59.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:59.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:59.208 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:59.242 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:59.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:59.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:59.344 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:59.378 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:59.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:59.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:59.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:25:59 smithi196 ceph-mon[84784]: pgmap v1143: 99 pgs: 1 peering, 98 active+clean; 1.3 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 849 op/s; 19 MiB/s, 310 objects/s recovering 2024-09-21T10:25:59.480 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:59.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:59.521 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:25:59.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:59.621 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:59.654 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:59.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:59.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:59.756 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:59.784 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:59.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:59.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:59.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:59.915 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:25:59.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:59.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:00.017 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:00.057 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:00.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:00.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:00.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:26:00.158 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:00.190 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:00.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:00.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:00.292 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:00.333 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:00.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:00.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:00.401 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:26:00.402 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:26:00.434 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:26:00.434 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:26:00.435 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:00.464 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:26:00.465 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:26:00.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:00.474 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:00.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:00.495 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:26:00.495 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:26:00.527 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:26:00.528 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:26:00.556 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:26:00.556 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:26:00.575 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:00.582 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:26:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:26:00.611 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:00.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:00.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:00.619 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:26:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:26:00.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:26:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:26:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:26:00.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:00.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:00.748 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:00.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:00.849 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:00.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:00.880 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:00.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:00.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:00.999 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:26:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:26:00] "GET /metrics HTTP/1.1" 200 36544 "" "Prometheus/2.51.0" 2024-09-21T10:26:01.009 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:01.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:01.111 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:01.141 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:01.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:01.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:01.279 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:01.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:01.297 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:01 smithi098 ceph-mon[97580]: pgmap v1144: 99 pgs: 99 active+clean; 1.2 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 768 op/s; 8.9 MiB/s, 142 objects/s recovering 2024-09-21T10:26:01.297 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:01 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1163935242' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:01.297 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:01 smithi098 ceph-mon[100914]: pgmap v1144: 99 pgs: 99 active+clean; 1.2 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 768 op/s; 8.9 MiB/s, 142 objects/s recovering 2024-09-21T10:26:01.297 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:01 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1163935242' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:01.381 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:01.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:01 smithi196 ceph-mon[84784]: pgmap v1144: 99 pgs: 99 active+clean; 1.2 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 768 op/s; 8.9 MiB/s, 142 objects/s recovering 2024-09-21T10:26:01.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:01 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1163935242' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:01.411 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:01.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:01.512 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:01.541 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:01.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:01.643 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:01.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.676 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:01.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:01.777 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:01.817 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:01.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:01.919 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:01.952 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:01.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:02.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:02.087 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:02.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:02.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:02.189 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:02.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:02.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:02.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:02.333 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:02.387 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:02.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:02.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:02.489 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:02.529 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:02.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:02.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:02.630 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:02.663 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:02.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:02.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:02.765 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:02.805 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:02.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:02.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:02.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:02.939 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:02.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:02.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:03.026 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:26:03.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:03.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:03.086 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:03.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:03.187 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:03.213 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:03.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:03.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:03.290 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:03.315 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:03.316 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:03 smithi098 ceph-mon[97580]: pgmap v1145: 99 pgs: 99 active+clean; 1.1 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 487 op/s; 7.1 MiB/s, 113 objects/s recovering 2024-09-21T10:26:03.316 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:03 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2286258217' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:03.317 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:03 smithi098 ceph-mon[100914]: pgmap v1145: 99 pgs: 99 active+clean; 1.1 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 487 op/s; 7.1 MiB/s, 113 objects/s recovering 2024-09-21T10:26:03.317 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:03 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2286258217' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:03.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:03.355 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:03.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:03.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:03 smithi196 ceph-mon[84784]: pgmap v1145: 99 pgs: 99 active+clean; 1.1 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 487 op/s; 7.1 MiB/s, 113 objects/s recovering 2024-09-21T10:26:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:03 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2286258217' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:03.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:03.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:03.484 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:03.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:03.585 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:03.619 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:03.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:03.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:03.721 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:03.752 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:03.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:03.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:03.782 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:26:03.853 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:03.888 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:03.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:03.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:03.990 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:04.024 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:04.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:04.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:04.126 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:04.175 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:04.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:04.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:04.219 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:04 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:04.219 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:04 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2286258217' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:26:04.219 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:04 smithi098 ceph-mon[97580]: osdmap e343: 8 total, 8 up, 7 in 2024-09-21T10:26:04.219 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:04 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:04.219 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:04 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2286258217' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:26:04.219 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:04 smithi098 ceph-mon[100914]: osdmap e343: 8 total, 8 up, 7 in 2024-09-21T10:26:04.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:04.315 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:04.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:04.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:04.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:04 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:04 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2286258217' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:26:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:04 smithi196 ceph-mon[84784]: osdmap e343: 8 total, 8 up, 7 in 2024-09-21T10:26:04.417 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:04.456 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:04.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:04.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:04.558 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:04.588 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:04.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:04.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:04.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:04.722 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:04.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:04.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:04.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:04.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:04.863 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:04.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:04.964 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:05.000 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:05.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T10:26:05.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:05.139 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:05.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:05.240 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:05.268 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:05.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:05.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:05 smithi098 ceph-mon[97580]: pgmap v1147: 99 pgs: 99 active+clean; 967 MiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 830 op/s; 6.7 MiB/s, 106 objects/s recovering 2024-09-21T10:26:05.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:05 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1531850935' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:26:05.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:05 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:26:05.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:05 smithi098 ceph-mon[100914]: pgmap v1147: 99 pgs: 99 active+clean; 967 MiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 830 op/s; 6.7 MiB/s, 106 objects/s recovering 2024-09-21T10:26:05.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:05 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1531850935' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:26:05.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:05 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:26:05.371 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:05.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:05 smithi196 ceph-mon[84784]: pgmap v1147: 99 pgs: 99 active+clean; 967 MiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 830 op/s; 6.7 MiB/s, 106 objects/s recovering 2024-09-21T10:26:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:05 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1531850935' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:26:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:05 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:26:05.407 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:05.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:05.509 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:05.547 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:05.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:05.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:05.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.680 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:05.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:05.781 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:05.826 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:05.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:05.928 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:05.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.970 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:05.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:06.071 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:06.108 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:06.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:06.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:06.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:06.237 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:06.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:06.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:06.330 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:06 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:26:06.330 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:06 smithi098 ceph-mon[100914]: osdmap e344: 8 total, 8 up, 7 in 2024-09-21T10:26:06.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:06 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:26:06.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:06 smithi098 ceph-mon[97580]: osdmap e344: 8 total, 8 up, 7 in 2024-09-21T10:26:06.339 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:06.370 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T10:26:06.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:06.374 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:06.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:06.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:06 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:26:06.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:06 smithi196 ceph-mon[84784]: osdmap e344: 8 total, 8 up, 7 in 2024-09-21T10:26:06.475 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:06.512 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:06.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:06.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:06.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:06.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:06.660 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:06.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:06.761 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:06.803 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:06.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:06.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:06.904 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:06.914 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:06.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:06.962 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:06.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:07.049 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:26:07.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:07.113 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:07 smithi098 ceph-mon[97580]: pgmap v1149: 99 pgs: 99 active+clean; 848 MiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 950 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:26:07.115 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:07 smithi098 ceph-mon[100914]: pgmap v1149: 99 pgs: 99 active+clean; 848 MiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 950 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:26:07.130 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:07.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:07.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:07.232 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:07.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:07.272 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:07.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:07.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:07 smithi196 ceph-mon[84784]: pgmap v1149: 99 pgs: 99 active+clean; 848 MiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 950 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:26:07.406 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:07.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:07.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:07.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:07.540 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:07.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:07.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:07.642 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:07.680 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:07.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:07.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:07.782 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:07.813 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:07.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:07.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:07.838 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:07.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:07.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:07.947 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:07.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:08.048 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:08.077 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:08.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:08.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:08.179 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:08.217 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:08.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:08.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:08.319 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:08.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:08.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:08.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:08.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:08.356 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:08.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:08.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:08.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:08.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:08.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:08.495 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:08.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:08.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:08.596 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:08.626 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:08.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:08.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:08.727 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:08.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:08.770 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:08.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:08.871 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:08.874 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:26:08.874 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:26:08.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:08.906 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:08.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:09.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:09.041 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:09.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:09.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:09.143 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:09.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:09.170 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:09.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:09.271 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:09.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:09.305 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:09.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:09.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:09 smithi098 ceph-mon[97580]: pgmap v1150: 99 pgs: 99 active+clean; 751 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 868 op/s 2024-09-21T10:26:09.344 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:09 smithi098 ceph-mon[100914]: pgmap v1150: 99 pgs: 99 active+clean; 751 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 868 op/s 2024-09-21T10:26:09.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:09 smithi196 ceph-mon[84784]: pgmap v1150: 99 pgs: 99 active+clean; 751 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 868 op/s 2024-09-21T10:26:09.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:09.439 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:09.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:09.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:09.540 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:09.570 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:09.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:09.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:09.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:09.715 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:09.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:09.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:09.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:09.851 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:09.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:09.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:09.953 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:09.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:09.989 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:09.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:10.090 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:10.123 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:10.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:10.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:10.224 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:10.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:10.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:10.356 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:10.384 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:10.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:10.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:10.486 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:10.514 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:10.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:10.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:10.615 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:10.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:26:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:26:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:26:10.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:10.644 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:10.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:10.733 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:26:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:26:10] "GET /metrics HTTP/1.1" 200 36520 "" "Prometheus/2.51.0" 2024-09-21T10:26:10.744 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:10.779 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:10.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:10.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:10.880 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:10.917 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:10.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:10.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:11.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:11.049 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:11.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:11.091 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:26:11.151 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:11.187 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:11.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:11.289 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:11.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:11.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:11 smithi098 ceph-mon[97580]: pgmap v1151: 99 pgs: 99 active+clean; 551 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.23k op/s 2024-09-21T10:26:11.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:11 smithi098 ceph-mon[100914]: pgmap v1151: 99 pgs: 99 active+clean; 551 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.23k op/s 2024-09-21T10:26:11.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:11 smithi196 ceph-mon[84784]: pgmap v1151: 99 pgs: 99 active+clean; 551 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.23k op/s 2024-09-21T10:26:11.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:11.452 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:11.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:11.554 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:11.581 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:11.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:11.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:11.711 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:11.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:11.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:11.838 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:11.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:11.869 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:26:11.940 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:11.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:12.080 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:12.113 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:12.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:12.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:12.215 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:12.256 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:12.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:12.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:12.358 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:12.366 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:12 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3927014350' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:12.387 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:12.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:12.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:12.489 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:12.534 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:12.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:12.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:12.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:12 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3927014350' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:12.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:12 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3927014350' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:12.636 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:12.665 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:12.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:12.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:12.767 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:12.776 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:26:12.776 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:26:12.776 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:26:12.777 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:26:12.777 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:26:12.777 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:26:12.777 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:26:12.777 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:26:12.777 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:26:12.777 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:26:12.777 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:26:12.778 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:26:12.778 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:26:12.778 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:26:12.778 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:26:12.778 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:26:12.802 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:12.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:12.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:12.904 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:12.946 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:12.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:12.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:13.048 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:13.068 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:13.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:13.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:13.170 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:13.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:13.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:13.301 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:13.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:13.339 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:13.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:13.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:13 smithi196 ceph-mon[84784]: pgmap v1152: 99 pgs: 99 active+clean; 533 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s 2024-09-21T10:26:13.441 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:13.476 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:13.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:13.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:13.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:13.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:13 smithi098 ceph-mon[97580]: pgmap v1152: 99 pgs: 99 active+clean; 533 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s 2024-09-21T10:26:13.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:13 smithi098 ceph-mon[100914]: pgmap v1152: 99 pgs: 99 active+clean; 533 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s 2024-09-21T10:26:13.609 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:13.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:13.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:13.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:13.736 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:13.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:13.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:13.838 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:13.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:13.876 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:13.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:13.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:14.002 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:14.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:14.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:14.104 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:14.154 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:14.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:14.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:14.256 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:14.278 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:26:14.278 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:26:14.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.278 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46025.0:262924 9.4 e344/341 9:2d8d4fa3:::benchmark_data_smithi098_254407_object130193:head v 344'16332, mlcod=344'16330)", 2024-09-21T10:26:14.278 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:14.265656+0000", 2024-09-21T10:26:14.278 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0095662309999999997, 2024-09-21T10:26:14.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0095940950000000004, 2024-09-21T10:26:14.279 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:14.279 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:26:14.279 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:14.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:14.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.265656+0000", 2024-09-21T10:26:14.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:14.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:14.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.265656+0000", 2024-09-21T10:26:14.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:14.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:14.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.265655+0000", 2024-09-21T10:26:14.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:26:14.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:14.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.265663+0000", 2024-09-21T10:26:14.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3999999999999992e-06 2024-09-21T10:26:14.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:14.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.265680+0000", 2024-09-21T10:26:14.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6466000000000001e-05 2024-09-21T10:26:14.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:14.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.265687+0000", 2024-09-21T10:26:14.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6869999999999997e-06 2024-09-21T10:26:14.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:14.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.265734+0000", 2024-09-21T10:26:14.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7942000000000003e-05 2024-09-21T10:26:14.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:14.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.265757+0000", 2024-09-21T10:26:14.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2657e-05 2024-09-21T10:26:14.282 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:14.282 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:14.282 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:14.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.282 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:262929 9.c 9:3b89a32e:::benchmark_data_smithi098_254407_object130198:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-09-21T10:26:14.283 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:14.267781+0000", 2024-09-21T10:26:14.283 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.007440884, 2024-09-21T10:26:14.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0075314520000000001, 2024-09-21T10:26:14.283 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:14.283 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:26:14.283 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:14.283 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:14.283 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:14.283 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 262929 2024-09-21T10:26:14.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.283 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:14.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:14.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.267781+0000", 2024-09-21T10:26:14.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:14.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:14.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.267781+0000", 2024-09-21T10:26:14.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:14.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:14.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.267780+0000", 2024-09-21T10:26:14.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:14.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:14.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.267788+0000", 2024-09-21T10:26:14.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4610000000000008e-06 2024-09-21T10:26:14.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.285 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:14.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.267808+0000", 2024-09-21T10:26:14.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9796000000000002e-05 2024-09-21T10:26:14.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.285 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:14.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.267815+0000", 2024-09-21T10:26:14.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.498e-06 2024-09-21T10:26:14.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.285 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:14.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.267868+0000", 2024-09-21T10:26:14.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2880000000000002e-05 2024-09-21T10:26:14.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.286 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:14.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.268208+0000", 2024-09-21T10:26:14.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00033940400000000002 2024-09-21T10:26:14.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.286 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:26:14.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.268544+0000", 2024-09-21T10:26:14.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000336053 2024-09-21T10:26:14.286 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:14.287 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:14.287 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:14.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.287 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46025.0:262927 9.4 e344/341 9:256197fc:::benchmark_data_smithi098_254407_object130196:head v 344'16333, mlcod=344'16330)", 2024-09-21T10:26:14.287 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:14.271030+0000", 2024-09-21T10:26:14.287 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.004191897, 2024-09-21T10:26:14.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043496339999999998, 2024-09-21T10:26:14.287 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:14.287 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:26:14.287 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:14.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:14.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.271030+0000", 2024-09-21T10:26:14.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:14.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:14.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.271030+0000", 2024-09-21T10:26:14.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:14.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:14.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.271029+0000", 2024-09-21T10:26:14.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:26:14.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:14.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.271037+0000", 2024-09-21T10:26:14.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8420000000000005e-06 2024-09-21T10:26:14.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:14.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.271054+0000", 2024-09-21T10:26:14.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6997999999999999e-05 2024-09-21T10:26:14.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:14.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.271060+0000", 2024-09-21T10:26:14.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0929999999999999e-06 2024-09-21T10:26:14.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:14.290 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.272496+0000", 2024-09-21T10:26:14.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001435689 2024-09-21T10:26:14.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.290 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:14.290 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.272541+0000", 2024-09-21T10:26:14.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5272000000000003e-05 2024-09-21T10:26:14.290 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:14.290 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:14.290 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:14.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.290 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46025.0:262931 9.1 e344/330 9:82d8e3d5:::benchmark_data_smithi098_254407_object130200:head v 344'16499, mlcod=344'16497)", 2024-09-21T10:26:14.290 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:14.274758+0000", 2024-09-21T10:26:14.291 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.00046447600000000002, 2024-09-21T10:26:14.291 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00067351900000000003, 2024-09-21T10:26:14.291 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:14.291 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:26:14.291 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:14.291 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.291 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:14.291 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.274758+0000", 2024-09-21T10:26:14.291 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:14.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.291 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.291 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:14.291 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.274758+0000", 2024-09-21T10:26:14.291 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:14.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.292 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:14.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.274756+0000", 2024-09-21T10:26:14.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:14.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.292 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:14.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.274766+0000", 2024-09-21T10:26:14.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3559999999999999e-06 2024-09-21T10:26:14.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:14.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.274784+0000", 2024-09-21T10:26:14.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8053000000000001e-05 2024-09-21T10:26:14.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:14.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.274791+0000", 2024-09-21T10:26:14.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2230000000000001e-06 2024-09-21T10:26:14.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:14.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.274862+0000", 2024-09-21T10:26:14.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1125999999999996e-05 2024-09-21T10:26:14.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:14.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.274891+0000", 2024-09-21T10:26:14.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8682000000000001e-05 2024-09-21T10:26:14.294 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:14.294 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:14.294 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:14.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.294 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.46025.0:262929 9.c e344/339)", 2024-09-21T10:26:14.294 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:14.274958+0000", 2024-09-21T10:26:14.294 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.00026385500000000001, 2024-09-21T10:26:14.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00052256799999999995, 2024-09-21T10:26:14.295 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:14.295 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "queued for pg", 2024-09-21T10:26:14.295 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:14.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:14.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.274958+0000", 2024-09-21T10:26:14.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:14.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:14.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.274958+0000", 2024-09-21T10:26:14.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:14.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:14.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.274958+0000", 2024-09-21T10:26:14.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:26:14.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:14.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.274966+0000", 2024-09-21T10:26:14.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5650000000000002e-06 2024-09-21T10:26:14.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:14.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.274978+0000", 2024-09-21T10:26:14.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2036e-05 2024-09-21T10:26:14.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:14.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:14.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:14.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:14.274984+0000", 2024-09-21T10:26:14.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5419999999999996e-06 2024-09-21T10:26:14.299 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:14.299 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:14.299 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:14.299 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:14.299 INFO:teuthology.orchestra.run.smithi196.stdout: ], 2024-09-21T10:26:14.299 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 5 2024-09-21T10:26:14.299 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:26:14.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:14.301 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:14.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:14.402 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:14.431 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:14.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:14.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:14.532 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:14.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:14.580 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:14.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:14.680 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:14.718 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:14.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:14.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:14.767 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:14.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:14.855 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:14.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:14.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:14.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:14.991 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:14.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:14.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:15.093 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:15.115 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T10:26:15.139 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:15.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:15.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:15.210 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:15 smithi098 ceph-mon[97580]: pgmap v1153: 99 pgs: 99 active+clean; 416 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 880 op/s 2024-09-21T10:26:15.210 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:15 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/735770771' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:26:15.211 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:15 smithi098 ceph-mon[100914]: pgmap v1153: 99 pgs: 99 active+clean; 416 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 880 op/s 2024-09-21T10:26:15.211 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:15 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/735770771' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:26:15.230 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:26:15.241 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:15.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:15.321 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:15.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:15 smithi196 ceph-mon[84784]: pgmap v1153: 99 pgs: 99 active+clean; 416 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 880 op/s 2024-09-21T10:26:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:15 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/735770771' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:26:15.422 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:15.466 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:15.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:15.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:15.569 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:15.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:15.623 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:15.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:15.663 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:26:15.724 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:15.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:15.761 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:15.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:15.862 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:15.887 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:15.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:15.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:15.989 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:16.023 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:16.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:16.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:16.158 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:16.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:16.259 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:16.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.301 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:16.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:16.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:16 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/735770771' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:26:16.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:16 smithi098 ceph-mon[97580]: osdmap e345: 8 total, 8 up, 7 in 2024-09-21T10:26:16.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:16 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/735770771' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:26:16.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:16 smithi098 ceph-mon[100914]: osdmap e345: 8 total, 8 up, 7 in 2024-09-21T10:26:16.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:16 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/735770771' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:26:16.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:16 smithi196 ceph-mon[84784]: osdmap e345: 8 total, 8 up, 7 in 2024-09-21T10:26:16.402 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:16.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.436 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:16.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:16.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:16.563 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:16.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:16.664 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:16.694 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:16.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:16.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:16.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:16.929 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:16.950 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:16.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:17.052 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:17.086 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:17.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:17.095 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Removed 132728 objects 2024-09-21T10:26:17.095 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Clean up completed and total clean up time :133.838 2024-09-21T10:26:17.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:17.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.209 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:17.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:17.310 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:17.313 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:17 smithi098 ceph-mon[97580]: pgmap v1155: 99 pgs: 99 active+clean; 218 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s 2024-09-21T10:26:17.313 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:17 smithi098 ceph-mon[100914]: pgmap v1155: 99 pgs: 99 active+clean; 218 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s 2024-09-21T10:26:17.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.332 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:17.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:17.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:17 smithi196 ceph-mon[84784]: pgmap v1155: 99 pgs: 99 active+clean; 218 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s 2024-09-21T10:26:17.433 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:17.453 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:17.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:17.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:17.576 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:17.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:17.645 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T10:26:17.645 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:26:17.678 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:17.709 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:17.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:17.811 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:17.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.834 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:17.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:17.935 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:17.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:18.045 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:26:18.058 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:18.079 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:18.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:18.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:18.145 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:18.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:18.202 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:18.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:18.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:18.303 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:18.325 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:18.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:18.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:18.427 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:18.448 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:18.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:18.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:18.550 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:18.572 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:18.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:18.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:18.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:18.696 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:18.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:18.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:18.798 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:18.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:18.823 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:18.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:18.912 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-21T10:26:18.913 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-21T10:26:18.923 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:18.979 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:18.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:19.080 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:19.100 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:26:19.100 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:26:19.100 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-21T10:26:19.100 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-21T10:26:19.100 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:26:19.103 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:19.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:19.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:26:19.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:19.226 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:19.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:19.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:19.328 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:19.350 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:19.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:19.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:19.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:19 smithi196 ceph-mon[84784]: pgmap v1156: 99 pgs: 99 active+clean; 131 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 992 op/s 2024-09-21T10:26:19.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:19 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1011931945' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:26:19.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:19.473 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:19.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:19.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:19.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:19.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:19 smithi098 ceph-mon[97580]: pgmap v1156: 99 pgs: 99 active+clean; 131 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 992 op/s 2024-09-21T10:26:19.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:19 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1011931945' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:26:19.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:19 smithi098 ceph-mon[100914]: pgmap v1156: 99 pgs: 99 active+clean; 131 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 992 op/s 2024-09-21T10:26:19.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:19 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1011931945' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:26:19.598 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:19.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:19.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:19.695 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:26:19.700 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:19.760 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:19.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:19.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:19.862 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:19.886 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:19.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:19.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:19.947 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:26:19.988 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:20.009 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:20.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:20.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:20.132 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:20.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:20.233 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:20.255 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:20.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:20.356 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:20.377 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:20.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:20.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:26:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:26:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:26:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:20 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:20 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1011931945' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:26:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:20 smithi196 ceph-mon[84784]: osdmap e346: 8 total, 8 up, 7 in 2024-09-21T10:26:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:20 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:26:20.479 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:20.498 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:20 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:20.499 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:20 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1011931945' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:26:20.499 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:20 smithi098 ceph-mon[100914]: osdmap e346: 8 total, 8 up, 7 in 2024-09-21T10:26:20.499 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:20 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:26:20.499 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:20 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:20.499 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:20 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1011931945' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:26:20.499 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:20 smithi098 ceph-mon[97580]: osdmap e346: 8 total, 8 up, 7 in 2024-09-21T10:26:20.499 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:20 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:26:20.502 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:20.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:20.604 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:20.625 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:20.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:20.727 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:20.748 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:20.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:20.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:26:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:26:20] "GET /metrics HTTP/1.1" 200 36520 "" "Prometheus/2.51.0" 2024-09-21T10:26:20.849 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:20.871 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:20.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:20.972 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:20.993 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:20.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:21.095 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:21.116 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:21.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:21.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:21.218 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:21.239 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:21.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:21.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:21.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:21.362 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:21.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:21.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:21.454 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:21 smithi196 ceph-mon[84784]: pgmap v1158: 99 pgs: 99 active+clean; 14 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-21T10:26:21.464 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:21.485 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:21.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:21.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:21.547 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:21 smithi098 ceph-mon[97580]: pgmap v1158: 99 pgs: 99 active+clean; 14 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-21T10:26:21.547 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:21 smithi098 ceph-mon[100914]: pgmap v1158: 99 pgs: 99 active+clean; 14 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-21T10:26:21.586 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:21.607 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:21.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:21.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:21.709 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:21.730 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:21.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:21.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:21.832 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:21.855 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:21.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:21.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:21.866 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:26:21.867 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-21T10:26:21.867 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-21T10:26:21.867 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:26:21.867 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.867 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181808 9.c 9.d25df03c (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:26:21.867 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.277707+0000", 2024-09-21T10:26:21.867 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 98.555829571000004, 2024-09-21T10:26:21.867 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0082070830000001, 2024-09-21T10:26:21.867 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:21.867 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:26:21.867 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:21.867 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:21.867 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:21.868 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181808 2024-09-21T10:26:21.868 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.868 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:21.868 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.868 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:21.868 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277707+0000", 2024-09-21T10:26:21.868 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.868 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.868 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.868 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:21.868 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277707+0000", 2024-09-21T10:26:21.868 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.868 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.868 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.868 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:21.868 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277705+0000", 2024-09-21T10:26:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277710+0000", 2024-09-21T10:26:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9729999999999997e-06 2024-09-21T10:26:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277717+0000", 2024-09-21T10:26:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7460000000000003e-06 2024-09-21T10:26:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277722+0000", 2024-09-21T10:26:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4260000000000002e-06 2024-09-21T10:26:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277974+0000", 2024-09-21T10:26:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025160500000000001 2024-09-21T10:26:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277980+0000", 2024-09-21T10:26:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0730000000000001e-06 2024-09-21T10:26:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.285892+0000", 2024-09-21T10:26:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0079123059999999 2024-09-21T10:26:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.285914+0000", 2024-09-21T10:26:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1756999999999999e-05 2024-09-21T10:26:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181841 9.c 9.8718e16c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:26:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.307618+0000", 2024-09-21T10:26:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 98.525918179000001, 2024-09-21T10:26:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97832473499999995, 2024-09-21T10:26:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:26:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181841 2024-09-21T10:26:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307618+0000", 2024-09-21T10:26:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307618+0000", 2024-09-21T10:26:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307617+0000", 2024-09-21T10:26:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307625+0000", 2024-09-21T10:26:21.874 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.473e-06 2024-09-21T10:26:21.874 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.874 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.874 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:21.874 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307642+0000", 2024-09-21T10:26:21.874 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7408e-05 2024-09-21T10:26:21.874 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.874 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.874 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:21.874 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307648+0000", 2024-09-21T10:26:21.874 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6400000000000002e-06 2024-09-21T10:26:21.874 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.874 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.874 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307699+0000", 2024-09-21T10:26:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1359000000000001e-05 2024-09-21T10:26:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307707+0000", 2024-09-21T10:26:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.328e-06 2024-09-21T10:26:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.285935+0000", 2024-09-21T10:26:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97822819699999997 2024-09-21T10:26:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.285943+0000", 2024-09-21T10:26:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0290000000000005e-06 2024-09-21T10:26:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181856 9.c 9.1a6bfdac (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:26:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.326694+0000", 2024-09-21T10:26:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 98.506842726000002, 2024-09-21T10:26:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.95929115300000001, 2024-09-21T10:26:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:26:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181856 2024-09-21T10:26:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326694+0000", 2024-09-21T10:26:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326694+0000", 2024-09-21T10:26:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326692+0000", 2024-09-21T10:26:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326704+0000", 2024-09-21T10:26:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1864e-05 2024-09-21T10:26:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326720+0000", 2024-09-21T10:26:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5603999999999998e-05 2024-09-21T10:26:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326727+0000", 2024-09-21T10:26:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.374e-06 2024-09-21T10:26:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326780+0000", 2024-09-21T10:26:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2669e-05 2024-09-21T10:26:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326788+0000", 2024-09-21T10:26:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1280000000000008e-06 2024-09-21T10:26:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.285962+0000", 2024-09-21T10:26:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.95917384299999997 2024-09-21T10:26:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.285985+0000", 2024-09-21T10:26:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3300000000000001e-05 2024-09-21T10:26:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181866 9.c 9.d4fb720c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:26:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.337971+0000", 2024-09-21T10:26:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 98.495565690000006, 2024-09-21T10:26:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94809235000000003, 2024-09-21T10:26:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:26:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181866 2024-09-21T10:26:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337971+0000", 2024-09-21T10:26:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337971+0000", 2024-09-21T10:26:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337969+0000", 2024-09-21T10:26:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337976+0000", 2024-09-21T10:26:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0319999999999998e-06 2024-09-21T10:26:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337992+0000", 2024-09-21T10:26:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5277e-05 2024-09-21T10:26:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337999+0000", 2024-09-21T10:26:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1720000000000001e-06 2024-09-21T10:26:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.884 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.338041+0000", 2024-09-21T10:26:21.884 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1777999999999999e-05 2024-09-21T10:26:21.884 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.884 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.884 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:21.884 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.338046+0000", 2024-09-21T10:26:21.884 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6119999999999998e-06 2024-09-21T10:26:21.884 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.884 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.884 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.884 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.286056+0000", 2024-09-21T10:26:21.885 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94800968200000002 2024-09-21T10:26:21.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.885 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:21.885 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.286063+0000", 2024-09-21T10:26:21.885 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1940000000000001e-06 2024-09-21T10:26:21.885 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.885 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:21.885 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.885 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243459 9.c 9:3515c86a:::benchmark_data_smithi098_254407_object110728:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:26:21.885 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:52.997457+0000", 2024-09-21T10:26:21.885 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 28.836079529999999, 2024-09-21T10:26:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.328995454, 2024-09-21T10:26:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:26:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243459 2024-09-21T10:26:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997457+0000", 2024-09-21T10:26:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997457+0000", 2024-09-21T10:26:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997456+0000", 2024-09-21T10:26:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997461+0000", 2024-09-21T10:26:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1320000000000002e-06 2024-09-21T10:26:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997484+0000", 2024-09-21T10:26:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3192000000000001e-05 2024-09-21T10:26:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997494+0000", 2024-09-21T10:26:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0175e-05 2024-09-21T10:26:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003474+0000", 2024-09-21T10:26:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059798409999999996 2024-09-21T10:26:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003481+0000", 2024-09-21T10:26:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4039999999999998e-06 2024-09-21T10:26:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.326400+0000", 2024-09-21T10:26:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3229192140000001 2024-09-21T10:26:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.326452+0000", 2024-09-21T10:26:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1841000000000001e-05 2024-09-21T10:26:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243460 9.c 9:3d3ff595:::benchmark_data_smithi098_254407_object110729:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:26:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:52.997502+0000", 2024-09-21T10:26:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 28.836034292000001, 2024-09-21T10:26:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3586443749999999, 2024-09-21T10:26:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:26:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243460 2024-09-21T10:26:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997502+0000", 2024-09-21T10:26:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997502+0000", 2024-09-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997502+0000", 2024-09-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997505+0000", 2024-09-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1889999999999999e-06 2024-09-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997512+0000", 2024-09-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8360000000000003e-06 2024-09-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997516+0000", 2024-09-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3100000000000002e-06 2024-09-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003820+0000", 2024-09-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0063037989999999997 2024-09-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003825+0000", 2024-09-21T10:26:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3410000000000002e-06 2024-09-21T10:26:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.326483+0000", 2024-09-21T10:26:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.322658127 2024-09-21T10:26:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for missing object", 2024-09-21T10:26:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.326563+0000", 2024-09-21T10:26:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0270000000000002e-05 2024-09-21T10:26:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356089+0000", 2024-09-21T10:26:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.029525136 2024-09-21T10:26:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356146+0000", 2024-09-21T10:26:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7917000000000001e-05 2024-09-21T10:26:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243465 9.c 9:36d01026:::benchmark_data_smithi098_254407_object110734:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:52.997520+0000", 2024-09-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 28.836016052000002, 2024-09-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.359137246, 2024-09-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243465 2024-09-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997520+0000", 2024-09-21T10:26:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997520+0000", 2024-09-21T10:26:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997520+0000", 2024-09-21T10:26:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:26:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997522+0000", 2024-09-21T10:26:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2139999999999999e-06 2024-09-21T10:26:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997528+0000", 2024-09-21T10:26:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.049e-06 2024-09-21T10:26:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997533+0000", 2024-09-21T10:26:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.578e-06 2024-09-21T10:26:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003841+0000", 2024-09-21T10:26:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0063081760000000004 2024-09-21T10:26:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003845+0000", 2024-09-21T10:26:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2760000000000002e-06 2024-09-21T10:26:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.326590+0000", 2024-09-21T10:26:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3227443409999999 2024-09-21T10:26:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.326677+0000", 2024-09-21T10:26:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7176999999999996e-05 2024-09-21T10:26:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:26:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.327203+0000", 2024-09-21T10:26:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00052658000000000002 2024-09-21T10:26:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.337718+0000", 2024-09-21T10:26:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.01051439 2024-09-21T10:26:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356287+0000", 2024-09-21T10:26:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018568972999999999 2024-09-21T10:26:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356519+0000", 2024-09-21T10:26:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000231921 2024-09-21T10:26:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356565+0000", 2024-09-21T10:26:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5957000000000001e-05 2024-09-21T10:26:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356658+0000", 2024-09-21T10:26:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3066999999999995e-05 2024-09-21T10:26:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243471 9.c 9:3e8385fd:::benchmark_data_smithi098_254407_object110740:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:26:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:52.997537+0000", 2024-09-21T10:26:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 28.835999224999998, 2024-09-21T10:26:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3592854779999999, 2024-09-21T10:26:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243471 2024-09-21T10:26:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997537+0000", 2024-09-21T10:26:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997537+0000", 2024-09-21T10:26:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997537+0000", 2024-09-21T10:26:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:26:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997539+0000", 2024-09-21T10:26:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3750000000000001e-06 2024-09-21T10:26:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997548+0000", 2024-09-21T10:26:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.02e-06 2024-09-21T10:26:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997552+0000", 2024-09-21T10:26:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9380000000000002e-06 2024-09-21T10:26:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003861+0000", 2024-09-21T10:26:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0063086239999999997 2024-09-21T10:26:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003865+0000", 2024-09-21T10:26:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1239999999999998e-06 2024-09-21T10:26:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.332121+0000", 2024-09-21T10:26:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.328256586 2024-09-21T10:26:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.332255+0000", 2024-09-21T10:26:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013350999999999999 2024-09-21T10:26:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:26:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.332401+0000", 2024-09-21T10:26:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000146465 2024-09-21T10:26:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.341719+0000", 2024-09-21T10:26:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0093176160000000008 2024-09-21T10:26:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356328+0000", 2024-09-21T10:26:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014608612 2024-09-21T10:26:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356695+0000", 2024-09-21T10:26:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00036710999999999997 2024-09-21T10:26:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356736+0000", 2024-09-21T10:26:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1112000000000002e-05 2024-09-21T10:26:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356823+0000", 2024-09-21T10:26:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6779000000000004e-05 2024-09-21T10:26:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243517 9.c 9:3e65f1ad:::benchmark_data_smithi098_254407_object110786:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:26:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.050602+0000", 2024-09-21T10:26:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 28.782934770000001, 2024-09-21T10:26:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.306614151, 2024-09-21T10:26:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243517 2024-09-21T10:26:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050602+0000", 2024-09-21T10:26:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050602+0000", 2024-09-21T10:26:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050599+0000", 2024-09-21T10:26:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999976 2024-09-21T10:26:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050607+0000", 2024-09-21T10:26:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0690000000000002e-06 2024-09-21T10:26:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050627+0000", 2024-09-21T10:26:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9422000000000001e-05 2024-09-21T10:26:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050634+0000", 2024-09-21T10:26:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1160000000000001e-06 2024-09-21T10:26:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050699+0000", 2024-09-21T10:26:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5191999999999995e-05 2024-09-21T10:26:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050709+0000", 2024-09-21T10:26:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3270000000000007e-06 2024-09-21T10:26:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.334376+0000", 2024-09-21T10:26:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.283667766 2024-09-21T10:26:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.334474+0000", 2024-09-21T10:26:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7481999999999999e-05 2024-09-21T10:26:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:26:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.334602+0000", 2024-09-21T10:26:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000128014 2024-09-21T10:26:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.341726+0000", 2024-09-21T10:26:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0071239490000000001 2024-09-21T10:26:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356357+0000", 2024-09-21T10:26:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014631183000000001 2024-09-21T10:26:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357084+0000", 2024-09-21T10:26:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00072674199999999995 2024-09-21T10:26:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357126+0000", 2024-09-21T10:26:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2732999999999999e-05 2024-09-21T10:26:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357216+0000", 2024-09-21T10:26:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9373e-05 2024-09-21T10:26:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243534 9.c 9:3cf10c44:::benchmark_data_smithi098_254407_object110803:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:26:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.063492+0000", 2024-09-21T10:26:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 28.77004414, 2024-09-21T10:26:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2940934849999999, 2024-09-21T10:26:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243534 2024-09-21T10:26:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063492+0000", 2024-09-21T10:26:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063492+0000", 2024-09-21T10:26:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063491+0000", 2024-09-21T10:26:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063498+0000", 2024-09-21T10:26:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9759999999999995e-06 2024-09-21T10:26:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063511+0000", 2024-09-21T10:26:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2238e-05 2024-09-21T10:26:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063516+0000", 2024-09-21T10:26:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6849999999999998e-06 2024-09-21T10:26:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063572+0000", 2024-09-21T10:26:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5648e-05 2024-09-21T10:26:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063579+0000", 2024-09-21T10:26:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2169999999999997e-06 2024-09-21T10:26:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.336131+0000", 2024-09-21T10:26:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.27255148 2024-09-21T10:26:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.336203+0000", 2024-09-21T10:26:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2214999999999999e-05 2024-09-21T10:26:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:26:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.336278+0000", 2024-09-21T10:26:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4629000000000007e-05 2024-09-21T10:26:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.343957+0000", 2024-09-21T10:26:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0076795850000000001 2024-09-21T10:26:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356395+0000", 2024-09-21T10:26:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012437325000000001 2024-09-21T10:26:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357453+0000", 2024-09-21T10:26:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001058743 2024-09-21T10:26:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357495+0000", 2024-09-21T10:26:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1498999999999997e-05 2024-09-21T10:26:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357586+0000", 2024-09-21T10:26:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1014999999999995e-05 2024-09-21T10:26:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243535 9.c 9:37d3df54:::benchmark_data_smithi098_254407_object110804:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:26:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.068354+0000", 2024-09-21T10:26:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 28.765182061000001, 2024-09-21T10:26:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2894149260000001, 2024-09-21T10:26:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243535 2024-09-21T10:26:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068354+0000", 2024-09-21T10:26:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068354+0000", 2024-09-21T10:26:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068352+0000", 2024-09-21T10:26:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068360+0000", 2024-09-21T10:26:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7209999999999994e-06 2024-09-21T10:26:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068377+0000", 2024-09-21T10:26:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7198e-05 2024-09-21T10:26:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068383+0000", 2024-09-21T10:26:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1160000000000004e-06 2024-09-21T10:26:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068465+0000", 2024-09-21T10:26:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1681999999999995e-05 2024-09-21T10:26:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068473+0000", 2024-09-21T10:26:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8310000000000001e-06 2024-09-21T10:26:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.337739+0000", 2024-09-21T10:26:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.269265758 2024-09-21T10:26:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.337850+0000", 2024-09-21T10:26:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000111327 2024-09-21T10:26:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:26:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.337980+0000", 2024-09-21T10:26:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000130149 2024-09-21T10:26:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.343962+0000", 2024-09-21T10:26:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059821479999999996 2024-09-21T10:26:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356421+0000", 2024-09-21T10:26:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012459328 2024-09-21T10:26:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357634+0000", 2024-09-21T10:26:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012120410000000001 2024-09-21T10:26:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357681+0000", 2024-09-21T10:26:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7024999999999998e-05 2024-09-21T10:26:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357769+0000", 2024-09-21T10:26:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8727999999999995e-05 2024-09-21T10:26:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243548 9.c 9:3c4fc234:::benchmark_data_smithi098_254407_object110817:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:26:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.078991+0000", 2024-09-21T10:26:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 28.754545488000002, 2024-09-21T10:26:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.278948075, 2024-09-21T10:26:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243548 2024-09-21T10:26:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.078991+0000", 2024-09-21T10:26:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.078991+0000", 2024-09-21T10:26:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.078989+0000", 2024-09-21T10:26:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.078996+0000", 2024-09-21T10:26:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.968e-06 2024-09-21T10:26:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.079013+0000", 2024-09-21T10:26:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6549e-05 2024-09-21T10:26:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.079019+0000", 2024-09-21T10:26:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6729999999999997e-06 2024-09-21T10:26:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.079063+0000", 2024-09-21T10:26:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4376000000000003e-05 2024-09-21T10:26:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.079070+0000", 2024-09-21T10:26:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6089999999999997e-06 2024-09-21T10:26:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.339499+0000", 2024-09-21T10:26:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.260429467 2024-09-21T10:26:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.339588+0000", 2024-09-21T10:26:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8708000000000005e-05 2024-09-21T10:26:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:26:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.339668+0000", 2024-09-21T10:26:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0549000000000004e-05 2024-09-21T10:26:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.350422+0000", 2024-09-21T10:26:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010753257 2024-09-21T10:26:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356457+0000", 2024-09-21T10:26:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0060354479999999997 2024-09-21T10:26:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357809+0000", 2024-09-21T10:26:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0013525049999999999 2024-09-21T10:26:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357848+0000", 2024-09-21T10:26:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8537000000000001e-05 2024-09-21T10:26:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357939+0000", 2024-09-21T10:26:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1001000000000005e-05 2024-09-21T10:26:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243570 9.c 9:342610f6:::benchmark_data_smithi098_254407_object110839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:26:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.101422+0000", 2024-09-21T10:26:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 28.732114498000001, 2024-09-21T10:26:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2566680909999999, 2024-09-21T10:26:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243570 2024-09-21T10:26:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101422+0000", 2024-09-21T10:26:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101422+0000", 2024-09-21T10:26:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101420+0000", 2024-09-21T10:26:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101428+0000", 2024-09-21T10:26:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8420000000000005e-06 2024-09-21T10:26:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:21.939 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101446+0000", 2024-09-21T10:26:21.939 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8026e-05 2024-09-21T10:26:21.939 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.939 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.939 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:21.939 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101452+0000", 2024-09-21T10:26:21.939 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4910000000000002e-06 2024-09-21T10:26:21.939 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.939 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.939 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.939 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101565+0000", 2024-09-21T10:26:21.939 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011296 2024-09-21T10:26:21.939 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.939 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.940 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:21.940 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101587+0000", 2024-09-21T10:26:21.940 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2072000000000001e-05 2024-09-21T10:26:21.940 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.940 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.940 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.940 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.341753+0000", 2024-09-21T10:26:21.940 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2401660510000001 2024-09-21T10:26:21.940 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.940 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.940 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:21.940 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.341857+0000", 2024-09-21T10:26:21.940 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000103174 2024-09-21T10:26:21.940 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:26:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.341986+0000", 2024-09-21T10:26:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012943199999999999 2024-09-21T10:26:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.355716+0000", 2024-09-21T10:26:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013729961000000001 2024-09-21T10:26:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356484+0000", 2024-09-21T10:26:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00076824599999999999 2024-09-21T10:26:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357972+0000", 2024-09-21T10:26:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014878319999999999 2024-09-21T10:26:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.358008+0000", 2024-09-21T10:26:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6263999999999997e-05 2024-09-21T10:26:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.358090+0000", 2024-09-21T10:26:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1607999999999995e-05 2024-09-21T10:26:21.943 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.943 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:21.943 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.943 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.943 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243592 9.c 9:3d6001ff:::benchmark_data_smithi098_254407_object110861:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:26:21.943 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.125239+0000", 2024-09-21T10:26:21.943 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 28.708297633000001, 2024-09-21T10:26:21.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.242184476, 2024-09-21T10:26:21.943 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:21.943 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:21.943 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:21.943 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:21.943 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:21.944 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243592 2024-09-21T10:26:21.944 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.944 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:21.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:21.944 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125239+0000", 2024-09-21T10:26:21.944 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.944 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:21.944 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125239+0000", 2024-09-21T10:26:21.944 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.944 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:21.945 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125237+0000", 2024-09-21T10:26:21.945 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:21.945 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.945 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.945 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:21.945 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125244+0000", 2024-09-21T10:26:21.945 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9600000000000003e-06 2024-09-21T10:26:21.945 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.945 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.945 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:21.945 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125256+0000", 2024-09-21T10:26:21.945 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1759e-05 2024-09-21T10:26:21.945 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.945 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.945 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125262+0000", 2024-09-21T10:26:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5139999999999998e-06 2024-09-21T10:26:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125306+0000", 2024-09-21T10:26:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3724999999999999e-05 2024-09-21T10:26:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125316+0000", 2024-09-21T10:26:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6500000000000008e-06 2024-09-21T10:26:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.344004+0000", 2024-09-21T10:26:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2186879720000001 2024-09-21T10:26:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.344092+0000", 2024-09-21T10:26:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8027000000000003e-05 2024-09-21T10:26:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:26:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.344168+0000", 2024-09-21T10:26:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6508999999999995e-05 2024-09-21T10:26:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.355720+0000", 2024-09-21T10:26:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011551532 2024-09-21T10:26:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356879+0000", 2024-09-21T10:26:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001159642 2024-09-21T10:26:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367261+0000", 2024-09-21T10:26:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010381081 2024-09-21T10:26:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367319+0000", 2024-09-21T10:26:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8953999999999998e-05 2024-09-21T10:26:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367423+0000", 2024-09-21T10:26:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000103762 2024-09-21T10:26:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243598 9.c 9:30869b3f:::benchmark_data_smithi098_254407_object110867:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:26:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.132539+0000", 2024-09-21T10:26:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 28.700997418, 2024-09-21T10:26:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2350737359999999, 2024-09-21T10:26:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243598 2024-09-21T10:26:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132539+0000", 2024-09-21T10:26:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132539+0000", 2024-09-21T10:26:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132537+0000", 2024-09-21T10:26:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132545+0000", 2024-09-21T10:26:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4660000000000002e-06 2024-09-21T10:26:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132564+0000", 2024-09-21T10:26:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8981999999999999e-05 2024-09-21T10:26:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132570+0000", 2024-09-21T10:26:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3400000000000003e-06 2024-09-21T10:26:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132754+0000", 2024-09-21T10:26:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000183821 2024-09-21T10:26:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132782+0000", 2024-09-21T10:26:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8096e-05 2024-09-21T10:26:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.345661+0000", 2024-09-21T10:26:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2128794350000001 2024-09-21T10:26:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.345758+0000", 2024-09-21T10:26:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6637000000000003e-05 2024-09-21T10:26:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:26:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.345832+0000", 2024-09-21T10:26:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3719000000000003e-05 2024-09-21T10:26:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.355724+0000", 2024-09-21T10:26:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0098921329999999991 2024-09-21T10:26:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356929+0000", 2024-09-21T10:26:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012046579999999999 2024-09-21T10:26:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367459+0000", 2024-09-21T10:26:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010530358 2024-09-21T10:26:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367509+0000", 2024-09-21T10:26:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0309999999999998e-05 2024-09-21T10:26:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367613+0000", 2024-09-21T10:26:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000103531 2024-09-21T10:26:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243614 9.c 9:3cd37905:::benchmark_data_smithi098_254407_object110883:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:26:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.149686+0000", 2024-09-21T10:26:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 28.683850800999998, 2024-09-21T10:26:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2181167049999999, 2024-09-21T10:26:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243614 2024-09-21T10:26:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149686+0000", 2024-09-21T10:26:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149686+0000", 2024-09-21T10:26:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149684+0000", 2024-09-21T10:26:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149691+0000", 2024-09-21T10:26:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1439999999999997e-06 2024-09-21T10:26:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149711+0000", 2024-09-21T10:26:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0307999999999999e-05 2024-09-21T10:26:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149723+0000", 2024-09-21T10:26:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1687e-05 2024-09-21T10:26:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149834+0000", 2024-09-21T10:26:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000110895 2024-09-21T10:26:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149870+0000", 2024-09-21T10:26:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.591e-05 2024-09-21T10:26:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.347247+0000", 2024-09-21T10:26:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1973775120000001 2024-09-21T10:26:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.347335+0000", 2024-09-21T10:26:21.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7427000000000002e-05 2024-09-21T10:26:21.961 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.961 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:26:21.961 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.347412+0000", 2024-09-21T10:26:21.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:21.962 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6972999999999996e-05 2024-09-21T10:26:21.962 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.962 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.962 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:21.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.355727+0000", 2024-09-21T10:26:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0083153749999999998 2024-09-21T10:26:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356967+0000", 2024-09-21T10:26:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001240285 2024-09-21T10:26:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367650+0000", 2024-09-21T10:26:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010682574 2024-09-21T10:26:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367693+0000", 2024-09-21T10:26:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3123e-05 2024-09-21T10:26:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367802+0000", 2024-09-21T10:26:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109219 2024-09-21T10:26:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243648 9.c 9:33fdd93d:::benchmark_data_smithi098_254407_object110917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:26:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.196470+0000", 2024-09-21T10:26:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 28.637066105999999, 2024-09-21T10:26:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1715113070000001, 2024-09-21T10:26:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243648 2024-09-21T10:26:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196470+0000", 2024-09-21T10:26:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196470+0000", 2024-09-21T10:26:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196469+0000", 2024-09-21T10:26:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196476+0000", 2024-09-21T10:26:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7000000000000008e-06 2024-09-21T10:26:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196492+0000", 2024-09-21T10:26:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5877000000000001e-05 2024-09-21T10:26:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196499+0000", 2024-09-21T10:26:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1550000000000001e-06 2024-09-21T10:26:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196545+0000", 2024-09-21T10:26:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5679000000000001e-05 2024-09-21T10:26:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196560+0000", 2024-09-21T10:26:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4919000000000001e-05 2024-09-21T10:26:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.348867+0000", 2024-09-21T10:26:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1523073770000001 2024-09-21T10:26:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.348947+0000", 2024-09-21T10:26:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9827999999999995e-05 2024-09-21T10:26:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:26:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.349016+0000", 2024-09-21T10:26:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9091000000000006e-05 2024-09-21T10:26:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.355766+0000", 2024-09-21T10:26:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0067494419999999996 2024-09-21T10:26:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357002+0000", 2024-09-21T10:26:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012364780000000001 2024-09-21T10:26:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367840+0000", 2024-09-21T10:26:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010837704 2024-09-21T10:26:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367884+0000", 2024-09-21T10:26:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4353e-05 2024-09-21T10:26:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367982+0000", 2024-09-21T10:26:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7347000000000002e-05 2024-09-21T10:26:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243665 9.c 9:3b9d021e:::benchmark_data_smithi098_254407_object110934:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:26:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.227151+0000", 2024-09-21T10:26:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 28.606384949999999, 2024-09-21T10:26:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.141037334, 2024-09-21T10:26:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243665 2024-09-21T10:26:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227151+0000", 2024-09-21T10:26:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227151+0000", 2024-09-21T10:26:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227150+0000", 2024-09-21T10:26:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227157+0000", 2024-09-21T10:26:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3030000000000002e-06 2024-09-21T10:26:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227169+0000", 2024-09-21T10:26:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1999e-05 2024-09-21T10:26:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227174+0000", 2024-09-21T10:26:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4580000000000001e-06 2024-09-21T10:26:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227202+0000", 2024-09-21T10:26:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8002999999999999e-05 2024-09-21T10:26:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227208+0000", 2024-09-21T10:26:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4099999999999999e-06 2024-09-21T10:26:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.350461+0000", 2024-09-21T10:26:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.123253126 2024-09-21T10:26:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.350554+0000", 2024-09-21T10:26:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2917000000000005e-05 2024-09-21T10:26:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:26:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.350643+0000", 2024-09-21T10:26:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9167999999999997e-05 2024-09-21T10:26:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.355771+0000", 2024-09-21T10:26:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051275189999999997 2024-09-21T10:26:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357271+0000", 2024-09-21T10:26:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015008459999999999 2024-09-21T10:26:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.368035+0000", 2024-09-21T10:26:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010763679 2024-09-21T10:26:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.368089+0000", 2024-09-21T10:26:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3399000000000002e-05 2024-09-21T10:26:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.368189+0000", 2024-09-21T10:26:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100296 2024-09-21T10:26:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243683 9.c 9:3b325d23:::benchmark_data_smithi098_254407_object110952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:26:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.260563+0000", 2024-09-21T10:26:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 28.572973883, 2024-09-21T10:26:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1154280519999999, 2024-09-21T10:26:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243683 2024-09-21T10:26:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260563+0000", 2024-09-21T10:26:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260563+0000", 2024-09-21T10:26:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260561+0000", 2024-09-21T10:26:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:26:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260567+0000", 2024-09-21T10:26:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6849999999999998e-06 2024-09-21T10:26:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260577+0000", 2024-09-21T10:26:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0217000000000001e-05 2024-09-21T10:26:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260582+0000", 2024-09-21T10:26:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3880000000000002e-06 2024-09-21T10:26:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260623+0000", 2024-09-21T10:26:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0915999999999999e-05 2024-09-21T10:26:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260630+0000", 2024-09-21T10:26:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5730000000000002e-06 2024-09-21T10:26:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.352085+0000", 2024-09-21T10:26:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0914547569999999 2024-09-21T10:26:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.352173+0000", 2024-09-21T10:26:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7819999999999996e-05 2024-09-21T10:26:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:26:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.352293+0000", 2024-09-21T10:26:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000120475 2024-09-21T10:26:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.355774+0000", 2024-09-21T10:26:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003481137 2024-09-21T10:26:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357314+0000", 2024-09-21T10:26:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001539285 2024-09-21T10:26:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.375813+0000", 2024-09-21T10:26:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018498905 2024-09-21T10:26:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.375865+0000", 2024-09-21T10:26:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2945000000000002e-05 2024-09-21T10:26:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.375991+0000", 2024-09-21T10:26:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012511 2024-09-21T10:26:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243684 9.c 9:3d7d9148:::benchmark_data_smithi098_254407_object110953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:26:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.263461+0000", 2024-09-21T10:26:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 28.570075714000001, 2024-09-21T10:26:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.115013201, 2024-09-21T10:26:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243684 2024-09-21T10:26:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263461+0000", 2024-09-21T10:26:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263461+0000", 2024-09-21T10:26:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263460+0000", 2024-09-21T10:26:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:26:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263465+0000", 2024-09-21T10:26:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6280000000000004e-06 2024-09-21T10:26:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263473+0000", 2024-09-21T10:26:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5290000000000008e-06 2024-09-21T10:26:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263477+0000", 2024-09-21T10:26:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9500000000000003e-06 2024-09-21T10:26:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263505+0000", 2024-09-21T10:26:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7491000000000002e-05 2024-09-21T10:26:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263510+0000", 2024-09-21T10:26:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4090000000000002e-06 2024-09-21T10:26:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.353714+0000", 2024-09-21T10:26:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0902042750000001 2024-09-21T10:26:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.353823+0000", 2024-09-21T10:26:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108884 2024-09-21T10:26:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:26:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.353950+0000", 2024-09-21T10:26:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012683599999999999 2024-09-21T10:26:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356144+0000", 2024-09-21T10:26:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021941030000000002 2024-09-21T10:26:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357352+0000", 2024-09-21T10:26:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012075149999999999 2024-09-21T10:26:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.378332+0000", 2024-09-21T10:26:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.020980784999999998 2024-09-21T10:26:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.991 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:21.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.378381+0000", 2024-09-21T10:26:21.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8603000000000002e-05 2024-09-21T10:26:21.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:21.991 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:21.991 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:21.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.378474+0000", 2024-09-21T10:26:21.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2977000000000001e-05 2024-09-21T10:26:21.991 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.991 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:21.991 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.991 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:21.991 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:21.991 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:26:22.013 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:22.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:22.115 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:22.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.137 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:22.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:22.238 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:22.259 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:22.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:22.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:22.383 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:22.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:22.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:22.507 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:22.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:22.609 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:22.630 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:22.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:22.698 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:26:22.709 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:22.713 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:22.732 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:22.756 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:22.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:22.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:22.878 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:26:22.879 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:26:22.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.881 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:22.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:22.982 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:23.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.004 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:23.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:23.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:23.127 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:23.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:23.203 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:26:23.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:23.249 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:23.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:23.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:23 smithi196 ceph-mon[84784]: pgmap v1159: 99 pgs: 99 active+clean; 4.6 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 823 op/s 2024-09-21T10:26:23.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:23 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:23.350 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:23.372 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:23.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:23.472 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:23 smithi098 ceph-mon[97580]: pgmap v1159: 99 pgs: 99 active+clean; 4.6 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 823 op/s 2024-09-21T10:26:23.472 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:23 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:23.472 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:23 smithi098 ceph-mon[100914]: pgmap v1159: 99 pgs: 99 active+clean; 4.6 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 823 op/s 2024-09-21T10:26:23.472 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:23 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:23.474 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:23.494 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:23.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:23.596 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:23.617 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:23.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:23.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:23.740 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:23.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:23.804 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1159,"stamp":"2024-09-21T10:26:21.526358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5983,"num_read_kb":13006,"num_write":266365,"num_write_kb":8507216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10829,"num_bytes_recovered":702883762,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113298,"ondisk_log_size":113298,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":299,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":5589692,"kb_used_data":129920,"kb_used_omap":30,"kb_used_meta":5459681,"kb_avail":650626372,"statfs":{"total":671965249536,"available":666241404928,"internally_reserved":0,"allocated":133038080,"data_stored":119753731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30993,"internal_metadata":5590714095},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,4,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-431751192,"num_objects":-6589,"num_object_clones":0,"num_object_copies":-19767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6589,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001669"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":82,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.066763+0000","last_change":"2024-09-21T10:25:39.893215+0000","last_active":"2024-09-21T10:26:05.066763+0000","last_peered":"2024-09-21T10:26:05.066763+0000","last_clean":"2024-09-21T10:26:05.066763+0000","last_became_active":"2024-09-21T10:25:39.892930+0000","last_became_peered":"2024-09-21T10:25:39.892930+0000","last_unstale":"2024-09-21T10:26:05.066763+0000","last_undegraded":"2024-09-21T10:26:05.066763+0000","last_fullsized":"2024-09-21T10:26:05.066763+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:11:59.615233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":375,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.066823+0000","last_change":"2024-09-21T10:24:44.305198+0000","last_active":"2024-09-21T10:26:05.066823+0000","last_peered":"2024-09-21T10:26:05.066823+0000","last_clean":"2024-09-21T10:26:05.066823+0000","last_became_active":"2024-09-21T10:24:44.297653+0000","last_became_peered":"2024-09-21T10:24:44.297653+0000","last_unstale":"2024-09-21T10:26:05.066823+0000","last_undegraded":"2024-09-21T10:26:05.066823+0000","last_fullsized":"2024-09-21T10:26:05.066823+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:19.320833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":275,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.067001+0000","last_change":"2024-09-21T10:18:33.465357+0000","last_active":"2024-09-21T10:26:05.067001+0000","last_peered":"2024-09-21T10:26:05.067001+0000","last_clean":"2024-09-21T10:26:05.067001+0000","last_became_active":"2024-09-21T10:18:33.464952+0000","last_became_peered":"2024-09-21T10:18:33.464952+0000","last_unstale":"2024-09-21T10:26:05.067001+0000","last_undegraded":"2024-09-21T10:26:05.067001+0000","last_fullsized":"2024-09-21T10:26:05.067001+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:15:41.090866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":151,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.074808+0000","last_change":"2024-09-21T10:18:33.475291+0000","last_active":"2024-09-21T10:26:05.074808+0000","last_peered":"2024-09-21T10:26:05.074808+0000","last_clean":"2024-09-21T10:26:05.074808+0000","last_became_active":"2024-09-21T10:18:33.474971+0000","last_became_peered":"2024-09-21T10:18:33.474971+0000","last_unstale":"2024-09-21T10:26:05.074808+0000","last_undegraded":"2024-09-21T10:26:05.074808+0000","last_fullsized":"2024-09-21T10:26:05.074808+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:39:40.179668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":567,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.133079+0000","last_change":"2024-09-21T10:18:33.475179+0000","last_active":"2024-09-21T10:26:15.133079+0000","last_peered":"2024-09-21T10:26:15.133079+0000","last_clean":"2024-09-21T10:26:15.133079+0000","last_became_active":"2024-09-21T10:18:33.474914+0000","last_became_peered":"2024-09-21T10:18:33.474914+0000","last_unstale":"2024-09-21T10:26:15.133079+0000","last_undegraded":"2024-09-21T10:26:15.133079+0000","last_fullsized":"2024-09-21T10:26:15.133079+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:15:32.186986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":34,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.065150+0000","last_change":"2024-09-21T10:24:44.304399+0000","last_active":"2024-09-21T10:26:05.065150+0000","last_peered":"2024-09-21T10:26:05.065150+0000","last_clean":"2024-09-21T10:26:05.065150+0000","last_became_active":"2024-09-21T10:24:44.303437+0000","last_became_peered":"2024-09-21T10:24:44.303437+0000","last_unstale":"2024-09-21T10:26:05.065150+0000","last_undegraded":"2024-09-21T10:26:05.065150+0000","last_fullsized":"2024-09-21T10:26:05.065150+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:32:56.638481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":28,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.133043+0000","last_change":"2024-09-21T10:25:39.876792+0000","last_active":"2024-09-21T10:26:15.133043+0000","last_peered":"2024-09-21T10:26:15.133043+0000","last_clean":"2024-09-21T10:26:15.133043+0000","last_became_active":"2024-09-21T10:25:39.876336+0000","last_became_peered":"2024-09-21T10:25:39.876336+0000","last_unstale":"2024-09-21T10:26:15.133043+0000","last_undegraded":"2024-09-21T10:26:15.133043+0000","last_fullsized":"2024-09-21T10:26:15.133043+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:06:57.688437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":317,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.068694+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:26:05.068694+0000","last_peered":"2024-09-21T10:26:05.068694+0000","last_clean":"2024-09-21T10:26:05.068694+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:26:05.068694+0000","last_undegraded":"2024-09-21T10:26:05.068694+0000","last_fullsized":"2024-09-21T10:26:05.068694+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"345'16610","reported_seq":16712,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:17.071334+0000","last_change":"2024-09-21T10:22:19.108552+0000","last_active":"2024-09-21T10:26:17.071334+0000","last_peered":"2024-09-21T10:26:17.071334+0000","last_clean":"2024-09-21T10:26:17.071334+0000","last_became_active":"2024-09-21T10:22:19.107377+0000","last_became_peered":"2024-09-21T10:22:19.107377+0000","last_unstale":"2024-09-21T10:26:17.071334+0000","last_undegraded":"2024-09-21T10:26:17.071334+0000","last_fullsized":"2024-09-21T10:26:17.071334+0000","mapping_epoch":300,"log_start":"317'8901","ondisk_log_start":"317'8901","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":7709,"log_dups_size":0,"ondisk_log_size":7709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:39.289538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16610,"num_write_kb":531520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":366,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.066672+0000","last_change":"2024-09-21T10:24:44.325196+0000","last_active":"2024-09-21T10:26:05.066672+0000","last_peered":"2024-09-21T10:26:05.066672+0000","last_clean":"2024-09-21T10:26:05.066672+0000","last_became_active":"2024-09-21T10:24:44.299614+0000","last_became_peered":"2024-09-21T10:24:44.299614+0000","last_unstale":"2024-09-21T10:26:05.066672+0000","last_undegraded":"2024-09-21T10:26:05.066672+0000","last_fullsized":"2024-09-21T10:26:05.066672+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:14.558663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":208,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.066301+0000","last_change":"2024-09-21T10:07:43.794677+0000","last_active":"2024-09-21T10:26:05.066301+0000","last_peered":"2024-09-21T10:26:05.066301+0000","last_clean":"2024-09-21T10:26:05.066301+0000","last_became_active":"2024-09-21T10:07:43.794348+0000","last_became_peered":"2024-09-21T10:07:43.794348+0000","last_unstale":"2024-09-21T10:26:05.066301+0000","last_undegraded":"2024-09-21T10:26:05.066301+0000","last_fullsized":"2024-09-21T10:26:05.066301+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:11.159014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"345'16480","reported_seq":20052,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:17.081622+0000","last_change":"2024-09-21T10:25:56.050379+0000","last_active":"2024-09-21T10:26:17.081622+0000","last_peered":"2024-09-21T10:26:17.081622+0000","last_clean":"2024-09-21T10:26:17.081622+0000","last_became_active":"2024-09-21T10:25:56.010811+0000","last_became_peered":"2024-09-21T10:25:56.010811+0000","last_unstale":"2024-09-21T10:26:17.081622+0000","last_undegraded":"2024-09-21T10:26:17.081622+0000","last_fullsized":"2024-09-21T10:26:17.081622+0000","mapping_epoch":341,"log_start":"318'10200","ondisk_log_start":"318'10200","created":300,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":6280,"log_dups_size":0,"ondisk_log_size":6280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:47.437587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16480,"num_write_kb":527360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1723,"num_bytes_recovered":112918528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":367,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.067397+0000","last_change":"2024-09-21T10:24:44.323182+0000","last_active":"2024-09-21T10:26:05.067397+0000","last_peered":"2024-09-21T10:26:05.067397+0000","last_clean":"2024-09-21T10:26:05.067397+0000","last_became_active":"2024-09-21T10:24:44.299698+0000","last_became_peered":"2024-09-21T10:24:44.299698+0000","last_unstale":"2024-09-21T10:26:05.067397+0000","last_undegraded":"2024-09-21T10:26:05.067397+0000","last_fullsized":"2024-09-21T10:26:05.067397+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:56.571057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":34,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.065293+0000","last_change":"2024-09-21T10:24:44.304586+0000","last_active":"2024-09-21T10:26:05.065293+0000","last_peered":"2024-09-21T10:26:05.065293+0000","last_clean":"2024-09-21T10:26:05.065293+0000","last_became_active":"2024-09-21T10:24:44.303815+0000","last_became_peered":"2024-09-21T10:24:44.303815+0000","last_unstale":"2024-09-21T10:26:05.065293+0000","last_undegraded":"2024-09-21T10:26:05.065293+0000","last_fullsized":"2024-09-21T10:26:05.065293+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:02.126454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"345'16542","reported_seq":16595,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:17.063152+0000","last_change":"2024-09-21T10:22:19.107933+0000","last_active":"2024-09-21T10:26:17.063152+0000","last_peered":"2024-09-21T10:26:17.063152+0000","last_clean":"2024-09-21T10:26:17.063152+0000","last_became_active":"2024-09-21T10:22:19.107577+0000","last_became_peered":"2024-09-21T10:22:19.107577+0000","last_unstale":"2024-09-21T10:26:17.063152+0000","last_undegraded":"2024-09-21T10:26:17.063152+0000","last_fullsized":"2024-09-21T10:26:17.063152+0000","mapping_epoch":300,"log_start":"318'9800","ondisk_log_start":"318'9800","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":6742,"log_dups_size":0,"ondisk_log_size":6742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:39:34.494252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16542,"num_write_kb":529344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":293,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.067469+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:26:05.067469+0000","last_peered":"2024-09-21T10:26:05.067469+0000","last_clean":"2024-09-21T10:26:05.067469+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:26:05.067469+0000","last_undegraded":"2024-09-21T10:26:05.067469+0000","last_fullsized":"2024-09-21T10:26:05.067469+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":308,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.068978+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:26:05.068978+0000","last_peered":"2024-09-21T10:26:05.068978+0000","last_clean":"2024-09-21T10:26:05.068978+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:26:05.068978+0000","last_undegraded":"2024-09-21T10:26:05.068978+0000","last_fullsized":"2024-09-21T10:26:05.068978+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"345'16722","reported_seq":16824,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:17.068333+0000","last_change":"2024-09-21T10:22:19.106238+0000","last_active":"2024-09-21T10:26:17.068333+0000","last_peered":"2024-09-21T10:26:17.068333+0000","last_clean":"2024-09-21T10:26:17.068333+0000","last_became_active":"2024-09-21T10:22:19.105893+0000","last_became_peered":"2024-09-21T10:22:19.105893+0000","last_unstale":"2024-09-21T10:26:17.068333+0000","last_undegraded":"2024-09-21T10:26:17.068333+0000","last_fullsized":"2024-09-21T10:26:17.068333+0000","mapping_epoch":300,"log_start":"317'8300","ondisk_log_start":"317'8300","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":8422,"log_dups_size":0,"ondisk_log_size":8422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:36:41.147519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16722,"num_write_kb":535104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":143,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.069546+0000","last_change":"2024-09-21T10:18:33.483457+0000","last_active":"2024-09-21T10:26:05.069546+0000","last_peered":"2024-09-21T10:26:05.069546+0000","last_clean":"2024-09-21T10:26:05.069546+0000","last_became_active":"2024-09-21T10:18:33.483140+0000","last_became_peered":"2024-09-21T10:18:33.483140+0000","last_unstale":"2024-09-21T10:26:05.069546+0000","last_undegraded":"2024-09-21T10:26:05.069546+0000","last_fullsized":"2024-09-21T10:26:05.069546+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:37:39.302566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":596,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.133720+0000","last_change":"2024-09-21T10:24:44.299417+0000","last_active":"2024-09-21T10:26:15.133720+0000","last_peered":"2024-09-21T10:26:15.133720+0000","last_clean":"2024-09-21T10:26:15.133720+0000","last_became_active":"2024-09-21T10:24:44.299116+0000","last_became_peered":"2024-09-21T10:24:44.299116+0000","last_unstale":"2024-09-21T10:26:15.133720+0000","last_undegraded":"2024-09-21T10:26:15.133720+0000","last_fullsized":"2024-09-21T10:26:15.133720+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:44.524841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"82'100","reported_seq":456,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.067022+0000","last_change":"2024-09-21T10:23:22.728269+0000","last_active":"2024-09-21T10:26:05.067022+0000","last_peered":"2024-09-21T10:26:05.067022+0000","last_clean":"2024-09-21T10:26:05.067022+0000","last_became_active":"2024-09-21T10:23:20.700241+0000","last_became_peered":"2024-09-21T10:23:20.700241+0000","last_unstale":"2024-09-21T10:26:05.067022+0000","last_undegraded":"2024-09-21T10:26:05.067022+0000","last_fullsized":"2024-09-21T10:26:05.067022+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.728195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.728195+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:31.235642+0000","scrub_duration":0.0069698140000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"345'16666","reported_seq":25050,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:17.078184+0000","last_change":"2024-09-21T10:25:19.440652+0000","last_active":"2024-09-21T10:26:17.078184+0000","last_peered":"2024-09-21T10:26:17.078184+0000","last_clean":"2024-09-21T10:26:17.078184+0000","last_became_active":"2024-09-21T10:25:19.367270+0000","last_became_peered":"2024-09-21T10:25:19.367270+0000","last_unstale":"2024-09-21T10:26:17.078184+0000","last_undegraded":"2024-09-21T10:26:17.078184+0000","last_fullsized":"2024-09-21T10:26:17.078184+0000","mapping_epoch":330,"log_start":"318'10100","ondisk_log_start":"318'10100","created":300,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":6566,"log_dups_size":0,"ondisk_log_size":6566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:31.003034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16666,"num_write_kb":533312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4153,"num_bytes_recovered":272171008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":38,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.066485+0000","last_change":"2024-09-21T10:24:44.327042+0000","last_active":"2024-09-21T10:26:05.066485+0000","last_peered":"2024-09-21T10:26:05.066485+0000","last_clean":"2024-09-21T10:26:05.066485+0000","last_became_active":"2024-09-21T10:24:44.303785+0000","last_became_peered":"2024-09-21T10:24:44.303785+0000","last_unstale":"2024-09-21T10:26:05.066485+0000","last_undegraded":"2024-09-21T10:26:05.066485+0000","last_fullsized":"2024-09-21T10:26:05.066485+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:11:19.180549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":34,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.067826+0000","last_change":"2024-09-21T10:24:44.306721+0000","last_active":"2024-09-21T10:26:05.067826+0000","last_peered":"2024-09-21T10:26:05.067826+0000","last_clean":"2024-09-21T10:26:05.067826+0000","last_became_active":"2024-09-21T10:24:44.306427+0000","last_became_peered":"2024-09-21T10:24:44.306427+0000","last_unstale":"2024-09-21T10:26:05.067826+0000","last_undegraded":"2024-09-21T10:26:05.067826+0000","last_fullsized":"2024-09-21T10:26:05.067826+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:33:08.866485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"82'100","reported_seq":88,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.128823+0000","last_change":"2024-09-21T10:23:27.234299+0000","last_active":"2024-09-21T10:26:15.128823+0000","last_peered":"2024-09-21T10:26:15.128823+0000","last_clean":"2024-09-21T10:26:15.128823+0000","last_became_active":"2024-09-21T10:23:20.698155+0000","last_became_peered":"2024-09-21T10:23:20.698155+0000","last_unstale":"2024-09-21T10:26:15.128823+0000","last_undegraded":"2024-09-21T10:26:15.128823+0000","last_fullsized":"2024-09-21T10:26:15.128823+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:27.234001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:27.234001+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:13:43.690329+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"345'16466","reported_seq":16568,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:17.060801+0000","last_change":"2024-09-21T10:22:18.649410+0000","last_active":"2024-09-21T10:26:17.060801+0000","last_peered":"2024-09-21T10:26:17.060801+0000","last_clean":"2024-09-21T10:26:17.060801+0000","last_became_active":"2024-09-21T10:22:18.648626+0000","last_became_peered":"2024-09-21T10:22:18.648626+0000","last_unstale":"2024-09-21T10:26:17.060801+0000","last_undegraded":"2024-09-21T10:26:17.060801+0000","last_fullsized":"2024-09-21T10:26:17.060801+0000","mapping_epoch":300,"log_start":"318'10200","ondisk_log_start":"318'10200","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":6266,"log_dups_size":0,"ondisk_log_size":6266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:42:50.134303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16466,"num_write_kb":526912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":359,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.069103+0000","last_change":"2024-09-21T10:24:44.309470+0000","last_active":"2024-09-21T10:26:05.069103+0000","last_peered":"2024-09-21T10:26:05.069103+0000","last_clean":"2024-09-21T10:26:05.069103+0000","last_became_active":"2024-09-21T10:24:44.308836+0000","last_became_peered":"2024-09-21T10:24:44.308836+0000","last_unstale":"2024-09-21T10:26:05.069103+0000","last_undegraded":"2024-09-21T10:26:05.069103+0000","last_fullsized":"2024-09-21T10:26:05.069103+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:09:44.622418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":34,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.068504+0000","last_change":"2024-09-21T10:24:44.305135+0000","last_active":"2024-09-21T10:26:05.068504+0000","last_peered":"2024-09-21T10:26:05.068504+0000","last_clean":"2024-09-21T10:26:05.068504+0000","last_became_active":"2024-09-21T10:24:44.304839+0000","last_became_peered":"2024-09-21T10:24:44.304839+0000","last_unstale":"2024-09-21T10:26:05.068504+0000","last_undegraded":"2024-09-21T10:26:05.068504+0000","last_fullsized":"2024-09-21T10:26:05.068504+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:02.264404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"345'16754","reported_seq":16807,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:17.088357+0000","last_change":"2024-09-21T10:22:19.106259+0000","last_active":"2024-09-21T10:26:17.088357+0000","last_peered":"2024-09-21T10:26:17.088357+0000","last_clean":"2024-09-21T10:26:17.088357+0000","last_became_active":"2024-09-21T10:22:19.105931+0000","last_became_peered":"2024-09-21T10:22:19.105931+0000","last_unstale":"2024-09-21T10:26:17.088357+0000","last_undegraded":"2024-09-21T10:26:17.088357+0000","last_fullsized":"2024-09-21T10:26:17.088357+0000","mapping_epoch":300,"log_start":"318'10000","ondisk_log_start":"318'10000","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":6754,"log_dups_size":0,"ondisk_log_size":6754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:38:07.993075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16754,"num_write_kb":536128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":28,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.132797+0000","last_change":"2024-09-21T10:25:39.872680+0000","last_active":"2024-09-21T10:26:15.132797+0000","last_peered":"2024-09-21T10:26:15.132797+0000","last_clean":"2024-09-21T10:26:15.132797+0000","last_became_active":"2024-09-21T10:25:39.872312+0000","last_became_peered":"2024-09-21T10:25:39.872312+0000","last_unstale":"2024-09-21T10:26:15.132797+0000","last_undegraded":"2024-09-21T10:26:15.132797+0000","last_fullsized":"2024-09-21T10:26:15.132797+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:54.942046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":34,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.069957+0000","last_change":"2024-09-21T10:24:44.310484+0000","last_active":"2024-09-21T10:26:05.069957+0000","last_peered":"2024-09-21T10:26:05.069957+0000","last_clean":"2024-09-21T10:26:05.069957+0000","last_became_active":"2024-09-21T10:24:44.307209+0000","last_became_peered":"2024-09-21T10:24:44.307209+0000","last_unstale":"2024-09-21T10:26:05.069957+0000","last_undegraded":"2024-09-21T10:26:05.069957+0000","last_fullsized":"2024-09-21T10:26:05.069957+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:48:36.767539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'100","reported_seq":83,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.129282+0000","last_change":"2024-09-21T10:23:26.197358+0000","last_active":"2024-09-21T10:26:15.129282+0000","last_peered":"2024-09-21T10:26:15.129282+0000","last_clean":"2024-09-21T10:26:15.129282+0000","last_became_active":"2024-09-21T10:23:20.695174+0000","last_became_peered":"2024-09-21T10:23:20.695174+0000","last_unstale":"2024-09-21T10:26:15.129282+0000","last_undegraded":"2024-09-21T10:26:15.129282+0000","last_fullsized":"2024-09-21T10:26:15.129282+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:26.197291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:26.197291+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:33.961031+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2","version":"345'16930","reported_seq":17032,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:17.093229+0000","last_change":"2024-09-21T10:22:18.649478+0000","last_active":"2024-09-21T10:26:17.093229+0000","last_peered":"2024-09-21T10:26:17.093229+0000","last_clean":"2024-09-21T10:26:17.093229+0000","last_became_active":"2024-09-21T10:22:18.649167+0000","last_became_peered":"2024-09-21T10:22:18.649167+0000","last_unstale":"2024-09-21T10:26:17.093229+0000","last_undegraded":"2024-09-21T10:26:17.093229+0000","last_fullsized":"2024-09-21T10:26:17.093229+0000","mapping_epoch":300,"log_start":"319'10676","ondisk_log_start":"319'10676","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":6254,"log_dups_size":0,"ondisk_log_size":6254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:17.191131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16930,"num_write_kb":541760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":222,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.130960+0000","last_change":"2024-09-21T10:24:44.308818+0000","last_active":"2024-09-21T10:26:15.130960+0000","last_peered":"2024-09-21T10:26:15.130960+0000","last_clean":"2024-09-21T10:26:15.130960+0000","last_became_active":"2024-09-21T10:24:44.308014+0000","last_became_peered":"2024-09-21T10:24:44.308014+0000","last_unstale":"2024-09-21T10:26:15.130960+0000","last_undegraded":"2024-09-21T10:26:15.130960+0000","last_fullsized":"2024-09-21T10:26:15.130960+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:36:14.730099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":585,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.131784+0000","last_change":"2024-09-21T10:24:44.309294+0000","last_active":"2024-09-21T10:26:15.131784+0000","last_peered":"2024-09-21T10:26:15.131784+0000","last_clean":"2024-09-21T10:26:15.131784+0000","last_became_active":"2024-09-21T10:24:44.308928+0000","last_became_peered":"2024-09-21T10:24:44.308928+0000","last_unstale":"2024-09-21T10:26:15.131784+0000","last_undegraded":"2024-09-21T10:26:15.131784+0000","last_fullsized":"2024-09-21T10:26:15.131784+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:50:20.141648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"345'16612","reported_seq":16714,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:17.073052+0000","last_change":"2024-09-21T10:22:19.107370+0000","last_active":"2024-09-21T10:26:17.073052+0000","last_peered":"2024-09-21T10:26:17.073052+0000","last_clean":"2024-09-21T10:26:17.073052+0000","last_became_active":"2024-09-21T10:22:19.106979+0000","last_became_peered":"2024-09-21T10:22:19.106979+0000","last_unstale":"2024-09-21T10:26:17.073052+0000","last_undegraded":"2024-09-21T10:26:17.073052+0000","last_fullsized":"2024-09-21T10:26:17.073052+0000","mapping_epoch":300,"log_start":"317'8869","ondisk_log_start":"317'8869","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":7743,"log_dups_size":0,"ondisk_log_size":7743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:10.360218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":531584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":447,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.066518+0000","last_change":"2024-09-21T10:23:18.790100+0000","last_active":"2024-09-21T10:26:05.066518+0000","last_peered":"2024-09-21T10:26:05.066518+0000","last_clean":"2024-09-21T10:26:05.066518+0000","last_became_active":"2024-09-21T10:23:14.677610+0000","last_became_peered":"2024-09-21T10:23:14.677610+0000","last_unstale":"2024-09-21T10:26:05.066518+0000","last_undegraded":"2024-09-21T10:26:05.066518+0000","last_fullsized":"2024-09-21T10:26:05.066518+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:18.790017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:18.790017+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:56.562241+0000","scrub_duration":0.0080400790000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":858,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.129423+0000","last_change":"2024-09-21T10:07:43.790490+0000","last_active":"2024-09-21T10:26:15.129423+0000","last_peered":"2024-09-21T10:26:15.129423+0000","last_clean":"2024-09-21T10:26:15.129423+0000","last_became_active":"2024-09-21T10:07:43.789986+0000","last_became_peered":"2024-09-21T10:07:43.789986+0000","last_unstale":"2024-09-21T10:26:15.129423+0000","last_undegraded":"2024-09-21T10:26:15.129423+0000","last_fullsized":"2024-09-21T10:26:15.129423+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:08.740356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":481,"num_read_kb":1580,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":106,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.068905+0000","last_change":"2024-09-21T10:24:44.309030+0000","last_active":"2024-09-21T10:26:05.068905+0000","last_peered":"2024-09-21T10:26:05.068905+0000","last_clean":"2024-09-21T10:26:05.068905+0000","last_became_active":"2024-09-21T10:24:44.308388+0000","last_became_peered":"2024-09-21T10:24:44.308388+0000","last_unstale":"2024-09-21T10:26:05.068905+0000","last_undegraded":"2024-09-21T10:26:05.068905+0000","last_fullsized":"2024-09-21T10:26:05.068905+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:56.273653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":331,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.070493+0000","last_change":"2024-09-21T10:24:44.308008+0000","last_active":"2024-09-21T10:26:05.070493+0000","last_peered":"2024-09-21T10:26:05.070493+0000","last_clean":"2024-09-21T10:26:05.070493+0000","last_became_active":"2024-09-21T10:24:44.307595+0000","last_became_peered":"2024-09-21T10:24:44.307595+0000","last_unstale":"2024-09-21T10:26:05.070493+0000","last_undegraded":"2024-09-21T10:26:05.070493+0000","last_fullsized":"2024-09-21T10:26:05.070493+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:33:49.448676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"82'100","reported_seq":90,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.130941+0000","last_change":"2024-09-21T10:24:44.309154+0000","last_active":"2024-09-21T10:26:15.130941+0000","last_peered":"2024-09-21T10:26:15.130941+0000","last_clean":"2024-09-21T10:26:15.130941+0000","last_became_active":"2024-09-21T10:24:44.308330+0000","last_became_peered":"2024-09-21T10:24:44.308330+0000","last_unstale":"2024-09-21T10:26:15.130941+0000","last_undegraded":"2024-09-21T10:26:15.130941+0000","last_fullsized":"2024-09-21T10:26:15.130941+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.641718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.641718+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:10.158739+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"345'16492","reported_seq":16594,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:17.093940+0000","last_change":"2024-09-21T10:22:19.108211+0000","last_active":"2024-09-21T10:26:17.093940+0000","last_peered":"2024-09-21T10:26:17.093940+0000","last_clean":"2024-09-21T10:26:17.093940+0000","last_became_active":"2024-09-21T10:22:19.107919+0000","last_became_peered":"2024-09-21T10:22:19.107919+0000","last_unstale":"2024-09-21T10:26:17.093940+0000","last_undegraded":"2024-09-21T10:26:17.093940+0000","last_fullsized":"2024-09-21T10:26:17.093940+0000","mapping_epoch":300,"log_start":"317'8775","ondisk_log_start":"317'8775","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":7717,"log_dups_size":0,"ondisk_log_size":7717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:20.144889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16492,"num_write_kb":527744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":689,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.066380+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:26:05.066380+0000","last_peered":"2024-09-21T10:26:05.066380+0000","last_clean":"2024-09-21T10:26:05.066380+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:26:05.066380+0000","last_undegraded":"2024-09-21T10:26:05.066380+0000","last_fullsized":"2024-09-21T10:26:05.066380+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":597,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.133088+0000","last_change":"2024-09-21T10:24:44.299102+0000","last_active":"2024-09-21T10:26:15.133088+0000","last_peered":"2024-09-21T10:26:15.133088+0000","last_clean":"2024-09-21T10:26:15.133088+0000","last_became_active":"2024-09-21T10:24:44.298723+0000","last_became_peered":"2024-09-21T10:24:44.298723+0000","last_unstale":"2024-09-21T10:26:15.133088+0000","last_undegraded":"2024-09-21T10:26:15.133088+0000","last_fullsized":"2024-09-21T10:26:15.133088+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:57:47.078328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":573,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.133280+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:26:15.133280+0000","last_peered":"2024-09-21T10:26:15.133280+0000","last_clean":"2024-09-21T10:26:15.133280+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:26:15.133280+0000","last_undegraded":"2024-09-21T10:26:15.133280+0000","last_fullsized":"2024-09-21T10:26:15.133280+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"82'100","reported_seq":236,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.133373+0000","last_change":"2024-09-21T10:24:44.294460+0000","last_active":"2024-09-21T10:26:15.133373+0000","last_peered":"2024-09-21T10:26:15.133373+0000","last_clean":"2024-09-21T10:26:15.133373+0000","last_became_active":"2024-09-21T10:24:44.293972+0000","last_became_peered":"2024-09-21T10:24:44.293972+0000","last_unstale":"2024-09-21T10:26:15.133373+0000","last_undegraded":"2024-09-21T10:26:15.133373+0000","last_fullsized":"2024-09-21T10:26:15.133373+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:26.199522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:26.199522+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:26.135171+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"345'16458","reported_seq":16560,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:16.866454+0000","last_change":"2024-09-21T10:22:19.108927+0000","last_active":"2024-09-21T10:26:16.866454+0000","last_peered":"2024-09-21T10:26:16.866454+0000","last_clean":"2024-09-21T10:26:16.866454+0000","last_became_active":"2024-09-21T10:22:19.108649+0000","last_became_peered":"2024-09-21T10:22:19.108649+0000","last_unstale":"2024-09-21T10:26:16.866454+0000","last_undegraded":"2024-09-21T10:26:16.866454+0000","last_fullsized":"2024-09-21T10:26:16.866454+0000","mapping_epoch":300,"log_start":"318'10200","ondisk_log_start":"318'10200","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":6258,"log_dups_size":0,"ondisk_log_size":6258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:02:10.256858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16458,"num_write_kb":526656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":1346,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.129218+0000","last_change":"2024-09-21T10:18:36.495308+0000","last_active":"2024-09-21T10:26:15.129218+0000","last_peered":"2024-09-21T10:26:15.129218+0000","last_clean":"2024-09-21T10:26:15.129218+0000","last_became_active":"2024-09-21T10:18:36.494662+0000","last_became_peered":"2024-09-21T10:18:36.494662+0000","last_unstale":"2024-09-21T10:26:15.129218+0000","last_undegraded":"2024-09-21T10:26:15.129218+0000","last_fullsized":"2024-09-21T10:26:15.129218+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:49.218676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":444,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.131042+0000","last_change":"2024-09-21T10:08:18.470748+0000","last_active":"2024-09-21T10:26:15.131042+0000","last_peered":"2024-09-21T10:26:15.131042+0000","last_clean":"2024-09-21T10:26:15.131042+0000","last_became_active":"2024-09-21T10:08:18.470227+0000","last_became_peered":"2024-09-21T10:08:18.470227+0000","last_unstale":"2024-09-21T10:26:15.131042+0000","last_undegraded":"2024-09-21T10:26:15.131042+0000","last_fullsized":"2024-09-21T10:26:15.131042+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:16.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":567,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.131417+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:26:15.131417+0000","last_peered":"2024-09-21T10:26:15.131417+0000","last_clean":"2024-09-21T10:26:15.131417+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:26:15.131417+0000","last_undegraded":"2024-09-21T10:26:15.131417+0000","last_fullsized":"2024-09-21T10:26:15.131417+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"82'100","reported_seq":229,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.132903+0000","last_change":"2024-09-21T10:23:21.277739+0000","last_active":"2024-09-21T10:26:15.132903+0000","last_peered":"2024-09-21T10:26:15.132903+0000","last_clean":"2024-09-21T10:26:15.132903+0000","last_became_active":"2024-09-21T10:23:20.689733+0000","last_became_peered":"2024-09-21T10:23:20.689733+0000","last_unstale":"2024-09-21T10:26:15.132903+0000","last_undegraded":"2024-09-21T10:26:15.132903+0000","last_fullsized":"2024-09-21T10:26:15.132903+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.277663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.277663+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:01:54.523189+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"345'16496","reported_seq":16598,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:17.067145+0000","last_change":"2024-09-21T10:22:19.106312+0000","last_active":"2024-09-21T10:26:17.067145+0000","last_peered":"2024-09-21T10:26:17.067145+0000","last_clean":"2024-09-21T10:26:17.067145+0000","last_became_active":"2024-09-21T10:22:19.106002+0000","last_became_peered":"2024-09-21T10:22:19.106002+0000","last_unstale":"2024-09-21T10:26:17.067145+0000","last_undegraded":"2024-09-21T10:26:17.067145+0000","last_fullsized":"2024-09-21T10:26:17.067145+0000","mapping_epoch":300,"log_start":"317'8748","ondisk_log_start":"317'8748","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":7748,"log_dups_size":0,"ondisk_log_size":7748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:24.317806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16496,"num_write_kb":527872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":1043,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.132375+0000","last_change":"2024-09-21T10:07:43.794767+0000","last_active":"2024-09-21T10:26:15.132375+0000","last_peered":"2024-09-21T10:26:15.132375+0000","last_clean":"2024-09-21T10:26:15.132375+0000","last_became_active":"2024-09-21T10:07:43.794062+0000","last_became_peered":"2024-09-21T10:07:43.794062+0000","last_unstale":"2024-09-21T10:26:15.132375+0000","last_undegraded":"2024-09-21T10:26:15.132375+0000","last_fullsized":"2024-09-21T10:26:15.132375+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:51.977955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":34,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.066577+0000","last_change":"2024-09-21T10:24:44.299492+0000","last_active":"2024-09-21T10:26:05.066577+0000","last_peered":"2024-09-21T10:26:05.066577+0000","last_clean":"2024-09-21T10:26:05.066577+0000","last_became_active":"2024-09-21T10:24:44.299187+0000","last_became_peered":"2024-09-21T10:24:44.299187+0000","last_unstale":"2024-09-21T10:26:05.066577+0000","last_undegraded":"2024-09-21T10:26:05.066577+0000","last_fullsized":"2024-09-21T10:26:05.066577+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:58.500386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":490,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.129678+0000","last_change":"2024-09-21T10:18:33.465815+0000","last_active":"2024-09-21T10:26:15.129678+0000","last_peered":"2024-09-21T10:26:15.129678+0000","last_clean":"2024-09-21T10:26:15.129678+0000","last_became_active":"2024-09-21T10:18:33.465459+0000","last_became_peered":"2024-09-21T10:18:33.465459+0000","last_unstale":"2024-09-21T10:26:15.129678+0000","last_undegraded":"2024-09-21T10:26:15.129678+0000","last_fullsized":"2024-09-21T10:26:15.129678+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:22:08.199800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"82'100","reported_seq":199,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.069340+0000","last_change":"2024-09-21T10:23:30.570074+0000","last_active":"2024-09-21T10:26:05.069340+0000","last_peered":"2024-09-21T10:26:05.069340+0000","last_clean":"2024-09-21T10:26:05.069340+0000","last_became_active":"2024-09-21T10:23:20.695270+0000","last_became_peered":"2024-09-21T10:23:20.695270+0000","last_unstale":"2024-09-21T10:26:05.069340+0000","last_undegraded":"2024-09-21T10:26:05.069340+0000","last_fullsized":"2024-09-21T10:26:05.069340+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:30.569982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:30.569982+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:06.011343+0000","scrub_duration":0.0051309349999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"345'16724","reported_seq":16826,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:17.090481+0000","last_change":"2024-09-21T10:22:19.107714+0000","last_active":"2024-09-21T10:26:17.090481+0000","last_peered":"2024-09-21T10:26:17.090481+0000","last_clean":"2024-09-21T10:26:17.090481+0000","last_became_active":"2024-09-21T10:22:19.107130+0000","last_became_peered":"2024-09-21T10:22:19.107130+0000","last_unstale":"2024-09-21T10:26:17.090481+0000","last_undegraded":"2024-09-21T10:26:17.090481+0000","last_fullsized":"2024-09-21T10:26:17.090481+0000","mapping_epoch":300,"log_start":"318'10400","ondisk_log_start":"318'10400","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":6324,"log_dups_size":0,"ondisk_log_size":6324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:50:53.512892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16724,"num_write_kb":535168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":233,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.129010+0000","last_change":"2024-09-21T10:18:33.467659+0000","last_active":"2024-09-21T10:26:15.129010+0000","last_peered":"2024-09-21T10:26:15.129010+0000","last_clean":"2024-09-21T10:26:15.129010+0000","last_became_active":"2024-09-21T10:18:33.467369+0000","last_became_peered":"2024-09-21T10:18:33.467369+0000","last_unstale":"2024-09-21T10:26:15.129010+0000","last_undegraded":"2024-09-21T10:26:15.129010+0000","last_fullsized":"2024-09-21T10:26:15.129010+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:16:41.951755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":1150,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.129755+0000","last_change":"2024-09-21T10:08:18.468995+0000","last_active":"2024-09-21T10:26:15.129755+0000","last_peered":"2024-09-21T10:26:15.129755+0000","last_clean":"2024-09-21T10:26:15.129755+0000","last_became_active":"2024-09-21T10:08:18.468243+0000","last_became_peered":"2024-09-21T10:08:18.468243+0000","last_unstale":"2024-09-21T10:26:15.129755+0000","last_undegraded":"2024-09-21T10:26:15.129755+0000","last_fullsized":"2024-09-21T10:26:15.129755+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:45.520968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":476,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.132519+0000","last_change":"2024-09-21T10:24:44.309614+0000","last_active":"2024-09-21T10:26:15.132519+0000","last_peered":"2024-09-21T10:26:15.132519+0000","last_clean":"2024-09-21T10:26:15.132519+0000","last_became_active":"2024-09-21T10:24:44.309187+0000","last_became_peered":"2024-09-21T10:24:44.309187+0000","last_unstale":"2024-09-21T10:26:15.132519+0000","last_undegraded":"2024-09-21T10:26:15.132519+0000","last_fullsized":"2024-09-21T10:26:15.132519+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:01.074803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'100","reported_seq":209,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.069530+0000","last_change":"2024-09-21T10:24:44.308946+0000","last_active":"2024-09-21T10:26:05.069530+0000","last_peered":"2024-09-21T10:26:05.069530+0000","last_clean":"2024-09-21T10:26:05.069530+0000","last_became_active":"2024-09-21T10:24:44.308558+0000","last_became_peered":"2024-09-21T10:24:44.308558+0000","last_unstale":"2024-09-21T10:26:05.069530+0000","last_undegraded":"2024-09-21T10:26:05.069530+0000","last_fullsized":"2024-09-21T10:26:05.069530+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:24.536711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:24.536711+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:11:00.667553+0000","scrub_duration":0.0047525650000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"345'16482","reported_seq":22410,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:16.864513+0000","last_change":"2024-09-21T10:25:36.867110+0000","last_active":"2024-09-21T10:26:16.864513+0000","last_peered":"2024-09-21T10:26:16.864513+0000","last_clean":"2024-09-21T10:26:16.864513+0000","last_became_active":"2024-09-21T10:25:36.838970+0000","last_became_peered":"2024-09-21T10:25:36.838970+0000","last_unstale":"2024-09-21T10:26:16.864513+0000","last_undegraded":"2024-09-21T10:26:16.864513+0000","last_fullsized":"2024-09-21T10:26:16.864513+0000","mapping_epoch":334,"log_start":"318'9800","ondisk_log_start":"318'9800","created":300,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":6682,"log_dups_size":0,"ondisk_log_size":6682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:10:40.672440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16482,"num_write_kb":527424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2923,"num_bytes_recovered":191496192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":445,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.129674+0000","last_change":"2024-09-21T10:08:18.469907+0000","last_active":"2024-09-21T10:26:15.129674+0000","last_peered":"2024-09-21T10:26:15.129674+0000","last_clean":"2024-09-21T10:26:15.129674+0000","last_became_active":"2024-09-21T10:08:18.469443+0000","last_became_peered":"2024-09-21T10:08:18.469443+0000","last_unstale":"2024-09-21T10:26:15.129674+0000","last_undegraded":"2024-09-21T10:26:15.129674+0000","last_fullsized":"2024-09-21T10:26:15.129674+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:15.645803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":767,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.071065+0000","last_change":"2024-09-21T10:24:44.334847+0000","last_active":"2024-09-21T10:26:05.071065+0000","last_peered":"2024-09-21T10:26:05.071065+0000","last_clean":"2024-09-21T10:26:05.071065+0000","last_became_active":"2024-09-21T10:24:44.301321+0000","last_became_peered":"2024-09-21T10:24:44.301321+0000","last_unstale":"2024-09-21T10:26:05.071065+0000","last_undegraded":"2024-09-21T10:26:05.071065+0000","last_fullsized":"2024-09-21T10:26:05.071065+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:37.628940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":551,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.133642+0000","last_change":"2024-09-21T09:59:54.267798+0000","last_active":"2024-09-21T10:26:15.133642+0000","last_peered":"2024-09-21T10:26:15.133642+0000","last_clean":"2024-09-21T10:26:15.133642+0000","last_became_active":"2024-09-21T09:59:54.265615+0000","last_became_peered":"2024-09-21T09:59:54.265615+0000","last_unstale":"2024-09-21T10:26:15.133642+0000","last_undegraded":"2024-09-21T10:26:15.133642+0000","last_fullsized":"2024-09-21T10:26:15.133642+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:30.363872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"82'100","reported_seq":34,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.068381+0000","last_change":"2024-09-21T10:24:44.296758+0000","last_active":"2024-09-21T10:26:05.068381+0000","last_peered":"2024-09-21T10:26:05.068381+0000","last_clean":"2024-09-21T10:26:05.068381+0000","last_became_active":"2024-09-21T10:24:44.296403+0000","last_became_peered":"2024-09-21T10:24:44.296403+0000","last_unstale":"2024-09-21T10:26:05.068381+0000","last_undegraded":"2024-09-21T10:26:05.068381+0000","last_fullsized":"2024-09-21T10:26:05.068381+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.105746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.105746+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:20:12.502081+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"345'16318","reported_seq":16372,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:17.076068+0000","last_change":"2024-09-21T10:22:19.106548+0000","last_active":"2024-09-21T10:26:17.076068+0000","last_peered":"2024-09-21T10:26:17.076068+0000","last_clean":"2024-09-21T10:26:17.076068+0000","last_became_active":"2024-09-21T10:22:19.105920+0000","last_became_peered":"2024-09-21T10:22:19.105920+0000","last_unstale":"2024-09-21T10:26:17.076068+0000","last_undegraded":"2024-09-21T10:26:17.076068+0000","last_fullsized":"2024-09-21T10:26:17.076068+0000","mapping_epoch":300,"log_start":"318'9600","ondisk_log_start":"318'9600","created":300,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":6718,"log_dups_size":0,"ondisk_log_size":6718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:30.951700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16318,"num_write_kb":522113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":108,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.066572+0000","last_change":"2024-09-21T10:18:33.489678+0000","last_active":"2024-09-21T10:26:05.066572+0000","last_peered":"2024-09-21T10:26:05.066572+0000","last_clean":"2024-09-21T10:26:05.066572+0000","last_became_active":"2024-09-21T10:18:33.474818+0000","last_became_peered":"2024-09-21T10:18:33.474818+0000","last_unstale":"2024-09-21T10:26:05.066572+0000","last_undegraded":"2024-09-21T10:26:05.066572+0000","last_fullsized":"2024-09-21T10:26:05.066572+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:49.828007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":870,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.131618+0000","last_change":"2024-09-21T10:24:46.844278+0000","last_active":"2024-09-21T10:26:15.131618+0000","last_peered":"2024-09-21T10:26:15.131618+0000","last_clean":"2024-09-21T10:26:15.131618+0000","last_became_active":"2024-09-21T10:24:46.840961+0000","last_became_peered":"2024-09-21T10:24:46.840961+0000","last_unstale":"2024-09-21T10:26:15.131618+0000","last_undegraded":"2024-09-21T10:26:15.131618+0000","last_fullsized":"2024-09-21T10:26:15.131618+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:49:48.823524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":137,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.066166+0000","last_change":"2024-09-21T10:18:33.479008+0000","last_active":"2024-09-21T10:26:05.066166+0000","last_peered":"2024-09-21T10:26:05.066166+0000","last_clean":"2024-09-21T10:26:05.066166+0000","last_became_active":"2024-09-21T10:18:33.478740+0000","last_became_peered":"2024-09-21T10:18:33.478740+0000","last_unstale":"2024-09-21T10:26:05.066166+0000","last_undegraded":"2024-09-21T10:26:05.066166+0000","last_fullsized":"2024-09-21T10:26:05.066166+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:16:52.594882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"82'100","reported_seq":460,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.066632+0000","last_change":"2024-09-21T10:23:21.759441+0000","last_active":"2024-09-21T10:26:05.066632+0000","last_peered":"2024-09-21T10:26:05.066632+0000","last_clean":"2024-09-21T10:26:05.066632+0000","last_became_active":"2024-09-21T10:23:20.700452+0000","last_became_peered":"2024-09-21T10:23:20.700452+0000","last_unstale":"2024-09-21T10:26:05.066632+0000","last_undegraded":"2024-09-21T10:26:05.066632+0000","last_fullsized":"2024-09-21T10:26:05.066632+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.759355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.759355+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:21:15.413337+0000","scrub_duration":0.001833559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"345'16706","reported_seq":20658,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:16.889957+0000","last_change":"2024-09-21T10:25:54.379015+0000","last_active":"2024-09-21T10:26:16.889957+0000","last_peered":"2024-09-21T10:26:16.889957+0000","last_clean":"2024-09-21T10:26:16.889957+0000","last_became_active":"2024-09-21T10:25:54.287868+0000","last_became_peered":"2024-09-21T10:25:54.287868+0000","last_unstale":"2024-09-21T10:26:16.889957+0000","last_undegraded":"2024-09-21T10:26:16.889957+0000","last_fullsized":"2024-09-21T10:26:16.889957+0000","mapping_epoch":339,"log_start":"318'9723","ondisk_log_start":"318'9723","created":300,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:17.626792+0000","last_clean_scrub_stamp":"2024-09-21T10:22:17.626792+0000","objects_scrubbed":0,"log_size":6983,"log_dups_size":0,"ondisk_log_size":6983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:58:51.915622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16706,"num_write_kb":534592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1928,"num_bytes_recovered":126287872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":610,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.132970+0000","last_change":"2024-09-21T10:24:44.307344+0000","last_active":"2024-09-21T10:26:15.132970+0000","last_peered":"2024-09-21T10:26:15.132970+0000","last_clean":"2024-09-21T10:26:15.132970+0000","last_became_active":"2024-09-21T10:24:44.298936+0000","last_became_peered":"2024-09-21T10:24:44.298936+0000","last_unstale":"2024-09-21T10:26:15.132970+0000","last_undegraded":"2024-09-21T10:26:15.132970+0000","last_fullsized":"2024-09-21T10:26:15.132970+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:58:26.025877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":571,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.069555+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:26:05.069555+0000","last_peered":"2024-09-21T10:26:05.069555+0000","last_clean":"2024-09-21T10:26:05.069555+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:26:05.069555+0000","last_undegraded":"2024-09-21T10:26:05.069555+0000","last_fullsized":"2024-09-21T10:26:05.069555+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":34,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.065351+0000","last_change":"2024-09-21T10:24:44.303781+0000","last_active":"2024-09-21T10:26:05.065351+0000","last_peered":"2024-09-21T10:26:05.065351+0000","last_clean":"2024-09-21T10:26:05.065351+0000","last_became_active":"2024-09-21T10:24:44.302058+0000","last_became_peered":"2024-09-21T10:24:44.302058+0000","last_unstale":"2024-09-21T10:26:05.065351+0000","last_undegraded":"2024-09-21T10:26:05.065351+0000","last_fullsized":"2024-09-21T10:26:05.065351+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:30.371229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":34,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.066645+0000","last_change":"2024-09-21T10:24:44.303417+0000","last_active":"2024-09-21T10:26:05.066645+0000","last_peered":"2024-09-21T10:26:05.066645+0000","last_clean":"2024-09-21T10:26:05.066645+0000","last_became_active":"2024-09-21T10:24:44.303069+0000","last_became_peered":"2024-09-21T10:24:44.303069+0000","last_unstale":"2024-09-21T10:26:05.066645+0000","last_undegraded":"2024-09-21T10:26:05.066645+0000","last_fullsized":"2024-09-21T10:26:05.066645+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:30:30.966228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":44,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.066231+0000","last_change":"2024-09-21T10:24:44.321421+0000","last_active":"2024-09-21T10:26:05.066231+0000","last_peered":"2024-09-21T10:26:05.066231+0000","last_clean":"2024-09-21T10:26:05.066231+0000","last_became_active":"2024-09-21T10:24:44.297082+0000","last_became_peered":"2024-09-21T10:24:44.297082+0000","last_unstale":"2024-09-21T10:26:05.066231+0000","last_undegraded":"2024-09-21T10:26:05.066231+0000","last_fullsized":"2024-09-21T10:26:05.066231+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:35:37.704881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":444,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.132442+0000","last_change":"2024-09-21T10:08:18.470378+0000","last_active":"2024-09-21T10:26:15.132442+0000","last_peered":"2024-09-21T10:26:15.132442+0000","last_clean":"2024-09-21T10:26:15.132442+0000","last_became_active":"2024-09-21T10:08:18.469482+0000","last_became_peered":"2024-09-21T10:08:18.469482+0000","last_unstale":"2024-09-21T10:26:15.132442+0000","last_undegraded":"2024-09-21T10:26:15.132442+0000","last_fullsized":"2024-09-21T10:26:15.132442+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:35.628724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":392,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.067088+0000","last_change":"2024-09-21T10:24:44.313834+0000","last_active":"2024-09-21T10:26:05.067088+0000","last_peered":"2024-09-21T10:26:05.067088+0000","last_clean":"2024-09-21T10:26:05.067088+0000","last_became_active":"2024-09-21T10:24:44.299135+0000","last_became_peered":"2024-09-21T10:24:44.299135+0000","last_unstale":"2024-09-21T10:26:05.067088+0000","last_undegraded":"2024-09-21T10:26:05.067088+0000","last_fullsized":"2024-09-21T10:26:05.067088+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:21:51.959148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":89,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.131168+0000","last_change":"2024-09-21T10:22:39.385291+0000","last_active":"2024-09-21T10:26:15.131168+0000","last_peered":"2024-09-21T10:26:15.131168+0000","last_clean":"2024-09-21T10:26:15.131168+0000","last_became_active":"2024-09-21T10:22:39.384875+0000","last_became_peered":"2024-09-21T10:22:39.384875+0000","last_unstale":"2024-09-21T10:26:15.131168+0000","last_undegraded":"2024-09-21T10:26:15.131168+0000","last_fullsized":"2024-09-21T10:26:15.131168+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:33:30.610497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":97,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.069608+0000","last_change":"2024-09-21T10:18:33.464989+0000","last_active":"2024-09-21T10:26:05.069608+0000","last_peered":"2024-09-21T10:26:05.069608+0000","last_clean":"2024-09-21T10:26:05.069608+0000","last_became_active":"2024-09-21T10:18:33.464617+0000","last_became_peered":"2024-09-21T10:18:33.464617+0000","last_unstale":"2024-09-21T10:26:05.069608+0000","last_undegraded":"2024-09-21T10:26:05.069608+0000","last_fullsized":"2024-09-21T10:26:05.069608+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:09.300797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":567,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.133260+0000","last_change":"2024-09-21T10:18:33.472415+0000","last_active":"2024-09-21T10:26:15.133260+0000","last_peered":"2024-09-21T10:26:15.133260+0000","last_clean":"2024-09-21T10:26:15.133260+0000","last_became_active":"2024-09-21T10:18:33.471746+0000","last_became_peered":"2024-09-21T10:18:33.471746+0000","last_unstale":"2024-09-21T10:26:15.133260+0000","last_undegraded":"2024-09-21T10:26:15.133260+0000","last_fullsized":"2024-09-21T10:26:15.133260+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:40:37.099458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":221,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.130757+0000","last_change":"2024-09-21T10:24:44.310195+0000","last_active":"2024-09-21T10:26:15.130757+0000","last_peered":"2024-09-21T10:26:15.130757+0000","last_clean":"2024-09-21T10:26:15.130757+0000","last_became_active":"2024-09-21T10:24:44.309600+0000","last_became_peered":"2024-09-21T10:24:44.309600+0000","last_unstale":"2024-09-21T10:26:15.130757+0000","last_undegraded":"2024-09-21T10:26:15.130757+0000","last_fullsized":"2024-09-21T10:26:15.130757+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:29:15.459007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":308,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.069480+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:26:05.069480+0000","last_peered":"2024-09-21T10:26:05.069480+0000","last_clean":"2024-09-21T10:26:05.069480+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:26:05.069480+0000","last_undegraded":"2024-09-21T10:26:05.069480+0000","last_fullsized":"2024-09-21T10:26:05.069480+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":233,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.132804+0000","last_change":"2024-09-21T10:18:33.479635+0000","last_active":"2024-09-21T10:26:15.132804+0000","last_peered":"2024-09-21T10:26:15.132804+0000","last_clean":"2024-09-21T10:26:15.132804+0000","last_became_active":"2024-09-21T10:18:33.478830+0000","last_became_peered":"2024-09-21T10:18:33.478830+0000","last_unstale":"2024-09-21T10:26:15.132804+0000","last_undegraded":"2024-09-21T10:26:15.132804+0000","last_fullsized":"2024-09-21T10:26:15.132804+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:16.950086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":584,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.133198+0000","last_change":"2024-09-21T10:18:33.480350+0000","last_active":"2024-09-21T10:26:15.133198+0000","last_peered":"2024-09-21T10:26:15.133198+0000","last_clean":"2024-09-21T10:26:15.133198+0000","last_became_active":"2024-09-21T10:18:33.479248+0000","last_became_peered":"2024-09-21T10:18:33.479248+0000","last_unstale":"2024-09-21T10:26:15.133198+0000","last_undegraded":"2024-09-21T10:26:15.133198+0000","last_fullsized":"2024-09-21T10:26:15.133198+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:25:28.780572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":584,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.129349+0000","last_change":"2024-09-21T10:24:44.309248+0000","last_active":"2024-09-21T10:26:15.129349+0000","last_peered":"2024-09-21T10:26:15.129349+0000","last_clean":"2024-09-21T10:26:15.129349+0000","last_became_active":"2024-09-21T10:24:44.308925+0000","last_became_peered":"2024-09-21T10:24:44.308925+0000","last_unstale":"2024-09-21T10:26:15.129349+0000","last_undegraded":"2024-09-21T10:26:15.129349+0000","last_fullsized":"2024-09-21T10:26:15.129349+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:40.267676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":345,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.130000+0000","last_change":"2024-09-21T10:24:44.308839+0000","last_active":"2024-09-21T10:26:15.130000+0000","last_peered":"2024-09-21T10:26:15.130000+0000","last_clean":"2024-09-21T10:26:15.130000+0000","last_became_active":"2024-09-21T10:24:44.308488+0000","last_became_peered":"2024-09-21T10:24:44.308488+0000","last_unstale":"2024-09-21T10:26:15.130000+0000","last_undegraded":"2024-09-21T10:26:15.130000+0000","last_fullsized":"2024-09-21T10:26:15.130000+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:22:11.961205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":34,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.066854+0000","last_change":"2024-09-21T10:24:44.305124+0000","last_active":"2024-09-21T10:26:05.066854+0000","last_peered":"2024-09-21T10:26:05.066854+0000","last_clean":"2024-09-21T10:26:05.066854+0000","last_became_active":"2024-09-21T10:24:44.304468+0000","last_became_peered":"2024-09-21T10:24:44.304468+0000","last_unstale":"2024-09-21T10:26:05.066854+0000","last_undegraded":"2024-09-21T10:26:05.066854+0000","last_fullsized":"2024-09-21T10:26:05.066854+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:37.904764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":223,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.129562+0000","last_change":"2024-09-21T10:18:33.480128+0000","last_active":"2024-09-21T10:26:15.129562+0000","last_peered":"2024-09-21T10:26:15.129562+0000","last_clean":"2024-09-21T10:26:15.129562+0000","last_became_active":"2024-09-21T10:18:33.479691+0000","last_became_peered":"2024-09-21T10:18:33.479691+0000","last_unstale":"2024-09-21T10:26:15.129562+0000","last_undegraded":"2024-09-21T10:26:15.129562+0000","last_fullsized":"2024-09-21T10:26:15.129562+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:43:49.311244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":581,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.133599+0000","last_change":"2024-09-21T10:24:44.299989+0000","last_active":"2024-09-21T10:26:15.133599+0000","last_peered":"2024-09-21T10:26:15.133599+0000","last_clean":"2024-09-21T10:26:15.133599+0000","last_became_active":"2024-09-21T10:24:44.299356+0000","last_became_peered":"2024-09-21T10:24:44.299356+0000","last_unstale":"2024-09-21T10:26:15.133599+0000","last_undegraded":"2024-09-21T10:26:15.133599+0000","last_fullsized":"2024-09-21T10:26:15.133599+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:58.867008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":34,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.066403+0000","last_change":"2024-09-21T10:24:44.304040+0000","last_active":"2024-09-21T10:26:05.066403+0000","last_peered":"2024-09-21T10:26:05.066403+0000","last_clean":"2024-09-21T10:26:05.066403+0000","last_became_active":"2024-09-21T10:24:44.303666+0000","last_became_peered":"2024-09-21T10:24:44.303666+0000","last_unstale":"2024-09-21T10:26:05.066403+0000","last_undegraded":"2024-09-21T10:26:05.066403+0000","last_fullsized":"2024-09-21T10:26:05.066403+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:19:35.061997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":38,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.068701+0000","last_change":"2024-09-21T10:24:44.329150+0000","last_active":"2024-09-21T10:26:05.068701+0000","last_peered":"2024-09-21T10:26:05.068701+0000","last_clean":"2024-09-21T10:26:05.068701+0000","last_became_active":"2024-09-21T10:24:44.306752+0000","last_became_peered":"2024-09-21T10:24:44.306752+0000","last_unstale":"2024-09-21T10:26:05.068701+0000","last_undegraded":"2024-09-21T10:26:05.068701+0000","last_fullsized":"2024-09-21T10:26:05.068701+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:16.592859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":34,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.066935+0000","last_change":"2024-09-21T10:24:44.298825+0000","last_active":"2024-09-21T10:26:05.066935+0000","last_peered":"2024-09-21T10:26:05.066935+0000","last_clean":"2024-09-21T10:26:05.066935+0000","last_became_active":"2024-09-21T10:24:44.298517+0000","last_became_peered":"2024-09-21T10:24:44.298517+0000","last_unstale":"2024-09-21T10:26:05.066935+0000","last_undegraded":"2024-09-21T10:26:05.066935+0000","last_fullsized":"2024-09-21T10:26:05.066935+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:10.288494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":38,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.068541+0000","last_change":"2024-09-21T10:24:44.324087+0000","last_active":"2024-09-21T10:26:05.068541+0000","last_peered":"2024-09-21T10:26:05.068541+0000","last_clean":"2024-09-21T10:26:05.068541+0000","last_became_active":"2024-09-21T10:24:44.298665+0000","last_became_peered":"2024-09-21T10:24:44.298665+0000","last_unstale":"2024-09-21T10:26:05.068541+0000","last_undegraded":"2024-09-21T10:26:05.068541+0000","last_fullsized":"2024-09-21T10:26:05.068541+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:59.819340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":236,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.129130+0000","last_change":"2024-09-21T10:18:33.474216+0000","last_active":"2024-09-21T10:26:15.129130+0000","last_peered":"2024-09-21T10:26:15.129130+0000","last_clean":"2024-09-21T10:26:15.129130+0000","last_became_active":"2024-09-21T10:18:33.473794+0000","last_became_peered":"2024-09-21T10:18:33.473794+0000","last_unstale":"2024-09-21T10:26:15.129130+0000","last_undegraded":"2024-09-21T10:26:15.129130+0000","last_fullsized":"2024-09-21T10:26:15.129130+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:38.956888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":93,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.069694+0000","last_change":"2024-09-21T10:18:33.483576+0000","last_active":"2024-09-21T10:26:05.069694+0000","last_peered":"2024-09-21T10:26:05.069694+0000","last_clean":"2024-09-21T10:26:05.069694+0000","last_became_active":"2024-09-21T10:18:33.483116+0000","last_became_peered":"2024-09-21T10:18:33.483116+0000","last_unstale":"2024-09-21T10:26:05.069694+0000","last_undegraded":"2024-09-21T10:26:05.069694+0000","last_fullsized":"2024-09-21T10:26:05.069694+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:48.964766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":34,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-21T10:26:05.065039+0000","last_change":"2024-09-21T10:24:44.298820+0000","last_active":"2024-09-21T10:26:05.065039+0000","last_peered":"2024-09-21T10:26:05.065039+0000","last_clean":"2024-09-21T10:26:05.065039+0000","last_became_active":"2024-09-21T10:24:44.298478+0000","last_became_peered":"2024-09-21T10:24:44.298478+0000","last_unstale":"2024-09-21T10:26:05.065039+0000","last_undegraded":"2024-09-21T10:26:05.065039+0000","last_fullsized":"2024-09-21T10:26:05.065039+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:59.567078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":566,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-21T10:26:15.133156+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:26:15.133156+0000","last_peered":"2024-09-21T10:26:15.133156+0000","last_clean":"2024-09-21T10:26:15.133156+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:26:15.133156+0000","last_undegraded":"2024-09-21T10:26:15.133156+0000","last_fullsized":"2024-09-21T10:26:15.133156+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":265458,"num_write_kb":8494593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10727,"num_bytes_recovered":702873600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":84869120,"data_stored":84869120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111166,"ondisk_log_size":111166,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":10162,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5982,"num_read_kb":13005,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1100,"ondisk_log_size":1100,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397938,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":901936,"kb_used_data":8872,"kb_used_omap":1,"kb_used_meta":893054,"kb_avail":92843216,"statfs":{"total":95995035648,"available":95071453184,"internally_reserved":0,"allocated":9084928,"data_stored":7176767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":914487640},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594166,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":999468,"kb_used_data":89828,"kb_used_omap":1,"kb_used_meta":909630,"kb_avail":92745684,"statfs":{"total":95995035648,"available":94971580416,"internally_reserved":0,"allocated":91983872,"data_stored":90080224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931461464},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790396,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":904864,"kb_used_data":9112,"kb_used_omap":1,"kb_used_meta":895742,"kb_avail":92840288,"statfs":{"total":95995035648,"available":95068454912,"internally_reserved":0,"allocated":9330688,"data_stored":7434839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":917240152},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986625,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310124,"kb_used_data":4452,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93435028,"statfs":{"total":95995035648,"available":95677468672,"internally_reserved":0,"allocated":4558848,"data_stored":2654705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":433791697187,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860714,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":858788,"kb_used_data":8792,"kb_used_omap":8,"kb_used_meta":849975,"kb_avail":92886364,"statfs":{"total":95995035648,"available":95115636736,"internally_reserved":0,"allocated":9003008,"data_stored":7107439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":870375276},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024239,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":771280,"kb_used_data":4424,"kb_used_omap":7,"kb_used_meta":766840,"kb_avail":92973872,"statfs":{"total":95995035648,"available":95205244928,"internally_reserved":0,"allocated":4530176,"data_stored":2649232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":785244722},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187775,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":843232,"kb_used_data":4440,"kb_used_omap":8,"kb_used_meta":838775,"kb_avail":92901920,"statfs":{"total":95995035648,"available":95131566080,"internally_reserved":0,"allocated":4546560,"data_stored":2650525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":858906609},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":84869120,"data_stored":84869120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:26:23.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:26:23.842 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:23.865 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:23.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:23.967 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:23.989 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:23.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:24.090 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:24.112 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:24.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:24.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:24.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:24.235 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:24.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:24.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:24.337 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:24.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:24 smithi098 ceph-mon[97580]: from='client.46259 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:26:24.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:24 smithi098 ceph-mon[100914]: from='client.46259 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:26:24.357 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:24.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:24.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:24.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:24 smithi196 ceph-mon[84784]: from='client.46259 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:26:24.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:26:24.459 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:24.481 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:24.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:24.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:24.583 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:24.603 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:24.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:24.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:24.705 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:24.726 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:24.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:24.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:24.827 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:24.849 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:24.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:24.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:24.951 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:24.972 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:24.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:24.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:25.074 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:25.095 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:25.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:25.197 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:25.212 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:26:25.212 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:26:25.212 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-21T10:26:25.212 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:26:25.217 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:25.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:25.319 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:25.340 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:25.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:25 smithi196 ceph-mon[84784]: pgmap v1160: 99 pgs: 99 active+clean; 4.6 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 406 op/s 2024-09-21T10:26:25.442 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:25.463 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:25.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:25.565 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:25.585 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:25 smithi098 ceph-mon[97580]: pgmap v1160: 99 pgs: 99 active+clean; 4.6 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 406 op/s 2024-09-21T10:26:25.585 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:25 smithi098 ceph-mon[100914]: pgmap v1160: 99 pgs: 99 active+clean; 4.6 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 406 op/s 2024-09-21T10:26:25.588 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:25.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:25.690 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:25.712 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:25.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:25.813 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:25.835 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:25.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:25.911 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:25.911 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:25.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:25.958 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:25.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:26.059 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:26.080 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:26.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:26.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:26.182 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:26.204 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:26.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:26.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:26.292 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:26:26.305 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:26.356 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:26.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:26.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:26.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:26.481 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:26.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:26.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:26.583 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:26.604 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:26.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:26.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:26.706 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:26.724 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:26:26.730 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:26.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:26.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:26.797 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:26:26.797 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:26:26.813 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:26:26.814 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:26:26.829 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:26:26.829 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:26:26.832 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:26.845 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:26:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:26:26.853 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:26.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:26.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:26.861 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:26:26.861 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:26:26.878 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:26:26.878 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:26:26.894 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:26:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:26:26.910 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:26:26.910 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:26:26.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:26.976 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:26.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:26.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:27.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:27.098 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:27.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:27.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:27.164 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'unique_pool_4' removed 2024-09-21T10:26:27.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:27.232 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:27.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:27.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:27.334 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:27.355 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:27.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:27.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:27 smithi196 ceph-mon[84784]: pgmap v1161: 99 pgs: 99 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 341 op/s 2024-09-21T10:26:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:27 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3743094041' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:26:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:27 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:26:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:27 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3992058029' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:27.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:27 smithi098 ceph-mon[97580]: pgmap v1161: 99 pgs: 99 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 341 op/s 2024-09-21T10:26:27.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:27 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3743094041' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:26:27.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:27 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:26:27.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:27 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3992058029' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:27.450 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:27 smithi098 ceph-mon[100914]: pgmap v1161: 99 pgs: 99 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 341 op/s 2024-09-21T10:26:27.450 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:27 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3743094041' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:26:27.450 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:27 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:26:27.450 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:27 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3992058029' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:27.457 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:27.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:27.480 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:27.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:27.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:27.603 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:27.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:27.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:27.704 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:27.726 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:27.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:27.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:27.828 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:27.849 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:27.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:27.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:27.951 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:27.972 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:27.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:27.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:28.073 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:28.095 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:28.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:28.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:28.197 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:28.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:28.220 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:28.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:28.321 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:28.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:28.344 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:28.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:28.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:28 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:26:28.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:28 smithi098 ceph-mon[97580]: osdmap e347: 8 total, 8 up, 7 in 2024-09-21T10:26:28.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:28 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:26:28.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:28 smithi098 ceph-mon[100914]: osdmap e347: 8 total, 8 up, 7 in 2024-09-21T10:26:28.420 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:28 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:26:28.420 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:28 smithi196 ceph-mon[84784]: osdmap e347: 8 total, 8 up, 7 in 2024-09-21T10:26:28.420 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:26:28.420 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:26:28.445 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:28.466 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:28.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:28.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:28.568 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:28.589 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:28.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:28.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:28.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:28.714 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:28.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:28.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:28.816 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:28.837 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:28.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:28.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:28.939 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:28.960 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:28.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:28.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:29.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:29.084 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:29.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:29.113 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:26:29.186 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:29.219 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:29.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:29.320 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:29.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:29.443 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:29.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:29.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:29 smithi098 ceph-mon[97580]: pgmap v1163: 83 pgs: 83 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:29.475 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:29 smithi098 ceph-mon[100914]: pgmap v1163: 83 pgs: 83 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:29.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:29.578 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:29 smithi196 ceph-mon[84784]: pgmap v1163: 83 pgs: 83 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:29.588 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:29.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:29.689 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:29.708 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:26:29.708 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:26:29.709 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-09-21T10:26:29.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_5 16 2024-09-21T10:26:29.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.712 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:29.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:29.814 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:29.837 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:29.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:29.939 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:29.961 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:29.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:30.063 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:30.085 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:30.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:30.186 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:30.209 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:30.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:30.259 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:26:30.311 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:30.333 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:30.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:30.380 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:26:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:26:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:26:30.434 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:30.457 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:30.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:30.559 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:30.563 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:30.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:30.682 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:30.704 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:30.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:30.803 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:26:30.805 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:30.826 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:30.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:30.925 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:26:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:26:30] "GET /metrics HTTP/1.1" 200 36510 "" "Prometheus/2.51.0" 2024-09-21T10:26:30.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:30.952 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:30.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:31.053 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:31.074 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:31.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:31.176 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:31.198 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:31.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:31.299 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:31.321 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:31.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:31.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:31 smithi098 ceph-mon[97580]: pgmap v1164: 83 pgs: 83 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:31.328 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:31 smithi098 ceph-mon[100914]: pgmap v1164: 83 pgs: 83 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:31.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:31 smithi196 ceph-mon[84784]: pgmap v1164: 83 pgs: 83 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:31.424 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:31.445 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:31.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:31.547 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:31.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:31.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:31.671 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:31.692 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:31.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:31.794 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:31.812 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:26:31.812 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:26:31.815 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:31.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:31.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:31.939 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:31.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:32.041 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:32.061 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:32.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:32.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:32.163 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:32.184 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:32.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:32.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:32.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:32.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:32.308 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:32.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:32.360 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:32.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:32.430 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:32.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:32.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:32.532 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:32.553 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:32.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:32.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:32.655 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:32.677 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:32.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:32.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:32.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:32.801 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:32.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:32.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:32.902 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:32.923 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:32.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:32.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:33.022 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:26:33.022 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:26:33.026 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:33.049 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:33.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:33.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:33.172 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:33.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:33.274 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:33.295 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:33.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:33.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:33 smithi098 ceph-mon[97580]: pgmap v1165: 83 pgs: 83 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:33.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:33 smithi098 ceph-mon[97580]: from='client.36357 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:26:33.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:33 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:33.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:33 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:33.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:33 smithi098 ceph-mon[100914]: pgmap v1165: 83 pgs: 83 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:33.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:33 smithi098 ceph-mon[100914]: from='client.36357 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:26:33.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:33 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:33.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:33 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:33.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:33 smithi196 ceph-mon[84784]: pgmap v1165: 83 pgs: 83 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:33.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:33 smithi196 ceph-mon[84784]: from='client.36357 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:26:33.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:33 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:33.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:33 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:33.397 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:33.418 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1165,"stamp":"2024-09-21T10:26:31.528866+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5982,"num_read_kb":13005,"num_write":907,"num_write_kb":12623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":10162,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2132,"ondisk_log_size":2132,"up":249,"acting":249,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":273,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":5553428,"kb_used_data":93632,"kb_used_omap":30,"kb_used_meta":5459681,"kb_avail":650662636,"statfs":{"total":671965249536,"available":666278539264,"internally_reserved":0,"allocated":95879168,"data_stored":82499651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30993,"internal_metadata":5590714095},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":58,"apply_latency_ms":58,"commit_latency_ns":58000000,"apply_latency_ns":58000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-265458,"num_write_kb":-8494593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-10727,"num_bytes_recovered":-702873600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002507"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":84,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.397150+0000","last_change":"2024-09-21T10:25:39.893215+0000","last_active":"2024-09-21T10:26:19.397150+0000","last_peered":"2024-09-21T10:26:19.397150+0000","last_clean":"2024-09-21T10:26:19.397150+0000","last_became_active":"2024-09-21T10:25:39.892930+0000","last_became_peered":"2024-09-21T10:25:39.892930+0000","last_unstale":"2024-09-21T10:26:19.397150+0000","last_undegraded":"2024-09-21T10:26:19.397150+0000","last_fullsized":"2024-09-21T10:26:19.397150+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:11:59.615233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":377,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.397015+0000","last_change":"2024-09-21T10:24:44.305198+0000","last_active":"2024-09-21T10:26:19.397015+0000","last_peered":"2024-09-21T10:26:19.397015+0000","last_clean":"2024-09-21T10:26:19.397015+0000","last_became_active":"2024-09-21T10:24:44.297653+0000","last_became_peered":"2024-09-21T10:24:44.297653+0000","last_unstale":"2024-09-21T10:26:19.397015+0000","last_undegraded":"2024-09-21T10:26:19.397015+0000","last_fullsized":"2024-09-21T10:26:19.397015+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:19.320833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":277,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.397159+0000","last_change":"2024-09-21T10:18:33.465357+0000","last_active":"2024-09-21T10:26:19.397159+0000","last_peered":"2024-09-21T10:26:19.397159+0000","last_clean":"2024-09-21T10:26:19.397159+0000","last_became_active":"2024-09-21T10:18:33.464952+0000","last_became_peered":"2024-09-21T10:18:33.464952+0000","last_unstale":"2024-09-21T10:26:19.397159+0000","last_undegraded":"2024-09-21T10:26:19.397159+0000","last_fullsized":"2024-09-21T10:26:19.397159+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:15:41.090866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":153,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.397002+0000","last_change":"2024-09-21T10:18:33.475291+0000","last_active":"2024-09-21T10:26:19.397002+0000","last_peered":"2024-09-21T10:26:19.397002+0000","last_clean":"2024-09-21T10:26:19.397002+0000","last_became_active":"2024-09-21T10:18:33.474971+0000","last_became_peered":"2024-09-21T10:18:33.474971+0000","last_unstale":"2024-09-21T10:26:19.397002+0000","last_undegraded":"2024-09-21T10:26:19.397002+0000","last_fullsized":"2024-09-21T10:26:19.397002+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:39:40.179668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":572,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.513377+0000","last_change":"2024-09-21T10:18:33.475179+0000","last_active":"2024-09-21T10:26:27.513377+0000","last_peered":"2024-09-21T10:26:27.513377+0000","last_clean":"2024-09-21T10:26:27.513377+0000","last_became_active":"2024-09-21T10:18:33.474914+0000","last_became_peered":"2024-09-21T10:18:33.474914+0000","last_unstale":"2024-09-21T10:26:27.513377+0000","last_undegraded":"2024-09-21T10:26:27.513377+0000","last_fullsized":"2024-09-21T10:26:27.513377+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:15:32.186986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":36,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:20.339647+0000","last_change":"2024-09-21T10:24:44.304399+0000","last_active":"2024-09-21T10:26:20.339647+0000","last_peered":"2024-09-21T10:26:20.339647+0000","last_clean":"2024-09-21T10:26:20.339647+0000","last_became_active":"2024-09-21T10:24:44.303437+0000","last_became_peered":"2024-09-21T10:24:44.303437+0000","last_unstale":"2024-09-21T10:26:20.339647+0000","last_undegraded":"2024-09-21T10:26:20.339647+0000","last_fullsized":"2024-09-21T10:26:20.339647+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:32:56.638481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":32,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.513162+0000","last_change":"2024-09-21T10:25:39.876792+0000","last_active":"2024-09-21T10:26:27.513162+0000","last_peered":"2024-09-21T10:26:27.513162+0000","last_clean":"2024-09-21T10:26:27.513162+0000","last_became_active":"2024-09-21T10:25:39.876336+0000","last_became_peered":"2024-09-21T10:25:39.876336+0000","last_unstale":"2024-09-21T10:26:27.513162+0000","last_undegraded":"2024-09-21T10:26:27.513162+0000","last_fullsized":"2024-09-21T10:26:27.513162+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:06:57.688437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":319,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.449237+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:26:19.449237+0000","last_peered":"2024-09-21T10:26:19.449237+0000","last_clean":"2024-09-21T10:26:19.449237+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:26:19.449237+0000","last_undegraded":"2024-09-21T10:26:19.449237+0000","last_fullsized":"2024-09-21T10:26:19.449237+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":368,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.397463+0000","last_change":"2024-09-21T10:24:44.325196+0000","last_active":"2024-09-21T10:26:19.397463+0000","last_peered":"2024-09-21T10:26:19.397463+0000","last_clean":"2024-09-21T10:26:19.397463+0000","last_became_active":"2024-09-21T10:24:44.299614+0000","last_became_peered":"2024-09-21T10:24:44.299614+0000","last_unstale":"2024-09-21T10:26:19.397463+0000","last_undegraded":"2024-09-21T10:26:19.397463+0000","last_fullsized":"2024-09-21T10:26:19.397463+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:14.558663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":210,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.396905+0000","last_change":"2024-09-21T10:07:43.794677+0000","last_active":"2024-09-21T10:26:19.396905+0000","last_peered":"2024-09-21T10:26:19.396905+0000","last_clean":"2024-09-21T10:26:19.396905+0000","last_became_active":"2024-09-21T10:07:43.794348+0000","last_became_peered":"2024-09-21T10:07:43.794348+0000","last_unstale":"2024-09-21T10:26:19.396905+0000","last_undegraded":"2024-09-21T10:26:19.396905+0000","last_fullsized":"2024-09-21T10:26:19.396905+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:11.159014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":369,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.397306+0000","last_change":"2024-09-21T10:24:44.323182+0000","last_active":"2024-09-21T10:26:19.397306+0000","last_peered":"2024-09-21T10:26:19.397306+0000","last_clean":"2024-09-21T10:26:19.397306+0000","last_became_active":"2024-09-21T10:24:44.299698+0000","last_became_peered":"2024-09-21T10:24:44.299698+0000","last_unstale":"2024-09-21T10:26:19.397306+0000","last_undegraded":"2024-09-21T10:26:19.397306+0000","last_fullsized":"2024-09-21T10:26:19.397306+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:56.571057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":36,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:20.338624+0000","last_change":"2024-09-21T10:24:44.304586+0000","last_active":"2024-09-21T10:26:20.338624+0000","last_peered":"2024-09-21T10:26:20.338624+0000","last_clean":"2024-09-21T10:26:20.338624+0000","last_became_active":"2024-09-21T10:24:44.303815+0000","last_became_peered":"2024-09-21T10:24:44.303815+0000","last_unstale":"2024-09-21T10:26:20.338624+0000","last_undegraded":"2024-09-21T10:26:20.338624+0000","last_fullsized":"2024-09-21T10:26:20.338624+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:02.126454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":295,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.397260+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:26:19.397260+0000","last_peered":"2024-09-21T10:26:19.397260+0000","last_clean":"2024-09-21T10:26:19.397260+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:26:19.397260+0000","last_undegraded":"2024-09-21T10:26:19.397260+0000","last_fullsized":"2024-09-21T10:26:19.397260+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":310,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.449186+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:26:19.449186+0000","last_peered":"2024-09-21T10:26:19.449186+0000","last_clean":"2024-09-21T10:26:19.449186+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:26:19.449186+0000","last_undegraded":"2024-09-21T10:26:19.449186+0000","last_fullsized":"2024-09-21T10:26:19.449186+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":145,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.449449+0000","last_change":"2024-09-21T10:18:33.483457+0000","last_active":"2024-09-21T10:26:19.449449+0000","last_peered":"2024-09-21T10:26:19.449449+0000","last_clean":"2024-09-21T10:26:19.449449+0000","last_became_active":"2024-09-21T10:18:33.483140+0000","last_became_peered":"2024-09-21T10:18:33.483140+0000","last_unstale":"2024-09-21T10:26:19.449449+0000","last_undegraded":"2024-09-21T10:26:19.449449+0000","last_fullsized":"2024-09-21T10:26:19.449449+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:37:39.302566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":600,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.513645+0000","last_change":"2024-09-21T10:24:44.299417+0000","last_active":"2024-09-21T10:26:27.513645+0000","last_peered":"2024-09-21T10:26:27.513645+0000","last_clean":"2024-09-21T10:26:27.513645+0000","last_became_active":"2024-09-21T10:24:44.299116+0000","last_became_peered":"2024-09-21T10:24:44.299116+0000","last_unstale":"2024-09-21T10:26:27.513645+0000","last_undegraded":"2024-09-21T10:26:27.513645+0000","last_fullsized":"2024-09-21T10:26:27.513645+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:44.524841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"82'100","reported_seq":458,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.396836+0000","last_change":"2024-09-21T10:23:22.728269+0000","last_active":"2024-09-21T10:26:19.396836+0000","last_peered":"2024-09-21T10:26:19.396836+0000","last_clean":"2024-09-21T10:26:19.396836+0000","last_became_active":"2024-09-21T10:23:20.700241+0000","last_became_peered":"2024-09-21T10:23:20.700241+0000","last_unstale":"2024-09-21T10:26:19.396836+0000","last_undegraded":"2024-09-21T10:26:19.396836+0000","last_fullsized":"2024-09-21T10:26:19.396836+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.728195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.728195+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:31.235642+0000","scrub_duration":0.0069698140000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":40,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:20.338927+0000","last_change":"2024-09-21T10:24:44.327042+0000","last_active":"2024-09-21T10:26:20.338927+0000","last_peered":"2024-09-21T10:26:20.338927+0000","last_clean":"2024-09-21T10:26:20.338927+0000","last_became_active":"2024-09-21T10:24:44.303785+0000","last_became_peered":"2024-09-21T10:24:44.303785+0000","last_unstale":"2024-09-21T10:26:20.338927+0000","last_undegraded":"2024-09-21T10:26:20.338927+0000","last_fullsized":"2024-09-21T10:26:20.338927+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:11:19.180549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":36,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:20.339349+0000","last_change":"2024-09-21T10:24:44.306721+0000","last_active":"2024-09-21T10:26:20.339349+0000","last_peered":"2024-09-21T10:26:20.339349+0000","last_clean":"2024-09-21T10:26:20.339349+0000","last_became_active":"2024-09-21T10:24:44.306427+0000","last_became_peered":"2024-09-21T10:24:44.306427+0000","last_unstale":"2024-09-21T10:26:20.339349+0000","last_undegraded":"2024-09-21T10:26:20.339349+0000","last_fullsized":"2024-09-21T10:26:20.339349+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:33:08.866485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"82'100","reported_seq":92,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.158505+0000","last_change":"2024-09-21T10:23:27.234299+0000","last_active":"2024-09-21T10:26:27.158505+0000","last_peered":"2024-09-21T10:26:27.158505+0000","last_clean":"2024-09-21T10:26:27.158505+0000","last_became_active":"2024-09-21T10:23:20.698155+0000","last_became_peered":"2024-09-21T10:23:20.698155+0000","last_unstale":"2024-09-21T10:26:27.158505+0000","last_undegraded":"2024-09-21T10:26:27.158505+0000","last_fullsized":"2024-09-21T10:26:27.158505+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:27.234001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:27.234001+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:13:43.690329+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":361,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.449266+0000","last_change":"2024-09-21T10:24:44.309470+0000","last_active":"2024-09-21T10:26:19.449266+0000","last_peered":"2024-09-21T10:26:19.449266+0000","last_clean":"2024-09-21T10:26:19.449266+0000","last_became_active":"2024-09-21T10:24:44.308836+0000","last_became_peered":"2024-09-21T10:24:44.308836+0000","last_unstale":"2024-09-21T10:26:19.449266+0000","last_undegraded":"2024-09-21T10:26:19.449266+0000","last_fullsized":"2024-09-21T10:26:19.449266+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:09:44.622418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":36,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:20.338726+0000","last_change":"2024-09-21T10:24:44.305135+0000","last_active":"2024-09-21T10:26:20.338726+0000","last_peered":"2024-09-21T10:26:20.338726+0000","last_clean":"2024-09-21T10:26:20.338726+0000","last_became_active":"2024-09-21T10:24:44.304839+0000","last_became_peered":"2024-09-21T10:24:44.304839+0000","last_unstale":"2024-09-21T10:26:20.338726+0000","last_undegraded":"2024-09-21T10:26:20.338726+0000","last_fullsized":"2024-09-21T10:26:20.338726+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:02.264404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":32,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.512819+0000","last_change":"2024-09-21T10:25:39.872680+0000","last_active":"2024-09-21T10:26:27.512819+0000","last_peered":"2024-09-21T10:26:27.512819+0000","last_clean":"2024-09-21T10:26:27.512819+0000","last_became_active":"2024-09-21T10:25:39.872312+0000","last_became_peered":"2024-09-21T10:25:39.872312+0000","last_unstale":"2024-09-21T10:26:27.512819+0000","last_undegraded":"2024-09-21T10:26:27.512819+0000","last_fullsized":"2024-09-21T10:26:27.512819+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:54.942046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":36,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:20.338923+0000","last_change":"2024-09-21T10:24:44.310484+0000","last_active":"2024-09-21T10:26:20.338923+0000","last_peered":"2024-09-21T10:26:20.338923+0000","last_clean":"2024-09-21T10:26:20.338923+0000","last_became_active":"2024-09-21T10:24:44.307209+0000","last_became_peered":"2024-09-21T10:24:44.307209+0000","last_unstale":"2024-09-21T10:26:20.338923+0000","last_undegraded":"2024-09-21T10:26:20.338923+0000","last_fullsized":"2024-09-21T10:26:20.338923+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:48:36.767539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'100","reported_seq":87,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.158623+0000","last_change":"2024-09-21T10:23:26.197358+0000","last_active":"2024-09-21T10:26:27.158623+0000","last_peered":"2024-09-21T10:26:27.158623+0000","last_clean":"2024-09-21T10:26:27.158623+0000","last_became_active":"2024-09-21T10:23:20.695174+0000","last_became_peered":"2024-09-21T10:23:20.695174+0000","last_unstale":"2024-09-21T10:26:27.158623+0000","last_undegraded":"2024-09-21T10:26:27.158623+0000","last_fullsized":"2024-09-21T10:26:27.158623+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:26.197291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:26.197291+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:33.961031+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":224,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.261843+0000","last_change":"2024-09-21T10:24:44.308818+0000","last_active":"2024-09-21T10:26:19.261843+0000","last_peered":"2024-09-21T10:26:19.261843+0000","last_clean":"2024-09-21T10:26:19.261843+0000","last_became_active":"2024-09-21T10:24:44.308014+0000","last_became_peered":"2024-09-21T10:24:44.308014+0000","last_unstale":"2024-09-21T10:26:19.261843+0000","last_undegraded":"2024-09-21T10:26:19.261843+0000","last_fullsized":"2024-09-21T10:26:19.261843+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:36:14.730099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":587,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.262293+0000","last_change":"2024-09-21T10:24:44.309294+0000","last_active":"2024-09-21T10:26:19.262293+0000","last_peered":"2024-09-21T10:26:19.262293+0000","last_clean":"2024-09-21T10:26:19.262293+0000","last_became_active":"2024-09-21T10:24:44.308928+0000","last_became_peered":"2024-09-21T10:24:44.308928+0000","last_unstale":"2024-09-21T10:26:19.262293+0000","last_undegraded":"2024-09-21T10:26:19.262293+0000","last_fullsized":"2024-09-21T10:26:19.262293+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:50:20.141648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":449,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.397023+0000","last_change":"2024-09-21T10:23:18.790100+0000","last_active":"2024-09-21T10:26:19.397023+0000","last_peered":"2024-09-21T10:26:19.397023+0000","last_clean":"2024-09-21T10:26:19.397023+0000","last_became_active":"2024-09-21T10:23:14.677610+0000","last_became_peered":"2024-09-21T10:23:14.677610+0000","last_unstale":"2024-09-21T10:26:19.397023+0000","last_undegraded":"2024-09-21T10:26:19.397023+0000","last_fullsized":"2024-09-21T10:26:19.397023+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:18.790017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:18.790017+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:56.562241+0000","scrub_duration":0.0080400790000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":863,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.158935+0000","last_change":"2024-09-21T10:07:43.790490+0000","last_active":"2024-09-21T10:26:27.158935+0000","last_peered":"2024-09-21T10:26:27.158935+0000","last_clean":"2024-09-21T10:26:27.158935+0000","last_became_active":"2024-09-21T10:07:43.789986+0000","last_became_peered":"2024-09-21T10:07:43.789986+0000","last_unstale":"2024-09-21T10:26:27.158935+0000","last_undegraded":"2024-09-21T10:26:27.158935+0000","last_fullsized":"2024-09-21T10:26:27.158935+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:19:08.740356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":481,"num_read_kb":1580,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":108,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.449111+0000","last_change":"2024-09-21T10:24:44.309030+0000","last_active":"2024-09-21T10:26:19.449111+0000","last_peered":"2024-09-21T10:26:19.449111+0000","last_clean":"2024-09-21T10:26:19.449111+0000","last_became_active":"2024-09-21T10:24:44.308388+0000","last_became_peered":"2024-09-21T10:24:44.308388+0000","last_unstale":"2024-09-21T10:26:19.449111+0000","last_undegraded":"2024-09-21T10:26:19.449111+0000","last_fullsized":"2024-09-21T10:26:19.449111+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:56.273653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":333,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.449432+0000","last_change":"2024-09-21T10:24:44.308008+0000","last_active":"2024-09-21T10:26:19.449432+0000","last_peered":"2024-09-21T10:26:19.449432+0000","last_clean":"2024-09-21T10:26:19.449432+0000","last_became_active":"2024-09-21T10:24:44.307595+0000","last_became_peered":"2024-09-21T10:24:44.307595+0000","last_unstale":"2024-09-21T10:26:19.449432+0000","last_undegraded":"2024-09-21T10:26:19.449432+0000","last_fullsized":"2024-09-21T10:26:19.449432+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:33:49.448676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"82'100","reported_seq":92,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.261783+0000","last_change":"2024-09-21T10:24:44.309154+0000","last_active":"2024-09-21T10:26:19.261783+0000","last_peered":"2024-09-21T10:26:19.261783+0000","last_clean":"2024-09-21T10:26:19.261783+0000","last_became_active":"2024-09-21T10:24:44.308330+0000","last_became_peered":"2024-09-21T10:24:44.308330+0000","last_unstale":"2024-09-21T10:26:19.261783+0000","last_undegraded":"2024-09-21T10:26:19.261783+0000","last_fullsized":"2024-09-21T10:26:19.261783+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.641718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.641718+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:10.158739+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":691,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.396892+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:26:19.396892+0000","last_peered":"2024-09-21T10:26:19.396892+0000","last_clean":"2024-09-21T10:26:19.396892+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:26:19.396892+0000","last_undegraded":"2024-09-21T10:26:19.396892+0000","last_fullsized":"2024-09-21T10:26:19.396892+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":601,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.512964+0000","last_change":"2024-09-21T10:24:44.299102+0000","last_active":"2024-09-21T10:26:27.512964+0000","last_peered":"2024-09-21T10:26:27.512964+0000","last_clean":"2024-09-21T10:26:27.512964+0000","last_became_active":"2024-09-21T10:24:44.298723+0000","last_became_peered":"2024-09-21T10:24:44.298723+0000","last_unstale":"2024-09-21T10:26:27.512964+0000","last_undegraded":"2024-09-21T10:26:27.512964+0000","last_fullsized":"2024-09-21T10:26:27.512964+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:57:47.078328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":577,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.513505+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:26:27.513505+0000","last_peered":"2024-09-21T10:26:27.513505+0000","last_clean":"2024-09-21T10:26:27.513505+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:26:27.513505+0000","last_undegraded":"2024-09-21T10:26:27.513505+0000","last_fullsized":"2024-09-21T10:26:27.513505+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"82'100","reported_seq":240,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.512880+0000","last_change":"2024-09-21T10:24:44.294460+0000","last_active":"2024-09-21T10:26:27.512880+0000","last_peered":"2024-09-21T10:26:27.512880+0000","last_clean":"2024-09-21T10:26:27.512880+0000","last_became_active":"2024-09-21T10:24:44.293972+0000","last_became_peered":"2024-09-21T10:24:44.293972+0000","last_unstale":"2024-09-21T10:26:27.512880+0000","last_undegraded":"2024-09-21T10:26:27.512880+0000","last_fullsized":"2024-09-21T10:26:27.512880+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:26.199522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:26.199522+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:26.135171+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":1350,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.158758+0000","last_change":"2024-09-21T10:18:36.495308+0000","last_active":"2024-09-21T10:26:27.158758+0000","last_peered":"2024-09-21T10:26:27.158758+0000","last_clean":"2024-09-21T10:26:27.158758+0000","last_became_active":"2024-09-21T10:18:36.494662+0000","last_became_peered":"2024-09-21T10:18:36.494662+0000","last_unstale":"2024-09-21T10:26:27.158758+0000","last_undegraded":"2024-09-21T10:26:27.158758+0000","last_fullsized":"2024-09-21T10:26:27.158758+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:49.218676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":447,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.261968+0000","last_change":"2024-09-21T10:08:18.470748+0000","last_active":"2024-09-21T10:26:19.261968+0000","last_peered":"2024-09-21T10:26:19.261968+0000","last_clean":"2024-09-21T10:26:19.261968+0000","last_became_active":"2024-09-21T10:08:18.470227+0000","last_became_peered":"2024-09-21T10:08:18.470227+0000","last_unstale":"2024-09-21T10:26:19.261968+0000","last_undegraded":"2024-09-21T10:26:19.261968+0000","last_fullsized":"2024-09-21T10:26:19.261968+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:16.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":571,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.160011+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:26:27.160011+0000","last_peered":"2024-09-21T10:26:27.160011+0000","last_clean":"2024-09-21T10:26:27.160011+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:26:27.160011+0000","last_undegraded":"2024-09-21T10:26:27.160011+0000","last_fullsized":"2024-09-21T10:26:27.160011+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"82'100","reported_seq":233,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.512826+0000","last_change":"2024-09-21T10:23:21.277739+0000","last_active":"2024-09-21T10:26:27.512826+0000","last_peered":"2024-09-21T10:26:27.512826+0000","last_clean":"2024-09-21T10:26:27.512826+0000","last_became_active":"2024-09-21T10:23:20.689733+0000","last_became_peered":"2024-09-21T10:23:20.689733+0000","last_unstale":"2024-09-21T10:26:27.512826+0000","last_undegraded":"2024-09-21T10:26:27.512826+0000","last_fullsized":"2024-09-21T10:26:27.512826+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.277663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.277663+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:01:54.523189+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":1045,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.262040+0000","last_change":"2024-09-21T10:07:43.794767+0000","last_active":"2024-09-21T10:26:19.262040+0000","last_peered":"2024-09-21T10:26:19.262040+0000","last_clean":"2024-09-21T10:26:19.262040+0000","last_became_active":"2024-09-21T10:07:43.794062+0000","last_became_peered":"2024-09-21T10:07:43.794062+0000","last_unstale":"2024-09-21T10:26:19.262040+0000","last_undegraded":"2024-09-21T10:26:19.262040+0000","last_fullsized":"2024-09-21T10:26:19.262040+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:51.977955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":36,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:20.338807+0000","last_change":"2024-09-21T10:24:44.299492+0000","last_active":"2024-09-21T10:26:20.338807+0000","last_peered":"2024-09-21T10:26:20.338807+0000","last_clean":"2024-09-21T10:26:20.338807+0000","last_became_active":"2024-09-21T10:24:44.299187+0000","last_became_peered":"2024-09-21T10:24:44.299187+0000","last_unstale":"2024-09-21T10:26:20.338807+0000","last_undegraded":"2024-09-21T10:26:20.338807+0000","last_fullsized":"2024-09-21T10:26:20.338807+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:58.500386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":494,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.159041+0000","last_change":"2024-09-21T10:18:33.465815+0000","last_active":"2024-09-21T10:26:27.159041+0000","last_peered":"2024-09-21T10:26:27.159041+0000","last_clean":"2024-09-21T10:26:27.159041+0000","last_became_active":"2024-09-21T10:18:33.465459+0000","last_became_peered":"2024-09-21T10:18:33.465459+0000","last_unstale":"2024-09-21T10:26:27.159041+0000","last_undegraded":"2024-09-21T10:26:27.159041+0000","last_fullsized":"2024-09-21T10:26:27.159041+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:22:08.199800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"82'100","reported_seq":201,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.449490+0000","last_change":"2024-09-21T10:23:30.570074+0000","last_active":"2024-09-21T10:26:19.449490+0000","last_peered":"2024-09-21T10:26:19.449490+0000","last_clean":"2024-09-21T10:26:19.449490+0000","last_became_active":"2024-09-21T10:23:20.695270+0000","last_became_peered":"2024-09-21T10:23:20.695270+0000","last_unstale":"2024-09-21T10:26:19.449490+0000","last_undegraded":"2024-09-21T10:26:19.449490+0000","last_fullsized":"2024-09-21T10:26:19.449490+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:30.569982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:30.569982+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:06.011343+0000","scrub_duration":0.0051309349999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":237,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.158891+0000","last_change":"2024-09-21T10:18:33.467659+0000","last_active":"2024-09-21T10:26:27.158891+0000","last_peered":"2024-09-21T10:26:27.158891+0000","last_clean":"2024-09-21T10:26:27.158891+0000","last_became_active":"2024-09-21T10:18:33.467369+0000","last_became_peered":"2024-09-21T10:18:33.467369+0000","last_unstale":"2024-09-21T10:26:27.158891+0000","last_undegraded":"2024-09-21T10:26:27.158891+0000","last_fullsized":"2024-09-21T10:26:27.158891+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:16:41.951755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":1152,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.262041+0000","last_change":"2024-09-21T10:08:18.468995+0000","last_active":"2024-09-21T10:26:19.262041+0000","last_peered":"2024-09-21T10:26:19.262041+0000","last_clean":"2024-09-21T10:26:19.262041+0000","last_became_active":"2024-09-21T10:08:18.468243+0000","last_became_peered":"2024-09-21T10:08:18.468243+0000","last_unstale":"2024-09-21T10:26:19.262041+0000","last_undegraded":"2024-09-21T10:26:19.262041+0000","last_fullsized":"2024-09-21T10:26:19.262041+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:45.520968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":478,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.262179+0000","last_change":"2024-09-21T10:24:44.309614+0000","last_active":"2024-09-21T10:26:19.262179+0000","last_peered":"2024-09-21T10:26:19.262179+0000","last_clean":"2024-09-21T10:26:19.262179+0000","last_became_active":"2024-09-21T10:24:44.309187+0000","last_became_peered":"2024-09-21T10:24:44.309187+0000","last_unstale":"2024-09-21T10:26:19.262179+0000","last_undegraded":"2024-09-21T10:26:19.262179+0000","last_fullsized":"2024-09-21T10:26:19.262179+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:01.074803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'100","reported_seq":211,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.449456+0000","last_change":"2024-09-21T10:24:44.308946+0000","last_active":"2024-09-21T10:26:19.449456+0000","last_peered":"2024-09-21T10:26:19.449456+0000","last_clean":"2024-09-21T10:26:19.449456+0000","last_became_active":"2024-09-21T10:24:44.308558+0000","last_became_peered":"2024-09-21T10:24:44.308558+0000","last_unstale":"2024-09-21T10:26:19.449456+0000","last_undegraded":"2024-09-21T10:26:19.449456+0000","last_fullsized":"2024-09-21T10:26:19.449456+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:24.536711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:24.536711+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:11:00.667553+0000","scrub_duration":0.0047525650000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":447,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.262032+0000","last_change":"2024-09-21T10:08:18.469907+0000","last_active":"2024-09-21T10:26:19.262032+0000","last_peered":"2024-09-21T10:26:19.262032+0000","last_clean":"2024-09-21T10:26:19.262032+0000","last_became_active":"2024-09-21T10:08:18.469443+0000","last_became_peered":"2024-09-21T10:08:18.469443+0000","last_unstale":"2024-09-21T10:26:19.262032+0000","last_undegraded":"2024-09-21T10:26:19.262032+0000","last_fullsized":"2024-09-21T10:26:19.262032+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:15.645803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":769,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.397301+0000","last_change":"2024-09-21T10:24:44.334847+0000","last_active":"2024-09-21T10:26:19.397301+0000","last_peered":"2024-09-21T10:26:19.397301+0000","last_clean":"2024-09-21T10:26:19.397301+0000","last_became_active":"2024-09-21T10:24:44.301321+0000","last_became_peered":"2024-09-21T10:24:44.301321+0000","last_unstale":"2024-09-21T10:26:19.397301+0000","last_undegraded":"2024-09-21T10:26:19.397301+0000","last_fullsized":"2024-09-21T10:26:19.397301+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:37.628940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":555,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.512729+0000","last_change":"2024-09-21T09:59:54.267798+0000","last_active":"2024-09-21T10:26:27.512729+0000","last_peered":"2024-09-21T10:26:27.512729+0000","last_clean":"2024-09-21T10:26:27.512729+0000","last_became_active":"2024-09-21T09:59:54.265615+0000","last_became_peered":"2024-09-21T09:59:54.265615+0000","last_unstale":"2024-09-21T10:26:27.512729+0000","last_undegraded":"2024-09-21T10:26:27.512729+0000","last_fullsized":"2024-09-21T10:26:27.512729+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:30.363872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"82'100","reported_seq":36,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:20.338845+0000","last_change":"2024-09-21T10:24:44.296758+0000","last_active":"2024-09-21T10:26:20.338845+0000","last_peered":"2024-09-21T10:26:20.338845+0000","last_clean":"2024-09-21T10:26:20.338845+0000","last_became_active":"2024-09-21T10:24:44.296403+0000","last_became_peered":"2024-09-21T10:24:44.296403+0000","last_unstale":"2024-09-21T10:26:20.338845+0000","last_undegraded":"2024-09-21T10:26:20.338845+0000","last_fullsized":"2024-09-21T10:26:20.338845+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.105746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.105746+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:20:12.502081+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":110,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.397458+0000","last_change":"2024-09-21T10:18:33.489678+0000","last_active":"2024-09-21T10:26:19.397458+0000","last_peered":"2024-09-21T10:26:19.397458+0000","last_clean":"2024-09-21T10:26:19.397458+0000","last_became_active":"2024-09-21T10:18:33.474818+0000","last_became_peered":"2024-09-21T10:18:33.474818+0000","last_unstale":"2024-09-21T10:26:19.397458+0000","last_undegraded":"2024-09-21T10:26:19.397458+0000","last_fullsized":"2024-09-21T10:26:19.397458+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:49.828007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":872,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.262367+0000","last_change":"2024-09-21T10:24:46.844278+0000","last_active":"2024-09-21T10:26:19.262367+0000","last_peered":"2024-09-21T10:26:19.262367+0000","last_clean":"2024-09-21T10:26:19.262367+0000","last_became_active":"2024-09-21T10:24:46.840961+0000","last_became_peered":"2024-09-21T10:24:46.840961+0000","last_unstale":"2024-09-21T10:26:19.262367+0000","last_undegraded":"2024-09-21T10:26:19.262367+0000","last_fullsized":"2024-09-21T10:26:19.262367+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:49:48.823524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":139,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.396770+0000","last_change":"2024-09-21T10:18:33.479008+0000","last_active":"2024-09-21T10:26:19.396770+0000","last_peered":"2024-09-21T10:26:19.396770+0000","last_clean":"2024-09-21T10:26:19.396770+0000","last_became_active":"2024-09-21T10:18:33.478740+0000","last_became_peered":"2024-09-21T10:18:33.478740+0000","last_unstale":"2024-09-21T10:26:19.396770+0000","last_undegraded":"2024-09-21T10:26:19.396770+0000","last_fullsized":"2024-09-21T10:26:19.396770+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:16:52.594882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"82'100","reported_seq":462,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.397050+0000","last_change":"2024-09-21T10:23:21.759441+0000","last_active":"2024-09-21T10:26:19.397050+0000","last_peered":"2024-09-21T10:26:19.397050+0000","last_clean":"2024-09-21T10:26:19.397050+0000","last_became_active":"2024-09-21T10:23:20.700452+0000","last_became_peered":"2024-09-21T10:23:20.700452+0000","last_unstale":"2024-09-21T10:26:19.397050+0000","last_undegraded":"2024-09-21T10:26:19.397050+0000","last_fullsized":"2024-09-21T10:26:19.397050+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.759355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.759355+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:21:15.413337+0000","scrub_duration":0.001833559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":614,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.513212+0000","last_change":"2024-09-21T10:24:44.307344+0000","last_active":"2024-09-21T10:26:27.513212+0000","last_peered":"2024-09-21T10:26:27.513212+0000","last_clean":"2024-09-21T10:26:27.513212+0000","last_became_active":"2024-09-21T10:24:44.298936+0000","last_became_peered":"2024-09-21T10:24:44.298936+0000","last_unstale":"2024-09-21T10:26:27.513212+0000","last_undegraded":"2024-09-21T10:26:27.513212+0000","last_fullsized":"2024-09-21T10:26:27.513212+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:58:26.025877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":573,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.449597+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:26:19.449597+0000","last_peered":"2024-09-21T10:26:19.449597+0000","last_clean":"2024-09-21T10:26:19.449597+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:26:19.449597+0000","last_undegraded":"2024-09-21T10:26:19.449597+0000","last_fullsized":"2024-09-21T10:26:19.449597+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":36,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:20.339352+0000","last_change":"2024-09-21T10:24:44.303781+0000","last_active":"2024-09-21T10:26:20.339352+0000","last_peered":"2024-09-21T10:26:20.339352+0000","last_clean":"2024-09-21T10:26:20.339352+0000","last_became_active":"2024-09-21T10:24:44.302058+0000","last_became_peered":"2024-09-21T10:24:44.302058+0000","last_unstale":"2024-09-21T10:26:20.339352+0000","last_undegraded":"2024-09-21T10:26:20.339352+0000","last_fullsized":"2024-09-21T10:26:20.339352+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:30.371229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":36,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:20.338669+0000","last_change":"2024-09-21T10:24:44.303417+0000","last_active":"2024-09-21T10:26:20.338669+0000","last_peered":"2024-09-21T10:26:20.338669+0000","last_clean":"2024-09-21T10:26:20.338669+0000","last_became_active":"2024-09-21T10:24:44.303069+0000","last_became_peered":"2024-09-21T10:24:44.303069+0000","last_unstale":"2024-09-21T10:26:20.338669+0000","last_undegraded":"2024-09-21T10:26:20.338669+0000","last_fullsized":"2024-09-21T10:26:20.338669+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:30:30.966228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":46,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:20.339475+0000","last_change":"2024-09-21T10:24:44.321421+0000","last_active":"2024-09-21T10:26:20.339475+0000","last_peered":"2024-09-21T10:26:20.339475+0000","last_clean":"2024-09-21T10:26:20.339475+0000","last_became_active":"2024-09-21T10:24:44.297082+0000","last_became_peered":"2024-09-21T10:24:44.297082+0000","last_unstale":"2024-09-21T10:26:20.339475+0000","last_undegraded":"2024-09-21T10:26:20.339475+0000","last_fullsized":"2024-09-21T10:26:20.339475+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:35:37.704881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":446,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.262046+0000","last_change":"2024-09-21T10:08:18.470378+0000","last_active":"2024-09-21T10:26:19.262046+0000","last_peered":"2024-09-21T10:26:19.262046+0000","last_clean":"2024-09-21T10:26:19.262046+0000","last_became_active":"2024-09-21T10:08:18.469482+0000","last_became_peered":"2024-09-21T10:08:18.469482+0000","last_unstale":"2024-09-21T10:26:19.262046+0000","last_undegraded":"2024-09-21T10:26:19.262046+0000","last_fullsized":"2024-09-21T10:26:19.262046+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:35.628724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":394,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.397177+0000","last_change":"2024-09-21T10:24:44.313834+0000","last_active":"2024-09-21T10:26:19.397177+0000","last_peered":"2024-09-21T10:26:19.397177+0000","last_clean":"2024-09-21T10:26:19.397177+0000","last_became_active":"2024-09-21T10:24:44.299135+0000","last_became_peered":"2024-09-21T10:24:44.299135+0000","last_unstale":"2024-09-21T10:26:19.397177+0000","last_undegraded":"2024-09-21T10:26:19.397177+0000","last_fullsized":"2024-09-21T10:26:19.397177+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:21:51.959148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":93,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.159030+0000","last_change":"2024-09-21T10:22:39.385291+0000","last_active":"2024-09-21T10:26:27.159030+0000","last_peered":"2024-09-21T10:26:27.159030+0000","last_clean":"2024-09-21T10:26:27.159030+0000","last_became_active":"2024-09-21T10:22:39.384875+0000","last_became_peered":"2024-09-21T10:22:39.384875+0000","last_unstale":"2024-09-21T10:26:27.159030+0000","last_undegraded":"2024-09-21T10:26:27.159030+0000","last_fullsized":"2024-09-21T10:26:27.159030+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:33:30.610497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":99,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.449132+0000","last_change":"2024-09-21T10:18:33.464989+0000","last_active":"2024-09-21T10:26:19.449132+0000","last_peered":"2024-09-21T10:26:19.449132+0000","last_clean":"2024-09-21T10:26:19.449132+0000","last_became_active":"2024-09-21T10:18:33.464617+0000","last_became_peered":"2024-09-21T10:18:33.464617+0000","last_unstale":"2024-09-21T10:26:19.449132+0000","last_undegraded":"2024-09-21T10:26:19.449132+0000","last_fullsized":"2024-09-21T10:26:19.449132+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:09.300797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":572,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.513563+0000","last_change":"2024-09-21T10:18:33.472415+0000","last_active":"2024-09-21T10:26:27.513563+0000","last_peered":"2024-09-21T10:26:27.513563+0000","last_clean":"2024-09-21T10:26:27.513563+0000","last_became_active":"2024-09-21T10:18:33.471746+0000","last_became_peered":"2024-09-21T10:18:33.471746+0000","last_unstale":"2024-09-21T10:26:27.513563+0000","last_undegraded":"2024-09-21T10:26:27.513563+0000","last_fullsized":"2024-09-21T10:26:27.513563+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:40:37.099458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":223,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.261743+0000","last_change":"2024-09-21T10:24:44.310195+0000","last_active":"2024-09-21T10:26:19.261743+0000","last_peered":"2024-09-21T10:26:19.261743+0000","last_clean":"2024-09-21T10:26:19.261743+0000","last_became_active":"2024-09-21T10:24:44.309600+0000","last_became_peered":"2024-09-21T10:24:44.309600+0000","last_unstale":"2024-09-21T10:26:19.261743+0000","last_undegraded":"2024-09-21T10:26:19.261743+0000","last_fullsized":"2024-09-21T10:26:19.261743+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:29:15.459007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":310,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.449552+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:26:19.449552+0000","last_peered":"2024-09-21T10:26:19.449552+0000","last_clean":"2024-09-21T10:26:19.449552+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:26:19.449552+0000","last_undegraded":"2024-09-21T10:26:19.449552+0000","last_fullsized":"2024-09-21T10:26:19.449552+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":237,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.512874+0000","last_change":"2024-09-21T10:18:33.479635+0000","last_active":"2024-09-21T10:26:27.512874+0000","last_peered":"2024-09-21T10:26:27.512874+0000","last_clean":"2024-09-21T10:26:27.512874+0000","last_became_active":"2024-09-21T10:18:33.478830+0000","last_became_peered":"2024-09-21T10:18:33.478830+0000","last_unstale":"2024-09-21T10:26:27.512874+0000","last_undegraded":"2024-09-21T10:26:27.512874+0000","last_fullsized":"2024-09-21T10:26:27.512874+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:16.950086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":588,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.513425+0000","last_change":"2024-09-21T10:18:33.480350+0000","last_active":"2024-09-21T10:26:27.513425+0000","last_peered":"2024-09-21T10:26:27.513425+0000","last_clean":"2024-09-21T10:26:27.513425+0000","last_became_active":"2024-09-21T10:18:33.479248+0000","last_became_peered":"2024-09-21T10:18:33.479248+0000","last_unstale":"2024-09-21T10:26:27.513425+0000","last_undegraded":"2024-09-21T10:26:27.513425+0000","last_fullsized":"2024-09-21T10:26:27.513425+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:25:28.780572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":586,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.261795+0000","last_change":"2024-09-21T10:24:44.309248+0000","last_active":"2024-09-21T10:26:19.261795+0000","last_peered":"2024-09-21T10:26:19.261795+0000","last_clean":"2024-09-21T10:26:19.261795+0000","last_became_active":"2024-09-21T10:24:44.308925+0000","last_became_peered":"2024-09-21T10:24:44.308925+0000","last_unstale":"2024-09-21T10:26:19.261795+0000","last_undegraded":"2024-09-21T10:26:19.261795+0000","last_fullsized":"2024-09-21T10:26:19.261795+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:40.267676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":347,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.262062+0000","last_change":"2024-09-21T10:24:44.308839+0000","last_active":"2024-09-21T10:26:19.262062+0000","last_peered":"2024-09-21T10:26:19.262062+0000","last_clean":"2024-09-21T10:26:19.262062+0000","last_became_active":"2024-09-21T10:24:44.308488+0000","last_became_peered":"2024-09-21T10:24:44.308488+0000","last_unstale":"2024-09-21T10:26:19.262062+0000","last_undegraded":"2024-09-21T10:26:19.262062+0000","last_fullsized":"2024-09-21T10:26:19.262062+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:22:11.961205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":36,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:20.338765+0000","last_change":"2024-09-21T10:24:44.305124+0000","last_active":"2024-09-21T10:26:20.338765+0000","last_peered":"2024-09-21T10:26:20.338765+0000","last_clean":"2024-09-21T10:26:20.338765+0000","last_became_active":"2024-09-21T10:24:44.304468+0000","last_became_peered":"2024-09-21T10:24:44.304468+0000","last_unstale":"2024-09-21T10:26:20.338765+0000","last_undegraded":"2024-09-21T10:26:20.338765+0000","last_fullsized":"2024-09-21T10:26:20.338765+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:37.904764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":227,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.158924+0000","last_change":"2024-09-21T10:18:33.480128+0000","last_active":"2024-09-21T10:26:27.158924+0000","last_peered":"2024-09-21T10:26:27.158924+0000","last_clean":"2024-09-21T10:26:27.158924+0000","last_became_active":"2024-09-21T10:18:33.479691+0000","last_became_peered":"2024-09-21T10:18:33.479691+0000","last_unstale":"2024-09-21T10:26:27.158924+0000","last_undegraded":"2024-09-21T10:26:27.158924+0000","last_fullsized":"2024-09-21T10:26:27.158924+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:43:49.311244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":585,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.513260+0000","last_change":"2024-09-21T10:24:44.299989+0000","last_active":"2024-09-21T10:26:27.513260+0000","last_peered":"2024-09-21T10:26:27.513260+0000","last_clean":"2024-09-21T10:26:27.513260+0000","last_became_active":"2024-09-21T10:24:44.299356+0000","last_became_peered":"2024-09-21T10:24:44.299356+0000","last_unstale":"2024-09-21T10:26:27.513260+0000","last_undegraded":"2024-09-21T10:26:27.513260+0000","last_fullsized":"2024-09-21T10:26:27.513260+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:58.867008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":36,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:20.339083+0000","last_change":"2024-09-21T10:24:44.304040+0000","last_active":"2024-09-21T10:26:20.339083+0000","last_peered":"2024-09-21T10:26:20.339083+0000","last_clean":"2024-09-21T10:26:20.339083+0000","last_became_active":"2024-09-21T10:24:44.303666+0000","last_became_peered":"2024-09-21T10:24:44.303666+0000","last_unstale":"2024-09-21T10:26:20.339083+0000","last_undegraded":"2024-09-21T10:26:20.339083+0000","last_fullsized":"2024-09-21T10:26:20.339083+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:19:35.061997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":40,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:20.339251+0000","last_change":"2024-09-21T10:24:44.329150+0000","last_active":"2024-09-21T10:26:20.339251+0000","last_peered":"2024-09-21T10:26:20.339251+0000","last_clean":"2024-09-21T10:26:20.339251+0000","last_became_active":"2024-09-21T10:24:44.306752+0000","last_became_peered":"2024-09-21T10:24:44.306752+0000","last_unstale":"2024-09-21T10:26:20.339251+0000","last_undegraded":"2024-09-21T10:26:20.339251+0000","last_fullsized":"2024-09-21T10:26:20.339251+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:16.592859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":36,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:20.338639+0000","last_change":"2024-09-21T10:24:44.298825+0000","last_active":"2024-09-21T10:26:20.338639+0000","last_peered":"2024-09-21T10:26:20.338639+0000","last_clean":"2024-09-21T10:26:20.338639+0000","last_became_active":"2024-09-21T10:24:44.298517+0000","last_became_peered":"2024-09-21T10:24:44.298517+0000","last_unstale":"2024-09-21T10:26:20.338639+0000","last_undegraded":"2024-09-21T10:26:20.338639+0000","last_fullsized":"2024-09-21T10:26:20.338639+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:10.288494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":40,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:20.338953+0000","last_change":"2024-09-21T10:24:44.324087+0000","last_active":"2024-09-21T10:26:20.338953+0000","last_peered":"2024-09-21T10:26:20.338953+0000","last_clean":"2024-09-21T10:26:20.338953+0000","last_became_active":"2024-09-21T10:24:44.298665+0000","last_became_peered":"2024-09-21T10:24:44.298665+0000","last_unstale":"2024-09-21T10:26:20.338953+0000","last_undegraded":"2024-09-21T10:26:20.338953+0000","last_fullsized":"2024-09-21T10:26:20.338953+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:59.819340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":240,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.158749+0000","last_change":"2024-09-21T10:18:33.474216+0000","last_active":"2024-09-21T10:26:27.158749+0000","last_peered":"2024-09-21T10:26:27.158749+0000","last_clean":"2024-09-21T10:26:27.158749+0000","last_became_active":"2024-09-21T10:18:33.473794+0000","last_became_peered":"2024-09-21T10:18:33.473794+0000","last_unstale":"2024-09-21T10:26:27.158749+0000","last_undegraded":"2024-09-21T10:26:27.158749+0000","last_fullsized":"2024-09-21T10:26:27.158749+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:38.956888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":95,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:19.449532+0000","last_change":"2024-09-21T10:18:33.483576+0000","last_active":"2024-09-21T10:26:19.449532+0000","last_peered":"2024-09-21T10:26:19.449532+0000","last_clean":"2024-09-21T10:26:19.449532+0000","last_became_active":"2024-09-21T10:18:33.483116+0000","last_became_peered":"2024-09-21T10:18:33.483116+0000","last_unstale":"2024-09-21T10:26:19.449532+0000","last_undegraded":"2024-09-21T10:26:19.449532+0000","last_fullsized":"2024-09-21T10:26:19.449532+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:48.964766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":36,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-21T10:26:20.338602+0000","last_change":"2024-09-21T10:24:44.298820+0000","last_active":"2024-09-21T10:26:20.338602+0000","last_peered":"2024-09-21T10:26:20.338602+0000","last_clean":"2024-09-21T10:26:20.338602+0000","last_became_active":"2024-09-21T10:24:44.298478+0000","last_became_peered":"2024-09-21T10:24:44.298478+0000","last_unstale":"2024-09-21T10:26:20.338602+0000","last_undegraded":"2024-09-21T10:26:20.338602+0000","last_fullsized":"2024-09-21T10:26:20.338602+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:59.567078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":570,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-21T10:26:27.513408+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:26:27.513408+0000","last_peered":"2024-09-21T10:26:27.513408+0000","last_clean":"2024-09-21T10:26:27.513408+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:26:27.513408+0000","last_undegraded":"2024-09-21T10:26:27.513408+0000","last_fullsized":"2024-09-21T10:26:27.513408+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":10162,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5982,"num_read_kb":13005,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1100,"ondisk_log_size":1100,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397940,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":901972,"kb_used_data":8904,"kb_used_omap":1,"kb_used_meta":893054,"kb_avail":92843180,"statfs":{"total":95995035648,"available":95071416320,"internally_reserved":0,"allocated":9117696,"data_stored":7193675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":914487640},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594168,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":963024,"kb_used_data":53380,"kb_used_omap":1,"kb_used_meta":909630,"kb_avail":92782128,"statfs":{"total":95995035648,"available":95008899072,"internally_reserved":0,"allocated":54661120,"data_stored":52741612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931461464},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790398,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":904900,"kb_used_data":9144,"kb_used_omap":1,"kb_used_meta":895742,"kb_avail":92840252,"statfs":{"total":95995035648,"available":95068418048,"internally_reserved":0,"allocated":9363456,"data_stored":7451747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":917240152},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986627,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310160,"kb_used_data":4484,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93434992,"statfs":{"total":95995035648,"available":95677431808,"internally_reserved":0,"allocated":4591616,"data_stored":2671613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":0,"seq":433791697189,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860716,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":858820,"kb_used_data":8824,"kb_used_omap":8,"kb_used_meta":849975,"kb_avail":92886332,"statfs":{"total":95995035648,"available":95115603968,"internally_reserved":0,"allocated":9035776,"data_stored":7124347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":870375276},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024241,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":771300,"kb_used_data":4440,"kb_used_omap":7,"kb_used_meta":766840,"kb_avail":92973852,"statfs":{"total":95995035648,"available":95205224448,"internally_reserved":0,"allocated":4546560,"data_stored":2657682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":785244722},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187777,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":843252,"kb_used_data":4456,"kb_used_omap":8,"kb_used_meta":838775,"kb_avail":92901900,"statfs":{"total":95995035648,"available":95131545600,"internally_reserved":0,"allocated":4562944,"data_stored":2658975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":858906609},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:26:33.421 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T10:26:33.421 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:26:33.422 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T10:26:33.422 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.710506 2024-09-21T10:26:33.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 5 0.7105059327514174 2024-09-21T10:26:33.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.423 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:33.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:33.524 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:33.544 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:33.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:33.646 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:33.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.667 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:33.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:33.705 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:26:33.769 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:33.809 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:33.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:33.911 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:33.932 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:33.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:34.011 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:26:34.033 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:34.054 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:34.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:34.156 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:34.178 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:34.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:34.280 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:34.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.302 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:34.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:34.403 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:34.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.425 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:34.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:34.526 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:34.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:34.648 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:34.669 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:34.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:34.771 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:34.792 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:34.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:34.893 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:34.915 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:34.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:35.016 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:35.039 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:35.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:35.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:35.043 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:35.142 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:35.164 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:35.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:35.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:35.181 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'unique_pool_5' created 2024-09-21T10:26:35.181 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:26:35.266 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:35.287 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:35.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:35.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:35.390 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:35.412 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:35.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:35.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:35.438 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:35 smithi196 ceph-mon[84784]: pgmap v1166: 83 pgs: 83 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:35.438 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:35 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/836258828' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:35.438 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:35 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:35.438 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:35 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2208742448' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-21T10:26:35.438 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:35 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-21T10:26:35.514 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:35.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:35.537 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:35.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:35.552 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:35 smithi098 ceph-mon[97580]: pgmap v1166: 83 pgs: 83 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:35.552 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:35 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/836258828' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:35.552 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:35 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:35.552 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:35 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2208742448' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-21T10:26:35.553 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:35 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-21T10:26:35.553 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:35 smithi098 ceph-mon[100914]: pgmap v1166: 83 pgs: 83 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:35.553 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:35 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/836258828' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:35.553 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:35 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:35.553 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:35 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2208742448' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-21T10:26:35.553 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:35 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-21T10:26:35.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:35.661 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:35.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:35.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:35.676 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:26:35.676 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:26:35.762 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:35.783 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:35.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:35.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:35.885 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:35.907 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:35.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:35.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:36.009 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:36.030 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:36.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:36.132 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:36.154 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:36.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:36.255 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:36.278 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:36.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:36.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:36 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:26:36.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:36 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:26:36.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:36 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:26:36.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:36 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-21T10:26:36.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:36 smithi098 ceph-mon[97580]: osdmap e348: 8 total, 8 up, 7 in 2024-09-21T10:26:36.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:36 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:26:36.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:36 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:26:36.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:36 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:26:36.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:36 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-21T10:26:36.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:36 smithi098 ceph-mon[100914]: osdmap e348: 8 total, 8 up, 7 in 2024-09-21T10:26:36.379 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:36.400 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:36.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:36.436 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:36 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:26:36.436 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:36 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:26:36.436 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:36 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:26:36.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:36 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-21T10:26:36.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:36 smithi196 ceph-mon[84784]: osdmap e348: 8 total, 8 up, 7 in 2024-09-21T10:26:36.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:26:36.502 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:36.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:36.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:36.646 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:36.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:36.720 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:26:36.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:36.769 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:36.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:36.871 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:36.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:36.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:37.015 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:37.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:37.117 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:37.138 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:37.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:37.240 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:37.263 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:37.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:37.362 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:37 smithi098 ceph-mon[97580]: pgmap v1168: 99 pgs: 16 unknown, 83 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:37.362 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:37 smithi098 ceph-mon[97580]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:26:37.362 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:37 smithi098 ceph-mon[97580]: osdmap e349: 8 total, 8 up, 7 in 2024-09-21T10:26:37.362 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:37 smithi098 ceph-mon[100914]: pgmap v1168: 99 pgs: 16 unknown, 83 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:37.363 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:37 smithi098 ceph-mon[100914]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:26:37.363 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:37 smithi098 ceph-mon[100914]: osdmap e349: 8 total, 8 up, 7 in 2024-09-21T10:26:37.364 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-09-21T10:26:37.364 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:37.387 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:37.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:37.488 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:37.505 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:37 smithi196 ceph-mon[84784]: pgmap v1168: 99 pgs: 16 unknown, 83 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:37.505 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:37 smithi196 ceph-mon[84784]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:26:37.505 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:37 smithi196 ceph-mon[84784]: osdmap e349: 8 total, 8 up, 7 in 2024-09-21T10:26:37.511 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:37.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:37.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:37.634 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:37.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:37.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:37.758 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:37.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:37.860 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:37.882 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:37.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:37.984 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:38.005 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:38.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:38.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:38.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:38.128 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:38.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:38.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:38.229 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:38.252 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:38.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:38.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:38.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:38 smithi098 ceph-mon[100914]: osdmap e350: 8 total, 8 up, 7 in 2024-09-21T10:26:38.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:38.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:38.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:38 smithi098 ceph-mon[97580]: osdmap e350: 8 total, 8 up, 7 in 2024-09-21T10:26:38.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:38.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:38.342 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:26:38.342 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:26:38.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:38.375 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:38.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:38.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:38 smithi196 ceph-mon[84784]: osdmap e350: 8 total, 8 up, 7 in 2024-09-21T10:26:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:38.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:38.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:38.498 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:38.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:38.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:38.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:38.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:38.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:38.791 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:38.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:38.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:38.892 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:38.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:38.914 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:38.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:38.972 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:26:39.015 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:39.036 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:39.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:39.137 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:39.160 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:39.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:39.196 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:26:39.196 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:26:39.197 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:26:39.197 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:26:39.197 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:26:39.197 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:26:39.197 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:26:39.197 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:26:39.197 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:26:39.198 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:26:39.198 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:26:39.198 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:26:39.198 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:26:39.199 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:26:39.216 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:26:39.216 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:26:39.220 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:26:39.262 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:39.283 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:39.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:39.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:39 smithi098 ceph-mon[97580]: pgmap v1171: 99 pgs: 15 unknown, 84 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:39.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:39 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/460402686' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:39.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:39 smithi098 ceph-mon[100914]: pgmap v1171: 99 pgs: 15 unknown, 84 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:39.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:39 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/460402686' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:39.385 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:39.407 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:39.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:39.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:39 smithi196 ceph-mon[84784]: pgmap v1171: 99 pgs: 15 unknown, 84 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:39.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:39 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/460402686' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:39.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:39.531 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:39.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:39.633 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:39.655 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:39.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:39.756 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:39.777 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:39.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:39.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:39.901 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:39.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:40.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:40.023 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:40.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:40.125 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:40.146 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:40.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:40.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:40.274 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:40.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:40.338 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.5 primary-affinity to 0.710506 (8465632) 2024-09-21T10:26:40.347 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:26:40.375 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:40.402 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:40 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1600010211' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.7105059327514174} : dispatch 2024-09-21T10:26:40.402 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:26:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:26:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:26:40.412 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:40.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:40.514 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:40.535 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:40.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:40 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1600010211' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.7105059327514174} : dispatch 2024-09-21T10:26:40.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:40 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1600010211' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.7105059327514174} : dispatch 2024-09-21T10:26:40.637 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:40.660 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:40.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:40.761 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:40.783 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:40.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:40.885 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:40.908 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:40.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:40.933 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:26:40.933 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-21T10:26:40.933 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-21T10:26:40.934 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:26:40.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.934 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.55000.0:214626 7.12 7.9d8ee1d2 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e255)", 2024-09-21T10:26:40.934 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:16:45.316385+0000", 2024-09-21T10:26:40.934 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 595.61056695599996, 2024-09-21T10:26:40.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.38870583199999997, 2024-09-21T10:26:40.934 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:40.934 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:26:40.934 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:40.934 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.55000", 2024-09-21T10:26:40.934 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/833849289", 2024-09-21T10:26:40.934 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 214626 2024-09-21T10:26:40.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.935 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:40.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:40.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:16:45.316385+0000", 2024-09-21T10:26:40.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:40.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:40.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:16:45.316385+0000", 2024-09-21T10:26:40.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:40.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:40.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:16:45.316383+0000", 2024-09-21T10:26:40.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:40.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:40.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:16:45.316391+0000", 2024-09-21T10:26:40.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2840000000000001e-06 2024-09-21T10:26:40.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:40.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:16:45.316407+0000", 2024-09-21T10:26:40.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5947000000000002e-05 2024-09-21T10:26:40.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:40.937 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:16:45.316413+0000", 2024-09-21T10:26:40.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0140000000000004e-06 2024-09-21T10:26:40.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.937 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.937 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:40.937 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:16:45.316455+0000", 2024-09-21T10:26:40.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2064000000000002e-05 2024-09-21T10:26:40.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.937 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.937 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:40.937 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:16:45.316462+0000", 2024-09-21T10:26:40.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9609999999999999e-06 2024-09-21T10:26:40.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.937 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.937 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:40.937 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:16:45.705080+0000", 2024-09-21T10:26:40.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.388618456 2024-09-21T10:26:40.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:40.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:16:45.705091+0000", 2024-09-21T10:26:40.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0689e-05 2024-09-21T10:26:40.938 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:40.938 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:40.938 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:40.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.938 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181812 9.1 9:8268420f:::benchmark_data_smithi098_254407_object49081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T10:26:40.938 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.268689+0000", 2024-09-21T10:26:40.938 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 117.65826279300001, 2024-09-21T10:26:40.939 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1774718389999999, 2024-09-21T10:26:40.939 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:40.939 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:40.939 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:40.939 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:40.939 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:40.939 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181812 2024-09-21T10:26:40.939 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.939 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:40.939 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.939 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:40.939 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268689+0000", 2024-09-21T10:26:40.939 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:40.939 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.939 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.939 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:40.940 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268689+0000", 2024-09-21T10:26:40.940 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:40.940 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.940 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.940 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:40.940 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268688+0000", 2024-09-21T10:26:40.940 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:40.940 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.940 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.940 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:40.940 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268695+0000", 2024-09-21T10:26:40.940 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6420000000000007e-06 2024-09-21T10:26:40.940 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:40.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268710+0000", 2024-09-21T10:26:40.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4959e-05 2024-09-21T10:26:40.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:40.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268716+0000", 2024-09-21T10:26:40.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0290000000000002e-06 2024-09-21T10:26:40.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:40.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.273397+0000", 2024-09-21T10:26:40.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046805370000000002 2024-09-21T10:26:40.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:40.942 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.273407+0000", 2024-09-21T10:26:40.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6970000000000005e-06 2024-09-21T10:26:40.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:40.942 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.292290+0000", 2024-09-21T10:26:40.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0188830849999999 2024-09-21T10:26:40.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:40.943 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.292305+0000", 2024-09-21T10:26:40.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4841000000000001e-05 2024-09-21T10:26:40.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.943 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.943 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:40.943 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.432527+0000", 2024-09-21T10:26:40.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1402229049999999 2024-09-21T10:26:40.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.943 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.943 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:40.943 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.432648+0000", 2024-09-21T10:26:40.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000120538 2024-09-21T10:26:40.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.943 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,4,7", 2024-09-21T10:26:40.944 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.432762+0000", 2024-09-21T10:26:40.944 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114478 2024-09-21T10:26:40.944 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:40.944 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.440164+0000", 2024-09-21T10:26:40.944 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0074017559999999998 2024-09-21T10:26:40.944 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:40.944 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.440418+0000", 2024-09-21T10:26:40.944 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025371399999999999 2024-09-21T10:26:40.944 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.945 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.945 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:40.945 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.440476+0000", 2024-09-21T10:26:40.945 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8146999999999997e-05 2024-09-21T10:26:40.945 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.945 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.945 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:40.945 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.445996+0000", 2024-09-21T10:26:40.945 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055200129999999998 2024-09-21T10:26:40.945 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.945 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.945 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:40.945 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.446054+0000", 2024-09-21T10:26:40.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8400999999999999e-05 2024-09-21T10:26:40.946 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.946 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.946 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:40.946 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.446161+0000", 2024-09-21T10:26:40.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000106586 2024-09-21T10:26:40.946 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:40.946 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:40.946 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:40.947 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.947 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.947 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181813 9.1 9:877e8199:::benchmark_data_smithi098_254407_object49082:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T10:26:40.947 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.268726+0000", 2024-09-21T10:26:40.947 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 117.658225735, 2024-09-21T10:26:40.947 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1922379680000001, 2024-09-21T10:26:40.947 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:40.947 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:40.947 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:40.947 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:40.947 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:40.947 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181813 2024-09-21T10:26:40.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.948 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:40.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:40.948 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268726+0000", 2024-09-21T10:26:40.948 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:40.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:40.948 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268726+0000", 2024-09-21T10:26:40.948 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:40.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:40.949 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268726+0000", 2024-09-21T10:26:40.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:26:40.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.949 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:40.949 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268728+0000", 2024-09-21T10:26:40.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7499999999999999e-06 2024-09-21T10:26:40.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.949 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:40.949 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268735+0000", 2024-09-21T10:26:40.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8700000000000003e-06 2024-09-21T10:26:40.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.950 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:40.950 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268739+0000", 2024-09-21T10:26:40.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8419999999999998e-06 2024-09-21T10:26:40.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.950 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:40.950 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.273431+0000", 2024-09-21T10:26:40.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046914460000000002 2024-09-21T10:26:40.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.950 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:40.950 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.273438+0000", 2024-09-21T10:26:40.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5809999999999999e-06 2024-09-21T10:26:40.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:40.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.292343+0000", 2024-09-21T10:26:40.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.018904467 2024-09-21T10:26:40.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:40.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.292352+0000", 2024-09-21T10:26:40.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0280000000000006e-06 2024-09-21T10:26:40.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:40.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.436686+0000", 2024-09-21T10:26:40.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1443347740000001 2024-09-21T10:26:40.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.952 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.952 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:40.952 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.436801+0000", 2024-09-21T10:26:40.952 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011475499999999999 2024-09-21T10:26:40.952 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.952 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.952 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,4,7", 2024-09-21T10:26:40.952 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.436931+0000", 2024-09-21T10:26:40.952 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012945200000000001 2024-09-21T10:26:40.952 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.952 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.952 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:40.952 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.440424+0000", 2024-09-21T10:26:40.952 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003493515 2024-09-21T10:26:40.953 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.953 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:40.953 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.444139+0000", 2024-09-21T10:26:40.953 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0037150170000000001 2024-09-21T10:26:40.953 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.953 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:40.953 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.454482+0000", 2024-09-21T10:26:40.953 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010343140000000001 2024-09-21T10:26:40.953 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.953 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:40.953 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.460725+0000", 2024-09-21T10:26:40.953 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0062427510000000004 2024-09-21T10:26:40.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:40.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.460845+0000", 2024-09-21T10:26:40.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119721 2024-09-21T10:26:40.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:40.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.460964+0000", 2024-09-21T10:26:40.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119419 2024-09-21T10:26:40.954 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:40.954 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:40.954 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:40.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.955 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212859 9.1 9:81de2bc9:::benchmark_data_smithi098_254407_object80128:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-21T10:26:40.955 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.362389+0000", 2024-09-21T10:26:40.955 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 82.564563093000004, 2024-09-21T10:26:40.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.029750589, 2024-09-21T10:26:40.955 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:40.955 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:26:40.955 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:40.955 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:40.955 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:40.955 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212859 2024-09-21T10:26:40.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.955 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:40.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.955 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:40.955 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362389+0000", 2024-09-21T10:26:40.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:40.956 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.956 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.956 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:40.956 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362389+0000", 2024-09-21T10:26:40.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:40.956 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.956 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.956 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:40.956 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362387+0000", 2024-09-21T10:26:40.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:40.956 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.956 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.956 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:40.956 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362395+0000", 2024-09-21T10:26:40.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6410000000000002e-06 2024-09-21T10:26:40.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.957 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:40.957 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362409+0000", 2024-09-21T10:26:40.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3872e-05 2024-09-21T10:26:40.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.957 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:40.957 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362415+0000", 2024-09-21T10:26:40.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0650000000000004e-06 2024-09-21T10:26:40.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.957 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:40.957 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362480+0000", 2024-09-21T10:26:40.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5152000000000002e-05 2024-09-21T10:26:40.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:40.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362601+0000", 2024-09-21T10:26:40.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000121327 2024-09-21T10:26:40.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,4,7", 2024-09-21T10:26:40.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362727+0000", 2024-09-21T10:26:40.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012533800000000001 2024-09-21T10:26:40.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:40.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.369166+0000", 2024-09-21T10:26:40.959 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006439274 2024-09-21T10:26:40.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.959 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:40.959 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.369292+0000", 2024-09-21T10:26:40.959 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012620200000000001 2024-09-21T10:26:40.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.959 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:40.959 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.369926+0000", 2024-09-21T10:26:40.959 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00063336299999999996 2024-09-21T10:26:40.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.959 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:40.959 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.374921+0000", 2024-09-21T10:26:40.960 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049951869999999999 2024-09-21T10:26:40.960 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.960 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.960 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:40.960 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.374935+0000", 2024-09-21T10:26:40.960 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3872e-05 2024-09-21T10:26:40.960 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.960 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.960 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:40.960 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.392097+0000", 2024-09-21T10:26:40.960 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0171621260000001 2024-09-21T10:26:40.960 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.960 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.960 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:40.961 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.392140+0000", 2024-09-21T10:26:40.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2700000000000001e-05 2024-09-21T10:26:40.961 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:40.961 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:40.961 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:40.961 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.961 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212886 9.1 9:80fef226:::benchmark_data_smithi098_254407_object80155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:26:40.961 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.391416+0000", 2024-09-21T10:26:40.961 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 82.535536426999997, 2024-09-21T10:26:40.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.049473646, 2024-09-21T10:26:40.961 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:40.961 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:40.961 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:40.962 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:40.962 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:40.962 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212886 2024-09-21T10:26:40.962 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.962 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:40.962 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.962 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:40.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391416+0000", 2024-09-21T10:26:40.962 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:40.962 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.962 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.962 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:40.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391416+0000", 2024-09-21T10:26:40.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:40.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.963 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:40.963 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391414+0000", 2024-09-21T10:26:40.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:40.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.963 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:40.963 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391422+0000", 2024-09-21T10:26:40.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2369999999999992e-06 2024-09-21T10:26:40.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.963 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:40.964 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391441+0000", 2024-09-21T10:26:40.964 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9313999999999998e-05 2024-09-21T10:26:40.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.964 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.964 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:40.964 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391448+0000", 2024-09-21T10:26:40.964 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8399999999999997e-06 2024-09-21T10:26:40.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.964 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.964 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:40.964 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391577+0000", 2024-09-21T10:26:40.964 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012900699999999999 2024-09-21T10:26:40.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.964 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.965 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:40.965 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391596+0000", 2024-09-21T10:26:40.965 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9448000000000001e-05 2024-09-21T10:26:40.965 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.965 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.965 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:40.965 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.392163+0000", 2024-09-21T10:26:40.965 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0005668080000001 2024-09-21T10:26:40.965 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.965 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.965 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:40.965 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.392226+0000", 2024-09-21T10:26:40.965 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2736000000000004e-05 2024-09-21T10:26:40.966 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.966 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.966 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:26:40.966 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.392302+0000", 2024-09-21T10:26:40.966 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6459999999999996e-05 2024-09-21T10:26:40.966 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.966 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.966 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:40.966 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.427385+0000", 2024-09-21T10:26:40.966 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.035082852999999997 2024-09-21T10:26:40.966 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.966 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.967 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:40.967 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440761+0000", 2024-09-21T10:26:40.967 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013375398 2024-09-21T10:26:40.967 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.967 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:40.967 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440797+0000", 2024-09-21T10:26:40.967 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6238000000000001e-05 2024-09-21T10:26:40.967 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.967 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:40.967 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440845+0000", 2024-09-21T10:26:40.967 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7781999999999998e-05 2024-09-21T10:26:40.967 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.968 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:40.968 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440889+0000", 2024-09-21T10:26:40.968 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4515999999999999e-05 2024-09-21T10:26:40.968 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:40.968 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:40.968 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:40.968 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.968 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.968 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212893 9.1 9:867ca339:::benchmark_data_smithi098_254407_object80162:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:26:40.968 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.401222+0000", 2024-09-21T10:26:40.968 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 82.525730041000003, 2024-09-21T10:26:40.968 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.039903265, 2024-09-21T10:26:40.968 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:40.968 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:40.968 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:40.969 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:40.969 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:40.969 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212893 2024-09-21T10:26:40.969 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.969 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:40.969 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.969 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:40.969 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401222+0000", 2024-09-21T10:26:40.969 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:40.969 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.969 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.969 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:40.969 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401222+0000", 2024-09-21T10:26:40.969 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:40.970 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:40.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401220+0000", 2024-09-21T10:26:40.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:40.970 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:40.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401228+0000", 2024-09-21T10:26:40.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3100000000000003e-06 2024-09-21T10:26:40.970 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:40.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401245+0000", 2024-09-21T10:26:40.971 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7504999999999999e-05 2024-09-21T10:26:40.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.971 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:40.971 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401252+0000", 2024-09-21T10:26:40.971 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5679999999999996e-06 2024-09-21T10:26:40.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.971 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:40.971 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401303+0000", 2024-09-21T10:26:40.971 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1232e-05 2024-09-21T10:26:40.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.971 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:40.972 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401313+0000", 2024-09-21T10:26:40.972 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0023999999999999e-05 2024-09-21T10:26:40.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.972 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:40.972 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.395122+0000", 2024-09-21T10:26:40.972 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99380883799999997 2024-09-21T10:26:40.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.972 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:40.972 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.395230+0000", 2024-09-21T10:26:40.972 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107917 2024-09-21T10:26:40.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.972 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:26:40.973 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.395302+0000", 2024-09-21T10:26:40.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1963000000000002e-05 2024-09-21T10:26:40.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.973 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.973 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:40.973 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.427393+0000", 2024-09-21T10:26:40.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.032091732999999997 2024-09-21T10:26:40.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.973 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.973 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:40.973 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440948+0000", 2024-09-21T10:26:40.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013554639 2024-09-21T10:26:40.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.973 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.974 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:40.974 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441060+0000", 2024-09-21T10:26:40.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011248599999999999 2024-09-21T10:26:40.974 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.974 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:40.974 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441101+0000", 2024-09-21T10:26:40.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0738999999999997e-05 2024-09-21T10:26:40.974 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.975 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.975 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:40.975 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441125+0000", 2024-09-21T10:26:40.975 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4076000000000001e-05 2024-09-21T10:26:40.975 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:40.976 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:40.976 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:40.976 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.976 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212896 9.1 9:84f721c8:::benchmark_data_smithi098_254407_object80165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:26:40.976 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.404604+0000", 2024-09-21T10:26:40.976 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 82.522347648999997, 2024-09-21T10:26:40.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0367745399999999, 2024-09-21T10:26:40.976 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:40.976 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:40.976 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:40.976 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:40.976 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:40.977 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212896 2024-09-21T10:26:40.977 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.977 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:40.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.977 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:40.977 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404604+0000", 2024-09-21T10:26:40.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:40.977 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.977 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:40.977 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404604+0000", 2024-09-21T10:26:40.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:40.977 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.978 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:40.978 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404603+0000", 2024-09-21T10:26:40.978 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:40.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.978 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.978 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:40.978 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404611+0000", 2024-09-21T10:26:40.978 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.619e-06 2024-09-21T10:26:40.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.978 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.978 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:40.978 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404629+0000", 2024-09-21T10:26:40.978 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8158000000000001e-05 2024-09-21T10:26:40.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:40.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404636+0000", 2024-09-21T10:26:40.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8249999999999999e-06 2024-09-21T10:26:40.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:40.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404685+0000", 2024-09-21T10:26:40.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8498000000000002e-05 2024-09-21T10:26:40.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:40.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404693+0000", 2024-09-21T10:26:40.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7309999999999999e-06 2024-09-21T10:26:40.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.980 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:40.980 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.397879+0000", 2024-09-21T10:26:40.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99318610699999998 2024-09-21T10:26:40.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.980 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:40.980 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.398010+0000", 2024-09-21T10:26:40.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013017000000000001 2024-09-21T10:26:40.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.980 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:26:40.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.398133+0000", 2024-09-21T10:26:40.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012345499999999999 2024-09-21T10:26:40.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:40.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.427399+0000", 2024-09-21T10:26:40.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.029265835 2024-09-21T10:26:40.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:40.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440996+0000", 2024-09-21T10:26:40.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013597122 2024-09-21T10:26:40.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:40.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441297+0000", 2024-09-21T10:26:40.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00030115399999999998 2024-09-21T10:26:40.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:40.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441343+0000", 2024-09-21T10:26:40.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5518e-05 2024-09-21T10:26:40.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:40.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441379+0000", 2024-09-21T10:26:40.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6149999999999998e-05 2024-09-21T10:26:40.983 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:40.983 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:40.983 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:40.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.983 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.983 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212935 9.1 9:83432cba:::benchmark_data_smithi098_254407_object80204:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:26:40.983 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.443322+0000", 2024-09-21T10:26:40.983 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 82.483630254000005, 2024-09-21T10:26:40.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99820324299999996, 2024-09-21T10:26:40.983 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:40.983 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:40.983 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:40.983 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:40.984 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:40.984 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212935 2024-09-21T10:26:40.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.984 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:40.984 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.984 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:40.984 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443322+0000", 2024-09-21T10:26:40.984 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:40.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.984 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.984 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:40.984 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443322+0000", 2024-09-21T10:26:40.984 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:40.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:40.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443320+0000", 2024-09-21T10:26:40.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999976 2024-09-21T10:26:40.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:40.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443328+0000", 2024-09-21T10:26:40.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0760000000000003e-06 2024-09-21T10:26:40.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:40.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443345+0000", 2024-09-21T10:26:40.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7479999999999999e-05 2024-09-21T10:26:40.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:40.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443351+0000", 2024-09-21T10:26:40.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2179999999999996e-06 2024-09-21T10:26:40.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:40.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443503+0000", 2024-09-21T10:26:40.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015199300000000001 2024-09-21T10:26:40.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:40.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443524+0000", 2024-09-21T10:26:40.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0154000000000001e-05 2024-09-21T10:26:40.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:40.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.401733+0000", 2024-09-21T10:26:40.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.95820901400000003 2024-09-21T10:26:40.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:40.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.401836+0000", 2024-09-21T10:26:40.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000103812 2024-09-21T10:26:40.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:26:40.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.401955+0000", 2024-09-21T10:26:40.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011872899999999999 2024-09-21T10:26:40.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:40.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433786+0000", 2024-09-21T10:26:40.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.031831163000000003 2024-09-21T10:26:40.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:40.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441037+0000", 2024-09-21T10:26:40.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0072507400000000003 2024-09-21T10:26:40.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:40.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441450+0000", 2024-09-21T10:26:40.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00041282899999999997 2024-09-21T10:26:40.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:40.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441491+0000", 2024-09-21T10:26:40.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1603999999999997e-05 2024-09-21T10:26:40.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:40.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441525+0000", 2024-09-21T10:26:40.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3592e-05 2024-09-21T10:26:40.990 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:40.990 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:40.990 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:40.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.990 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212946 9.1 9:86b77d80:::benchmark_data_smithi098_254407_object80215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:26:40.990 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.455455+0000", 2024-09-21T10:26:40.990 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 82.471496575000003, 2024-09-21T10:26:40.990 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.98628469600000002, 2024-09-21T10:26:40.990 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:40.990 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:40.990 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:40.990 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:40.990 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:40.991 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212946 2024-09-21T10:26:40.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.991 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:40.991 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.991 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:40.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455455+0000", 2024-09-21T10:26:40.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:40.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.991 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.991 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:40.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455455+0000", 2024-09-21T10:26:40.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:40.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.992 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:40.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455454+0000", 2024-09-21T10:26:40.992 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:40.992 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.992 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:40.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455462+0000", 2024-09-21T10:26:40.992 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6349999999999995e-06 2024-09-21T10:26:40.992 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.992 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:40.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455479+0000", 2024-09-21T10:26:40.992 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6521e-05 2024-09-21T10:26:40.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.993 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.993 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:40.993 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455485+0000", 2024-09-21T10:26:40.993 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5289999999999997e-06 2024-09-21T10:26:40.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.993 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.993 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:40.993 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455531+0000", 2024-09-21T10:26:40.993 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5420000000000002e-05 2024-09-21T10:26:40.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.993 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.993 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:40.993 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455539+0000", 2024-09-21T10:26:40.994 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3769999999999996e-06 2024-09-21T10:26:40.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.994 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.994 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:40.994 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.404976+0000", 2024-09-21T10:26:40.994 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94943732199999997 2024-09-21T10:26:40.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.994 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.994 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:40.994 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.405084+0000", 2024-09-21T10:26:40.994 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010813400000000001 2024-09-21T10:26:40.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.994 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.994 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:26:40.995 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.405217+0000", 2024-09-21T10:26:40.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013250100000000001 2024-09-21T10:26:40.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.995 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:40.995 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433790+0000", 2024-09-21T10:26:40.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.028573111000000002 2024-09-21T10:26:40.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.995 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:40.995 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441178+0000", 2024-09-21T10:26:40.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0073877259999999998 2024-09-21T10:26:40.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.995 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:40.996 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441667+0000", 2024-09-21T10:26:40.996 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00048886199999999996 2024-09-21T10:26:40.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.996 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.996 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:40.996 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441706+0000", 2024-09-21T10:26:40.996 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8828000000000002e-05 2024-09-21T10:26:40.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.996 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.996 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:40.996 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441740+0000", 2024-09-21T10:26:40.996 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4625999999999997e-05 2024-09-21T10:26:40.996 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:40.996 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:40.997 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:40.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.997 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.997 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212950 9.1 9:83ffba2d:::benchmark_data_smithi098_254407_object80219:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:26:40.997 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.459261+0000", 2024-09-21T10:26:40.997 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 82.467690614999995, 2024-09-21T10:26:40.997 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.98262656500000001, 2024-09-21T10:26:40.997 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:40.997 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:40.997 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:40.997 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:40.997 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:40.997 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212950 2024-09-21T10:26:40.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.998 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:40.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.998 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:40.998 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459261+0000", 2024-09-21T10:26:40.998 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:40.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.998 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:40.998 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459261+0000", 2024-09-21T10:26:40.998 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:40.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.998 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:40.999 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459260+0000", 2024-09-21T10:26:40.999 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:40.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:40.999 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459267+0000", 2024-09-21T10:26:40.999 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7970000000000001e-06 2024-09-21T10:26:40.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:40.999 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459289+0000", 2024-09-21T10:26:40.999 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1185000000000001e-05 2024-09-21T10:26:40.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:40.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:40.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:41.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459295+0000", 2024-09-21T10:26:41.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5130000000000002e-06 2024-09-21T10:26:41.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.000 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.000 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:41.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459340+0000", 2024-09-21T10:26:41.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4555000000000003e-05 2024-09-21T10:26:41.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.000 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.000 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:41.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459349+0000", 2024-09-21T10:26:41.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4150000000000005e-06 2024-09-21T10:26:41.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.000 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.000 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:41.001 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.408479+0000", 2024-09-21T10:26:41.001 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94912973899999997 2024-09-21T10:26:41.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.001 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:41.001 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.408589+0000", 2024-09-21T10:26:41.001 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109972 2024-09-21T10:26:41.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.001 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:26:41.001 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.408712+0000", 2024-09-21T10:26:41.001 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012358900000000001 2024-09-21T10:26:41.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.002 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:41.002 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433794+0000", 2024-09-21T10:26:41.002 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.025081782 2024-09-21T10:26:41.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.002 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.002 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:41.002 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441575+0000", 2024-09-21T10:26:41.002 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.007781123 2024-09-21T10:26:41.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.002 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.002 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:41.002 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441795+0000", 2024-09-21T10:26:41.002 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000219268 2024-09-21T10:26:41.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.002 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:41.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441841+0000", 2024-09-21T10:26:41.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6286999999999999e-05 2024-09-21T10:26:41.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:41.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441888+0000", 2024-09-21T10:26:41.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7077999999999999e-05 2024-09-21T10:26:41.003 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.003 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:41.003 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.004 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.004 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212956 9.1 9:85c97ada:::benchmark_data_smithi098_254407_object80225:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:26:41.004 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.465260+0000", 2024-09-21T10:26:41.004 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 82.461691732999995, 2024-09-21T10:26:41.004 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97679754200000002, 2024-09-21T10:26:41.004 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:41.004 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:41.004 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:41.004 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:41.004 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:41.004 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212956 2024-09-21T10:26:41.004 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.004 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:41.004 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.005 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:41.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465260+0000", 2024-09-21T10:26:41.005 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:41.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.005 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.005 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:41.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465260+0000", 2024-09-21T10:26:41.005 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:41.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.005 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.005 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:41.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465258+0000", 2024-09-21T10:26:41.005 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:41.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.005 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:41.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465268+0000", 2024-09-21T10:26:41.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5109999999999992e-06 2024-09-21T10:26:41.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.006 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:41.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465290+0000", 2024-09-21T10:26:41.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2013000000000001e-05 2024-09-21T10:26:41.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.006 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:41.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465297+0000", 2024-09-21T10:26:41.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.311e-06 2024-09-21T10:26:41.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.007 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:41.007 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465456+0000", 2024-09-21T10:26:41.007 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000159347 2024-09-21T10:26:41.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.007 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:41.007 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465488+0000", 2024-09-21T10:26:41.007 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1739000000000002e-05 2024-09-21T10:26:41.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.007 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:41.008 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.411613+0000", 2024-09-21T10:26:41.008 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94612508699999998 2024-09-21T10:26:41.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.008 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.008 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:41.008 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.411728+0000", 2024-09-21T10:26:41.008 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011490200000000001 2024-09-21T10:26:41.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.008 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.008 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:26:41.008 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.411855+0000", 2024-09-21T10:26:41.008 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000126971 2024-09-21T10:26:41.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.009 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.009 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:41.009 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433798+0000", 2024-09-21T10:26:41.009 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.021942560999999999 2024-09-21T10:26:41.009 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.009 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.009 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:41.009 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441610+0000", 2024-09-21T10:26:41.009 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0078126959999999992 2024-09-21T10:26:41.009 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.009 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.009 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:41.009 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441984+0000", 2024-09-21T10:26:41.009 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000373956 2024-09-21T10:26:41.010 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.010 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.010 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:41.010 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442024+0000", 2024-09-21T10:26:41.010 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9459999999999998e-05 2024-09-21T10:26:41.010 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.010 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.010 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:41.010 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442058+0000", 2024-09-21T10:26:41.010 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4010000000000001e-05 2024-09-21T10:26:41.010 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.010 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:41.010 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.010 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.010 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.011 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212974 9.1 9:8bf7ce4b:::benchmark_data_smithi098_254407_object80243:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:26:41.011 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.484850+0000", 2024-09-21T10:26:41.011 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 82.442101871999995, 2024-09-21T10:26:41.011 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.95736883500000003, 2024-09-21T10:26:41.011 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:41.011 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:41.011 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:41.011 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:41.011 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:41.011 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212974 2024-09-21T10:26:41.011 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.011 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:41.011 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.011 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:41.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.484850+0000", 2024-09-21T10:26:41.012 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:41.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.012 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.012 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:41.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.484850+0000", 2024-09-21T10:26:41.012 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:41.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.012 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.012 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:41.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.484848+0000", 2024-09-21T10:26:41.012 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:41.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.012 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.012 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:41.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.484857+0000", 2024-09-21T10:26:41.013 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1589999999999999e-06 2024-09-21T10:26:41.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.013 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.013 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:41.013 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.484883+0000", 2024-09-21T10:26:41.013 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5681999999999999e-05 2024-09-21T10:26:41.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.013 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.013 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:41.013 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.484890+0000", 2024-09-21T10:26:41.013 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4200000000000001e-06 2024-09-21T10:26:41.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.013 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.013 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:41.013 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.485094+0000", 2024-09-21T10:26:41.014 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00020403399999999999 2024-09-21T10:26:41.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.014 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.014 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:41.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.485127+0000", 2024-09-21T10:26:41.014 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2221000000000002e-05 2024-09-21T10:26:41.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.014 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.014 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:41.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.414677+0000", 2024-09-21T10:26:41.014 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.929550023 2024-09-21T10:26:41.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.014 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.014 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:41.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.414781+0000", 2024-09-21T10:26:41.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104038 2024-09-21T10:26:41.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.015 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:26:41.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.414897+0000", 2024-09-21T10:26:41.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011673899999999999 2024-09-21T10:26:41.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.015 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:41.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433801+0000", 2024-09-21T10:26:41.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018903169000000001 2024-09-21T10:26:41.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.015 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:41.016 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441942+0000", 2024-09-21T10:26:41.016 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0081409929999999991 2024-09-21T10:26:41.016 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.016 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.016 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:41.016 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442168+0000", 2024-09-21T10:26:41.016 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00022609 2024-09-21T10:26:41.016 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.016 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.016 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:41.016 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442198+0000", 2024-09-21T10:26:41.016 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.042e-05 2024-09-21T10:26:41.016 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.016 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.016 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:41.017 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442219+0000", 2024-09-21T10:26:41.017 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0837000000000001e-05 2024-09-21T10:26:41.017 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.017 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:41.017 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.017 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.017 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.017 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212980 9.1 9:8e228128:::benchmark_data_smithi098_254407_object80249:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:26:41.017 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.495564+0000", 2024-09-21T10:26:41.017 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 82.431388005000002, 2024-09-21T10:26:41.017 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.946840558, 2024-09-21T10:26:41.017 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:41.017 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:41.018 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:41.018 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:41.018 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:41.018 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212980 2024-09-21T10:26:41.018 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.018 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:41.018 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.018 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:41.018 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495564+0000", 2024-09-21T10:26:41.018 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:41.018 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.018 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.018 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:41.019 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495564+0000", 2024-09-21T10:26:41.019 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:41.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.019 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.019 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:41.019 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495562+0000", 2024-09-21T10:26:41.019 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:41.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.019 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.019 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:41.019 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495569+0000", 2024-09-21T10:26:41.019 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.781e-06 2024-09-21T10:26:41.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.020 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.020 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:41.020 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495580+0000", 2024-09-21T10:26:41.020 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0601999999999999e-05 2024-09-21T10:26:41.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.020 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.020 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:41.020 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495586+0000", 2024-09-21T10:26:41.020 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9750000000000004e-06 2024-09-21T10:26:41.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.020 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.020 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:41.020 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495640+0000", 2024-09-21T10:26:41.021 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3989000000000001e-05 2024-09-21T10:26:41.021 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.021 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.021 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:41.021 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495651+0000", 2024-09-21T10:26:41.021 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1012000000000001e-05 2024-09-21T10:26:41.021 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.021 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.021 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:41.021 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.417846+0000", 2024-09-21T10:26:41.021 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.92219546299999999 2024-09-21T10:26:41.021 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.021 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.021 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:41.022 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.417972+0000", 2024-09-21T10:26:41.022 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000125695 2024-09-21T10:26:41.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.022 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.022 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:26:41.022 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.418097+0000", 2024-09-21T10:26:41.022 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012517 2024-09-21T10:26:41.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.022 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.022 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:41.022 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433804+0000", 2024-09-21T10:26:41.022 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015706759000000001 2024-09-21T10:26:41.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.023 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.023 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:41.023 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442128+0000", 2024-09-21T10:26:41.023 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0083244349999999998 2024-09-21T10:26:41.023 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.023 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.023 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:41.023 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442338+0000", 2024-09-21T10:26:41.023 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00020961600000000001 2024-09-21T10:26:41.023 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.023 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.023 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:41.023 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442372+0000", 2024-09-21T10:26:41.024 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4573999999999998e-05 2024-09-21T10:26:41.024 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.024 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.024 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:41.024 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442405+0000", 2024-09-21T10:26:41.024 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2186e-05 2024-09-21T10:26:41.024 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.024 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:41.024 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.024 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.024 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.024 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212982 9.1 9:8bbf57c4:::benchmark_data_smithi098_254407_object80251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:26:41.024 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.495812+0000", 2024-09-21T10:26:41.025 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 82.431139611999996, 2024-09-21T10:26:41.025 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94673693199999998, 2024-09-21T10:26:41.025 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:41.025 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:41.025 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:41.025 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:41.025 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:41.025 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212982 2024-09-21T10:26:41.025 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.025 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:41.025 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.025 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:41.025 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495812+0000", 2024-09-21T10:26:41.026 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:41.026 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.026 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.026 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:41.026 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495812+0000", 2024-09-21T10:26:41.026 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:41.026 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.026 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.026 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:41.026 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495812+0000", 2024-09-21T10:26:41.026 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:26:41.026 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.026 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.026 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:41.027 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495816+0000", 2024-09-21T10:26:41.027 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3039999999999998e-06 2024-09-21T10:26:41.027 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.027 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.027 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:41.027 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495824+0000", 2024-09-21T10:26:41.027 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5120000000000002e-06 2024-09-21T10:26:41.027 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.027 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.027 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:41.027 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495828+0000", 2024-09-21T10:26:41.027 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6240000000000001e-06 2024-09-21T10:26:41.028 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.028 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.028 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:41.028 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495870+0000", 2024-09-21T10:26:41.028 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1554999999999998e-05 2024-09-21T10:26:41.028 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.028 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.028 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:41.028 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495887+0000", 2024-09-21T10:26:41.028 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7122999999999999e-05 2024-09-21T10:26:41.028 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.028 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.028 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:41.028 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.420943+0000", 2024-09-21T10:26:41.028 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.92505649000000001 2024-09-21T10:26:41.029 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.029 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.029 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:41.029 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.421059+0000", 2024-09-21T10:26:41.029 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000115108 2024-09-21T10:26:41.029 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.029 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.029 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:26:41.029 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.421167+0000", 2024-09-21T10:26:41.029 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108929 2024-09-21T10:26:41.029 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.029 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.029 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:41.029 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433807+0000", 2024-09-21T10:26:41.030 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012639262 2024-09-21T10:26:41.030 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.030 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.030 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:41.030 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442271+0000", 2024-09-21T10:26:41.030 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0084638600000000001 2024-09-21T10:26:41.030 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.030 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.030 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:41.030 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442444+0000", 2024-09-21T10:26:41.030 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017356899999999999 2024-09-21T10:26:41.030 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.030 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.031 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:41.031 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442515+0000", 2024-09-21T10:26:41.031 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1299999999999998e-05 2024-09-21T10:26:41.031 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.031 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.031 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:41.031 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442549+0000", 2024-09-21T10:26:41.031 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3889e-05 2024-09-21T10:26:41.031 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.031 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:41.031 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.031 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.031 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.031 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:213024 9.1 9:8ab0c9ed:::benchmark_data_smithi098_254407_object80293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:26:41.031 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.545957+0000", 2024-09-21T10:26:41.032 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 82.380994533999996, 2024-09-21T10:26:41.032 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.896714071, 2024-09-21T10:26:41.032 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:41.032 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:41.032 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:41.032 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:41.032 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:41.032 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 213024 2024-09-21T10:26:41.032 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.032 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:41.032 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.032 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:41.032 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.545957+0000", 2024-09-21T10:26:41.033 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:41.033 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.033 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.033 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:41.033 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.545957+0000", 2024-09-21T10:26:41.033 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:41.033 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.033 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.033 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:41.033 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.545956+0000", 2024-09-21T10:26:41.033 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:41.033 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.033 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.033 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:41.033 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.545964+0000", 2024-09-21T10:26:41.033 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6629999999999999e-06 2024-09-21T10:26:41.034 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.034 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.034 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:41.034 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.545986+0000", 2024-09-21T10:26:41.034 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1220999999999999e-05 2024-09-21T10:26:41.034 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.034 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.034 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:41.034 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.545993+0000", 2024-09-21T10:26:41.034 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.396e-06 2024-09-21T10:26:41.034 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.034 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.034 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:41.034 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.546156+0000", 2024-09-21T10:26:41.034 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016294899999999999 2024-09-21T10:26:41.035 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.035 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.035 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:41.035 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.546178+0000", 2024-09-21T10:26:41.035 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1971999999999999e-05 2024-09-21T10:26:41.035 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.035 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.035 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:41.035 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.424032+0000", 2024-09-21T10:26:41.035 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.87785439099999996 2024-09-21T10:26:41.035 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.035 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.035 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:41.036 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.424140+0000", 2024-09-21T10:26:41.036 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107507 2024-09-21T10:26:41.036 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.036 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.036 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:26:41.036 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.424258+0000", 2024-09-21T10:26:41.036 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011840700000000001 2024-09-21T10:26:41.036 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.036 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.036 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:41.036 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.437127+0000", 2024-09-21T10:26:41.036 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012869234 2024-09-21T10:26:41.036 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.036 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.036 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:41.036 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442310+0000", 2024-09-21T10:26:41.037 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051825589999999998 2024-09-21T10:26:41.037 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.037 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.037 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:41.037 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442599+0000", 2024-09-21T10:26:41.037 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00028935099999999997 2024-09-21T10:26:41.037 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.037 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.037 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:41.037 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442639+0000", 2024-09-21T10:26:41.037 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9762000000000003e-05 2024-09-21T10:26:41.037 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.037 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.038 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:41.038 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442672+0000", 2024-09-21T10:26:41.038 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2480000000000001e-05 2024-09-21T10:26:41.038 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.038 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:41.038 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.038 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.038 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.038 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:213027 9.1 9:8d848b4e:::benchmark_data_smithi098_254407_object80296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:26:41.038 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.549492+0000", 2024-09-21T10:26:41.038 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 82.377459993000002, 2024-09-21T10:26:41.038 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.89346427100000003, 2024-09-21T10:26:41.038 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:41.038 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:41.038 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:41.039 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:41.039 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:41.039 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 213027 2024-09-21T10:26:41.039 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.039 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:41.039 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.039 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:41.039 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549492+0000", 2024-09-21T10:26:41.039 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:41.039 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.039 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.039 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:41.039 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549492+0000", 2024-09-21T10:26:41.039 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:41.039 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.040 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.040 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:41.040 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549490+0000", 2024-09-21T10:26:41.040 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:41.040 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.040 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.040 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:41.040 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549498+0000", 2024-09-21T10:26:41.040 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4209999999999995e-06 2024-09-21T10:26:41.040 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.040 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.040 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:41.040 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549518+0000", 2024-09-21T10:26:41.041 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0057000000000001e-05 2024-09-21T10:26:41.041 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.041 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.041 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:41.041 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549526+0000", 2024-09-21T10:26:41.041 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4730000000000003e-06 2024-09-21T10:26:41.041 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.041 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.041 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:41.041 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549593+0000", 2024-09-21T10:26:41.041 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7409000000000005e-05 2024-09-21T10:26:41.041 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.041 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.042 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:41.042 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549603+0000", 2024-09-21T10:26:41.042 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5810000000000003e-06 2024-09-21T10:26:41.042 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.042 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.042 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:41.042 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.427438+0000", 2024-09-21T10:26:41.042 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.87783562999999998 2024-09-21T10:26:41.042 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.042 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.042 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:41.042 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.427559+0000", 2024-09-21T10:26:41.042 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000120377 2024-09-21T10:26:41.042 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.043 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.043 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:26:41.043 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.427666+0000", 2024-09-21T10:26:41.043 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107382 2024-09-21T10:26:41.043 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.043 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.043 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:41.043 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.437138+0000", 2024-09-21T10:26:41.043 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0094713769999999996 2024-09-21T10:26:41.043 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.043 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.043 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:41.043 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442725+0000", 2024-09-21T10:26:41.044 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.005586995 2024-09-21T10:26:41.044 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:41.045 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:26:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:26:40] "GET /metrics HTTP/1.1" 200 36512 "" "Prometheus/2.51.0" 2024-09-21T10:26:41.045 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.046 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.046 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:41.046 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442870+0000", 2024-09-21T10:26:41.046 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014551399999999999 2024-09-21T10:26:41.046 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.046 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.046 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:41.046 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442920+0000", 2024-09-21T10:26:41.046 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0371999999999998e-05 2024-09-21T10:26:41.046 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.046 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.046 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:41.046 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442956+0000", 2024-09-21T10:26:41.047 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5805e-05 2024-09-21T10:26:41.047 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.047 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:41.047 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.047 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.047 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.047 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:213030 9.1 9:8c3d5a31:::benchmark_data_smithi098_254407_object80299:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:26:41.047 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.556579+0000", 2024-09-21T10:26:41.047 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 82.370372700000004, 2024-09-21T10:26:41.047 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.88649370000000005, 2024-09-21T10:26:41.047 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:41.047 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:41.048 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:41.048 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:41.048 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:41.048 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 213030 2024-09-21T10:26:41.048 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.048 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:41.048 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.048 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:41.048 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556579+0000", 2024-09-21T10:26:41.048 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:41.048 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.048 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.048 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:41.049 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556579+0000", 2024-09-21T10:26:41.049 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:41.049 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.049 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.049 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:41.049 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556578+0000", 2024-09-21T10:26:41.049 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:41.049 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.049 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.049 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:41.049 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556585+0000", 2024-09-21T10:26:41.049 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6159999999999996e-06 2024-09-21T10:26:41.049 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.049 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.050 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:41.050 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556600+0000", 2024-09-21T10:26:41.050 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.454e-05 2024-09-21T10:26:41.050 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.050 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.050 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:41.050 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556607+0000", 2024-09-21T10:26:41.050 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8900000000000001e-06 2024-09-21T10:26:41.050 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.050 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.050 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:41.050 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556650+0000", 2024-09-21T10:26:41.050 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3727000000000003e-05 2024-09-21T10:26:41.051 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.051 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.051 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:41.051 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556658+0000", 2024-09-21T10:26:41.051 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5079999999999999e-06 2024-09-21T10:26:41.051 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.051 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.051 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:41.051 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.430408+0000", 2024-09-21T10:26:41.051 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.87375053700000005 2024-09-21T10:26:41.051 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.051 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.051 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:41.051 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.430547+0000", 2024-09-21T10:26:41.052 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013809400000000001 2024-09-21T10:26:41.052 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.052 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.052 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:26:41.052 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.430679+0000", 2024-09-21T10:26:41.052 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013244900000000001 2024-09-21T10:26:41.052 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.052 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.052 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:41.052 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440488+0000", 2024-09-21T10:26:41.052 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0098090959999999998 2024-09-21T10:26:41.052 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.052 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.053 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:41.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442765+0000", 2024-09-21T10:26:41.053 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002277276 2024-09-21T10:26:41.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.053 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.053 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:41.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443000+0000", 2024-09-21T10:26:41.053 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000234682 2024-09-21T10:26:41.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.053 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.053 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:41.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443042+0000", 2024-09-21T10:26:41.053 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2126000000000003e-05 2024-09-21T10:26:41.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.054 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.054 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:41.054 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443073+0000", 2024-09-21T10:26:41.054 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0825999999999999e-05 2024-09-21T10:26:41.054 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.054 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:41.054 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.054 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.054 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:213040 9.1 9:8e7e94d6:::benchmark_data_smithi098_254407_object80309:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:26:41.054 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.575656+0000", 2024-09-21T10:26:41.054 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 82.351295848000007, 2024-09-21T10:26:41.054 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.86753614499999998, 2024-09-21T10:26:41.055 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:41.055 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:41.055 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:41.055 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:41.055 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:41.055 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 213040 2024-09-21T10:26:41.055 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.055 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:41.055 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.055 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:41.055 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575656+0000", 2024-09-21T10:26:41.055 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:41.055 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.055 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.055 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:41.056 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575656+0000", 2024-09-21T10:26:41.056 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:41.056 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.056 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.056 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:41.056 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575654+0000", 2024-09-21T10:26:41.056 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:41.056 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.056 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.056 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:41.056 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575664+0000", 2024-09-21T10:26:41.056 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7340000000000003e-06 2024-09-21T10:26:41.056 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.056 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.056 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:41.057 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575682+0000", 2024-09-21T10:26:41.057 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8410000000000002e-05 2024-09-21T10:26:41.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.057 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.057 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:41.057 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575691+0000", 2024-09-21T10:26:41.057 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3559999999999993e-06 2024-09-21T10:26:41.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.057 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.057 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:41.057 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575742+0000", 2024-09-21T10:26:41.057 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0698999999999997e-05 2024-09-21T10:26:41.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.057 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.057 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:41.058 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575752+0000", 2024-09-21T10:26:41.058 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0288999999999999e-05 2024-09-21T10:26:41.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.058 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.058 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:41.058 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433825+0000", 2024-09-21T10:26:41.058 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.85807313799999996 2024-09-21T10:26:41.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.058 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.058 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:41.058 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433937+0000", 2024-09-21T10:26:41.058 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011184900000000001 2024-09-21T10:26:41.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.058 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.058 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:26:41.059 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.434061+0000", 2024-09-21T10:26:41.059 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012429100000000001 2024-09-21T10:26:41.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.059 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.059 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:41.059 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440716+0000", 2024-09-21T10:26:41.059 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006655134 2024-09-21T10:26:41.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.059 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.059 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:41.059 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442806+0000", 2024-09-21T10:26:41.059 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002089835 2024-09-21T10:26:41.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.059 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.059 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:41.060 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443118+0000", 2024-09-21T10:26:41.060 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00031147500000000002 2024-09-21T10:26:41.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.060 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.060 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:41.060 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443158+0000", 2024-09-21T10:26:41.060 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0670000000000002e-05 2024-09-21T10:26:41.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.060 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.060 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:41.060 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443192+0000", 2024-09-21T10:26:41.060 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4109999999999997e-05 2024-09-21T10:26:41.060 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.060 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:41.061 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.061 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.061 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.061 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:213046 9.1 9:8035648d:::benchmark_data_smithi098_254407_object80315:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:26:41.061 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.597939+0000", 2024-09-21T10:26:41.061 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 82.329012923999997, 2024-09-21T10:26:41.061 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.84537171499999997, 2024-09-21T10:26:41.061 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:41.061 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:41.061 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:41.061 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:41.061 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:41.061 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 213046 2024-09-21T10:26:41.061 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.061 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:41.062 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.062 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:41.062 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.597939+0000", 2024-09-21T10:26:41.062 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:41.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.062 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.062 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:41.062 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.597939+0000", 2024-09-21T10:26:41.062 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:41.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.062 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.062 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:41.062 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.597938+0000", 2024-09-21T10:26:41.062 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:41.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.063 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.063 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:41.063 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.597945+0000", 2024-09-21T10:26:41.063 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5959999999999998e-06 2024-09-21T10:26:41.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.063 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.063 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:41.063 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.597959+0000", 2024-09-21T10:26:41.063 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3986e-05 2024-09-21T10:26:41.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.063 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.063 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:41.063 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.597964+0000", 2024-09-21T10:26:41.063 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0289999999999996e-06 2024-09-21T10:26:41.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.064 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.064 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:41.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.598015+0000", 2024-09-21T10:26:41.064 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0376e-05 2024-09-21T10:26:41.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.064 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.064 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:26:41.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.598024+0000", 2024-09-21T10:26:41.064 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8899999999999996e-06 2024-09-21T10:26:41.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.064 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.064 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:41.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.437180+0000", 2024-09-21T10:26:41.064 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83915633099999998 2024-09-21T10:26:41.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.064 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.065 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:41.065 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.437294+0000", 2024-09-21T10:26:41.065 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114388 2024-09-21T10:26:41.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.065 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.065 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:26:41.065 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.437423+0000", 2024-09-21T10:26:41.065 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000128768 2024-09-21T10:26:41.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.065 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.065 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:41.065 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440953+0000", 2024-09-21T10:26:41.065 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0035302250000000001 2024-09-21T10:26:41.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.065 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.066 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:41.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442834+0000", 2024-09-21T10:26:41.066 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0018801919999999999 2024-09-21T10:26:41.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.066 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.066 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:41.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443242+0000", 2024-09-21T10:26:41.066 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00040894199999999998 2024-09-21T10:26:41.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.066 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.066 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:41.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443282+0000", 2024-09-21T10:26:41.066 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9504e-05 2024-09-21T10:26:41.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.067 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.067 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:41.067 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443311+0000", 2024-09-21T10:26:41.067 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8816999999999999e-05 2024-09-21T10:26:41.067 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.067 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:41.067 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.067 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.067 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:238082 9.1 9:8844e02b:::benchmark_data_smithi098_254407_object105351:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-09-21T10:26:41.067 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:47.291888+0000", 2024-09-21T10:26:41.067 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 53.635064217999997, 2024-09-21T10:26:41.067 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.058846923000000002, 2024-09-21T10:26:41.067 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:26:41.067 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:41.068 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:26:41.068 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:26:41.068 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:26:41.068 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 238082 2024-09-21T10:26:41.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.068 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:26:41.068 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.068 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:26:41.068 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:47.291888+0000", 2024-09-21T10:26:41.068 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:41.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.068 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.068 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:26:41.068 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:47.291888+0000", 2024-09-21T10:26:41.069 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:26:41.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.069 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.069 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:26:41.069 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:47.291886+0000", 2024-09-21T10:26:41.069 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:41.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.069 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.069 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:26:41.069 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:47.291893+0000", 2024-09-21T10:26:41.069 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3170000000000004e-06 2024-09-21T10:26:41.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.069 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.069 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:26:41.069 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:47.291911+0000", 2024-09-21T10:26:41.070 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7408999999999999e-05 2024-09-21T10:26:41.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.070 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.070 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:26:41.070 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:47.291917+0000", 2024-09-21T10:26:41.070 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3740000000000003e-06 2024-09-21T10:26:41.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.070 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.070 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:26:41.070 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:47.292067+0000", 2024-09-21T10:26:41.070 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014950299999999999 2024-09-21T10:26:41.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.070 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.070 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:26:41.070 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:47.292339+0000", 2024-09-21T10:26:41.071 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000272107 2024-09-21T10:26:41.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.071 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.071 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:26:41.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:47.292666+0000", 2024-09-21T10:26:41.071 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00032763999999999997 2024-09-21T10:26:41.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.071 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.071 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:26:41.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:47.300074+0000", 2024-09-21T10:26:41.071 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0074073869999999997 2024-09-21T10:26:41.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.071 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.071 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:41.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:47.302219+0000", 2024-09-21T10:26:41.072 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021449440000000002 2024-09-21T10:26:41.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.072 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.072 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:41.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:47.350598+0000", 2024-09-21T10:26:41.072 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.048379243000000002 2024-09-21T10:26:41.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.072 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.072 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:26:41.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:47.350689+0000", 2024-09-21T10:26:41.072 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1477999999999994e-05 2024-09-21T10:26:41.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:26:41.072 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:26:41.072 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:26:41.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:47.350735+0000", 2024-09-21T10:26:41.073 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5318999999999998e-05 2024-09-21T10:26:41.073 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.073 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:41.073 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.073 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:26:41.073 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:26:41.073 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:26:41.074 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:41.074 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:41.094 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:41.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:41.195 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:41.215 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:41.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:41.317 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:41.338 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:41.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:41.439 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:41.460 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:41.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:41.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:41.583 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:41.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:41.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:41 smithi098 ceph-mon[97580]: pgmap v1172: 99 pgs: 3 unknown, 96 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:41.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:41 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1600010211' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7105059327514174}]': finished 2024-09-21T10:26:41.587 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:41 smithi098 ceph-mon[97580]: osdmap e351: 8 total, 8 up, 7 in 2024-09-21T10:26:41.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:41 smithi098 ceph-mon[100914]: pgmap v1172: 99 pgs: 3 unknown, 96 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:41.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:41 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1600010211' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7105059327514174}]': finished 2024-09-21T10:26:41.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:41 smithi098 ceph-mon[100914]: osdmap e351: 8 total, 8 up, 7 in 2024-09-21T10:26:41.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:41 smithi196 ceph-mon[84784]: pgmap v1172: 99 pgs: 3 unknown, 96 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:41.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:41 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1600010211' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7105059327514174}]': finished 2024-09-21T10:26:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:41 smithi196 ceph-mon[84784]: osdmap e351: 8 total, 8 up, 7 in 2024-09-21T10:26:41.685 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:41.706 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:41.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:41.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:41.830 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:41.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:41.931 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:41.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:41.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:42.054 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:42.072 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:26:42.076 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:42.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:42.178 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:42.201 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:42.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:42.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:26:42.303 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:42.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.325 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:42.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:42.331 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:26:42.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:42 smithi196 ceph-mon[84784]: osdmap e352: 8 total, 8 up, 7 in 2024-09-21T10:26:42.426 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:42.448 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:42.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:42.550 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:42.571 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:42.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:42.580 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:42 smithi098 ceph-mon[97580]: osdmap e352: 8 total, 8 up, 7 in 2024-09-21T10:26:42.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:42 smithi098 ceph-mon[100914]: osdmap e352: 8 total, 8 up, 7 in 2024-09-21T10:26:42.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:42.694 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:42.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:42.796 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:42.817 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:42.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:42.918 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:42.939 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:42.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:43.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:43.063 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:43.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:43.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:43.164 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:43.186 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:43.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:43.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:43.287 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:43.309 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:43.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:43.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:43.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:43 smithi196 ceph-mon[84784]: pgmap v1175: 99 pgs: 99 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:43.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:43.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:43.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:43.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:43.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:43.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:43.558 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:43.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:43.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:43 smithi098 ceph-mon[97580]: pgmap v1175: 99 pgs: 99 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:43.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:43 smithi098 ceph-mon[100914]: pgmap v1175: 99 pgs: 99 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:26:43.659 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:43.680 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:43.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:43.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:43.781 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:43.802 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:43.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:43.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:43.904 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:43.925 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:43.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:43.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:43.975 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:26:43.976 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:26:43.976 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-21T10:26:43.976 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:26:44.027 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:44.048 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:44.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:44.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:44.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:44.171 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:44.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:44.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:44.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:44.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:44.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:44.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:44.417 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:44.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:44.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:44.519 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:44.539 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:44.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:44.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:44.614 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:26:44.641 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:44.656 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:44.664 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:44.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:44.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:44.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:44.788 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:44.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:44.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:44.862 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:26:44.890 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:44.911 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:44.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:44.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:45.012 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:45.034 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:45.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:45.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:45.157 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:45.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:45.259 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:45.284 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:45.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:45.285 INFO:teuthology.orchestra.run.smithi098.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-09-21T10:26:45.385 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:45.407 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:45.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:45.509 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:45.531 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:45.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:45.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:45 smithi098 ceph-mon[97580]: pgmap v1176: 99 pgs: 99 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 161 B/s rd, 0 op/s 2024-09-21T10:26:45.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:45 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1060324925' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:26:45.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:45 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:45.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:45 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:45.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:45 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:26:45.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:45 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:26:45.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:45 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:45.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:45 smithi098 ceph-mon[100914]: pgmap v1176: 99 pgs: 99 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 161 B/s rd, 0 op/s 2024-09-21T10:26:45.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:45 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1060324925' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:26:45.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:45 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:45.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:45 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:45.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:45 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:26:45.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:45 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:26:45.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:45 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:45.632 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:45.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:45 smithi196 ceph-mon[84784]: pgmap v1176: 99 pgs: 99 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 161 B/s rd, 0 op/s 2024-09-21T10:26:45.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:45 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1060324925' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:26:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:45 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:45 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:45 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:26:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:45 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:26:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:45 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:45.654 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:45.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:45.756 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:45.777 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:45.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:45.879 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:45.901 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:45.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:46.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:46.025 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:46.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:46.074 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 4] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:26:46.075 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:26:46.077 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-21T10:26:46.077 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:26:46.078 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:26:46.078 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-21T10:26:46.126 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:46.147 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:46.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:46.249 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:46.270 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:46.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:46.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:46 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1060324925' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:26:46.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:46 smithi196 ceph-mon[84784]: osdmap e353: 8 total, 8 up, 7 in 2024-09-21T10:26:46.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:46 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1060324925' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:26:46.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:46 smithi098 ceph-mon[97580]: osdmap e353: 8 total, 8 up, 7 in 2024-09-21T10:26:46.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:46 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1060324925' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:26:46.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:46 smithi098 ceph-mon[100914]: osdmap e353: 8 total, 8 up, 7 in 2024-09-21T10:26:46.373 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:46.394 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:46.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:46.496 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:46.518 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:46.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:46.568 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:26:46.568 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:26:46.568 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-21T10:26:46.568 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-21T10:26:46.568 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:26:46.620 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:46.641 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:46.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:46.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:46.763 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:46.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:46.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:46.887 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:46.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:46.988 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:47.010 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:47.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:47.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:47.098 DEBUG:teuthology.orchestra.run.smithi098:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-09-21T10:26:47.100 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:26:47.101 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:26:47.112 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:47.133 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:47.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:47.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:47.177 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:hints = 1 2024-09-21T10:26:47.178 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:26:47.178 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Object prefix: benchmark_data_smithi098_281896 2024-09-21T10:26:47.178 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:26:47.178 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:26:47.235 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:47.275 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:47.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:47.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:47.306 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:26:47.378 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:47.415 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:47.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:47.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:47.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:47.551 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:47.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:47.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:47.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:47 smithi098 ceph-mon[97580]: pgmap v1178: 99 pgs: 99 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 170 B/s rd, 0 op/s 2024-09-21T10:26:47.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:47 smithi098 ceph-mon[97580]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:26:47.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:47 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:26:47.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:47 smithi098 ceph-mon[100914]: pgmap v1178: 99 pgs: 99 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 170 B/s rd, 0 op/s 2024-09-21T10:26:47.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:47 smithi098 ceph-mon[100914]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:26:47.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:47 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:26:47.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:47 smithi196 ceph-mon[84784]: pgmap v1178: 99 pgs: 99 active+clean; 4.6 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 170 B/s rd, 0 op/s 2024-09-21T10:26:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:47 smithi196 ceph-mon[84784]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:26:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:47 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:26:47.653 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:47.690 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:47.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:47.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:47.734 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:26:47.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:47.827 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:47.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:47.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:47.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:47.962 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:47.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:47.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:48.064 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:48.090 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:48.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:48.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:48.178 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 1 16 3629 3613 225.8 225.812 0.00372482 0.00441782 2024-09-21T10:26:48.192 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:48.228 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:48.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:48.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:48.249 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:48.309 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:26:48.309 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:26:48.319 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:26:48.330 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:48.338 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:26:48.338 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:26:48.364 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:48.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:48.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:48.371 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:26:48.371 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:26:48.399 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:26:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:26:48.430 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:26:48.430 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:26:48.457 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:26:48.457 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:26:48.465 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:48.486 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:26:48.486 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:26:48.500 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:48.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:48.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:48.518 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:26:48.519 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:26:48.519 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:48 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1928014351' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:26:48.519 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:48 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:48.519 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:48 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3525741743' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:48.519 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:48 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1928014351' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:26:48.519 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:48 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:48.520 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:48 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3525741743' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:48.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:48.637 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:48.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:48.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:48 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1928014351' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:26:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:48 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:48 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3525741743' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:48.739 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:48.776 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:48.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:48.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:48.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:48.912 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:48.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:48.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:49.014 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:49.048 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:49.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:49.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:49.150 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:49.178 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 2 16 7231 7215 225.453 225.125 0.00796901 0.00442429 2024-09-21T10:26:49.196 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:49.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:49.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:49.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:49.329 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:49.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:49.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:49.423 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:49 smithi098 ceph-mon[97580]: pgmap v1179: 99 pgs: 99 active+clean; 7.3 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 140 B/s rd, 378 KiB/s wr, 6 op/s 2024-09-21T10:26:49.424 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:49 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:49.424 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:49 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1928014351' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:26:49.424 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:49 smithi098 ceph-mon[97580]: osdmap e354: 8 total, 8 up, 7 in 2024-09-21T10:26:49.424 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:49 smithi098 ceph-mon[100914]: pgmap v1179: 99 pgs: 99 active+clean; 7.3 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 140 B/s rd, 378 KiB/s wr, 6 op/s 2024-09-21T10:26:49.424 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:49 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:49.425 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:49 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1928014351' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:26:49.425 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:49 smithi098 ceph-mon[100914]: osdmap e354: 8 total, 8 up, 7 in 2024-09-21T10:26:49.431 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:49.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:49.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:49.572 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:49.606 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:49.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:49.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:49.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:49 smithi196 ceph-mon[84784]: pgmap v1179: 99 pgs: 99 active+clean; 7.3 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 140 B/s rd, 378 KiB/s wr, 6 op/s 2024-09-21T10:26:49.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:49 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:49.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:49 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1928014351' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:26:49.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:49 smithi196 ceph-mon[84784]: osdmap e354: 8 total, 8 up, 7 in 2024-09-21T10:26:49.708 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:49.747 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:49.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:49.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:49.786 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:26:49.786 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:26:49.850 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:49.885 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:49.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:49.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:49.988 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:50.024 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:50.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:50.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:50.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:50.163 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:50.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:50.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:50.178 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 3 16 10528 10512 218.984 206.062 0.00317089 0.00456037 2024-09-21T10:26:50.265 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:50.300 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:50.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:50.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:50.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:26:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:26:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:26:50.402 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:50.440 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:50.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:50.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:50.514 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:26:50.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:50.597 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:50.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:50.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:50.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:50.737 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:50.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:50.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:50.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:50.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:26:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:26:50] "GET /metrics HTTP/1.1" 200 36512 "" "Prometheus/2.51.0" 2024-09-21T10:26:50.875 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:50.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:50.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:50.977 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:51.011 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:51.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:51.055 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:51.113 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:51.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.154 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:51.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:51.179 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 4 16 13784 13768 215.109 203.5 0.00409916 0.00464146 2024-09-21T10:26:51.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:51.289 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:51.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:51.391 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:51.423 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:51.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:51.469 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:26:51.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:51 smithi098 ceph-mon[97580]: pgmap v1181: 99 pgs: 99 active+clean; 170 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 127 B/s rd, 21 MiB/s wr, 330 op/s 2024-09-21T10:26:51.503 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:51 smithi098 ceph-mon[100914]: pgmap v1181: 99 pgs: 99 active+clean; 170 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 127 B/s rd, 21 MiB/s wr, 330 op/s 2024-09-21T10:26:51.525 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:51.560 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:51.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:51.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:51 smithi196 ceph-mon[84784]: pgmap v1181: 99 pgs: 99 active+clean; 170 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 127 B/s rd, 21 MiB/s wr, 330 op/s 2024-09-21T10:26:51.662 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:51.695 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:51.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:51.797 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:51.830 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:51.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:51.932 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:51.968 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:51.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:51.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:26:52.071 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:52.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:52.092 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:52.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:52.179 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 5 16 16589 16573 207.147 175.312 0.00438044 0.00482092 2024-09-21T10:26:52.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:52.234 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:52.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:52.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:52.336 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:52.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:52.376 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:52.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:52.477 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:52.512 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:52.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:52.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:52.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:52.648 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:52.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:52.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:52.750 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:52.783 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:52.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:52.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:52.885 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:52.921 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:52.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:53.023 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:53.056 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:53.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:53.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:53.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:53.179 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 6 16 19172 19156 199.526 161.438 0.00156349 0.00499777 2024-09-21T10:26:53.183 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:53.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:53.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:53.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:53.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:53.339 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:53.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:53.440 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:53.477 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:53.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:53.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:53.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:53.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:53 smithi098 ceph-mon[97580]: pgmap v1182: 99 pgs: 99 active+clean; 242 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 30 MiB/s wr, 475 op/s 2024-09-21T10:26:53.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:53.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:53.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:53 smithi098 ceph-mon[100914]: pgmap v1182: 99 pgs: 99 active+clean; 242 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 30 MiB/s wr, 475 op/s 2024-09-21T10:26:53.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:53.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:53.612 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:53.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:53.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:53.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:53 smithi196 ceph-mon[84784]: pgmap v1182: 99 pgs: 99 active+clean; 242 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 30 MiB/s wr, 475 op/s 2024-09-21T10:26:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:26:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:53.713 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:53.749 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:53.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:53.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:53.851 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:53.886 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:53.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:53.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:53.988 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:54.022 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:54.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:54.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:54.156 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:54.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:54.179 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 7 16 21791 21775 194.404 163.688 0.011599 0.00513882 2024-09-21T10:26:54.257 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:54.297 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:54.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:54.398 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:54.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:54.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:54.567 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:54.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:54.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:54.690 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:54.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:54.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:54.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.815 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:54.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:54.916 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:54.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.939 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:54.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:55.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:55.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.076 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:55.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:55.177 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:55.179 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 8 16 23325 23309 182.087 95.875 0.00359219 0.00548737 2024-09-21T10:26:55.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.214 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:55.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:55.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:55.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:55 smithi098 ceph-mon[97580]: pgmap v1183: 99 pgs: 99 active+clean; 925 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 112 MiB/s wr, 1.78k op/s 2024-09-21T10:26:55.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:55 smithi098 ceph-mon[100914]: pgmap v1183: 99 pgs: 99 active+clean; 925 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 112 MiB/s wr, 1.78k op/s 2024-09-21T10:26:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:55.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:55.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:55 smithi196 ceph-mon[84784]: pgmap v1183: 99 pgs: 99 active+clean; 925 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 112 MiB/s wr, 1.78k op/s 2024-09-21T10:26:55.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:55.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:55.584 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:55.616 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:55.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:55.652 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:26:55.652 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "" 2024-09-21T10:26:55.652 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:26:55.717 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:55.754 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:55.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:55.855 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:55.888 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:55.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:55.990 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:56.024 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:56.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:56.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:56.157 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:56.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:56.179 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 9 16 25988 25972 180.346 166.438 0.0032565 0.0055399 2024-09-21T10:26:56.258 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:56.304 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:56.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:56.406 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:56.439 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:56.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:56.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:56.574 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:56.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:56.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:56.710 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:56.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:56.812 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:56.845 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:56.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:56.947 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:56.981 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:56.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:56.992 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:57.055 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:57.083 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:57.117 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:57.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:57.179 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 10 16 28537 28521 178.242 159.312 0.00452718 0.00560484 2024-09-21T10:26:57.218 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:57.252 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:57.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:57.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:57.389 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:57.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:57.490 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:57.525 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:57.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:57.544 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:57 smithi098 ceph-mon[97580]: pgmap v1184: 99 pgs: 99 active+clean; 1.2 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 120 MiB/s wr, 1.93k op/s 2024-09-21T10:26:57.544 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:57 smithi098 ceph-mon[100914]: pgmap v1184: 99 pgs: 99 active+clean; 1.2 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 120 MiB/s wr, 1.93k op/s 2024-09-21T10:26:57.627 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:57.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:57 smithi196 ceph-mon[84784]: pgmap v1184: 99 pgs: 99 active+clean; 1.2 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 120 MiB/s wr, 1.93k op/s 2024-09-21T10:26:57.665 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:57.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:57.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:57.801 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:57.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:57.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:57.936 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:57.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:58.038 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:58.071 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:58.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:58.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:58.172 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:58.180 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 11 16 30958 30942 175.792 151.312 0.00542888 0.00568376 2024-09-21T10:26:58.208 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:58.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:58.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:58.310 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:58.347 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:58.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:58.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:58.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:58.480 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:58.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:58.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:58.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:58.615 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:58.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:58.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:58.716 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:58.752 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:58.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:58.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:58.854 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:58.889 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:58.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:58.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:58.991 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:59.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.026 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:59.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:59.127 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:59.154 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:59.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:59.179 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 12 16 33224 33208 172.943 141.625 0.00219192 0.00575836 2024-09-21T10:26:59.256 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:59.291 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:59.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:59.393 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:59.432 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:59.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:59.534 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:59.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.569 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:59.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:59.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:26:59 smithi098 ceph-mon[97580]: pgmap v1185: 99 pgs: 99 active+clean; 1.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-09-21T10:26:59.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:26:59 smithi098 ceph-mon[100914]: pgmap v1185: 99 pgs: 99 active+clean; 1.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-09-21T10:26:59.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:26:59 smithi196 ceph-mon[84784]: pgmap v1185: 99 pgs: 99 active+clean; 1.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-09-21T10:26:59.671 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:59.705 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:26:59.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:59.806 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:59.845 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:59.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:59.947 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:59.978 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:26:59.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:00.079 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:00.113 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:00.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:00.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:00.179 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 13 16 35243 35227 169.345 126.188 0.00428179 0.00590059 2024-09-21T10:27:00.215 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:00.251 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:00.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:00.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:00.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:00.380 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:00.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:00.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:00.482 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:00.518 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:00.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:00.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:00.619 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:00.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:27:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:27:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:27:00.656 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:00.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:00.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:00.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:00.791 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:00.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:00.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:00.893 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:00.935 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:00.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:00.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:01.036 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:01.069 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:01.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:01.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:01.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:27:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:27:00] "GET /metrics HTTP/1.1" 200 36564 "" "Prometheus/2.51.0" 2024-09-21T10:27:01.171 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:01.179 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 14 16 37342 37326 166.619 131.188 0.0124268 0.00599605 2024-09-21T10:27:01.205 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:01.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:01.307 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:01.327 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:01.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:01.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:01.429 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:01.467 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:01.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:01.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:01.569 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:01 smithi098 ceph-mon[97580]: pgmap v1186: 99 pgs: 99 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-09-21T10:27:01.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:01 smithi098 ceph-mon[100914]: pgmap v1186: 99 pgs: 99 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-09-21T10:27:01.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:01.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:01.624 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:27:01.624 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:27:01.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:01 smithi196 ceph-mon[84784]: pgmap v1186: 99 pgs: 99 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-09-21T10:27:01.707 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:01.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:01.741 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:01.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:01.842 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:01.874 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:01.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:01.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:01.976 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:02.012 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:02.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:02.038 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 4] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:27:02.038 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:27:02.038 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_5 2024-09-21T10:27:02.038 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:27:02.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:02.148 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:02.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:02.184 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 15 16 39259 39243 163.443 119.812 0.00844741 0.0061132 2024-09-21T10:27:02.250 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:02.291 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:02.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:02.393 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:02.427 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:02.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:02.529 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:02.567 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:02.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:02.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:02.705 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:02.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:02.707 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:27:02.707 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:27:02.741 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:27:02.741 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:27:02.773 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:27:02.774 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:27:02.807 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:02.811 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:27:02.811 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:27:02.844 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:02.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:02.847 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:27:02.847 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:27:02.879 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:27:02.879 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:27:02.910 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:27:02.910 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:27:02.942 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:27:02.942 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:27:02.946 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:02.983 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:02.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:03.086 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:03.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:03.121 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:03.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:03.185 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 16 16 41261 41245 161.048 125.125 0.00778008 0.00620402 2024-09-21T10:27:03.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:03.261 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:03.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:03.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:03.363 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:03.394 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:03.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:03.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:03.496 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:03.527 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:03 smithi098 ceph-mon[97580]: pgmap v1187: 99 pgs: 99 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 148 MiB/s wr, 2.36k op/s 2024-09-21T10:27:03.527 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:03 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1515379297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:03.527 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:03 smithi098 ceph-mon[100914]: pgmap v1187: 99 pgs: 99 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 148 MiB/s wr, 2.36k op/s 2024-09-21T10:27:03.527 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:03 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1515379297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:03.529 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:27:03.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:03.542 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:03.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:03.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:03 smithi196 ceph-mon[84784]: pgmap v1187: 99 pgs: 99 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 148 MiB/s wr, 2.36k op/s 2024-09-21T10:27:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:03 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1515379297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:03.643 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:03.679 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:03.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:03.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:03.781 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:03.817 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:03.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:03.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:03.919 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:03.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:03.954 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:03.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:04.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:04.089 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:04.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:04.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:04.185 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 17 16 43286 43270 159.02 126.562 0.00726497 0.00628386 2024-09-21T10:27:04.191 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:04.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:04.225 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:04.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:04.290 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:27:04.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:04.390 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:04.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:04.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:04.492 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:04.524 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:04.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:04.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:04.560 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:04.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:04 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2185994963' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:27:04.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:04 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2185994963' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:27:04.627 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:04.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:04 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2185994963' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:27:04.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:04.667 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:04.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:04.768 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:04.804 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:04.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:04.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:04.906 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:04.939 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:04.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:04.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:05.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:05.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:05.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:05.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:05.185 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 18 16 45211 45195 156.869 120.312 0.00971101 0.00636984 2024-09-21T10:27:05.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:05.216 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:05.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:05.317 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:05.350 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:05.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:05.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:05.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:05.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:05.486 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:05.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:05.588 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:05.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:05 smithi098 ceph-mon[97580]: pgmap v1188: 99 pgs: 99 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 179 MiB/s wr, 2.87k op/s 2024-09-21T10:27:05.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:05 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:27:05.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:05 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2185994963' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:27:05.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:05 smithi098 ceph-mon[97580]: osdmap e355: 8 total, 8 up, 7 in 2024-09-21T10:27:05.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:05 smithi098 ceph-mon[100914]: pgmap v1188: 99 pgs: 99 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 179 MiB/s wr, 2.87k op/s 2024-09-21T10:27:05.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:05 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:27:05.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:05 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2185994963' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:27:05.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:05 smithi098 ceph-mon[100914]: osdmap e355: 8 total, 8 up, 7 in 2024-09-21T10:27:05.621 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:05.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:05.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:05.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:05 smithi196 ceph-mon[84784]: pgmap v1188: 99 pgs: 99 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 179 MiB/s wr, 2.87k op/s 2024-09-21T10:27:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:05 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:27:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:05 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2185994963' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:27:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:05 smithi196 ceph-mon[84784]: osdmap e355: 8 total, 8 up, 7 in 2024-09-21T10:27:05.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:05.759 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:05.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:05.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:05.862 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:05.895 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:05.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:05.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:05.997 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:06.034 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:06.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:06.136 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:06.173 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:06.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:06.185 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 19 16 47127 47111 154.916 119.75 0.0130656 0.00645035 2024-09-21T10:27:06.242 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:27:06.274 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:06.323 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:06.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:06.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:06.463 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:06.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:06.565 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:06.602 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:06.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:06.704 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:06.738 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:06.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:06.843 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:06.880 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:06.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:06.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:07.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:07.015 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:07.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:07.116 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:07.154 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:07.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:07.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:07.185 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:27:07.184204+0000 min lat: 0.00114942 max lat: 0.424342 avg lat: 0.00653272 2024-09-21T10:27:07.185 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:27:07.185 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 20 16 48978 48962 152.954 115.688 0.0152233 0.00653272 2024-09-21T10:27:07.256 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:07.296 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:07.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:07.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:07.397 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:07.430 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:07.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:07.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:07.532 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:07.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:07.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:07.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:07 smithi098 ceph-mon[97580]: pgmap v1190: 99 pgs: 99 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 138 MiB/s wr, 2.20k op/s 2024-09-21T10:27:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:07.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:07 smithi098 ceph-mon[100914]: pgmap v1190: 99 pgs: 99 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 138 MiB/s wr, 2.20k op/s 2024-09-21T10:27:07.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:07.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:07 smithi196 ceph-mon[84784]: pgmap v1190: 99 pgs: 99 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 138 MiB/s wr, 2.20k op/s 2024-09-21T10:27:07.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:07.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:07.710 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:07.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:07.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:07.739 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:27:07.811 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:07.872 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:07.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:07.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:07.974 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:08.009 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:08.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:08.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:08.146 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:08.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:08.185 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 21 16 50703 50687 150.802 107.812 0.0240163 0.00662454 2024-09-21T10:27:08.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:08.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.287 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:08.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:08.388 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:08.424 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:08.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:08.526 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:08.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.557 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:08.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:08.658 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:08.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.695 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:08.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:08.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:08.817 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:08.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:08.919 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:08.943 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:27:08.945 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:27:08.957 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:08.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:09.059 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:09.097 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:09.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:09.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:09.185 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 22 16 52305 52289 148.499 100.125 0.00652435 0.00672917 2024-09-21T10:27:09.200 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:09.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:09.236 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:09.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:09.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:09.368 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:09.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:09.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:09.470 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:09.506 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:09.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:09.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:09.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:09 smithi098 ceph-mon[97580]: pgmap v1191: 99 pgs: 99 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-21T10:27:09.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:09 smithi098 ceph-mon[100914]: pgmap v1191: 99 pgs: 99 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-21T10:27:09.608 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:09.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:09 smithi196 ceph-mon[84784]: pgmap v1191: 99 pgs: 99 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-21T10:27:09.644 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:09.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:09.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:09.746 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:09.781 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:09.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:09.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:09.835 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:27:09.835 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:27:09.836 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-21T10:27:09.836 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-21T10:27:09.836 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:27:09.882 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:09.917 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:09.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:09.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:10.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:10.054 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:10.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:10.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:10.156 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:10.185 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 23 16 53999 53983 146.646 105.875 0.011228 0.00681372 2024-09-21T10:27:10.195 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:10.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:10.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:10.297 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:10.342 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:10.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:10.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:10.386 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:10 smithi098 ceph-mon[97580]: from='client.46337 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:27:10.386 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:10 smithi098 ceph-mon[100914]: from='client.46337 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:27:10.444 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:10.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:10.479 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:10.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:10.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:10.616 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:10.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:10.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:10.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:10 smithi196 ceph-mon[84784]: from='client.46337 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:27:10.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:27:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:27:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:27:10.717 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:10.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:10.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:10.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:10.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:10.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:10.897 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:10.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:10.935 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:27:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:27:10] "GET /metrics HTTP/1.1" 200 36577 "" "Prometheus/2.51.0" 2024-09-21T10:27:10.976 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:27:10.999 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:11.032 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:11.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:11.133 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:11.175 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:11.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:11.186 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 24 16 55537 55521 144.541 96.125 0.0101243 0.0069133 2024-09-21T10:27:11.276 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:11.310 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:11.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:11.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:11.454 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:11.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:11.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:11.586 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:11.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:11.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:11 smithi098 ceph-mon[97580]: pgmap v1192: 99 pgs: 99 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-09-21T10:27:11.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:11 smithi098 ceph-mon[100914]: pgmap v1192: 99 pgs: 99 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-09-21T10:27:11.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:11 smithi196 ceph-mon[84784]: pgmap v1192: 99 pgs: 99 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-09-21T10:27:11.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:11.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.721 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:11.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:11.822 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:11.853 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:11.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:11.955 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:11.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.996 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:11.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:12.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:12.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:12.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:12.153 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1191,"stamp":"2024-09-21T10:27:07.538807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2755270605,"num_objects":41997,"num_object_clones":0,"num_object_copies":125991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41997,"num_whiteouts":0,"num_read":5983,"num_read_kb":13006,"num_write":42876,"num_write_kb":2698639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":10162,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44101,"ondisk_log_size":44101,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":297,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13489144,"kb_used_data":7932224,"kb_used_omap":30,"kb_used_meta":5556833,"kb_avail":642726920,"statfs":{"total":671965249536,"available":658152366080,"internally_reserved":0,"allocated":8122597376,"data_stored":8108812573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30993,"internal_metadata":5690197743},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1438777344,"num_objects":21954,"num_object_clones":0,"num_object_copies":65862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21954,"num_write_kb":1405056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003963"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":80,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:46.068310+0000","last_change":"2024-09-21T10:26:41.278510+0000","last_active":"2024-09-21T10:26:46.068310+0000","last_peered":"2024-09-21T10:26:46.068310+0000","last_clean":"2024-09-21T10:26:46.068310+0000","last_became_active":"2024-09-21T10:26:41.278198+0000","last_became_peered":"2024-09-21T10:26:41.278198+0000","last_unstale":"2024-09-21T10:26:46.068310+0000","last_undegraded":"2024-09-21T10:26:46.068310+0000","last_fullsized":"2024-09-21T10:26:46.068310+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:01:50.721912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":384,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.181817+0000","last_change":"2024-09-21T10:24:44.305198+0000","last_active":"2024-09-21T10:26:47.181817+0000","last_peered":"2024-09-21T10:26:47.181817+0000","last_clean":"2024-09-21T10:26:47.181817+0000","last_became_active":"2024-09-21T10:24:44.297653+0000","last_became_peered":"2024-09-21T10:24:44.297653+0000","last_unstale":"2024-09-21T10:26:47.181817+0000","last_undegraded":"2024-09-21T10:26:47.181817+0000","last_fullsized":"2024-09-21T10:26:47.181817+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:19.320833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":211,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:46.068240+0000","last_change":"2024-09-21T10:26:41.304971+0000","last_active":"2024-09-21T10:26:46.068240+0000","last_peered":"2024-09-21T10:26:46.068240+0000","last_clean":"2024-09-21T10:26:46.068240+0000","last_became_active":"2024-09-21T10:26:41.304562+0000","last_became_peered":"2024-09-21T10:26:41.304562+0000","last_unstale":"2024-09-21T10:26:46.068240+0000","last_undegraded":"2024-09-21T10:26:46.068240+0000","last_fullsized":"2024-09-21T10:26:46.068240+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:33:50.716233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":160,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.181416+0000","last_change":"2024-09-21T10:18:33.475291+0000","last_active":"2024-09-21T10:26:47.181416+0000","last_peered":"2024-09-21T10:26:47.181416+0000","last_clean":"2024-09-21T10:26:47.181416+0000","last_became_active":"2024-09-21T10:18:33.474971+0000","last_became_peered":"2024-09-21T10:18:33.474971+0000","last_unstale":"2024-09-21T10:26:47.181416+0000","last_undegraded":"2024-09-21T10:26:47.181416+0000","last_fullsized":"2024-09-21T10:26:47.181416+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:39:40.179668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":586,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:26:48.329512+0000","last_change":"2024-09-21T10:18:33.475179+0000","last_active":"2024-09-21T10:26:48.329512+0000","last_peered":"2024-09-21T10:26:48.329512+0000","last_clean":"2024-09-21T10:26:48.329512+0000","last_became_active":"2024-09-21T10:18:33.474914+0000","last_became_peered":"2024-09-21T10:18:33.474914+0000","last_unstale":"2024-09-21T10:26:48.329512+0000","last_undegraded":"2024-09-21T10:26:48.329512+0000","last_fullsized":"2024-09-21T10:26:48.329512+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:15:32.186986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":43,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:45.288351+0000","last_change":"2024-09-21T10:24:44.304399+0000","last_active":"2024-09-21T10:26:45.288351+0000","last_peered":"2024-09-21T10:26:45.288351+0000","last_clean":"2024-09-21T10:26:45.288351+0000","last_became_active":"2024-09-21T10:24:44.303437+0000","last_became_peered":"2024-09-21T10:24:44.303437+0000","last_unstale":"2024-09-21T10:26:45.288351+0000","last_undegraded":"2024-09-21T10:26:45.288351+0000","last_fullsized":"2024-09-21T10:26:45.288351+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:32:56.638481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":46,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:26:48.328715+0000","last_change":"2024-09-21T10:25:39.876792+0000","last_active":"2024-09-21T10:26:48.328715+0000","last_peered":"2024-09-21T10:26:48.328715+0000","last_clean":"2024-09-21T10:26:48.328715+0000","last_became_active":"2024-09-21T10:25:39.876336+0000","last_became_peered":"2024-09-21T10:25:39.876336+0000","last_unstale":"2024-09-21T10:26:48.328715+0000","last_undegraded":"2024-09-21T10:26:48.328715+0000","last_fullsized":"2024-09-21T10:26:48.328715+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:06:57.688437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":326,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.181864+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:26:47.181864+0000","last_peered":"2024-09-21T10:26:47.181864+0000","last_clean":"2024-09-21T10:26:47.181864+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:26:47.181864+0000","last_undegraded":"2024-09-21T10:26:47.181864+0000","last_fullsized":"2024-09-21T10:26:47.181864+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"354'2596","reported_seq":2619,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:27:03.542135+0000","last_change":"2024-09-21T10:26:36.989699+0000","last_active":"2024-09-21T10:27:03.542135+0000","last_peered":"2024-09-21T10:27:03.542135+0000","last_clean":"2024-09-21T10:27:03.542135+0000","last_became_active":"2024-09-21T10:26:36.989318+0000","last_became_peered":"2024-09-21T10:26:36.989318+0000","last_unstale":"2024-09-21T10:27:03.542135+0000","last_undegraded":"2024-09-21T10:27:03.542135+0000","last_fullsized":"2024-09-21T10:27:03.542135+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":2596,"log_dups_size":0,"ondisk_log_size":2596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:13:32.624712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170131456,"num_objects":2596,"num_object_clones":0,"num_object_copies":7788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2596,"num_write_kb":166144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":375,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.182253+0000","last_change":"2024-09-21T10:24:44.325196+0000","last_active":"2024-09-21T10:26:47.182253+0000","last_peered":"2024-09-21T10:26:47.182253+0000","last_clean":"2024-09-21T10:26:47.182253+0000","last_became_active":"2024-09-21T10:24:44.299614+0000","last_became_peered":"2024-09-21T10:24:44.299614+0000","last_unstale":"2024-09-21T10:26:47.182253+0000","last_undegraded":"2024-09-21T10:26:47.182253+0000","last_fullsized":"2024-09-21T10:26:47.182253+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:14.558663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":107,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:46.067974+0000","last_change":"2024-09-21T10:26:41.305571+0000","last_active":"2024-09-21T10:26:46.067974+0000","last_peered":"2024-09-21T10:26:46.067974+0000","last_clean":"2024-09-21T10:26:46.067974+0000","last_became_active":"2024-09-21T10:26:41.305135+0000","last_became_peered":"2024-09-21T10:26:41.305135+0000","last_unstale":"2024-09-21T10:26:46.067974+0000","last_undegraded":"2024-09-21T10:26:46.067974+0000","last_fullsized":"2024-09-21T10:26:46.067974+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:49:55.452467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"354'2486","reported_seq":2500,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:27:02.912207+0000","last_change":"2024-09-21T10:26:36.989381+0000","last_active":"2024-09-21T10:27:02.912207+0000","last_peered":"2024-09-21T10:27:02.912207+0000","last_clean":"2024-09-21T10:27:02.912207+0000","last_became_active":"2024-09-21T10:26:36.988959+0000","last_became_peered":"2024-09-21T10:26:36.988959+0000","last_unstale":"2024-09-21T10:27:02.912207+0000","last_undegraded":"2024-09-21T10:27:02.912207+0000","last_fullsized":"2024-09-21T10:27:02.912207+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":2486,"log_dups_size":0,"ondisk_log_size":2486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:16:10.102647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162922496,"num_objects":2486,"num_object_clones":0,"num_object_copies":7458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2486,"num_write_kb":159104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":376,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.182415+0000","last_change":"2024-09-21T10:24:44.323182+0000","last_active":"2024-09-21T10:26:47.182415+0000","last_peered":"2024-09-21T10:26:47.182415+0000","last_clean":"2024-09-21T10:26:47.182415+0000","last_became_active":"2024-09-21T10:24:44.299698+0000","last_became_peered":"2024-09-21T10:24:44.299698+0000","last_unstale":"2024-09-21T10:26:47.182415+0000","last_undegraded":"2024-09-21T10:26:47.182415+0000","last_fullsized":"2024-09-21T10:26:47.182415+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:56.571057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":43,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:45.288466+0000","last_change":"2024-09-21T10:24:44.304586+0000","last_active":"2024-09-21T10:26:45.288466+0000","last_peered":"2024-09-21T10:26:45.288466+0000","last_clean":"2024-09-21T10:26:45.288466+0000","last_became_active":"2024-09-21T10:24:44.303815+0000","last_became_peered":"2024-09-21T10:24:44.303815+0000","last_unstale":"2024-09-21T10:26:45.288466+0000","last_undegraded":"2024-09-21T10:26:45.288466+0000","last_fullsized":"2024-09-21T10:26:45.288466+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:43:02.126454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"354'2561","reported_seq":2584,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:27:03.526023+0000","last_change":"2024-09-21T10:26:36.196791+0000","last_active":"2024-09-21T10:27:03.526023+0000","last_peered":"2024-09-21T10:27:03.526023+0000","last_clean":"2024-09-21T10:27:03.526023+0000","last_became_active":"2024-09-21T10:26:36.196478+0000","last_became_peered":"2024-09-21T10:26:36.196478+0000","last_unstale":"2024-09-21T10:27:03.526023+0000","last_undegraded":"2024-09-21T10:27:03.526023+0000","last_fullsized":"2024-09-21T10:27:03.526023+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":2561,"log_dups_size":0,"ondisk_log_size":2561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:03:16.836679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167837696,"num_objects":2561,"num_object_clones":0,"num_object_copies":7683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2561,"num_write_kb":163904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":302,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.182654+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:26:47.182654+0000","last_peered":"2024-09-21T10:26:47.182654+0000","last_clean":"2024-09-21T10:26:47.182654+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:26:47.182654+0000","last_undegraded":"2024-09-21T10:26:47.182654+0000","last_fullsized":"2024-09-21T10:26:47.182654+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":317,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.181998+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:26:47.181998+0000","last_peered":"2024-09-21T10:26:47.181998+0000","last_clean":"2024-09-21T10:26:47.181998+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:26:47.181998+0000","last_undegraded":"2024-09-21T10:26:47.181998+0000","last_fullsized":"2024-09-21T10:26:47.181998+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"355'2688","reported_seq":2713,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:04.755023+0000","last_change":"2024-09-21T10:26:36.196873+0000","last_active":"2024-09-21T10:27:04.755023+0000","last_peered":"2024-09-21T10:27:04.755023+0000","last_clean":"2024-09-21T10:27:04.755023+0000","last_became_active":"2024-09-21T10:26:36.196596+0000","last_became_peered":"2024-09-21T10:26:36.196596+0000","last_unstale":"2024-09-21T10:27:04.755023+0000","last_undegraded":"2024-09-21T10:27:04.755023+0000","last_fullsized":"2024-09-21T10:27:04.755023+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":2688,"log_dups_size":0,"ondisk_log_size":2688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:32:15.576035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":2688,"num_object_clones":0,"num_object_copies":8064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2688,"num_write_kb":172032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":152,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.185114+0000","last_change":"2024-09-21T10:18:33.483457+0000","last_active":"2024-09-21T10:26:47.185114+0000","last_peered":"2024-09-21T10:26:47.185114+0000","last_clean":"2024-09-21T10:26:47.185114+0000","last_became_active":"2024-09-21T10:18:33.483140+0000","last_became_peered":"2024-09-21T10:18:33.483140+0000","last_unstale":"2024-09-21T10:26:47.185114+0000","last_undegraded":"2024-09-21T10:26:47.185114+0000","last_fullsized":"2024-09-21T10:26:47.185114+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:37:39.302566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":615,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:26:48.328508+0000","last_change":"2024-09-21T10:24:44.299417+0000","last_active":"2024-09-21T10:26:48.328508+0000","last_peered":"2024-09-21T10:26:48.328508+0000","last_clean":"2024-09-21T10:26:48.328508+0000","last_became_active":"2024-09-21T10:24:44.299116+0000","last_became_peered":"2024-09-21T10:24:44.299116+0000","last_unstale":"2024-09-21T10:26:48.328508+0000","last_undegraded":"2024-09-21T10:26:48.328508+0000","last_fullsized":"2024-09-21T10:26:48.328508+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:44.524841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"355'3127","reported_seq":3152,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:07.327869+0000","last_change":"2024-09-21T10:26:36.196642+0000","last_active":"2024-09-21T10:27:07.327869+0000","last_peered":"2024-09-21T10:27:07.327869+0000","last_clean":"2024-09-21T10:27:07.327869+0000","last_became_active":"2024-09-21T10:26:36.196357+0000","last_became_peered":"2024-09-21T10:26:36.196357+0000","last_unstale":"2024-09-21T10:27:07.327869+0000","last_undegraded":"2024-09-21T10:27:07.327869+0000","last_fullsized":"2024-09-21T10:27:07.327869+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":3127,"log_dups_size":0,"ondisk_log_size":3127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:09:43.765205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204931072,"num_objects":3127,"num_object_clones":0,"num_object_copies":9381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3127,"num_write_kb":200128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"82'100","reported_seq":465,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.181439+0000","last_change":"2024-09-21T10:23:22.728269+0000","last_active":"2024-09-21T10:26:47.181439+0000","last_peered":"2024-09-21T10:26:47.181439+0000","last_clean":"2024-09-21T10:26:47.181439+0000","last_became_active":"2024-09-21T10:23:20.700241+0000","last_became_peered":"2024-09-21T10:23:20.700241+0000","last_unstale":"2024-09-21T10:26:47.181439+0000","last_undegraded":"2024-09-21T10:26:47.181439+0000","last_fullsized":"2024-09-21T10:26:47.181439+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.728195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.728195+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:31.235642+0000","scrub_duration":0.0069698140000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":47,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:45.288598+0000","last_change":"2024-09-21T10:24:44.327042+0000","last_active":"2024-09-21T10:26:45.288598+0000","last_peered":"2024-09-21T10:26:45.288598+0000","last_clean":"2024-09-21T10:26:45.288598+0000","last_became_active":"2024-09-21T10:24:44.303785+0000","last_became_peered":"2024-09-21T10:24:44.303785+0000","last_unstale":"2024-09-21T10:26:45.288598+0000","last_undegraded":"2024-09-21T10:26:45.288598+0000","last_fullsized":"2024-09-21T10:26:45.288598+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:11:19.180549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":43,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:45.288675+0000","last_change":"2024-09-21T10:24:44.306721+0000","last_active":"2024-09-21T10:26:45.288675+0000","last_peered":"2024-09-21T10:26:45.288675+0000","last_clean":"2024-09-21T10:26:45.288675+0000","last_became_active":"2024-09-21T10:24:44.306427+0000","last_became_peered":"2024-09-21T10:24:44.306427+0000","last_unstale":"2024-09-21T10:26:45.288675+0000","last_undegraded":"2024-09-21T10:26:45.288675+0000","last_fullsized":"2024-09-21T10:26:45.288675+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:33:08.866485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"355'2808","reported_seq":2832,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:04.775325+0000","last_change":"2024-09-21T10:26:36.989578+0000","last_active":"2024-09-21T10:27:04.775325+0000","last_peered":"2024-09-21T10:27:04.775325+0000","last_clean":"2024-09-21T10:27:04.775325+0000","last_became_active":"2024-09-21T10:26:36.989250+0000","last_became_peered":"2024-09-21T10:26:36.989250+0000","last_unstale":"2024-09-21T10:27:04.775325+0000","last_undegraded":"2024-09-21T10:27:04.775325+0000","last_fullsized":"2024-09-21T10:27:04.775325+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":2808,"log_dups_size":0,"ondisk_log_size":2808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:08:31.403222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184025088,"num_objects":2808,"num_object_clones":0,"num_object_copies":8424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2808,"num_write_kb":179712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'100","reported_seq":108,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:04.302658+0000","last_change":"2024-09-21T10:23:27.234299+0000","last_active":"2024-09-21T10:27:04.302658+0000","last_peered":"2024-09-21T10:27:04.302658+0000","last_clean":"2024-09-21T10:27:04.302658+0000","last_became_active":"2024-09-21T10:23:20.698155+0000","last_became_peered":"2024-09-21T10:23:20.698155+0000","last_unstale":"2024-09-21T10:27:04.302658+0000","last_undegraded":"2024-09-21T10:27:04.302658+0000","last_fullsized":"2024-09-21T10:27:04.302658+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:27.234001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:27.234001+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:13:43.690329+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":368,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.182098+0000","last_change":"2024-09-21T10:24:44.309470+0000","last_active":"2024-09-21T10:26:47.182098+0000","last_peered":"2024-09-21T10:26:47.182098+0000","last_clean":"2024-09-21T10:26:47.182098+0000","last_became_active":"2024-09-21T10:24:44.308836+0000","last_became_peered":"2024-09-21T10:24:44.308836+0000","last_unstale":"2024-09-21T10:26:47.182098+0000","last_undegraded":"2024-09-21T10:26:47.182098+0000","last_fullsized":"2024-09-21T10:26:47.182098+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:09:44.622418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":43,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:45.288896+0000","last_change":"2024-09-21T10:24:44.305135+0000","last_active":"2024-09-21T10:26:45.288896+0000","last_peered":"2024-09-21T10:26:45.288896+0000","last_clean":"2024-09-21T10:26:45.288896+0000","last_became_active":"2024-09-21T10:24:44.304839+0000","last_became_peered":"2024-09-21T10:24:44.304839+0000","last_unstale":"2024-09-21T10:26:45.288896+0000","last_undegraded":"2024-09-21T10:26:45.288896+0000","last_fullsized":"2024-09-21T10:26:45.288896+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:02.264404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"354'2570","reported_seq":2584,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:27:02.918721+0000","last_change":"2024-09-21T10:26:36.194994+0000","last_active":"2024-09-21T10:27:02.918721+0000","last_peered":"2024-09-21T10:27:02.918721+0000","last_clean":"2024-09-21T10:27:02.918721+0000","last_became_active":"2024-09-21T10:26:36.194404+0000","last_became_peered":"2024-09-21T10:26:36.194404+0000","last_unstale":"2024-09-21T10:27:02.918721+0000","last_undegraded":"2024-09-21T10:27:02.918721+0000","last_fullsized":"2024-09-21T10:27:02.918721+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":2570,"log_dups_size":0,"ondisk_log_size":2570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:10:51.551059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168427520,"num_objects":2570,"num_object_clones":0,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2570,"num_write_kb":164480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":46,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:26:48.327878+0000","last_change":"2024-09-21T10:25:39.872680+0000","last_active":"2024-09-21T10:26:48.327878+0000","last_peered":"2024-09-21T10:26:48.327878+0000","last_clean":"2024-09-21T10:26:48.327878+0000","last_became_active":"2024-09-21T10:25:39.872312+0000","last_became_peered":"2024-09-21T10:25:39.872312+0000","last_unstale":"2024-09-21T10:26:48.327878+0000","last_undegraded":"2024-09-21T10:26:48.327878+0000","last_fullsized":"2024-09-21T10:26:48.327878+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:54.942046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":43,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:45.288940+0000","last_change":"2024-09-21T10:24:44.310484+0000","last_active":"2024-09-21T10:26:45.288940+0000","last_peered":"2024-09-21T10:26:45.288940+0000","last_clean":"2024-09-21T10:26:45.288940+0000","last_became_active":"2024-09-21T10:24:44.307209+0000","last_became_peered":"2024-09-21T10:24:44.307209+0000","last_unstale":"2024-09-21T10:26:45.288940+0000","last_undegraded":"2024-09-21T10:26:45.288940+0000","last_fullsized":"2024-09-21T10:26:45.288940+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:48:36.767539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"354'2610","reported_seq":2624,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:27:02.904610+0000","last_change":"2024-09-21T10:26:36.196826+0000","last_active":"2024-09-21T10:27:02.904610+0000","last_peered":"2024-09-21T10:27:02.904610+0000","last_clean":"2024-09-21T10:27:02.904610+0000","last_became_active":"2024-09-21T10:26:36.196355+0000","last_became_peered":"2024-09-21T10:26:36.196355+0000","last_unstale":"2024-09-21T10:27:02.904610+0000","last_undegraded":"2024-09-21T10:27:02.904610+0000","last_fullsized":"2024-09-21T10:27:02.904610+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":2610,"log_dups_size":0,"ondisk_log_size":2610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:58.869309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171048960,"num_objects":2610,"num_object_clones":0,"num_object_copies":7830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2610,"num_write_kb":167040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'100","reported_seq":103,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:04.302493+0000","last_change":"2024-09-21T10:23:26.197358+0000","last_active":"2024-09-21T10:27:04.302493+0000","last_peered":"2024-09-21T10:27:04.302493+0000","last_clean":"2024-09-21T10:27:04.302493+0000","last_became_active":"2024-09-21T10:23:20.695174+0000","last_became_peered":"2024-09-21T10:23:20.695174+0000","last_unstale":"2024-09-21T10:27:04.302493+0000","last_undegraded":"2024-09-21T10:27:04.302493+0000","last_fullsized":"2024-09-21T10:27:04.302493+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:26.197291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:26.197291+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:33.961031+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":242,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:04.305832+0000","last_change":"2024-09-21T10:24:44.308818+0000","last_active":"2024-09-21T10:27:04.305832+0000","last_peered":"2024-09-21T10:27:04.305832+0000","last_clean":"2024-09-21T10:27:04.305832+0000","last_became_active":"2024-09-21T10:24:44.308014+0000","last_became_peered":"2024-09-21T10:24:44.308014+0000","last_unstale":"2024-09-21T10:27:04.305832+0000","last_undegraded":"2024-09-21T10:27:04.305832+0000","last_fullsized":"2024-09-21T10:27:04.305832+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:36:14.730099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":605,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:04.302891+0000","last_change":"2024-09-21T10:24:44.309294+0000","last_active":"2024-09-21T10:27:04.302891+0000","last_peered":"2024-09-21T10:27:04.302891+0000","last_clean":"2024-09-21T10:27:04.302891+0000","last_became_active":"2024-09-21T10:24:44.308928+0000","last_became_peered":"2024-09-21T10:24:44.308928+0000","last_unstale":"2024-09-21T10:27:04.302891+0000","last_undegraded":"2024-09-21T10:27:04.302891+0000","last_fullsized":"2024-09-21T10:27:04.302891+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:50:20.141648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"354'2621","reported_seq":2635,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:27:02.915171+0000","last_change":"2024-09-21T10:26:36.197465+0000","last_active":"2024-09-21T10:27:02.915171+0000","last_peered":"2024-09-21T10:27:02.915171+0000","last_clean":"2024-09-21T10:27:02.915171+0000","last_became_active":"2024-09-21T10:26:36.196105+0000","last_became_peered":"2024-09-21T10:26:36.196105+0000","last_unstale":"2024-09-21T10:27:02.915171+0000","last_undegraded":"2024-09-21T10:27:02.915171+0000","last_fullsized":"2024-09-21T10:27:02.915171+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":2621,"log_dups_size":0,"ondisk_log_size":2621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:16.586685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171769856,"num_objects":2621,"num_object_clones":0,"num_object_copies":7863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2621,"num_write_kb":167744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":456,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.181267+0000","last_change":"2024-09-21T10:23:18.790100+0000","last_active":"2024-09-21T10:26:47.181267+0000","last_peered":"2024-09-21T10:26:47.181267+0000","last_clean":"2024-09-21T10:26:47.181267+0000","last_became_active":"2024-09-21T10:23:14.677610+0000","last_became_peered":"2024-09-21T10:23:14.677610+0000","last_unstale":"2024-09-21T10:26:47.181267+0000","last_undegraded":"2024-09-21T10:26:47.181267+0000","last_fullsized":"2024-09-21T10:26:47.181267+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:18.790017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:18.790017+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:56.562241+0000","scrub_duration":0.0080400790000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":549,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:46.068198+0000","last_change":"2024-09-21T10:26:41.279374+0000","last_active":"2024-09-21T10:26:46.068198+0000","last_peered":"2024-09-21T10:26:46.068198+0000","last_clean":"2024-09-21T10:26:46.068198+0000","last_became_active":"2024-09-21T10:26:41.278994+0000","last_became_peered":"2024-09-21T10:26:41.278994+0000","last_unstale":"2024-09-21T10:26:46.068198+0000","last_undegraded":"2024-09-21T10:26:46.068198+0000","last_fullsized":"2024-09-21T10:26:46.068198+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:49:57.821175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":482,"num_read_kb":1581,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":88,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:46.068120+0000","last_change":"2024-09-21T10:26:41.305053+0000","last_active":"2024-09-21T10:26:46.068120+0000","last_peered":"2024-09-21T10:26:46.068120+0000","last_clean":"2024-09-21T10:26:46.068120+0000","last_became_active":"2024-09-21T10:26:41.304677+0000","last_became_peered":"2024-09-21T10:26:41.304677+0000","last_unstale":"2024-09-21T10:26:46.068120+0000","last_undegraded":"2024-09-21T10:26:46.068120+0000","last_fullsized":"2024-09-21T10:26:46.068120+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:11:11.992908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":340,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.184357+0000","last_change":"2024-09-21T10:24:44.308008+0000","last_active":"2024-09-21T10:26:47.184357+0000","last_peered":"2024-09-21T10:26:47.184357+0000","last_clean":"2024-09-21T10:26:47.184357+0000","last_became_active":"2024-09-21T10:24:44.307595+0000","last_became_peered":"2024-09-21T10:24:44.307595+0000","last_unstale":"2024-09-21T10:26:47.184357+0000","last_undegraded":"2024-09-21T10:26:47.184357+0000","last_fullsized":"2024-09-21T10:26:47.184357+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:33:49.448676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"354'2606","reported_seq":2629,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:27:03.528221+0000","last_change":"2024-09-21T10:26:36.197089+0000","last_active":"2024-09-21T10:27:03.528221+0000","last_peered":"2024-09-21T10:27:03.528221+0000","last_clean":"2024-09-21T10:27:03.528221+0000","last_became_active":"2024-09-21T10:26:36.196846+0000","last_became_peered":"2024-09-21T10:26:36.196846+0000","last_unstale":"2024-09-21T10:27:03.528221+0000","last_undegraded":"2024-09-21T10:27:03.528221+0000","last_fullsized":"2024-09-21T10:27:03.528221+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":2606,"log_dups_size":0,"ondisk_log_size":2606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:33:42.447848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170786816,"num_objects":2606,"num_object_clones":0,"num_object_copies":7818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2606,"num_write_kb":166784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"82'100","reported_seq":110,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:04.303704+0000","last_change":"2024-09-21T10:24:44.309154+0000","last_active":"2024-09-21T10:27:04.303704+0000","last_peered":"2024-09-21T10:27:04.303704+0000","last_clean":"2024-09-21T10:27:04.303704+0000","last_became_active":"2024-09-21T10:24:44.308330+0000","last_became_peered":"2024-09-21T10:24:44.308330+0000","last_unstale":"2024-09-21T10:27:04.303704+0000","last_undegraded":"2024-09-21T10:27:04.303704+0000","last_fullsized":"2024-09-21T10:27:04.303704+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.641718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.641718+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:10.158739+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":698,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.181282+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:26:47.181282+0000","last_peered":"2024-09-21T10:26:47.181282+0000","last_clean":"2024-09-21T10:26:47.181282+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:26:47.181282+0000","last_undegraded":"2024-09-21T10:26:47.181282+0000","last_fullsized":"2024-09-21T10:26:47.181282+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":616,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:26:48.328117+0000","last_change":"2024-09-21T10:24:44.299102+0000","last_active":"2024-09-21T10:26:48.328117+0000","last_peered":"2024-09-21T10:26:48.328117+0000","last_clean":"2024-09-21T10:26:48.328117+0000","last_became_active":"2024-09-21T10:24:44.298723+0000","last_became_peered":"2024-09-21T10:24:44.298723+0000","last_unstale":"2024-09-21T10:26:48.328117+0000","last_undegraded":"2024-09-21T10:26:48.328117+0000","last_fullsized":"2024-09-21T10:26:48.328117+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:57:47.078328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":591,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:26:48.329705+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:26:48.329705+0000","last_peered":"2024-09-21T10:26:48.329705+0000","last_clean":"2024-09-21T10:26:48.329705+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:26:48.329705+0000","last_undegraded":"2024-09-21T10:26:48.329705+0000","last_fullsized":"2024-09-21T10:26:48.329705+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"354'2607","reported_seq":2622,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:27:02.901375+0000","last_change":"2024-09-21T10:26:41.278294+0000","last_active":"2024-09-21T10:27:02.901375+0000","last_peered":"2024-09-21T10:27:02.901375+0000","last_clean":"2024-09-21T10:27:02.901375+0000","last_became_active":"2024-09-21T10:26:41.277965+0000","last_became_peered":"2024-09-21T10:26:41.277965+0000","last_unstale":"2024-09-21T10:27:02.901375+0000","last_undegraded":"2024-09-21T10:27:02.901375+0000","last_fullsized":"2024-09-21T10:27:02.901375+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":348,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":2607,"log_dups_size":0,"ondisk_log_size":2607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:30.913837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170852352,"num_objects":2607,"num_object_clones":0,"num_object_copies":7821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2607,"num_write_kb":166848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"82'100","reported_seq":254,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:26:48.329422+0000","last_change":"2024-09-21T10:24:44.294460+0000","last_active":"2024-09-21T10:26:48.329422+0000","last_peered":"2024-09-21T10:26:48.329422+0000","last_clean":"2024-09-21T10:26:48.329422+0000","last_became_active":"2024-09-21T10:24:44.293972+0000","last_became_peered":"2024-09-21T10:24:44.293972+0000","last_unstale":"2024-09-21T10:26:48.329422+0000","last_undegraded":"2024-09-21T10:26:48.329422+0000","last_fullsized":"2024-09-21T10:26:48.329422+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:26.199522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:26.199522+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:26.135171+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":1367,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:04.308428+0000","last_change":"2024-09-21T10:18:36.495308+0000","last_active":"2024-09-21T10:27:04.308428+0000","last_peered":"2024-09-21T10:27:04.308428+0000","last_clean":"2024-09-21T10:27:04.308428+0000","last_became_active":"2024-09-21T10:18:36.494662+0000","last_became_peered":"2024-09-21T10:18:36.494662+0000","last_unstale":"2024-09-21T10:27:04.308428+0000","last_undegraded":"2024-09-21T10:27:04.308428+0000","last_fullsized":"2024-09-21T10:27:04.308428+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:49.218676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":465,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:04.305919+0000","last_change":"2024-09-21T10:08:18.470748+0000","last_active":"2024-09-21T10:27:04.305919+0000","last_peered":"2024-09-21T10:27:04.305919+0000","last_clean":"2024-09-21T10:27:04.305919+0000","last_became_active":"2024-09-21T10:08:18.470227+0000","last_became_peered":"2024-09-21T10:08:18.470227+0000","last_unstale":"2024-09-21T10:27:04.305919+0000","last_undegraded":"2024-09-21T10:27:04.305919+0000","last_fullsized":"2024-09-21T10:27:04.305919+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:16.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":587,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:04.305296+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:27:04.305296+0000","last_peered":"2024-09-21T10:27:04.305296+0000","last_clean":"2024-09-21T10:27:04.305296+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:27:04.305296+0000","last_undegraded":"2024-09-21T10:27:04.305296+0000","last_fullsized":"2024-09-21T10:27:04.305296+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"354'2604","reported_seq":2618,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:27:02.927944+0000","last_change":"2024-09-21T10:26:36.989641+0000","last_active":"2024-09-21T10:27:02.927944+0000","last_peered":"2024-09-21T10:27:02.927944+0000","last_clean":"2024-09-21T10:27:02.927944+0000","last_became_active":"2024-09-21T10:26:36.989340+0000","last_became_peered":"2024-09-21T10:26:36.989340+0000","last_unstale":"2024-09-21T10:27:02.927944+0000","last_undegraded":"2024-09-21T10:27:02.927944+0000","last_fullsized":"2024-09-21T10:27:02.927944+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":2604,"log_dups_size":0,"ondisk_log_size":2604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:50.213332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170655744,"num_objects":2604,"num_object_clones":0,"num_object_copies":7812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2604,"num_write_kb":166656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"82'100","reported_seq":247,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:26:48.328030+0000","last_change":"2024-09-21T10:23:21.277739+0000","last_active":"2024-09-21T10:26:48.328030+0000","last_peered":"2024-09-21T10:26:48.328030+0000","last_clean":"2024-09-21T10:26:48.328030+0000","last_became_active":"2024-09-21T10:23:20.689733+0000","last_became_peered":"2024-09-21T10:23:20.689733+0000","last_unstale":"2024-09-21T10:26:48.328030+0000","last_undegraded":"2024-09-21T10:26:48.328030+0000","last_fullsized":"2024-09-21T10:26:48.328030+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.277663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.277663+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:01:54.523189+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":1064,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:04.306530+0000","last_change":"2024-09-21T10:07:43.794767+0000","last_active":"2024-09-21T10:27:04.306530+0000","last_peered":"2024-09-21T10:27:04.306530+0000","last_clean":"2024-09-21T10:27:04.306530+0000","last_became_active":"2024-09-21T10:07:43.794062+0000","last_became_peered":"2024-09-21T10:07:43.794062+0000","last_unstale":"2024-09-21T10:27:04.306530+0000","last_undegraded":"2024-09-21T10:27:04.306530+0000","last_fullsized":"2024-09-21T10:27:04.306530+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:51.977955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":43,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:45.288673+0000","last_change":"2024-09-21T10:24:44.299492+0000","last_active":"2024-09-21T10:26:45.288673+0000","last_peered":"2024-09-21T10:26:45.288673+0000","last_clean":"2024-09-21T10:26:45.288673+0000","last_became_active":"2024-09-21T10:24:44.299187+0000","last_became_peered":"2024-09-21T10:24:44.299187+0000","last_unstale":"2024-09-21T10:26:45.288673+0000","last_undegraded":"2024-09-21T10:26:45.288673+0000","last_fullsized":"2024-09-21T10:26:45.288673+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:58.500386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":511,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:04.303358+0000","last_change":"2024-09-21T10:18:33.465815+0000","last_active":"2024-09-21T10:27:04.303358+0000","last_peered":"2024-09-21T10:27:04.303358+0000","last_clean":"2024-09-21T10:27:04.303358+0000","last_became_active":"2024-09-21T10:18:33.465459+0000","last_became_peered":"2024-09-21T10:18:33.465459+0000","last_unstale":"2024-09-21T10:27:04.303358+0000","last_undegraded":"2024-09-21T10:27:04.303358+0000","last_fullsized":"2024-09-21T10:27:04.303358+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:22:08.199800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"354'2492","reported_seq":2506,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:27:02.896066+0000","last_change":"2024-09-21T10:26:36.196471+0000","last_active":"2024-09-21T10:27:02.896066+0000","last_peered":"2024-09-21T10:27:02.896066+0000","last_clean":"2024-09-21T10:27:02.896066+0000","last_became_active":"2024-09-21T10:26:36.196167+0000","last_became_peered":"2024-09-21T10:26:36.196167+0000","last_unstale":"2024-09-21T10:27:02.896066+0000","last_undegraded":"2024-09-21T10:27:02.896066+0000","last_fullsized":"2024-09-21T10:27:02.896066+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":2492,"log_dups_size":0,"ondisk_log_size":2492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:16.488795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163315712,"num_objects":2492,"num_object_clones":0,"num_object_copies":7476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2492,"num_write_kb":159488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"82'100","reported_seq":208,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.182826+0000","last_change":"2024-09-21T10:23:30.570074+0000","last_active":"2024-09-21T10:26:47.182826+0000","last_peered":"2024-09-21T10:26:47.182826+0000","last_clean":"2024-09-21T10:26:47.182826+0000","last_became_active":"2024-09-21T10:23:20.695270+0000","last_became_peered":"2024-09-21T10:23:20.695270+0000","last_unstale":"2024-09-21T10:26:47.182826+0000","last_undegraded":"2024-09-21T10:26:47.182826+0000","last_fullsized":"2024-09-21T10:26:47.182826+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:30.569982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:30.569982+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:06.011343+0000","scrub_duration":0.0051309349999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":142,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:46.068332+0000","last_change":"2024-09-21T10:26:41.305269+0000","last_active":"2024-09-21T10:26:46.068332+0000","last_peered":"2024-09-21T10:26:46.068332+0000","last_clean":"2024-09-21T10:26:46.068332+0000","last_became_active":"2024-09-21T10:26:41.304670+0000","last_became_peered":"2024-09-21T10:26:41.304670+0000","last_unstale":"2024-09-21T10:26:46.068332+0000","last_undegraded":"2024-09-21T10:26:46.068332+0000","last_fullsized":"2024-09-21T10:26:46.068332+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:30:18.379876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":1171,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:04.303390+0000","last_change":"2024-09-21T10:08:18.468995+0000","last_active":"2024-09-21T10:27:04.303390+0000","last_peered":"2024-09-21T10:27:04.303390+0000","last_clean":"2024-09-21T10:27:04.303390+0000","last_became_active":"2024-09-21T10:08:18.468243+0000","last_became_peered":"2024-09-21T10:08:18.468243+0000","last_unstale":"2024-09-21T10:27:04.303390+0000","last_undegraded":"2024-09-21T10:27:04.303390+0000","last_fullsized":"2024-09-21T10:27:04.303390+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:45.520968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":497,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:04.306779+0000","last_change":"2024-09-21T10:24:44.309614+0000","last_active":"2024-09-21T10:27:04.306779+0000","last_peered":"2024-09-21T10:27:04.306779+0000","last_clean":"2024-09-21T10:27:04.306779+0000","last_became_active":"2024-09-21T10:24:44.309187+0000","last_became_peered":"2024-09-21T10:24:44.309187+0000","last_unstale":"2024-09-21T10:27:04.306779+0000","last_undegraded":"2024-09-21T10:27:04.306779+0000","last_fullsized":"2024-09-21T10:27:04.306779+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:01.074803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"354'2596","reported_seq":2610,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:27:02.911501+0000","last_change":"2024-09-21T10:26:36.196357+0000","last_active":"2024-09-21T10:27:02.911501+0000","last_peered":"2024-09-21T10:27:02.911501+0000","last_clean":"2024-09-21T10:27:02.911501+0000","last_became_active":"2024-09-21T10:26:36.196060+0000","last_became_peered":"2024-09-21T10:26:36.196060+0000","last_unstale":"2024-09-21T10:27:02.911501+0000","last_undegraded":"2024-09-21T10:27:02.911501+0000","last_fullsized":"2024-09-21T10:27:02.911501+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":2596,"log_dups_size":0,"ondisk_log_size":2596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:46:34.643144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170131456,"num_objects":2596,"num_object_clones":0,"num_object_copies":7788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2596,"num_write_kb":166144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"82'100","reported_seq":218,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.183036+0000","last_change":"2024-09-21T10:24:44.308946+0000","last_active":"2024-09-21T10:26:47.183036+0000","last_peered":"2024-09-21T10:26:47.183036+0000","last_clean":"2024-09-21T10:26:47.183036+0000","last_became_active":"2024-09-21T10:24:44.308558+0000","last_became_peered":"2024-09-21T10:24:44.308558+0000","last_unstale":"2024-09-21T10:26:47.183036+0000","last_undegraded":"2024-09-21T10:26:47.183036+0000","last_fullsized":"2024-09-21T10:26:47.183036+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:24.536711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:24.536711+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:11:00.667553+0000","scrub_duration":0.0047525650000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":466,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:04.303277+0000","last_change":"2024-09-21T10:08:18.469907+0000","last_active":"2024-09-21T10:27:04.303277+0000","last_peered":"2024-09-21T10:27:04.303277+0000","last_clean":"2024-09-21T10:27:04.303277+0000","last_became_active":"2024-09-21T10:08:18.469443+0000","last_became_peered":"2024-09-21T10:08:18.469443+0000","last_unstale":"2024-09-21T10:27:04.303277+0000","last_undegraded":"2024-09-21T10:27:04.303277+0000","last_fullsized":"2024-09-21T10:27:04.303277+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:15.645803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":776,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.182729+0000","last_change":"2024-09-21T10:24:44.334847+0000","last_active":"2024-09-21T10:26:47.182729+0000","last_peered":"2024-09-21T10:26:47.182729+0000","last_clean":"2024-09-21T10:26:47.182729+0000","last_became_active":"2024-09-21T10:24:44.301321+0000","last_became_peered":"2024-09-21T10:24:44.301321+0000","last_unstale":"2024-09-21T10:26:47.182729+0000","last_undegraded":"2024-09-21T10:26:47.182729+0000","last_fullsized":"2024-09-21T10:26:47.182729+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:37.628940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":28,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:46.068124+0000","last_change":"2024-09-21T10:26:41.278937+0000","last_active":"2024-09-21T10:26:46.068124+0000","last_peered":"2024-09-21T10:26:46.068124+0000","last_clean":"2024-09-21T10:26:46.068124+0000","last_became_active":"2024-09-21T10:26:41.277726+0000","last_became_peered":"2024-09-21T10:26:41.277726+0000","last_unstale":"2024-09-21T10:26:46.068124+0000","last_undegraded":"2024-09-21T10:26:46.068124+0000","last_fullsized":"2024-09-21T10:26:46.068124+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:04.741057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"354'2519","reported_seq":2533,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:27:02.919683+0000","last_change":"2024-09-21T10:26:36.196641+0000","last_active":"2024-09-21T10:27:02.919683+0000","last_peered":"2024-09-21T10:27:02.919683+0000","last_clean":"2024-09-21T10:27:02.919683+0000","last_became_active":"2024-09-21T10:26:36.196351+0000","last_became_peered":"2024-09-21T10:26:36.196351+0000","last_unstale":"2024-09-21T10:27:02.919683+0000","last_undegraded":"2024-09-21T10:27:02.919683+0000","last_fullsized":"2024-09-21T10:27:02.919683+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":2519,"log_dups_size":0,"ondisk_log_size":2519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:41:54.280247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165085184,"num_objects":2519,"num_object_clones":0,"num_object_copies":7557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2519,"num_write_kb":161216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'100","reported_seq":43,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:45.288824+0000","last_change":"2024-09-21T10:24:44.296758+0000","last_active":"2024-09-21T10:26:45.288824+0000","last_peered":"2024-09-21T10:26:45.288824+0000","last_clean":"2024-09-21T10:26:45.288824+0000","last_became_active":"2024-09-21T10:24:44.296403+0000","last_became_peered":"2024-09-21T10:24:44.296403+0000","last_unstale":"2024-09-21T10:26:45.288824+0000","last_undegraded":"2024-09-21T10:26:45.288824+0000","last_fullsized":"2024-09-21T10:26:45.288824+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.105746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.105746+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:20:12.502081+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":117,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.182164+0000","last_change":"2024-09-21T10:18:33.489678+0000","last_active":"2024-09-21T10:26:47.182164+0000","last_peered":"2024-09-21T10:26:47.182164+0000","last_clean":"2024-09-21T10:26:47.182164+0000","last_became_active":"2024-09-21T10:18:33.474818+0000","last_became_peered":"2024-09-21T10:18:33.474818+0000","last_unstale":"2024-09-21T10:26:47.182164+0000","last_undegraded":"2024-09-21T10:26:47.182164+0000","last_fullsized":"2024-09-21T10:26:47.182164+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:49.828007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":890,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:04.306020+0000","last_change":"2024-09-21T10:24:46.844278+0000","last_active":"2024-09-21T10:27:04.306020+0000","last_peered":"2024-09-21T10:27:04.306020+0000","last_clean":"2024-09-21T10:27:04.306020+0000","last_became_active":"2024-09-21T10:24:46.840961+0000","last_became_peered":"2024-09-21T10:24:46.840961+0000","last_unstale":"2024-09-21T10:27:04.306020+0000","last_undegraded":"2024-09-21T10:27:04.306020+0000","last_fullsized":"2024-09-21T10:27:04.306020+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:49:48.823524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":146,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.181089+0000","last_change":"2024-09-21T10:18:33.479008+0000","last_active":"2024-09-21T10:26:47.181089+0000","last_peered":"2024-09-21T10:26:47.181089+0000","last_clean":"2024-09-21T10:26:47.181089+0000","last_became_active":"2024-09-21T10:18:33.478740+0000","last_became_peered":"2024-09-21T10:18:33.478740+0000","last_unstale":"2024-09-21T10:26:47.181089+0000","last_undegraded":"2024-09-21T10:26:47.181089+0000","last_fullsized":"2024-09-21T10:26:47.181089+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:16:52.594882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"354'2478","reported_seq":2492,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:27:02.921064+0000","last_change":"2024-09-21T10:26:36.195498+0000","last_active":"2024-09-21T10:27:02.921064+0000","last_peered":"2024-09-21T10:27:02.921064+0000","last_clean":"2024-09-21T10:27:02.921064+0000","last_became_active":"2024-09-21T10:26:36.195217+0000","last_became_peered":"2024-09-21T10:26:36.195217+0000","last_unstale":"2024-09-21T10:27:02.921064+0000","last_undegraded":"2024-09-21T10:27:02.921064+0000","last_fullsized":"2024-09-21T10:27:02.921064+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":2478,"log_dups_size":0,"ondisk_log_size":2478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:55:41.167304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162398208,"num_objects":2478,"num_object_clones":0,"num_object_copies":7434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2478,"num_write_kb":158592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"82'100","reported_seq":469,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.182355+0000","last_change":"2024-09-21T10:23:21.759441+0000","last_active":"2024-09-21T10:26:47.182355+0000","last_peered":"2024-09-21T10:26:47.182355+0000","last_clean":"2024-09-21T10:26:47.182355+0000","last_became_active":"2024-09-21T10:23:20.700452+0000","last_became_peered":"2024-09-21T10:23:20.700452+0000","last_unstale":"2024-09-21T10:26:47.182355+0000","last_undegraded":"2024-09-21T10:26:47.182355+0000","last_fullsized":"2024-09-21T10:26:47.182355+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.759355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.759355+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:21:15.413337+0000","scrub_duration":0.001833559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":628,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:26:48.329397+0000","last_change":"2024-09-21T10:24:44.307344+0000","last_active":"2024-09-21T10:26:48.329397+0000","last_peered":"2024-09-21T10:26:48.329397+0000","last_clean":"2024-09-21T10:26:48.329397+0000","last_became_active":"2024-09-21T10:24:44.298936+0000","last_became_peered":"2024-09-21T10:24:44.298936+0000","last_unstale":"2024-09-21T10:26:48.329397+0000","last_undegraded":"2024-09-21T10:26:48.329397+0000","last_fullsized":"2024-09-21T10:26:48.329397+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:58:26.025877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":580,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.183506+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:26:47.183506+0000","last_peered":"2024-09-21T10:26:47.183506+0000","last_clean":"2024-09-21T10:26:47.183506+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:26:47.183506+0000","last_undegraded":"2024-09-21T10:26:47.183506+0000","last_fullsized":"2024-09-21T10:26:47.183506+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":43,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:45.288362+0000","last_change":"2024-09-21T10:24:44.303781+0000","last_active":"2024-09-21T10:26:45.288362+0000","last_peered":"2024-09-21T10:26:45.288362+0000","last_clean":"2024-09-21T10:26:45.288362+0000","last_became_active":"2024-09-21T10:24:44.302058+0000","last_became_peered":"2024-09-21T10:24:44.302058+0000","last_unstale":"2024-09-21T10:26:45.288362+0000","last_undegraded":"2024-09-21T10:26:45.288362+0000","last_fullsized":"2024-09-21T10:26:45.288362+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:30.371229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":43,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:45.288672+0000","last_change":"2024-09-21T10:24:44.303417+0000","last_active":"2024-09-21T10:26:45.288672+0000","last_peered":"2024-09-21T10:26:45.288672+0000","last_clean":"2024-09-21T10:26:45.288672+0000","last_became_active":"2024-09-21T10:24:44.303069+0000","last_became_peered":"2024-09-21T10:24:44.303069+0000","last_unstale":"2024-09-21T10:26:45.288672+0000","last_undegraded":"2024-09-21T10:26:45.288672+0000","last_fullsized":"2024-09-21T10:26:45.288672+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:30:30.966228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":53,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:45.288689+0000","last_change":"2024-09-21T10:24:44.321421+0000","last_active":"2024-09-21T10:26:45.288689+0000","last_peered":"2024-09-21T10:26:45.288689+0000","last_clean":"2024-09-21T10:26:45.288689+0000","last_became_active":"2024-09-21T10:24:44.297082+0000","last_became_peered":"2024-09-21T10:24:44.297082+0000","last_unstale":"2024-09-21T10:26:45.288689+0000","last_undegraded":"2024-09-21T10:26:45.288689+0000","last_fullsized":"2024-09-21T10:26:45.288689+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:35:37.704881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":465,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:04.306609+0000","last_change":"2024-09-21T10:08:18.470378+0000","last_active":"2024-09-21T10:27:04.306609+0000","last_peered":"2024-09-21T10:27:04.306609+0000","last_clean":"2024-09-21T10:27:04.306609+0000","last_became_active":"2024-09-21T10:08:18.469482+0000","last_became_peered":"2024-09-21T10:08:18.469482+0000","last_unstale":"2024-09-21T10:27:04.306609+0000","last_undegraded":"2024-09-21T10:27:04.306609+0000","last_fullsized":"2024-09-21T10:27:04.306609+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:35.628724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":401,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.182532+0000","last_change":"2024-09-21T10:24:44.313834+0000","last_active":"2024-09-21T10:26:47.182532+0000","last_peered":"2024-09-21T10:26:47.182532+0000","last_clean":"2024-09-21T10:26:47.182532+0000","last_became_active":"2024-09-21T10:24:44.299135+0000","last_became_peered":"2024-09-21T10:24:44.299135+0000","last_unstale":"2024-09-21T10:26:47.182532+0000","last_undegraded":"2024-09-21T10:26:47.182532+0000","last_fullsized":"2024-09-21T10:26:47.182532+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:21:51.959148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":125,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:46.068320+0000","last_change":"2024-09-21T10:26:41.644952+0000","last_active":"2024-09-21T10:26:46.068320+0000","last_peered":"2024-09-21T10:26:46.068320+0000","last_clean":"2024-09-21T10:26:46.068320+0000","last_became_active":"2024-09-21T10:26:41.644592+0000","last_became_peered":"2024-09-21T10:26:41.644592+0000","last_unstale":"2024-09-21T10:26:46.068320+0000","last_undegraded":"2024-09-21T10:26:46.068320+0000","last_fullsized":"2024-09-21T10:26:46.068320+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:44:44.183877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":106,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.181882+0000","last_change":"2024-09-21T10:18:33.464989+0000","last_active":"2024-09-21T10:26:47.181882+0000","last_peered":"2024-09-21T10:26:47.181882+0000","last_clean":"2024-09-21T10:26:47.181882+0000","last_became_active":"2024-09-21T10:18:33.464617+0000","last_became_peered":"2024-09-21T10:18:33.464617+0000","last_unstale":"2024-09-21T10:26:47.181882+0000","last_undegraded":"2024-09-21T10:26:47.181882+0000","last_fullsized":"2024-09-21T10:26:47.181882+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:09.300797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":586,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:26:48.328911+0000","last_change":"2024-09-21T10:18:33.472415+0000","last_active":"2024-09-21T10:26:48.328911+0000","last_peered":"2024-09-21T10:26:48.328911+0000","last_clean":"2024-09-21T10:26:48.328911+0000","last_became_active":"2024-09-21T10:18:33.471746+0000","last_became_peered":"2024-09-21T10:18:33.471746+0000","last_unstale":"2024-09-21T10:26:48.328911+0000","last_undegraded":"2024-09-21T10:26:48.328911+0000","last_fullsized":"2024-09-21T10:26:48.328911+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:40:37.099458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":241,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:04.305620+0000","last_change":"2024-09-21T10:24:44.310195+0000","last_active":"2024-09-21T10:27:04.305620+0000","last_peered":"2024-09-21T10:27:04.305620+0000","last_clean":"2024-09-21T10:27:04.305620+0000","last_became_active":"2024-09-21T10:24:44.309600+0000","last_became_peered":"2024-09-21T10:24:44.309600+0000","last_unstale":"2024-09-21T10:27:04.305620+0000","last_undegraded":"2024-09-21T10:27:04.305620+0000","last_fullsized":"2024-09-21T10:27:04.305620+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:29:15.459007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":317,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.183331+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:26:47.183331+0000","last_peered":"2024-09-21T10:26:47.183331+0000","last_clean":"2024-09-21T10:26:47.183331+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:26:47.183331+0000","last_undegraded":"2024-09-21T10:26:47.183331+0000","last_fullsized":"2024-09-21T10:26:47.183331+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":251,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:26:48.328278+0000","last_change":"2024-09-21T10:18:33.479635+0000","last_active":"2024-09-21T10:26:48.328278+0000","last_peered":"2024-09-21T10:26:48.328278+0000","last_clean":"2024-09-21T10:26:48.328278+0000","last_became_active":"2024-09-21T10:18:33.478830+0000","last_became_peered":"2024-09-21T10:18:33.478830+0000","last_unstale":"2024-09-21T10:26:48.328278+0000","last_undegraded":"2024-09-21T10:26:48.328278+0000","last_fullsized":"2024-09-21T10:26:48.328278+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:16.950086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":602,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:26:48.329594+0000","last_change":"2024-09-21T10:18:33.480350+0000","last_active":"2024-09-21T10:26:48.329594+0000","last_peered":"2024-09-21T10:26:48.329594+0000","last_clean":"2024-09-21T10:26:48.329594+0000","last_became_active":"2024-09-21T10:18:33.479248+0000","last_became_peered":"2024-09-21T10:18:33.479248+0000","last_unstale":"2024-09-21T10:26:48.329594+0000","last_undegraded":"2024-09-21T10:26:48.329594+0000","last_fullsized":"2024-09-21T10:26:48.329594+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:25:28.780572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":604,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:04.302684+0000","last_change":"2024-09-21T10:24:44.309248+0000","last_active":"2024-09-21T10:27:04.302684+0000","last_peered":"2024-09-21T10:27:04.302684+0000","last_clean":"2024-09-21T10:27:04.302684+0000","last_became_active":"2024-09-21T10:24:44.308925+0000","last_became_peered":"2024-09-21T10:24:44.308925+0000","last_unstale":"2024-09-21T10:27:04.302684+0000","last_undegraded":"2024-09-21T10:27:04.302684+0000","last_fullsized":"2024-09-21T10:27:04.302684+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:40.267676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":365,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:04.304698+0000","last_change":"2024-09-21T10:24:44.308839+0000","last_active":"2024-09-21T10:27:04.304698+0000","last_peered":"2024-09-21T10:27:04.304698+0000","last_clean":"2024-09-21T10:27:04.304698+0000","last_became_active":"2024-09-21T10:24:44.308488+0000","last_became_peered":"2024-09-21T10:24:44.308488+0000","last_unstale":"2024-09-21T10:27:04.304698+0000","last_undegraded":"2024-09-21T10:27:04.304698+0000","last_fullsized":"2024-09-21T10:27:04.304698+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:22:11.961205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":43,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:45.288491+0000","last_change":"2024-09-21T10:24:44.305124+0000","last_active":"2024-09-21T10:26:45.288491+0000","last_peered":"2024-09-21T10:26:45.288491+0000","last_clean":"2024-09-21T10:26:45.288491+0000","last_became_active":"2024-09-21T10:24:44.304468+0000","last_became_peered":"2024-09-21T10:24:44.304468+0000","last_unstale":"2024-09-21T10:26:45.288491+0000","last_undegraded":"2024-09-21T10:26:45.288491+0000","last_fullsized":"2024-09-21T10:26:45.288491+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:37.904764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":243,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:04.303199+0000","last_change":"2024-09-21T10:18:33.480128+0000","last_active":"2024-09-21T10:27:04.303199+0000","last_peered":"2024-09-21T10:27:04.303199+0000","last_clean":"2024-09-21T10:27:04.303199+0000","last_became_active":"2024-09-21T10:18:33.479691+0000","last_became_peered":"2024-09-21T10:18:33.479691+0000","last_unstale":"2024-09-21T10:27:04.303199+0000","last_undegraded":"2024-09-21T10:27:04.303199+0000","last_fullsized":"2024-09-21T10:27:04.303199+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:43:49.311244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":599,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:26:48.328341+0000","last_change":"2024-09-21T10:24:44.299989+0000","last_active":"2024-09-21T10:26:48.328341+0000","last_peered":"2024-09-21T10:26:48.328341+0000","last_clean":"2024-09-21T10:26:48.328341+0000","last_became_active":"2024-09-21T10:24:44.299356+0000","last_became_peered":"2024-09-21T10:24:44.299356+0000","last_unstale":"2024-09-21T10:26:48.328341+0000","last_undegraded":"2024-09-21T10:26:48.328341+0000","last_fullsized":"2024-09-21T10:26:48.328341+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:58.867008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":43,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:45.288588+0000","last_change":"2024-09-21T10:24:44.304040+0000","last_active":"2024-09-21T10:26:45.288588+0000","last_peered":"2024-09-21T10:26:45.288588+0000","last_clean":"2024-09-21T10:26:45.288588+0000","last_became_active":"2024-09-21T10:24:44.303666+0000","last_became_peered":"2024-09-21T10:24:44.303666+0000","last_unstale":"2024-09-21T10:26:45.288588+0000","last_undegraded":"2024-09-21T10:26:45.288588+0000","last_fullsized":"2024-09-21T10:26:45.288588+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:19:35.061997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":47,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:45.288764+0000","last_change":"2024-09-21T10:24:44.329150+0000","last_active":"2024-09-21T10:26:45.288764+0000","last_peered":"2024-09-21T10:26:45.288764+0000","last_clean":"2024-09-21T10:26:45.288764+0000","last_became_active":"2024-09-21T10:24:44.306752+0000","last_became_peered":"2024-09-21T10:24:44.306752+0000","last_unstale":"2024-09-21T10:26:45.288764+0000","last_undegraded":"2024-09-21T10:26:45.288764+0000","last_fullsized":"2024-09-21T10:26:45.288764+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:16.592859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":43,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:45.288504+0000","last_change":"2024-09-21T10:24:44.298825+0000","last_active":"2024-09-21T10:26:45.288504+0000","last_peered":"2024-09-21T10:26:45.288504+0000","last_clean":"2024-09-21T10:26:45.288504+0000","last_became_active":"2024-09-21T10:24:44.298517+0000","last_became_peered":"2024-09-21T10:24:44.298517+0000","last_unstale":"2024-09-21T10:26:45.288504+0000","last_undegraded":"2024-09-21T10:26:45.288504+0000","last_fullsized":"2024-09-21T10:26:45.288504+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:10.288494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":47,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:45.288772+0000","last_change":"2024-09-21T10:24:44.324087+0000","last_active":"2024-09-21T10:26:45.288772+0000","last_peered":"2024-09-21T10:26:45.288772+0000","last_clean":"2024-09-21T10:26:45.288772+0000","last_became_active":"2024-09-21T10:24:44.298665+0000","last_became_peered":"2024-09-21T10:24:44.298665+0000","last_unstale":"2024-09-21T10:26:45.288772+0000","last_undegraded":"2024-09-21T10:26:45.288772+0000","last_fullsized":"2024-09-21T10:26:45.288772+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:59.819340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":256,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-21T10:27:04.308342+0000","last_change":"2024-09-21T10:18:33.474216+0000","last_active":"2024-09-21T10:27:04.308342+0000","last_peered":"2024-09-21T10:27:04.308342+0000","last_clean":"2024-09-21T10:27:04.308342+0000","last_became_active":"2024-09-21T10:18:33.473794+0000","last_became_peered":"2024-09-21T10:18:33.473794+0000","last_unstale":"2024-09-21T10:27:04.308342+0000","last_undegraded":"2024-09-21T10:27:04.308342+0000","last_fullsized":"2024-09-21T10:27:04.308342+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:38.956888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":102,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:47.183180+0000","last_change":"2024-09-21T10:18:33.483576+0000","last_active":"2024-09-21T10:26:47.183180+0000","last_peered":"2024-09-21T10:26:47.183180+0000","last_clean":"2024-09-21T10:26:47.183180+0000","last_became_active":"2024-09-21T10:18:33.483116+0000","last_became_peered":"2024-09-21T10:18:33.483116+0000","last_unstale":"2024-09-21T10:26:47.183180+0000","last_undegraded":"2024-09-21T10:26:47.183180+0000","last_fullsized":"2024-09-21T10:26:47.183180+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:48.964766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":43,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-21T10:26:45.288338+0000","last_change":"2024-09-21T10:24:44.298820+0000","last_active":"2024-09-21T10:26:45.288338+0000","last_peered":"2024-09-21T10:26:45.288338+0000","last_clean":"2024-09-21T10:26:45.288338+0000","last_became_active":"2024-09-21T10:24:44.298478+0000","last_became_peered":"2024-09-21T10:24:44.298478+0000","last_unstale":"2024-09-21T10:26:45.288338+0000","last_undegraded":"2024-09-21T10:26:45.288338+0000","last_fullsized":"2024-09-21T10:26:45.288338+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:59.567078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":585,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-21T10:26:48.328858+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:26:48.328858+0000","last_peered":"2024-09-21T10:26:48.328858+0000","last_clean":"2024-09-21T10:26:48.328858+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:26:48.328858+0000","last_undegraded":"2024-09-21T10:26:48.328858+0000","last_fullsized":"2024-09-21T10:26:48.328858+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":2750480384,"num_objects":41969,"num_object_clones":0,"num_object_copies":125907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41969,"num_write_kb":2686016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8304852992,"data_stored":8304852992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41969,"ondisk_log_size":41969,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":10162,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5983,"num_read_kb":13006,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1100,"ondisk_log_size":1100,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397947,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1706880,"kb_used_data":801336,"kb_used_omap":1,"kb_used_meta":905534,"kb_avail":92038272,"statfs":{"total":95995035648,"available":94247190528,"internally_reserved":0,"allocated":820568064,"data_stored":818589543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":927267160},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594175,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2037308,"kb_used_data":1131828,"kb_used_omap":1,"kb_used_meta":905470,"kb_avail":91707844,"statfs":{"total":95995035648,"available":93908832256,"internally_reserved":0,"allocated":1158991872,"data_stored":1157017864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":927201624},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790405,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1698416,"kb_used_data":790184,"kb_used_omap":1,"kb_used_meta":908222,"kb_avail":92046736,"statfs":{"total":95995035648,"available":94255857664,"internally_reserved":0,"allocated":809148416,"data_stored":807182207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":930019672},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986634,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1863100,"kb_used_data":1553268,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91882052,"statfs":{"total":95995035648,"available":94087221248,"internally_reserved":0,"allocated":1590546432,"data_stored":1588571929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":433791697196,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860723,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2085424,"kb_used_data":1213224,"kb_used_omap":8,"kb_used_meta":872183,"kb_avail":91659728,"statfs":{"total":95995035648,"available":93859561472,"internally_reserved":0,"allocated":1242341376,"data_stored":1240375447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":893116268},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024249,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1562032,"kb_used_data":781288,"kb_used_omap":7,"kb_used_meta":780728,"kb_avail":92183120,"statfs":{"total":95995035648,"available":94395514880,"internally_reserved":0,"allocated":800038912,"data_stored":798079674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":799466034},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187784,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2535984,"kb_used_data":1661096,"kb_used_omap":8,"kb_used_meta":874871,"kb_avail":91209168,"statfs":{"total":95995035648,"available":93398188032,"internally_reserved":0,"allocated":1700962304,"data_stored":1698995909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":895868913},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1723138048,"data_stored":1723138048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":810483712,"data_stored":810483712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1262092288,"data_stored":1262092288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1666318336,"data_stored":1666318336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":838729728,"data_stored":838729728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1171390464,"data_stored":1171390464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":832700416,"data_stored":832700416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:27:12.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_5 pgp_num 16 2024-09-21T10:27:12.186 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 25 16 57059 57043 142.564 95.125 0.0204629 0.00700917 2024-09-21T10:27:12.243 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:12.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:12.295 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:12.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:12.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:12.397 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:12.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:12.438 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:12.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:12.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:12.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:12.577 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:12.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:12.678 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:12.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:12.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:12.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:12.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:12.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:12.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:12.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:12.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:12.993 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:12.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:13.094 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:13.128 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:13.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:13.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:13.186 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 26 16 58575 58559 140.725 94.75 0.016402 0.00710097 2024-09-21T10:27:13.229 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:13.272 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:13.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:13.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:13.373 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:13.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:13.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:13.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:13.550 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:13.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:13.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:13.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:13 smithi098 ceph-mon[97580]: pgmap v1193: 99 pgs: 99 active+clean; 3.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 123 MiB/s wr, 1.98k op/s 2024-09-21T10:27:13.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:13 smithi098 ceph-mon[100914]: pgmap v1193: 99 pgs: 99 active+clean; 3.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 123 MiB/s wr, 1.98k op/s 2024-09-21T10:27:13.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:13 smithi196 ceph-mon[84784]: pgmap v1193: 99 pgs: 99 active+clean; 3.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 123 MiB/s wr, 1.98k op/s 2024-09-21T10:27:13.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:13.689 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:13.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:13.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:13.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:13.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:13.823 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:13.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:13.923 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:13.956 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:13.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:13.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:14.058 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:14.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:14.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:14.186 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 27 16 60191 60175 139.254 101 0.00640036 0.00717614 2024-09-21T10:27:14.202 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:14.241 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:14.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:14.343 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:14.380 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:14.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:14.482 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:14.514 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:14.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:14.616 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:14.660 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:14.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:14.761 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:14.797 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:14.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:14.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:14.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:15.033 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:15.070 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:15.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:15.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:15.172 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:15.186 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 28 16 61734 61718 137.725 96.4375 0.0179886 0.00725544 2024-09-21T10:27:15.209 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:15.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:15.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:15.311 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:15.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:15.344 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:15.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:15.445 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:15.475 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:15.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:15.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:15.576 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:15.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:15 smithi098 ceph-mon[97580]: pgmap v1194: 99 pgs: 99 active+clean; 3.4 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-21T10:27:15.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:15 smithi098 ceph-mon[100914]: pgmap v1194: 99 pgs: 99 active+clean; 3.4 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-21T10:27:15.606 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:15.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:15.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:15 smithi196 ceph-mon[84784]: pgmap v1194: 99 pgs: 99 active+clean; 3.4 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-21T10:27:15.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:15.742 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:15.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:15.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:15.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:15.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:15.883 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:15.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:15.984 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:16.020 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:16.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:16.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:16.146 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:16.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:16.186 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 29 16 63280 63264 136.308 96.625 0.0068437 0.00733142 2024-09-21T10:27:16.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:16.280 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:16.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:16.382 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:16.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:16.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:16.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:16.549 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:16.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:16.651 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:16.694 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:16.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:16.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:16.834 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:16.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:16.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.973 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:16.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:17.074 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:17.109 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:17.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:17.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:17.186 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 30 16 64654 64638 134.627 85.875 0.00877045 0.00742348 2024-09-21T10:27:17.211 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:17.247 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:17.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:17.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:17.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:17.385 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:17.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:17.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:17.487 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:17.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:17.524 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:17.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:17.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:17 smithi098 ceph-mon[97580]: pgmap v1195: 99 pgs: 99 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-21T10:27:17.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:17 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1711408614' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:17.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:17 smithi098 ceph-mon[100914]: pgmap v1195: 99 pgs: 99 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-21T10:27:17.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:17 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1711408614' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:17.626 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:17.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:17 smithi196 ceph-mon[84784]: pgmap v1195: 99 pgs: 99 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-21T10:27:17.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:17 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1711408614' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:17.664 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:17.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:17.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:17.765 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:17.803 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:17.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:17.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:17.873 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:27:17.873 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:27:17.873 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:27:17.873 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:27:17.873 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:27:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:27:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:27:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:27:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:27:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:27:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:27:17.874 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:27:17.874 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:27:17.874 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:27:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:27:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:27:17.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:17.940 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:17.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:17.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:18.042 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:18.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.082 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:18.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:18.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:18.186 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 31 16 66106 66090 133.211 90.75 0.00598552 0.00749428 2024-09-21T10:27:18.203 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:18.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:18.305 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:18.327 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:18.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:18.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:18.464 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:18.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:18.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:18.603 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:18.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:18.705 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:18.739 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:18.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:18.841 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:18.876 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:18.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:18.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:19.008 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:19.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:19.110 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:19.131 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:19.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:19.186 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 32 16 67365 67349 131.508 78.6875 0.00922509 0.00759996 2024-09-21T10:27:19.233 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:19.265 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:19.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:19.366 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:19.400 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:19.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:19.502 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:19.534 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:19.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:19.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:19 smithi098 ceph-mon[97580]: pgmap v1196: 99 pgs: 99 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-09-21T10:27:19.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:19 smithi098 ceph-mon[100914]: pgmap v1196: 99 pgs: 99 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-09-21T10:27:19.635 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:19.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:19 smithi196 ceph-mon[84784]: pgmap v1196: 99 pgs: 99 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-09-21T10:27:19.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.671 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:19.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:19.772 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:19.811 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:19.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:19.914 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:19.939 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:19.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:20.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:20.074 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:20.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:20.175 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:20.186 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 33 16 68842 68826 130.32 92.3125 0.00942389 0.00766766 2024-09-21T10:27:20.213 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:20.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:20.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:20.346 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:20.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:20.448 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:20.487 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:20.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:20.588 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:20.627 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:20.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:20.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:27:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:27:20.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:27:20.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:20.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:20.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:20.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:27:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:27:20] "GET /metrics HTTP/1.1" 200 36577 "" "Prometheus/2.51.0" 2024-09-21T10:27:20.851 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:20.872 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:20.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:20.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:21.004 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:21.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:21.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:21.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:21.138 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:21.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:21.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:21.186 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 34 16 69926 69910 128.48 67.75 0.0112211 0.00777834 2024-09-21T10:27:21.240 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:21.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:21.273 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:21.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:21.373 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:21.413 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:21.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:21.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:21.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:21.518 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:21.537 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:21.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:21.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:21.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:21 smithi098 ceph-mon[97580]: pgmap v1197: 99 pgs: 99 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-09-21T10:27:21.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:21 smithi098 ceph-mon[100914]: pgmap v1197: 99 pgs: 99 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-09-21T10:27:21.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:21 smithi196 ceph-mon[84784]: pgmap v1197: 99 pgs: 99 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-09-21T10:27:21.677 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:21.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:21.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:21.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:21.819 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:21.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:21.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:21.921 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:21.956 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:21.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:21.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:22.058 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:22.092 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:22.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:22.186 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 35 16 71242 71226 127.159 82.25 0.00963601 0.00786011 2024-09-21T10:27:22.195 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:22.226 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:22.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:22.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:22.361 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:22.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:22.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:22.496 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:22.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:22.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:22.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:22.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:22.624 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:22.726 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:22.759 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:22.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:22.860 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:22.895 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:22.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:22.997 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:23.030 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:23.132 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:23.167 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:23.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:23.187 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 36 16 72666 72650 126.099 89 0.0125345 0.00792568 2024-09-21T10:27:23.269 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:23.306 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:23.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:23.407 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:23.436 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:23.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:23.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:27:23.538 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:23.572 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:23.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:23.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:23 smithi098 ceph-mon[97580]: pgmap v1198: 99 pgs: 99 active+clean; 4.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-21T10:27:23.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:23 smithi098 ceph-mon[100914]: pgmap v1198: 99 pgs: 99 active+clean; 4.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-21T10:27:23.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:23 smithi196 ceph-mon[84784]: pgmap v1198: 99 pgs: 99 active+clean; 4.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-21T10:27:23.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:23.708 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:23.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:23.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:23.847 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:23.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:23.949 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:23.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.988 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:23.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:24.089 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:24.128 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:24.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:24.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:24.187 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 37 16 74071 74055 125.064 87.8125 0.0119325 0.00799097 2024-09-21T10:27:24.233 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:24.272 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:24.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:24.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:24.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:24.406 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:24.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:24.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:24.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:24.547 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:24.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:24.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:24.648 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:24.684 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:24.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:24.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:24.786 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:24.859 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:24.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:24.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:24.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:24.997 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:24.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:24.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:25.096 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:27:25.097 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:27:25.098 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:25.147 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:25.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:25.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:25.187 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 38 16 75455 75439 124.048 86.5 0.00873054 0.00805669 2024-09-21T10:27:25.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:25.280 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:25.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:25.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:25.382 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:25.414 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:25.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:25.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:25.516 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:25.550 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:25.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:25.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:25.596 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:25 smithi098 ceph-mon[97580]: pgmap v1199: 99 pgs: 99 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-21T10:27:25.596 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:25 smithi098 ceph-mon[100914]: pgmap v1199: 99 pgs: 99 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-21T10:27:25.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:25 smithi196 ceph-mon[84784]: pgmap v1199: 99 pgs: 99 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-21T10:27:25.652 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:25.683 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:25.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:25.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:25.785 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:25.823 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:25.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:25.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:25.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:25.963 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:25.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:25.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:26.065 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:26.100 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:26.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:26.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:26.187 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 39 16 76814 76798 123.045 84.9375 0.0117261 0.00812209 2024-09-21T10:27:26.203 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:26.235 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:26.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:26.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:26.337 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:26.378 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:26.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:26.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:26.480 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:26.504 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:26.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:26.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:26.605 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:26.638 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:26.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:26 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1864170859' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:27:26.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:26.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:26.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:26.776 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:26.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:26.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:26.780 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:26 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1864170859' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:27:26.781 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:26 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1864170859' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:27:26.817 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:27:26.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:26.913 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:26.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:26.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:27.014 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:27.054 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:27.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:27.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:27.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:27.188 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:27:27.187216+0000 min lat: 0.00114942 max lat: 0.424342 avg lat: 0.00819376 2024-09-21T10:27:27.188 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:27:27.188 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 40 16 78102 78086 121.979 80.5 0.0187532 0.00819376 2024-09-21T10:27:27.197 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:27.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:27.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:27.274 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:27:27.299 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:27.336 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:27.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:27.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:27.438 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:27.474 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:27.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:27.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:27.576 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:27.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:27 smithi098 ceph-mon[97580]: pgmap v1200: 99 pgs: 99 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-21T10:27:27.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:27 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:27:27.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:27 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1864170859' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:27:27.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:27 smithi098 ceph-mon[97580]: osdmap e356: 8 total, 8 up, 7 in 2024-09-21T10:27:27.608 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:27 smithi098 ceph-mon[100914]: pgmap v1200: 99 pgs: 99 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-21T10:27:27.608 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:27 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:27:27.609 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:27 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1864170859' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:27:27.609 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:27 smithi098 ceph-mon[100914]: osdmap e356: 8 total, 8 up, 7 in 2024-09-21T10:27:27.610 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:27.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:27.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:27.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:27 smithi196 ceph-mon[84784]: pgmap v1200: 99 pgs: 99 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-21T10:27:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:27 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:27:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:27 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1864170859' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:27:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:27 smithi196 ceph-mon[84784]: osdmap e356: 8 total, 8 up, 7 in 2024-09-21T10:27:27.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:27.741 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:27.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:27.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:27.843 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:27.880 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:27.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:27.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:27.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:28.007 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:28.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:28.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:28.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:28.114 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:28.166 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:28.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:28.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:28.174 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:28.189 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 41 16 79441 79425 121.044 83.6875 0.0112824 0.00825644 2024-09-21T10:27:28.268 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:28.307 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:28.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:28.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:28.374 INFO:teuthology.orchestra.run.smithi098.stderr:set pool 10 pgp_num to 16 2024-09-21T10:27:28.409 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:28.445 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:28.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:28.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:28.547 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:28.589 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:28.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:28.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:28.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:28 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2360886920' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:27:28.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:28 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:27:28.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:28 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2360886920' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:27:28.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:28 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:27:28.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:28 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2360886920' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:27:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:28 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:27:28.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:28.723 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:28.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:28.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:28.827 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:28.859 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:28.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:28.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:28.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:29.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:29.002 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:29.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:29.103 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:29.138 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:29.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:29.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:29.189 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 42 16 80765 80749 120.132 82.75 0.0127023 0.00831906 2024-09-21T10:27:29.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:29.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:29.281 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:29.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:29.382 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:29.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:29.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:29.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:29.469 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:29 smithi098 ceph-mon[97580]: pgmap v1202: 99 pgs: 99 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T10:27:29.469 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:29 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T10:27:29.469 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:29 smithi098 ceph-mon[97580]: osdmap e357: 8 total, 8 up, 7 in 2024-09-21T10:27:29.469 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:29 smithi098 ceph-mon[100914]: pgmap v1202: 99 pgs: 99 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T10:27:29.469 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:29 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T10:27:29.469 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:29 smithi098 ceph-mon[100914]: osdmap e357: 8 total, 8 up, 7 in 2024-09-21T10:27:29.522 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:29.555 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:29.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:29.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:29 smithi196 ceph-mon[84784]: pgmap v1202: 99 pgs: 99 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T10:27:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:29 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T10:27:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:29 smithi196 ceph-mon[84784]: osdmap e357: 8 total, 8 up, 7 in 2024-09-21T10:27:29.656 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:29.685 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:29.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:29.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:29.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:29.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:29.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:29.921 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:29.954 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:29.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:29.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:30.048 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:27:30.056 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:30.091 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:30.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:30.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:30.189 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 43 16 82050 82034 119.206 80.3125 0.00976771 0.00838384 2024-09-21T10:27:30.192 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:30.225 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:30.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:30.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:30.326 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:30.369 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:30.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:30.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:30.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:30.511 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:30.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:30.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:30.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:30.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:27:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:27:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:27:30.646 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:30.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:30.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:30.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:30.779 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:30.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:30.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:30.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:30.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:30.912 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:30.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:30.940 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:27:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:27:30] "GET /metrics HTTP/1.1" 200 36570 "" "Prometheus/2.51.0" 2024-09-21T10:27:31.013 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:31.058 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:31.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:31.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:31.160 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:31.189 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 44 16 83356 83340 118.35 81.625 0.0110351 0.00844455 2024-09-21T10:27:31.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:31.191 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:31.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:31.292 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:31.326 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:31.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:31.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:31.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:31.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:31.470 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:31.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:31.498 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:31.571 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:31.577 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:31 smithi098 ceph-mon[97580]: pgmap v1204: 99 pgs: 99 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:27:31.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:31 smithi098 ceph-mon[100914]: pgmap v1204: 99 pgs: 99 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:27:31.604 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:31.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:31.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:31.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:31 smithi196 ceph-mon[84784]: pgmap v1204: 99 pgs: 99 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:27:31.706 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:31.741 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:31.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:31.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:31.843 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:31.881 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:31.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:31.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:31.982 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:32.020 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:32.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:32.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:32.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:32.154 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:32.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:32.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:32.189 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 45 16 84627 84611 117.486 79.4375 0.0104744 0.00850741 2024-09-21T10:27:32.256 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:32.290 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:32.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:32.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:32.392 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:32.423 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:32.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:32.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:32.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:32.574 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:32.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:32.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:32.676 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:32.711 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:32.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:32.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:32.753 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:27:32.753 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:27:32.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:32.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:32.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:32.949 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:32.983 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:32.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:32.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:33.085 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:33.125 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:33.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:33.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:33.190 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 46 16 85882 85866 116.637 78.4375 0.0043806 0.008568 2024-09-21T10:27:33.227 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:33.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:33.261 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:33.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:33.362 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:33.397 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:33.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:33.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:33.499 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:33.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:33.530 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:33.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:33.631 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:33.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:33 smithi196 ceph-mon[84784]: pgmap v1205: 99 pgs: 99 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-21T10:27:33.661 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:33 smithi098 ceph-mon[97580]: pgmap v1205: 99 pgs: 99 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-21T10:27:33.661 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:33 smithi098 ceph-mon[100914]: pgmap v1205: 99 pgs: 99 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-21T10:27:33.666 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:33.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:33.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:33.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:33.801 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:33.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:33.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:33.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:33.938 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:33.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:33.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:34.040 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:34.077 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:34.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:34.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:34.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:34.190 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 47 16 87120 87104 115.801 77.375 0.0143466 0.00863066 2024-09-21T10:27:34.210 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:34.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:34.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:34.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:34.342 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:34.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:34.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:34.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:34.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:34.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:34.583 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:34.623 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:34.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:34.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:34.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:34.734 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:27:34.735 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:27:34.763 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:34.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:34.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:34.768 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:27:34.768 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:27:34.796 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:27:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:27:34.824 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:27:34.824 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:27:34.854 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:27:34.854 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:27:34.864 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:34.885 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:27:34.885 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:27:34.901 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:34.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:34.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:34.914 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:27:34.914 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:27:34.942 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:27:34.942 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:27:35.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:35.034 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:35.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:35.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:35.136 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:35.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:35.171 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:35.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:35.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:27:35.190 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 48 16 88344 88328 114.983 76.5 0.00927989 0.00869192 2024-09-21T10:27:35.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:35.307 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:35.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:35.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:35.379 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 4] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:27:35.379 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:27:35.379 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-21T10:27:35.379 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:27:35.380 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:27:35.380 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-21T10:27:35.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:35.421 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:35 smithi098 ceph-mon[97580]: pgmap v1206: 99 pgs: 99 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-09-21T10:27:35.421 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:35 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1370689986' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:35.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:35 smithi098 ceph-mon[100914]: pgmap v1206: 99 pgs: 99 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-09-21T10:27:35.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:35 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1370689986' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:35.470 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:35.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:35.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:35.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:35.614 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:35.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:35.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:35.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:35 smithi196 ceph-mon[84784]: pgmap v1206: 99 pgs: 99 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-09-21T10:27:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:35 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1370689986' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:35.716 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:35.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:35.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:35.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:35.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:35.896 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:35.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:35.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:35.998 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:36.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:36.029 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:36.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:36.130 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:36.164 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:36.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:36.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:36.190 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 49 16 89559 89543 114.186 75.9375 0.0157219 0.008753 2024-09-21T10:27:36.266 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:36.311 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:36.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:36.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:36.413 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:36.446 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:36.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:36.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:36.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:36.580 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:36.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:36.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:36.654 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:36.666 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:36.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:36.750 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:36.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:36.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:36.852 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:36.889 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:36.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:36.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:36.990 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:37.024 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:37.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:37.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:37.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:37.162 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:37.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:37.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:37.190 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 50 16 90724 90708 113.358 72.8125 0.0138514 0.00881663 2024-09-21T10:27:37.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:37.309 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:37.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:37.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:37.396 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:27:37.411 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:37.448 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:37.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:37.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:37.550 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:37.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:37.583 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:37.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:37 smithi196 ceph-mon[84784]: pgmap v1207: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-21T10:27:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:37 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1531826986' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:27:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:37.684 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:37.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:37.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:37.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:37.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:37.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:37 smithi098 ceph-mon[97580]: pgmap v1207: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-21T10:27:37.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:37 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1531826986' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:27:37.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:37.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:37 smithi098 ceph-mon[100914]: pgmap v1207: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-21T10:27:37.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:37 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1531826986' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:27:37.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:37.865 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:37.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:37.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:37.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:37.991 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:27:38.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:38.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:38.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:38.140 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:38.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:38.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:38.190 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 51 16 91913 91897 112.593 74.3125 0.0145974 0.00887631 2024-09-21T10:27:38.242 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:38.277 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:38.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:38.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:38.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:38.414 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:38.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:38.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:38.516 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:38.548 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:38.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:38.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:38 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:27:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:38 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:27:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:38 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1531826986' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:27:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:38 smithi196 ceph-mon[84784]: osdmap e358: 8 total, 8 up, 7 in 2024-09-21T10:27:38.642 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:38 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:27:38.642 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:38 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:27:38.642 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:38 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1531826986' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:27:38.642 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:38 smithi098 ceph-mon[97580]: osdmap e358: 8 total, 8 up, 7 in 2024-09-21T10:27:38.642 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:38 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:27:38.642 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:38 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:27:38.642 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:38 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1531826986' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:27:38.642 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:38 smithi098 ceph-mon[100914]: osdmap e358: 8 total, 8 up, 7 in 2024-09-21T10:27:38.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:38.685 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:38.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:38.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:38.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:38.818 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:38.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:38.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:38.920 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:38.950 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:38.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:38.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:39.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:39.082 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:39.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:39.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:39.183 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:39.190 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 52 16 92995 92979 111.728 67.625 0.0512097 0.008943 2024-09-21T10:27:39.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:39.224 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:39.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:39.325 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:39.363 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:39.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:39.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:39.465 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:39.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:39.502 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:39.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:39.603 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:39.638 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:39.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:39.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:39.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:39.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:39.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:39.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:39 smithi098 ceph-mon[97580]: pgmap v1209: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 60 MiB/s wr, 966 op/s 2024-09-21T10:27:39.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:39 smithi098 ceph-mon[100914]: pgmap v1209: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 60 MiB/s wr, 966 op/s 2024-09-21T10:27:39.875 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:39 smithi196 ceph-mon[84784]: pgmap v1209: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 60 MiB/s wr, 966 op/s 2024-09-21T10:27:39.907 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:39.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:39.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:40.009 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:40.049 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:40.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:40.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:40.151 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:40.187 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:40.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:40.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:40.190 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 53 16 94150 94134 110.982 72.1875 0.0112901 0.00900482 2024-09-21T10:27:40.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:40.316 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:40.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:40.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:40.418 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:40.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:40.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:40.550 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:40.578 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:40.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:40.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:40.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:27:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:27:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:27:40.680 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:40.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:40.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:40.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:40.822 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:40.860 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:40.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:40.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:40.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:40.996 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:40.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:40.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:41.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:27:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:27:40] "GET /metrics HTTP/1.1" 200 36581 "" "Prometheus/2.51.0" 2024-09-21T10:27:41.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:41.143 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:41.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:41.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:41.190 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 54 16 95320 95304 110.281 73.125 0.0106214 0.00906297 2024-09-21T10:27:41.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:41.282 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:41.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:41.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:41.383 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:41.424 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:41.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:41.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:41.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:41.563 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:41.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:41.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:41.665 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:41.700 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:41.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:41.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:41.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:41.837 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:41.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:41.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:41.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:41 smithi098 ceph-mon[97580]: pgmap v1210: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:27:41.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:41 smithi098 ceph-mon[100914]: pgmap v1210: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:27:41.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:41 smithi196 ceph-mon[84784]: pgmap v1210: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:27:41.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:41.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:41.976 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:41.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:42.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:42.108 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:42.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:42.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:42.190 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 55 16 96396 96380 109.498 67.25 0.0118146 0.0091274 2024-09-21T10:27:42.211 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:42.236 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:42.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:42.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:42.337 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:42.387 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:42.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:42.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:42.488 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:42.531 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:42.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:42.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:42.632 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:42.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:42.664 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:42.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:42.764 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:42.804 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:42.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:42.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:42.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:42 smithi098 ceph-mon[100914]: pgmap v1211: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-21T10:27:42.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:42 smithi098 ceph-mon[97580]: pgmap v1211: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-21T10:27:42.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:42 smithi196 ceph-mon[84784]: pgmap v1211: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-21T10:27:42.906 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:42.943 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:42.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:42.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:43.045 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:43.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:43.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:43.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:43.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:43.191 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 56 16 97520 97504 108.797 70.25 0.00971723 0.00918637 2024-09-21T10:27:43.203 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:43.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:43.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:43.304 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:43.347 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:43.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:43.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:43.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:43.479 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:43.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:43.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:43.581 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:43.621 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:43.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:43.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:43.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:43.758 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:43.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:43.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:43.860 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:43.897 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:43.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:43.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:43.999 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:44.037 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:44.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:44.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:44.139 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:44.182 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:44.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:44.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:44.191 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 57 16 98604 98588 108.077 67.75 0.00819031 0.00924758 2024-09-21T10:27:44.283 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:44.316 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:44.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:44.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:44.418 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:44.425 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:44.452 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:44.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:44.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:44.554 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:44.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:44.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:44.606 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:44 smithi098 ceph-mon[100914]: pgmap v1212: 99 pgs: 99 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-09-21T10:27:44.693 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:44.731 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:44.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:44.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:44.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:44.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:44.873 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:44.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:44.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:44 smithi196 ceph-mon[84784]: pgmap v1212: 99 pgs: 99 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-09-21T10:27:44.912 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:44 smithi098 ceph-mon[97580]: pgmap v1212: 99 pgs: 99 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-09-21T10:27:44.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:45.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:45.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:45.110 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:45.146 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:45.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:45.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:45.195 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 58 16 99707 99691 107.395 68.9375 0.0114411 0.00930623 2024-09-21T10:27:45.247 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:45.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:45.289 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:45.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:45.303 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:27:45.303 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:27:45.303 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-21T10:27:45.304 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-21T10:27:45.304 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:27:45.390 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:45.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:45.429 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:45.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:45.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:45.560 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:45.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:45.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:45.656 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:27:45.657 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "" 2024-09-21T10:27:45.657 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:27:45.662 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:45.718 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:45.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:45.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:45.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:45.841 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:27:45.872 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:45 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:27:45.872 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:45 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:27:45.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:45.876 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:45.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:45.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:45 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:27:45.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:46.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:46.013 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:46.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:46.114 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:46.153 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:46.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:46.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:46.195 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 59 16 100700 100684 106.627 62.0625 0.0299125 0.00937358 2024-09-21T10:27:46.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:46.294 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:46.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:46.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:46.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:46.435 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:46.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:46.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:46.445 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:27:46.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:46.567 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:46.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:46.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:46.669 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:46.704 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:46.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:46.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:46.806 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:46.841 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:46.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:46.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:46.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:46 smithi098 ceph-mon[97580]: pgmap v1213: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-21T10:27:46.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:46 smithi098 ceph-mon[100914]: pgmap v1213: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-21T10:27:46.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:46 smithi196 ceph-mon[84784]: pgmap v1213: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-21T10:27:46.943 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:46.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:46.971 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:46.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:47.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:47.099 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:47.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:47.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:47.195 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:27:47.193904+0000 min lat: 0.00114942 max lat: 0.424342 avg lat: 0.00943581 2024-09-21T10:27:47.195 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:27:47.195 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 60 16 101734 101718 105.927 64.625 0.0154271 0.00943581 2024-09-21T10:27:47.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:47.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:47.248 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:47.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:47.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:47.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:47.391 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:47.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:47.492 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:47.522 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:47.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:47.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:47.624 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:47.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:47.671 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:47.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:47.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:47.797 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:47.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:47.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:47.899 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:47.926 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:47.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:47.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:48.027 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:48.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:48.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:48.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:48.195 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 61 16 102649 102633 105.127 57.1875 0.0166346 0.00950787 2024-09-21T10:27:48.196 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:48.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:48.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:48.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:48.329 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:48.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:48.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:48.431 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:48.469 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:48.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:48.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:48.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:48.592 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:48.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:48.694 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:48.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:48.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:48.826 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:48.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:48 smithi098 ceph-mon[97580]: pgmap v1214: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T10:27:48.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:48 smithi098 ceph-mon[100914]: pgmap v1214: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T10:27:48.862 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:48.863 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:48.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:48.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:48.871 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:48.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:48 smithi196 ceph-mon[84784]: pgmap v1214: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T10:27:48.965 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:49.009 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:49.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:49.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:49.111 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:49.145 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:49.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:49.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:49.196 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 62 16 103652 103636 104.442 62.6875 0.0103328 0.00956992 2024-09-21T10:27:49.247 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:49.269 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:27:49.269 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:27:49.269 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:27:49.269 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:27:49.269 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:27:49.270 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:27:49.271 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:49.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:49.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:49.274 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:27:49.274 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:27:49.357 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:27:49.357 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:27:49.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:49.385 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:27:49.385 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:27:49.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:49.409 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:27:49.409 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:27:49.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:49.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:49.441 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:27:49.441 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:27:49.511 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:49.546 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:49.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:49.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:49.647 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:49.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:49.669 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:49.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:49.770 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:49.799 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:49.800 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:49 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2055243160' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:49.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:49.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:49.801 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:49 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2055243160' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:49.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:49 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2055243160' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:49.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:49.934 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:49.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:49.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:50.036 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:50.066 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:50.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:50.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:50.168 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:50.196 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 63 16 104513 104497 103.639 53.8125 0.013706 0.00964452 2024-09-21T10:27:50.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:50.198 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:50.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:50.298 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:50.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:50.333 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:50.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:50.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:50.468 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:50.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:50.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:50.476 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:27:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:27:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:27:50.571 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:50.610 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:50.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:50.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:50.713 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:50.745 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:50.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:50.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:50.847 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:50.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:27:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:27:50] "GET /metrics HTTP/1.1" 200 36581 "" "Prometheus/2.51.0" 2024-09-21T10:27:50.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:50 smithi098 ceph-mon[97580]: pgmap v1215: 99 pgs: 99 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-21T10:27:50.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:50 smithi098 ceph-mon[100914]: pgmap v1215: 99 pgs: 99 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-21T10:27:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:50 smithi196 ceph-mon[84784]: pgmap v1215: 99 pgs: 99 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-21T10:27:50.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:50.894 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:50.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:50.995 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:51.040 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:51.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:51.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:51.141 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:51.177 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:51.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:51.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:51.196 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 64 16 105553 105537 103.035 65 0.00894694 0.00970069 2024-09-21T10:27:51.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:51.305 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:51.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:51.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:51.407 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:51.437 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:51.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:51.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:51.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:51.581 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:51.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:51.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:51.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:51.714 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:51.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:51.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:51.816 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:51.849 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:51.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:51.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:51.937 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 4] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:27:51.937 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:27:51.938 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-09-21T10:27:51.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd in 3 2024-09-21T10:27:51.951 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:52.006 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:52.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:52.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:52.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:52.146 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:52.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:52.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:52.196 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 65 16 106567 106551 102.425 63.375 0.0119001 0.00975836 2024-09-21T10:27:52.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:52.299 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:52.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:52.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:52.316 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:27:52.316 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:27:52.400 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:52.447 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:52.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:52.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:52.549 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:52.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:27:52.597 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:52.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:52.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:52.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:52.749 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:52.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:52.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:52.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:52.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:52.897 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:52.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:52.998 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:53.049 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:53.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:53.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:53.152 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:53.179 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:53.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:53.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:53.196 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 66 16 107561 107545 101.814 62.125 0.0156395 0.00981697 2024-09-21T10:27:53.280 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:53.308 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:53.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:53.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:53.410 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:53.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:53.440 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:53.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:53.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:53.576 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:53.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:53.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:53.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:53 smithi098 ceph-mon[100914]: pgmap v1216: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 60 MiB/s wr, 958 op/s 2024-09-21T10:27:53.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:27:53.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:53.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:27:53.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:27:53.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:53 smithi098 ceph-mon[97580]: pgmap v1216: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 60 MiB/s wr, 958 op/s 2024-09-21T10:27:53.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:27:53.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:53.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:27:53.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:27:53.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:53 smithi196 ceph-mon[84784]: pgmap v1216: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 60 MiB/s wr, 958 op/s 2024-09-21T10:27:53.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:27:53.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:27:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:27:53.678 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:53.710 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:53.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:53.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:53.811 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:53.844 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:53.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:53.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:53.945 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:53.976 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:53.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:53.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:54.061 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:27:54.078 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:54.156 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:54.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:54.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:54.196 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 67 16 108633 108617 101.295 67 0.0232399 0.00986785 2024-09-21T10:27:54.258 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:54.295 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:54.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:54.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:54.397 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:54.450 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:54.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:54.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:54.552 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:54.590 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:54.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:54.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:54.692 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:54.728 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:54.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:54.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:54.831 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:54.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:54.864 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:54.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:54.965 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:55.009 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:55.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:55.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:55.111 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:55.142 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:55.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:55.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:55.196 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 68 16 109664 109648 100.753 64.4375 0.0101716 0.00991987 2024-09-21T10:27:55.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:55.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:55.280 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:55.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:55.381 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:55.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:55.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:55.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:55.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:55.555 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:55.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:55.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:55.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:55 smithi098 ceph-mon[97580]: pgmap v1217: 99 pgs: 99 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 73 MiB/s wr, 1.18k op/s 2024-09-21T10:27:55.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:55 smithi098 ceph-mon[100914]: pgmap v1217: 99 pgs: 99 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 73 MiB/s wr, 1.18k op/s 2024-09-21T10:27:55.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:55 smithi196 ceph-mon[84784]: pgmap v1217: 99 pgs: 99 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 73 MiB/s wr, 1.18k op/s 2024-09-21T10:27:55.657 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:55.689 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:55.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:55.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:55.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:55.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:55.827 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:55.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:55.929 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:55.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:55.970 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:55.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:56.071 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:56.104 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:56.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:56.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:56.196 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 69 16 110746 110730 100.273 67.625 0.0130385 0.00996762 2024-09-21T10:27:56.206 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:56.241 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:56.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:56.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:56.288 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:27:56.343 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:56.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:56.382 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:56.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:56.483 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:56.521 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:56.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:56.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:56.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:56 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1601164367' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:27:56.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:56 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:27:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:56 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1601164367' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:27:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:56 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:27:56.623 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:56.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:56 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1601164367' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:27:56.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:56 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:27:56.655 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:56.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:56.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:56.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:56.790 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:56.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:56.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:56.892 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:56.930 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:56.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:56.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:57.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:57.060 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:57.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:57.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:57.162 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:57.198 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 70 16 111809 111793 99.7893 66.4375 0.0102112 0.0100163 2024-09-21T10:27:57.208 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:57.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:57.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:57.310 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:57.347 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:57.349 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:57.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:57.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:57.443 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:57 smithi098 ceph-mon[97580]: pgmap v1218: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 62 MiB/s wr, 985 op/s 2024-09-21T10:27:57.443 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:57 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:27:57.443 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:57 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:27:57.443 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:57 smithi098 ceph-mon[97580]: osdmap e359: 8 total, 8 up, 7 in 2024-09-21T10:27:57.444 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:57 smithi098 ceph-mon[100914]: pgmap v1218: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 62 MiB/s wr, 985 op/s 2024-09-21T10:27:57.444 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:57 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:27:57.444 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:57 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:27:57.444 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:57 smithi098 ceph-mon[100914]: osdmap e359: 8 total, 8 up, 7 in 2024-09-21T10:27:57.451 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:57.486 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:57.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:57.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:57.588 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:57.619 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:57.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:57.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:57.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:57 smithi196 ceph-mon[84784]: pgmap v1218: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 62 MiB/s wr, 985 op/s 2024-09-21T10:27:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:57 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:27:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:57 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:27:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:57 smithi196 ceph-mon[84784]: osdmap e359: 8 total, 8 up, 7 in 2024-09-21T10:27:57.721 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:57.760 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:57.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:57.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:57.861 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:57.892 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:57.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:57.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:57.994 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:58.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:58.035 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:58.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:58.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:58.171 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:58.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:58.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:58.196 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 71 16 112872 112856 99.3195 66.4375 0.0151749 0.0100637 2024-09-21T10:27:58.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:58.314 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:58.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:58.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:58.416 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:58.446 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:58.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:58.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:58.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:58.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:58.589 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:58.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:58.690 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:58.727 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:58.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:58.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:58.830 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:58.864 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:58.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:58.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:58.966 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:58.970 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:58.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:58.999 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:58.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:59.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:59.137 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:59.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:59.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:59.196 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 72 16 113932 113916 98.8602 66.25 0.00877872 0.010111 2024-09-21T10:27:59.211 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:58 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a[97576]: 2024-09-21T10:27:58.937+0000 7f65561f9640 -1 mon.a@0(leader).osd e359 definitely_dead 0 2024-09-21T10:27:59.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:59.274 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:59.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:59.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:59.276 INFO:teuthology.orchestra.run.smithi098.stderr:marked in osd.3. 2024-09-21T10:27:59.375 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:59.396 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:59.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:59.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:59.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:59.519 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:59.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:59.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:59.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:59 smithi098 ceph-mon[97580]: pgmap v1220: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T10:27:59.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:59 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2774924810' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-21T10:27:59.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:27:59 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-21T10:27:59.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:59 smithi098 ceph-mon[100914]: pgmap v1220: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T10:27:59.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:59 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2774924810' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-21T10:27:59.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:27:59 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-21T10:27:59.621 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:59.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:59 smithi196 ceph-mon[84784]: pgmap v1220: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T10:27:59.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:59 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2774924810' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-21T10:27:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:27:59 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-21T10:27:59.642 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:59.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:59.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:59.744 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:59.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:59.766 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:27:59.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:59.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:27:59.868 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:59.889 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:27:59.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:59.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:59.991 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:00.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:00.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:00.113 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:00.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:00.140 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:00.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:00.197 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 73 16 114041 114025 97.5994 6.8125 0.00351643 0.0101165 2024-09-21T10:28:00.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:00.263 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:00.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:00.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:00.338 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-09-21T10:28:00.364 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:00.385 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:00.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:00.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:00.487 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:00.512 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:00.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:00.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:00.578 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:00 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-21T10:28:00.578 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:00 smithi098 ceph-mon[97580]: osdmap e360: 8 total, 8 up, 8 in 2024-09-21T10:28:00.578 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:00 smithi098 ceph-mon[97580]: osdmap e361: 8 total, 8 up, 8 in 2024-09-21T10:28:00.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:00 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-21T10:28:00.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:00 smithi098 ceph-mon[100914]: osdmap e360: 8 total, 8 up, 8 in 2024-09-21T10:28:00.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:00 smithi098 ceph-mon[100914]: osdmap e361: 8 total, 8 up, 8 in 2024-09-21T10:28:00.614 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:00.638 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:00.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:00.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:00 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-21T10:28:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:00 smithi196 ceph-mon[84784]: osdmap e360: 8 total, 8 up, 8 in 2024-09-21T10:28:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:00 smithi196 ceph-mon[84784]: osdmap e361: 8 total, 8 up, 8 in 2024-09-21T10:28:00.640 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:28:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:28:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:28:00.740 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:00.765 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:00.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:00.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:00.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:28:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:28:00] "GET /metrics HTTP/1.1" 200 36581 "" "Prometheus/2.51.0" 2024-09-21T10:28:00.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:00.887 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:00.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:00.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:00.989 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:01.014 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:28:01.014 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:28:01.015 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:01.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:01.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:01.022 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:28:01.022 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:28:01.038 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:28:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:28:01.054 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:28:01.054 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:28:01.070 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:28:01.070 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:28:01.085 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:28:01.086 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:28:01.102 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:28:01.102 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:28:01.116 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:01.117 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:28:01.117 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:28:01.142 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:01.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:01.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:01.197 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 74 16 114041 114025 96.2807 0 - 0.0101165 2024-09-21T10:28:01.243 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:01.268 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:01.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:01.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:01.370 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:01.392 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:01.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:01.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:01.493 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:01.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:01.527 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:01.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:01 smithi098 ceph-mon[97580]: pgmap v1222: 99 pgs: 99 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-21T10:28:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:01 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1199562092' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:01 smithi098 ceph-mon[97580]: osdmap e362: 8 total, 8 up, 8 in 2024-09-21T10:28:01.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:01 smithi098 ceph-mon[100914]: pgmap v1222: 99 pgs: 99 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-21T10:28:01.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:01 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1199562092' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:01.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:01 smithi098 ceph-mon[100914]: osdmap e362: 8 total, 8 up, 8 in 2024-09-21T10:28:01.628 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:01.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:01 smithi196 ceph-mon[84784]: pgmap v1222: 99 pgs: 99 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-21T10:28:01.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:01 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1199562092' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:01 smithi196 ceph-mon[84784]: osdmap e362: 8 total, 8 up, 8 in 2024-09-21T10:28:01.651 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:01.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:01.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:01.752 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:01.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:01.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:01.892 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:01.919 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:01.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:01.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:02.021 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:02.052 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:02.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:02.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:02.154 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:02.180 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:02.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:02.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:02.197 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 75 16 114491 114475 95.372 14.0625 0.064551 0.0104755 2024-09-21T10:28:02.281 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:02.319 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:02.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:02.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:02.421 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:02.451 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:02.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:02.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:02.553 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:02.586 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:02.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:02.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:02.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:02 smithi098 ceph-mon[97580]: 10.f starting backfill to osd.3 from (0'0,0'0] MAX to 359'7061 2024-09-21T10:28:02.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:02 smithi098 ceph-mon[97580]: 10.7 starting backfill to osd.3 from (0'0,0'0] MAX to 359'7053 2024-09-21T10:28:02.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:02 smithi098 ceph-mon[100914]: 10.f starting backfill to osd.3 from (0'0,0'0] MAX to 359'7061 2024-09-21T10:28:02.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:02 smithi098 ceph-mon[100914]: 10.7 starting backfill to osd.3 from (0'0,0'0] MAX to 359'7053 2024-09-21T10:28:02.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:02 smithi196 ceph-mon[84784]: 10.f starting backfill to osd.3 from (0'0,0'0] MAX to 359'7061 2024-09-21T10:28:02.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:02 smithi196 ceph-mon[84784]: 10.7 starting backfill to osd.3 from (0'0,0'0] MAX to 359'7053 2024-09-21T10:28:02.689 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:02.710 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:02.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:02.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:02.812 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:02.841 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:02.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:02.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:02.943 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:02.977 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:02.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:02.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:03.078 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:03.120 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:03.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:03.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:03.197 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 76 15 115153 115138 94.6624 41.4375 0.00516467 0.0105576 2024-09-21T10:28:03.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:03.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:03.264 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:03.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:03.364 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:03.390 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:03.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:03.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:03.492 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:03.524 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:03.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:03.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:03.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:03 smithi098 ceph-mon[97580]: pgmap v1225: 99 pgs: 6 peering, 93 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 62 MiB/s wr, 986 op/s 2024-09-21T10:28:03.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:03 smithi098 ceph-mon[97580]: osdmap e363: 8 total, 8 up, 8 in 2024-09-21T10:28:03.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:03 smithi098 ceph-mon[100914]: pgmap v1225: 99 pgs: 6 peering, 93 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 62 MiB/s wr, 986 op/s 2024-09-21T10:28:03.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:03 smithi098 ceph-mon[100914]: osdmap e363: 8 total, 8 up, 8 in 2024-09-21T10:28:03.626 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:03.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:03 smithi196 ceph-mon[84784]: pgmap v1225: 99 pgs: 6 peering, 93 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 62 MiB/s wr, 986 op/s 2024-09-21T10:28:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:03 smithi196 ceph-mon[84784]: osdmap e363: 8 total, 8 up, 8 in 2024-09-21T10:28:03.655 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:03.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:03.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:03.757 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:03.797 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:03.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:03.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:03.899 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:03.925 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:03.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:03.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:04.027 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:04.056 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:04.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:04.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:04.158 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:04.197 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 77 16 115905 115889 94.0426 46.9375 0.0176422 0.0106271 2024-09-21T10:28:04.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:04.200 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:04.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:04.301 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:04.328 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:04.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:04.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:04.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:04.465 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:04.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:04.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:04.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:04.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:04 smithi098 ceph-mon[97580]: osdmap e364: 8 total, 8 up, 8 in 2024-09-21T10:28:04.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:04 smithi098 ceph-mon[100914]: osdmap e364: 8 total, 8 up, 8 in 2024-09-21T10:28:04.604 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:04.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:04.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:04.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:04 smithi196 ceph-mon[84784]: osdmap e364: 8 total, 8 up, 8 in 2024-09-21T10:28:04.706 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:04.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:04.737 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:04.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:04.838 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:04.880 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:04.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:04.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:04.982 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:05.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:05.011 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:05.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:05.112 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:05.145 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:05.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:05.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:05.197 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 78 16 116674 116658 93.4532 48.0625 0.0364541 0.0106929 2024-09-21T10:28:05.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:05.276 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:05.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:05.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:05.339 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 4, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:28:05.339 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T10:28:05.378 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:05.385 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:05 smithi098 ceph-mon[97580]: pgmap v1228: 99 pgs: 2 active+remapped+backfill_wait, 6 peering, 91 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 17 KiB/s rd, 35 MiB/s wr, 565 op/s; 14204/343761 objects misplaced (4.132%); 813 KiB/s, 11 objects/s recovering 2024-09-21T10:28:05.385 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:05 smithi098 ceph-mon[100914]: pgmap v1228: 99 pgs: 2 active+remapped+backfill_wait, 6 peering, 91 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 17 KiB/s rd, 35 MiB/s wr, 565 op/s; 14204/343761 objects misplaced (4.132%); 813 KiB/s, 11 objects/s recovering 2024-09-21T10:28:05.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:05.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:05.539 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:05.550 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:05.599 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:05.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:05.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:05.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:05 smithi196 ceph-mon[84784]: pgmap v1228: 99 pgs: 2 active+remapped+backfill_wait, 6 peering, 91 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 17 KiB/s rd, 35 MiB/s wr, 565 op/s; 14204/343761 objects misplaced (4.132%); 813 KiB/s, 11 objects/s recovering 2024-09-21T10:28:05.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:05.755 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:05.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:05.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:05.857 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:05.883 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:05.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:05.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:05.985 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:06.013 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:06.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:06.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:06.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:06.142 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:06.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:06.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:06.197 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 79 16 117337 117321 92.7948 41.4375 0.0294077 0.0107674 2024-09-21T10:28:06.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:06.269 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:28:06.269 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:28:06.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.269 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:117352 10.5 10.def09e05 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-09-21T10:28:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:06.227511+0000", 2024-09-21T10:28:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.036269750000000003, 2024-09-21T10:28:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036297603999999997, 2024-09-21T10:28:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-21T10:28:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:28:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:28:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:28:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 117352 2024-09-21T10:28:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.227511+0000", 2024-09-21T10:28:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.227511+0000", 2024-09-21T10:28:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.227512+0000", 2024-09-21T10:28:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.102e-06 2024-09-21T10:28:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.227534+0000", 2024-09-21T10:28:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1942000000000001e-05 2024-09-21T10:28:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.227546+0000", 2024-09-21T10:28:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1897e-05 2024-09-21T10:28:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.227552+0000", 2024-09-21T10:28:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6359999999999997e-06 2024-09-21T10:28:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:117360 10.3 10:c490660c:::benchmark_data_smithi098_281896_object117359:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-09-21T10:28:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:06.237399+0000", 2024-09-21T10:28:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.026381509000000001, 2024-09-21T10:28:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.026478781, 2024-09-21T10:28:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:28:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:28:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:28:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 117360 2024-09-21T10:28:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.237399+0000", 2024-09-21T10:28:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.237399+0000", 2024-09-21T10:28:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.237401+0000", 2024-09-21T10:28:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.719e-06 2024-09-21T10:28:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.237424+0000", 2024-09-21T10:28:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2915e-05 2024-09-21T10:28:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.237437+0000", 2024-09-21T10:28:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3426e-05 2024-09-21T10:28:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.237444+0000", 2024-09-21T10:28:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6660000000000002e-06 2024-09-21T10:28:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.237508+0000", 2024-09-21T10:28:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4580999999999997e-05 2024-09-21T10:28:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.237743+0000", 2024-09-21T10:28:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023468600000000001 2024-09-21T10:28:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:28:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.237917+0000", 2024-09-21T10:28:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000173998 2024-09-21T10:28:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.242670+0000", 2024-09-21T10:28:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0047524300000000002 2024-09-21T10:28:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:28:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.246284+0000", 2024-09-21T10:28:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036146609999999999 2024-09-21T10:28:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.254197+0000", 2024-09-21T10:28:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0079128359999999995 2024-09-21T10:28:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:28:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.255204+0000", 2024-09-21T10:28:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010069790000000001 2024-09-21T10:28:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:117370 10.5 10.e72637e5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-09-21T10:28:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:06.247989+0000", 2024-09-21T10:28:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.015791273000000002, 2024-09-21T10:28:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01596148, 2024-09-21T10:28:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-21T10:28:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:28:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:28:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:28:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 117370 2024-09-21T10:28:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.247989+0000", 2024-09-21T10:28:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.247989+0000", 2024-09-21T10:28:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.247991+0000", 2024-09-21T10:28:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6360000000000001e-06 2024-09-21T10:28:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.248013+0000", 2024-09-21T10:28:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1588000000000001e-05 2024-09-21T10:28:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.248027+0000", 2024-09-21T10:28:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4851000000000001e-05 2024-09-21T10:28:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.248036+0000", 2024-09-21T10:28:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0989999999999999e-06 2024-09-21T10:28:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:117371 10.3 10:c15236b0:::benchmark_data_smithi098_281896_object117370:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-09-21T10:28:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:06.249005+0000", 2024-09-21T10:28:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.014775922, 2024-09-21T10:28:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01499088, 2024-09-21T10:28:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:28:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:28:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:28:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:28:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 117371 2024-09-21T10:28:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.249005+0000", 2024-09-21T10:28:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.249005+0000", 2024-09-21T10:28:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.249006+0000", 2024-09-21T10:28:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.95e-07 2024-09-21T10:28:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.249026+0000", 2024-09-21T10:28:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0598000000000002e-05 2024-09-21T10:28:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.249039+0000", 2024-09-21T10:28:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2713e-05 2024-09-21T10:28:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.249046+0000", 2024-09-21T10:28:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7309999999999996e-06 2024-09-21T10:28:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.249239+0000", 2024-09-21T10:28:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019363599999999999 2024-09-21T10:28:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.249444+0000", 2024-09-21T10:28:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020462599999999999 2024-09-21T10:28:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:28:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.249641+0000", 2024-09-21T10:28:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000197228 2024-09-21T10:28:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.46322.0:117360 10.3 e364/361 ondisk, result = 0)", 2024-09-21T10:28:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:06.253272+0000", 2024-09-21T10:28:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.010508889, 2024-09-21T10:28:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010775814999999999, 2024-09-21T10:28:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:28:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.253272+0000", 2024-09-21T10:28:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.253272+0000", 2024-09-21T10:28:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.253273+0000", 2024-09-21T10:28:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1289999999999999e-06 2024-09-21T10:28:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.253278+0000", 2024-09-21T10:28:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6420000000000004e-06 2024-09-21T10:28:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.253294+0000", 2024-09-21T10:28:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5413999999999998e-05 2024-09-21T10:28:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.253302+0000", 2024-09-21T10:28:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5709999999999998e-06 2024-09-21T10:28:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.254183+0000", 2024-09-21T10:28:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088059200000000003 2024-09-21T10:28:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.254191+0000", 2024-09-21T10:28:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8830000000000006e-06 2024-09-21T10:28:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.46322.0:117371 10.3 e364/361)", 2024-09-21T10:28:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:06.253500+0000", 2024-09-21T10:28:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.010280526, 2024-09-21T10:28:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010585169, 2024-09-21T10:28:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-21T10:28:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.253500+0000", 2024-09-21T10:28:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.253500+0000", 2024-09-21T10:28:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.253501+0000", 2024-09-21T10:28:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9999999999999997e-07 2024-09-21T10:28:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.253505+0000", 2024-09-21T10:28:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0210000000000001e-06 2024-09-21T10:28:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.253518+0000", 2024-09-21T10:28:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2848e-05 2024-09-21T10:28:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.253524+0000", 2024-09-21T10:28:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2670000000000003e-06 2024-09-21T10:28:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:117377 10.3 10.685b05f3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-09-21T10:28:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:06.256866+0000", 2024-09-21T10:28:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0069145279999999996, 2024-09-21T10:28:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0072592849999999999, 2024-09-21T10:28:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-21T10:28:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:28:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:28:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:28:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 117377 2024-09-21T10:28:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.256866+0000", 2024-09-21T10:28:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.256866+0000", 2024-09-21T10:28:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.256869+0000", 2024-09-21T10:28:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8959999999999999e-06 2024-09-21T10:28:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.256905+0000", 2024-09-21T10:28:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5960000000000001e-05 2024-09-21T10:28:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.256929+0000", 2024-09-21T10:28:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3904999999999999e-05 2024-09-21T10:28:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.256941+0000", 2024-09-21T10:28:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1787000000000001e-05 2024-09-21T10:28:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:117351 10.5 10:a757f23d:::benchmark_data_smithi098_281896_object117350:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-09-21T10:28:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:06.226993+0000", 2024-09-21T10:28:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.036787394000000001, 2024-09-21T10:28:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037179070000000002, 2024-09-21T10:28:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:28:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:28:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:28:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:28:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 117351 2024-09-21T10:28:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.226993+0000", 2024-09-21T10:28:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.226993+0000", 2024-09-21T10:28:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.226995+0000", 2024-09-21T10:28:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.894e-06 2024-09-21T10:28:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.227019+0000", 2024-09-21T10:28:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4389e-05 2024-09-21T10:28:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.227034+0000", 2024-09-21T10:28:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4941e-05 2024-09-21T10:28:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.227043+0000", 2024-09-21T10:28:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2810000000000008e-06 2024-09-21T10:28:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.258254+0000", 2024-09-21T10:28:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.031211383999999998 2024-09-21T10:28:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.259645+0000", 2024-09-21T10:28:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001391006 2024-09-21T10:28:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:28:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:06.259864+0000", 2024-09-21T10:28:06.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00021935600000000001 2024-09-21T10:28:06.301 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:06.301 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:06.301 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:06.301 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:06.301 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-21T10:28:06.301 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 8 2024-09-21T10:28:06.301 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:28:06.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:06.302 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:06.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:06.403 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:06.438 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:06.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:06.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:06.540 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:06.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:06.583 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:06.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:06.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:06 smithi196 ceph-mon[84784]: Health check failed: Degraded data redundancy: 14163/345870 objects degraded (4.095%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:06.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:06.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:06.715 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:06.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:06.801 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:06 smithi098 ceph-mon[100914]: Health check failed: Degraded data redundancy: 14163/345870 objects degraded (4.095%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:06.802 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:06 smithi098 ceph-mon[97580]: Health check failed: Degraded data redundancy: 14163/345870 objects degraded (4.095%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:06.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:06.842 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:06.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:06.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:06.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:06.969 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:06.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:06.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:07.070 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:07.104 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:07.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:07.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:07.198 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:2024-09-21T10:28:07.197117+0000 min lat: 0.00114942 max lat: 2.42112 avg lat: 0.0108368 2024-09-21T10:28:07.198 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:28:07.198 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 80 16 118096 118080 92.2272 47.4375 0.0652115 0.0108368 2024-09-21T10:28:07.207 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:07.255 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:07.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:07.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:07.356 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:07.397 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:07.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:07.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:07.499 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:07.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:07.532 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:07.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:07.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:28:07.633 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:07 smithi196 ceph-mon[84784]: pgmap v1229: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 13 KiB/s rd, 28 MiB/s wr, 450 op/s; 14163/345870 objects degraded (4.095%); 21207/345870 objects misplaced (6.131%); 965 KiB/s, 16 objects/s recovering 2024-09-21T10:28:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:28:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:07.650 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:07 smithi098 ceph-mon[97580]: pgmap v1229: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 13 KiB/s rd, 28 MiB/s wr, 450 op/s; 14163/345870 objects degraded (4.095%); 21207/345870 objects misplaced (6.131%); 965 KiB/s, 16 objects/s recovering 2024-09-21T10:28:07.651 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:28:07.651 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:07.652 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:07 smithi098 ceph-mon[100914]: pgmap v1229: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 13 KiB/s rd, 28 MiB/s wr, 450 op/s; 14163/345870 objects degraded (4.095%); 21207/345870 objects misplaced (6.131%); 965 KiB/s, 16 objects/s recovering 2024-09-21T10:28:07.652 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:28:07.652 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:07.660 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:07.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:07.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:07.762 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:07.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:07.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:07.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:07.956 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:07.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:07.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:08.058 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:08.104 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:08.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:08.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:08.198 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 81 16 118843 118827 91.665 46.6875 0.0102287 0.0109029 2024-09-21T10:28:08.205 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:08.229 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:08.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:08.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:08.331 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:08.370 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:08.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:08.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:08.501 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:08.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:08.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:08.603 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:08.633 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:08.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:08.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:08.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:08.778 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:08.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:08.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:08.882 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:08.916 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:08.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:08.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:09.018 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:09.041 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:09.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:09.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:09.143 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:09.167 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:09.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:09.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:09.199 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 82 16 119580 119564 91.1079 46.0625 0.0813141 0.0109704 2024-09-21T10:28:09.268 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:09.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:09.323 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:09.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:09.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:28:09.419 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:09 smithi098 ceph-mon[97580]: pgmap v1230: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 11 KiB/s rd, 25 MiB/s wr, 412 op/s; 14163/346521 objects degraded (4.087%); 21207/346521 objects misplaced (6.120%); 812 KiB/s, 13 objects/s recovering 2024-09-21T10:28:09.420 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:09 smithi098 ceph-mon[100914]: pgmap v1230: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 11 KiB/s rd, 25 MiB/s wr, 412 op/s; 14163/346521 objects degraded (4.087%); 21207/346521 objects misplaced (6.120%); 812 KiB/s, 13 objects/s recovering 2024-09-21T10:28:09.424 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:09.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:09.488 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:09.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:09.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:09.640 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:09.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:09.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:09.741 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:09.755 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:09 smithi196 ceph-mon[84784]: pgmap v1230: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 11 KiB/s rd, 25 MiB/s wr, 412 op/s; 14163/346521 objects degraded (4.087%); 21207/346521 objects misplaced (6.120%); 812 KiB/s, 13 objects/s recovering 2024-09-21T10:28:09.779 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:09.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:09.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:09.880 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:09.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:09.941 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:09.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:10.036 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:10.042 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:10.070 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:10.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:10.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:10.171 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:10.200 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 83 16 120154 120138 90.4417 35.875 0.0168207 0.0110497 2024-09-21T10:28:10.242 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:10.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:10.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:10.344 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:10.390 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:10.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:10.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:10.402 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:10.404 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:10.492 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:10.538 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:10.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:10.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:10.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:28:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:28:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:28:10.642 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:10.676 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:10.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:10.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:10.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:10.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:10.813 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:10.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:10.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:28:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:28:10] "GET /metrics HTTP/1.1" 200 36621 "" "Prometheus/2.51.0" 2024-09-21T10:28:10.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:10.950 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:10.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:10.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:11.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:11.091 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:11.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:11.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:11.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:11.202 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 84 16 120815 120799 89.8569 41.3125 0.0807001 0.0111221 2024-09-21T10:28:11.234 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:11.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:11.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:11.336 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:11.367 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:11.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:11.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:11.416 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:28:11.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:11.499 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:11.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:11.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:11.600 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:11.632 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:11.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:11.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:11.734 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:11.756 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:11 smithi098 ceph-mon[97580]: pgmap v1231: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.2 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 8.6 KiB/s rd, 44 MiB/s wr, 701 op/s; 14163/355563 objects degraded (3.983%); 21569/355563 objects misplaced (6.066%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:28:11.756 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:11 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2989363957' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:28:11.756 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:11 smithi098 ceph-mon[100914]: pgmap v1231: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.2 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 8.6 KiB/s rd, 44 MiB/s wr, 701 op/s; 14163/355563 objects degraded (3.983%); 21569/355563 objects misplaced (6.066%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:28:11.756 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:11 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2989363957' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:28:11.760 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:11.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:11.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:11.862 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:11.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:11 smithi196 ceph-mon[84784]: pgmap v1231: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.2 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 8.6 KiB/s rd, 44 MiB/s wr, 701 op/s; 14163/355563 objects degraded (3.983%); 21569/355563 objects misplaced (6.066%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:28:11.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:11 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2989363957' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:28:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:11.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:11.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:12.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:12.045 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:12.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:12.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:12.147 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:12.198 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:12.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:12.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:12.200 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 85 16 121534 121518 89.3284 44.9375 0.00662038 0.011185 2024-09-21T10:28:12.300 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:12.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:12.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:12.441 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:12.469 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:12.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:12.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:12.572 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:12.604 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:12.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:12.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:12.706 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:12.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:12 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:28:12.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:12 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2989363957' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:28:12.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:12 smithi098 ceph-mon[97580]: osdmap e365: 8 total, 8 up, 8 in 2024-09-21T10:28:12.708 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:12 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:28:12.708 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:12 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2989363957' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:28:12.708 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:12 smithi098 ceph-mon[100914]: osdmap e365: 8 total, 8 up, 8 in 2024-09-21T10:28:12.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:12.734 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:12.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:12.835 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:12.866 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:12.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:12.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:12 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:28:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:12 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2989363957' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:28:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:12 smithi196 ceph-mon[84784]: osdmap e365: 8 total, 8 up, 8 in 2024-09-21T10:28:12.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:13.015 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:13.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:13.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:13.117 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:13.143 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:13.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:13.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:13.200 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 86 16 122227 122211 88.7934 43.3125 0.00782995 0.0112559 2024-09-21T10:28:13.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:13.281 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:13.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:13.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:13.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:13.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:13.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:13.524 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:13.553 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:13.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:13.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:13.649 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:13 smithi098 ceph-mon[97580]: pgmap v1233: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.3 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 8.4 KiB/s rd, 33 MiB/s wr, 536 op/s; 14163/356925 objects degraded (3.968%); 21569/356925 objects misplaced (6.043%); 650 KiB/s, 10 objects/s recovering 2024-09-21T10:28:13.650 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:13 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 14163/356925 objects degraded (3.968%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:13.650 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:13 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:28:13.650 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:13 smithi098 ceph-mon[100914]: pgmap v1233: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.3 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 8.4 KiB/s rd, 33 MiB/s wr, 536 op/s; 14163/356925 objects degraded (3.968%); 21569/356925 objects misplaced (6.043%); 650 KiB/s, 10 objects/s recovering 2024-09-21T10:28:13.650 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:13 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 14163/356925 objects degraded (3.968%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:13.650 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:13 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:28:13.655 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:13.683 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:13.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:13.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:13.785 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:13.809 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:13.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:13.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:13.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:13 smithi196 ceph-mon[84784]: pgmap v1233: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.3 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 8.4 KiB/s rd, 33 MiB/s wr, 536 op/s; 14163/356925 objects degraded (3.968%); 21569/356925 objects misplaced (6.043%); 650 KiB/s, 10 objects/s recovering 2024-09-21T10:28:13.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:13 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 14163/356925 objects degraded (3.968%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:13.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:13 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:28:13.911 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:13.936 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:13.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:13.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:14.037 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:14.070 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:14.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:14.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:14.172 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:14.201 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 87 16 122939 122923 88.2835 44.5 0.0089777 0.0113199 2024-09-21T10:28:14.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:14.215 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:14.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:14.316 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:14.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:14.351 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:14.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:14.452 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:14.480 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:14.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:14.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:14.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:14.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:14 smithi098 ceph-mon[97580]: pgmap v1234: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 42 MiB/s wr, 672 op/s; 14163/363945 objects degraded (3.892%); 22020/363945 objects misplaced (6.050%); 499 KiB/s, 8 objects/s recovering 2024-09-21T10:28:14.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:14 smithi098 ceph-mon[100914]: pgmap v1234: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 42 MiB/s wr, 672 op/s; 14163/363945 objects degraded (3.892%); 22020/363945 objects misplaced (6.050%); 499 KiB/s, 8 objects/s recovering 2024-09-21T10:28:14.618 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:14.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:14.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:14.719 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:14.746 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:14.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:14.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:14.848 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:14.877 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:14.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:14.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:14 smithi196 ceph-mon[84784]: pgmap v1234: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 42 MiB/s wr, 672 op/s; 14163/363945 objects degraded (3.892%); 22020/363945 objects misplaced (6.050%); 499 KiB/s, 8 objects/s recovering 2024-09-21T10:28:14.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:15.021 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:15.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:15.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:15.123 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:15.164 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:15.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:15.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:15.201 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 88 16 123679 123663 87.8059 46.25 0.013432 0.0113835 2024-09-21T10:28:15.267 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:15.298 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:15.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:15.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:15.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:15.447 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:15.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:15.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:15.549 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:15.579 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:15.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:15.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:15.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:15.734 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:15.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:15.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:15.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:15.873 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:15.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:15.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:15.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:16.022 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:16.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:16.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:16.132 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:16.160 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:16.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:16.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:16.201 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 89 16 124375 124359 87.3081 43.5 0.0172672 0.0114474 2024-09-21T10:28:16.262 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:16.287 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:16.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:16.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:16.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:16.429 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:16.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:16.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:16.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:16.565 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:16.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:16.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:16.631 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:28:16.631 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:28:16.667 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:16.670 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:28:16.671 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:28:16.697 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:16.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:16.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:16.702 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:28:16.702 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:28:16.734 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:28:16.735 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:28:16.763 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:28:16.763 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:28:16.793 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:28:16.793 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:28:16.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:16.826 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:16.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:16.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:16.828 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:28:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:28:16.854 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:28:16.854 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:28:16.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:16.969 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:16.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:16.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:17.071 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:17.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:17.119 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:17.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:17.201 INFO:tasks.radosbench.radosbench.0.smithi098.stdout: 90 2 125022 125020 86.7971 41.3125 0.0504468 0.011515 2024-09-21T10:28:17.221 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:17.241 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:17.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:17.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:17.343 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:17.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:17 smithi098 ceph-mon[97580]: pgmap v1235: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 44 MiB/s wr, 707 op/s; 14163/367107 objects degraded (3.858%); 21962/367107 objects misplaced (5.982%); 685 KiB/s, 10 objects/s recovering 2024-09-21T10:28:17.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:17 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:28:17.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:17 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:28:17.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:17 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1059954659' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:17.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:17 smithi098 ceph-mon[100914]: pgmap v1235: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 44 MiB/s wr, 707 op/s; 14163/367107 objects degraded (3.858%); 21962/367107 objects misplaced (5.982%); 685 KiB/s, 10 objects/s recovering 2024-09-21T10:28:17.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:17 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:28:17.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:17 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:28:17.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:17 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1059954659' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:17.365 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:17.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:17.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:17.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:17 smithi196 ceph-mon[84784]: pgmap v1235: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 44 MiB/s wr, 707 op/s; 14163/367107 objects degraded (3.858%); 21962/367107 objects misplaced (5.982%); 685 KiB/s, 10 objects/s recovering 2024-09-21T10:28:17.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:17 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:28:17.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:17 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:28:17.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:17 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1059954659' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:17.466 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:17.488 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:17.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:17.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:17.589 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:17.611 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:17.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:17.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:17.712 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:17.734 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:17.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:17.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:17.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:17.858 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:17.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:17.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:17.960 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:17.982 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:17.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:17.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:18.038 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T10:28:18.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:18.105 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:18.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:18.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:18.201 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Total time run: 90.0291 2024-09-21T10:28:18.202 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Total writes made: 125022 2024-09-21T10:28:18.202 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Write size: 65536 2024-09-21T10:28:18.202 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Object size: 65536 2024-09-21T10:28:18.202 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Bandwidth (MB/sec): 86.7928 2024-09-21T10:28:18.202 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Stddev Bandwidth: 44.3459 2024-09-21T10:28:18.202 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Max bandwidth (MB/sec): 225.812 2024-09-21T10:28:18.202 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Min bandwidth (MB/sec): 0 2024-09-21T10:28:18.202 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Average IOPS: 1388 2024-09-21T10:28:18.202 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Stddev IOPS: 709.534 2024-09-21T10:28:18.202 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Max IOPS: 3613 2024-09-21T10:28:18.202 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Min IOPS: 0 2024-09-21T10:28:18.203 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Average Latency(s): 0.0115156 2024-09-21T10:28:18.203 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Stddev Latency(s): 0.0281279 2024-09-21T10:28:18.203 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Max latency(s): 2.42112 2024-09-21T10:28:18.203 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Min latency(s): 0.00114942 2024-09-21T10:28:18.206 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:28:18.207 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:18.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:18.240 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:18.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:18.341 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:18.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:18 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1216515612' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:28:18.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:18 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1216515612' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:28:18.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:18 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1216515612' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:28:18.392 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:18.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:18.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:18.494 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:18.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:18.526 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:18.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:18.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:18.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:18.675 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:18.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:18.776 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:18.802 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:18.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:18.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:18.904 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:18.933 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:18.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:18.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:19.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:19.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:19.071 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:19.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:19.172 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:19.207 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:19.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:19.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:19.308 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:19.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:19 smithi098 ceph-mon[97580]: pgmap v1236: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.5 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 44 MiB/s wr, 706 op/s; 14163/367716 objects degraded (3.852%); 21962/367716 objects misplaced (5.973%); 685 KiB/s, 10 objects/s recovering 2024-09-21T10:28:19.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:19 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 14163/367716 objects degraded (3.852%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:19.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:19 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1216515612' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:28:19.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:19 smithi098 ceph-mon[97580]: osdmap e366: 8 total, 8 up, 8 in 2024-09-21T10:28:19.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:19 smithi098 ceph-mon[100914]: pgmap v1236: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.5 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 44 MiB/s wr, 706 op/s; 14163/367716 objects degraded (3.852%); 21962/367716 objects misplaced (5.973%); 685 KiB/s, 10 objects/s recovering 2024-09-21T10:28:19.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:19 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 14163/367716 objects degraded (3.852%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:19.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:19 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1216515612' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:28:19.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:19 smithi098 ceph-mon[100914]: osdmap e366: 8 total, 8 up, 8 in 2024-09-21T10:28:19.349 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:19.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:19.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:19.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:19 smithi196 ceph-mon[84784]: pgmap v1236: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.5 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 44 MiB/s wr, 706 op/s; 14163/367716 objects degraded (3.852%); 21962/367716 objects misplaced (5.973%); 685 KiB/s, 10 objects/s recovering 2024-09-21T10:28:19.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:19 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 14163/367716 objects degraded (3.852%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:19.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:19 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1216515612' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:28:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:19 smithi196 ceph-mon[84784]: osdmap e366: 8 total, 8 up, 8 in 2024-09-21T10:28:19.451 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:19.482 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:19.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:19.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:19.583 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:19.616 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:19.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:19.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:19.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:19.757 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:19.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:19.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:19.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:19.885 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:19.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:19.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:19.987 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:20.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:20.030 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:20.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:20.131 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:20.152 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:20.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:20.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:20.253 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:20.280 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:20.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:20.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:20.382 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:20.408 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:20.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:20.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:20.509 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:20.545 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:20.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:20.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:20.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:28:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:28:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:28:20.647 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:20.672 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:20.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:20.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:20.774 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:20.824 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:20.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:20.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:20.926 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:20.950 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:20.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:20.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:20.967 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:28:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:28:20] "GET /metrics HTTP/1.1" 200 36621 "" "Prometheus/2.51.0" 2024-09-21T10:28:21.054 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:21.084 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:21.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:21.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:21.185 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:21.225 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:21.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:21.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:21.312 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:21 smithi098 ceph-mon[97580]: pgmap v1238: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 45 MiB/s wr, 726 op/s; 14163/374385 objects degraded (3.783%); 22190/374385 objects misplaced (5.927%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:28:21.312 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:21 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:28:21.312 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:21 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:28:21.312 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:21 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:28:21.313 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:21 smithi098 ceph-mon[100914]: pgmap v1238: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 45 MiB/s wr, 726 op/s; 14163/374385 objects degraded (3.783%); 22190/374385 objects misplaced (5.927%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:28:21.313 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:21 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:28:21.313 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:21 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:28:21.313 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:21 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:28:21.327 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:21.358 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:21.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:21.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:21.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:21 smithi196 ceph-mon[84784]: pgmap v1238: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 45 MiB/s wr, 726 op/s; 14163/374385 objects degraded (3.783%); 22190/374385 objects misplaced (5.927%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:28:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:21 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:28:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:21 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:28:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:21 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:28:21.460 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:21.488 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:21.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:21.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:21.590 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:21.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:21.616 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:21.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:21.717 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:21.751 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:21.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:21.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:21.853 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:21.878 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:21.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:21.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:21.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:22.012 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:22.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:22.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:22.113 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:22.138 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:22.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:22.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:22.240 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:22.287 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:22.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:22.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:22.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:22.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:22.428 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:22.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:22.469 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T10:28:22.529 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:22.597 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:22.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:22.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:22.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:22.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:22.749 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:22.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:22.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:22.901 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:22.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:22.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:23.002 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:23.028 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:23.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:23.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:23.112 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:28:23.112 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:28:23.130 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:23.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:23.155 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:23.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:23.190 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:28:23.256 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:23.302 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:23.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:23.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:23.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:23 smithi098 ceph-mon[97580]: pgmap v1239: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 38 MiB/s wr, 628 op/s; 14163/374523 objects degraded (3.782%); 22190/374523 objects misplaced (5.925%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:28:23.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:23 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:23.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:23 smithi098 ceph-mon[100914]: pgmap v1239: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 38 MiB/s wr, 628 op/s; 14163/374523 objects degraded (3.782%); 22190/374523 objects misplaced (5.925%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:28:23.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:23 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:23.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:23 smithi196 ceph-mon[84784]: pgmap v1239: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 38 MiB/s wr, 628 op/s; 14163/374523 objects degraded (3.782%); 22190/374523 objects misplaced (5.925%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:28:23.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:23 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:23.405 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:23.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:23.448 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:23.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:23.544 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:23.549 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:23.572 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:23.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:23.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:23.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:23.677 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:28:23.712 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:23.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:23.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:23.813 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:23.837 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:23.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:23.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:23.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:23.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:23.981 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:23.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:24.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:28:24.082 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:24.147 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:24.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:24.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:24.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:24 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 14163/367227 objects degraded (3.857%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:24.242 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:24 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 14163/367227 objects degraded (3.857%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:24.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:24.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:24.327 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:24.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:24.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:24 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 14163/367227 objects degraded (3.857%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:24.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:24.453 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:24.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:24.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:24.555 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:24.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:24.613 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:24.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:24.714 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:24.747 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:24.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:24.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:24.849 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:24.886 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:24.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:24.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:24.987 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:25.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:25.020 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:25.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:25.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:25.161 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:25.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:25.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:25.263 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:25.289 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:25.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:25.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:25.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:25 smithi196 ceph-mon[84784]: pgmap v1240: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.5 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 23 MiB/s wr, 637 op/s; 14163/367227 objects degraded (3.857%); 21752/367227 objects misplaced (5.923%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:28:25.391 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:25.422 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:25.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:25.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:25.428 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:25 smithi098 ceph-mon[97580]: pgmap v1240: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.5 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 23 MiB/s wr, 637 op/s; 14163/367227 objects degraded (3.857%); 21752/367227 objects misplaced (5.923%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:28:25.428 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:25 smithi098 ceph-mon[100914]: pgmap v1240: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.5 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 23 MiB/s wr, 637 op/s; 14163/367227 objects degraded (3.857%); 21752/367227 objects misplaced (5.923%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:28:25.525 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:25.554 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:25.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:25.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:25.656 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:25.693 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:25.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:25.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:25.794 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:25.820 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:25.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:25.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:25.922 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:25.944 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:25.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:25.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:26.046 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:26.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:26.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:26.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:26.173 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:26.206 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:26.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:26.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:26.239 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:26.308 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:26.347 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:26.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:26.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:26.449 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:26.474 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:26.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:26.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:26.575 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:26.601 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:26.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:26.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:26.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:26.743 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:26.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:26.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:26.845 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:26.871 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:26.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:26.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:26.972 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:27.005 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:27.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:27.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:27.107 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:27.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:27.151 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:27.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:27.252 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:27.290 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:27.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:27.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:27.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:27 smithi098 ceph-mon[97580]: pgmap v1241: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 17 MiB/s wr, 641 op/s; 14163/363960 objects degraded (3.891%); 21691/363960 objects misplaced (5.960%); 1.1 MiB/s, 16 objects/s recovering 2024-09-21T10:28:27.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:27 smithi098 ceph-mon[100914]: pgmap v1241: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 17 MiB/s wr, 641 op/s; 14163/363960 objects degraded (3.891%); 21691/363960 objects misplaced (5.960%); 1.1 MiB/s, 16 objects/s recovering 2024-09-21T10:28:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:27 smithi196 ceph-mon[84784]: pgmap v1241: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 17 MiB/s wr, 641 op/s; 14163/363960 objects degraded (3.891%); 21691/363960 objects misplaced (5.960%); 1.1 MiB/s, 16 objects/s recovering 2024-09-21T10:28:27.392 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:27.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:27.417 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:27.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:27.518 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:27.546 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:27.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:27.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:27.648 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:27.673 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:27.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:27.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:27.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:27.815 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:27.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:27.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:27.916 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:27.943 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:27.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:27.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:28.044 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:28.086 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:28.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:28.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:28.188 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:28.217 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:28.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:28.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:28.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:28.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:28.354 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:28.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:28.455 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:28.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:28.502 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:28.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:28.576 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:28.576 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:28.602 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:28.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:28.631 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:28.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:28.732 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:28.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:28.758 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:28.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:28.859 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:28.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:28.898 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:28.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:28.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:29.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:29.043 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:29.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T10:28:29.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:29.188 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:29 smithi098 ceph-mon[97580]: pgmap v1242: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 15 MiB/s wr, 643 op/s; 14163/363264 objects degraded (3.899%); 21691/363264 objects misplaced (5.971%); 1.1 MiB/s, 16 objects/s recovering 2024-09-21T10:28:29.188 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:29 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4287789531' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:28:29.189 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:29 smithi098 ceph-mon[100914]: pgmap v1242: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 15 MiB/s wr, 643 op/s; 14163/363264 objects degraded (3.899%); 21691/363264 objects misplaced (5.971%); 1.1 MiB/s, 16 objects/s recovering 2024-09-21T10:28:29.189 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:29 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4287789531' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:28:29.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:29.190 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:29.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:29.290 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:29.318 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:29.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:29.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:29.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:29 smithi196 ceph-mon[84784]: pgmap v1242: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 15 MiB/s wr, 643 op/s; 14163/363264 objects degraded (3.899%); 21691/363264 objects misplaced (5.971%); 1.1 MiB/s, 16 objects/s recovering 2024-09-21T10:28:29.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:29 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4287789531' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:28:29.420 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:29.445 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:29.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:29.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:29.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:29.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:29.580 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:29.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:29.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:29.719 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:29.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:29.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:29.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:29.855 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:29.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:29.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:29.957 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:30.006 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:30.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:30.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:30.108 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:30.134 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:30.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:30.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:30.259 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T10:28:30.265 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:30.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:30.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:30.367 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:30.399 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:30.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:30.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:30.430 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:30 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4287789531' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:28:30.430 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:30 smithi098 ceph-mon[97580]: osdmap e367: 8 total, 8 up, 8 in 2024-09-21T10:28:30.430 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:30 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4287789531' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:28:30.430 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:30 smithi098 ceph-mon[100914]: osdmap e367: 8 total, 8 up, 8 in 2024-09-21T10:28:30.500 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:30.522 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:30.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:30.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:30.624 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:30.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:30 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4287789531' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:28:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:30 smithi196 ceph-mon[84784]: osdmap e367: 8 total, 8 up, 8 in 2024-09-21T10:28:30.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:28:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:28:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:28:30.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:30.655 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:30.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:30.756 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:30.791 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:30.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:30.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:30.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:28:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:28:30] "GET /metrics HTTP/1.1" 200 36608 "" "Prometheus/2.51.0" 2024-09-21T10:28:30.893 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:30.919 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:30.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:30.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:31.021 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:31.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:31.054 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:31.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:31.155 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:31.185 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:31.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:31.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:31.286 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:31.317 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:31.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:31.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:31.419 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:31.450 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:31.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:31.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:31.551 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:31.583 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:31.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:31.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:31.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:31 smithi098 ceph-mon[97580]: pgmap v1244: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 1.3 MiB/s wr, 687 op/s; 14273/354981 objects degraded (4.021%); 21117/354981 objects misplaced (5.949%); 761 KiB/s, 11 objects/s recovering 2024-09-21T10:28:31.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:31 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 14273/354981 objects degraded (4.021%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:31.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:31 smithi098 ceph-mon[100914]: pgmap v1244: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 1.3 MiB/s wr, 687 op/s; 14273/354981 objects degraded (4.021%); 21117/354981 objects misplaced (5.949%); 761 KiB/s, 11 objects/s recovering 2024-09-21T10:28:31.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:31 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 14273/354981 objects degraded (4.021%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:31.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:31 smithi196 ceph-mon[84784]: pgmap v1244: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 1.3 MiB/s wr, 687 op/s; 14273/354981 objects degraded (4.021%); 21117/354981 objects misplaced (5.949%); 761 KiB/s, 11 objects/s recovering 2024-09-21T10:28:31.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:31 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 14273/354981 objects degraded (4.021%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:31.685 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:31.716 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:31.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:31.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:31.818 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:31.845 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:31.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:31.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:31.947 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:31.980 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:31.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:31.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:32.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:32.114 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:32.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:32.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:32.216 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:32.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:32.243 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:32.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:32.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:32.372 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:32.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:32.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:32.475 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:32.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:32.516 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:32.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:32.617 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:32.647 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:32.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:32.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:32.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:32.781 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:32.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:32.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:32.882 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:32.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:32.920 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:32.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:33.021 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:33.073 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:33.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:33.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:33.076 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:28:33.076 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:28:33.107 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:28:33.107 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:28:33.139 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:28:33.140 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:28:33.175 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:33.192 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:28:33.192 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:28:33.206 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:33.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:33.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:33.223 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:28:33.223 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:28:33.256 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:28:33.256 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:28:33.286 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:28:33.286 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:28:33.308 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:33.313 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:28:33.313 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:28:33.340 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:33.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:33.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:33.442 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:33.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:33.480 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:33.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:33.570 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:33 smithi098 ceph-mon[97580]: pgmap v1245: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 102 B/s wr, 699 op/s; 14273/353529 objects degraded (4.037%); 21117/353529 objects misplaced (5.973%); 761 KiB/s, 11 objects/s recovering 2024-09-21T10:28:33.570 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:33 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1546395088' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:33.570 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:33 smithi098 ceph-mon[100914]: pgmap v1245: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 102 B/s wr, 699 op/s; 14273/353529 objects degraded (4.037%); 21117/353529 objects misplaced (5.973%); 761 KiB/s, 11 objects/s recovering 2024-09-21T10:28:33.571 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:33 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1546395088' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:33.581 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:33.606 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:33.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:33.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:33.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:33 smithi196 ceph-mon[84784]: pgmap v1245: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 102 B/s wr, 699 op/s; 14273/353529 objects degraded (4.037%); 21117/353529 objects misplaced (5.973%); 761 KiB/s, 11 objects/s recovering 2024-09-21T10:28:33.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:33 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1546395088' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:33.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:33.746 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:33.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:33.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:33.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:33.879 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:33.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:33.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:33.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:34.016 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:34.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:34.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:34.118 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:34.149 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:34.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:34.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:34.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:34.277 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:34.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:34.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:34.378 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:34.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:34.420 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:34.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:34.521 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:34.556 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:34.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:34.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:34.658 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:34.684 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:34.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:34.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:34.786 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:34.816 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:34.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:34.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:34.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:34.953 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:34.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:34.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:35.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:35.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:35.109 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:35.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:35.210 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:35.234 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:35.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:35.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:35.317 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:35 smithi098 ceph-mon[97580]: pgmap v1246: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 14273/345900 objects degraded (4.126%); 20697/345900 objects misplaced (5.984%); 761 KiB/s, 11 objects/s recovering 2024-09-21T10:28:35.317 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:35 smithi098 ceph-mon[100914]: pgmap v1246: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 14273/345900 objects degraded (4.126%); 20697/345900 objects misplaced (5.984%); 761 KiB/s, 11 objects/s recovering 2024-09-21T10:28:35.335 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:35.370 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:35.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:35.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:35.471 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:35.505 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:35.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:35.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:35.607 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:35.636 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:35.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:35.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:35.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:35 smithi196 ceph-mon[84784]: pgmap v1246: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 14273/345900 objects degraded (4.126%); 20697/345900 objects misplaced (5.984%); 761 KiB/s, 11 objects/s recovering 2024-09-21T10:28:35.738 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:35.779 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:35.785 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:35.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:35.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:35.886 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:35.913 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:35.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:35.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:36.014 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:36.039 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:36.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:36.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:36.141 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:36.165 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:36.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:36.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:36.266 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:36.276 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:36 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 14273/345900 objects degraded (4.126%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:36.277 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:36 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 14273/345900 objects degraded (4.126%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:36.291 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:36.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:36.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:36.393 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:36.420 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:36.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:36.522 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:36.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:36.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:36.577 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:28:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-21T10:28:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-21T10:28:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:28:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:167429 8.f 8:fda3abdd:::benchmark_data_smithi098_226587_object63759:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-21T10:28:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:20:45.090144+0000", 2024-09-21T10:28:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 471.48135311300001, 2024-09-21T10:28:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.00801798, 2024-09-21T10:28:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:28:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:28:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:28:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 167429 2024-09-21T10:28:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:45.090144+0000", 2024-09-21T10:28:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:45.090144+0000", 2024-09-21T10:28:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:45.090145+0000", 2024-09-21T10:28:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2900000000000003e-07 2024-09-21T10:28:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:45.090148+0000", 2024-09-21T10:28:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.129e-06 2024-09-21T10:28:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:45.090159+0000", 2024-09-21T10:28:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0995000000000001e-05 2024-09-21T10:28:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:45.090164+0000", 2024-09-21T10:28:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2900000000000002e-06 2024-09-21T10:28:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:45.090206+0000", 2024-09-21T10:28:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2077999999999999e-05 2024-09-21T10:28:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:28:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:45.090213+0000", 2024-09-21T10:28:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9759999999999998e-06 2024-09-21T10:28:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:46.081306+0000", 2024-09-21T10:28:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.99109267199999995 2024-09-21T10:28:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:46.081415+0000", 2024-09-21T10:28:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108978 2024-09-21T10:28:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:28:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:46.081542+0000", 2024-09-21T10:28:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012722699999999999 2024-09-21T10:28:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:46.089016+0000", 2024-09-21T10:28:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0074734839999999999 2024-09-21T10:28:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:28:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:46.089027+0000", 2024-09-21T10:28:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1068e-05 2024-09-21T10:28:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:46.098052+0000", 2024-09-21T10:28:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0090255349999999995 2024-09-21T10:28:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:28:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:46.098117+0000", 2024-09-21T10:28:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4979000000000002e-05 2024-09-21T10:28:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:28:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:46.098162+0000", 2024-09-21T10:28:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4839999999999998e-05 2024-09-21T10:28:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.45794.0:167434 8.f 8:fae57b12:::benchmark_data_smithi098_226587_object63764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-21T10:28:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:20:45.094029+0000", 2024-09-21T10:28:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 471.47746819600002, 2024-09-21T10:28:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0043766080000001, 2024-09-21T10:28:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:28:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.45794", 2024-09-21T10:28:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2789621500", 2024-09-21T10:28:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 167434 2024-09-21T10:28:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:45.094029+0000", 2024-09-21T10:28:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:45.094029+0000", 2024-09-21T10:28:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:45.094031+0000", 2024-09-21T10:28:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.739e-06 2024-09-21T10:28:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:45.094034+0000", 2024-09-21T10:28:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4640000000000002e-06 2024-09-21T10:28:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:45.094043+0000", 2024-09-21T10:28:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5969999999999992e-06 2024-09-21T10:28:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:45.094049+0000", 2024-09-21T10:28:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7910000000000001e-06 2024-09-21T10:28:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:45.098370+0000", 2024-09-21T10:28:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004320997 2024-09-21T10:28:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:28:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:45.098380+0000", 2024-09-21T10:28:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0259e-05 2024-09-21T10:28:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:46.084966+0000", 2024-09-21T10:28:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.98658582399999994 2024-09-21T10:28:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:46.085041+0000", 2024-09-21T10:28:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5030999999999994e-05 2024-09-21T10:28:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:28:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:46.085135+0000", 2024-09-21T10:28:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4518999999999994e-05 2024-09-21T10:28:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:46.089050+0000", 2024-09-21T10:28:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039150110000000004 2024-09-21T10:28:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:28:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:46.098004+0000", 2024-09-21T10:28:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0089533240000000004 2024-09-21T10:28:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:46.098310+0000", 2024-09-21T10:28:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00030600499999999998 2024-09-21T10:28:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:28:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:46.098366+0000", 2024-09-21T10:28:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6555999999999999e-05 2024-09-21T10:28:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:28:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:20:46.098406+0000", 2024-09-21T10:28:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9490999999999998e-05 2024-09-21T10:28:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46025.0:265409 9.6 9:66bb34ee:::benchmark_data_smithi098_254407_object132678:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-09-21T10:28:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:26:16.865237+0000", 2024-09-21T10:28:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 139.70626016099999, 2024-09-21T10:28:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.184752904, 2024-09-21T10:28:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:28:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46025", 2024-09-21T10:28:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:28:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 265409 2024-09-21T10:28:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.865237+0000", 2024-09-21T10:28:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.865237+0000", 2024-09-21T10:28:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.865239+0000", 2024-09-21T10:28:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7439999999999999e-06 2024-09-21T10:28:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.865243+0000", 2024-09-21T10:28:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5940000000000004e-06 2024-09-21T10:28:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.865257+0000", 2024-09-21T10:28:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3168e-05 2024-09-21T10:28:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.865264+0000", 2024-09-21T10:28:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7069999999999992e-06 2024-09-21T10:28:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.865347+0000", 2024-09-21T10:28:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2398999999999995e-05 2024-09-21T10:28:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.865449+0000", 2024-09-21T10:28:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000102352 2024-09-21T10:28:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:28:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.865562+0000", 2024-09-21T10:28:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011291800000000001 2024-09-21T10:28:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.869888+0000", 2024-09-21T10:28:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043258319999999999 2024-09-21T10:28:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:28:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.869903+0000", 2024-09-21T10:28:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5571000000000001e-05 2024-09-21T10:28:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:17.049906+0000", 2024-09-21T10:28:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.180002422 2024-09-21T10:28:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:28:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:17.049953+0000", 2024-09-21T10:28:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7017000000000001e-05 2024-09-21T10:28:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:28:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:17.049990+0000", 2024-09-21T10:28:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.718e-05 2024-09-21T10:28:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:114049 10.5 10:a91b5b21:::benchmark_data_smithi098_281896_object114048:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-09-21T10:28:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:01.455401+0000", 2024-09-21T10:28:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 35.116095876999999, 2024-09-21T10:28:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.223158365, 2024-09-21T10:28:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:28:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:28:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:28:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 114049 2024-09-21T10:28:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.455401+0000", 2024-09-21T10:28:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.455401+0000", 2024-09-21T10:28:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.455403+0000", 2024-09-21T10:28:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7290000000000001e-06 2024-09-21T10:28:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.455429+0000", 2024-09-21T10:28:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5422000000000002e-05 2024-09-21T10:28:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.455442+0000", 2024-09-21T10:28:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3037000000000001e-05 2024-09-21T10:28:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.455450+0000", 2024-09-21T10:28:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3219999999999993e-06 2024-09-21T10:28:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.497851+0000", 2024-09-21T10:28:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.042401094 2024-09-21T10:28:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:28:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.497857+0000", 2024-09-21T10:28:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4130000000000003e-06 2024-09-21T10:28:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.649683+0000", 2024-09-21T10:28:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.15182572799999999 2024-09-21T10:28:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.649865+0000", 2024-09-21T10:28:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018164200000000001 2024-09-21T10:28:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:28:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.650040+0000", 2024-09-21T10:28:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017542400000000001 2024-09-21T10:28:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.664461+0000", 2024-09-21T10:28:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014420448000000001 2024-09-21T10:28:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:28:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.664519+0000", 2024-09-21T10:28:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8797e-05 2024-09-21T10:28:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.673850+0000", 2024-09-21T10:28:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0093306540000000007 2024-09-21T10:28:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:28:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.673953+0000", 2024-09-21T10:28:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000103106 2024-09-21T10:28:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:28:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.678560+0000", 2024-09-21T10:28:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046065489999999997 2024-09-21T10:28:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:114069 10.5 10:a106a2ea:::benchmark_data_smithi098_281896_object114068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-09-21T10:28:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:01.497035+0000", 2024-09-21T10:28:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 35.074462142999998, 2024-09-21T10:28:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.18686325600000001, 2024-09-21T10:28:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:28:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:28:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:28:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 114069 2024-09-21T10:28:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.497035+0000", 2024-09-21T10:28:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.497035+0000", 2024-09-21T10:28:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.497037+0000", 2024-09-21T10:28:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7710000000000001e-06 2024-09-21T10:28:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.497059+0000", 2024-09-21T10:28:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1840000000000001e-05 2024-09-21T10:28:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.497072+0000", 2024-09-21T10:28:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2911999999999999e-05 2024-09-21T10:28:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.497080+0000", 2024-09-21T10:28:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0979999999999994e-06 2024-09-21T10:28:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.497870+0000", 2024-09-21T10:28:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00079043099999999997 2024-09-21T10:28:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:28:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.497874+0000", 2024-09-21T10:28:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3780000000000001e-06 2024-09-21T10:28:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.654540+0000", 2024-09-21T10:28:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.15666666900000001 2024-09-21T10:28:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.654706+0000", 2024-09-21T10:28:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016584 2024-09-21T10:28:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:28:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.654876+0000", 2024-09-21T10:28:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017001000000000001 2024-09-21T10:28:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.664505+0000", 2024-09-21T10:28:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0096288929999999995 2024-09-21T10:28:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:28:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.664526+0000", 2024-09-21T10:28:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0803999999999999e-05 2024-09-21T10:28:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.678631+0000", 2024-09-21T10:28:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014105695 2024-09-21T10:28:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:28:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.678761+0000", 2024-09-21T10:28:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012910100000000001 2024-09-21T10:28:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:28:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.683898+0000", 2024-09-21T10:28:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051378140000000001 2024-09-21T10:28:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:114077 10.3 10:c932c26e:::benchmark_data_smithi098_281896_object114076:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-09-21T10:28:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:01.525273+0000", 2024-09-21T10:28:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 35.046224105999997, 2024-09-21T10:28:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.19998084099999999, 2024-09-21T10:28:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:28:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:28:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:28:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 114077 2024-09-21T10:28:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.525273+0000", 2024-09-21T10:28:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.525273+0000", 2024-09-21T10:28:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.525275+0000", 2024-09-21T10:28:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6780000000000001e-06 2024-09-21T10:28:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.525310+0000", 2024-09-21T10:28:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5423999999999998e-05 2024-09-21T10:28:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.525328+0000", 2024-09-21T10:28:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8151e-05 2024-09-21T10:28:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.525337+0000", 2024-09-21T10:28:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0589999999999992e-06 2024-09-21T10:28:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.525402+0000", 2024-09-21T10:28:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4238000000000004e-05 2024-09-21T10:28:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:28:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.525412+0000", 2024-09-21T10:28:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8719999999999997e-06 2024-09-21T10:28:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.686377+0000", 2024-09-21T10:28:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.16096533099999999 2024-09-21T10:28:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.686539+0000", 2024-09-21T10:28:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016166600000000001 2024-09-21T10:28:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:28:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.686700+0000", 2024-09-21T10:28:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016121500000000001 2024-09-21T10:28:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.692189+0000", 2024-09-21T10:28:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054890980000000004 2024-09-21T10:28:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:28:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.701100+0000", 2024-09-21T10:28:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0089110319999999993 2024-09-21T10:28:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.719247+0000", 2024-09-21T10:28:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018147513000000001 2024-09-21T10:28:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:28:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.719457+0000", 2024-09-21T10:28:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020962300000000001 2024-09-21T10:28:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:28:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.725254+0000", 2024-09-21T10:28:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057969409999999999 2024-09-21T10:28:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:114085 10.3 10:c5f5ce7e:::benchmark_data_smithi098_281896_object114084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-09-21T10:28:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:01.549413+0000", 2024-09-21T10:28:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 35.022084282999998, 2024-09-21T10:28:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.181503585, 2024-09-21T10:28:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:28:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:28:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:28:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 114085 2024-09-21T10:28:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.549413+0000", 2024-09-21T10:28:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.549413+0000", 2024-09-21T10:28:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.549415+0000", 2024-09-21T10:28:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.75e-06 2024-09-21T10:28:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.549437+0000", 2024-09-21T10:28:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2529000000000001e-05 2024-09-21T10:28:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.549454+0000", 2024-09-21T10:28:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6373999999999999e-05 2024-09-21T10:28:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.549462+0000", 2024-09-21T10:28:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3499999999999997e-06 2024-09-21T10:28:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.549532+0000", 2024-09-21T10:28:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0485e-05 2024-09-21T10:28:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:28:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.549544+0000", 2024-09-21T10:28:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1083999999999999e-05 2024-09-21T10:28:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.692219+0000", 2024-09-21T10:28:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.14267558399999999 2024-09-21T10:28:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.692368+0000", 2024-09-21T10:28:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014861200000000001 2024-09-21T10:28:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:28:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.692525+0000", 2024-09-21T10:28:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000157114 2024-09-21T10:28:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:28:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.701111+0000", 2024-09-21T10:28:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0085857799999999995 2024-09-21T10:28:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.703457+0000", 2024-09-21T10:28:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002346744 2024-09-21T10:28:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.725305+0000", 2024-09-21T10:28:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.021847887 2024-09-21T10:28:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:28:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.725354+0000", 2024-09-21T10:28:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8726000000000001e-05 2024-09-21T10:28:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:28:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.730917+0000", 2024-09-21T10:28:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055625659999999997 2024-09-21T10:28:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:114728 10.5 10:a6448361:::benchmark_data_smithi098_281896_object114727:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-21T10:28:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:02.533408+0000", 2024-09-21T10:28:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 34.038088811000001, 2024-09-21T10:28:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.216344283, 2024-09-21T10:28:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:28:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:28:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:28:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 114728 2024-09-21T10:28:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.533408+0000", 2024-09-21T10:28:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.533408+0000", 2024-09-21T10:28:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.533410+0000", 2024-09-21T10:28:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4869999999999999e-06 2024-09-21T10:28:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.533437+0000", 2024-09-21T10:28:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7410999999999999e-05 2024-09-21T10:28:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.533448+0000", 2024-09-21T10:28:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0917000000000001e-05 2024-09-21T10:28:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.533459+0000", 2024-09-21T10:28:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0363999999999999e-05 2024-09-21T10:28:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.545536+0000", 2024-09-21T10:28:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012076988 2024-09-21T10:28:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.545705+0000", 2024-09-21T10:28:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016944200000000001 2024-09-21T10:28:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:28:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.546015+0000", 2024-09-21T10:28:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00030984300000000002 2024-09-21T10:28:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.557937+0000", 2024-09-21T10:28:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011921932 2024-09-21T10:28:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:28:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.557952+0000", 2024-09-21T10:28:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5170999999999999e-05 2024-09-21T10:28:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.744455+0000", 2024-09-21T10:28:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.18650330400000001 2024-09-21T10:28:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:28:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.744517+0000", 2024-09-21T10:28:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1334000000000006e-05 2024-09-21T10:28:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:28:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.749753+0000", 2024-09-21T10:28:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052360899999999997 2024-09-21T10:28:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:114735 10.5 10:a3b215c2:::benchmark_data_smithi098_281896_object114734:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-21T10:28:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:02.534475+0000", 2024-09-21T10:28:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 34.037021973000002, 2024-09-21T10:28:36.636 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.22054879599999999, 2024-09-21T10:28:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:28:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:28:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:28:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 114735 2024-09-21T10:28:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.534475+0000", 2024-09-21T10:28:36.637 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.534475+0000", 2024-09-21T10:28:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.534477+0000", 2024-09-21T10:28:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-21T10:28:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:36.638 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.534497+0000", 2024-09-21T10:28:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0078e-05 2024-09-21T10:28:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.534515+0000", 2024-09-21T10:28:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8036000000000001e-05 2024-09-21T10:28:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.534528+0000", 2024-09-21T10:28:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3369e-05 2024-09-21T10:28:36.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.552136+0000", 2024-09-21T10:28:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01760774 2024-09-21T10:28:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.552314+0000", 2024-09-21T10:28:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017825800000000001 2024-09-21T10:28:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:28:36.640 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.552471+0000", 2024-09-21T10:28:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000156532 2024-09-21T10:28:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:28:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.577378+0000", 2024-09-21T10:28:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.024906924 2024-09-21T10:28:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.577440+0000", 2024-09-21T10:28:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2334000000000003e-05 2024-09-21T10:28:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.641 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.749808+0000", 2024-09-21T10:28:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.17236832199999999 2024-09-21T10:28:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:28:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.749916+0000", 2024-09-21T10:28:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010746799999999999 2024-09-21T10:28:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:28:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.755024+0000", 2024-09-21T10:28:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051083179999999997 2024-09-21T10:28:36.642 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:114736 10.5 10:a153896d:::benchmark_data_smithi098_281896_object114735:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-21T10:28:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:02.535607+0000", 2024-09-21T10:28:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 34.035889883999999, 2024-09-21T10:28:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.22527894700000001, 2024-09-21T10:28:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:28:36.643 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:28:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:28:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 114736 2024-09-21T10:28:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.535607+0000", 2024-09-21T10:28:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.535607+0000", 2024-09-21T10:28:36.644 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.535609+0000", 2024-09-21T10:28:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3599999999999999e-06 2024-09-21T10:28:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.535654+0000", 2024-09-21T10:28:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5602999999999997e-05 2024-09-21T10:28:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.535666+0000", 2024-09-21T10:28:36.645 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2054e-05 2024-09-21T10:28:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.535674+0000", 2024-09-21T10:28:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7030000000000006e-06 2024-09-21T10:28:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.558050+0000", 2024-09-21T10:28:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.022376255000000001 2024-09-21T10:28:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.646 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.558251+0000", 2024-09-21T10:28:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020019100000000001 2024-09-21T10:28:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:28:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.558393+0000", 2024-09-21T10:28:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000142931 2024-09-21T10:28:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:28:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.577385+0000", 2024-09-21T10:28:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018991642999999999 2024-09-21T10:28:36.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.577498+0000", 2024-09-21T10:28:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000112521 2024-09-21T10:28:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.755073+0000", 2024-09-21T10:28:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.177575545 2024-09-21T10:28:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:28:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.755201+0000", 2024-09-21T10:28:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012760700000000001 2024-09-21T10:28:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:28:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.760886+0000", 2024-09-21T10:28:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056855339999999999 2024-09-21T10:28:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:114756 10.5 10:a9ea4135:::benchmark_data_smithi098_281896_object114755:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-21T10:28:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:02.561872+0000", 2024-09-21T10:28:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 34.009624879, 2024-09-21T10:28:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.20555221200000001, 2024-09-21T10:28:36.649 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:36.650 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:36.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:36 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 14273/345900 objects degraded (4.126%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:28:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:28:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:28:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 114756 2024-09-21T10:28:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:36.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.561872+0000", 2024-09-21T10:28:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.561872+0000", 2024-09-21T10:28:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.561874+0000", 2024-09-21T10:28:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7540000000000001e-06 2024-09-21T10:28:36.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.561903+0000", 2024-09-21T10:28:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8479999999999998e-05 2024-09-21T10:28:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.561917+0000", 2024-09-21T10:28:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4191e-05 2024-09-21T10:28:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:36.653 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.561925+0000", 2024-09-21T10:28:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5089999999999993e-06 2024-09-21T10:28:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.562239+0000", 2024-09-21T10:28:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031390200000000002 2024-09-21T10:28:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.562422+0000", 2024-09-21T10:28:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000183003 2024-09-21T10:28:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.654 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:28:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.562595+0000", 2024-09-21T10:28:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017252200000000001 2024-09-21T10:28:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:28:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.577512+0000", 2024-09-21T10:28:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014916992 2024-09-21T10:28:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.577547+0000", 2024-09-21T10:28:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5769000000000003e-05 2024-09-21T10:28:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.760953+0000", 2024-09-21T10:28:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.18340527400000001 2024-09-21T10:28:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:28:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.761221+0000", 2024-09-21T10:28:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00026867699999999999 2024-09-21T10:28:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:28:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.767425+0000", 2024-09-21T10:28:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062031389999999999 2024-09-21T10:28:36.656 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:114759 10.3 10:ccb9f190:::benchmark_data_smithi098_281896_object114758:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-21T10:28:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:02.562515+0000", 2024-09-21T10:28:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 34.008982097000001, 2024-09-21T10:28:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.18183801599999999, 2024-09-21T10:28:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:28:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:28:36.657 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:28:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 114759 2024-09-21T10:28:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.562515+0000", 2024-09-21T10:28:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.562515+0000", 2024-09-21T10:28:36.658 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.562516+0000", 2024-09-21T10:28:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.02e-06 2024-09-21T10:28:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.562537+0000", 2024-09-21T10:28:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0530000000000002e-05 2024-09-21T10:28:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.562549+0000", 2024-09-21T10:28:36.659 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2191000000000001e-05 2024-09-21T10:28:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.562556+0000", 2024-09-21T10:28:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5549999999999997e-06 2024-09-21T10:28:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.562617+0000", 2024-09-21T10:28:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0454000000000001e-05 2024-09-21T10:28:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.660 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.562799+0000", 2024-09-21T10:28:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018246899999999999 2024-09-21T10:28:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:28:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.562982+0000", 2024-09-21T10:28:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018293300000000001 2024-09-21T10:28:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.567370+0000", 2024-09-21T10:28:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043879469999999997 2024-09-21T10:28:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:28:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.569062+0000", 2024-09-21T10:28:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001691344 2024-09-21T10:28:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.575799+0000", 2024-09-21T10:28:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0067372090000000001 2024-09-21T10:28:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:28:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.575860+0000", 2024-09-21T10:28:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1476999999999994e-05 2024-09-21T10:28:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:28:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.744353+0000", 2024-09-21T10:28:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.16849288700000001 2024-09-21T10:28:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:114760 10.5 10:ab5b8f37:::benchmark_data_smithi098_281896_object114759:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-21T10:28:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:02.563166+0000", 2024-09-21T10:28:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 34.008330854, 2024-09-21T10:28:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.21826158900000001, 2024-09-21T10:28:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:36.663 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:28:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:28:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:28:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 114760 2024-09-21T10:28:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.563166+0000", 2024-09-21T10:28:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.563166+0000", 2024-09-21T10:28:36.664 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.563168+0000", 2024-09-21T10:28:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.201e-06 2024-09-21T10:28:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.563222+0000", 2024-09-21T10:28:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4141999999999998e-05 2024-09-21T10:28:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:36.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.563235+0000", 2024-09-21T10:28:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.33e-05 2024-09-21T10:28:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.563243+0000", 2024-09-21T10:28:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8869999999999992e-06 2024-09-21T10:28:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.569889+0000", 2024-09-21T10:28:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0066461630000000001 2024-09-21T10:28:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:36.666 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.570071+0000", 2024-09-21T10:28:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000182287 2024-09-21T10:28:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:28:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.570256+0000", 2024-09-21T10:28:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018436 2024-09-21T10:28:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.579264+0000", 2024-09-21T10:28:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0090080300000000002 2024-09-21T10:28:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:28:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.744409+0000", 2024-09-21T10:28:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.16514512200000001 2024-09-21T10:28:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.769251+0000", 2024-09-21T10:28:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.024842426000000001 2024-09-21T10:28:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:28:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.769351+0000", 2024-09-21T10:28:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9467000000000002e-05 2024-09-21T10:28:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:28:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.781428+0000", 2024-09-21T10:28:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012077203999999999 2024-09-21T10:28:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.46322.0:114759 10.3 e363/361 ondisk, result = 0)", 2024-09-21T10:28:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:02.575675+0000", 2024-09-21T10:28:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.995821906000003, 2024-09-21T10:28:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.16869122, 2024-09-21T10:28:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:36.669 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:28:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.575675+0000", 2024-09-21T10:28:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.575675+0000", 2024-09-21T10:28:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.575677+0000", 2024-09-21T10:28:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2440000000000001e-06 2024-09-21T10:28:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.575683+0000", 2024-09-21T10:28:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6660000000000002e-06 2024-09-21T10:28:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.575699+0000", 2024-09-21T10:28:36.671 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5421999999999999e-05 2024-09-21T10:28:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.575707+0000", 2024-09-21T10:28:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5730000000000008e-06 2024-09-21T10:28:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.575782+0000", 2024-09-21T10:28:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4852999999999996e-05 2024-09-21T10:28:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.575792+0000", 2024-09-21T10:28:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9920000000000002e-06 2024-09-21T10:28:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:28:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.744367+0000", 2024-09-21T10:28:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.16857447 2024-09-21T10:28:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:114767 10.3 10:cfdc2cde:::benchmark_data_smithi098_281896_object114766:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-21T10:28:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:02.576301+0000", 2024-09-21T10:28:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.995196069999999, 2024-09-21T10:28:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.18460564300000001, 2024-09-21T10:28:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:28:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:28:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:28:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 114767 2024-09-21T10:28:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.576301+0000", 2024-09-21T10:28:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.576301+0000", 2024-09-21T10:28:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.576303+0000", 2024-09-21T10:28:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7099999999999999e-06 2024-09-21T10:28:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.576326+0000", 2024-09-21T10:28:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3487000000000001e-05 2024-09-21T10:28:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.576337+0000", 2024-09-21T10:28:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0959e-05 2024-09-21T10:28:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.576344+0000", 2024-09-21T10:28:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9249999999999998e-06 2024-09-21T10:28:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.744411+0000", 2024-09-21T10:28:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.168067155 2024-09-21T10:28:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.744598+0000", 2024-09-21T10:28:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000186646 2024-09-21T10:28:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:28:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.744805+0000", 2024-09-21T10:28:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020698799999999999 2024-09-21T10:28:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.748975+0000", 2024-09-21T10:28:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0041695899999999999 2024-09-21T10:28:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:28:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.753697+0000", 2024-09-21T10:28:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0047225749999999997 2024-09-21T10:28:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.754822+0000", 2024-09-21T10:28:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00112469 2024-09-21T10:28:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:28:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.754883+0000", 2024-09-21T10:28:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1001000000000001e-05 2024-09-21T10:28:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:28:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.760907+0000", 2024-09-21T10:28:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0060239170000000002 2024-09-21T10:28:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:114768 10.e e363/348 10:78c364d3:::benchmark_data_smithi098_281896_object114767:head v 363'7222, mlcod=363'7220)", 2024-09-21T10:28:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:02.576764+0000", 2024-09-21T10:28:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.994733136000001, 2024-09-21T10:28:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.16752075299999999, 2024-09-21T10:28:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.576764+0000", 2024-09-21T10:28:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.576764+0000", 2024-09-21T10:28:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.576765+0000", 2024-09-21T10:28:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2799999999999995e-07 2024-09-21T10:28:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.576854+0000", 2024-09-21T10:28:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9375000000000004e-05 2024-09-21T10:28:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.576865+0000", 2024-09-21T10:28:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0623e-05 2024-09-21T10:28:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.576871+0000", 2024-09-21T10:28:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2910000000000004e-06 2024-09-21T10:28:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.576920+0000", 2024-09-21T10:28:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8844999999999997e-05 2024-09-21T10:28:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.576939+0000", 2024-09-21T10:28:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9143999999999998e-05 2024-09-21T10:28:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:28:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.580270+0000", 2024-09-21T10:28:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033305779999999998 2024-09-21T10:28:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:28:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.744285+0000", 2024-09-21T10:28:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.16401506900000001 2024-09-21T10:28:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:114772 10.9 e363/351 10:9fcfe052:::benchmark_data_smithi098_281896_object114771:head v 363'7336, mlcod=363'7334)", 2024-09-21T10:28:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:02.585150+0000", 2024-09-21T10:28:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.986347262999999, 2024-09-21T10:28:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.17071482800000001, 2024-09-21T10:28:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.585150+0000", 2024-09-21T10:28:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.585150+0000", 2024-09-21T10:28:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.585152+0000", 2024-09-21T10:28:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.956e-06 2024-09-21T10:28:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.585186+0000", 2024-09-21T10:28:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3599000000000001e-05 2024-09-21T10:28:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.745207+0000", 2024-09-21T10:28:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.16002175399999999 2024-09-21T10:28:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.745216+0000", 2024-09-21T10:28:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7819999999999999e-06 2024-09-21T10:28:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.745281+0000", 2024-09-21T10:28:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4888999999999995e-05 2024-09-21T10:28:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.745297+0000", 2024-09-21T10:28:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6033e-05 2024-09-21T10:28:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:28:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.755831+0000", 2024-09-21T10:28:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010533934999999999 2024-09-21T10:28:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:28:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.755865+0000", 2024-09-21T10:28:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3880000000000001e-05 2024-09-21T10:28:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:114774 10.8 e363/348 10:1f7f13ed:::benchmark_data_smithi098_281896_object114773:head v 363'7127, mlcod=363'7125)", 2024-09-21T10:28:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:02.587888+0000", 2024-09-21T10:28:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.983608873000001, 2024-09-21T10:28:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.16897221900000001, 2024-09-21T10:28:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.587888+0000", 2024-09-21T10:28:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.587888+0000", 2024-09-21T10:28:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.587890+0000", 2024-09-21T10:28:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1489999999999999e-06 2024-09-21T10:28:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.587911+0000", 2024-09-21T10:28:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1603e-05 2024-09-21T10:28:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.745200+0000", 2024-09-21T10:28:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.157288699 2024-09-21T10:28:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.745209+0000", 2024-09-21T10:28:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8689999999999993e-06 2024-09-21T10:28:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.750084+0000", 2024-09-21T10:28:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048750180000000001 2024-09-21T10:28:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.750099+0000", 2024-09-21T10:28:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5362999999999998e-05 2024-09-21T10:28:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:28:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.756808+0000", 2024-09-21T10:28:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.006709067 2024-09-21T10:28:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:28:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.756861+0000", 2024-09-21T10:28:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2451000000000003e-05 2024-09-21T10:28:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:136218 10.3 10:cb407e02:::benchmark_data_smithi098_281896_object11193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-21T10:28:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:33.825437+0000", 2024-09-21T10:28:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 2.746060484, 2024-09-21T10:28:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.13842306400000001, 2024-09-21T10:28:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:28:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:28:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:28:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 136218 2024-09-21T10:28:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.825437+0000", 2024-09-21T10:28:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.825437+0000", 2024-09-21T10:28:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.825438+0000", 2024-09-21T10:28:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7489999999999999e-06 2024-09-21T10:28:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.825444+0000", 2024-09-21T10:28:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9719999999999997e-06 2024-09-21T10:28:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.825459+0000", 2024-09-21T10:28:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4167e-05 2024-09-21T10:28:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.825466+0000", 2024-09-21T10:28:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1520000000000003e-06 2024-09-21T10:28:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.837367+0000", 2024-09-21T10:28:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011901180000000001 2024-09-21T10:28:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.837667+0000", 2024-09-21T10:28:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00030006899999999998 2024-09-21T10:28:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:28:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.838037+0000", 2024-09-21T10:28:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037025699999999998 2024-09-21T10:28:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.848292+0000", 2024-09-21T10:28:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010254655 2024-09-21T10:28:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:28:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.848328+0000", 2024-09-21T10:28:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6455999999999998e-05 2024-09-21T10:28:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.958660+0000", 2024-09-21T10:28:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.110331652 2024-09-21T10:28:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:28:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.958722+0000", 2024-09-21T10:28:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1719999999999999e-05 2024-09-21T10:28:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:28:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.963860+0000", 2024-09-21T10:28:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005138035 2024-09-21T10:28:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:136231 10.3 10:c2d96113:::benchmark_data_smithi098_281896_object11206:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-21T10:28:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:33.843739+0000", 2024-09-21T10:28:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 2.7277580970000002, 2024-09-21T10:28:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.140585988, 2024-09-21T10:28:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:28:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:28:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:28:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:28:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:28:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 136231 2024-09-21T10:28:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:28:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:28:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.843739+0000", 2024-09-21T10:28:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:28:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.843739+0000", 2024-09-21T10:28:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:28:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:28:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.843740+0000", 2024-09-21T10:28:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0699999999999996e-07 2024-09-21T10:28:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:28:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.843744+0000", 2024-09-21T10:28:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8319999999999999e-06 2024-09-21T10:28:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:28:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.843751+0000", 2024-09-21T10:28:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5259999999999996e-06 2024-09-21T10:28:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:28:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.843756+0000", 2024-09-21T10:28:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1909999999999999e-06 2024-09-21T10:28:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:28:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.854607+0000", 2024-09-21T10:28:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010850688000000001 2024-09-21T10:28:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:28:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.854767+0000", 2024-09-21T10:28:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000159498 2024-09-21T10:28:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:28:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.854912+0000", 2024-09-21T10:28:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014533800000000001 2024-09-21T10:28:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.860248+0000", 2024-09-21T10:28:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053361390000000002 2024-09-21T10:28:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:28:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.860318+0000", 2024-09-21T10:28:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.991e-05 2024-09-21T10:28:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.976733+0000", 2024-09-21T10:28:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.116415247 2024-09-21T10:28:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:28:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.976894+0000", 2024-09-21T10:28:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000161201 2024-09-21T10:28:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:28:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:28:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:28:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.984325+0000", 2024-09-21T10:28:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0074307109999999996 2024-09-21T10:28:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:28:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:28:36.706 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:28:36.706 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:28:36.707 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:36.733 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:36.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:36.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:36.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:36.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:36.887 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:36.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:36.988 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:37.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:37.046 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:37.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:37.147 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:37.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:37.194 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:37.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:37.295 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:37.318 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:37.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:37.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:37.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:37 smithi098 ceph-mon[97580]: pgmap v1247: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 14412/342387 objects degraded (4.209%); 20491/342387 objects misplaced (5.985%); 800 KiB/s, 12 objects/s recovering 2024-09-21T10:28:37.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:37 smithi098 ceph-mon[100914]: pgmap v1247: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 14412/342387 objects degraded (4.209%); 20491/342387 objects misplaced (5.985%); 800 KiB/s, 12 objects/s recovering 2024-09-21T10:28:37.420 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:37.453 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:37.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:37.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:37.555 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:37.596 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:37.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:37.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:37 smithi196 ceph-mon[84784]: pgmap v1247: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 14412/342387 objects degraded (4.209%); 20491/342387 objects misplaced (5.985%); 800 KiB/s, 12 objects/s recovering 2024-09-21T10:28:37.697 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:37.740 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:37.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:37.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:37.841 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:37.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:37.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:37.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:28:37.979 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:38.008 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:38.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:38.109 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:38.138 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:38.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:38.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:38.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:38.282 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:38.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:38.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:38.384 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:38.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:38.423 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:38.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:38.524 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:38.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:38.566 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:38.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:38.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:38.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:38.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:38.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:38.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:38.706 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:38.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:38.807 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:38.844 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:38.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:38.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:38.946 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:38.975 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:38.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:38.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:39.077 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:39.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:39.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:39.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:39.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:39.243 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:39.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:39.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:39.344 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:39.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:39 smithi098 ceph-mon[97580]: pgmap v1248: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 14412/341691 objects degraded (4.218%); 20491/341691 objects misplaced (5.997%); 800 KiB/s, 12 objects/s recovering 2024-09-21T10:28:39.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:39 smithi098 ceph-mon[100914]: pgmap v1248: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 14412/341691 objects degraded (4.218%); 20491/341691 objects misplaced (5.997%); 800 KiB/s, 12 objects/s recovering 2024-09-21T10:28:39.377 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:39.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:39.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:39.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:39.510 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:39.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:39.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:39.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:39 smithi196 ceph-mon[84784]: pgmap v1248: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 14412/341691 objects degraded (4.218%); 20491/341691 objects misplaced (5.997%); 800 KiB/s, 12 objects/s recovering 2024-09-21T10:28:39.612 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:39.655 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:39.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:39.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:39.757 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:39.791 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:39.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:39.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:39.892 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:39.921 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:39.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:39.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:40.022 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:40.051 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:40.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:40.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:40.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:40.187 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:40.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:40.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:40.289 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:40.319 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:40.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:40.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:40.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:28:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:28:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:28:40.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:40.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:40.483 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:40.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:40.584 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:40.608 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:40.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:40.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:40.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:40.745 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:40.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:40.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:40.847 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:40.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:28:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:28:40] "GET /metrics HTTP/1.1" 200 36614 "" "Prometheus/2.51.0" 2024-09-21T10:28:40.868 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:40.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:40.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:40.969 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:40.998 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:40.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:40.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:41.100 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:41.142 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:41.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:41.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:41.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:41.268 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:41.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:41.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:41.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:41 smithi098 ceph-mon[97580]: pgmap v1249: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.8 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 14581/333027 objects degraded (4.378%); 19793/333027 objects misplaced (5.943%); 785 KiB/s, 12 objects/s recovering 2024-09-21T10:28:41.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:41 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 14581/333027 objects degraded (4.378%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:41.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:41 smithi098 ceph-mon[100914]: pgmap v1249: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.8 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 14581/333027 objects degraded (4.378%); 19793/333027 objects misplaced (5.943%); 785 KiB/s, 12 objects/s recovering 2024-09-21T10:28:41.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:41 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 14581/333027 objects degraded (4.378%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:41.370 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:41.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:41.426 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:41.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:41.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:41.553 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:41.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:41.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:41.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:41 smithi196 ceph-mon[84784]: pgmap v1249: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.8 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 14581/333027 objects degraded (4.378%); 19793/333027 objects misplaced (5.943%); 785 KiB/s, 12 objects/s recovering 2024-09-21T10:28:41.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:41 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 14581/333027 objects degraded (4.378%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:41.655 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:41.694 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:41.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:41.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:41.796 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:41.821 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:41.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:41.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:41.922 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:41.948 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:41.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:41.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:42.049 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:42.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:42.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:42.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:42.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:42.199 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:42.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:42.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:42.230 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:28:42.301 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:42.338 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:42.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:42.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:42.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:42.444 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:42 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/972997364' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:28:42.445 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:42 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/972997364' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:28:42.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:42.477 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:42.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:42.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:42.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:42.612 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:42.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:42.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:42 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/972997364' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:28:42.713 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:42.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:42.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:42.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:42.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:42.890 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:42.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:42.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:42.992 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:43.034 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:43.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:43.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:43.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:43.182 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:43.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:43.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T10:28:43.265 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:43.284 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:43.307 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:43.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:43.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:43.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:43 smithi098 ceph-mon[97580]: pgmap v1250: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.8 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 643 op/s; 14581/331788 objects degraded (4.395%); 19793/331788 objects misplaced (5.966%); 682 KiB/s, 10 objects/s recovering 2024-09-21T10:28:43.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:43 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:28:43.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:43 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/972997364' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:28:43.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:43 smithi098 ceph-mon[97580]: osdmap e368: 8 total, 8 up, 8 in 2024-09-21T10:28:43.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:43 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/426861062' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:28:43.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:43 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:28:43.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:43 smithi098 ceph-mon[100914]: pgmap v1250: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.8 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 643 op/s; 14581/331788 objects degraded (4.395%); 19793/331788 objects misplaced (5.966%); 682 KiB/s, 10 objects/s recovering 2024-09-21T10:28:43.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:43 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:28:43.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:43 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/972997364' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:28:43.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:43 smithi098 ceph-mon[100914]: osdmap e368: 8 total, 8 up, 8 in 2024-09-21T10:28:43.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:43 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/426861062' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:28:43.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:43 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:28:43.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:43.446 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:43.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:43.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:43.548 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:43.579 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:43.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:43.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:43.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:43 smithi196 ceph-mon[84784]: pgmap v1250: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.8 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 643 op/s; 14581/331788 objects degraded (4.395%); 19793/331788 objects misplaced (5.966%); 682 KiB/s, 10 objects/s recovering 2024-09-21T10:28:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:43 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:28:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:43 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/972997364' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:28:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:43 smithi196 ceph-mon[84784]: osdmap e368: 8 total, 8 up, 8 in 2024-09-21T10:28:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:43 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/426861062' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:28:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:43 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:28:43.682 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:43.713 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:43.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:43.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:43.814 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:43.840 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:43.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:43.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:43.942 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:43.967 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:43.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:43.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:44.069 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:44.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:44.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:44.212 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:44.236 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:44.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:44.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:44.338 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:44.361 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:44.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:44.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:44.462 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:44.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:44.496 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:44.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:44.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:44 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:28:44.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:44 smithi098 ceph-mon[97580]: osdmap e369: 8 total, 8 up, 8 in 2024-09-21T10:28:44.524 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:44 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:28:44.524 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:44 smithi098 ceph-mon[100914]: osdmap e369: 8 total, 8 up, 8 in 2024-09-21T10:28:44.597 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:44.636 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:44.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:44.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:44.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:44 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:28:44.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:44 smithi196 ceph-mon[84784]: osdmap e369: 8 total, 8 up, 8 in 2024-09-21T10:28:44.738 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:44.766 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:44.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:44.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:44.868 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:44.874 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T10:28:44.899 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:44.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:44.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:45.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:45.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:45.039 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:45.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:45.140 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:45.160 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:28:45.160 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:28:45.164 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:45.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:45.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:45.204 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:28:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:28:45.238 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:28:45.238 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:28:45.265 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:45.270 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:28:45.270 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:28:45.276 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:28:45.294 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:45.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:45.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:45.305 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:28:45.305 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:28:45.333 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:28:45.333 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:28:45.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:45 smithi098 ceph-mon[97580]: pgmap v1253: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 724 op/s; 14581/324996 objects degraded (4.487%); 19390/324996 objects misplaced (5.966%); 488 KiB/s, 7 objects/s recovering 2024-09-21T10:28:45.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:45 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1146862296' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:45.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:45 smithi098 ceph-mon[100914]: pgmap v1253: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 724 op/s; 14581/324996 objects degraded (4.487%); 19390/324996 objects misplaced (5.966%); 488 KiB/s, 7 objects/s recovering 2024-09-21T10:28:45.349 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:45 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1146862296' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:45.362 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:28:45.363 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:28:45.395 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:28:45.395 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:28:45.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:45.459 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:45.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:45.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:45.561 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:45.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:45.608 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:45.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:45.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:45 smithi196 ceph-mon[84784]: pgmap v1253: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 724 op/s; 14581/324996 objects degraded (4.487%); 19390/324996 objects misplaced (5.966%); 488 KiB/s, 7 objects/s recovering 2024-09-21T10:28:45.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:45 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1146862296' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:45.710 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:45.734 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:45.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:45.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:45.836 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:45.864 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:45.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:45.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:45.966 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:45.995 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:45.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:45.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:46.096 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:46.129 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:46.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:46.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:46.231 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:46.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:46.266 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:46.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:46.367 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:46.407 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:46.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:46.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:46.509 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:46.553 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:46.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:46.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:46.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:46 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 14581/324996 objects degraded (4.487%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:46.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:46 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 14581/324996 objects degraded (4.487%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:46.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:46 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 14581/324996 objects degraded (4.487%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:46.656 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:46.691 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:46.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:46.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:46.793 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:46.821 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:46.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:46.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:46.923 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:46.970 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:46.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:46.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:47.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:47.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:47.107 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:47.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:47.208 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:47.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:47.258 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:47.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:47.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:47.396 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:47.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:47.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:47.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:47.530 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:47.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:47.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:47.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:47 smithi098 ceph-mon[97580]: pgmap v1254: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 829 op/s; 14725/321777 objects degraded (4.576%); 19180/321777 objects misplaced (5.961%); 1015 KiB/s, 15 objects/s recovering 2024-09-21T10:28:47.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:47 smithi098 ceph-mon[100914]: pgmap v1254: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 829 op/s; 14725/321777 objects degraded (4.576%); 19180/321777 objects misplaced (5.961%); 1015 KiB/s, 15 objects/s recovering 2024-09-21T10:28:47.634 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:47.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:47 smithi196 ceph-mon[84784]: pgmap v1254: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 829 op/s; 14725/321777 objects degraded (4.576%); 19180/321777 objects misplaced (5.961%); 1015 KiB/s, 15 objects/s recovering 2024-09-21T10:28:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:47.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:47.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:47.768 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:47.810 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:47.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:47.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:47.912 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:47.954 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:47.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:47.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:48.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:48.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:48.095 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:48.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:48.196 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:48.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:48.237 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:48.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:48.338 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:48.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:48.391 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:48.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:48.492 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:48.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:48.518 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:48.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:48.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:48.654 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:48.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:48.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:48.756 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:48.781 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:48.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:48.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:48.883 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:48.911 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:48.911 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:48.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:48.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:48.939 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:49.013 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:49.038 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:49.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:49.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:49.139 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:49.180 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:49.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:49.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:49.282 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:49.306 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:49.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:49.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:49.408 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:49.435 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:49.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:49.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:49.465 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:49 smithi098 ceph-mon[97580]: pgmap v1255: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 492 op/s; 14725/321195 objects degraded (4.584%); 19180/321195 objects misplaced (5.971%); 528 KiB/s, 8 objects/s recovering 2024-09-21T10:28:49.465 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:49 smithi098 ceph-mon[100914]: pgmap v1255: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 492 op/s; 14725/321195 objects degraded (4.584%); 19180/321195 objects misplaced (5.971%); 528 KiB/s, 8 objects/s recovering 2024-09-21T10:28:49.537 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:49.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:49.567 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:49.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:49.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:49 smithi196 ceph-mon[84784]: pgmap v1255: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 492 op/s; 14725/321195 objects degraded (4.584%); 19180/321195 objects misplaced (5.971%); 528 KiB/s, 8 objects/s recovering 2024-09-21T10:28:49.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:49.696 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:49.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:49.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:49.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:28:49.797 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:49.821 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:49.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:49.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:49.922 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:49.947 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:49.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:49.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:50.049 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:50.077 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:50.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:50.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:50.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:50.200 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:50.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:50.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:50.301 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:50.323 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:50.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:50.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:50.424 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:50.445 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:50.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:50.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:50.547 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:50.575 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:50.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:50.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:50.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:28:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:28:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:28:50.678 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:50.705 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:50.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:50.807 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:50.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:50.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:50.948 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:50.977 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:50.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:50.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:51.079 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:51.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:28:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:28:50] "GET /metrics HTTP/1.1" 200 36614 "" "Prometheus/2.51.0" 2024-09-21T10:28:51.122 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:51.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:51.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:51.224 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:51.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:51.279 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:51.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:51.379 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:51.410 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:51.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:51.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:51.445 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:51 smithi098 ceph-mon[97580]: pgmap v1256: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.4 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 803 op/s; 14890/312498 objects degraded (4.765%); 18510/312498 objects misplaced (5.923%); 1016 KiB/s, 15 objects/s recovering 2024-09-21T10:28:51.445 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:51 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 14890/312498 objects degraded (4.765%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:51.445 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:51 smithi098 ceph-mon[100914]: pgmap v1256: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.4 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 803 op/s; 14890/312498 objects degraded (4.765%); 18510/312498 objects misplaced (5.923%); 1016 KiB/s, 15 objects/s recovering 2024-09-21T10:28:51.445 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:51 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 14890/312498 objects degraded (4.765%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:51.512 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:51.537 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:51.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:51.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:51.638 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:51 smithi196 ceph-mon[84784]: pgmap v1256: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.4 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 803 op/s; 14890/312498 objects degraded (4.765%); 18510/312498 objects misplaced (5.923%); 1016 KiB/s, 15 objects/s recovering 2024-09-21T10:28:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:51 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 14890/312498 objects degraded (4.765%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:51.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:51.686 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:51.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:51.787 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:51.819 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:51.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:51.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:51.921 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:51.948 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:51.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:51.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:52.050 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:52.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:52.079 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:52.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:52.180 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:52.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:52.223 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:52.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:52.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:52.322 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:52.324 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:52.348 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:52.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:52.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:52.449 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:52.478 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:52.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:52.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:52.580 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:52.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:52.609 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:52.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:52.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:52.710 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:52.736 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:52.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:52.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:52.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:52.869 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:52.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:52.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:52.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:53.001 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:53.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:53.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:53.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:53.127 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:53.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:53.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:53.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:53.274 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:53.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:53.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:53.298 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:28:53.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:53.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:53 smithi196 ceph-mon[84784]: pgmap v1257: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 733 op/s; 14890/311208 objects degraded (4.785%); 18510/311208 objects misplaced (5.948%); 869 KiB/s, 13 objects/s recovering 2024-09-21T10:28:53.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:53 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3656601650' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:28:53.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:53 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:28:53.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:53.422 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:53.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:53.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:53.564 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:53.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:53.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:53.592 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:53 smithi098 ceph-mon[97580]: pgmap v1257: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 733 op/s; 14890/311208 objects degraded (4.785%); 18510/311208 objects misplaced (5.948%); 869 KiB/s, 13 objects/s recovering 2024-09-21T10:28:53.592 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:53 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3656601650' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:28:53.592 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:53 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:28:53.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:53 smithi098 ceph-mon[100914]: pgmap v1257: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 733 op/s; 14890/311208 objects degraded (4.785%); 18510/311208 objects misplaced (5.948%); 869 KiB/s, 13 objects/s recovering 2024-09-21T10:28:53.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:53 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3656601650' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:28:53.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:53 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:28:53.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:53.691 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:28:53.691 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:28:53.703 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:53.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:53.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:53.805 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:53.844 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:53.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:53.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:53.945 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:53.968 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:53.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:53.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:54.070 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:54.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:54.108 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:54.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:54.209 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:54.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:54.257 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:54.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:54.358 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:54.396 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:54.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:54.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:54.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:54.534 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:54.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:54.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:54.582 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:54 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:28:54.582 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:54 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:28:54.582 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:54 smithi098 ceph-mon[97580]: osdmap e370: 8 total, 8 up, 8 in 2024-09-21T10:28:54.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:54 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:28:54.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:54 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:28:54.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:54 smithi098 ceph-mon[100914]: osdmap e370: 8 total, 8 up, 8 in 2024-09-21T10:28:54.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:28:54.636 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:54.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:54 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:28:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:54 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:28:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:54 smithi196 ceph-mon[84784]: osdmap e370: 8 total, 8 up, 8 in 2024-09-21T10:28:54.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:54.664 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:54.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:54.765 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:54.813 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:54.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:54.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:54.914 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:54.991 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:54.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:54.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:55.092 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:55.146 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:55.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:55.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:55.248 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:55.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:55.285 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:55.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:55.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:55.428 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:55 smithi098 ceph-mon[97580]: pgmap v1259: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 14890/304212 objects degraded (4.895%); 18095/304212 objects misplaced (5.948%); 813 KiB/s, 12 objects/s recovering 2024-09-21T10:28:55.428 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:55 smithi098 ceph-mon[100914]: pgmap v1259: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 14890/304212 objects degraded (4.895%); 18095/304212 objects misplaced (5.948%); 813 KiB/s, 12 objects/s recovering 2024-09-21T10:28:55.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:55.437 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:55.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:55.538 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:55.573 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:55.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:55.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:55.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:55 smithi196 ceph-mon[84784]: pgmap v1259: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 14890/304212 objects degraded (4.895%); 18095/304212 objects misplaced (5.948%); 813 KiB/s, 12 objects/s recovering 2024-09-21T10:28:55.675 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:55.702 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:55.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:55.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:55.803 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:55.833 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:55.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:55.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:55.935 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:55.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:55.965 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:55.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:56.066 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:56.099 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:56.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:56.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:56.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:56.230 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:56.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:56.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:56.317 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T10:28:56.332 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:56.372 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:56.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:56.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:56.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:56.496 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:56.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:56.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:56.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:56 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3993888853' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:28:56.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:56 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:28:56.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:56 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 14890/304212 objects degraded (4.895%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:56 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3993888853' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:28:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:56 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:28:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:56 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 14890/304212 objects degraded (4.895%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:56.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:56.621 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:56.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:56.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:56 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3993888853' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:28:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:56 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:28:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:56 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 14890/304212 objects degraded (4.895%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:28:56.722 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:56.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:56.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:56.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:56.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:56.866 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:56.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:56.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:56.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:57.000 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:57.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:57.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:57.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:57.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:57.147 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:57.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:57.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:57.273 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:57.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:57.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:57.375 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:57.404 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:57.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:57.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:57.505 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:57.538 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:57.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:57.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:57.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:57 smithi098 ceph-mon[97580]: pgmap v1260: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 687 op/s; 15031/301155 objects degraded (4.991%); 17892/301155 objects misplaced (5.941%); 787 KiB/s, 12 objects/s recovering 2024-09-21T10:28:57.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:57 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:28:57.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:57 smithi098 ceph-mon[97580]: osdmap e371: 8 total, 8 up, 8 in 2024-09-21T10:28:57.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:57 smithi098 ceph-mon[100914]: pgmap v1260: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 687 op/s; 15031/301155 objects degraded (4.991%); 17892/301155 objects misplaced (5.941%); 787 KiB/s, 12 objects/s recovering 2024-09-21T10:28:57.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:57 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:28:57.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:57 smithi098 ceph-mon[100914]: osdmap e371: 8 total, 8 up, 8 in 2024-09-21T10:28:57.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:57 smithi196 ceph-mon[84784]: pgmap v1260: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 687 op/s; 15031/301155 objects degraded (4.991%); 17892/301155 objects misplaced (5.941%); 787 KiB/s, 12 objects/s recovering 2024-09-21T10:28:57.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:57 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:28:57.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:57 smithi196 ceph-mon[84784]: osdmap e371: 8 total, 8 up, 8 in 2024-09-21T10:28:57.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:57.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:57.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:57.790 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:57.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:57.833 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:57.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:57.934 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:57.966 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:57.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:57.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:58.067 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:58.095 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:58.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:58.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:58.197 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:58.225 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:58.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:58.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:58.326 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:58.347 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:58.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:58.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:58.449 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:58.476 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:58.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:58.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:58.535 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:58.577 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:58.609 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:58.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:58.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:58.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:58.751 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:58.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:58.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:58.853 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:58.884 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:58.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:58.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:58.986 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:59.012 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:59.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:59.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:59.114 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:59.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:59.142 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:59.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:59.243 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:59.268 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:59.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:59.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:59.301 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:28:59.370 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:59.395 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:59.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:59.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:59.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:59.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:59.547 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:59.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:59.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:28:59 smithi098 ceph-mon[97580]: pgmap v1262: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 498 op/s; 15031/300540 objects degraded (5.001%); 17892/300540 objects misplaced (5.953%); 496 KiB/s, 7 objects/s recovering 2024-09-21T10:28:59.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:28:59 smithi098 ceph-mon[100914]: pgmap v1262: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 498 op/s; 15031/300540 objects degraded (5.001%); 17892/300540 objects misplaced (5.953%); 496 KiB/s, 7 objects/s recovering 2024-09-21T10:28:59.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:28:59 smithi196 ceph-mon[84784]: pgmap v1262: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 498 op/s; 15031/300540 objects degraded (5.001%); 17892/300540 objects misplaced (5.953%); 496 KiB/s, 7 objects/s recovering 2024-09-21T10:28:59.649 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:59.672 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:28:59.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:59.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:59.692 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T10:28:59.774 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:59.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:59.842 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:59.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:59.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:59.990 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:28:59.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:59.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:00.092 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:00.122 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:00.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:00.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:00.224 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:00.266 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:00.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:00.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:00.367 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:00.401 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:00.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:00.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:00.502 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:00.530 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:00.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:00.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:00.632 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:00.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:29:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:29:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:29:00.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:00.673 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:00.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:00.774 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:00.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:00.813 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:00.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:00.913 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:00.951 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:00.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:00.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:00.997 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:29:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:29:00] "GET /metrics HTTP/1.1" 200 36614 "" "Prometheus/2.51.0" 2024-09-21T10:29:01.053 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:01.096 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:01.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:01.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:01.198 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:01.223 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:01.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:01.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:01.325 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:01.360 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:01.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:01.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:01.461 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:01.494 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:01.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:01.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:01.596 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:01 smithi098 ceph-mon[97580]: pgmap v1263: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 765 op/s; 15168/292824 objects degraded (5.180%); 17317/292824 objects misplaced (5.914%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:29:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:01 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 15168/292824 objects degraded (5.180%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:01.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:01 smithi098 ceph-mon[100914]: pgmap v1263: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 765 op/s; 15168/292824 objects degraded (5.180%); 17317/292824 objects misplaced (5.914%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:29:01.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:01 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 15168/292824 objects degraded (5.180%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:01.633 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:01.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:01.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:01.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:01 smithi196 ceph-mon[84784]: pgmap v1263: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 765 op/s; 15168/292824 objects degraded (5.180%); 17317/292824 objects misplaced (5.914%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:29:01.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:01 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 15168/292824 objects degraded (5.180%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:01.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:01.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:01.803 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:01.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:01.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:01.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:01.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:02.062 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:02.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:02.124 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:02.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:02.225 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:02.250 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:02.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:02.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:02.352 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:02.384 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:02.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:02.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:02.486 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:02.525 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:02.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:02.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:02.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:02.661 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:02.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:02.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:02.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:02.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:02.799 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:02.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:02.899 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:02.922 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:02.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:02.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:03.024 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:03.050 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:03.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:03.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:03.151 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:03.190 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:03.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:03.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:03.292 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:03.317 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:03.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:03.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:03.378 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:29:03.378 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:29:03.418 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:03.452 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:03.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:03.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:03.554 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:03.589 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:03.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:03.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:03.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:03 smithi098 ceph-mon[97580]: pgmap v1264: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 508 op/s; 15168/291573 objects degraded (5.202%); 17317/291573 objects misplaced (5.939%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:29:03.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:03 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/479211366' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:03.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:03 smithi098 ceph-mon[100914]: pgmap v1264: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 508 op/s; 15168/291573 objects degraded (5.202%); 17317/291573 objects misplaced (5.939%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:29:03.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:03 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/479211366' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:03.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:03 smithi196 ceph-mon[84784]: pgmap v1264: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 508 op/s; 15168/291573 objects degraded (5.202%); 17317/291573 objects misplaced (5.939%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:29:03.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:03 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/479211366' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:03.691 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:03.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:03.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:03.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:03.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:03.855 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:03.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:03.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:03.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:03.985 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:03.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:03.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:04.087 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:04.127 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:04.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:04.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:04.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:04.261 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:04.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:04.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:04.363 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:04.396 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:04.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:04.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:04.498 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:04.521 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:04.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:04.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:04.623 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:04.648 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:29:04.649 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:29:04.649 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:29:04.649 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:29:04.649 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:29:04.649 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:29:04.649 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:29:04.649 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:29:04.649 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:29:04.649 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:29:04.649 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:29:04.649 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:29:04.650 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:29:04.650 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:29:04.650 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:29:04.650 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:29:04.657 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:04.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:04.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:04.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:04.799 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:04.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:04.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:04.862 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:04.866 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:04.902 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:04.926 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:04.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:04.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:05.028 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:05.065 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:05.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:05.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:05.167 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:05.194 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:05.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:05.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:05.295 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:05.331 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:05.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:05.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:05.433 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:05.497 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:05.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:05.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:05.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:05 smithi098 ceph-mon[97580]: pgmap v1265: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 15168/284214 objects degraded (5.337%); 16857/284214 objects misplaced (5.931%); 915 KiB/s, 14 objects/s recovering 2024-09-21T10:29:05.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:05 smithi098 ceph-mon[100914]: pgmap v1265: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 15168/284214 objects degraded (5.337%); 16857/284214 objects misplaced (5.931%); 915 KiB/s, 14 objects/s recovering 2024-09-21T10:29:05.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:05.628 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:05.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:05.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:05.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:05 smithi196 ceph-mon[84784]: pgmap v1265: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 15168/284214 objects degraded (5.337%); 16857/284214 objects misplaced (5.931%); 915 KiB/s, 14 objects/s recovering 2024-09-21T10:29:05.730 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:05.762 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:05.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:05.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:05.863 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:05.905 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:05.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:05.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:05.955 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:29:06.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:06.072 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:06.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:06.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:06.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:06.245 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:06.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:06.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:06.347 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:06.395 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:06.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:06.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:06.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:06 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 15168/284214 objects degraded (5.337%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:06.449 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:06 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 15168/284214 objects degraded (5.337%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:06.497 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:06.525 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:06.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:06.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:06.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:06.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:06 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 15168/284214 objects degraded (5.337%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:06.659 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:06.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:06.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:06.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:06.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:06.794 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:06.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:06.895 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:06.923 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:06.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:06.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:07.024 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:07.052 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:07.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:07.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:07.153 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:07.180 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:07.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:07.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:07.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:07.316 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:07.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:07.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:07.418 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:07.443 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:29:07.446 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:07.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:07.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:07.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:07.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:07.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:07.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:07 smithi196 ceph-mon[84784]: pgmap v1266: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 668 op/s; 15299/281094 objects degraded (5.443%); 16653/281094 objects misplaced (5.924%); 985 KiB/s, 15 objects/s recovering 2024-09-21T10:29:07.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:07.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:07.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:07.719 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:07 smithi098 ceph-mon[97580]: pgmap v1266: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 668 op/s; 15299/281094 objects degraded (5.443%); 16653/281094 objects misplaced (5.924%); 985 KiB/s, 15 objects/s recovering 2024-09-21T10:29:07.719 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:07.719 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:07.719 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:07 smithi098 ceph-mon[100914]: pgmap v1266: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 668 op/s; 15299/281094 objects degraded (5.443%); 16653/281094 objects misplaced (5.924%); 985 KiB/s, 15 objects/s recovering 2024-09-21T10:29:07.719 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:07.719 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:07.773 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:07.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:07.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:07.875 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:07.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:07.912 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:07.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:08.013 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:08.043 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:08.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:08.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:08.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:08.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:08.188 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:08.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:08.289 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:08.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:08.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:08.431 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:08.471 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:08.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:08.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:08.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:08.607 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:08.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:08.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:08.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:08.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:08.749 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:08.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:08.850 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:08.877 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:08.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:08.978 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:09.003 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:09.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:09.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:09.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:09.149 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:09.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:09.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:09.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:09.275 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:09.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:09.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:09.365 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T10:29:09.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:09.429 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:09.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:09.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:09.531 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:09.572 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:09.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:09.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:09.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:09 smithi196 ceph-mon[84784]: pgmap v1267: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 612 op/s; 15299/280434 objects degraded (5.455%); 16653/280434 objects misplaced (5.938%); 874 KiB/s, 13 objects/s recovering 2024-09-21T10:29:09.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:09 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/783115103' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:29:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:09 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:29:09.673 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:09.698 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:09.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:09.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:09.800 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:09.831 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:09.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:09.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:09.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:09 smithi098 ceph-mon[97580]: pgmap v1267: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 612 op/s; 15299/280434 objects degraded (5.455%); 16653/280434 objects misplaced (5.938%); 874 KiB/s, 13 objects/s recovering 2024-09-21T10:29:09.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:09 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/783115103' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:29:09.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:09 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:29:09.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:09 smithi098 ceph-mon[100914]: pgmap v1267: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 612 op/s; 15299/280434 objects degraded (5.455%); 16653/280434 objects misplaced (5.938%); 874 KiB/s, 13 objects/s recovering 2024-09-21T10:29:09.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:09 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/783115103' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:29:09.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:09 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:29:09.932 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:09.959 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:09.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:09.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:10.061 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:10.090 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:10.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:10.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:10.103 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:10.192 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:10.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:10.219 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:10.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:10.320 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:10.351 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:10.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:10.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:10.453 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:10.491 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:10.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:10.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:10.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:10.628 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:10.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:10.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:10.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:10 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:29:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:10 smithi196 ceph-mon[84784]: osdmap e372: 8 total, 8 up, 8 in 2024-09-21T10:29:10.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:29:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:29:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:29:10.658 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:10 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:29:10.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:10 smithi098 ceph-mon[97580]: osdmap e372: 8 total, 8 up, 8 in 2024-09-21T10:29:10.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:10 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:29:10.659 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:10 smithi098 ceph-mon[100914]: osdmap e372: 8 total, 8 up, 8 in 2024-09-21T10:29:10.729 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:10.770 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:10.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:10.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:10.872 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:10.895 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:10.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:10.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:10.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T10:29:10.946 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:29:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:29:10] "GET /metrics HTTP/1.1" 200 36612 "" "Prometheus/2.51.0" 2024-09-21T10:29:10.996 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:11.028 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:11.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:11.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:11.130 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:11.153 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:11.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:11.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:11.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:11.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:11.292 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:11.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:11.393 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:11.420 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:11.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:11.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:11.522 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:11.570 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:11.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:11.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:11.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:11 smithi098 ceph-mon[97580]: pgmap v1269: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 15444/271410 objects degraded (5.690%); 15998/271410 objects misplaced (5.894%); 902 KiB/s, 14 objects/s recovering 2024-09-21T10:29:11.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:11 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 15444/271410 objects degraded (5.690%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:11.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:11 smithi098 ceph-mon[100914]: pgmap v1269: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 15444/271410 objects degraded (5.690%); 15998/271410 objects misplaced (5.894%); 902 KiB/s, 14 objects/s recovering 2024-09-21T10:29:11.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:11 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 15444/271410 objects degraded (5.690%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:11.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:11 smithi196 ceph-mon[84784]: pgmap v1269: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 15444/271410 objects degraded (5.690%); 15998/271410 objects misplaced (5.894%); 902 KiB/s, 14 objects/s recovering 2024-09-21T10:29:11.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:11 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 15444/271410 objects degraded (5.690%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:11.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:11.697 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:11.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:11.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:11.798 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:11.849 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:11.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:11.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:11.951 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:11.985 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:11.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:11.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:12.086 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:12.111 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:12.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:12.213 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:12.241 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:12.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:12.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:12.343 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:12.382 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:12.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:12.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:12.399 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:29:12.428 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:12 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3466942897' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:29:12.428 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:12 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3466942897' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:29:12.483 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:12.515 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:12.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:12.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:12.617 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:12.646 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:12.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:12.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:12.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:12.772 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:12.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:12.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:12.794 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:12.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:12 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3466942897' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:29:12.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:12.929 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:12.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:13.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:13.068 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:13.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:13.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:13.170 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:13.213 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:13.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:13.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:13.315 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:13.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:13.346 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:13.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:13.447 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:13.473 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:13.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:13.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:13.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:13.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:13.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:13.651 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:13 smithi098 ceph-mon[97580]: pgmap v1270: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 15444/270153 objects degraded (5.717%); 15998/270153 objects misplaced (5.922%); 902 KiB/s, 14 objects/s recovering 2024-09-21T10:29:13.651 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:13 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:29:13.652 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:13 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3466942897' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:29:13.652 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:13 smithi098 ceph-mon[97580]: osdmap e373: 8 total, 8 up, 8 in 2024-09-21T10:29:13.652 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:13 smithi098 ceph-mon[100914]: pgmap v1270: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 15444/270153 objects degraded (5.717%); 15998/270153 objects misplaced (5.922%); 902 KiB/s, 14 objects/s recovering 2024-09-21T10:29:13.652 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:13 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:29:13.652 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:13 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3466942897' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:29:13.652 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:13 smithi098 ceph-mon[100914]: osdmap e373: 8 total, 8 up, 8 in 2024-09-21T10:29:13.719 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:13.748 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:13.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:13.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:13.850 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:13.879 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:13.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:13.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:13 smithi196 ceph-mon[84784]: pgmap v1270: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 15444/270153 objects degraded (5.717%); 15998/270153 objects misplaced (5.922%); 902 KiB/s, 14 objects/s recovering 2024-09-21T10:29:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:13 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:29:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:13 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3466942897' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:29:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:13 smithi196 ceph-mon[84784]: osdmap e373: 8 total, 8 up, 8 in 2024-09-21T10:29:13.981 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:14.017 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:14.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:14.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:14.118 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:14.147 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:14.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:14.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:14.237 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:14.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:14.314 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:14.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:14.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:14.416 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:14.461 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:14.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:14.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:14.563 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:14.589 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:14.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:14.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:14.690 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:14.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:14.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:14.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:14.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:14.854 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:14.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:14.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:14.956 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:14.989 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:14.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:14.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:15.091 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:15.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:15.131 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:15.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:15.232 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:15.255 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:15.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:15.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:15.357 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:15.384 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:15.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:15.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:15.407 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:29:15.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:15 smithi098 ceph-mon[97580]: pgmap v1272: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 742 op/s; 15444/263280 objects degraded (5.866%); 15629/263280 objects misplaced (5.936%); 544 KiB/s, 8 objects/s recovering 2024-09-21T10:29:15.431 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:15 smithi098 ceph-mon[100914]: pgmap v1272: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 742 op/s; 15444/263280 objects degraded (5.866%); 15629/263280 objects misplaced (5.936%); 544 KiB/s, 8 objects/s recovering 2024-09-21T10:29:15.485 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:15.513 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:15.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:15.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:15.604 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:29:15.605 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-21T10:29:15.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-21T10:29:15.605 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:29:15.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.605 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:115317 10.4 e364/361 10:2b512cea:::benchmark_data_smithi098_281896_object115316:head v 364'7119, mlcod=353'23)", 2024-09-21T10:29:15.605 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:03.471259+0000", 2024-09-21T10:29:15.605 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 72.127929487000003, 2024-09-21T10:29:15.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.099525844000000002, 2024-09-21T10:29:15.605 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:29:15.605 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:29:15.605 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:15.605 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:29:15.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:29:15.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:03.471259+0000", 2024-09-21T10:29:15.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.606 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:29:15.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:03.471259+0000", 2024-09-21T10:29:15.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.606 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:29:15.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:03.471260+0000", 2024-09-21T10:29:15.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1179999999999999e-06 2024-09-21T10:29:15.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.607 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:29:15.607 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:03.471356+0000", 2024-09-21T10:29:15.607 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5717000000000005e-05 2024-09-21T10:29:15.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.607 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:29:15.607 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:03.471376+0000", 2024-09-21T10:29:15.607 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0194e-05 2024-09-21T10:29:15.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.607 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:29:15.607 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:03.471402+0000", 2024-09-21T10:29:15.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5856000000000001e-05 2024-09-21T10:29:15.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.608 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:29:15.608 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:03.481652+0000", 2024-09-21T10:29:15.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010249945999999999 2024-09-21T10:29:15.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.608 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:29:15.608 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:03.481708+0000", 2024-09-21T10:29:15.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6079999999999998e-05 2024-09-21T10:29:15.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:29:15.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:03.570697+0000", 2024-09-21T10:29:15.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.088988810000000002 2024-09-21T10:29:15.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:29:15.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:03.570785+0000", 2024-09-21T10:29:15.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8122999999999996e-05 2024-09-21T10:29:15.609 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.609 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:29:15.609 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.610 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:115323 10.4 e364/361 10:24d6bf5b:::benchmark_data_smithi098_281896_object115322:head v 364'7120, mlcod=353'23)", 2024-09-21T10:29:15.610 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:03.479541+0000", 2024-09-21T10:29:15.610 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 72.119646986999996, 2024-09-21T10:29:15.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.10341998099999999, 2024-09-21T10:29:15.610 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:29:15.610 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:29:15.610 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:15.610 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:29:15.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.610 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:29:15.610 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:03.479541+0000", 2024-09-21T10:29:15.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.611 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:29:15.611 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:03.479541+0000", 2024-09-21T10:29:15.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.611 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:29:15.611 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:03.479543+0000", 2024-09-21T10:29:15.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.099e-06 2024-09-21T10:29:15.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:29:15.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:03.479565+0000", 2024-09-21T10:29:15.612 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2365e-05 2024-09-21T10:29:15.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:29:15.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:03.479581+0000", 2024-09-21T10:29:15.612 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6192999999999999e-05 2024-09-21T10:29:15.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:29:15.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:03.479588+0000", 2024-09-21T10:29:15.612 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.34e-06 2024-09-21T10:29:15.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.613 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:29:15.613 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:03.491749+0000", 2024-09-21T10:29:15.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012160851 2024-09-21T10:29:15.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.613 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:29:15.613 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:03.491769+0000", 2024-09-21T10:29:15.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9884000000000001e-05 2024-09-21T10:29:15.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.613 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:29:15.613 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:03.582784+0000", 2024-09-21T10:29:15.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.091014387000000002 2024-09-21T10:29:15.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.614 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:29:15.614 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:03.582961+0000", 2024-09-21T10:29:15.614 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017786200000000001 2024-09-21T10:29:15.614 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.614 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:29:15.614 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.614 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:136218 10.3 e367/361 10:cb407e02:::benchmark_data_smithi098_281896_object11193:head v 367'8572, mlcod=0'0)", 2024-09-21T10:29:15.614 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:33.841214+0000", 2024-09-21T10:29:15.614 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.757974851999997, 2024-09-21T10:29:15.614 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.117310699, 2024-09-21T10:29:15.614 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:29:15.614 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:29:15.614 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:15.615 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:29:15.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.615 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:29:15.615 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.841214+0000", 2024-09-21T10:29:15.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.615 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:29:15.615 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.841214+0000", 2024-09-21T10:29:15.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.615 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:29:15.615 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.841214+0000", 2024-09-21T10:29:15.616 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7599999999999998e-07 2024-09-21T10:29:15.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.616 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:29:15.616 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.841218+0000", 2024-09-21T10:29:15.616 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0489999999999997e-06 2024-09-21T10:29:15.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.616 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:29:15.616 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.841224+0000", 2024-09-21T10:29:15.616 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8089999999999998e-06 2024-09-21T10:29:15.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.616 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:29:15.616 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.841228+0000", 2024-09-21T10:29:15.616 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0679999999999998e-06 2024-09-21T10:29:15.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.617 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:29:15.617 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.846410+0000", 2024-09-21T10:29:15.617 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051826859999999997 2024-09-21T10:29:15.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.617 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:29:15.617 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.846439+0000", 2024-09-21T10:29:15.617 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8826000000000001e-05 2024-09-21T10:29:15.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.617 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:29:15.617 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.958364+0000", 2024-09-21T10:29:15.617 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11192506100000001 2024-09-21T10:29:15.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.618 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:29:15.618 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.958524+0000", 2024-09-21T10:29:15.618 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015992400000000001 2024-09-21T10:29:15.618 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.618 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:29:15.618 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.618 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:136227 10.3 e367/361 10:c3791142:::benchmark_data_smithi098_281896_object11202:head v 367'8573, mlcod=0'0)", 2024-09-21T10:29:15.618 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:33.844316+0000", 2024-09-21T10:29:15.618 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.754872783000003, 2024-09-21T10:29:15.618 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11434214700000001, 2024-09-21T10:29:15.619 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:29:15.619 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:29:15.619 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:15.619 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:29:15.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:29:15.619 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.844316+0000", 2024-09-21T10:29:15.619 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:29:15.619 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.844316+0000", 2024-09-21T10:29:15.619 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.620 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:29:15.620 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.844317+0000", 2024-09-21T10:29:15.620 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.054e-06 2024-09-21T10:29:15.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.620 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:29:15.620 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.844323+0000", 2024-09-21T10:29:15.620 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6800000000000004e-06 2024-09-21T10:29:15.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.620 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:29:15.620 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.844337+0000", 2024-09-21T10:29:15.621 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3157e-05 2024-09-21T10:29:15.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.621 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:29:15.621 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.844343+0000", 2024-09-21T10:29:15.621 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8719999999999996e-06 2024-09-21T10:29:15.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.621 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:29:15.621 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.857006+0000", 2024-09-21T10:29:15.621 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012662229000000001 2024-09-21T10:29:15.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.621 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:29:15.621 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.857031+0000", 2024-09-21T10:29:15.622 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5826999999999999e-05 2024-09-21T10:29:15.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.622 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:29:15.622 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.958529+0000", 2024-09-21T10:29:15.622 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.101497063 2024-09-21T10:29:15.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.622 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:29:15.622 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.958658+0000", 2024-09-21T10:29:15.622 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012926499999999999 2024-09-21T10:29:15.622 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.622 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:29:15.622 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.623 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:136230 10.3 e367/361 10:c8f3e9dd:::benchmark_data_smithi098_281896_object11205:head v 367'8574, mlcod=0'0)", 2024-09-21T10:29:15.623 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:33.849139+0000", 2024-09-21T10:29:15.623 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.750049435999998, 2024-09-21T10:29:15.623 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.109555916, 2024-09-21T10:29:15.623 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:29:15.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:29:15.623 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:15.623 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:29:15.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.623 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:29:15.623 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.849139+0000", 2024-09-21T10:29:15.623 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.624 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:29:15.624 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.849139+0000", 2024-09-21T10:29:15.624 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.624 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:29:15.624 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.849141+0000", 2024-09-21T10:29:15.624 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6899999999999999e-06 2024-09-21T10:29:15.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.624 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:29:15.624 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.849150+0000", 2024-09-21T10:29:15.625 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4909999999999994e-06 2024-09-21T10:29:15.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.625 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:29:15.625 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.849165+0000", 2024-09-21T10:29:15.625 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4541e-05 2024-09-21T10:29:15.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.625 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:29:15.625 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.849174+0000", 2024-09-21T10:29:15.625 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8580000000000006e-06 2024-09-21T10:29:15.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.626 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:29:15.626 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.867122+0000", 2024-09-21T10:29:15.626 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017948583000000001 2024-09-21T10:29:15.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.626 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:29:15.626 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.867151+0000", 2024-09-21T10:29:15.626 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8387000000000001e-05 2024-09-21T10:29:15.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.626 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:29:15.626 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.958663+0000", 2024-09-21T10:29:15.626 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.091512074999999998 2024-09-21T10:29:15.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.627 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:29:15.627 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.958695+0000", 2024-09-21T10:29:15.627 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2291e-05 2024-09-21T10:29:15.627 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.627 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:29:15.627 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.627 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:136231 10.3 e367/361 10:c2d96113:::benchmark_data_smithi098_281896_object11206:head v 367'8575, mlcod=0'0)", 2024-09-21T10:29:15.627 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:33.855084+0000", 2024-09-21T10:29:15.627 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.744104116999999, 2024-09-21T10:29:15.627 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.103637456, 2024-09-21T10:29:15.627 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:29:15.628 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:29:15.628 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:15.628 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:29:15.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.628 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:29:15.628 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.855084+0000", 2024-09-21T10:29:15.628 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.628 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:29:15.628 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.855084+0000", 2024-09-21T10:29:15.628 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.628 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:29:15.629 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.855086+0000", 2024-09-21T10:29:15.629 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5179999999999999e-06 2024-09-21T10:29:15.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.629 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:29:15.629 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.855095+0000", 2024-09-21T10:29:15.629 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2539999999999998e-06 2024-09-21T10:29:15.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.629 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:29:15.629 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.855116+0000", 2024-09-21T10:29:15.629 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0517e-05 2024-09-21T10:29:15.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.629 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:29:15.630 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.855125+0000", 2024-09-21T10:29:15.630 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9970000000000004e-06 2024-09-21T10:29:15.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.630 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:29:15.630 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.877257+0000", 2024-09-21T10:29:15.630 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.022131904000000001 2024-09-21T10:29:15.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.630 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:29:15.630 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.877319+0000", 2024-09-21T10:29:15.630 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2759e-05 2024-09-21T10:29:15.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.631 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:29:15.631 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.958697+0000", 2024-09-21T10:29:15.631 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.081377737000000006 2024-09-21T10:29:15.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.631 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:29:15.631 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:33.958722+0000", 2024-09-21T10:29:15.631 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4769999999999998e-05 2024-09-21T10:29:15.631 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.631 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:29:15.631 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.631 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:137477 10.4 e367/361 10:2cc2c4bf:::benchmark_data_smithi098_281896_object12452:head v 367'8490, mlcod=353'122)", 2024-09-21T10:29:15.631 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:35.610035+0000", 2024-09-21T10:29:15.632 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 39.989153803999997, 2024-09-21T10:29:15.632 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.10383943399999999, 2024-09-21T10:29:15.632 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:29:15.632 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:29:15.632 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:15.632 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:29:15.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.632 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:29:15.632 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:35.610035+0000", 2024-09-21T10:29:15.632 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.632 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:15.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.634 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:29:15.634 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:35.610035+0000", 2024-09-21T10:29:15.634 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.634 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:29:15.634 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:35.610036+0000", 2024-09-21T10:29:15.634 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.017e-06 2024-09-21T10:29:15.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.634 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:29:15.634 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:35.610042+0000", 2024-09-21T10:29:15.635 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5270000000000004e-06 2024-09-21T10:29:15.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.635 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:29:15.635 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:35.610058+0000", 2024-09-21T10:29:15.635 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5308999999999999e-05 2024-09-21T10:29:15.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.635 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:29:15.635 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:35.610065+0000", 2024-09-21T10:29:15.635 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2760000000000003e-06 2024-09-21T10:29:15.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.635 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:29:15.636 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:35.631577+0000", 2024-09-21T10:29:15.636 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.021511747000000001 2024-09-21T10:29:15.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.636 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:29:15.636 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:35.631603+0000", 2024-09-21T10:29:15.636 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6951e-05 2024-09-21T10:29:15.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.636 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:29:15.636 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:35.713812+0000", 2024-09-21T10:29:15.636 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.082208332999999995 2024-09-21T10:29:15.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.636 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:29:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:35.713874+0000", 2024-09-21T10:29:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2273999999999994e-05 2024-09-21T10:29:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:29:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:139644 10.5 e367/361 10:a3c9d25d:::benchmark_data_smithi098_281896_object14619:head v 367'8624, mlcod=0'0)", 2024-09-21T10:29:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:38.653543+0000", 2024-09-21T10:29:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.945645915999997, 2024-09-21T10:29:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.10160860100000001, 2024-09-21T10:29:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:29:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:29:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:29:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:29:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:38.653543+0000", 2024-09-21T10:29:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:29:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:38.653543+0000", 2024-09-21T10:29:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:29:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:38.653543+0000", 2024-09-21T10:29:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8299999999999998e-07 2024-09-21T10:29:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:29:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:38.653547+0000", 2024-09-21T10:29:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3939999999999998e-06 2024-09-21T10:29:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:29:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:38.653558+0000", 2024-09-21T10:29:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0376e-05 2024-09-21T10:29:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:29:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:38.653563+0000", 2024-09-21T10:29:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2209999999999996e-06 2024-09-21T10:29:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:29:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:38.671463+0000", 2024-09-21T10:29:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017899750999999998 2024-09-21T10:29:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:29:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:38.671488+0000", 2024-09-21T10:29:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5493000000000001e-05 2024-09-21T10:29:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:29:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:38.755102+0000", 2024-09-21T10:29:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.083613913999999998 2024-09-21T10:29:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:29:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:38.755151+0000", 2024-09-21T10:29:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9069000000000001e-05 2024-09-21T10:29:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:29:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:142538 10.3 e368/361 10:c030bcc5:::benchmark_data_smithi098_281896_object17513:head v 368'8949, mlcod=0'0)", 2024-09-21T10:29:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:43.057126+0000", 2024-09-21T10:29:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.542062784999999, 2024-09-21T10:29:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.119454185, 2024-09-21T10:29:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:29:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:29:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:29:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:29:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.057126+0000", 2024-09-21T10:29:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:29:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.057126+0000", 2024-09-21T10:29:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:29:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.057127+0000", 2024-09-21T10:29:15.643 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.449e-06 2024-09-21T10:29:15.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.643 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:29:15.643 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.057134+0000", 2024-09-21T10:29:15.643 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9879999999999998e-06 2024-09-21T10:29:15.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.643 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:29:15.643 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.057151+0000", 2024-09-21T10:29:15.643 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6405999999999998e-05 2024-09-21T10:29:15.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.643 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:29:15.643 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.057158+0000", 2024-09-21T10:29:15.643 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8939999999999993e-06 2024-09-21T10:29:15.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.644 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:29:15.644 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.103993+0000", 2024-09-21T10:29:15.644 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.046834703999999998 2024-09-21T10:29:15.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.644 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:29:15.644 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.104018+0000", 2024-09-21T10:29:15.644 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4436000000000001e-05 2024-09-21T10:29:15.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.644 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:29:15.644 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.176526+0000", 2024-09-21T10:29:15.644 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.072508609000000002 2024-09-21T10:29:15.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.645 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:29:15.645 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.176580+0000", 2024-09-21T10:29:15.645 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3699000000000002e-05 2024-09-21T10:29:15.645 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.645 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:29:15.645 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.645 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:142540 10.3 e368/361 10:cf7a15cb:::benchmark_data_smithi098_281896_object17515:head v 368'8950, mlcod=0'0)", 2024-09-21T10:29:15.645 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:43.061008+0000", 2024-09-21T10:29:15.645 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.538180437000001, 2024-09-21T10:29:15.645 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11559902499999999, 2024-09-21T10:29:15.645 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:29:15.645 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:29:15.646 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:15.646 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:29:15.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.646 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:29:15.646 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.061008+0000", 2024-09-21T10:29:15.646 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.646 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:29:15.646 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.061008+0000", 2024-09-21T10:29:15.646 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.646 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:29:15.646 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.061009+0000", 2024-09-21T10:29:15.647 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.248e-06 2024-09-21T10:29:15.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.647 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:29:15.647 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.061017+0000", 2024-09-21T10:29:15.647 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7910000000000005e-06 2024-09-21T10:29:15.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.647 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:29:15.647 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.061069+0000", 2024-09-21T10:29:15.647 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1978000000000003e-05 2024-09-21T10:29:15.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.647 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:29:15.648 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.061076+0000", 2024-09-21T10:29:15.648 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2520000000000002e-06 2024-09-21T10:29:15.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.648 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:29:15.648 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.117079+0000", 2024-09-21T10:29:15.648 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.056002244999999999 2024-09-21T10:29:15.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.648 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:29:15.648 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.117104+0000", 2024-09-21T10:29:15.648 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5248e-05 2024-09-21T10:29:15.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.649 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:29:15.649 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.176584+0000", 2024-09-21T10:29:15.649 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.059480102999999999 2024-09-21T10:29:15.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.649 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:29:15.649 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.176607+0000", 2024-09-21T10:29:15.649 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3159999999999998e-05 2024-09-21T10:29:15.649 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.649 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:29:15.649 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.649 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:142566 10.3 e368/361 10:cf439dc4:::benchmark_data_smithi098_281896_object17541:head v 368'8951, mlcod=0'0)", 2024-09-21T10:29:15.649 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:43.071682+0000", 2024-09-21T10:29:15.650 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.527506078999998, 2024-09-21T10:29:15.650 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.10495608100000001, 2024-09-21T10:29:15.650 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:29:15.650 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:29:15.650 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:15.650 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:29:15.650 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.650 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:29:15.650 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.071682+0000", 2024-09-21T10:29:15.650 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.650 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.650 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:29:15.651 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.071682+0000", 2024-09-21T10:29:15.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.651 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:29:15.651 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.071684+0000", 2024-09-21T10:29:15.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.218e-06 2024-09-21T10:29:15.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.651 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:29:15.651 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.071691+0000", 2024-09-21T10:29:15.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7100000000000007e-06 2024-09-21T10:29:15.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.652 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:29:15.652 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.071707+0000", 2024-09-21T10:29:15.652 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5270000000000001e-05 2024-09-21T10:29:15.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.652 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.652 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:29:15.652 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.071714+0000", 2024-09-21T10:29:15.652 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.951e-06 2024-09-21T10:29:15.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.652 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.652 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:29:15.652 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.127776+0000", 2024-09-21T10:29:15.652 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.056062887999999998 2024-09-21T10:29:15.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.653 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:29:15.653 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.127800+0000", 2024-09-21T10:29:15.653 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3728999999999999e-05 2024-09-21T10:29:15.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.653 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:29:15.653 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.176611+0000", 2024-09-21T10:29:15.653 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.048810680000000002 2024-09-21T10:29:15.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.653 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:29:15.653 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.176638+0000", 2024-09-21T10:29:15.653 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7634999999999999e-05 2024-09-21T10:29:15.654 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.654 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:29:15.654 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.654 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:142567 10.3 e368/361 10:ced60e62:::benchmark_data_smithi098_281896_object17542:head v 368'8952, mlcod=0'0)", 2024-09-21T10:29:15.654 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:43.074933+0000", 2024-09-21T10:29:15.654 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.524255478000001, 2024-09-21T10:29:15.654 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.10172980500000001, 2024-09-21T10:29:15.654 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:29:15.654 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:29:15.654 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:15.654 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:29:15.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.654 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:29:15.655 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.074933+0000", 2024-09-21T10:29:15.655 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.655 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:29:15.655 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.074933+0000", 2024-09-21T10:29:15.655 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.655 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:29:15.655 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.074934+0000", 2024-09-21T10:29:15.655 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4249999999999999e-06 2024-09-21T10:29:15.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.656 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:29:15.656 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.074941+0000", 2024-09-21T10:29:15.656 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1739999999999997e-06 2024-09-21T10:29:15.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.656 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:29:15.656 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.074956+0000", 2024-09-21T10:29:15.656 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5699999999999999e-05 2024-09-21T10:29:15.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.656 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:29:15.656 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.074964+0000", 2024-09-21T10:29:15.656 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9270000000000005e-06 2024-09-21T10:29:15.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.657 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.657 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:29:15.657 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.141264+0000", 2024-09-21T10:29:15.657 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.066299318999999995 2024-09-21T10:29:15.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.657 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.657 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:29:15.657 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.141292+0000", 2024-09-21T10:29:15.657 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8422e-05 2024-09-21T10:29:15.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.657 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.657 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:29:15.657 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.176641+0000", 2024-09-21T10:29:15.657 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035348638000000002 2024-09-21T10:29:15.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.658 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:29:15.658 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:43.176663+0000", 2024-09-21T10:29:15.658 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2200000000000001e-05 2024-09-21T10:29:15.658 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.658 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:29:15.658 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.658 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:156599 10.3 e371/361 10:cd41b47e:::benchmark_data_smithi098_281896_object31574:head v 371'9872, mlcod=0'0)", 2024-09-21T10:29:15.658 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:03.709138+0000", 2024-09-21T10:29:15.658 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 11.890050048999999, 2024-09-21T10:29:15.658 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.098479381000000005, 2024-09-21T10:29:15.658 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:29:15.659 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:29:15.659 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:15.659 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:29:15.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.659 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:29:15.659 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:03.709138+0000", 2024-09-21T10:29:15.659 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.659 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:29:15.659 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:03.709138+0000", 2024-09-21T10:29:15.659 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.659 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:29:15.659 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:03.709140+0000", 2024-09-21T10:29:15.660 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.243e-06 2024-09-21T10:29:15.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.660 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:29:15.660 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:03.709148+0000", 2024-09-21T10:29:15.660 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4519999999999997e-06 2024-09-21T10:29:15.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.660 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:29:15.660 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:03.709165+0000", 2024-09-21T10:29:15.660 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6416999999999999e-05 2024-09-21T10:29:15.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.661 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:29:15.661 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:03.709173+0000", 2024-09-21T10:29:15.661 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5399999999999996e-06 2024-09-21T10:29:15.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.661 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:29:15.661 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:03.714051+0000", 2024-09-21T10:29:15.661 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048780869999999997 2024-09-21T10:29:15.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.661 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:29:15.661 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:03.714071+0000", 2024-09-21T10:29:15.661 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9732e-05 2024-09-21T10:29:15.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.662 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:29:15.662 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:03.807557+0000", 2024-09-21T10:29:15.662 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.093486458999999994 2024-09-21T10:29:15.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.662 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:29:15.662 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:03.807618+0000", 2024-09-21T10:29:15.662 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0451000000000001e-05 2024-09-21T10:29:15.662 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.662 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:29:15.662 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.662 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:160578 10.3 e371/361 10:c3ba3c86:::benchmark_data_smithi098_281896_object35553:head v 371'10111, mlcod=0'0)", 2024-09-21T10:29:15.663 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:09.321026+0000", 2024-09-21T10:29:15.663 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 6.2781627650000003, 2024-09-21T10:29:15.663 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.123462081, 2024-09-21T10:29:15.663 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:29:15.663 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:29:15.663 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:15.663 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:29:15.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.663 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:29:15.663 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.321026+0000", 2024-09-21T10:29:15.663 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.663 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:29:15.663 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.321026+0000", 2024-09-21T10:29:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:29:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.321027+0000", 2024-09-21T10:29:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.277e-06 2024-09-21T10:29:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:29:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.321035+0000", 2024-09-21T10:29:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2530000000000004e-06 2024-09-21T10:29:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:29:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.321047+0000", 2024-09-21T10:29:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1411e-05 2024-09-21T10:29:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:29:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.321054+0000", 2024-09-21T10:29:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7950000000000008e-06 2024-09-21T10:29:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:29:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.370908+0000", 2024-09-21T10:29:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.049853498000000003 2024-09-21T10:29:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:29:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.370938+0000", 2024-09-21T10:29:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0080999999999999e-05 2024-09-21T10:29:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:29:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.444303+0000", 2024-09-21T10:29:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.073365185999999999 2024-09-21T10:29:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:29:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.444488+0000", 2024-09-21T10:29:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018458 2024-09-21T10:29:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:29:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:160616 10.3 e371/361 10:cbe0f9f6:::benchmark_data_smithi098_281896_object35591:head v 371'10112, mlcod=0'0)", 2024-09-21T10:29:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:09.343890+0000", 2024-09-21T10:29:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 6.2552987729999998, 2024-09-21T10:29:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.100750015, 2024-09-21T10:29:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:29:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:29:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:29:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:29:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.343890+0000", 2024-09-21T10:29:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:29:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.343890+0000", 2024-09-21T10:29:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:29:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.343891+0000", 2024-09-21T10:29:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0839999999999999e-06 2024-09-21T10:29:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:29:15.669 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.343897+0000", 2024-09-21T10:29:15.669 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1680000000000001e-06 2024-09-21T10:29:15.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.669 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:29:15.669 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.343909+0000", 2024-09-21T10:29:15.669 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1793999999999999e-05 2024-09-21T10:29:15.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.669 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:29:15.669 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.343914+0000", 2024-09-21T10:29:15.669 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6779999999999997e-06 2024-09-21T10:29:15.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:29:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.393540+0000", 2024-09-21T10:29:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.049625781000000001 2024-09-21T10:29:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:29:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.393595+0000", 2024-09-21T10:29:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4954999999999998e-05 2024-09-21T10:29:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:29:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.444491+0000", 2024-09-21T10:29:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.05089631 2024-09-21T10:29:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:29:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.444640+0000", 2024-09-21T10:29:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000148245 2024-09-21T10:29:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:29:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:160626 10.3 e371/361 10:c0569e7d:::benchmark_data_smithi098_281896_object35601:head v 371'10113, mlcod=0'0)", 2024-09-21T10:29:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:09.367090+0000", 2024-09-21T10:29:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 6.2320986859999996, 2024-09-21T10:29:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.102492216, 2024-09-21T10:29:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:29:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:29:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:29:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:29:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.367090+0000", 2024-09-21T10:29:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:29:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.367090+0000", 2024-09-21T10:29:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:29:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.367091+0000", 2024-09-21T10:29:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.39e-06 2024-09-21T10:29:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:29:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.367115+0000", 2024-09-21T10:29:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3598999999999999e-05 2024-09-21T10:29:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:29:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.367135+0000", 2024-09-21T10:29:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0721999999999999e-05 2024-09-21T10:29:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.674 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:29:15.674 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.367146+0000", 2024-09-21T10:29:15.674 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0495e-05 2024-09-21T10:29:15.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.675 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:29:15.675 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.408134+0000", 2024-09-21T10:29:15.675 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.040988384000000003 2024-09-21T10:29:15.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.675 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:29:15.675 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.408206+0000", 2024-09-21T10:29:15.675 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1779999999999994e-05 2024-09-21T10:29:15.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.675 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:29:15.676 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.469448+0000", 2024-09-21T10:29:15.676 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.061241666 2024-09-21T10:29:15.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.676 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:29:15.676 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.469582+0000", 2024-09-21T10:29:15.676 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013417999999999999 2024-09-21T10:29:15.676 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.676 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:29:15.676 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.676 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:161950 10.5 e372/361 10:acb11375:::benchmark_data_smithi098_281896_object36925:head v 372'9922, mlcod=0'0)", 2024-09-21T10:29:15.676 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:11.358331+0000", 2024-09-21T10:29:15.676 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.2408572070000004, 2024-09-21T10:29:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.097909880000000005, 2024-09-21T10:29:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:29:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:29:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:29:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:29:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:11.358331+0000", 2024-09-21T10:29:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:29:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:11.358331+0000", 2024-09-21T10:29:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:29:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:11.358333+0000", 2024-09-21T10:29:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5600000000000001e-06 2024-09-21T10:29:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:29:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:11.358340+0000", 2024-09-21T10:29:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4109999999999999e-06 2024-09-21T10:29:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:29:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:11.358359+0000", 2024-09-21T10:29:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8811999999999999e-05 2024-09-21T10:29:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:29:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:11.358367+0000", 2024-09-21T10:29:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.96e-06 2024-09-21T10:29:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:29:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:11.358532+0000", 2024-09-21T10:29:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016483700000000001 2024-09-21T10:29:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:29:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:11.358559+0000", 2024-09-21T10:29:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6699999999999998e-05 2024-09-21T10:29:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:29:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:11.455508+0000", 2024-09-21T10:29:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.096949908000000001 2024-09-21T10:29:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:29:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:11.456241+0000", 2024-09-21T10:29:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00073269199999999996 2024-09-21T10:29:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:29:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:162030 10.5 e372/361 10:a0c61dff:::benchmark_data_smithi098_281896_object37005:head v 372'9928, mlcod=0'0)", 2024-09-21T10:29:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:11.489555+0000", 2024-09-21T10:29:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.1096333659999997, 2024-09-21T10:29:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.097177959999999994, 2024-09-21T10:29:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:29:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:29:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:29:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:29:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:11.489555+0000", 2024-09-21T10:29:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:29:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:11.489555+0000", 2024-09-21T10:29:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:29:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:11.489556+0000", 2024-09-21T10:29:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3909999999999999e-06 2024-09-21T10:29:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:29:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:11.489565+0000", 2024-09-21T10:29:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5690000000000005e-06 2024-09-21T10:29:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:29:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:11.489586+0000", 2024-09-21T10:29:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1129e-05 2024-09-21T10:29:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:29:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:11.489595+0000", 2024-09-21T10:29:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3130000000000004e-06 2024-09-21T10:29:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:29:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:11.489655+0000", 2024-09-21T10:29:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9630999999999999e-05 2024-09-21T10:29:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:29:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:11.489679+0000", 2024-09-21T10:29:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.338e-05 2024-09-21T10:29:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:29:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:11.586571+0000", 2024-09-21T10:29:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.096892193000000001 2024-09-21T10:29:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:29:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:11.586733+0000", 2024-09-21T10:29:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000162354 2024-09-21T10:29:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:29:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:163061 10.5 e373/361 10:af211453:::benchmark_data_smithi098_281896_object38036:head v 373'10001, mlcod=0'0)", 2024-09-21T10:29:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:13.037995+0000", 2024-09-21T10:29:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 2.5611933709999999, 2024-09-21T10:29:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.097771685999999997, 2024-09-21T10:29:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:29:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:29:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:29:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:29:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:13.037995+0000", 2024-09-21T10:29:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:29:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:13.037995+0000", 2024-09-21T10:29:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:29:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:13.037997+0000", 2024-09-21T10:29:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6899999999999999e-06 2024-09-21T10:29:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:29:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:13.038006+0000", 2024-09-21T10:29:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1810000000000007e-06 2024-09-21T10:29:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:29:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:13.038021+0000", 2024-09-21T10:29:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4854999999999999e-05 2024-09-21T10:29:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:29:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:13.038031+0000", 2024-09-21T10:29:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7710000000000001e-06 2024-09-21T10:29:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:29:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:13.052078+0000", 2024-09-21T10:29:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014047514000000001 2024-09-21T10:29:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:29:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:13.052147+0000", 2024-09-21T10:29:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8398999999999993e-05 2024-09-21T10:29:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:29:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:13.135718+0000", 2024-09-21T10:29:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.083571736999999993 2024-09-21T10:29:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:29:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:13.135767+0000", 2024-09-21T10:29:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8538999999999997e-05 2024-09-21T10:29:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:29:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:164623 10.3 e373/361 10:c80ce5b4:::benchmark_data_smithi098_281896_object39598:head v 373'10366, mlcod=0'0)", 2024-09-21T10:29:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:15.199778+0000", 2024-09-21T10:29:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.39941047499999999, 2024-09-21T10:29:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.086910987999999995, 2024-09-21T10:29:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:29:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:29:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:29:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:29:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:15.199778+0000", 2024-09-21T10:29:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:29:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:15.199778+0000", 2024-09-21T10:29:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:29:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:29:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:15.199780+0000", 2024-09-21T10:29:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.765e-06 2024-09-21T10:29:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:29:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:15.199789+0000", 2024-09-21T10:29:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5300000000000002e-06 2024-09-21T10:29:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:29:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:15.199803+0000", 2024-09-21T10:29:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3997e-05 2024-09-21T10:29:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:29:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:15.199812+0000", 2024-09-21T10:29:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2010000000000005e-06 2024-09-21T10:29:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:29:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:15.212185+0000", 2024-09-21T10:29:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012372808000000001 2024-09-21T10:29:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:29:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:15.212213+0000", 2024-09-21T10:29:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7847999999999998e-05 2024-09-21T10:29:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:29:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:15.286587+0000", 2024-09-21T10:29:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.074373394999999995 2024-09-21T10:29:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:29:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:29:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:29:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:15.286689+0000", 2024-09-21T10:29:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000102444 2024-09-21T10:29:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:29:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:29:15.694 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:29:15.694 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:29:15.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:15.694 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:15.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:15.795 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:15.824 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:15.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:15.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:15.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:15 smithi196 ceph-mon[84784]: pgmap v1272: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 742 op/s; 15444/263280 objects degraded (5.866%); 15629/263280 objects misplaced (5.936%); 544 KiB/s, 8 objects/s recovering 2024-09-21T10:29:15.926 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:15.970 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:15.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:15.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:16.072 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:16.093 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:16.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:16.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:16.195 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:16.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:16.223 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:16.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:16.324 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:16.358 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:16.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:16.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:16.460 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:16.504 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:16.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:16.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:16.606 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:16.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:16.644 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:16.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:16.745 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:16.785 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:16.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:16.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:16 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 15444/263280 objects degraded (5.866%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:16.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:16 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 15444/263280 objects degraded (5.866%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:16.887 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:16 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 15444/263280 objects degraded (5.866%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:16.913 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:16.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:16.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:17.015 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:17.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:17.046 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:17.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:17.147 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:17.200 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:17.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:17.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:17.301 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:17.326 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:17.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:17.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:17.331 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:29:17.331 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:29:17.365 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:29:17.365 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:29:17.405 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:29:17.405 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:29:17.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:17.454 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:17.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:17.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:17.455 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:29:17.455 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:29:17.492 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:29:17.492 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:29:17.524 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:29:17.524 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:29:17.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:17.556 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:29:17.557 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:29:17.581 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:17.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:17.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:17.589 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:29:17.589 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:29:17.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:17 smithi098 ceph-mon[97580]: pgmap v1273: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 848 op/s; 15577/260070 objects degraded (5.990%); 15426/260070 objects misplaced (5.931%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:29:17.590 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:17 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/979941593' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:17.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:17 smithi098 ceph-mon[100914]: pgmap v1273: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 848 op/s; 15577/260070 objects degraded (5.990%); 15426/260070 objects misplaced (5.931%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:29:17.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:17 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/979941593' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:17.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:17.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:17.724 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:17.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:17.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:17.861 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:17.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:17.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:17.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:17 smithi196 ceph-mon[84784]: pgmap v1273: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 848 op/s; 15577/260070 objects degraded (5.990%); 15426/260070 objects misplaced (5.931%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:29:17.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:17 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/979941593' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:17.963 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:17.994 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:17.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:17.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:18.096 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:18.123 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:18.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:18.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:18.224 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:18.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:18.261 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:18.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:18.362 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:18.394 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:18.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:18.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:18.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:18.532 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:18.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:18.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:18.633 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:18.664 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:18.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:18.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:18.765 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:18.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:18.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:18.911 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:18.955 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:18.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:18.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:19.057 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:19.086 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:19.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:19.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:19.188 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:19.214 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:19.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:19.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:19.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:19.353 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:19.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:19.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:19.432 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:19.449 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T10:29:19.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:19.486 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:29:19.519 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:19.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:19.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:19.621 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:19.660 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:19.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:19.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:19.758 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:19 smithi098 ceph-mon[97580]: pgmap v1274: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 486 op/s; 15577/259395 objects degraded (6.005%); 15426/259395 objects misplaced (5.947%); 544 KiB/s, 8 objects/s recovering 2024-09-21T10:29:19.758 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:19 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/904572438' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:29:19.758 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:19 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:29:19.758 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:19 smithi098 ceph-mon[100914]: pgmap v1274: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 486 op/s; 15577/259395 objects degraded (6.005%); 15426/259395 objects misplaced (5.947%); 544 KiB/s, 8 objects/s recovering 2024-09-21T10:29:19.758 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:19 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/904572438' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:29:19.759 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:19 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:29:19.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:19.806 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:19.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:19.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:19.832 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:29:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:19 smithi196 ceph-mon[84784]: pgmap v1274: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 486 op/s; 15577/259395 objects degraded (6.005%); 15426/259395 objects misplaced (5.947%); 544 KiB/s, 8 objects/s recovering 2024-09-21T10:29:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:19 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/904572438' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:29:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:19 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:29:19.908 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:19.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:19.949 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:19.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:20.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:20.082 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:20.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:20.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:20.183 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:20.220 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:20.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:20.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:20.322 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:20.349 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:20.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:20.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:20.451 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:20.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:29:20.476 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:20.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:20.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:20.478 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:29:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:29:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:29:20.579 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:20.604 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:20.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:20.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:20.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:20 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:29:20.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:20 smithi098 ceph-mon[97580]: osdmap e374: 8 total, 8 up, 8 in 2024-09-21T10:29:20.659 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:20 smithi098 ceph-mon[97580]: pgmap v1276: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 802 op/s; 15689/250887 objects degraded (6.253%); 14808/250887 objects misplaced (5.902%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:29:20.660 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:20 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:29:20.660 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:20 smithi098 ceph-mon[100914]: osdmap e374: 8 total, 8 up, 8 in 2024-09-21T10:29:20.660 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:20 smithi098 ceph-mon[100914]: pgmap v1276: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 802 op/s; 15689/250887 objects degraded (6.253%); 14808/250887 objects misplaced (5.902%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:29:20.705 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:20.732 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T10:29:20.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:20.762 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:20.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:20.863 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:20.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:20 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:29:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:20 smithi196 ceph-mon[84784]: osdmap e374: 8 total, 8 up, 8 in 2024-09-21T10:29:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:20 smithi196 ceph-mon[84784]: pgmap v1276: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 802 op/s; 15689/250887 objects degraded (6.253%); 14808/250887 objects misplaced (5.902%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:29:20.911 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:20.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:20.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:21.006 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:29:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:29:20] "GET /metrics HTTP/1.1" 200 36612 "" "Prometheus/2.51.0" 2024-09-21T10:29:21.014 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:21.040 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:21.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:21.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:21.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:21.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:21.208 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:21.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:21.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:21.341 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:21.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:21.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:21.442 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:21.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:21.487 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:21.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:21.588 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:21.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:21 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 15689/250887 objects degraded (6.253%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:21.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:21 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:29:21.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:21 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 15689/250887 objects degraded (6.253%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:21.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:21 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:29:21.626 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:21.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:21.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:21.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:21.760 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:21.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:21.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:21.862 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:21.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:21 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 15689/250887 objects degraded (6.253%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:21.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:21 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:29:21.903 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:21.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:21.938 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:21.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:22.039 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:22.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:22.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:22.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:22.173 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:22.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:22.238 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:22.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:22.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:22.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:22.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:22.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:22.511 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:22.538 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:22.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:22.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:22.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:22.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:22.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:22 smithi098 ceph-mon[97580]: pgmap v1277: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 745 op/s; 15689/249582 objects degraded (6.286%); 14808/249582 objects misplaced (5.933%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:29:22.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:22.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:22.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:22.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:22.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:22.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:22.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:22 smithi098 ceph-mon[100914]: pgmap v1277: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 745 op/s; 15689/249582 objects degraded (6.286%); 14808/249582 objects misplaced (5.933%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:29:22.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:22.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:22.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:22.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:22 smithi196 ceph-mon[84784]: pgmap v1277: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 745 op/s; 15689/249582 objects degraded (6.286%); 14808/249582 objects misplaced (5.933%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:29:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:22.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:22.683 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:22.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:22.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:22.697 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:22.785 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:22.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:22.836 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:22.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:22.838 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:29:22.838 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:29:22.938 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:22.964 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:22.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:22.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:23.066 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:23.091 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:23.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:23.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:23.193 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:23.225 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:23.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:23.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:23.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:23.374 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:23.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:23.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:23.475 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:23.507 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:23.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:23.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:29:23.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:23.645 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:23.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:23.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:23.747 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:23.748 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:23 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2942317435' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:29:23.748 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:23 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:29:23.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:23.780 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:23.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:23.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:23 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2942317435' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:29:23.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:23 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:29:23.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:23 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2942317435' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:29:23.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:23 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:29:23.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:23.909 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:23.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:23.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:24.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:24.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:24.055 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:24.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:24.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:29:24.156 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:24.187 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:24.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:24.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:24.289 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:24.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:24.316 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:24.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:24.417 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:24.453 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:24.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:24.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:24.517 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:29:24.554 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:24.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:24.607 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:24.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:24.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:24.746 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:24.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:24.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:24.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:24.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:24.885 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:24.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:24 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:29:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:24 smithi196 ceph-mon[84784]: pgmap v1278: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 688 op/s; 15689/242625 objects degraded (6.466%); 14399/242625 objects misplaced (5.935%); 979 KiB/s, 15 objects/s recovering 2024-09-21T10:29:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:24 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:29:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:24 smithi196 ceph-mon[84784]: osdmap e375: 8 total, 8 up, 8 in 2024-09-21T10:29:24.986 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:24.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:24 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:29:24.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:24 smithi098 ceph-mon[97580]: pgmap v1278: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 688 op/s; 15689/242625 objects degraded (6.466%); 14399/242625 objects misplaced (5.935%); 979 KiB/s, 15 objects/s recovering 2024-09-21T10:29:24.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:24 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:29:24.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:24 smithi098 ceph-mon[97580]: osdmap e375: 8 total, 8 up, 8 in 2024-09-21T10:29:24.995 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:24 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:29:24.995 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:24 smithi098 ceph-mon[100914]: pgmap v1278: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 688 op/s; 15689/242625 objects degraded (6.466%); 14399/242625 objects misplaced (5.935%); 979 KiB/s, 15 objects/s recovering 2024-09-21T10:29:24.995 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:24 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:29:24.995 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:24 smithi098 ceph-mon[100914]: osdmap e375: 8 total, 8 up, 8 in 2024-09-21T10:29:25.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:25.030 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:25.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:25.131 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:25.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:25.191 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:25.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:25.291 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:25.326 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:25.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:25.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:25.428 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:25.463 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:25.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:25.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:25.564 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:25.591 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:25.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:25.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:25.692 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:25.725 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:25.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:25.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:25.827 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:25.856 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:25.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:25.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:25.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:25.998 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:25.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:25.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:26.099 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:26.135 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:26.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:26.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:26.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:26.262 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:26.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:26.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:26.364 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:26.395 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:26.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:26.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:26.496 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:26.532 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:26.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:26.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:26.634 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:26.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:26 smithi196 ceph-mon[84784]: pgmap v1280: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 828 op/s; 15821/239508 objects degraded (6.606%); 14196/239508 objects misplaced (5.927%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:29:26.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:26 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 15821/239508 objects degraded (6.606%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:26.661 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:26.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:26.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:26.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:26.801 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:26.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:26.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:26.903 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:26.929 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:26.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:26.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:27.031 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:27.066 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:27.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:27.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:26 smithi098 ceph-mon[97580]: pgmap v1280: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 828 op/s; 15821/239508 objects degraded (6.606%); 14196/239508 objects misplaced (5.927%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:29:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:26 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 15821/239508 objects degraded (6.606%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:27.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:26 smithi098 ceph-mon[100914]: pgmap v1280: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 828 op/s; 15821/239508 objects degraded (6.606%); 14196/239508 objects misplaced (5.927%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:29:27.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:26 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 15821/239508 objects degraded (6.606%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:27.168 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:27.207 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:27.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:27.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:27.309 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:27.338 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:27.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:27.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:27.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:27.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:27.475 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:27.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:27.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:27.613 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:27.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:27.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:27.715 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:27.752 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:27.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:27.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:27.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:27.887 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:27.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:27.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:27.932 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:29:27.932 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:29:27.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:28.033 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:28.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:28.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:28.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:28.174 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:28.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:28.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:28.276 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:28.311 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:28.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:28.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:28.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:28.450 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:28.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:28.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:28.551 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:28.585 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:28.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:28.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:28.635 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:29:28.687 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:28.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:28.729 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:28.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:28.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:28.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:28.866 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:28.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:28 smithi196 ceph-mon[84784]: pgmap v1281: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 494 op/s; 15821/238785 objects degraded (6.626%); 14196/238785 objects misplaced (5.945%); 558 KiB/s, 8 objects/s recovering 2024-09-21T10:29:28.967 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:28.999 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:28.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:29.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:29.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:28 smithi098 ceph-mon[97580]: pgmap v1281: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 494 op/s; 15821/238785 objects degraded (6.626%); 14196/238785 objects misplaced (5.945%); 558 KiB/s, 8 objects/s recovering 2024-09-21T10:29:29.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:28 smithi098 ceph-mon[100914]: pgmap v1281: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 494 op/s; 15821/238785 objects degraded (6.626%); 14196/238785 objects misplaced (5.945%); 558 KiB/s, 8 objects/s recovering 2024-09-21T10:29:29.101 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:29.153 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:29.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:29.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:29.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:29.292 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:29.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:29.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:29.394 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:29.422 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:29.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:29.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:29.524 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:29.578 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:29:29.588 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:29.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:29.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:29.689 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:29.754 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:29.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:29.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:29.856 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:29.897 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:29.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:29.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:29.999 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:30.031 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:30.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:30.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:30.133 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:30.176 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:30.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:30.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:30.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:30.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:30.311 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:30.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:30.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:29:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:29:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:29:30.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:30.441 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:30.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:30.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:30.543 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:30.583 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:30.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:30.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:30.685 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:30.707 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:30.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:30.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:30.715 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:29:30.715 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:29:30.767 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:29:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:29:30.803 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:29:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:29:30] "GET /metrics HTTP/1.1" 200 36616 "" "Prometheus/2.51.0" 2024-09-21T10:29:30.804 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:29:30.804 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:29:30.804 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:30 smithi098 ceph-mon[97580]: pgmap v1282: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 704 op/s; 15963/229740 objects degraded (6.948%); 13539/229740 objects misplaced (5.893%); 985 KiB/s, 15 objects/s recovering 2024-09-21T10:29:30.804 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:30 smithi098 ceph-mon[100914]: pgmap v1282: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 704 op/s; 15963/229740 objects degraded (6.948%); 13539/229740 objects misplaced (5.893%); 985 KiB/s, 15 objects/s recovering 2024-09-21T10:29:30.809 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:30.834 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:30.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:30.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:30.846 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:29:30.846 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:29:30.884 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:29:30.885 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:29:30.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:30 smithi196 ceph-mon[84784]: pgmap v1282: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 704 op/s; 15963/229740 objects degraded (6.948%); 13539/229740 objects misplaced (5.893%); 985 KiB/s, 15 objects/s recovering 2024-09-21T10:29:30.913 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:29:30.914 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:29:30.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:30.949 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:29:30.949 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:29:30.961 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:30.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:30.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:30.984 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:29:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:29:31.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:31.105 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:31.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:31.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:31.206 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:31.249 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:31.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:31.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:31.350 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:31.384 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:31.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:31.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:31.486 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:31.517 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:31.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:31.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:31.618 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:31.656 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:31.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:31.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:31.758 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:31.784 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:31.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:31.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:31.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:31 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3848812454' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:31.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:31 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 15963/229740 objects degraded (6.948%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:31.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:31 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3848812454' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:31.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:31 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 15963/229740 objects degraded (6.948%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:31.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:31.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:31 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3848812454' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:31.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:31 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 15963/229740 objects degraded (6.948%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:31.925 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:31.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:31.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:32.027 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:32.070 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:32.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:32.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:32.127 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:29:32.172 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:32.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:32.211 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:32.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:32.312 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:32.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:32.338 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:32.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:32.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:32.465 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:32.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:32.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:32.567 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:32.592 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:32.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:32.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:32.694 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:32.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:32.736 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:32.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:32.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:32.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:32 smithi098 ceph-mon[97580]: pgmap v1283: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 15963/228387 objects degraded (6.989%); 13539/228387 objects misplaced (5.928%); 985 KiB/s, 15 objects/s recovering 2024-09-21T10:29:32.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:32 smithi098 ceph-mon[100914]: pgmap v1283: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 15963/228387 objects degraded (6.989%); 13539/228387 objects misplaced (5.928%); 985 KiB/s, 15 objects/s recovering 2024-09-21T10:29:32.872 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:32.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:32.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:32.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:32 smithi196 ceph-mon[84784]: pgmap v1283: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 15963/228387 objects degraded (6.989%); 13539/228387 objects misplaced (5.928%); 985 KiB/s, 15 objects/s recovering 2024-09-21T10:29:32.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:33.041 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:33.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:33.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:33.143 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:33.187 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:33.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:33.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:33.289 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:33.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:33.325 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:33.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:33.377 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:29:33.377 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-21T10:29:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-21T10:29:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:29:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181808 9.c 9.d25df03c (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:29:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.277707+0000", 2024-09-21T10:29:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 290.09558192399999, 2024-09-21T10:29:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0082070830000001, 2024-09-21T10:29:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:29:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181808 2024-09-21T10:29:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277707+0000", 2024-09-21T10:29:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277707+0000", 2024-09-21T10:29:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277705+0000", 2024-09-21T10:29:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277710+0000", 2024-09-21T10:29:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9729999999999997e-06 2024-09-21T10:29:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277717+0000", 2024-09-21T10:29:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7460000000000003e-06 2024-09-21T10:29:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277722+0000", 2024-09-21T10:29:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4260000000000002e-06 2024-09-21T10:29:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277974+0000", 2024-09-21T10:29:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025160500000000001 2024-09-21T10:29:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277980+0000", 2024-09-21T10:29:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0730000000000001e-06 2024-09-21T10:29:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.285892+0000", 2024-09-21T10:29:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0079123059999999 2024-09-21T10:29:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.285914+0000", 2024-09-21T10:29:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1756999999999999e-05 2024-09-21T10:29:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181841 9.c 9.8718e16c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:29:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.307618+0000", 2024-09-21T10:29:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 290.06567053200001, 2024-09-21T10:29:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97832473499999995, 2024-09-21T10:29:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:29:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181841 2024-09-21T10:29:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307618+0000", 2024-09-21T10:29:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307618+0000", 2024-09-21T10:29:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307617+0000", 2024-09-21T10:29:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307625+0000", 2024-09-21T10:29:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.473e-06 2024-09-21T10:29:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307642+0000", 2024-09-21T10:29:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7408e-05 2024-09-21T10:29:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307648+0000", 2024-09-21T10:29:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6400000000000002e-06 2024-09-21T10:29:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307699+0000", 2024-09-21T10:29:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1359000000000001e-05 2024-09-21T10:29:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307707+0000", 2024-09-21T10:29:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.328e-06 2024-09-21T10:29:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.285935+0000", 2024-09-21T10:29:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97822819699999997 2024-09-21T10:29:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.285943+0000", 2024-09-21T10:29:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0290000000000005e-06 2024-09-21T10:29:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181856 9.c 9.1a6bfdac (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:29:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.326694+0000", 2024-09-21T10:29:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 290.04659507899999, 2024-09-21T10:29:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.95929115300000001, 2024-09-21T10:29:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:29:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181856 2024-09-21T10:29:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326694+0000", 2024-09-21T10:29:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326694+0000", 2024-09-21T10:29:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326692+0000", 2024-09-21T10:29:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326704+0000", 2024-09-21T10:29:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1864e-05 2024-09-21T10:29:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326720+0000", 2024-09-21T10:29:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5603999999999998e-05 2024-09-21T10:29:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326727+0000", 2024-09-21T10:29:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.374e-06 2024-09-21T10:29:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326780+0000", 2024-09-21T10:29:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2669e-05 2024-09-21T10:29:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326788+0000", 2024-09-21T10:29:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1280000000000008e-06 2024-09-21T10:29:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.285962+0000", 2024-09-21T10:29:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.95917384299999997 2024-09-21T10:29:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.285985+0000", 2024-09-21T10:29:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3300000000000001e-05 2024-09-21T10:29:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181866 9.c 9.d4fb720c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:29:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.337971+0000", 2024-09-21T10:29:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 290.03531804300002, 2024-09-21T10:29:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94809235000000003, 2024-09-21T10:29:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:29:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181866 2024-09-21T10:29:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337971+0000", 2024-09-21T10:29:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337971+0000", 2024-09-21T10:29:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337969+0000", 2024-09-21T10:29:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337976+0000", 2024-09-21T10:29:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0319999999999998e-06 2024-09-21T10:29:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337992+0000", 2024-09-21T10:29:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5277e-05 2024-09-21T10:29:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337999+0000", 2024-09-21T10:29:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1720000000000001e-06 2024-09-21T10:29:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.338041+0000", 2024-09-21T10:29:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1777999999999999e-05 2024-09-21T10:29:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.338046+0000", 2024-09-21T10:29:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6119999999999998e-06 2024-09-21T10:29:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.286056+0000", 2024-09-21T10:29:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94800968200000002 2024-09-21T10:29:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.286063+0000", 2024-09-21T10:29:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1940000000000001e-06 2024-09-21T10:29:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243459 9.c 9:3515c86a:::benchmark_data_smithi098_254407_object110728:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:29:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:52.997457+0000", 2024-09-21T10:29:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 220.37583188299999, 2024-09-21T10:29:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.328995454, 2024-09-21T10:29:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:29:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243459 2024-09-21T10:29:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997457+0000", 2024-09-21T10:29:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997457+0000", 2024-09-21T10:29:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997456+0000", 2024-09-21T10:29:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997461+0000", 2024-09-21T10:29:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1320000000000002e-06 2024-09-21T10:29:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997484+0000", 2024-09-21T10:29:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3192000000000001e-05 2024-09-21T10:29:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997494+0000", 2024-09-21T10:29:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0175e-05 2024-09-21T10:29:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003474+0000", 2024-09-21T10:29:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059798409999999996 2024-09-21T10:29:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003481+0000", 2024-09-21T10:29:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4039999999999998e-06 2024-09-21T10:29:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.326400+0000", 2024-09-21T10:29:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3229192140000001 2024-09-21T10:29:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.326452+0000", 2024-09-21T10:29:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1841000000000001e-05 2024-09-21T10:29:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243460 9.c 9:3d3ff595:::benchmark_data_smithi098_254407_object110729:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:29:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:52.997502+0000", 2024-09-21T10:29:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 220.37578664500001, 2024-09-21T10:29:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3586443749999999, 2024-09-21T10:29:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:29:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243460 2024-09-21T10:29:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997502+0000", 2024-09-21T10:29:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997502+0000", 2024-09-21T10:29:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997502+0000", 2024-09-21T10:29:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:29:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997505+0000", 2024-09-21T10:29:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1889999999999999e-06 2024-09-21T10:29:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997512+0000", 2024-09-21T10:29:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8360000000000003e-06 2024-09-21T10:29:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997516+0000", 2024-09-21T10:29:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3100000000000002e-06 2024-09-21T10:29:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003820+0000", 2024-09-21T10:29:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0063037989999999997 2024-09-21T10:29:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003825+0000", 2024-09-21T10:29:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3410000000000002e-06 2024-09-21T10:29:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.326483+0000", 2024-09-21T10:29:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.322658127 2024-09-21T10:29:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for missing object", 2024-09-21T10:29:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.326563+0000", 2024-09-21T10:29:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0270000000000002e-05 2024-09-21T10:29:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356089+0000", 2024-09-21T10:29:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.029525136 2024-09-21T10:29:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356146+0000", 2024-09-21T10:29:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7917000000000001e-05 2024-09-21T10:29:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243465 9.c 9:36d01026:::benchmark_data_smithi098_254407_object110734:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:29:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:52.997520+0000", 2024-09-21T10:29:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 220.375768405, 2024-09-21T10:29:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.359137246, 2024-09-21T10:29:33.409 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:33.409 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:33.409 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:33.409 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:33.409 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:33.409 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243465 2024-09-21T10:29:33.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.409 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:33.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:33.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997520+0000", 2024-09-21T10:29:33.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:33.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997520+0000", 2024-09-21T10:29:33.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:33.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997520+0000", 2024-09-21T10:29:33.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:29:33.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:33.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997522+0000", 2024-09-21T10:29:33.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2139999999999999e-06 2024-09-21T10:29:33.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:33.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997528+0000", 2024-09-21T10:29:33.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.049e-06 2024-09-21T10:29:33.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:33.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997533+0000", 2024-09-21T10:29:33.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.578e-06 2024-09-21T10:29:33.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003841+0000", 2024-09-21T10:29:33.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0063081760000000004 2024-09-21T10:29:33.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:33.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003845+0000", 2024-09-21T10:29:33.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2760000000000002e-06 2024-09-21T10:29:33.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.326590+0000", 2024-09-21T10:29:33.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3227443409999999 2024-09-21T10:29:33.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.326677+0000", 2024-09-21T10:29:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7176999999999996e-05 2024-09-21T10:29:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:29:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.327203+0000", 2024-09-21T10:29:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00052658000000000002 2024-09-21T10:29:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.337718+0000", 2024-09-21T10:29:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.01051439 2024-09-21T10:29:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356287+0000", 2024-09-21T10:29:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018568972999999999 2024-09-21T10:29:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356519+0000", 2024-09-21T10:29:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000231921 2024-09-21T10:29:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356565+0000", 2024-09-21T10:29:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5957000000000001e-05 2024-09-21T10:29:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356658+0000", 2024-09-21T10:29:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3066999999999995e-05 2024-09-21T10:29:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243471 9.c 9:3e8385fd:::benchmark_data_smithi098_254407_object110740:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:29:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:52.997537+0000", 2024-09-21T10:29:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 220.37575157800001, 2024-09-21T10:29:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3592854779999999, 2024-09-21T10:29:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243471 2024-09-21T10:29:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997537+0000", 2024-09-21T10:29:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997537+0000", 2024-09-21T10:29:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997537+0000", 2024-09-21T10:29:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:29:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997539+0000", 2024-09-21T10:29:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3750000000000001e-06 2024-09-21T10:29:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997548+0000", 2024-09-21T10:29:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.02e-06 2024-09-21T10:29:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997552+0000", 2024-09-21T10:29:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9380000000000002e-06 2024-09-21T10:29:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003861+0000", 2024-09-21T10:29:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0063086239999999997 2024-09-21T10:29:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003865+0000", 2024-09-21T10:29:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1239999999999998e-06 2024-09-21T10:29:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.332121+0000", 2024-09-21T10:29:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.328256586 2024-09-21T10:29:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.332255+0000", 2024-09-21T10:29:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013350999999999999 2024-09-21T10:29:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:29:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.332401+0000", 2024-09-21T10:29:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000146465 2024-09-21T10:29:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.341719+0000", 2024-09-21T10:29:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0093176160000000008 2024-09-21T10:29:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356328+0000", 2024-09-21T10:29:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014608612 2024-09-21T10:29:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356695+0000", 2024-09-21T10:29:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00036710999999999997 2024-09-21T10:29:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356736+0000", 2024-09-21T10:29:33.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1112000000000002e-05 2024-09-21T10:29:33.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:33.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356823+0000", 2024-09-21T10:29:33.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6779000000000004e-05 2024-09-21T10:29:33.422 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.422 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:33.422 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.422 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243517 9.c 9:3e65f1ad:::benchmark_data_smithi098_254407_object110786:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:29:33.422 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.050602+0000", 2024-09-21T10:29:33.422 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 220.32268712300001, 2024-09-21T10:29:33.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.306614151, 2024-09-21T10:29:33.423 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:33.423 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:33.423 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:33.423 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:33.423 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:33.423 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243517 2024-09-21T10:29:33.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.423 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:33.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:33.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050602+0000", 2024-09-21T10:29:33.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:33.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050602+0000", 2024-09-21T10:29:33.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:33.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050599+0000", 2024-09-21T10:29:33.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999976 2024-09-21T10:29:33.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:33.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050607+0000", 2024-09-21T10:29:33.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0690000000000002e-06 2024-09-21T10:29:33.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:33.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050627+0000", 2024-09-21T10:29:33.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9422000000000001e-05 2024-09-21T10:29:33.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:33.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050634+0000", 2024-09-21T10:29:33.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1160000000000001e-06 2024-09-21T10:29:33.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050699+0000", 2024-09-21T10:29:33.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5191999999999995e-05 2024-09-21T10:29:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050709+0000", 2024-09-21T10:29:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3270000000000007e-06 2024-09-21T10:29:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.334376+0000", 2024-09-21T10:29:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.283667766 2024-09-21T10:29:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.334474+0000", 2024-09-21T10:29:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7481999999999999e-05 2024-09-21T10:29:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:29:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.334602+0000", 2024-09-21T10:29:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000128014 2024-09-21T10:29:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.341726+0000", 2024-09-21T10:29:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0071239490000000001 2024-09-21T10:29:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356357+0000", 2024-09-21T10:29:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014631183000000001 2024-09-21T10:29:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357084+0000", 2024-09-21T10:29:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00072674199999999995 2024-09-21T10:29:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357126+0000", 2024-09-21T10:29:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2732999999999999e-05 2024-09-21T10:29:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357216+0000", 2024-09-21T10:29:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9373e-05 2024-09-21T10:29:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243534 9.c 9:3cf10c44:::benchmark_data_smithi098_254407_object110803:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:29:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.063492+0000", 2024-09-21T10:29:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 220.30979649299999, 2024-09-21T10:29:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2940934849999999, 2024-09-21T10:29:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243534 2024-09-21T10:29:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063492+0000", 2024-09-21T10:29:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063492+0000", 2024-09-21T10:29:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063491+0000", 2024-09-21T10:29:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063498+0000", 2024-09-21T10:29:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9759999999999995e-06 2024-09-21T10:29:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063511+0000", 2024-09-21T10:29:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2238e-05 2024-09-21T10:29:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063516+0000", 2024-09-21T10:29:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6849999999999998e-06 2024-09-21T10:29:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063572+0000", 2024-09-21T10:29:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5648e-05 2024-09-21T10:29:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063579+0000", 2024-09-21T10:29:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2169999999999997e-06 2024-09-21T10:29:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.336131+0000", 2024-09-21T10:29:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.27255148 2024-09-21T10:29:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.336203+0000", 2024-09-21T10:29:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2214999999999999e-05 2024-09-21T10:29:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:29:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.336278+0000", 2024-09-21T10:29:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4629000000000007e-05 2024-09-21T10:29:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:33.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.343957+0000", 2024-09-21T10:29:33.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0076795850000000001 2024-09-21T10:29:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356395+0000", 2024-09-21T10:29:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012437325000000001 2024-09-21T10:29:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357453+0000", 2024-09-21T10:29:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001058743 2024-09-21T10:29:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357495+0000", 2024-09-21T10:29:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1498999999999997e-05 2024-09-21T10:29:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357586+0000", 2024-09-21T10:29:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1014999999999995e-05 2024-09-21T10:29:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243535 9.c 9:37d3df54:::benchmark_data_smithi098_254407_object110804:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:29:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.068354+0000", 2024-09-21T10:29:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 220.304934414, 2024-09-21T10:29:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2894149260000001, 2024-09-21T10:29:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243535 2024-09-21T10:29:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068354+0000", 2024-09-21T10:29:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068354+0000", 2024-09-21T10:29:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068352+0000", 2024-09-21T10:29:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068360+0000", 2024-09-21T10:29:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7209999999999994e-06 2024-09-21T10:29:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068377+0000", 2024-09-21T10:29:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7198e-05 2024-09-21T10:29:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068383+0000", 2024-09-21T10:29:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1160000000000004e-06 2024-09-21T10:29:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068465+0000", 2024-09-21T10:29:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1681999999999995e-05 2024-09-21T10:29:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068473+0000", 2024-09-21T10:29:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8310000000000001e-06 2024-09-21T10:29:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.337739+0000", 2024-09-21T10:29:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.269265758 2024-09-21T10:29:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.337850+0000", 2024-09-21T10:29:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000111327 2024-09-21T10:29:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:29:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.337980+0000", 2024-09-21T10:29:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000130149 2024-09-21T10:29:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.343962+0000", 2024-09-21T10:29:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059821479999999996 2024-09-21T10:29:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356421+0000", 2024-09-21T10:29:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012459328 2024-09-21T10:29:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357634+0000", 2024-09-21T10:29:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012120410000000001 2024-09-21T10:29:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357681+0000", 2024-09-21T10:29:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7024999999999998e-05 2024-09-21T10:29:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357769+0000", 2024-09-21T10:29:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8727999999999995e-05 2024-09-21T10:29:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243548 9.c 9:3c4fc234:::benchmark_data_smithi098_254407_object110817:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:29:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.078991+0000", 2024-09-21T10:29:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 220.294297841, 2024-09-21T10:29:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.278948075, 2024-09-21T10:29:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243548 2024-09-21T10:29:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.078991+0000", 2024-09-21T10:29:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.078991+0000", 2024-09-21T10:29:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.078989+0000", 2024-09-21T10:29:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.078996+0000", 2024-09-21T10:29:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.968e-06 2024-09-21T10:29:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.079013+0000", 2024-09-21T10:29:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6549e-05 2024-09-21T10:29:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.079019+0000", 2024-09-21T10:29:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6729999999999997e-06 2024-09-21T10:29:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.079063+0000", 2024-09-21T10:29:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4376000000000003e-05 2024-09-21T10:29:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.079070+0000", 2024-09-21T10:29:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6089999999999997e-06 2024-09-21T10:29:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.339499+0000", 2024-09-21T10:29:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.260429467 2024-09-21T10:29:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.339588+0000", 2024-09-21T10:29:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8708000000000005e-05 2024-09-21T10:29:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:29:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.339668+0000", 2024-09-21T10:29:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0549000000000004e-05 2024-09-21T10:29:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.350422+0000", 2024-09-21T10:29:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010753257 2024-09-21T10:29:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356457+0000", 2024-09-21T10:29:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0060354479999999997 2024-09-21T10:29:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357809+0000", 2024-09-21T10:29:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0013525049999999999 2024-09-21T10:29:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357848+0000", 2024-09-21T10:29:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8537000000000001e-05 2024-09-21T10:29:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357939+0000", 2024-09-21T10:29:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1001000000000005e-05 2024-09-21T10:29:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243570 9.c 9:342610f6:::benchmark_data_smithi098_254407_object110839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:29:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.101422+0000", 2024-09-21T10:29:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 220.271866851, 2024-09-21T10:29:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2566680909999999, 2024-09-21T10:29:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243570 2024-09-21T10:29:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101422+0000", 2024-09-21T10:29:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101422+0000", 2024-09-21T10:29:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101420+0000", 2024-09-21T10:29:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101428+0000", 2024-09-21T10:29:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8420000000000005e-06 2024-09-21T10:29:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101446+0000", 2024-09-21T10:29:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8026e-05 2024-09-21T10:29:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101452+0000", 2024-09-21T10:29:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4910000000000002e-06 2024-09-21T10:29:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101565+0000", 2024-09-21T10:29:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011296 2024-09-21T10:29:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101587+0000", 2024-09-21T10:29:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2072000000000001e-05 2024-09-21T10:29:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.341753+0000", 2024-09-21T10:29:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2401660510000001 2024-09-21T10:29:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.341857+0000", 2024-09-21T10:29:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000103174 2024-09-21T10:29:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:29:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.341986+0000", 2024-09-21T10:29:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012943199999999999 2024-09-21T10:29:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.355716+0000", 2024-09-21T10:29:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013729961000000001 2024-09-21T10:29:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356484+0000", 2024-09-21T10:29:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00076824599999999999 2024-09-21T10:29:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357972+0000", 2024-09-21T10:29:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014878319999999999 2024-09-21T10:29:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.358008+0000", 2024-09-21T10:29:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6263999999999997e-05 2024-09-21T10:29:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.358090+0000", 2024-09-21T10:29:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1607999999999995e-05 2024-09-21T10:29:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243592 9.c 9:3d6001ff:::benchmark_data_smithi098_254407_object110861:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:29:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.125239+0000", 2024-09-21T10:29:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 220.24804998600001, 2024-09-21T10:29:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.242184476, 2024-09-21T10:29:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243592 2024-09-21T10:29:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125239+0000", 2024-09-21T10:29:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125239+0000", 2024-09-21T10:29:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125237+0000", 2024-09-21T10:29:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125244+0000", 2024-09-21T10:29:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9600000000000003e-06 2024-09-21T10:29:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125256+0000", 2024-09-21T10:29:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1759e-05 2024-09-21T10:29:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125262+0000", 2024-09-21T10:29:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5139999999999998e-06 2024-09-21T10:29:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125306+0000", 2024-09-21T10:29:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3724999999999999e-05 2024-09-21T10:29:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125316+0000", 2024-09-21T10:29:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6500000000000008e-06 2024-09-21T10:29:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.344004+0000", 2024-09-21T10:29:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2186879720000001 2024-09-21T10:29:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.344092+0000", 2024-09-21T10:29:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8027000000000003e-05 2024-09-21T10:29:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:29:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.344168+0000", 2024-09-21T10:29:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6508999999999995e-05 2024-09-21T10:29:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.355720+0000", 2024-09-21T10:29:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011551532 2024-09-21T10:29:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356879+0000", 2024-09-21T10:29:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001159642 2024-09-21T10:29:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367261+0000", 2024-09-21T10:29:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010381081 2024-09-21T10:29:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367319+0000", 2024-09-21T10:29:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8953999999999998e-05 2024-09-21T10:29:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367423+0000", 2024-09-21T10:29:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000103762 2024-09-21T10:29:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243598 9.c 9:30869b3f:::benchmark_data_smithi098_254407_object110867:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:29:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.132539+0000", 2024-09-21T10:29:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 220.240749771, 2024-09-21T10:29:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2350737359999999, 2024-09-21T10:29:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243598 2024-09-21T10:29:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132539+0000", 2024-09-21T10:29:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132539+0000", 2024-09-21T10:29:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132537+0000", 2024-09-21T10:29:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132545+0000", 2024-09-21T10:29:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4660000000000002e-06 2024-09-21T10:29:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132564+0000", 2024-09-21T10:29:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8981999999999999e-05 2024-09-21T10:29:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132570+0000", 2024-09-21T10:29:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3400000000000003e-06 2024-09-21T10:29:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132754+0000", 2024-09-21T10:29:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000183821 2024-09-21T10:29:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132782+0000", 2024-09-21T10:29:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8096e-05 2024-09-21T10:29:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.345661+0000", 2024-09-21T10:29:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2128794350000001 2024-09-21T10:29:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.345758+0000", 2024-09-21T10:29:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6637000000000003e-05 2024-09-21T10:29:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:29:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.345832+0000", 2024-09-21T10:29:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3719000000000003e-05 2024-09-21T10:29:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.355724+0000", 2024-09-21T10:29:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0098921329999999991 2024-09-21T10:29:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356929+0000", 2024-09-21T10:29:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012046579999999999 2024-09-21T10:29:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367459+0000", 2024-09-21T10:29:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010530358 2024-09-21T10:29:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367509+0000", 2024-09-21T10:29:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0309999999999998e-05 2024-09-21T10:29:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367613+0000", 2024-09-21T10:29:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000103531 2024-09-21T10:29:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243614 9.c 9:3cd37905:::benchmark_data_smithi098_254407_object110883:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:29:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.149686+0000", 2024-09-21T10:29:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 220.22360315399999, 2024-09-21T10:29:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2181167049999999, 2024-09-21T10:29:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243614 2024-09-21T10:29:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149686+0000", 2024-09-21T10:29:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149686+0000", 2024-09-21T10:29:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149684+0000", 2024-09-21T10:29:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149691+0000", 2024-09-21T10:29:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1439999999999997e-06 2024-09-21T10:29:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149711+0000", 2024-09-21T10:29:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0307999999999999e-05 2024-09-21T10:29:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149723+0000", 2024-09-21T10:29:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1687e-05 2024-09-21T10:29:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149834+0000", 2024-09-21T10:29:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000110895 2024-09-21T10:29:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149870+0000", 2024-09-21T10:29:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.591e-05 2024-09-21T10:29:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.347247+0000", 2024-09-21T10:29:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1973775120000001 2024-09-21T10:29:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.347335+0000", 2024-09-21T10:29:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7427000000000002e-05 2024-09-21T10:29:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:29:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.347412+0000", 2024-09-21T10:29:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6972999999999996e-05 2024-09-21T10:29:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.355727+0000", 2024-09-21T10:29:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0083153749999999998 2024-09-21T10:29:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356967+0000", 2024-09-21T10:29:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001240285 2024-09-21T10:29:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367650+0000", 2024-09-21T10:29:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010682574 2024-09-21T10:29:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367693+0000", 2024-09-21T10:29:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3123e-05 2024-09-21T10:29:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367802+0000", 2024-09-21T10:29:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109219 2024-09-21T10:29:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243648 9.c 9:33fdd93d:::benchmark_data_smithi098_254407_object110917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:29:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.196470+0000", 2024-09-21T10:29:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 220.176818459, 2024-09-21T10:29:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1715113070000001, 2024-09-21T10:29:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243648 2024-09-21T10:29:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196470+0000", 2024-09-21T10:29:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196470+0000", 2024-09-21T10:29:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196469+0000", 2024-09-21T10:29:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196476+0000", 2024-09-21T10:29:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7000000000000008e-06 2024-09-21T10:29:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196492+0000", 2024-09-21T10:29:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5877000000000001e-05 2024-09-21T10:29:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196499+0000", 2024-09-21T10:29:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1550000000000001e-06 2024-09-21T10:29:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196545+0000", 2024-09-21T10:29:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5679000000000001e-05 2024-09-21T10:29:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196560+0000", 2024-09-21T10:29:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4919000000000001e-05 2024-09-21T10:29:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.348867+0000", 2024-09-21T10:29:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1523073770000001 2024-09-21T10:29:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.348947+0000", 2024-09-21T10:29:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9827999999999995e-05 2024-09-21T10:29:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:29:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.349016+0000", 2024-09-21T10:29:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9091000000000006e-05 2024-09-21T10:29:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.355766+0000", 2024-09-21T10:29:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0067494419999999996 2024-09-21T10:29:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357002+0000", 2024-09-21T10:29:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012364780000000001 2024-09-21T10:29:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367840+0000", 2024-09-21T10:29:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010837704 2024-09-21T10:29:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367884+0000", 2024-09-21T10:29:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4353e-05 2024-09-21T10:29:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367982+0000", 2024-09-21T10:29:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7347000000000002e-05 2024-09-21T10:29:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243665 9.c 9:3b9d021e:::benchmark_data_smithi098_254407_object110934:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:29:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.227151+0000", 2024-09-21T10:29:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 220.14613730299999, 2024-09-21T10:29:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.141037334, 2024-09-21T10:29:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243665 2024-09-21T10:29:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227151+0000", 2024-09-21T10:29:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227151+0000", 2024-09-21T10:29:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227150+0000", 2024-09-21T10:29:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227157+0000", 2024-09-21T10:29:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3030000000000002e-06 2024-09-21T10:29:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227169+0000", 2024-09-21T10:29:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1999e-05 2024-09-21T10:29:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227174+0000", 2024-09-21T10:29:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4580000000000001e-06 2024-09-21T10:29:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227202+0000", 2024-09-21T10:29:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8002999999999999e-05 2024-09-21T10:29:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227208+0000", 2024-09-21T10:29:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4099999999999999e-06 2024-09-21T10:29:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.350461+0000", 2024-09-21T10:29:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.123253126 2024-09-21T10:29:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.350554+0000", 2024-09-21T10:29:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2917000000000005e-05 2024-09-21T10:29:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:29:33.489 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.350643+0000", 2024-09-21T10:29:33.489 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9167999999999997e-05 2024-09-21T10:29:33.489 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.489 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.489 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.355771+0000", 2024-09-21T10:29:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051275189999999997 2024-09-21T10:29:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357271+0000", 2024-09-21T10:29:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015008459999999999 2024-09-21T10:29:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.368035+0000", 2024-09-21T10:29:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010763679 2024-09-21T10:29:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.368089+0000", 2024-09-21T10:29:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3399000000000002e-05 2024-09-21T10:29:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.368189+0000", 2024-09-21T10:29:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100296 2024-09-21T10:29:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243683 9.c 9:3b325d23:::benchmark_data_smithi098_254407_object110952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:29:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.260563+0000", 2024-09-21T10:29:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 220.11272623599999, 2024-09-21T10:29:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1154280519999999, 2024-09-21T10:29:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243683 2024-09-21T10:29:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260563+0000", 2024-09-21T10:29:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260563+0000", 2024-09-21T10:29:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260561+0000", 2024-09-21T10:29:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:29:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260567+0000", 2024-09-21T10:29:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6849999999999998e-06 2024-09-21T10:29:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260577+0000", 2024-09-21T10:29:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0217000000000001e-05 2024-09-21T10:29:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260582+0000", 2024-09-21T10:29:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3880000000000002e-06 2024-09-21T10:29:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260623+0000", 2024-09-21T10:29:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0915999999999999e-05 2024-09-21T10:29:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260630+0000", 2024-09-21T10:29:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5730000000000002e-06 2024-09-21T10:29:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.352085+0000", 2024-09-21T10:29:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0914547569999999 2024-09-21T10:29:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.352173+0000", 2024-09-21T10:29:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7819999999999996e-05 2024-09-21T10:29:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:29:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.352293+0000", 2024-09-21T10:29:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000120475 2024-09-21T10:29:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.355774+0000", 2024-09-21T10:29:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003481137 2024-09-21T10:29:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357314+0000", 2024-09-21T10:29:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001539285 2024-09-21T10:29:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.375813+0000", 2024-09-21T10:29:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018498905 2024-09-21T10:29:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.375865+0000", 2024-09-21T10:29:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2945000000000002e-05 2024-09-21T10:29:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.375991+0000", 2024-09-21T10:29:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012511 2024-09-21T10:29:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243684 9.c 9:3d7d9148:::benchmark_data_smithi098_254407_object110953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:29:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.263461+0000", 2024-09-21T10:29:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 220.109828067, 2024-09-21T10:29:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.115013201, 2024-09-21T10:29:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243684 2024-09-21T10:29:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263461+0000", 2024-09-21T10:29:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263461+0000", 2024-09-21T10:29:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263460+0000", 2024-09-21T10:29:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:29:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263465+0000", 2024-09-21T10:29:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6280000000000004e-06 2024-09-21T10:29:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263473+0000", 2024-09-21T10:29:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5290000000000008e-06 2024-09-21T10:29:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263477+0000", 2024-09-21T10:29:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9500000000000003e-06 2024-09-21T10:29:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263505+0000", 2024-09-21T10:29:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7491000000000002e-05 2024-09-21T10:29:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263510+0000", 2024-09-21T10:29:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4090000000000002e-06 2024-09-21T10:29:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.353714+0000", 2024-09-21T10:29:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0902042750000001 2024-09-21T10:29:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.353823+0000", 2024-09-21T10:29:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108884 2024-09-21T10:29:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:29:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.353950+0000", 2024-09-21T10:29:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012683599999999999 2024-09-21T10:29:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356144+0000", 2024-09-21T10:29:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021941030000000002 2024-09-21T10:29:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357352+0000", 2024-09-21T10:29:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012075149999999999 2024-09-21T10:29:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.378332+0000", 2024-09-21T10:29:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.020980784999999998 2024-09-21T10:29:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.378381+0000", 2024-09-21T10:29:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8603000000000002e-05 2024-09-21T10:29:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.378474+0000", 2024-09-21T10:29:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2977000000000001e-05 2024-09-21T10:29:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:33.505 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:29:33.541 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:33.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:33.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:33.644 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:33.673 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:33.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:33.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:33.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:33.816 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:33.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:33.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:33.918 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:33.948 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:33.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:33.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:34.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:34.105 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:34.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:34.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:34.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:33 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:34.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:33 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:34.208 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:34.233 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:33 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:34.233 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:33 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:34.233 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:33 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:34.233 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:33 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:34.243 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:34.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:34.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:34.309 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:34.345 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:34.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:34.434 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:34.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:34.488 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:29:34.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:34.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:34.590 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:34.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:34.690 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:34.730 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:34.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:34.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:34.831 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:34.870 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:34.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:34.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:34.972 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:35.008 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:35.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:35.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:35.016 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:29:35.059 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T10:29:35.111 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:35.139 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:35.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:35.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:35.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:35.296 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:35.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:35.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:35.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:35 smithi098 ceph-mon[97580]: pgmap v1284: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 712 op/s; 15963/221244 objects degraded (7.215%); 13109/221244 objects misplaced (5.925%); 985 KiB/s, 15 objects/s recovering 2024-09-21T10:29:35.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:35 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:35.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:35 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:35.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:35 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2159498899' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:29:35.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:35 smithi098 ceph-mon[100914]: pgmap v1284: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 712 op/s; 15963/221244 objects degraded (7.215%); 13109/221244 objects misplaced (5.925%); 985 KiB/s, 15 objects/s recovering 2024-09-21T10:29:35.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:35 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:35.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:35 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:35.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:35 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2159498899' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:29:35.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:35 smithi196 ceph-mon[84784]: pgmap v1284: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 712 op/s; 15963/221244 objects degraded (7.215%); 13109/221244 objects misplaced (5.925%); 985 KiB/s, 15 objects/s recovering 2024-09-21T10:29:35.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:35 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:35.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:35 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:35.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:35 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2159498899' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:29:35.398 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:35.438 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:35.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:35.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:35.540 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:35.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:35.575 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:35.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:35.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:35.710 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:35.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:35.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:35.811 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:35.870 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:29:35.879 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:35.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:35.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:35.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:36.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:36.015 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:36.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:36.115 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:36.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:36.158 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:36.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:36.259 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:36.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:36.294 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:36.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:36.345 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:36 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2159498899' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:29:36.345 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:36 smithi098 ceph-mon[97580]: osdmap e376: 8 total, 8 up, 8 in 2024-09-21T10:29:36.345 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:36 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 15963/221244 objects degraded (7.215%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:36.345 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:36 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2159498899' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:29:36.345 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:36 smithi098 ceph-mon[100914]: osdmap e376: 8 total, 8 up, 8 in 2024-09-21T10:29:36.345 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:36 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 15963/221244 objects degraded (7.215%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:36.386 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:36 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2159498899' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:29:36.386 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:36 smithi196 ceph-mon[84784]: osdmap e376: 8 total, 8 up, 8 in 2024-09-21T10:29:36.386 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:36 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 15963/221244 objects degraded (7.215%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:36.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T10:29:36.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:36.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:36.444 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:36.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:36.545 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:36.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:36.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:36.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:36.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:36.698 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:36.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:36.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:36.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:36.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:36.842 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:36.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:36.942 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:36.967 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:36.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:36.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:37.013 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:37.069 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:37.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:37 smithi098 ceph-mon[97580]: pgmap v1286: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 16098/218046 objects degraded (7.383%); 12897/218046 objects misplaced (5.915%); 1024 KiB/s, 15 objects/s recovering 2024-09-21T10:29:37.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:37.121 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:37.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:37.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:37.277 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:37.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:37.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:37.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:37 smithi098 ceph-mon[100914]: pgmap v1286: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 16098/218046 objects degraded (7.383%); 12897/218046 objects misplaced (5.915%); 1024 KiB/s, 15 objects/s recovering 2024-09-21T10:29:37.379 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:37.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:37 smithi196 ceph-mon[84784]: pgmap v1286: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 16098/218046 objects degraded (7.383%); 12897/218046 objects misplaced (5.915%); 1024 KiB/s, 15 objects/s recovering 2024-09-21T10:29:37.405 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:37.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:37.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:37.507 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:37.534 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:37.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:37.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:37.636 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:37.661 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:37.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:37.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:37.762 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:37.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:37.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:37.896 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:37.932 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:37.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:37.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:38.033 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:38.075 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:38.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:38.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:38.177 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:38.210 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:38.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:38.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:38.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:38.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:38.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:38.350 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:38.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:38.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:38.452 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:38.481 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:38.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:38.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:38.582 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:38.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:38.611 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:38.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:38.712 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:38.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:38.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:38.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:38.850 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:38.877 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:38.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:38.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:38.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:39.018 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:39.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:39.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:39.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:39.154 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:39.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:39.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:39.189 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:39 smithi098 ceph-mon[97580]: pgmap v1287: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 16098/217377 objects degraded (7.406%); 12897/217377 objects misplaced (5.933%); 1024 KiB/s, 15 objects/s recovering 2024-09-21T10:29:39.189 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:39 smithi098 ceph-mon[100914]: pgmap v1287: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 16098/217377 objects degraded (7.406%); 12897/217377 objects misplaced (5.933%); 1024 KiB/s, 15 objects/s recovering 2024-09-21T10:29:39.256 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:39.287 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:39.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:39.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:39.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:39 smithi196 ceph-mon[84784]: pgmap v1287: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 16098/217377 objects degraded (7.406%); 12897/217377 objects misplaced (5.933%); 1024 KiB/s, 15 objects/s recovering 2024-09-21T10:29:39.389 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:39.413 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:39.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:39.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:39.514 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:39.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:39.539 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:39.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:39.640 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:39.666 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:39.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:39.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:39.767 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:39.803 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:29:39.804 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:29:39.804 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:39.804 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46322.0:181506 10.9 10:9342d7b4:::benchmark_data_smithi098_281896_object56481:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-21T10:29:39.804 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:29:39.794682+0000", 2024-09-21T10:29:39.804 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0062040009999999998, 2024-09-21T10:29:39.804 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0062442740000000002, 2024-09-21T10:29:39.804 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:39.804 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:29:39.804 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:39.804 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46322", 2024-09-21T10:29:39.804 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:29:39.804 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181506 2024-09-21T10:29:39.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:39.804 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:39.804 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:39.805 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:39.805 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:39.794682+0000", 2024-09-21T10:29:39.805 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:39.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:39.805 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:39.805 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:39.805 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:39.794682+0000", 2024-09-21T10:29:39.805 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:39.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:39.805 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:39.805 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:39.805 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:39.794680+0000", 2024-09-21T10:29:39.805 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:39.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:39.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:39.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:39.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:39.794685+0000", 2024-09-21T10:29:39.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1379999999999998e-06 2024-09-21T10:29:39.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:39.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:39.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:39.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:39.794694+0000", 2024-09-21T10:29:39.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2430000000000005e-06 2024-09-21T10:29:39.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:39.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:39.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:39.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:39.794698+0000", 2024-09-21T10:29:39.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1350000000000002e-06 2024-09-21T10:29:39.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:39.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:39.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:39.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:39.794745+0000", 2024-09-21T10:29:39.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6839000000000003e-05 2024-09-21T10:29:39.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:39.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:39.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:39.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:39.794862+0000", 2024-09-21T10:29:39.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000117442 2024-09-21T10:29:39.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:39.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:39.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:29:39.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:39.794994+0000", 2024-09-21T10:29:39.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000132158 2024-09-21T10:29:39.807 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:39.808 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:39.808 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:39.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:39.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:39.808 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46322.0:181507 10.7 e376/361 10:eddda4c4:::benchmark_data_smithi098_281896_object56482:head v 376'11144, mlcod=376'11142)", 2024-09-21T10:29:39.808 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:29:39.798427+0000", 2024-09-21T10:29:39.808 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0024587340000000002, 2024-09-21T10:29:39.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025733660000000001, 2024-09-21T10:29:39.808 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:39.808 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:29:39.808 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:39.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:39.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:39.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:39.798427+0000", 2024-09-21T10:29:39.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:39.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:39.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:39.809 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:39.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:39.798427+0000", 2024-09-21T10:29:39.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:39.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:39.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:39.809 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:39.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:39.798426+0000", 2024-09-21T10:29:39.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:29:39.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:39.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:39.809 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:39.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:39.798436+0000", 2024-09-21T10:29:39.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6609999999999995e-06 2024-09-21T10:29:39.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:39.810 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:39.810 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:39.810 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:39.798456+0000", 2024-09-21T10:29:39.810 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9541000000000001e-05 2024-09-21T10:29:39.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:39.810 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:39.810 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:39.810 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:39.798463+0000", 2024-09-21T10:29:39.810 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2130000000000002e-06 2024-09-21T10:29:39.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:39.810 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:39.810 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:39.811 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:39.798524+0000", 2024-09-21T10:29:39.811 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1217999999999996e-05 2024-09-21T10:29:39.811 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:39.811 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:39.811 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:39.811 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:39.798552+0000", 2024-09-21T10:29:39.811 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7727999999999999e-05 2024-09-21T10:29:39.811 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:39.811 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:39.811 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:39.811 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:39.811 INFO:teuthology.orchestra.run.smithi196.stdout: ], 2024-09-21T10:29:39.811 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 2 2024-09-21T10:29:39.811 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:29:39.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:39.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:39.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:39.913 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:39.956 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:39.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:39.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:40.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:40.093 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:40.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:40.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:40.195 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:29:40.253 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:40.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:40.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:40.355 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:40.386 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:40.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:40.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:40.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:40 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:40 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:40 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/589612038' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:29:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:40 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:29:40.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:29:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:29:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:29:40.487 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:40.511 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:40.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:40.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:40.564 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:29:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:40 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:40 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:40 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/589612038' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:29:40.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:40 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:29:40.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:40 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:40.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:40 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:40.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:40 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/589612038' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:29:40.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:40 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:29:40.613 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:40.654 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:40.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:40.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:40.756 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:40.803 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:40.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:40.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:40.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:40.940 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:40.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:40.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:40.974 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:29:41.042 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:41.045 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:29:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:29:40] "GET /metrics HTTP/1.1" 200 36616 "" "Prometheus/2.51.0" 2024-09-21T10:29:41.096 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:41.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:41.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:41.198 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:41.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:41.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:41.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:41 smithi196 ceph-mon[84784]: pgmap v1288: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 674 op/s; 16215/209505 objects degraded (7.740%); 12328/209505 objects misplaced (5.884%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:29:41.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:41 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:29:41.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:41 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:29:41.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:41 smithi196 ceph-mon[84784]: osdmap e377: 8 total, 8 up, 8 in 2024-09-21T10:29:41.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:41 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 16215/209505 objects degraded (7.740%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:41.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:41 smithi098 ceph-mon[97580]: pgmap v1288: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 674 op/s; 16215/209505 objects degraded (7.740%); 12328/209505 objects misplaced (5.884%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:29:41.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:41 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:29:41.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:41 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:29:41.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:41 smithi098 ceph-mon[97580]: osdmap e377: 8 total, 8 up, 8 in 2024-09-21T10:29:41.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:41 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 16215/209505 objects degraded (7.740%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:41.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:41 smithi098 ceph-mon[100914]: pgmap v1288: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 674 op/s; 16215/209505 objects degraded (7.740%); 12328/209505 objects misplaced (5.884%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:29:41.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:41 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:29:41.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:41 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:29:41.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:41 smithi098 ceph-mon[100914]: osdmap e377: 8 total, 8 up, 8 in 2024-09-21T10:29:41.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:41 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 16215/209505 objects degraded (7.740%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:41.348 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:41.377 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:41.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:41.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:41.419 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:41.479 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:41.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:41.523 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:41.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:41.624 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:41.663 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:41.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:41.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:41.765 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:41.789 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:41.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:41.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:41.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:41.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:41.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:42.027 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:42.051 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:42.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:42.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:42.135 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:29:42.135 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:29:42.152 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:42.168 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:29:42.168 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:29:42.192 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:42.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:42.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:42.199 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:29:42.199 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:29:42.234 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:29:42.234 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:29:42.265 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:29:42.265 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:29:42.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:42.294 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:29:42.294 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:29:42.326 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:29:42.326 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:29:42.328 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:42.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:42.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:42.356 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:29:42.356 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:29:42.430 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:42.465 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:42.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:42.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:42.486 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:42 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2233065662' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:42.487 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:42 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2233065662' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:42.567 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:42.590 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:42.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:42.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:42.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:42 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2233065662' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:42.692 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:42.719 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:42.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:42.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:42.821 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:42.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:42.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:42.959 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:42.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:42.987 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:42.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:43.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:43.091 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:29:43.137 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:43.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:43.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:43.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:43.265 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:43.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:43.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:43.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:43 smithi098 ceph-mon[97580]: pgmap v1290: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 540 op/s; 16215/208257 objects degraded (7.786%); 12328/208257 objects misplaced (5.920%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:29:43.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:43 smithi098 ceph-mon[100914]: pgmap v1290: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 540 op/s; 16215/208257 objects degraded (7.786%); 12328/208257 objects misplaced (5.920%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:29:43.367 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:43.394 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:43.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:43.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:43.495 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:43.522 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:43.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:43.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:43.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:43.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:43 smithi196 ceph-mon[84784]: pgmap v1290: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 540 op/s; 16215/208257 objects degraded (7.786%); 12328/208257 objects misplaced (5.920%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:29:43.668 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:43.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:43.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:43.770 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:43.797 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:43.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:43.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:43.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:43.939 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:43.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:43.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:44.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:44.110 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:44.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:44.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:44.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:44.241 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:44.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:44.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:44.343 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:44.384 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:44.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:44.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:44.487 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:44.518 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:44.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:44.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:44.620 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:44.664 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:44.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:44.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:44.766 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:44.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:44.799 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:44.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:44.900 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:44.934 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:44.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:44.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:45.023 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:29:45.023 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:29:45.035 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:45.075 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:45.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:45.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:45.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:45.216 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:45.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:45.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:45.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:45.349 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:45.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:45.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:45.450 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:45.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:45.481 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:45.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:45.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:45.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:45 smithi098 ceph-mon[97580]: pgmap v1291: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 701 op/s; 16215/201327 objects degraded (8.054%); 11900/201327 objects misplaced (5.911%); 629 KiB/s, 9 objects/s recovering 2024-09-21T10:29:45.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:45 smithi098 ceph-mon[100914]: pgmap v1291: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 701 op/s; 16215/201327 objects degraded (8.054%); 11900/201327 objects misplaced (5.911%); 629 KiB/s, 9 objects/s recovering 2024-09-21T10:29:45.622 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:45.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:45.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:45.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:45 smithi196 ceph-mon[84784]: pgmap v1291: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 701 op/s; 16215/201327 objects degraded (8.054%); 11900/201327 objects misplaced (5.911%); 629 KiB/s, 9 objects/s recovering 2024-09-21T10:29:45.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:29:45.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:45.760 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:29:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:45.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:45.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:45.888 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:45.929 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:45.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:45.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:46.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:46.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:46.068 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:46.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:46.138 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:29:46.169 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:46.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:46.203 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:46.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:46.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:46.332 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:46.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:46.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:46.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:46.470 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:46.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:46.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:46.536 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:46 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 16215/201327 objects degraded (8.054%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:46.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:46.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:46 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 16215/201327 objects degraded (8.054%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:46.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:46 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 16215/201327 objects degraded (8.054%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:46.623 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:46.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:46.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:46.725 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:46.756 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:46.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:46.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:46.858 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:46.888 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:46.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:46.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:46.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:47.033 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:47.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:47.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:47.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:47.170 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:47.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:47.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:47.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:47.310 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:47.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:47.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:47.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:47 smithi098 ceph-mon[97580]: pgmap v1292: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 665 op/s; 16334/198078 objects degraded (8.246%); 11693/198078 objects misplaced (5.903%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:29:47.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:47 smithi098 ceph-mon[100914]: pgmap v1292: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 665 op/s; 16334/198078 objects degraded (8.246%); 11693/198078 objects misplaced (5.903%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:29:47.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:47.446 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:47.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:47.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:47.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:47.564 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:47.570 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:47 smithi196 ceph-mon[84784]: pgmap v1292: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 665 op/s; 16334/198078 objects degraded (8.246%); 11693/198078 objects misplaced (5.903%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:29:47.641 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:47.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:47.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:47.743 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:47.775 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:47.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:47.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:47.876 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:47.914 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:47.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:47.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:48.012 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:29:48.012 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:29:48.016 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:48.058 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:48.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:48.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:48.160 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:48.184 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:48.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:48.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:48.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:48.338 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:48.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:48.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:48.440 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:48.471 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:48.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:48.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:48.572 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:48.597 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:48.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:48.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:48.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:48.732 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:29:48.759 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:48.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:48.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:48.861 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:48.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:48.934 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:48.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:49.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:49.052 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:29:49.095 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:49.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:49.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:49.197 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:49.251 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:49.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:49.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:49.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:49 smithi098 ceph-mon[97580]: pgmap v1293: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 661 op/s; 16334/197511 objects degraded (8.270%); 11693/197511 objects misplaced (5.920%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:29:49.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:49 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:49.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:49 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:49.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:49 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4293243878' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:29:49.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:49 smithi098 ceph-mon[100914]: pgmap v1293: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 661 op/s; 16334/197511 objects degraded (8.270%); 11693/197511 objects misplaced (5.920%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:29:49.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:49 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:49.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:49 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:49.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:49 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4293243878' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:29:49.353 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:49.377 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:49.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:49.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:49.468 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:49 smithi196 ceph-mon[84784]: pgmap v1293: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 661 op/s; 16334/197511 objects degraded (8.270%); 11693/197511 objects misplaced (5.920%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:29:49.468 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:49 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:49.468 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:49 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:49.468 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:49 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4293243878' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:29:49.480 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:49.514 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:49.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:49.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:49.616 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:49.654 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:49.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:49.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:49.700 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T10:29:49.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:49.789 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:49.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:49.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:49.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:49.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:49.937 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:49.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:50.038 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:50.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:50.071 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:50.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:50.171 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:50.202 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:50.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:50.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:50.304 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:50.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:50.343 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:50.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:50.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:50.474 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:29:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:29:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:29:50.475 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:50 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:29:50.475 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:50 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:29:50.475 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:50 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:50.475 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:50 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4293243878' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:29:50.475 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:50 smithi196 ceph-mon[84784]: osdmap e378: 8 total, 8 up, 8 in 2024-09-21T10:29:50.487 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:50.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:50.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:50.589 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:50.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:50 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:29:50.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:50 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:29:50.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:50 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:50.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:50 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4293243878' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:29:50.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:50 smithi098 ceph-mon[97580]: osdmap e378: 8 total, 8 up, 8 in 2024-09-21T10:29:50.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:50 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:29:50.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:50 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:29:50.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:50 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:29:50.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:50 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4293243878' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:29:50.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:50 smithi098 ceph-mon[100914]: osdmap e378: 8 total, 8 up, 8 in 2024-09-21T10:29:50.622 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:50.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:50.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:50.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:50.759 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:50.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:50.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:50.860 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:50.899 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:50.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:50.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:50.910 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:29:50.910 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:29:51.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:51.029 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:29:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:29:50] "GET /metrics HTTP/1.1" 200 36616 "" "Prometheus/2.51.0" 2024-09-21T10:29:51.037 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:51.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:51.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:51.068 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T10:29:51.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:29:51.139 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:51.177 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:51.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:51.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:51.279 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:51.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:51.327 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:51.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:51.428 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:51.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:51.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:51.581 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:51.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:51 smithi098 ceph-mon[97580]: pgmap v1294: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 679 op/s; 16494/189111 objects degraded (8.722%); 11039/189111 objects misplaced (5.837%); 1017 KiB/s, 15 objects/s recovering 2024-09-21T10:29:51.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:51 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 16494/189111 objects degraded (8.722%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:51.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:51 smithi098 ceph-mon[100914]: pgmap v1294: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 679 op/s; 16494/189111 objects degraded (8.722%); 11039/189111 objects misplaced (5.837%); 1017 KiB/s, 15 objects/s recovering 2024-09-21T10:29:51.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:51 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 16494/189111 objects degraded (8.722%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:51.608 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:51.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:51.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:51.633 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:51 smithi196 ceph-mon[84784]: pgmap v1294: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 679 op/s; 16494/189111 objects degraded (8.722%); 11039/189111 objects misplaced (5.837%); 1017 KiB/s, 15 objects/s recovering 2024-09-21T10:29:51.633 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:51 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 16494/189111 objects degraded (8.722%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:51.671 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:29:51.710 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:51.739 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:51.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:51.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:51.840 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:51.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:51.868 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:51.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:51.969 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:51.998 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:51.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:51.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:52.070 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:29:52.100 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:52.136 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:52.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:52.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:52.238 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:52.280 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:52.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:52.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:52.382 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:52.412 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:52.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:52.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:52.463 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:52 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2784067528' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:29:52.463 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:52 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:29:52.463 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:52.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:52.551 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:52.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:52.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:52.653 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:52.683 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:52.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:52.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:52.785 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:52.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:52.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:52.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:52.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:52 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2784067528' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:29:52.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:52 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:29:52.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:52.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:52 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2784067528' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:29:52.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:52 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:29:52.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:52.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:52.940 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:52.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:52.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:53.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:53.082 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:53.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:53.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:53.184 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:53.214 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:53.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:53.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:53.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:53.360 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:29:53.391 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:53.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:53.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:53.493 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:53.538 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:53.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:53.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:53.563 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:53 smithi098 ceph-mon[97580]: pgmap v1296: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 16494/187785 objects degraded (8.783%); 11039/187785 objects misplaced (5.879%); 1017 KiB/s, 15 objects/s recovering 2024-09-21T10:29:53.563 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:53 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:29:53.563 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:53 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:29:53.563 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:53 smithi098 ceph-mon[97580]: osdmap e379: 8 total, 8 up, 8 in 2024-09-21T10:29:53.563 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:53 smithi098 ceph-mon[100914]: pgmap v1296: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 16494/187785 objects degraded (8.783%); 11039/187785 objects misplaced (5.879%); 1017 KiB/s, 15 objects/s recovering 2024-09-21T10:29:53.563 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:53 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:29:53.563 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:53 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:29:53.563 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:53 smithi098 ceph-mon[100914]: osdmap e379: 8 total, 8 up, 8 in 2024-09-21T10:29:53.629 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:53 smithi196 ceph-mon[84784]: pgmap v1296: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 16494/187785 objects degraded (8.783%); 11039/187785 objects misplaced (5.879%); 1017 KiB/s, 15 objects/s recovering 2024-09-21T10:29:53.629 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:53 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:29:53.630 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:53 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:29:53.630 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:53 smithi196 ceph-mon[84784]: osdmap e379: 8 total, 8 up, 8 in 2024-09-21T10:29:53.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:53.678 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:53.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:53.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:53.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:53.805 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:53.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:53.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:53.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:53.908 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-21T10:29:53.908 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:29:53.937 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:53.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:53.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:54.039 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:54.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:54.070 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:54.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:54.171 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:54.218 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:54.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:54.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:54.320 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:54.354 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:54.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:54.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:54.455 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:54.479 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:54.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:54.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:54.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:54.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:54.629 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:54.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:54.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:54.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:54.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:54.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:54.859 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:54.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:54.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:55.013 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:55.052 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:55.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:55.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:55.105 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:55.155 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:55.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:55.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:55.306 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:55.335 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:55.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:55.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:55.366 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:29:55.437 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:55.495 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:55.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:55.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:55.597 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:55 smithi196 ceph-mon[84784]: pgmap v1298: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 16494/180792 objects degraded (9.123%); 10607/180792 objects misplaced (5.867%); 568 KiB/s, 8 objects/s recovering 2024-09-21T10:29:55.641 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:55.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:55.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:55.651 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:55 smithi098 ceph-mon[97580]: pgmap v1298: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 16494/180792 objects degraded (9.123%); 10607/180792 objects misplaced (5.867%); 568 KiB/s, 8 objects/s recovering 2024-09-21T10:29:55.651 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:55 smithi098 ceph-mon[100914]: pgmap v1298: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 16494/180792 objects degraded (9.123%); 10607/180792 objects misplaced (5.867%); 568 KiB/s, 8 objects/s recovering 2024-09-21T10:29:55.743 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:55.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:55.780 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:55.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:55.801 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:29:55.882 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:55.911 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:55.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:55.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:56.013 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:56.056 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:56.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:56.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:56.158 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:56.196 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:56.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:56.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:56.298 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:56.323 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:56.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:56.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:56.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:56.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:56.460 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:56.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:56.561 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:56.590 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:56.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:56.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:56.597 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:29:56.636 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:56 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 16494/180792 objects degraded (9.123%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:56.636 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:56 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 16494/180792 objects degraded (9.123%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:56.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:56 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 16494/180792 objects degraded (9.123%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:29:56.692 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:56.720 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:56.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:56.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:56.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:56.879 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:56.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:56.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:56.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:57.003 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:29:57.003 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:29:57.031 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:57.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:57.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:57.040 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:29:57.041 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:29:57.074 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:29:57.075 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:29:57.116 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:29:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:29:57.133 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:57.149 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:29:57.150 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:29:57.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:57.173 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:57.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:57.178 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:29:57.178 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:29:57.208 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:29:57.209 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:29:57.241 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:29:57.241 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:29:57.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:57.302 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:57.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:57.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:57.405 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:57.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:57.438 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:57.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:57.538 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:57.570 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:57.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:57.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:57.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:57 smithi098 ceph-mon[97580]: pgmap v1299: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 837 op/s; 16609/177396 objects degraded (9.363%); 10401/177396 objects misplaced (5.863%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:29:57.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:57 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1713246779' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:57.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:57 smithi098 ceph-mon[100914]: pgmap v1299: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 837 op/s; 16609/177396 objects degraded (9.363%); 10401/177396 objects misplaced (5.863%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:29:57.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:57 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1713246779' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:57.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:57 smithi196 ceph-mon[84784]: pgmap v1299: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 837 op/s; 16609/177396 objects degraded (9.363%); 10401/177396 objects misplaced (5.863%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:29:57.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:57 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1713246779' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:57.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:57.686 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:29:57.687 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-21T10:29:57.687 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-21T10:29:57.687 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:29:57.687 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.687 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181812 9.1 9:8268420f:::benchmark_data_smithi098_254407_object49081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T10:29:57.687 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.268689+0000", 2024-09-21T10:29:57.687 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 314.41282883000002, 2024-09-21T10:29:57.687 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1774718389999999, 2024-09-21T10:29:57.687 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:57.687 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:57.687 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:57.687 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:57.688 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:57.688 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181812 2024-09-21T10:29:57.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.688 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:57.688 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.688 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:57.688 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268689+0000", 2024-09-21T10:29:57.688 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.688 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.688 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:57.688 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268689+0000", 2024-09-21T10:29:57.688 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.689 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.689 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:57.689 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268688+0000", 2024-09-21T10:29:57.689 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:57.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.689 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.689 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:57.689 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268695+0000", 2024-09-21T10:29:57.689 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6420000000000007e-06 2024-09-21T10:29:57.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.689 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.689 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:57.689 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268710+0000", 2024-09-21T10:29:57.689 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4959e-05 2024-09-21T10:29:57.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.690 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.690 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:57.690 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268716+0000", 2024-09-21T10:29:57.690 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0290000000000002e-06 2024-09-21T10:29:57.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.690 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.690 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.690 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.273397+0000", 2024-09-21T10:29:57.690 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046805370000000002 2024-09-21T10:29:57.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.690 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.690 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:57.690 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.273407+0000", 2024-09-21T10:29:57.691 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6970000000000005e-06 2024-09-21T10:29:57.691 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.691 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.691 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.691 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.292290+0000", 2024-09-21T10:29:57.691 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0188830849999999 2024-09-21T10:29:57.691 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.691 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.691 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:57.691 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.292305+0000", 2024-09-21T10:29:57.691 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4841000000000001e-05 2024-09-21T10:29:57.691 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.691 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.691 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.692 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.432527+0000", 2024-09-21T10:29:57.692 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1402229049999999 2024-09-21T10:29:57.692 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.692 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.692 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:57.692 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.432648+0000", 2024-09-21T10:29:57.692 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000120538 2024-09-21T10:29:57.692 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.692 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.692 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,4,7", 2024-09-21T10:29:57.692 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.432762+0000", 2024-09-21T10:29:57.692 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114478 2024-09-21T10:29:57.692 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.692 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.693 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:57.693 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.440164+0000", 2024-09-21T10:29:57.693 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0074017559999999998 2024-09-21T10:29:57.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.693 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.693 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.693 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.440418+0000", 2024-09-21T10:29:57.693 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025371399999999999 2024-09-21T10:29:57.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.693 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.693 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.694 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.440476+0000", 2024-09-21T10:29:57.694 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8146999999999997e-05 2024-09-21T10:29:57.694 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.694 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.694 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.694 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.445996+0000", 2024-09-21T10:29:57.694 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055200129999999998 2024-09-21T10:29:57.694 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.694 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.694 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:57.694 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.446054+0000", 2024-09-21T10:29:57.694 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8400999999999999e-05 2024-09-21T10:29:57.694 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.694 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.695 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:57.695 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.446161+0000", 2024-09-21T10:29:57.695 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000106586 2024-09-21T10:29:57.695 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.695 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:57.695 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.695 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.695 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.695 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181813 9.1 9:877e8199:::benchmark_data_smithi098_254407_object49082:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T10:29:57.695 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.268726+0000", 2024-09-21T10:29:57.695 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 314.41279177199999, 2024-09-21T10:29:57.695 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1922379680000001, 2024-09-21T10:29:57.695 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:57.695 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:57.696 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:57.696 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:57.696 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:57.696 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181813 2024-09-21T10:29:57.696 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.696 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:57.696 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.696 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:57.696 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268726+0000", 2024-09-21T10:29:57.696 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.696 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.696 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.696 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:57.696 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268726+0000", 2024-09-21T10:29:57.697 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.697 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.697 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:57.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268726+0000", 2024-09-21T10:29:57.697 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:29:57.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.697 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.697 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:57.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268728+0000", 2024-09-21T10:29:57.697 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7499999999999999e-06 2024-09-21T10:29:57.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.697 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.697 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:57.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268735+0000", 2024-09-21T10:29:57.698 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8700000000000003e-06 2024-09-21T10:29:57.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:57.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268739+0000", 2024-09-21T10:29:57.698 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8419999999999998e-06 2024-09-21T10:29:57.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.273431+0000", 2024-09-21T10:29:57.698 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046914460000000002 2024-09-21T10:29:57.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:57.699 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.273438+0000", 2024-09-21T10:29:57.699 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5809999999999999e-06 2024-09-21T10:29:57.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.699 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.699 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.699 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.292343+0000", 2024-09-21T10:29:57.699 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.018904467 2024-09-21T10:29:57.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.699 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.699 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:57.699 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.292352+0000", 2024-09-21T10:29:57.699 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0280000000000006e-06 2024-09-21T10:29:57.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.699 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.700 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.436686+0000", 2024-09-21T10:29:57.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1443347740000001 2024-09-21T10:29:57.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.700 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.700 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:57.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.436801+0000", 2024-09-21T10:29:57.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011475499999999999 2024-09-21T10:29:57.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.700 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.700 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,4,7", 2024-09-21T10:29:57.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.436931+0000", 2024-09-21T10:29:57.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012945200000000001 2024-09-21T10:29:57.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.701 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.701 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:57.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.440424+0000", 2024-09-21T10:29:57.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003493515 2024-09-21T10:29:57.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.701 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.702 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.702 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.444139+0000", 2024-09-21T10:29:57.702 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0037150170000000001 2024-09-21T10:29:57.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.702 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.702 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.702 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.454482+0000", 2024-09-21T10:29:57.702 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010343140000000001 2024-09-21T10:29:57.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.702 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.702 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.702 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.460725+0000", 2024-09-21T10:29:57.702 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0062427510000000004 2024-09-21T10:29:57.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.703 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.703 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:57.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.460845+0000", 2024-09-21T10:29:57.703 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119721 2024-09-21T10:29:57.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.703 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.703 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:57.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.460964+0000", 2024-09-21T10:29:57.703 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119419 2024-09-21T10:29:57.703 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.703 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:57.703 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.703 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.703 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212859 9.1 9:81de2bc9:::benchmark_data_smithi098_254407_object80128:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-21T10:29:57.704 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.362389+0000", 2024-09-21T10:29:57.704 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 279.31912913000002, 2024-09-21T10:29:57.704 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.029750589, 2024-09-21T10:29:57.704 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:57.704 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:29:57.704 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:57.704 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:57.704 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:57.704 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212859 2024-09-21T10:29:57.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.704 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:57.704 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.704 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:57.704 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362389+0000", 2024-09-21T10:29:57.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.705 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:57.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362389+0000", 2024-09-21T10:29:57.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.705 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:57.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362387+0000", 2024-09-21T10:29:57.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:57.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.705 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:57.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362395+0000", 2024-09-21T10:29:57.706 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6410000000000002e-06 2024-09-21T10:29:57.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.706 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.706 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:57.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362409+0000", 2024-09-21T10:29:57.706 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3872e-05 2024-09-21T10:29:57.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.706 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.706 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:57.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362415+0000", 2024-09-21T10:29:57.706 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0650000000000004e-06 2024-09-21T10:29:57.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.706 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.707 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362480+0000", 2024-09-21T10:29:57.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5152000000000002e-05 2024-09-21T10:29:57.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.707 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.707 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:57.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362601+0000", 2024-09-21T10:29:57.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000121327 2024-09-21T10:29:57.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.707 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.707 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,4,7", 2024-09-21T10:29:57.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362727+0000", 2024-09-21T10:29:57.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012533800000000001 2024-09-21T10:29:57.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.708 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.708 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.708 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.369166+0000", 2024-09-21T10:29:57.708 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006439274 2024-09-21T10:29:57.708 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.708 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.708 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.708 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.369292+0000", 2024-09-21T10:29:57.708 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012620200000000001 2024-09-21T10:29:57.708 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.708 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.708 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:57.708 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.369926+0000", 2024-09-21T10:29:57.708 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00063336299999999996 2024-09-21T10:29:57.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.709 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.709 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.709 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.374921+0000", 2024-09-21T10:29:57.709 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049951869999999999 2024-09-21T10:29:57.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.709 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.709 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:57.709 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.374935+0000", 2024-09-21T10:29:57.709 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3872e-05 2024-09-21T10:29:57.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.709 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.709 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.709 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.392097+0000", 2024-09-21T10:29:57.709 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0171621260000001 2024-09-21T10:29:57.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.710 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.710 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:57.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.392140+0000", 2024-09-21T10:29:57.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2700000000000001e-05 2024-09-21T10:29:57.710 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.710 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:57.710 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.710 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.710 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212886 9.1 9:80fef226:::benchmark_data_smithi098_254407_object80155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:29:57.710 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.391416+0000", 2024-09-21T10:29:57.710 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 279.29010246399997, 2024-09-21T10:29:57.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.049473646, 2024-09-21T10:29:57.711 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:57.711 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:57.711 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:57.711 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:57.711 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:57.711 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212886 2024-09-21T10:29:57.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.711 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:57.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.711 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:57.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391416+0000", 2024-09-21T10:29:57.711 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.712 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.712 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:57.712 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391416+0000", 2024-09-21T10:29:57.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.712 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.712 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:57.712 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391414+0000", 2024-09-21T10:29:57.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:57.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.712 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.712 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:57.712 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391422+0000", 2024-09-21T10:29:57.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2369999999999992e-06 2024-09-21T10:29:57.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:57.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391441+0000", 2024-09-21T10:29:57.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9313999999999998e-05 2024-09-21T10:29:57.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:57.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391448+0000", 2024-09-21T10:29:57.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8399999999999997e-06 2024-09-21T10:29:57.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391577+0000", 2024-09-21T10:29:57.714 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012900699999999999 2024-09-21T10:29:57.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.714 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.714 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:57.714 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391596+0000", 2024-09-21T10:29:57.714 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9448000000000001e-05 2024-09-21T10:29:57.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.714 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.714 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.714 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.392163+0000", 2024-09-21T10:29:57.714 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0005668080000001 2024-09-21T10:29:57.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.714 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.715 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:57.715 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.392226+0000", 2024-09-21T10:29:57.715 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2736000000000004e-05 2024-09-21T10:29:57.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.715 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.715 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:29:57.715 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.392302+0000", 2024-09-21T10:29:57.715 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6459999999999996e-05 2024-09-21T10:29:57.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.715 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.715 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:57.715 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.427385+0000", 2024-09-21T10:29:57.715 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.035082852999999997 2024-09-21T10:29:57.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.716 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440761+0000", 2024-09-21T10:29:57.716 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013375398 2024-09-21T10:29:57.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.716 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440797+0000", 2024-09-21T10:29:57.716 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6238000000000001e-05 2024-09-21T10:29:57.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.716 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:57.717 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440845+0000", 2024-09-21T10:29:57.717 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7781999999999998e-05 2024-09-21T10:29:57.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.717 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.717 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:57.717 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440889+0000", 2024-09-21T10:29:57.717 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4515999999999999e-05 2024-09-21T10:29:57.717 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.717 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:57.717 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.717 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.717 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212893 9.1 9:867ca339:::benchmark_data_smithi098_254407_object80162:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:29:57.718 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.401222+0000", 2024-09-21T10:29:57.718 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 279.28029607799999, 2024-09-21T10:29:57.718 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.039903265, 2024-09-21T10:29:57.718 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:57.718 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:57.718 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:57.718 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:57.718 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:57.718 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212893 2024-09-21T10:29:57.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.718 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:57.718 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.719 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:57.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401222+0000", 2024-09-21T10:29:57.719 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.719 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.719 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:57.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401222+0000", 2024-09-21T10:29:57.719 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.719 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.719 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:57.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401220+0000", 2024-09-21T10:29:57.719 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:57.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.720 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.720 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:57.720 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401228+0000", 2024-09-21T10:29:57.720 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3100000000000003e-06 2024-09-21T10:29:57.720 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.720 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.720 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:57.720 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401245+0000", 2024-09-21T10:29:57.720 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7504999999999999e-05 2024-09-21T10:29:57.720 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.720 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.720 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:57.720 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401252+0000", 2024-09-21T10:29:57.721 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5679999999999996e-06 2024-09-21T10:29:57.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.721 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.721 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.721 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401303+0000", 2024-09-21T10:29:57.721 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1232e-05 2024-09-21T10:29:57.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.721 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.721 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:57.721 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401313+0000", 2024-09-21T10:29:57.721 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0023999999999999e-05 2024-09-21T10:29:57.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.721 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.721 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.722 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.395122+0000", 2024-09-21T10:29:57.722 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99380883799999997 2024-09-21T10:29:57.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.722 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.722 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:57.722 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.395230+0000", 2024-09-21T10:29:57.722 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107917 2024-09-21T10:29:57.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.722 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.722 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:29:57.722 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.395302+0000", 2024-09-21T10:29:57.722 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1963000000000002e-05 2024-09-21T10:29:57.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.722 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.723 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:57.723 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.427393+0000", 2024-09-21T10:29:57.723 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.032091732999999997 2024-09-21T10:29:57.723 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.723 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.723 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.723 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440948+0000", 2024-09-21T10:29:57.723 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013554639 2024-09-21T10:29:57.723 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.723 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.723 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.723 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441060+0000", 2024-09-21T10:29:57.723 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011248599999999999 2024-09-21T10:29:57.723 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.724 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.724 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:57.724 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441101+0000", 2024-09-21T10:29:57.724 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0738999999999997e-05 2024-09-21T10:29:57.724 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.724 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.724 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:57.724 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441125+0000", 2024-09-21T10:29:57.724 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4076000000000001e-05 2024-09-21T10:29:57.724 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.724 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:57.724 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.724 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.724 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.724 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212896 9.1 9:84f721c8:::benchmark_data_smithi098_254407_object80165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:29:57.725 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.404604+0000", 2024-09-21T10:29:57.725 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 279.276913686, 2024-09-21T10:29:57.725 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0367745399999999, 2024-09-21T10:29:57.725 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:57.725 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:57.725 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:57.725 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:57.725 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:57.725 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212896 2024-09-21T10:29:57.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.725 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:57.725 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.725 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:57.725 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404604+0000", 2024-09-21T10:29:57.725 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.726 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.726 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:57.726 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404604+0000", 2024-09-21T10:29:57.726 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.726 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.726 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:57.726 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404603+0000", 2024-09-21T10:29:57.726 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:57.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.726 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.726 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:57.727 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404611+0000", 2024-09-21T10:29:57.727 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.619e-06 2024-09-21T10:29:57.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.727 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.727 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:57.727 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404629+0000", 2024-09-21T10:29:57.727 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8158000000000001e-05 2024-09-21T10:29:57.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.727 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.727 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:57.727 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404636+0000", 2024-09-21T10:29:57.727 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8249999999999999e-06 2024-09-21T10:29:57.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.727 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.728 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.728 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404685+0000", 2024-09-21T10:29:57.728 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8498000000000002e-05 2024-09-21T10:29:57.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.728 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.728 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:57.728 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404693+0000", 2024-09-21T10:29:57.728 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7309999999999999e-06 2024-09-21T10:29:57.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.728 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.728 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.728 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.397879+0000", 2024-09-21T10:29:57.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99318610699999998 2024-09-21T10:29:57.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.729 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.729 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:57.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.398010+0000", 2024-09-21T10:29:57.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013017000000000001 2024-09-21T10:29:57.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.729 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.729 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:29:57.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.398133+0000", 2024-09-21T10:29:57.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012345499999999999 2024-09-21T10:29:57.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.729 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.729 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:57.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.427399+0000", 2024-09-21T10:29:57.730 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.029265835 2024-09-21T10:29:57.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.730 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.730 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440996+0000", 2024-09-21T10:29:57.730 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013597122 2024-09-21T10:29:57.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.730 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.730 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441297+0000", 2024-09-21T10:29:57.730 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00030115399999999998 2024-09-21T10:29:57.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.730 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.731 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:57.731 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441343+0000", 2024-09-21T10:29:57.731 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5518e-05 2024-09-21T10:29:57.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.731 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.731 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:57.731 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441379+0000", 2024-09-21T10:29:57.731 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6149999999999998e-05 2024-09-21T10:29:57.731 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.731 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:57.731 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.732 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.732 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212935 9.1 9:83432cba:::benchmark_data_smithi098_254407_object80204:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:29:57.732 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.443322+0000", 2024-09-21T10:29:57.732 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 279.23819629100001, 2024-09-21T10:29:57.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99820324299999996, 2024-09-21T10:29:57.732 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:57.732 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:57.732 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:57.732 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:57.732 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:57.732 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212935 2024-09-21T10:29:57.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.733 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:57.733 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.733 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:57.733 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443322+0000", 2024-09-21T10:29:57.733 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.733 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.733 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:57.733 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443322+0000", 2024-09-21T10:29:57.733 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.733 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.733 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:57.734 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443320+0000", 2024-09-21T10:29:57.734 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999976 2024-09-21T10:29:57.734 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.734 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.734 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:57.734 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443328+0000", 2024-09-21T10:29:57.734 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0760000000000003e-06 2024-09-21T10:29:57.734 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.734 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.734 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:57.734 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443345+0000", 2024-09-21T10:29:57.734 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7479999999999999e-05 2024-09-21T10:29:57.734 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.734 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.734 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:57.735 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443351+0000", 2024-09-21T10:29:57.735 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2179999999999996e-06 2024-09-21T10:29:57.735 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.735 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.735 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.735 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443503+0000", 2024-09-21T10:29:57.735 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015199300000000001 2024-09-21T10:29:57.735 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.735 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.735 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:57.735 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443524+0000", 2024-09-21T10:29:57.735 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0154000000000001e-05 2024-09-21T10:29:57.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.736 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.736 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.736 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.401733+0000", 2024-09-21T10:29:57.736 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.95820901400000003 2024-09-21T10:29:57.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.736 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.736 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:57.736 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.401836+0000", 2024-09-21T10:29:57.736 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000103812 2024-09-21T10:29:57.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.736 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.737 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:29:57.737 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.401955+0000", 2024-09-21T10:29:57.737 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011872899999999999 2024-09-21T10:29:57.737 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.737 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.737 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:57.737 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433786+0000", 2024-09-21T10:29:57.737 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.031831163000000003 2024-09-21T10:29:57.737 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.737 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.737 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.737 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441037+0000", 2024-09-21T10:29:57.737 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0072507400000000003 2024-09-21T10:29:57.737 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.737 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.738 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.738 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441450+0000", 2024-09-21T10:29:57.738 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00041282899999999997 2024-09-21T10:29:57.738 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.738 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.738 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:57.738 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441491+0000", 2024-09-21T10:29:57.738 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1603999999999997e-05 2024-09-21T10:29:57.738 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.738 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.738 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:57.738 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441525+0000", 2024-09-21T10:29:57.738 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3592e-05 2024-09-21T10:29:57.739 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.739 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:57.739 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.739 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.739 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212946 9.1 9:86b77d80:::benchmark_data_smithi098_254407_object80215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:29:57.739 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.455455+0000", 2024-09-21T10:29:57.739 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 279.22606261200002, 2024-09-21T10:29:57.739 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.98628469600000002, 2024-09-21T10:29:57.739 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:57.739 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:57.739 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:57.739 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:57.740 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:57.740 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212946 2024-09-21T10:29:57.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.740 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:57.740 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.740 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:57.740 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455455+0000", 2024-09-21T10:29:57.740 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.740 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.740 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:57.740 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455455+0000", 2024-09-21T10:29:57.741 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.741 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.741 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.741 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:57.741 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455454+0000", 2024-09-21T10:29:57.741 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:57.741 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.741 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.741 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:57.741 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455462+0000", 2024-09-21T10:29:57.741 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6349999999999995e-06 2024-09-21T10:29:57.741 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.741 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.741 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:57.741 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455479+0000", 2024-09-21T10:29:57.742 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6521e-05 2024-09-21T10:29:57.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.742 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.742 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:57.742 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455485+0000", 2024-09-21T10:29:57.742 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5289999999999997e-06 2024-09-21T10:29:57.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.742 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.742 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.742 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455531+0000", 2024-09-21T10:29:57.742 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5420000000000002e-05 2024-09-21T10:29:57.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.742 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.742 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:57.743 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455539+0000", 2024-09-21T10:29:57.743 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3769999999999996e-06 2024-09-21T10:29:57.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.743 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:57.744 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:57.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:57.745 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.404976+0000", 2024-09-21T10:29:57.745 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94943732199999997 2024-09-21T10:29:57.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.745 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.745 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:57.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.405084+0000", 2024-09-21T10:29:57.745 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010813400000000001 2024-09-21T10:29:57.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.745 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.745 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:29:57.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.405217+0000", 2024-09-21T10:29:57.746 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013250100000000001 2024-09-21T10:29:57.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.746 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.746 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:57.746 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433790+0000", 2024-09-21T10:29:57.746 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.028573111000000002 2024-09-21T10:29:57.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.746 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.746 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.746 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441178+0000", 2024-09-21T10:29:57.746 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0073877259999999998 2024-09-21T10:29:57.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.746 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.746 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.746 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441667+0000", 2024-09-21T10:29:57.747 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00048886199999999996 2024-09-21T10:29:57.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.747 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.747 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:57.747 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441706+0000", 2024-09-21T10:29:57.747 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8828000000000002e-05 2024-09-21T10:29:57.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.747 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.747 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:57.747 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441740+0000", 2024-09-21T10:29:57.747 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4625999999999997e-05 2024-09-21T10:29:57.747 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.747 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:57.747 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.748 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.748 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212950 9.1 9:83ffba2d:::benchmark_data_smithi098_254407_object80219:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:29:57.748 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.459261+0000", 2024-09-21T10:29:57.748 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 279.222256652, 2024-09-21T10:29:57.748 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.98262656500000001, 2024-09-21T10:29:57.748 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:57.748 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:57.748 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:57.748 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:57.748 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:57.748 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212950 2024-09-21T10:29:57.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.748 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:57.748 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.748 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:57.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459261+0000", 2024-09-21T10:29:57.749 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.749 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.749 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.749 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:57.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459261+0000", 2024-09-21T10:29:57.749 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.749 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.749 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.749 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:57.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459260+0000", 2024-09-21T10:29:57.749 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:57.749 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.749 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.749 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:57.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459267+0000", 2024-09-21T10:29:57.750 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7970000000000001e-06 2024-09-21T10:29:57.750 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.750 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.750 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:57.750 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459289+0000", 2024-09-21T10:29:57.750 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1185000000000001e-05 2024-09-21T10:29:57.750 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.750 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.750 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:57.750 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459295+0000", 2024-09-21T10:29:57.750 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5130000000000002e-06 2024-09-21T10:29:57.750 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.750 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.750 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.750 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459340+0000", 2024-09-21T10:29:57.751 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4555000000000003e-05 2024-09-21T10:29:57.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.751 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.751 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:57.751 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459349+0000", 2024-09-21T10:29:57.751 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4150000000000005e-06 2024-09-21T10:29:57.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.751 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.751 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.751 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.408479+0000", 2024-09-21T10:29:57.751 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94912973899999997 2024-09-21T10:29:57.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.751 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.751 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:57.751 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.408589+0000", 2024-09-21T10:29:57.752 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109972 2024-09-21T10:29:57.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.752 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.752 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:29:57.752 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.408712+0000", 2024-09-21T10:29:57.752 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012358900000000001 2024-09-21T10:29:57.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.752 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.752 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:57.752 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433794+0000", 2024-09-21T10:29:57.752 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.025081782 2024-09-21T10:29:57.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.752 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.752 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.752 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441575+0000", 2024-09-21T10:29:57.752 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.007781123 2024-09-21T10:29:57.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.753 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.753 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.753 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441795+0000", 2024-09-21T10:29:57.753 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000219268 2024-09-21T10:29:57.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.753 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.753 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:57.753 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441841+0000", 2024-09-21T10:29:57.753 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6286999999999999e-05 2024-09-21T10:29:57.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.753 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.753 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:57.753 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441888+0000", 2024-09-21T10:29:57.753 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7077999999999999e-05 2024-09-21T10:29:57.753 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.754 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:57.754 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.754 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.754 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212956 9.1 9:85c97ada:::benchmark_data_smithi098_254407_object80225:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:29:57.754 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.465260+0000", 2024-09-21T10:29:57.754 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 279.21625777000003, 2024-09-21T10:29:57.754 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97679754200000002, 2024-09-21T10:29:57.754 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:57.754 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:57.754 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:57.754 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:57.754 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:57.754 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212956 2024-09-21T10:29:57.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.755 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:57.755 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.755 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:57.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465260+0000", 2024-09-21T10:29:57.755 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.755 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.755 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:57.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465260+0000", 2024-09-21T10:29:57.755 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.755 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.755 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:57.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465258+0000", 2024-09-21T10:29:57.755 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:57.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.756 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.756 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:57.756 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465268+0000", 2024-09-21T10:29:57.756 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5109999999999992e-06 2024-09-21T10:29:57.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.756 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.756 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:57.756 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465290+0000", 2024-09-21T10:29:57.756 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2013000000000001e-05 2024-09-21T10:29:57.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.756 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.756 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:57.756 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465297+0000", 2024-09-21T10:29:57.756 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.311e-06 2024-09-21T10:29:57.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.757 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.757 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.757 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465456+0000", 2024-09-21T10:29:57.757 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000159347 2024-09-21T10:29:57.757 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.757 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.757 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:57.757 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465488+0000", 2024-09-21T10:29:57.757 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1739000000000002e-05 2024-09-21T10:29:57.757 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.757 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.757 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.757 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.411613+0000", 2024-09-21T10:29:57.758 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94612508699999998 2024-09-21T10:29:57.758 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.758 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.758 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:57.758 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.411728+0000", 2024-09-21T10:29:57.758 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011490200000000001 2024-09-21T10:29:57.758 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.758 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.758 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:29:57.758 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.411855+0000", 2024-09-21T10:29:57.758 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000126971 2024-09-21T10:29:57.758 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.758 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.759 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:57.759 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433798+0000", 2024-09-21T10:29:57.759 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.021942560999999999 2024-09-21T10:29:57.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.759 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.759 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.759 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441610+0000", 2024-09-21T10:29:57.759 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0078126959999999992 2024-09-21T10:29:57.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.759 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.759 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.759 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441984+0000", 2024-09-21T10:29:57.759 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000373956 2024-09-21T10:29:57.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.760 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.760 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:57.760 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442024+0000", 2024-09-21T10:29:57.760 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9459999999999998e-05 2024-09-21T10:29:57.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.760 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.760 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:57.760 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442058+0000", 2024-09-21T10:29:57.760 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4010000000000001e-05 2024-09-21T10:29:57.760 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.760 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:57.760 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.760 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.760 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212974 9.1 9:8bf7ce4b:::benchmark_data_smithi098_254407_object80243:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:29:57.760 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.484850+0000", 2024-09-21T10:29:57.761 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 279.19666790899998, 2024-09-21T10:29:57.761 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.95736883500000003, 2024-09-21T10:29:57.761 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:57.761 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:57.761 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:57.761 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:57.761 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:57.761 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212974 2024-09-21T10:29:57.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.761 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:57.761 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.761 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:57.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.484850+0000", 2024-09-21T10:29:57.761 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.762 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.762 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:57.762 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.484850+0000", 2024-09-21T10:29:57.762 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.762 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.762 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:57.762 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.484848+0000", 2024-09-21T10:29:57.762 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:57.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.762 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.762 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:57.762 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.484857+0000", 2024-09-21T10:29:57.763 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1589999999999999e-06 2024-09-21T10:29:57.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.763 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.763 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:57.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.484883+0000", 2024-09-21T10:29:57.763 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5681999999999999e-05 2024-09-21T10:29:57.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.763 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.763 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:57.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.484890+0000", 2024-09-21T10:29:57.763 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4200000000000001e-06 2024-09-21T10:29:57.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.763 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.763 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.485094+0000", 2024-09-21T10:29:57.763 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00020403399999999999 2024-09-21T10:29:57.764 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.764 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.764 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:57.764 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.485127+0000", 2024-09-21T10:29:57.764 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2221000000000002e-05 2024-09-21T10:29:57.764 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.764 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.764 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.764 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.414677+0000", 2024-09-21T10:29:57.764 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.929550023 2024-09-21T10:29:57.764 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.764 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.764 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:57.764 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.414781+0000", 2024-09-21T10:29:57.765 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104038 2024-09-21T10:29:57.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.765 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.765 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:29:57.765 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.414897+0000", 2024-09-21T10:29:57.765 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011673899999999999 2024-09-21T10:29:57.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.765 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.765 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:57.765 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433801+0000", 2024-09-21T10:29:57.765 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018903169000000001 2024-09-21T10:29:57.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.765 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.765 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.765 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441942+0000", 2024-09-21T10:29:57.765 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0081409929999999991 2024-09-21T10:29:57.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.766 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.766 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442168+0000", 2024-09-21T10:29:57.766 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00022609 2024-09-21T10:29:57.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.766 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.766 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:57.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442198+0000", 2024-09-21T10:29:57.766 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.042e-05 2024-09-21T10:29:57.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.766 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.766 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:57.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442219+0000", 2024-09-21T10:29:57.767 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0837000000000001e-05 2024-09-21T10:29:57.767 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.767 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:57.767 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.767 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.767 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212980 9.1 9:8e228128:::benchmark_data_smithi098_254407_object80249:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:29:57.767 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.495564+0000", 2024-09-21T10:29:57.767 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 279.18595404199999, 2024-09-21T10:29:57.767 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.946840558, 2024-09-21T10:29:57.767 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:57.767 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:57.767 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:57.767 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:57.767 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:57.767 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212980 2024-09-21T10:29:57.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.768 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:57.768 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.768 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:57.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495564+0000", 2024-09-21T10:29:57.768 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.768 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.768 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:57.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495564+0000", 2024-09-21T10:29:57.768 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.768 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.768 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:57.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495562+0000", 2024-09-21T10:29:57.768 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:57.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.769 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.769 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:57.769 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495569+0000", 2024-09-21T10:29:57.769 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.781e-06 2024-09-21T10:29:57.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.769 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.769 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:57.769 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495580+0000", 2024-09-21T10:29:57.769 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0601999999999999e-05 2024-09-21T10:29:57.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.769 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.769 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:57.769 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495586+0000", 2024-09-21T10:29:57.769 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9750000000000004e-06 2024-09-21T10:29:57.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.770 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.770 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.770 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495640+0000", 2024-09-21T10:29:57.770 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3989000000000001e-05 2024-09-21T10:29:57.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.770 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.770 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:57.770 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495651+0000", 2024-09-21T10:29:57.770 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1012000000000001e-05 2024-09-21T10:29:57.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.770 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.770 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.770 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.417846+0000", 2024-09-21T10:29:57.771 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.92219546299999999 2024-09-21T10:29:57.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.771 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.771 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:57.771 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.417972+0000", 2024-09-21T10:29:57.771 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000125695 2024-09-21T10:29:57.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.771 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.771 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:29:57.771 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.418097+0000", 2024-09-21T10:29:57.771 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012517 2024-09-21T10:29:57.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.771 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.771 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:57.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433804+0000", 2024-09-21T10:29:57.772 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015706759000000001 2024-09-21T10:29:57.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.772 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442128+0000", 2024-09-21T10:29:57.772 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0083244349999999998 2024-09-21T10:29:57.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.772 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442338+0000", 2024-09-21T10:29:57.772 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00020961600000000001 2024-09-21T10:29:57.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:57.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442372+0000", 2024-09-21T10:29:57.773 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4573999999999998e-05 2024-09-21T10:29:57.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:57.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442405+0000", 2024-09-21T10:29:57.773 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2186e-05 2024-09-21T10:29:57.773 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.773 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:57.773 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.773 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212982 9.1 9:8bbf57c4:::benchmark_data_smithi098_254407_object80251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:29:57.773 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.495812+0000", 2024-09-21T10:29:57.774 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 279.185705649, 2024-09-21T10:29:57.774 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94673693199999998, 2024-09-21T10:29:57.774 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:57.774 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:57.774 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:57.774 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:57.774 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:57.774 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212982 2024-09-21T10:29:57.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.774 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:57.774 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.774 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:57.774 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495812+0000", 2024-09-21T10:29:57.774 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.775 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.775 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:57.775 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495812+0000", 2024-09-21T10:29:57.775 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.775 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.775 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:57.775 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495812+0000", 2024-09-21T10:29:57.775 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:29:57.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.775 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.775 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:57.775 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495816+0000", 2024-09-21T10:29:57.775 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3039999999999998e-06 2024-09-21T10:29:57.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.775 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:57.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495824+0000", 2024-09-21T10:29:57.776 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5120000000000002e-06 2024-09-21T10:29:57.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:57.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495828+0000", 2024-09-21T10:29:57.776 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6240000000000001e-06 2024-09-21T10:29:57.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495870+0000", 2024-09-21T10:29:57.776 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1554999999999998e-05 2024-09-21T10:29:57.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:57.777 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495887+0000", 2024-09-21T10:29:57.777 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7122999999999999e-05 2024-09-21T10:29:57.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.777 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.777 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.777 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.420943+0000", 2024-09-21T10:29:57.777 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.92505649000000001 2024-09-21T10:29:57.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.777 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.777 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:57.777 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.421059+0000", 2024-09-21T10:29:57.777 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000115108 2024-09-21T10:29:57.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.777 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.777 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:29:57.778 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.421167+0000", 2024-09-21T10:29:57.778 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108929 2024-09-21T10:29:57.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.778 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.778 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:57.778 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433807+0000", 2024-09-21T10:29:57.778 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012639262 2024-09-21T10:29:57.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.778 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.778 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.778 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442271+0000", 2024-09-21T10:29:57.778 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0084638600000000001 2024-09-21T10:29:57.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.778 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.778 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442444+0000", 2024-09-21T10:29:57.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017356899999999999 2024-09-21T10:29:57.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.779 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.779 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:57.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442515+0000", 2024-09-21T10:29:57.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1299999999999998e-05 2024-09-21T10:29:57.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.779 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.779 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:57.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442549+0000", 2024-09-21T10:29:57.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3889e-05 2024-09-21T10:29:57.779 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.780 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:57.780 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.780 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.780 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:213024 9.1 9:8ab0c9ed:::benchmark_data_smithi098_254407_object80293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:29:57.780 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.545957+0000", 2024-09-21T10:29:57.780 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 279.13556057099998, 2024-09-21T10:29:57.780 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.896714071, 2024-09-21T10:29:57.780 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:57.780 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:57.780 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:57.780 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:57.780 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:57.780 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 213024 2024-09-21T10:29:57.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.780 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:57.781 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.781 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:57.781 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.545957+0000", 2024-09-21T10:29:57.781 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.781 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.781 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:57.781 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.545957+0000", 2024-09-21T10:29:57.781 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.781 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.781 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:57.781 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.545956+0000", 2024-09-21T10:29:57.781 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:57.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.782 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.782 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:57.782 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.545964+0000", 2024-09-21T10:29:57.782 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6629999999999999e-06 2024-09-21T10:29:57.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.782 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.782 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:57.782 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.545986+0000", 2024-09-21T10:29:57.782 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1220999999999999e-05 2024-09-21T10:29:57.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.782 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.782 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:57.782 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.545993+0000", 2024-09-21T10:29:57.782 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.396e-06 2024-09-21T10:29:57.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.782 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.546156+0000", 2024-09-21T10:29:57.783 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016294899999999999 2024-09-21T10:29:57.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:57.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.546178+0000", 2024-09-21T10:29:57.783 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1971999999999999e-05 2024-09-21T10:29:57.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.424032+0000", 2024-09-21T10:29:57.783 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.87785439099999996 2024-09-21T10:29:57.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.784 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:57.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.424140+0000", 2024-09-21T10:29:57.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107507 2024-09-21T10:29:57.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.784 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.784 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:29:57.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.424258+0000", 2024-09-21T10:29:57.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011840700000000001 2024-09-21T10:29:57.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.784 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.784 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:57.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.437127+0000", 2024-09-21T10:29:57.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012869234 2024-09-21T10:29:57.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.784 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.784 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.785 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442310+0000", 2024-09-21T10:29:57.785 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051825589999999998 2024-09-21T10:29:57.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.785 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.785 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442599+0000", 2024-09-21T10:29:57.785 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00028935099999999997 2024-09-21T10:29:57.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.785 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:57.785 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442639+0000", 2024-09-21T10:29:57.785 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9762000000000003e-05 2024-09-21T10:29:57.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.785 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:57.786 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442672+0000", 2024-09-21T10:29:57.786 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2480000000000001e-05 2024-09-21T10:29:57.786 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.786 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:57.786 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.786 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.786 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.786 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:213027 9.1 9:8d848b4e:::benchmark_data_smithi098_254407_object80296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:29:57.786 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.549492+0000", 2024-09-21T10:29:57.786 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 279.13202603000002, 2024-09-21T10:29:57.786 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.89346427100000003, 2024-09-21T10:29:57.786 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:57.786 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:57.786 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:57.786 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:57.786 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:57.787 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 213027 2024-09-21T10:29:57.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.787 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:57.787 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.787 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:57.787 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549492+0000", 2024-09-21T10:29:57.787 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.787 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.787 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:57.787 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549492+0000", 2024-09-21T10:29:57.787 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.787 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.787 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:57.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549490+0000", 2024-09-21T10:29:57.788 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:57.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.788 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.788 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:57.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549498+0000", 2024-09-21T10:29:57.788 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4209999999999995e-06 2024-09-21T10:29:57.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.788 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.788 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:57.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549518+0000", 2024-09-21T10:29:57.788 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0057000000000001e-05 2024-09-21T10:29:57.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.788 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.788 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:57.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549526+0000", 2024-09-21T10:29:57.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4730000000000003e-06 2024-09-21T10:29:57.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549593+0000", 2024-09-21T10:29:57.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7409000000000005e-05 2024-09-21T10:29:57.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:57.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549603+0000", 2024-09-21T10:29:57.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5810000000000003e-06 2024-09-21T10:29:57.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.427438+0000", 2024-09-21T10:29:57.790 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.87783562999999998 2024-09-21T10:29:57.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.790 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.790 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:57.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.427559+0000", 2024-09-21T10:29:57.790 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000120377 2024-09-21T10:29:57.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.790 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.790 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:29:57.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.427666+0000", 2024-09-21T10:29:57.790 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107382 2024-09-21T10:29:57.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.791 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.791 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:57.791 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.437138+0000", 2024-09-21T10:29:57.791 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0094713769999999996 2024-09-21T10:29:57.791 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.791 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.791 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.791 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442725+0000", 2024-09-21T10:29:57.791 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.005586995 2024-09-21T10:29:57.791 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.791 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.791 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.791 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442870+0000", 2024-09-21T10:29:57.791 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014551399999999999 2024-09-21T10:29:57.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.792 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.792 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:57.792 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442920+0000", 2024-09-21T10:29:57.792 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0371999999999998e-05 2024-09-21T10:29:57.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.792 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.792 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:57.792 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442956+0000", 2024-09-21T10:29:57.792 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5805e-05 2024-09-21T10:29:57.792 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.792 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:57.792 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.793 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.793 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:213030 9.1 9:8c3d5a31:::benchmark_data_smithi098_254407_object80299:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:29:57.793 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.556579+0000", 2024-09-21T10:29:57.793 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 279.12493873699998, 2024-09-21T10:29:57.793 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.88649370000000005, 2024-09-21T10:29:57.793 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:57.793 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:57.793 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:57.793 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:57.793 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:57.793 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 213030 2024-09-21T10:29:57.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.793 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:57.793 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.793 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:57.793 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556579+0000", 2024-09-21T10:29:57.794 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.794 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.794 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:57.794 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556579+0000", 2024-09-21T10:29:57.794 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.794 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.794 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:57.794 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556578+0000", 2024-09-21T10:29:57.794 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:57.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.794 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.794 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:57.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556585+0000", 2024-09-21T10:29:57.795 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6159999999999996e-06 2024-09-21T10:29:57.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.795 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.795 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:57.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556600+0000", 2024-09-21T10:29:57.795 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.454e-05 2024-09-21T10:29:57.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.795 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.795 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:57.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556607+0000", 2024-09-21T10:29:57.795 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8900000000000001e-06 2024-09-21T10:29:57.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.795 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.795 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.796 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556650+0000", 2024-09-21T10:29:57.796 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3727000000000003e-05 2024-09-21T10:29:57.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.796 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.796 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:57.797 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556658+0000", 2024-09-21T10:29:57.797 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5079999999999999e-06 2024-09-21T10:29:57.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.797 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.797 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.797 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.430408+0000", 2024-09-21T10:29:57.797 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.87375053700000005 2024-09-21T10:29:57.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.797 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.797 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:57.797 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.430547+0000", 2024-09-21T10:29:57.797 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013809400000000001 2024-09-21T10:29:57.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.797 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.798 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:29:57.798 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.430679+0000", 2024-09-21T10:29:57.798 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013244900000000001 2024-09-21T10:29:57.798 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.798 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.798 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:57.798 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440488+0000", 2024-09-21T10:29:57.798 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0098090959999999998 2024-09-21T10:29:57.798 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.798 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.798 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.798 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442765+0000", 2024-09-21T10:29:57.798 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002277276 2024-09-21T10:29:57.799 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.799 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.799 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.799 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443000+0000", 2024-09-21T10:29:57.799 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000234682 2024-09-21T10:29:57.799 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.799 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.799 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:57.799 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443042+0000", 2024-09-21T10:29:57.799 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2126000000000003e-05 2024-09-21T10:29:57.799 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.799 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.799 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:57.800 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443073+0000", 2024-09-21T10:29:57.800 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0825999999999999e-05 2024-09-21T10:29:57.800 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.800 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:57.800 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.800 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.800 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:213040 9.1 9:8e7e94d6:::benchmark_data_smithi098_254407_object80309:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:29:57.800 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.575656+0000", 2024-09-21T10:29:57.800 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 279.10586188500002, 2024-09-21T10:29:57.800 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.86753614499999998, 2024-09-21T10:29:57.800 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:57.800 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:57.801 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:57.801 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:57.801 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:57.801 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 213040 2024-09-21T10:29:57.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.801 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:57.801 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.801 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:57.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575656+0000", 2024-09-21T10:29:57.801 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.801 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.801 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:57.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575656+0000", 2024-09-21T10:29:57.802 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.802 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.802 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:57.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575654+0000", 2024-09-21T10:29:57.802 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:57.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.802 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.802 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:57.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575664+0000", 2024-09-21T10:29:57.802 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7340000000000003e-06 2024-09-21T10:29:57.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.802 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.802 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:57.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575682+0000", 2024-09-21T10:29:57.803 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8410000000000002e-05 2024-09-21T10:29:57.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.803 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.803 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:57.803 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575691+0000", 2024-09-21T10:29:57.803 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3559999999999993e-06 2024-09-21T10:29:57.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.803 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.803 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.803 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575742+0000", 2024-09-21T10:29:57.803 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0698999999999997e-05 2024-09-21T10:29:57.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.803 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.804 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:57.804 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575752+0000", 2024-09-21T10:29:57.804 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0288999999999999e-05 2024-09-21T10:29:57.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.804 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.804 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.804 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433825+0000", 2024-09-21T10:29:57.804 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.85807313799999996 2024-09-21T10:29:57.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.804 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.804 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:57.804 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433937+0000", 2024-09-21T10:29:57.804 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011184900000000001 2024-09-21T10:29:57.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.804 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.805 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:29:57.805 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.434061+0000", 2024-09-21T10:29:57.805 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012429100000000001 2024-09-21T10:29:57.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.805 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.805 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:57.805 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440716+0000", 2024-09-21T10:29:57.805 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006655134 2024-09-21T10:29:57.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.805 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.805 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.805 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442806+0000", 2024-09-21T10:29:57.805 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002089835 2024-09-21T10:29:57.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443118+0000", 2024-09-21T10:29:57.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00031147500000000002 2024-09-21T10:29:57.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:57.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443158+0000", 2024-09-21T10:29:57.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0670000000000002e-05 2024-09-21T10:29:57.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:57.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443192+0000", 2024-09-21T10:29:57.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4109999999999997e-05 2024-09-21T10:29:57.806 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.807 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:57.807 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.807 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:213046 9.1 9:8035648d:::benchmark_data_smithi098_254407_object80315:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:29:57.807 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.597939+0000", 2024-09-21T10:29:57.807 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 279.083578961, 2024-09-21T10:29:57.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.84537171499999997, 2024-09-21T10:29:57.807 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:57.807 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:57.807 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:57.807 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:29:57.807 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:29:57.807 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 213046 2024-09-21T10:29:57.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.808 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:57.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:57.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.597939+0000", 2024-09-21T10:29:57.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:57.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.597939+0000", 2024-09-21T10:29:57.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:57.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.597938+0000", 2024-09-21T10:29:57.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:57.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.809 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:57.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.597945+0000", 2024-09-21T10:29:57.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5959999999999998e-06 2024-09-21T10:29:57.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.809 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:57.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.597959+0000", 2024-09-21T10:29:57.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3986e-05 2024-09-21T10:29:57.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.809 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:57.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.597964+0000", 2024-09-21T10:29:57.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0289999999999996e-06 2024-09-21T10:29:57.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.810 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.810 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.810 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.598015+0000", 2024-09-21T10:29:57.810 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0376e-05 2024-09-21T10:29:57.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.810 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.810 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:29:57.810 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.598024+0000", 2024-09-21T10:29:57.810 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8899999999999996e-06 2024-09-21T10:29:57.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.810 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.810 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.811 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.437180+0000", 2024-09-21T10:29:57.811 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83915633099999998 2024-09-21T10:29:57.811 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.811 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.811 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:57.811 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.437294+0000", 2024-09-21T10:29:57.811 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114388 2024-09-21T10:29:57.811 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.811 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.811 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:29:57.811 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.437423+0000", 2024-09-21T10:29:57.811 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000128768 2024-09-21T10:29:57.812 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.812 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.812 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:57.812 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440953+0000", 2024-09-21T10:29:57.812 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0035302250000000001 2024-09-21T10:29:57.812 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.812 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.812 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.812 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442834+0000", 2024-09-21T10:29:57.812 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0018801919999999999 2024-09-21T10:29:57.812 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.812 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.812 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.813 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443242+0000", 2024-09-21T10:29:57.813 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00040894199999999998 2024-09-21T10:29:57.813 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.813 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.813 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:57.813 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443282+0000", 2024-09-21T10:29:57.813 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9504e-05 2024-09-21T10:29:57.813 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.813 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.813 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:57.813 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443311+0000", 2024-09-21T10:29:57.813 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8816999999999999e-05 2024-09-21T10:29:57.813 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.813 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:57.813 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.813 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.814 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.814 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46322.0:180590 10.b 10:d1a46ba4:::benchmark_data_smithi098_281896_object55565:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-21T10:29:57.814 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:29:38.404835+0000", 2024-09-21T10:29:57.814 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 19.276683352999999, 2024-09-21T10:29:57.814 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16545469700000001, 2024-09-21T10:29:57.814 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:57.814 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:57.814 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:57.814 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46322", 2024-09-21T10:29:57.814 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:29:57.814 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 180590 2024-09-21T10:29:57.814 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.814 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:57.814 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.815 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:57.815 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.404835+0000", 2024-09-21T10:29:57.815 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.815 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.815 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:57.815 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.404835+0000", 2024-09-21T10:29:57.815 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.815 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.815 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:57.815 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.404834+0000", 2024-09-21T10:29:57.815 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:29:57.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.816 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.816 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:57.816 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.404842+0000", 2024-09-21T10:29:57.816 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6539999999999999e-06 2024-09-21T10:29:57.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.816 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.816 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:57.816 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.404855+0000", 2024-09-21T10:29:57.816 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3716e-05 2024-09-21T10:29:57.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.816 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.816 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:57.816 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.404862+0000", 2024-09-21T10:29:57.817 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.922e-06 2024-09-21T10:29:57.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.817 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.817 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.404916+0000", 2024-09-21T10:29:57.817 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4187999999999997e-05 2024-09-21T10:29:57.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.817 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.817 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:57.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.405026+0000", 2024-09-21T10:29:57.817 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010914 2024-09-21T10:29:57.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.817 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.817 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:29:57.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.405103+0000", 2024-09-21T10:29:57.818 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7511999999999999e-05 2024-09-21T10:29:57.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.818 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.818 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:57.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.412556+0000", 2024-09-21T10:29:57.818 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0074533359999999996 2024-09-21T10:29:57.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.818 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.818 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.415354+0000", 2024-09-21T10:29:57.818 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002797669 2024-09-21T10:29:57.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.818 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.818 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.570198+0000", 2024-09-21T10:29:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.15484372399999999 2024-09-21T10:29:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.570248+0000", 2024-09-21T10:29:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0430999999999999e-05 2024-09-21T10:29:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.570289+0000", 2024-09-21T10:29:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1165999999999999e-05 2024-09-21T10:29:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.819 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46322.0:180591 10.b 10:db5bbccd:::benchmark_data_smithi098_281896_object55566:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-21T10:29:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:29:38.405530+0000", 2024-09-21T10:29:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 19.275988276, 2024-09-21T10:29:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16484871000000001, 2024-09-21T10:29:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:29:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:29:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46322", 2024-09-21T10:29:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:29:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 180591 2024-09-21T10:29:57.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.821 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:29:57.821 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.821 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:29:57.821 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.405530+0000", 2024-09-21T10:29:57.821 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.821 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.821 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.821 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:29:57.821 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.405530+0000", 2024-09-21T10:29:57.821 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:29:57.821 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.821 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.821 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:29:57.821 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.405529+0000", 2024-09-21T10:29:57.821 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:29:57.822 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.822 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.822 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:29:57.822 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.405534+0000", 2024-09-21T10:29:57.822 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3519999999999997e-06 2024-09-21T10:29:57.822 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.822 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.822 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:29:57.822 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.405548+0000", 2024-09-21T10:29:57.822 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.342e-05 2024-09-21T10:29:57.822 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.822 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.822 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:29:57.822 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.405553+0000", 2024-09-21T10:29:57.823 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.604e-06 2024-09-21T10:29:57.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.823 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.823 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:29:57.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.408376+0000", 2024-09-21T10:29:57.823 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002822256 2024-09-21T10:29:57.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.823 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.823 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:29:57.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.408493+0000", 2024-09-21T10:29:57.823 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011741 2024-09-21T10:29:57.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.823 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.823 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:29:57.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.408609+0000", 2024-09-21T10:29:57.824 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011647899999999999 2024-09-21T10:29:57.824 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.824 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.824 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:29:57.824 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.412765+0000", 2024-09-21T10:29:57.824 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041552760000000003 2024-09-21T10:29:57.824 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.824 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.824 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.824 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.415405+0000", 2024-09-21T10:29:57.824 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026403809999999998 2024-09-21T10:29:57.824 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.824 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.824 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:57.824 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.570326+0000", 2024-09-21T10:29:57.825 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.15492055599999999 2024-09-21T10:29:57.825 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.825 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.825 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:29:57.825 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.570357+0000", 2024-09-21T10:29:57.825 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1127999999999997e-05 2024-09-21T10:29:57.825 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:29:57.825 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:29:57.825 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:29:57.825 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:38.570378+0000", 2024-09-21T10:29:57.825 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1718000000000001e-05 2024-09-21T10:29:57.825 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.825 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:57.825 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.825 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:29:57.826 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:29:57.826 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:29:57.845 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:57.868 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:57.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:57.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:57.969 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:58.012 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:58.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:58.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:58.113 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:58.154 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:58.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:58.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:58.255 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:58.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:58.291 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:58.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:58.391 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:58.426 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:58.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:58.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:58.527 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:58.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:58.548 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:58.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:58.640 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:29:58.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:58.683 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:58.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:58.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:58.786 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:58.825 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:58.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:58.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:58.926 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:58.980 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:58.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:58.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:59.034 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:29:59.083 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:59.118 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:59.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:59.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:59.219 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:59.254 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:59.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:59.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:59.356 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:59.379 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:59.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:59.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:59.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:29:59 smithi196 ceph-mon[84784]: pgmap v1300: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 513 op/s; 16609/176787 objects degraded (9.395%); 10401/176787 objects misplaced (5.883%); 728 KiB/s, 11 objects/s recovering 2024-09-21T10:29:59.481 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:59.503 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:59.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:59.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:59.604 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:59.636 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:59.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:59.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:59.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:29:59 smithi098 ceph-mon[97580]: pgmap v1300: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 513 op/s; 16609/176787 objects degraded (9.395%); 10401/176787 objects misplaced (5.883%); 728 KiB/s, 11 objects/s recovering 2024-09-21T10:29:59.705 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:29:59 smithi098 ceph-mon[100914]: pgmap v1300: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 513 op/s; 16609/176787 objects degraded (9.395%); 10401/176787 objects misplaced (5.883%); 728 KiB/s, 11 objects/s recovering 2024-09-21T10:29:59.737 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:59.765 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:29:59.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:59.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:59.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:59.902 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:29:59.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:59.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:00.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:00.051 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:00.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:00.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:00.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:00.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:00.194 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:00.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:00.295 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:00.314 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:30:00.320 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:30:00.338 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:00.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:00.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:00.439 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:00.460 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:00.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:00.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:00.517 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:00 smithi098 ceph-mon[97580]: Health detail: HEALTH_WARN Degraded data redundancy: 16609/176787 objects degraded (9.395%), 3 pgs degraded, 3 pgs undersized 2024-09-21T10:30:00.517 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:00 smithi098 ceph-mon[97580]: [WRN] PG_DEGRADED: Degraded data redundancy: 16609/176787 objects degraded (9.395%), 3 pgs degraded, 3 pgs undersized 2024-09-21T10:30:00.517 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:00 smithi098 ceph-mon[97580]: pg 10.3 is stuck undersized for 116s, current state active+recovery_wait+undersized+degraded+remapped, last acting [2,5] 2024-09-21T10:30:00.517 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:00 smithi098 ceph-mon[97580]: pg 10.4 is stuck undersized for 116s, current state active+recovering+undersized+degraded+remapped, last acting [0,4] 2024-09-21T10:30:00.517 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:00 smithi098 ceph-mon[97580]: pg 10.5 is stuck undersized for 116s, current state active+recovery_wait+undersized+degraded+remapped, last acting [2,0] 2024-09-21T10:30:00.517 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:00 smithi098 ceph-mon[100914]: Health detail: HEALTH_WARN Degraded data redundancy: 16609/176787 objects degraded (9.395%), 3 pgs degraded, 3 pgs undersized 2024-09-21T10:30:00.517 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:00 smithi098 ceph-mon[100914]: [WRN] PG_DEGRADED: Degraded data redundancy: 16609/176787 objects degraded (9.395%), 3 pgs degraded, 3 pgs undersized 2024-09-21T10:30:00.517 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:00 smithi098 ceph-mon[100914]: pg 10.3 is stuck undersized for 116s, current state active+recovery_wait+undersized+degraded+remapped, last acting [2,5] 2024-09-21T10:30:00.517 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:00 smithi098 ceph-mon[100914]: pg 10.4 is stuck undersized for 116s, current state active+recovering+undersized+degraded+remapped, last acting [0,4] 2024-09-21T10:30:00.517 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:00 smithi098 ceph-mon[100914]: pg 10.5 is stuck undersized for 116s, current state active+recovery_wait+undersized+degraded+remapped, last acting [2,0] 2024-09-21T10:30:00.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:00 smithi196 ceph-mon[84784]: Health detail: HEALTH_WARN Degraded data redundancy: 16609/176787 objects degraded (9.395%), 3 pgs degraded, 3 pgs undersized 2024-09-21T10:30:00.532 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:30:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:30:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:30:00.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:00.599 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:00.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:00.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:00.701 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:00.723 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:00.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:00.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:00.825 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:00.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:30:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:30:00] "GET /metrics HTTP/1.1" 200 36615 "" "Prometheus/2.51.0" 2024-09-21T10:30:00.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:00.853 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:00.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:00 smithi196 ceph-mon[84784]: [WRN] PG_DEGRADED: Degraded data redundancy: 16609/176787 objects degraded (9.395%), 3 pgs degraded, 3 pgs undersized 2024-09-21T10:30:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:00 smithi196 ceph-mon[84784]: pg 10.3 is stuck undersized for 116s, current state active+recovery_wait+undersized+degraded+remapped, last acting [2,5] 2024-09-21T10:30:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:00 smithi196 ceph-mon[84784]: pg 10.4 is stuck undersized for 116s, current state active+recovering+undersized+degraded+remapped, last acting [0,4] 2024-09-21T10:30:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:00 smithi196 ceph-mon[84784]: pg 10.5 is stuck undersized for 116s, current state active+recovery_wait+undersized+degraded+remapped, last acting [2,0] 2024-09-21T10:30:00.954 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:00.995 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:00.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:00.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:01.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:01.144 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:01.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:01.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:01.246 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:01.273 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:01.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:01.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:01.374 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:01.404 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:01.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:01.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:01.505 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:01.535 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:01.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:01.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:01 smithi098 ceph-mon[97580]: pgmap v1301: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 698 op/s; 16731/168345 objects degraded (9.939%); 9771/168345 objects misplaced (5.804%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:30:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:01 smithi098 ceph-mon[97580]: from='client.36573 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:30:01.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:01 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 16731/168345 objects degraded (9.939%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:01.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:01 smithi098 ceph-mon[100914]: pgmap v1301: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 698 op/s; 16731/168345 objects degraded (9.939%); 9771/168345 objects misplaced (5.804%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:30:01.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:01 smithi098 ceph-mon[100914]: from='client.36573 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:30:01.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:01 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 16731/168345 objects degraded (9.939%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:01.636 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:01.662 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:01.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:01.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:01.741 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:01 smithi196 ceph-mon[84784]: pgmap v1301: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 698 op/s; 16731/168345 objects degraded (9.939%); 9771/168345 objects misplaced (5.804%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:30:01.741 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:01 smithi196 ceph-mon[84784]: from='client.36573 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:30:01.741 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:01 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 16731/168345 objects degraded (9.939%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:01.764 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:01.771 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1301,"stamp":"2024-09-21T10:29:59.607472+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3680233867,"num_objects":56115,"num_object_clones":0,"num_object_copies":168345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16731,"num_objects_misplaced":9771,"num_objects_unfound":0,"num_objects_dirty":56115,"num_whiteouts":0,"num_read":6012,"num_read_kb":13076,"num_write":194879,"num_write_kb":8014180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1871,"num_bytes_recovered":115538898,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129190,"ondisk_log_size":129190,"up":297,"acting":294,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":302,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":18187160,"kb_used_data":11712536,"kb_used_omap":35,"kb_used_meta":6474332,"kb_avail":731774056,"statfs":{"total":767960285184,"available":749336633344,"internally_reserved":0,"allocated":11993636864,"data_stored":11976342942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36353,"internal_metadata":6629716479},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,4,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-453640192,"num_objects":-6922,"num_object_clones":0,"num_object_copies":-20766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":237,"num_objects_misplaced":-1268,"num_objects_unfound":0,"num_objects_dirty":-6922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6922,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":11927552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.904306"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":48,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.389471+0000","last_change":"2024-09-21T10:28:00.289895+0000","last_active":"2024-09-21T10:29:52.389471+0000","last_peered":"2024-09-21T10:29:52.389471+0000","last_clean":"2024-09-21T10:29:52.389471+0000","last_became_active":"2024-09-21T10:28:00.289622+0000","last_became_peered":"2024-09-21T10:28:00.289622+0000","last_unstale":"2024-09-21T10:29:52.389471+0000","last_undegraded":"2024-09-21T10:29:52.389471+0000","last_fullsized":"2024-09-21T10:29:52.389471+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:35:31.514813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":409,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.704385+0000","last_change":"2024-09-21T10:24:44.305198+0000","last_active":"2024-09-21T10:29:49.704385+0000","last_peered":"2024-09-21T10:29:49.704385+0000","last_clean":"2024-09-21T10:29:49.704385+0000","last_became_active":"2024-09-21T10:24:44.297653+0000","last_became_peered":"2024-09-21T10:24:44.297653+0000","last_unstale":"2024-09-21T10:29:49.704385+0000","last_undegraded":"2024-09-21T10:29:49.704385+0000","last_fullsized":"2024-09-21T10:29:49.704385+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:19.320833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":246,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.713148+0000","last_change":"2024-09-21T10:28:00.294917+0000","last_active":"2024-09-21T10:29:49.713148+0000","last_peered":"2024-09-21T10:29:49.713148+0000","last_clean":"2024-09-21T10:29:49.713148+0000","last_became_active":"2024-09-21T10:28:00.294134+0000","last_became_peered":"2024-09-21T10:28:00.294134+0000","last_unstale":"2024-09-21T10:29:49.713148+0000","last_undegraded":"2024-09-21T10:29:49.713148+0000","last_fullsized":"2024-09-21T10:29:49.713148+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:29:05.780212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":195,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.703697+0000","last_change":"2024-09-21T10:28:00.287636+0000","last_active":"2024-09-21T10:29:49.703697+0000","last_peered":"2024-09-21T10:29:49.703697+0000","last_clean":"2024-09-21T10:29:49.703697+0000","last_became_active":"2024-09-21T10:28:00.287308+0000","last_became_peered":"2024-09-21T10:28:00.287308+0000","last_unstale":"2024-09-21T10:29:49.703697+0000","last_undegraded":"2024-09-21T10:29:49.703697+0000","last_fullsized":"2024-09-21T10:29:49.703697+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:41:46.132365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":644,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.383785+0000","last_change":"2024-09-21T10:28:00.297783+0000","last_active":"2024-09-21T10:29:52.383785+0000","last_peered":"2024-09-21T10:29:52.383785+0000","last_clean":"2024-09-21T10:29:52.383785+0000","last_became_active":"2024-09-21T10:28:00.297024+0000","last_became_peered":"2024-09-21T10:28:00.297024+0000","last_unstale":"2024-09-21T10:29:52.383785+0000","last_undegraded":"2024-09-21T10:29:52.383785+0000","last_fullsized":"2024-09-21T10:29:52.383785+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:24.382523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":68,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.697071+0000","last_change":"2024-09-21T10:24:44.304399+0000","last_active":"2024-09-21T10:29:49.697071+0000","last_peered":"2024-09-21T10:29:49.697071+0000","last_clean":"2024-09-21T10:29:49.697071+0000","last_became_active":"2024-09-21T10:24:44.303437+0000","last_became_peered":"2024-09-21T10:24:44.303437+0000","last_unstale":"2024-09-21T10:29:49.697071+0000","last_undegraded":"2024-09-21T10:29:49.697071+0000","last_fullsized":"2024-09-21T10:29:49.697071+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:32:56.638481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":48,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.393742+0000","last_change":"2024-09-21T10:28:00.293505+0000","last_active":"2024-09-21T10:29:52.393742+0000","last_peered":"2024-09-21T10:29:52.393742+0000","last_clean":"2024-09-21T10:29:52.393742+0000","last_became_active":"2024-09-21T10:28:00.293173+0000","last_became_peered":"2024-09-21T10:28:00.293173+0000","last_unstale":"2024-09-21T10:29:52.393742+0000","last_undegraded":"2024-09-21T10:29:52.393742+0000","last_fullsized":"2024-09-21T10:29:52.393742+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:16.878847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":351,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.707444+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:29:49.707444+0000","last_peered":"2024-09-21T10:29:49.707444+0000","last_clean":"2024-09-21T10:29:49.707444+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:29:49.707444+0000","last_undegraded":"2024-09-21T10:29:49.707444+0000","last_fullsized":"2024-09-21T10:29:49.707444+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"379'12260","reported_seq":12335,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:59.248772+0000","last_change":"2024-09-21T10:26:36.989699+0000","last_active":"2024-09-21T10:29:59.248772+0000","last_peered":"2024-09-21T10:29:59.248772+0000","last_clean":"2024-09-21T10:29:59.248772+0000","last_became_active":"2024-09-21T10:26:36.989318+0000","last_became_peered":"2024-09-21T10:26:36.989318+0000","last_unstale":"2024-09-21T10:29:59.248772+0000","last_undegraded":"2024-09-21T10:29:59.248772+0000","last_fullsized":"2024-09-21T10:29:59.248772+0000","mapping_epoch":348,"log_start":"356'4900","ondisk_log_start":"356'4900","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7360,"log_dups_size":0,"ondisk_log_size":7360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:54:53.567344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222691328,"num_objects":3398,"num_object_clones":0,"num_object_copies":10194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12260,"num_write_kb":501056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":400,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.704738+0000","last_change":"2024-09-21T10:24:44.325196+0000","last_active":"2024-09-21T10:29:49.704738+0000","last_peered":"2024-09-21T10:29:49.704738+0000","last_clean":"2024-09-21T10:29:49.704738+0000","last_became_active":"2024-09-21T10:24:44.299614+0000","last_became_peered":"2024-09-21T10:24:44.299614+0000","last_unstale":"2024-09-21T10:29:49.704738+0000","last_undegraded":"2024-09-21T10:29:49.704738+0000","last_fullsized":"2024-09-21T10:29:49.704738+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:14.558663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":132,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.712598+0000","last_change":"2024-09-21T10:26:41.305571+0000","last_active":"2024-09-21T10:29:49.712598+0000","last_peered":"2024-09-21T10:29:49.712598+0000","last_clean":"2024-09-21T10:29:49.712598+0000","last_became_active":"2024-09-21T10:26:41.305135+0000","last_became_peered":"2024-09-21T10:26:41.305135+0000","last_unstale":"2024-09-21T10:29:49.712598+0000","last_undegraded":"2024-09-21T10:29:49.712598+0000","last_fullsized":"2024-09-21T10:29:49.712598+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:49:55.452467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"379'11932","reported_seq":11982,"reported_epoch":379,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:29:58.106523+0000","last_change":"2024-09-21T10:28:01.438839+0000","last_active":"2024-09-21T10:29:58.106523+0000","last_peered":"2024-09-21T10:29:58.106523+0000","last_clean":"2024-09-21T10:27:59.232048+0000","last_became_active":"2024-09-21T10:28:01.433549+0000","last_became_peered":"2024-09-21T10:28:01.433549+0000","last_unstale":"2024-09-21T10:29:58.106523+0000","last_undegraded":"2024-09-21T10:29:58.106523+0000","last_fullsized":"2024-09-21T10:29:58.106523+0000","mapping_epoch":361,"log_start":"355'4200","ondisk_log_start":"355'4200","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7732,"log_dups_size":0,"ondisk_log_size":7732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:58.936666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229769216,"num_objects":3506,"num_object_clones":0,"num_object_copies":10518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3506,"num_objects_unfound":0,"num_objects_dirty":3506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11932,"num_write_kb":494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,5,2],"avail_no_missing":["6","2","5"],"object_location_counts":[{"shards":"2,5,6","objects":3506}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":401,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.704887+0000","last_change":"2024-09-21T10:24:44.323182+0000","last_active":"2024-09-21T10:29:49.704887+0000","last_peered":"2024-09-21T10:29:49.704887+0000","last_clean":"2024-09-21T10:29:49.704887+0000","last_became_active":"2024-09-21T10:24:44.299698+0000","last_became_peered":"2024-09-21T10:24:44.299698+0000","last_unstale":"2024-09-21T10:29:49.704887+0000","last_undegraded":"2024-09-21T10:29:49.704887+0000","last_fullsized":"2024-09-21T10:29:49.704887+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:56.571057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":78,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.697204+0000","last_change":"2024-09-21T10:28:00.281253+0000","last_active":"2024-09-21T10:29:49.697204+0000","last_peered":"2024-09-21T10:29:49.697204+0000","last_clean":"2024-09-21T10:29:49.697204+0000","last_became_active":"2024-09-21T10:28:00.280773+0000","last_became_peered":"2024-09-21T10:28:00.280773+0000","last_unstale":"2024-09-21T10:29:49.697204+0000","last_undegraded":"2024-09-21T10:29:49.697204+0000","last_fullsized":"2024-09-21T10:29:49.697204+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:32:48.557279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"379'12013","reported_seq":15507,"reported_epoch":379,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T10:29:59.409567+0000","last_change":"2024-09-21T10:28:25.623040+0000","last_active":"2024-09-21T10:29:59.409567+0000","last_peered":"2024-09-21T10:29:59.409567+0000","last_clean":"2024-09-21T10:27:59.294389+0000","last_became_active":"2024-09-21T10:28:01.608572+0000","last_became_peered":"2024-09-21T10:28:01.608572+0000","last_unstale":"2024-09-21T10:29:59.409567+0000","last_undegraded":"2024-09-21T10:28:25.614349+0000","last_fullsized":"2024-09-21T10:28:01.272643+0000","mapping_epoch":361,"log_start":"355'4600","ondisk_log_start":"355'4600","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7413,"log_dups_size":0,"ondisk_log_size":7413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227082240,"num_objects":3465,"num_object_clones":0,"num_object_copies":10395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2568,"num_objects_misplaced":2752,"num_objects_unfound":0,"num_objects_dirty":3465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12013,"num_write_kb":495296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1708,"num_bytes_recovered":111935488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":2752},{"shards":"0,4","objects":-1855}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":327,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.704207+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:29:49.704207+0000","last_peered":"2024-09-21T10:29:49.704207+0000","last_clean":"2024-09-21T10:29:49.704207+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:29:49.704207+0000","last_undegraded":"2024-09-21T10:29:49.704207+0000","last_fullsized":"2024-09-21T10:29:49.704207+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":342,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.707865+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:29:49.707865+0000","last_peered":"2024-09-21T10:29:49.707865+0000","last_clean":"2024-09-21T10:29:49.707865+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:29:49.707865+0000","last_undegraded":"2024-09-21T10:29:49.707865+0000","last_fullsized":"2024-09-21T10:29:49.707865+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"379'11812","reported_seq":11886,"reported_epoch":379,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:29:55.356871+0000","last_change":"2024-09-21T10:28:01.644360+0000","last_active":"2024-09-21T10:29:55.356871+0000","last_peered":"2024-09-21T10:29:55.356871+0000","last_clean":"2024-09-21T10:27:59.276032+0000","last_became_active":"2024-09-21T10:28:01.624219+0000","last_became_peered":"2024-09-21T10:28:01.624219+0000","last_unstale":"2024-09-21T10:29:55.356871+0000","last_undegraded":"2024-09-21T10:28:01.350374+0000","last_fullsized":"2024-09-21T10:28:01.295088+0000","mapping_epoch":361,"log_start":"354'2400","ondisk_log_start":"354'2400","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":9412,"log_dups_size":0,"ondisk_log_size":9412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233701376,"num_objects":3566,"num_object_clones":0,"num_object_copies":10698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11812,"num_write_kb":492096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0],"avail_no_missing":["2","0","7"],"object_location_counts":[{"shards":"0,2","objects":-3448},{"shards":"0,2,7","objects":2891}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":48,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.389388+0000","last_change":"2024-09-21T10:28:00.288968+0000","last_active":"2024-09-21T10:29:52.389388+0000","last_peered":"2024-09-21T10:29:52.389388+0000","last_clean":"2024-09-21T10:29:52.389388+0000","last_became_active":"2024-09-21T10:28:00.288244+0000","last_became_peered":"2024-09-21T10:28:00.288244+0000","last_unstale":"2024-09-21T10:29:52.389388+0000","last_undegraded":"2024-09-21T10:29:52.389388+0000","last_fullsized":"2024-09-21T10:29:52.389388+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:42.089701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":666,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.383103+0000","last_change":"2024-09-21T10:24:44.299417+0000","last_active":"2024-09-21T10:29:52.383103+0000","last_peered":"2024-09-21T10:29:52.383103+0000","last_clean":"2024-09-21T10:29:52.383103+0000","last_became_active":"2024-09-21T10:24:44.299116+0000","last_became_peered":"2024-09-21T10:24:44.299116+0000","last_unstale":"2024-09-21T10:29:52.383103+0000","last_undegraded":"2024-09-21T10:29:52.383103+0000","last_fullsized":"2024-09-21T10:29:52.383103+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:44.524841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"379'12278","reported_seq":12353,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:57.374611+0000","last_change":"2024-09-21T10:26:36.196642+0000","last_active":"2024-09-21T10:29:57.374611+0000","last_peered":"2024-09-21T10:29:57.374611+0000","last_clean":"2024-09-21T10:29:57.374611+0000","last_became_active":"2024-09-21T10:26:36.196357+0000","last_became_peered":"2024-09-21T10:26:36.196357+0000","last_unstale":"2024-09-21T10:29:57.374611+0000","last_undegraded":"2024-09-21T10:29:57.374611+0000","last_fullsized":"2024-09-21T10:29:57.374611+0000","mapping_epoch":348,"log_start":"355'3700","ondisk_log_start":"355'3700","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":8578,"log_dups_size":0,"ondisk_log_size":8578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:54:19.391031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235012096,"num_objects":3586,"num_object_clones":0,"num_object_copies":10758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12278,"num_write_kb":507648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"82'100","reported_seq":500,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.703845+0000","last_change":"2024-09-21T10:28:00.285919+0000","last_active":"2024-09-21T10:29:49.703845+0000","last_peered":"2024-09-21T10:29:49.703845+0000","last_clean":"2024-09-21T10:29:49.703845+0000","last_became_active":"2024-09-21T10:28:00.285324+0000","last_became_peered":"2024-09-21T10:28:00.285324+0000","last_unstale":"2024-09-21T10:29:49.703845+0000","last_undegraded":"2024-09-21T10:29:49.703845+0000","last_fullsized":"2024-09-21T10:29:49.703845+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.728195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.728195+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:00.841610+0000","scrub_duration":0.0069698140000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":52,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.389000+0000","last_change":"2024-09-21T10:28:00.303125+0000","last_active":"2024-09-21T10:29:52.389000+0000","last_peered":"2024-09-21T10:29:52.389000+0000","last_clean":"2024-09-21T10:29:52.389000+0000","last_became_active":"2024-09-21T10:28:00.293684+0000","last_became_peered":"2024-09-21T10:28:00.293684+0000","last_unstale":"2024-09-21T10:29:52.389000+0000","last_undegraded":"2024-09-21T10:29:52.389000+0000","last_fullsized":"2024-09-21T10:29:52.389000+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:04.495913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":78,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.697177+0000","last_change":"2024-09-21T10:28:00.283456+0000","last_active":"2024-09-21T10:29:49.697177+0000","last_peered":"2024-09-21T10:29:49.697177+0000","last_clean":"2024-09-21T10:29:49.697177+0000","last_became_active":"2024-09-21T10:28:00.282881+0000","last_became_peered":"2024-09-21T10:28:00.282881+0000","last_unstale":"2024-09-21T10:29:49.697177+0000","last_undegraded":"2024-09-21T10:29:49.697177+0000","last_fullsized":"2024-09-21T10:29:49.697177+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:43.681481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"379'12122","reported_seq":12200,"reported_epoch":379,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:29:55.347121+0000","last_change":"2024-09-21T10:28:01.680872+0000","last_active":"2024-09-21T10:29:55.347121+0000","last_peered":"2024-09-21T10:29:55.347121+0000","last_clean":"2024-09-21T10:27:59.257182+0000","last_became_active":"2024-09-21T10:28:01.651875+0000","last_became_peered":"2024-09-21T10:28:01.651875+0000","last_unstale":"2024-09-21T10:29:55.347121+0000","last_undegraded":"2024-09-21T10:28:01.349133+0000","last_fullsized":"2024-09-21T10:28:01.272587+0000","mapping_epoch":361,"log_start":"354'2700","ondisk_log_start":"354'2700","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":9422,"log_dups_size":0,"ondisk_log_size":9422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237371392,"num_objects":3622,"num_object_clones":0,"num_object_copies":10866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":503808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5],"avail_no_missing":["2","4","5"],"object_location_counts":[{"shards":"2,4,5","objects":2899},{"shards":"2,5","objects":-3527}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'100","reported_seq":169,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.373920+0000","last_change":"2024-09-21T10:28:00.301695+0000","last_active":"2024-09-21T10:29:52.373920+0000","last_peered":"2024-09-21T10:29:52.373920+0000","last_clean":"2024-09-21T10:29:52.373920+0000","last_became_active":"2024-09-21T10:28:00.289856+0000","last_became_peered":"2024-09-21T10:28:00.289856+0000","last_unstale":"2024-09-21T10:29:52.373920+0000","last_undegraded":"2024-09-21T10:29:52.373920+0000","last_fullsized":"2024-09-21T10:29:52.373920+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:27.234001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:27.234001+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:02.389036+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":393,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.708049+0000","last_change":"2024-09-21T10:24:44.309470+0000","last_active":"2024-09-21T10:29:49.708049+0000","last_peered":"2024-09-21T10:29:49.708049+0000","last_clean":"2024-09-21T10:29:49.708049+0000","last_became_active":"2024-09-21T10:24:44.308836+0000","last_became_peered":"2024-09-21T10:24:44.308836+0000","last_unstale":"2024-09-21T10:29:49.708049+0000","last_undegraded":"2024-09-21T10:29:49.708049+0000","last_fullsized":"2024-09-21T10:29:49.708049+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:09:44.622418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":68,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.697566+0000","last_change":"2024-09-21T10:24:44.305135+0000","last_active":"2024-09-21T10:29:49.697566+0000","last_peered":"2024-09-21T10:29:49.697566+0000","last_clean":"2024-09-21T10:29:49.697566+0000","last_became_active":"2024-09-21T10:24:44.304839+0000","last_became_peered":"2024-09-21T10:24:44.304839+0000","last_unstale":"2024-09-21T10:29:49.697566+0000","last_undegraded":"2024-09-21T10:29:49.697566+0000","last_fullsized":"2024-09-21T10:29:49.697566+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:02.264404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"379'12088","reported_seq":12127,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:58.065305+0000","last_change":"2024-09-21T10:26:36.194994+0000","last_active":"2024-09-21T10:29:58.065305+0000","last_peered":"2024-09-21T10:29:58.065305+0000","last_clean":"2024-09-21T10:29:58.065305+0000","last_became_active":"2024-09-21T10:26:36.194404+0000","last_became_peered":"2024-09-21T10:26:36.194404+0000","last_unstale":"2024-09-21T10:29:58.065305+0000","last_undegraded":"2024-09-21T10:29:58.065305+0000","last_fullsized":"2024-09-21T10:29:58.065305+0000","mapping_epoch":348,"log_start":"355'4300","ondisk_log_start":"355'4300","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7788,"log_dups_size":0,"ondisk_log_size":7788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:10:51.551059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224657408,"num_objects":3428,"num_object_clones":0,"num_object_copies":10284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12088,"num_write_kb":496512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":97,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.382601+0000","last_change":"2024-09-21T10:25:39.872680+0000","last_active":"2024-09-21T10:29:52.382601+0000","last_peered":"2024-09-21T10:29:52.382601+0000","last_clean":"2024-09-21T10:29:52.382601+0000","last_became_active":"2024-09-21T10:25:39.872312+0000","last_became_peered":"2024-09-21T10:25:39.872312+0000","last_unstale":"2024-09-21T10:29:52.382601+0000","last_undegraded":"2024-09-21T10:29:52.382601+0000","last_fullsized":"2024-09-21T10:29:52.382601+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:54.942046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":68,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.698510+0000","last_change":"2024-09-21T10:24:44.310484+0000","last_active":"2024-09-21T10:29:49.698510+0000","last_peered":"2024-09-21T10:29:49.698510+0000","last_clean":"2024-09-21T10:29:49.698510+0000","last_became_active":"2024-09-21T10:24:44.307209+0000","last_became_peered":"2024-09-21T10:24:44.307209+0000","last_unstale":"2024-09-21T10:29:49.698510+0000","last_undegraded":"2024-09-21T10:29:49.698510+0000","last_fullsized":"2024-09-21T10:29:49.698510+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:48:36.767539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"379'12228","reported_seq":12267,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:58.101034+0000","last_change":"2024-09-21T10:26:36.196826+0000","last_active":"2024-09-21T10:29:58.101034+0000","last_peered":"2024-09-21T10:29:58.101034+0000","last_clean":"2024-09-21T10:29:58.101034+0000","last_became_active":"2024-09-21T10:26:36.196355+0000","last_became_peered":"2024-09-21T10:26:36.196355+0000","last_unstale":"2024-09-21T10:29:58.101034+0000","last_undegraded":"2024-09-21T10:29:58.101034+0000","last_fullsized":"2024-09-21T10:29:58.101034+0000","mapping_epoch":348,"log_start":"357'5500","ondisk_log_start":"357'5500","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":6728,"log_dups_size":0,"ondisk_log_size":6728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:58.869309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223608832,"num_objects":3412,"num_object_clones":0,"num_object_copies":10236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12228,"num_write_kb":500480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'100","reported_seq":153,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.374033+0000","last_change":"2024-09-21T10:23:26.197358+0000","last_active":"2024-09-21T10:29:52.374033+0000","last_peered":"2024-09-21T10:29:52.374033+0000","last_clean":"2024-09-21T10:29:52.374033+0000","last_became_active":"2024-09-21T10:23:20.695174+0000","last_became_peered":"2024-09-21T10:23:20.695174+0000","last_unstale":"2024-09-21T10:29:52.374033+0000","last_undegraded":"2024-09-21T10:29:52.374033+0000","last_fullsized":"2024-09-21T10:29:52.374033+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:26.197291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:26.197291+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:33.961031+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":48,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.389076+0000","last_change":"2024-09-21T10:28:00.287560+0000","last_active":"2024-09-21T10:29:52.389076+0000","last_peered":"2024-09-21T10:29:52.389076+0000","last_clean":"2024-09-21T10:29:52.389076+0000","last_became_active":"2024-09-21T10:28:00.285076+0000","last_became_peered":"2024-09-21T10:28:00.285076+0000","last_unstale":"2024-09-21T10:29:52.389076+0000","last_undegraded":"2024-09-21T10:29:52.389076+0000","last_fullsized":"2024-09-21T10:29:52.389076+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:49.950523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":661,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.407584+0000","last_change":"2024-09-21T10:28:00.282062+0000","last_active":"2024-09-21T10:29:52.407584+0000","last_peered":"2024-09-21T10:29:52.407584+0000","last_clean":"2024-09-21T10:29:52.407584+0000","last_became_active":"2024-09-21T10:28:00.281593+0000","last_became_peered":"2024-09-21T10:28:00.281593+0000","last_unstale":"2024-09-21T10:29:52.407584+0000","last_undegraded":"2024-09-21T10:29:52.407584+0000","last_fullsized":"2024-09-21T10:29:52.407584+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:45.704840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"379'12122","reported_seq":12161,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:58.100204+0000","last_change":"2024-09-21T10:26:36.197465+0000","last_active":"2024-09-21T10:29:58.100204+0000","last_peered":"2024-09-21T10:29:58.100204+0000","last_clean":"2024-09-21T10:29:58.100204+0000","last_became_active":"2024-09-21T10:26:36.196105+0000","last_became_peered":"2024-09-21T10:26:36.196105+0000","last_unstale":"2024-09-21T10:29:58.100204+0000","last_undegraded":"2024-09-21T10:29:58.100204+0000","last_fullsized":"2024-09-21T10:29:58.100204+0000","mapping_epoch":348,"log_start":"355'4400","ondisk_log_start":"355'4400","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7722,"log_dups_size":0,"ondisk_log_size":7722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:16.586685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228458496,"num_objects":3486,"num_object_clones":0,"num_object_copies":10458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":499456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":481,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.703661+0000","last_change":"2024-09-21T10:23:18.790100+0000","last_active":"2024-09-21T10:29:49.703661+0000","last_peered":"2024-09-21T10:29:49.703661+0000","last_clean":"2024-09-21T10:29:49.703661+0000","last_became_active":"2024-09-21T10:23:14.677610+0000","last_became_peered":"2024-09-21T10:23:14.677610+0000","last_unstale":"2024-09-21T10:29:49.703661+0000","last_undegraded":"2024-09-21T10:29:49.703661+0000","last_fullsized":"2024-09-21T10:29:49.703661+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:18.790017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:18.790017+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:56.562241+0000","scrub_duration":0.0080400790000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":574,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.713331+0000","last_change":"2024-09-21T10:26:41.279374+0000","last_active":"2024-09-21T10:29:49.713331+0000","last_peered":"2024-09-21T10:29:49.713331+0000","last_clean":"2024-09-21T10:29:49.713331+0000","last_became_active":"2024-09-21T10:26:41.278994+0000","last_became_peered":"2024-09-21T10:26:41.278994+0000","last_unstale":"2024-09-21T10:29:49.713331+0000","last_undegraded":"2024-09-21T10:29:49.713331+0000","last_fullsized":"2024-09-21T10:29:49.713331+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:49:57.821175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":482,"num_read_kb":1581,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":48,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.389186+0000","last_change":"2024-09-21T10:28:00.295436+0000","last_active":"2024-09-21T10:29:52.389186+0000","last_peered":"2024-09-21T10:29:52.389186+0000","last_clean":"2024-09-21T10:29:52.389186+0000","last_became_active":"2024-09-21T10:28:00.295135+0000","last_became_peered":"2024-09-21T10:28:00.295135+0000","last_unstale":"2024-09-21T10:29:52.389186+0000","last_undegraded":"2024-09-21T10:29:52.389186+0000","last_fullsized":"2024-09-21T10:29:52.389186+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:14:03.027395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":365,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.707907+0000","last_change":"2024-09-21T10:24:44.308008+0000","last_active":"2024-09-21T10:29:49.707907+0000","last_peered":"2024-09-21T10:29:49.707907+0000","last_clean":"2024-09-21T10:29:49.707907+0000","last_became_active":"2024-09-21T10:24:44.307595+0000","last_became_peered":"2024-09-21T10:24:44.307595+0000","last_unstale":"2024-09-21T10:29:49.707907+0000","last_undegraded":"2024-09-21T10:29:49.707907+0000","last_fullsized":"2024-09-21T10:29:49.707907+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:33:49.448676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"379'12017","reported_seq":12092,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:59.311920+0000","last_change":"2024-09-21T10:26:36.197089+0000","last_active":"2024-09-21T10:29:59.311920+0000","last_peered":"2024-09-21T10:29:59.311920+0000","last_clean":"2024-09-21T10:29:59.311920+0000","last_became_active":"2024-09-21T10:26:36.196846+0000","last_became_peered":"2024-09-21T10:26:36.196846+0000","last_unstale":"2024-09-21T10:29:59.311920+0000","last_undegraded":"2024-09-21T10:29:59.311920+0000","last_fullsized":"2024-09-21T10:29:59.311920+0000","mapping_epoch":348,"log_start":"355'4600","ondisk_log_start":"355'4600","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7417,"log_dups_size":0,"ondisk_log_size":7417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:14:52.494647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226820096,"num_objects":3461,"num_object_clones":0,"num_object_copies":10383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12017,"num_write_kb":495296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"82'100","reported_seq":160,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.407729+0000","last_change":"2024-09-21T10:24:44.309154+0000","last_active":"2024-09-21T10:29:52.407729+0000","last_peered":"2024-09-21T10:29:52.407729+0000","last_clean":"2024-09-21T10:29:52.407729+0000","last_became_active":"2024-09-21T10:24:44.308330+0000","last_became_peered":"2024-09-21T10:24:44.308330+0000","last_unstale":"2024-09-21T10:29:52.407729+0000","last_undegraded":"2024-09-21T10:29:52.407729+0000","last_fullsized":"2024-09-21T10:29:52.407729+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.641718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.641718+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:10.158739+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":723,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.703588+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:29:49.703588+0000","last_peered":"2024-09-21T10:29:49.703588+0000","last_clean":"2024-09-21T10:29:49.703588+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:29:49.703588+0000","last_undegraded":"2024-09-21T10:29:49.703588+0000","last_fullsized":"2024-09-21T10:29:49.703588+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":674,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.382698+0000","last_change":"2024-09-21T10:28:00.296831+0000","last_active":"2024-09-21T10:29:52.382698+0000","last_peered":"2024-09-21T10:29:52.382698+0000","last_clean":"2024-09-21T10:29:52.382698+0000","last_became_active":"2024-09-21T10:28:00.296527+0000","last_became_peered":"2024-09-21T10:28:00.296527+0000","last_unstale":"2024-09-21T10:29:52.382698+0000","last_undegraded":"2024-09-21T10:29:52.382698+0000","last_fullsized":"2024-09-21T10:29:52.382698+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:18.024868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":642,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.383920+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:29:52.383920+0000","last_peered":"2024-09-21T10:29:52.383920+0000","last_clean":"2024-09-21T10:29:52.383920+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:29:52.383920+0000","last_undegraded":"2024-09-21T10:29:52.383920+0000","last_fullsized":"2024-09-21T10:29:52.383920+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"379'12495","reported_seq":12535,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:57.967133+0000","last_change":"2024-09-21T10:26:41.278294+0000","last_active":"2024-09-21T10:29:57.967133+0000","last_peered":"2024-09-21T10:29:57.967133+0000","last_clean":"2024-09-21T10:29:57.967133+0000","last_became_active":"2024-09-21T10:26:41.277965+0000","last_became_peered":"2024-09-21T10:26:41.277965+0000","last_unstale":"2024-09-21T10:29:57.967133+0000","last_undegraded":"2024-09-21T10:29:57.967133+0000","last_fullsized":"2024-09-21T10:29:57.967133+0000","mapping_epoch":351,"log_start":"355'2800","ondisk_log_start":"355'2800","created":348,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":9695,"log_dups_size":0,"ondisk_log_size":9695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:30.913837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233635840,"num_objects":3565,"num_object_clones":0,"num_object_copies":10695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12495,"num_write_kb":513920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"82'100","reported_seq":306,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.383317+0000","last_change":"2024-09-21T10:24:44.294460+0000","last_active":"2024-09-21T10:29:52.383317+0000","last_peered":"2024-09-21T10:29:52.383317+0000","last_clean":"2024-09-21T10:29:52.383317+0000","last_became_active":"2024-09-21T10:24:44.293972+0000","last_became_peered":"2024-09-21T10:24:44.293972+0000","last_unstale":"2024-09-21T10:29:52.383317+0000","last_undegraded":"2024-09-21T10:29:52.383317+0000","last_fullsized":"2024-09-21T10:29:52.383317+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:26.199522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:26.199522+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:26.135171+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":1452,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.376934+0000","last_change":"2024-09-21T10:28:03.346064+0000","last_active":"2024-09-21T10:29:52.376934+0000","last_peered":"2024-09-21T10:29:52.376934+0000","last_clean":"2024-09-21T10:29:52.376934+0000","last_became_active":"2024-09-21T10:28:03.345313+0000","last_became_peered":"2024-09-21T10:28:03.345313+0000","last_unstale":"2024-09-21T10:29:52.376934+0000","last_undegraded":"2024-09-21T10:29:52.376934+0000","last_fullsized":"2024-09-21T10:29:52.376934+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:32.097521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":514,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.412903+0000","last_change":"2024-09-21T10:08:18.470748+0000","last_active":"2024-09-21T10:29:52.412903+0000","last_peered":"2024-09-21T10:29:52.412903+0000","last_clean":"2024-09-21T10:29:52.412903+0000","last_became_active":"2024-09-21T10:08:18.470227+0000","last_became_peered":"2024-09-21T10:08:18.470227+0000","last_unstale":"2024-09-21T10:29:52.412903+0000","last_undegraded":"2024-09-21T10:29:52.412903+0000","last_fullsized":"2024-09-21T10:29:52.412903+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:16.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":637,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.378040+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:29:52.378040+0000","last_peered":"2024-09-21T10:29:52.378040+0000","last_clean":"2024-09-21T10:29:52.378040+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:29:52.378040+0000","last_undegraded":"2024-09-21T10:29:52.378040+0000","last_fullsized":"2024-09-21T10:29:52.378040+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"379'12309","reported_seq":12348,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:58.088470+0000","last_change":"2024-09-21T10:26:36.989641+0000","last_active":"2024-09-21T10:29:58.088470+0000","last_peered":"2024-09-21T10:29:58.088470+0000","last_clean":"2024-09-21T10:29:58.088470+0000","last_became_active":"2024-09-21T10:26:36.989340+0000","last_became_peered":"2024-09-21T10:26:36.989340+0000","last_unstale":"2024-09-21T10:29:58.088470+0000","last_undegraded":"2024-09-21T10:29:58.088470+0000","last_fullsized":"2024-09-21T10:29:58.088470+0000","mapping_epoch":348,"log_start":"355'4600","ondisk_log_start":"355'4600","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7709,"log_dups_size":0,"ondisk_log_size":7709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:50.213332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229965824,"num_objects":3509,"num_object_clones":0,"num_object_copies":10527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12309,"num_write_kb":506176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"82'100","reported_seq":299,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.382646+0000","last_change":"2024-09-21T10:23:21.277739+0000","last_active":"2024-09-21T10:29:52.382646+0000","last_peered":"2024-09-21T10:29:52.382646+0000","last_clean":"2024-09-21T10:29:52.382646+0000","last_became_active":"2024-09-21T10:23:20.689733+0000","last_became_peered":"2024-09-21T10:23:20.689733+0000","last_unstale":"2024-09-21T10:29:52.382646+0000","last_undegraded":"2024-09-21T10:29:52.382646+0000","last_fullsized":"2024-09-21T10:29:52.382646+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.277663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.277663+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:01:54.523189+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":1113,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.406580+0000","last_change":"2024-09-21T10:07:43.794767+0000","last_active":"2024-09-21T10:29:52.406580+0000","last_peered":"2024-09-21T10:29:52.406580+0000","last_clean":"2024-09-21T10:29:52.406580+0000","last_became_active":"2024-09-21T10:07:43.794062+0000","last_became_peered":"2024-09-21T10:07:43.794062+0000","last_unstale":"2024-09-21T10:29:52.406580+0000","last_undegraded":"2024-09-21T10:29:52.406580+0000","last_fullsized":"2024-09-21T10:29:52.406580+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:51.977955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":68,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.697324+0000","last_change":"2024-09-21T10:24:44.299492+0000","last_active":"2024-09-21T10:29:49.697324+0000","last_peered":"2024-09-21T10:29:49.697324+0000","last_clean":"2024-09-21T10:29:49.697324+0000","last_became_active":"2024-09-21T10:24:44.299187+0000","last_became_peered":"2024-09-21T10:24:44.299187+0000","last_unstale":"2024-09-21T10:29:49.697324+0000","last_undegraded":"2024-09-21T10:29:49.697324+0000","last_fullsized":"2024-09-21T10:29:49.697324+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:58.500386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":567,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.374334+0000","last_change":"2024-09-21T10:28:00.289453+0000","last_active":"2024-09-21T10:29:52.374334+0000","last_peered":"2024-09-21T10:29:52.374334+0000","last_clean":"2024-09-21T10:29:52.374334+0000","last_became_active":"2024-09-21T10:28:00.289030+0000","last_became_peered":"2024-09-21T10:28:00.289030+0000","last_unstale":"2024-09-21T10:29:52.374334+0000","last_undegraded":"2024-09-21T10:29:52.374334+0000","last_fullsized":"2024-09-21T10:29:52.374334+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:21.959998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"379'11995","reported_seq":12032,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:57.970072+0000","last_change":"2024-09-21T10:26:36.196471+0000","last_active":"2024-09-21T10:29:57.970072+0000","last_peered":"2024-09-21T10:29:57.970072+0000","last_clean":"2024-09-21T10:29:57.970072+0000","last_became_active":"2024-09-21T10:26:36.196167+0000","last_became_peered":"2024-09-21T10:26:36.196167+0000","last_unstale":"2024-09-21T10:29:57.970072+0000","last_undegraded":"2024-09-21T10:29:57.970072+0000","last_fullsized":"2024-09-21T10:29:57.970072+0000","mapping_epoch":348,"log_start":"355'4100","ondisk_log_start":"355'4100","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7895,"log_dups_size":0,"ondisk_log_size":7895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:16.488795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228392960,"num_objects":3485,"num_object_clones":0,"num_object_copies":10455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11995,"num_write_kb":495360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"82'100","reported_seq":243,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.708245+0000","last_change":"2024-09-21T10:28:00.294997+0000","last_active":"2024-09-21T10:29:49.708245+0000","last_peered":"2024-09-21T10:29:49.708245+0000","last_clean":"2024-09-21T10:29:49.708245+0000","last_became_active":"2024-09-21T10:28:00.294595+0000","last_became_peered":"2024-09-21T10:28:00.294595+0000","last_unstale":"2024-09-21T10:29:49.708245+0000","last_undegraded":"2024-09-21T10:29:49.708245+0000","last_fullsized":"2024-09-21T10:29:49.708245+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:30.569982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:30.569982+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:01.422939+0000","scrub_duration":0.0051309349999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":177,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.713388+0000","last_change":"2024-09-21T10:28:00.294966+0000","last_active":"2024-09-21T10:29:49.713388+0000","last_peered":"2024-09-21T10:29:49.713388+0000","last_clean":"2024-09-21T10:29:49.713388+0000","last_became_active":"2024-09-21T10:28:00.294552+0000","last_became_peered":"2024-09-21T10:28:00.294552+0000","last_unstale":"2024-09-21T10:29:49.713388+0000","last_undegraded":"2024-09-21T10:29:49.713388+0000","last_fullsized":"2024-09-21T10:29:49.713388+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:47.292125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":1220,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.407464+0000","last_change":"2024-09-21T10:08:18.468995+0000","last_active":"2024-09-21T10:29:52.407464+0000","last_peered":"2024-09-21T10:29:52.407464+0000","last_clean":"2024-09-21T10:29:52.407464+0000","last_became_active":"2024-09-21T10:08:18.468243+0000","last_became_peered":"2024-09-21T10:08:18.468243+0000","last_unstale":"2024-09-21T10:29:52.407464+0000","last_undegraded":"2024-09-21T10:29:52.407464+0000","last_fullsized":"2024-09-21T10:29:52.407464+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:45.520968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":546,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.406805+0000","last_change":"2024-09-21T10:24:44.309614+0000","last_active":"2024-09-21T10:29:52.406805+0000","last_peered":"2024-09-21T10:29:52.406805+0000","last_clean":"2024-09-21T10:29:52.406805+0000","last_became_active":"2024-09-21T10:24:44.309187+0000","last_became_peered":"2024-09-21T10:24:44.309187+0000","last_unstale":"2024-09-21T10:29:52.406805+0000","last_undegraded":"2024-09-21T10:29:52.406805+0000","last_fullsized":"2024-09-21T10:29:52.406805+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:01.074803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"379'12176","reported_seq":12216,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:57.991924+0000","last_change":"2024-09-21T10:26:36.196357+0000","last_active":"2024-09-21T10:29:57.991924+0000","last_peered":"2024-09-21T10:29:57.991924+0000","last_clean":"2024-09-21T10:29:57.991924+0000","last_became_active":"2024-09-21T10:26:36.196060+0000","last_became_peered":"2024-09-21T10:26:36.196060+0000","last_unstale":"2024-09-21T10:29:57.991924+0000","last_undegraded":"2024-09-21T10:29:57.991924+0000","last_fullsized":"2024-09-21T10:29:57.991924+0000","mapping_epoch":348,"log_start":"355'4400","ondisk_log_start":"355'4400","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7776,"log_dups_size":0,"ondisk_log_size":7776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:46:34.643144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228917272,"num_objects":3494,"num_object_clones":0,"num_object_copies":10482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3494,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12176,"num_write_kb":501377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"82'100","reported_seq":253,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.707775+0000","last_change":"2024-09-21T10:28:00.295277+0000","last_active":"2024-09-21T10:29:49.707775+0000","last_peered":"2024-09-21T10:29:49.707775+0000","last_clean":"2024-09-21T10:29:49.707775+0000","last_became_active":"2024-09-21T10:28:00.294016+0000","last_became_peered":"2024-09-21T10:28:00.294016+0000","last_unstale":"2024-09-21T10:29:49.707775+0000","last_undegraded":"2024-09-21T10:29:49.707775+0000","last_fullsized":"2024-09-21T10:29:49.707775+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:24.536711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:24.536711+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:28:49.984470+0000","scrub_duration":0.0047525650000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":515,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.407377+0000","last_change":"2024-09-21T10:08:18.469907+0000","last_active":"2024-09-21T10:29:52.407377+0000","last_peered":"2024-09-21T10:29:52.407377+0000","last_clean":"2024-09-21T10:29:52.407377+0000","last_became_active":"2024-09-21T10:08:18.469443+0000","last_became_peered":"2024-09-21T10:08:18.469443+0000","last_unstale":"2024-09-21T10:29:52.407377+0000","last_undegraded":"2024-09-21T10:29:52.407377+0000","last_fullsized":"2024-09-21T10:29:52.407377+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:15.645803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":801,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.704276+0000","last_change":"2024-09-21T10:24:44.334847+0000","last_active":"2024-09-21T10:29:49.704276+0000","last_peered":"2024-09-21T10:29:49.704276+0000","last_clean":"2024-09-21T10:29:49.704276+0000","last_became_active":"2024-09-21T10:24:44.301321+0000","last_became_peered":"2024-09-21T10:24:44.301321+0000","last_unstale":"2024-09-21T10:29:49.704276+0000","last_undegraded":"2024-09-21T10:29:49.704276+0000","last_fullsized":"2024-09-21T10:29:49.704276+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:37.628940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":53,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.713073+0000","last_change":"2024-09-21T10:26:41.278937+0000","last_active":"2024-09-21T10:29:49.713073+0000","last_peered":"2024-09-21T10:29:49.713073+0000","last_clean":"2024-09-21T10:29:49.713073+0000","last_became_active":"2024-09-21T10:26:41.277726+0000","last_became_peered":"2024-09-21T10:26:41.277726+0000","last_unstale":"2024-09-21T10:29:49.713073+0000","last_undegraded":"2024-09-21T10:29:49.713073+0000","last_fullsized":"2024-09-21T10:29:49.713073+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:04.741057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"379'12184","reported_seq":12223,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:58.041215+0000","last_change":"2024-09-21T10:26:36.196641+0000","last_active":"2024-09-21T10:29:58.041215+0000","last_peered":"2024-09-21T10:29:58.041215+0000","last_clean":"2024-09-21T10:29:58.041215+0000","last_became_active":"2024-09-21T10:26:36.196351+0000","last_became_peered":"2024-09-21T10:26:36.196351+0000","last_unstale":"2024-09-21T10:29:58.041215+0000","last_undegraded":"2024-09-21T10:29:58.041215+0000","last_fullsized":"2024-09-21T10:29:58.041215+0000","mapping_epoch":348,"log_start":"357'5500","ondisk_log_start":"357'5500","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":6684,"log_dups_size":0,"ondisk_log_size":6684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:41:54.280247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234225664,"num_objects":3574,"num_object_clones":0,"num_object_copies":10722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12184,"num_write_kb":504256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'100","reported_seq":68,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.697506+0000","last_change":"2024-09-21T10:24:44.296758+0000","last_active":"2024-09-21T10:29:49.697506+0000","last_peered":"2024-09-21T10:29:49.697506+0000","last_clean":"2024-09-21T10:29:49.697506+0000","last_became_active":"2024-09-21T10:24:44.296403+0000","last_became_peered":"2024-09-21T10:24:44.296403+0000","last_unstale":"2024-09-21T10:29:49.697506+0000","last_undegraded":"2024-09-21T10:29:49.697506+0000","last_fullsized":"2024-09-21T10:29:49.697506+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.105746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.105746+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:20:12.502081+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":52,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.393616+0000","last_change":"2024-09-21T10:28:00.300952+0000","last_active":"2024-09-21T10:29:52.393616+0000","last_peered":"2024-09-21T10:29:52.393616+0000","last_clean":"2024-09-21T10:29:52.393616+0000","last_became_active":"2024-09-21T10:28:00.293337+0000","last_became_peered":"2024-09-21T10:28:00.293337+0000","last_unstale":"2024-09-21T10:29:52.393616+0000","last_undegraded":"2024-09-21T10:29:52.393616+0000","last_fullsized":"2024-09-21T10:29:52.393616+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:58.409761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":940,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.407831+0000","last_change":"2024-09-21T10:24:46.844278+0000","last_active":"2024-09-21T10:29:52.407831+0000","last_peered":"2024-09-21T10:29:52.407831+0000","last_clean":"2024-09-21T10:29:52.407831+0000","last_became_active":"2024-09-21T10:24:46.840961+0000","last_became_peered":"2024-09-21T10:24:46.840961+0000","last_unstale":"2024-09-21T10:29:52.407831+0000","last_undegraded":"2024-09-21T10:29:52.407831+0000","last_fullsized":"2024-09-21T10:29:52.407831+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:49:48.823524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":48,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.388818+0000","last_change":"2024-09-21T10:28:00.293864+0000","last_active":"2024-09-21T10:29:52.388818+0000","last_peered":"2024-09-21T10:29:52.388818+0000","last_clean":"2024-09-21T10:29:52.388818+0000","last_became_active":"2024-09-21T10:28:00.293494+0000","last_became_peered":"2024-09-21T10:28:00.293494+0000","last_unstale":"2024-09-21T10:29:52.388818+0000","last_undegraded":"2024-09-21T10:29:52.388818+0000","last_fullsized":"2024-09-21T10:29:52.388818+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:37.098068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"379'11927","reported_seq":12070,"reported_epoch":379,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:29:58.066261+0000","last_change":"2024-09-21T10:28:01.631803+0000","last_active":"2024-09-21T10:29:58.066261+0000","last_peered":"2024-09-21T10:29:58.066261+0000","last_clean":"2024-09-21T10:27:59.267411+0000","last_became_active":"2024-09-21T10:28:01.425385+0000","last_became_peered":"2024-09-21T10:28:01.425385+0000","last_unstale":"2024-09-21T10:29:58.066261+0000","last_undegraded":"2024-09-21T10:29:58.066261+0000","last_fullsized":"2024-09-21T10:29:58.066261+0000","mapping_epoch":361,"log_start":"355'4200","ondisk_log_start":"355'4200","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7727,"log_dups_size":0,"ondisk_log_size":7727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:51:26.681334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231407616,"num_objects":3531,"num_object_clones":0,"num_object_copies":10593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3513,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11927,"num_write_kb":494656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":2949120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,2,4],"avail_no_missing":["6","2","4"],"object_location_counts":[{"shards":"2,4,6","objects":3531}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"82'100","reported_seq":55,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.389548+0000","last_change":"2024-09-21T10:28:00.305255+0000","last_active":"2024-09-21T10:29:52.389548+0000","last_peered":"2024-09-21T10:29:52.389548+0000","last_clean":"2024-09-21T10:29:52.389548+0000","last_became_active":"2024-09-21T10:28:00.290981+0000","last_became_peered":"2024-09-21T10:28:00.290981+0000","last_unstale":"2024-09-21T10:29:52.389548+0000","last_undegraded":"2024-09-21T10:29:52.389548+0000","last_fullsized":"2024-09-21T10:29:52.389548+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.759355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.759355+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":69,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":679,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.383664+0000","last_change":"2024-09-21T10:24:44.307344+0000","last_active":"2024-09-21T10:29:52.383664+0000","last_peered":"2024-09-21T10:29:52.383664+0000","last_clean":"2024-09-21T10:29:52.383664+0000","last_became_active":"2024-09-21T10:24:44.298936+0000","last_became_peered":"2024-09-21T10:24:44.298936+0000","last_unstale":"2024-09-21T10:29:52.383664+0000","last_undegraded":"2024-09-21T10:29:52.383664+0000","last_fullsized":"2024-09-21T10:29:52.383664+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:58:26.025877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":605,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.708399+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:29:49.708399+0000","last_peered":"2024-09-21T10:29:49.708399+0000","last_clean":"2024-09-21T10:29:49.708399+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:29:49.708399+0000","last_undegraded":"2024-09-21T10:29:49.708399+0000","last_fullsized":"2024-09-21T10:29:49.708399+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":68,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.697141+0000","last_change":"2024-09-21T10:24:44.303781+0000","last_active":"2024-09-21T10:29:49.697141+0000","last_peered":"2024-09-21T10:29:49.697141+0000","last_clean":"2024-09-21T10:29:49.697141+0000","last_became_active":"2024-09-21T10:24:44.302058+0000","last_became_peered":"2024-09-21T10:24:44.302058+0000","last_unstale":"2024-09-21T10:29:49.697141+0000","last_undegraded":"2024-09-21T10:29:49.697141+0000","last_fullsized":"2024-09-21T10:29:49.697141+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:30.371229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":68,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.697387+0000","last_change":"2024-09-21T10:24:44.303417+0000","last_active":"2024-09-21T10:29:49.697387+0000","last_peered":"2024-09-21T10:29:49.697387+0000","last_clean":"2024-09-21T10:29:49.697387+0000","last_became_active":"2024-09-21T10:24:44.303069+0000","last_became_peered":"2024-09-21T10:24:44.303069+0000","last_unstale":"2024-09-21T10:29:49.697387+0000","last_undegraded":"2024-09-21T10:29:49.697387+0000","last_fullsized":"2024-09-21T10:29:49.697387+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:30:30.966228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":78,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.697187+0000","last_change":"2024-09-21T10:24:44.321421+0000","last_active":"2024-09-21T10:29:49.697187+0000","last_peered":"2024-09-21T10:29:49.697187+0000","last_clean":"2024-09-21T10:29:49.697187+0000","last_became_active":"2024-09-21T10:24:44.297082+0000","last_became_peered":"2024-09-21T10:24:44.297082+0000","last_unstale":"2024-09-21T10:29:49.697187+0000","last_undegraded":"2024-09-21T10:29:49.697187+0000","last_fullsized":"2024-09-21T10:29:49.697187+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:35:37.704881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":514,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.406667+0000","last_change":"2024-09-21T10:08:18.470378+0000","last_active":"2024-09-21T10:29:52.406667+0000","last_peered":"2024-09-21T10:29:52.406667+0000","last_clean":"2024-09-21T10:29:52.406667+0000","last_became_active":"2024-09-21T10:08:18.469482+0000","last_became_peered":"2024-09-21T10:08:18.469482+0000","last_unstale":"2024-09-21T10:29:52.406667+0000","last_undegraded":"2024-09-21T10:29:52.406667+0000","last_fullsized":"2024-09-21T10:29:52.406667+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:35.628724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":442,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.704096+0000","last_change":"2024-09-21T10:28:00.296771+0000","last_active":"2024-09-21T10:29:49.704096+0000","last_peered":"2024-09-21T10:29:49.704096+0000","last_clean":"2024-09-21T10:29:49.704096+0000","last_became_active":"2024-09-21T10:28:00.285336+0000","last_became_peered":"2024-09-21T10:28:00.285336+0000","last_unstale":"2024-09-21T10:29:49.704096+0000","last_undegraded":"2024-09-21T10:29:49.704096+0000","last_fullsized":"2024-09-21T10:29:49.704096+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:06:50.848492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":150,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.713433+0000","last_change":"2024-09-21T10:26:41.644952+0000","last_active":"2024-09-21T10:29:49.713433+0000","last_peered":"2024-09-21T10:29:49.713433+0000","last_clean":"2024-09-21T10:29:49.713433+0000","last_became_active":"2024-09-21T10:26:41.644592+0000","last_became_peered":"2024-09-21T10:26:41.644592+0000","last_unstale":"2024-09-21T10:29:49.713433+0000","last_undegraded":"2024-09-21T10:29:49.713433+0000","last_fullsized":"2024-09-21T10:29:49.713433+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:44:44.183877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":48,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.389704+0000","last_change":"2024-09-21T10:28:00.294338+0000","last_active":"2024-09-21T10:29:52.389704+0000","last_peered":"2024-09-21T10:29:52.389704+0000","last_clean":"2024-09-21T10:29:52.389704+0000","last_became_active":"2024-09-21T10:28:00.294044+0000","last_became_peered":"2024-09-21T10:28:00.294044+0000","last_unstale":"2024-09-21T10:29:52.389704+0000","last_undegraded":"2024-09-21T10:29:52.389704+0000","last_fullsized":"2024-09-21T10:29:52.389704+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:43:30.999988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":644,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.382738+0000","last_change":"2024-09-21T10:28:00.296935+0000","last_active":"2024-09-21T10:29:52.382738+0000","last_peered":"2024-09-21T10:29:52.382738+0000","last_clean":"2024-09-21T10:29:52.382738+0000","last_became_active":"2024-09-21T10:28:00.296562+0000","last_became_peered":"2024-09-21T10:28:00.296562+0000","last_unstale":"2024-09-21T10:29:52.382738+0000","last_undegraded":"2024-09-21T10:29:52.382738+0000","last_fullsized":"2024-09-21T10:29:52.382738+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:06.863310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":48,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.388993+0000","last_change":"2024-09-21T10:28:00.294718+0000","last_active":"2024-09-21T10:29:52.388993+0000","last_peered":"2024-09-21T10:29:52.388993+0000","last_clean":"2024-09-21T10:29:52.388993+0000","last_became_active":"2024-09-21T10:28:00.294299+0000","last_became_peered":"2024-09-21T10:28:00.294299+0000","last_unstale":"2024-09-21T10:29:52.388993+0000","last_undegraded":"2024-09-21T10:29:52.388993+0000","last_fullsized":"2024-09-21T10:29:52.388993+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:41:46.881336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":342,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.708308+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:29:49.708308+0000","last_peered":"2024-09-21T10:29:49.708308+0000","last_clean":"2024-09-21T10:29:49.708308+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:29:49.708308+0000","last_undegraded":"2024-09-21T10:29:49.708308+0000","last_fullsized":"2024-09-21T10:29:49.708308+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":48,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.388908+0000","last_change":"2024-09-21T10:28:00.293872+0000","last_active":"2024-09-21T10:29:52.388908+0000","last_peered":"2024-09-21T10:29:52.388908+0000","last_clean":"2024-09-21T10:29:52.388908+0000","last_became_active":"2024-09-21T10:28:00.293320+0000","last_became_peered":"2024-09-21T10:28:00.293320+0000","last_unstale":"2024-09-21T10:29:52.388908+0000","last_undegraded":"2024-09-21T10:29:52.388908+0000","last_fullsized":"2024-09-21T10:29:52.388908+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:45:43.909307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":661,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.383878+0000","last_change":"2024-09-21T10:28:00.297304+0000","last_active":"2024-09-21T10:29:52.383878+0000","last_peered":"2024-09-21T10:29:52.383878+0000","last_clean":"2024-09-21T10:29:52.383878+0000","last_became_active":"2024-09-21T10:28:00.296631+0000","last_became_peered":"2024-09-21T10:28:00.296631+0000","last_unstale":"2024-09-21T10:29:52.383878+0000","last_undegraded":"2024-09-21T10:29:52.383878+0000","last_fullsized":"2024-09-21T10:29:52.383878+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:22:16.633581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":660,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.406443+0000","last_change":"2024-09-21T10:28:00.284924+0000","last_active":"2024-09-21T10:29:52.406443+0000","last_peered":"2024-09-21T10:29:52.406443+0000","last_clean":"2024-09-21T10:29:52.406443+0000","last_became_active":"2024-09-21T10:28:00.284633+0000","last_became_peered":"2024-09-21T10:28:00.284633+0000","last_unstale":"2024-09-21T10:29:52.406443+0000","last_undegraded":"2024-09-21T10:29:52.406443+0000","last_fullsized":"2024-09-21T10:29:52.406443+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:23:09.808151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":48,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.389548+0000","last_change":"2024-09-21T10:28:00.283403+0000","last_active":"2024-09-21T10:29:52.389548+0000","last_peered":"2024-09-21T10:29:52.389548+0000","last_clean":"2024-09-21T10:29:52.389548+0000","last_became_active":"2024-09-21T10:28:00.283089+0000","last_became_peered":"2024-09-21T10:28:00.283089+0000","last_unstale":"2024-09-21T10:29:52.389548+0000","last_undegraded":"2024-09-21T10:29:52.389548+0000","last_fullsized":"2024-09-21T10:29:52.389548+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:42.213644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":68,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.697560+0000","last_change":"2024-09-21T10:24:44.305124+0000","last_active":"2024-09-21T10:29:49.697560+0000","last_peered":"2024-09-21T10:29:49.697560+0000","last_clean":"2024-09-21T10:29:49.697560+0000","last_became_active":"2024-09-21T10:24:44.304468+0000","last_became_peered":"2024-09-21T10:24:44.304468+0000","last_unstale":"2024-09-21T10:29:49.697560+0000","last_undegraded":"2024-09-21T10:29:49.697560+0000","last_fullsized":"2024-09-21T10:29:49.697560+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:37.904764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":299,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.374210+0000","last_change":"2024-09-21T10:28:00.286646+0000","last_active":"2024-09-21T10:29:52.374210+0000","last_peered":"2024-09-21T10:29:52.374210+0000","last_clean":"2024-09-21T10:29:52.374210+0000","last_became_active":"2024-09-21T10:28:00.286300+0000","last_became_peered":"2024-09-21T10:28:00.286300+0000","last_unstale":"2024-09-21T10:29:52.374210+0000","last_undegraded":"2024-09-21T10:29:52.374210+0000","last_fullsized":"2024-09-21T10:29:52.374210+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:58.215385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":651,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.383010+0000","last_change":"2024-09-21T10:24:44.299989+0000","last_active":"2024-09-21T10:29:52.383010+0000","last_peered":"2024-09-21T10:29:52.383010+0000","last_clean":"2024-09-21T10:29:52.383010+0000","last_became_active":"2024-09-21T10:24:44.299356+0000","last_became_peered":"2024-09-21T10:24:44.299356+0000","last_unstale":"2024-09-21T10:29:52.383010+0000","last_undegraded":"2024-09-21T10:29:52.383010+0000","last_fullsized":"2024-09-21T10:29:52.383010+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:58.867008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":78,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.697180+0000","last_change":"2024-09-21T10:28:00.281942+0000","last_active":"2024-09-21T10:29:49.697180+0000","last_peered":"2024-09-21T10:29:49.697180+0000","last_clean":"2024-09-21T10:29:49.697180+0000","last_became_active":"2024-09-21T10:28:00.281600+0000","last_became_peered":"2024-09-21T10:28:00.281600+0000","last_unstale":"2024-09-21T10:29:49.697180+0000","last_undegraded":"2024-09-21T10:29:49.697180+0000","last_fullsized":"2024-09-21T10:29:49.697180+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:21.144244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":72,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.697280+0000","last_change":"2024-09-21T10:24:44.329150+0000","last_active":"2024-09-21T10:29:49.697280+0000","last_peered":"2024-09-21T10:29:49.697280+0000","last_clean":"2024-09-21T10:29:49.697280+0000","last_became_active":"2024-09-21T10:24:44.306752+0000","last_became_peered":"2024-09-21T10:24:44.306752+0000","last_unstale":"2024-09-21T10:29:49.697280+0000","last_undegraded":"2024-09-21T10:29:49.697280+0000","last_fullsized":"2024-09-21T10:29:49.697280+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:16.592859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":78,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.697592+0000","last_change":"2024-09-21T10:28:00.286009+0000","last_active":"2024-09-21T10:29:49.697592+0000","last_peered":"2024-09-21T10:29:49.697592+0000","last_clean":"2024-09-21T10:29:49.697592+0000","last_became_active":"2024-09-21T10:28:00.285327+0000","last_became_peered":"2024-09-21T10:28:00.285327+0000","last_unstale":"2024-09-21T10:29:49.697592+0000","last_undegraded":"2024-09-21T10:29:49.697592+0000","last_fullsized":"2024-09-21T10:29:49.697592+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:34.985354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":72,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.697413+0000","last_change":"2024-09-21T10:24:44.324087+0000","last_active":"2024-09-21T10:29:49.697413+0000","last_peered":"2024-09-21T10:29:49.697413+0000","last_clean":"2024-09-21T10:29:49.697413+0000","last_became_active":"2024-09-21T10:24:44.298665+0000","last_became_peered":"2024-09-21T10:24:44.298665+0000","last_unstale":"2024-09-21T10:29:49.697413+0000","last_undegraded":"2024-09-21T10:29:49.697413+0000","last_fullsized":"2024-09-21T10:29:49.697413+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:59.819340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":48,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.389633+0000","last_change":"2024-09-21T10:28:00.293054+0000","last_active":"2024-09-21T10:29:52.389633+0000","last_peered":"2024-09-21T10:29:52.389633+0000","last_clean":"2024-09-21T10:29:52.389633+0000","last_became_active":"2024-09-21T10:28:00.292723+0000","last_became_peered":"2024-09-21T10:28:00.292723+0000","last_unstale":"2024-09-21T10:29:52.389633+0000","last_undegraded":"2024-09-21T10:29:52.389633+0000","last_fullsized":"2024-09-21T10:29:52.389633+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:01:45.933570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":48,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.389433+0000","last_change":"2024-09-21T10:28:00.285515+0000","last_active":"2024-09-21T10:29:52.389433+0000","last_peered":"2024-09-21T10:29:52.389433+0000","last_clean":"2024-09-21T10:29:52.389433+0000","last_became_active":"2024-09-21T10:28:00.285203+0000","last_became_peered":"2024-09-21T10:28:00.285203+0000","last_unstale":"2024-09-21T10:29:52.389433+0000","last_undegraded":"2024-09-21T10:29:52.389433+0000","last_fullsized":"2024-09-21T10:29:52.389433+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:41.982228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":68,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T10:29:49.696905+0000","last_change":"2024-09-21T10:24:44.298820+0000","last_active":"2024-09-21T10:29:49.696905+0000","last_peered":"2024-09-21T10:29:49.696905+0000","last_clean":"2024-09-21T10:29:49.696905+0000","last_became_active":"2024-09-21T10:24:44.298478+0000","last_became_peered":"2024-09-21T10:24:44.298478+0000","last_unstale":"2024-09-21T10:29:49.696905+0000","last_undegraded":"2024-09-21T10:29:49.696905+0000","last_fullsized":"2024-09-21T10:29:49.696905+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:59.567078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":636,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.382623+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:29:52.382623+0000","last_peered":"2024-09-21T10:29:52.382623+0000","last_clean":"2024-09-21T10:29:52.382623+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:29:52.382623+0000","last_undegraded":"2024-09-21T10:29:52.382623+0000","last_fullsized":"2024-09-21T10:29:52.382623+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":3675717656,"num_objects":56088,"num_object_clones":0,"num_object_copies":168264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16731,"num_objects_misplaced":9771,"num_objects_unfound":0,"num_objects_dirty":56088,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193958,"num_write_kb":8001409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1753,"num_bytes_recovered":114884608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11867271168,"data_stored":11867258952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127058,"ondisk_log_size":127058,"up":48,"acting":45,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":10252,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5983,"num_read_kb":13006,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":316358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":69,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":644038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1100,"ondisk_log_size":1100,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397982,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2263696,"kb_used_data":1358152,"kb_used_omap":1,"kb_used_meta":905534,"kb_avail":91481456,"statfs":{"total":95995035648,"available":93677010944,"internally_reserved":0,"allocated":1390747648,"data_stored":1388575372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":927267160},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594210,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2483596,"kb_used_data":1582276,"kb_used_omap":1,"kb_used_meta":901310,"kb_avail":91261556,"statfs":{"total":95995035648,"available":93451833344,"internally_reserved":0,"allocated":1620250624,"data_stored":1618082341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922941784},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790440,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2043456,"kb_used_data":1147704,"kb_used_omap":1,"kb_used_meta":895742,"kb_avail":91701696,"statfs":{"total":95995035648,"available":93902536704,"internally_reserved":0,"allocated":1175248896,"data_stored":1173088932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":917240152},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986669,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2799376,"kb_used_data":2493704,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":90945776,"statfs":{"total":95995035648,"available":93128474624,"internally_reserved":0,"allocated":2553552896,"data_stored":2551380566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":101,"seq":433791697231,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":804092,"kb_used_data":141876,"kb_used_omap":5,"kb_used_meta":662202,"kb_avail":92941060,"statfs":{"total":95995035648,"available":95171645440,"internally_reserved":0,"allocated":145281024,"data_stored":143124914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":678095632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860757,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2612480,"kb_used_data":1644536,"kb_used_omap":8,"kb_used_meta":967799,"kb_avail":91132672,"statfs":{"total":95995035648,"available":93319856128,"internally_reserved":0,"allocated":1684004864,"data_stored":1681845180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":991027052},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024283,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1941680,"kb_used_data":1133096,"kb_used_omap":7,"kb_used_meta":808504,"kb_avail":91803472,"statfs":{"total":95995035648,"available":94006755328,"internally_reserved":0,"allocated":1160290304,"data_stored":1158145245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":827908658},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187819,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3238784,"kb_used_data":2211192,"kb_used_omap":8,"kb_used_meta":1027575,"kb_avail":90506368,"statfs":{"total":95995035648,"available":92678520832,"internally_reserved":0,"allocated":2264260608,"data_stored":2262100392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":1052237809},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2248478720,"data_stored":2248474648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1152516096,"data_stored":1152516096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1668022272,"data_stored":1668022272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":139591680,"data_stored":139591680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2526154752,"data_stored":2526150680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1149632512,"data_stored":1149632512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1607340032,"data_stored":1607335960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1375535104,"data_stored":1375535104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:30:01.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:30:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:01.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:01.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:01.837 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:01.909 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:01.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:01.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:02.083 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:02.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:02.139 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:02.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:02.240 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:02.285 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:02.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:02.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:02.420 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:02.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:02.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:02.522 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:02.559 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:02.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:02.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:02.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:02.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:02.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:02.801 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:02.833 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:02.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:02.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:02.934 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:02.966 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:02.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:02.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:03.042 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:30:03.067 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:03.103 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:03.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:03.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:03.205 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:03.234 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:03.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:03.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:03.319 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:30:03.319 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:30:03.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:03.319 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46322.0:197420 10.1 e379/348 10:87b3f3ef:::benchmark_data_smithi098_281896_object72395:head v 379'12420, mlcod=379'12418)", 2024-09-21T10:30:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:30:03.315898+0000", 2024-09-21T10:30:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0011694539999999999, 2024-09-21T10:30:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001191945, 2024-09-21T10:30:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:30:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:30:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:30:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:30:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:03.315898+0000", 2024-09-21T10:30:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:30:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:03.315898+0000", 2024-09-21T10:30:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:30:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:03.315897+0000", 2024-09-21T10:30:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:30:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:30:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:03.315907+0000", 2024-09-21T10:30:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8330000000000006e-06 2024-09-21T10:30:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:30:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:03.315927+0000", 2024-09-21T10:30:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0669000000000001e-05 2024-09-21T10:30:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:30:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:03.315935+0000", 2024-09-21T10:30:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6820000000000003e-06 2024-09-21T10:30:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:03.315984+0000", 2024-09-21T10:30:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9095999999999999e-05 2024-09-21T10:30:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:30:03.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:03.316004+0000", 2024-09-21T10:30:03.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0029000000000001e-05 2024-09-21T10:30:03.323 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:03.323 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:30:03.323 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:03.323 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:03.323 INFO:teuthology.orchestra.run.smithi196.stdout: ], 2024-09-21T10:30:03.323 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 1 2024-09-21T10:30:03.323 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:30:03.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:03.400 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:03.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:03.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:03.480 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:03.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:03.567 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:03.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:03.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:03.605 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:03 smithi098 ceph-mon[100914]: pgmap v1302: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 688 op/s; 16731/167118 objects degraded (10.011%); 9771/167118 objects misplaced (5.847%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:30:03.606 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:03 smithi098 ceph-mon[97580]: pgmap v1302: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 688 op/s; 16731/167118 objects degraded (10.011%); 9771/167118 objects misplaced (5.847%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:30:03.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:03 smithi196 ceph-mon[84784]: pgmap v1302: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 688 op/s; 16731/167118 objects degraded (10.011%); 9771/167118 objects misplaced (5.847%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:30:03.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:03.698 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:03.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:03.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:03.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:03.828 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:03.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:03.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:03.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:03.974 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:03.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:03.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:04.018 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:30:04.076 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:04.116 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:04.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:04.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:04.218 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:04.253 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:04.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:04.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:04.355 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:04.365 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:30:04.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:04.395 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:04.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:04.499 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:04.523 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:04.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:04.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:04.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:04.664 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:04.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:04.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:04.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:04.794 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:04.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:04.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:04.896 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:04.927 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:04.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:04.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:05.029 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:05.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:05.056 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:05.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:05.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:05.192 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:05.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:05.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:05.294 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:05.337 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:05.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:05.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:05.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:05.494 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:05.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:05.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:05.595 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:05.596 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:05 smithi196 ceph-mon[84784]: pgmap v1303: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 812 op/s; 16731/160398 objects degraded (10.431%); 9340/160398 objects misplaced (5.823%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:30:05.596 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:05 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1586933932' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:30:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:05.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:05.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:05.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:05.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:05.773 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:05.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:05.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:05 smithi098 ceph-mon[97580]: pgmap v1303: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 812 op/s; 16731/160398 objects degraded (10.431%); 9340/160398 objects misplaced (5.823%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:30:05.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:05 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1586933932' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:30:05.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:05 smithi098 ceph-mon[100914]: pgmap v1303: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 812 op/s; 16731/160398 objects degraded (10.431%); 9340/160398 objects misplaced (5.823%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:30:05.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:05 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1586933932' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:30:05.874 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:05.923 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:05.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:05.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:06.024 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:06.056 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:06.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:06.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:06.158 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:06.182 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:06.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:06.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:06.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:06.313 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:30:06.313 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:30:06.313 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-21T10:30:06.313 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-21T10:30:06.313 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:30:06.318 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:06.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:06.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:06.372 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:30:06.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:06.455 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:06.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:06.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:06.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:06.595 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:06.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:06.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:06.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:06 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:30:06.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:06 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1586933932' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:30:06.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:06 smithi196 ceph-mon[84784]: osdmap e380: 8 total, 8 up, 8 in 2024-09-21T10:30:06.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:06 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 16731/160398 objects degraded (10.431%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:06.697 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:06.724 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:06.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:06.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:06.826 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:06.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:06 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:30:06.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:06 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1586933932' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:30:06.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:06 smithi098 ceph-mon[97580]: osdmap e380: 8 total, 8 up, 8 in 2024-09-21T10:30:06.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:06 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 16731/160398 objects degraded (10.431%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:06.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:06 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:30:06.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:06 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1586933932' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:30:06.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:06 smithi098 ceph-mon[100914]: osdmap e380: 8 total, 8 up, 8 in 2024-09-21T10:30:06.849 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:06 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 16731/160398 objects degraded (10.431%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:06.864 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:06.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:06.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:06.965 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:06.999 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:06.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:07.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:07.091 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:30:07.100 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:07.140 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:07.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:07.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:07.242 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:07.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:07.282 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:07.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:07.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:07.418 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:07.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:07.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:07.440 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:30:07.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:07.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:07.556 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:07.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:07.657 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:07.700 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:07.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:07.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:07.710 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:07 smithi196 ceph-mon[84784]: pgmap v1305: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 665 op/s; 16843/157410 objects degraded (10.700%); 9119/157410 objects misplaced (5.793%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:30:07.710 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:07.748 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:07 smithi098 ceph-mon[100914]: pgmap v1305: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 665 op/s; 16843/157410 objects degraded (10.700%); 9119/157410 objects misplaced (5.793%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:30:07.748 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:07.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:07 smithi098 ceph-mon[97580]: pgmap v1305: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 665 op/s; 16843/157410 objects degraded (10.700%); 9119/157410 objects misplaced (5.793%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:30:07.749 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:07.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:07.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:07.843 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:07.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:07.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:07.981 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:07.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:07.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:08.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:08.110 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:08.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:08.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:08.211 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:08.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:08.236 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:08.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:08.338 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:08.372 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:08.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:08.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:08.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:08.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:08.504 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:08.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:08.605 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:08.637 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:08.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:08.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:08.745 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:08.786 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:08.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:08.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:08.888 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:08.927 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:08.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:08.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:09.029 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:09.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:09.066 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:09.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:09.167 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:09.203 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:09.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:09.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:09.304 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:09.320 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:30:09.320 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-21T10:30:09.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-21T10:30:09.320 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:30:09.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.320 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:52171 9.c 9:360aee28:::benchmark_data_smithi098_254407_object52170:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-21T10:30:09.320 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:22:56.544104+0000", 2024-09-21T10:30:09.320 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 432.769546569, 2024-09-21T10:30:09.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.157414996, 2024-09-21T10:30:09.320 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:30:09.320 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:30:09.320 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:30:09.320 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:30:09.320 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:30:09.321 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 52171 2024-09-21T10:30:09.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.321 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:30:09.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:30:09.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.544104+0000", 2024-09-21T10:30:09.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:30:09.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.544104+0000", 2024-09-21T10:30:09.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:30:09.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.544098+0000", 2024-09-21T10:30:09.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999943 2024-09-21T10:30:09.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:30:09.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.544136+0000", 2024-09-21T10:30:09.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8164000000000003e-05 2024-09-21T10:30:09.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:30:09.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.544146+0000", 2024-09-21T10:30:09.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5119999999999997e-06 2024-09-21T10:30:09.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:30:09.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.544150+0000", 2024-09-21T10:30:09.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2880000000000003e-06 2024-09-21T10:30:09.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.544190+0000", 2024-09-21T10:30:09.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0119e-05 2024-09-21T10:30:09.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:30:09.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.544310+0000", 2024-09-21T10:30:09.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119418 2024-09-21T10:30:09.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:30:09.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.544445+0000", 2024-09-21T10:30:09.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013546200000000001 2024-09-21T10:30:09.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:30:09.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.547672+0000", 2024-09-21T10:30:09.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032270559999999998 2024-09-21T10:30:09.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.547861+0000", 2024-09-21T10:30:09.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00018934900000000001 2024-09-21T10:30:09.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.701428+0000", 2024-09-21T10:30:09.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.153566906 2024-09-21T10:30:09.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:30:09.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.701475+0000", 2024-09-21T10:30:09.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6854999999999998e-05 2024-09-21T10:30:09.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:30:09.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:22:56.701519+0000", 2024-09-21T10:30:09.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3587999999999997e-05 2024-09-21T10:30:09.325 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.326 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:30:09.326 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.326 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181814 9.e 9:74c3af90:::benchmark_data_smithi098_254407_object49083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T10:30:09.326 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.273477+0000", 2024-09-21T10:30:09.326 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 326.04017387599998, 2024-09-21T10:30:09.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.160252872, 2024-09-21T10:30:09.326 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:30:09.326 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:30:09.326 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:30:09.326 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:30:09.326 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:30:09.326 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181814 2024-09-21T10:30:09.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.327 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:30:09.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:30:09.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.273477+0000", 2024-09-21T10:30:09.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:30:09.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.273477+0000", 2024-09-21T10:30:09.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:30:09.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.273475+0000", 2024-09-21T10:30:09.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:30:09.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:30:09.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.273486+0000", 2024-09-21T10:30:09.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0481e-05 2024-09-21T10:30:09.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:30:09.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.275927+0000", 2024-09-21T10:30:09.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024413059999999999 2024-09-21T10:30:09.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:30:09.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.275937+0000", 2024-09-21T10:30:09.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0273e-05 2024-09-21T10:30:09.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277114+0000", 2024-09-21T10:30:09.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011773440000000001 2024-09-21T10:30:09.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:30:09.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277227+0000", 2024-09-21T10:30:09.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000112969 2024-09-21T10:30:09.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:30:09.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277336+0000", 2024-09-21T10:30:09.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108162 2024-09-21T10:30:09.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:30:09.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.286117+0000", 2024-09-21T10:30:09.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0087815770000000005 2024-09-21T10:30:09.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.294408+0000", 2024-09-21T10:30:09.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0082903879999999992 2024-09-21T10:30:09.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:30:09.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.294415+0000", 2024-09-21T10:30:09.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2869999999999999e-06 2024-09-21T10:30:09.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.297546+0000", 2024-09-21T10:30:09.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0031308510000001 2024-09-21T10:30:09.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:30:09.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.297561+0000", 2024-09-21T10:30:09.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5545000000000001e-05 2024-09-21T10:30:09.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.425226+0000", 2024-09-21T10:30:09.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1276647829999999 2024-09-21T10:30:09.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for missing object", 2024-09-21T10:30:09.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.425366+0000", 2024-09-21T10:30:09.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000139529 2024-09-21T10:30:09.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433657+0000", 2024-09-21T10:30:09.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0082909950000000007 2024-09-21T10:30:09.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:30:09.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433729+0000", 2024-09-21T10:30:09.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2879000000000005e-05 2024-09-21T10:30:09.332 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.333 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:30:09.333 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.333 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181817 9.e 9:7fac4b1e:::benchmark_data_smithi098_254407_object49086:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T10:30:09.333 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.275953+0000", 2024-09-21T10:30:09.333 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 326.03769691500003, 2024-09-21T10:30:09.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1579490689999998, 2024-09-21T10:30:09.333 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:30:09.333 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:30:09.333 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:30:09.333 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:30:09.333 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:30:09.333 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181817 2024-09-21T10:30:09.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.334 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:30:09.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:30:09.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.275953+0000", 2024-09-21T10:30:09.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:30:09.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.275953+0000", 2024-09-21T10:30:09.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:30:09.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.275952+0000", 2024-09-21T10:30:09.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:30:09.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:30:09.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.275958+0000", 2024-09-21T10:30:09.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4840000000000003e-06 2024-09-21T10:30:09.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:30:09.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.275965+0000", 2024-09-21T10:30:09.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2110000000000001e-06 2024-09-21T10:30:09.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:30:09.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.275969+0000", 2024-09-21T10:30:09.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2560000000000004e-06 2024-09-21T10:30:09.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.294432+0000", 2024-09-21T10:30:09.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018462203999999999 2024-09-21T10:30:09.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:30:09.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.294437+0000", 2024-09-21T10:30:09.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.271e-06 2024-09-21T10:30:09.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.298237+0000", 2024-09-21T10:30:09.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.003800574 2024-09-21T10:30:09.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:30:09.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.298263+0000", 2024-09-21T10:30:09.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5819000000000001e-05 2024-09-21T10:30:09.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.425691+0000", 2024-09-21T10:30:09.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.127427889 2024-09-21T10:30:09.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:30:09.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.425818+0000", 2024-09-21T10:30:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012697000000000001 2024-09-21T10:30:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1,4", 2024-09-21T10:30:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.425958+0000", 2024-09-21T10:30:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014029900000000001 2024-09-21T10:30:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:30:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433272+0000", 2024-09-21T10:30:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0073139520000000003 2024-09-21T10:30:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433315+0000", 2024-09-21T10:30:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2237999999999998e-05 2024-09-21T10:30:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433345+0000", 2024-09-21T10:30:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0508999999999999e-05 2024-09-21T10:30:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433759+0000", 2024-09-21T10:30:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00041381300000000002 2024-09-21T10:30:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:30:09.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433797+0000", 2024-09-21T10:30:09.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7857e-05 2024-09-21T10:30:09.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:30:09.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433903+0000", 2024-09-21T10:30:09.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105803 2024-09-21T10:30:09.340 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.340 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:30:09.340 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.340 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181827 9.e 9:752f527b:::benchmark_data_smithi098_254407_object49096:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:30:09.340 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.295979+0000", 2024-09-21T10:30:09.341 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 326.01767185900002, 2024-09-21T10:30:09.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.138117657, 2024-09-21T10:30:09.341 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:30:09.341 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:30:09.341 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:30:09.341 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:30:09.341 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:30:09.341 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181827 2024-09-21T10:30:09.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.341 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:30:09.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:30:09.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.295979+0000", 2024-09-21T10:30:09.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:30:09.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.295979+0000", 2024-09-21T10:30:09.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:30:09.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.295977+0000", 2024-09-21T10:30:09.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:30:09.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:30:09.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.295982+0000", 2024-09-21T10:30:09.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0239999999999997e-06 2024-09-21T10:30:09.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:30:09.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.295991+0000", 2024-09-21T10:30:09.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3080000000000008e-06 2024-09-21T10:30:09.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:30:09.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.295995+0000", 2024-09-21T10:30:09.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5410000000000002e-06 2024-09-21T10:30:09.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.296052+0000", 2024-09-21T10:30:09.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6929000000000003e-05 2024-09-21T10:30:09.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:30:09.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.296082+0000", 2024-09-21T10:30:09.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0080999999999999e-05 2024-09-21T10:30:09.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.298286+0000", 2024-09-21T10:30:09.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.002203583 2024-09-21T10:30:09.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:30:09.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.298293+0000", 2024-09-21T10:30:09.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9820000000000002e-06 2024-09-21T10:30:09.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.429659+0000", 2024-09-21T10:30:09.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.131366028 2024-09-21T10:30:09.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.345 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:30:09.345 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.429795+0000", 2024-09-21T10:30:09.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000136594 2024-09-21T10:30:09.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1,4", 2024-09-21T10:30:09.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.429931+0000", 2024-09-21T10:30:09.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000135807 2024-09-21T10:30:09.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:30:09.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433279+0000", 2024-09-21T10:30:09.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0033479209999999998 2024-09-21T10:30:09.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433374+0000", 2024-09-21T10:30:09.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5358000000000004e-05 2024-09-21T10:30:09.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433935+0000", 2024-09-21T10:30:09.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00056065800000000001 2024-09-21T10:30:09.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433965+0000", 2024-09-21T10:30:09.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0321e-05 2024-09-21T10:30:09.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:30:09.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.434007+0000", 2024-09-21T10:30:09.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1180000000000002e-05 2024-09-21T10:30:09.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.348 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:30:09.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.434096+0000", 2024-09-21T10:30:09.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9605999999999996e-05 2024-09-21T10:30:09.348 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.348 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:30:09.348 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.348 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181829 9.e 9:7ab751bc:::benchmark_data_smithi098_254407_object49098:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:30:09.348 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.296018+0000", 2024-09-21T10:30:09.348 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 326.01763266099999, 2024-09-21T10:30:09.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1587496179999999, 2024-09-21T10:30:09.349 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:30:09.349 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:30:09.349 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:30:09.349 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:30:09.349 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:30:09.349 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181829 2024-09-21T10:30:09.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.349 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:30:09.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:30:09.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.296018+0000", 2024-09-21T10:30:09.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:30:09.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.296018+0000", 2024-09-21T10:30:09.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:30:09.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.296017+0000", 2024-09-21T10:30:09.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:30:09.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:30:09.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.296020+0000", 2024-09-21T10:30:09.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1300000000000001e-06 2024-09-21T10:30:09.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:30:09.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.296028+0000", 2024-09-21T10:30:09.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7500000000000003e-06 2024-09-21T10:30:09.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:30:09.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.296032+0000", 2024-09-21T10:30:09.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0980000000000004e-06 2024-09-21T10:30:09.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.296265+0000", 2024-09-21T10:30:09.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000233012 2024-09-21T10:30:09.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:30:09.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.296273+0000", 2024-09-21T10:30:09.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0749999999999998e-06 2024-09-21T10:30:09.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.298339+0000", 2024-09-21T10:30:09.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.002066219 2024-09-21T10:30:09.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:30:09.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.298347+0000", 2024-09-21T10:30:09.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0659999999999998e-06 2024-09-21T10:30:09.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.434117+0000", 2024-09-21T10:30:09.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.135770629 2024-09-21T10:30:09.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:30:09.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.434218+0000", 2024-09-21T10:30:09.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100985 2024-09-21T10:30:09.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1,4", 2024-09-21T10:30:09.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.434325+0000", 2024-09-21T10:30:09.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107205 2024-09-21T10:30:09.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.445178+0000", 2024-09-21T10:30:09.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010852288 2024-09-21T10:30:09.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:30:09.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.445186+0000", 2024-09-21T10:30:09.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9130000000000003e-06 2024-09-21T10:30:09.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.446747+0000", 2024-09-21T10:30:09.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015616580000000001 2024-09-21T10:30:09.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.454616+0000", 2024-09-21T10:30:09.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0078688309999999997 2024-09-21T10:30:09.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:30:09.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.454662+0000", 2024-09-21T10:30:09.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5759e-05 2024-09-21T10:30:09.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:30:09.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.454767+0000", 2024-09-21T10:30:09.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105568 2024-09-21T10:30:09.355 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.355 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:30:09.355 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.355 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181852 9.e 9:74989e04:::benchmark_data_smithi098_254407_object49121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:30:09.356 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.320821+0000", 2024-09-21T10:30:09.356 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 325.992829266, 2024-09-21T10:30:09.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1341375820000001, 2024-09-21T10:30:09.356 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:30:09.356 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:30:09.356 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:30:09.356 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:30:09.356 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:30:09.356 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181852 2024-09-21T10:30:09.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.356 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:30:09.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.356 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:30:09.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.320821+0000", 2024-09-21T10:30:09.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:30:09.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.320821+0000", 2024-09-21T10:30:09.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:30:09.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.320820+0000", 2024-09-21T10:30:09.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:30:09.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:30:09.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.320824+0000", 2024-09-21T10:30:09.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0879999999999997e-06 2024-09-21T10:30:09.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:30:09.358 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.320835+0000", 2024-09-21T10:30:09.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.062e-05 2024-09-21T10:30:09.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:30:09.358 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.320840+0000", 2024-09-21T10:30:09.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9629999999999997e-06 2024-09-21T10:30:09.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.358 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.320906+0000", 2024-09-21T10:30:09.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5568999999999995e-05 2024-09-21T10:30:09.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:30:09.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.320913+0000", 2024-09-21T10:30:09.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5499999999999997e-06 2024-09-21T10:30:09.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.298371+0000", 2024-09-21T10:30:09.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97745800699999996 2024-09-21T10:30:09.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:30:09.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.298378+0000", 2024-09-21T10:30:09.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7039999999999997e-06 2024-09-21T10:30:09.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.438127+0000", 2024-09-21T10:30:09.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1397487129999999 2024-09-21T10:30:09.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:30:09.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.438232+0000", 2024-09-21T10:30:09.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105609 2024-09-21T10:30:09.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1,4", 2024-09-21T10:30:09.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.438340+0000", 2024-09-21T10:30:09.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010830199999999999 2024-09-21T10:30:09.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:30:09.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.445191+0000", 2024-09-21T10:30:09.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0068508939999999997 2024-09-21T10:30:09.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.453574+0000", 2024-09-21T10:30:09.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0083831280000000001 2024-09-21T10:30:09.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.453934+0000", 2024-09-21T10:30:09.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00036004800000000002 2024-09-21T10:30:09.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.454812+0000", 2024-09-21T10:30:09.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00087731799999999998 2024-09-21T10:30:09.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:30:09.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.454856+0000", 2024-09-21T10:30:09.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3945e-05 2024-09-21T10:30:09.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:30:09.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.454959+0000", 2024-09-21T10:30:09.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000102971 2024-09-21T10:30:09.363 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.363 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:30:09.363 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.363 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181865 9.e 9:795ade75:::benchmark_data_smithi098_254407_object49134:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:30:09.363 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.337664+0000", 2024-09-21T10:30:09.364 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 325.97598631, 2024-09-21T10:30:09.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1178443219999998, 2024-09-21T10:30:09.364 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:30:09.364 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:30:09.364 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:30:09.364 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:30:09.364 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:30:09.364 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181865 2024-09-21T10:30:09.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.364 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:30:09.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:30:09.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337664+0000", 2024-09-21T10:30:09.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:30:09.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337664+0000", 2024-09-21T10:30:09.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:30:09.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337663+0000", 2024-09-21T10:30:09.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:30:09.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:30:09.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337668+0000", 2024-09-21T10:30:09.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0579999999999997e-06 2024-09-21T10:30:09.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:30:09.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337678+0000", 2024-09-21T10:30:09.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.026e-05 2024-09-21T10:30:09.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:30:09.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337683+0000", 2024-09-21T10:30:09.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7790000000000003e-06 2024-09-21T10:30:09.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337726+0000", 2024-09-21T10:30:09.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3711999999999998e-05 2024-09-21T10:30:09.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:30:09.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337733+0000", 2024-09-21T10:30:09.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4030000000000004e-06 2024-09-21T10:30:09.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.298435+0000", 2024-09-21T10:30:09.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.96070259599999996 2024-09-21T10:30:09.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:30:09.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.298446+0000", 2024-09-21T10:30:09.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0958000000000001e-05 2024-09-21T10:30:09.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.441414+0000", 2024-09-21T10:30:09.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.142967466 2024-09-21T10:30:09.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:30:09.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.441533+0000", 2024-09-21T10:30:09.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119294 2024-09-21T10:30:09.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1,4", 2024-09-21T10:30:09.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.441673+0000", 2024-09-21T10:30:09.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014025100000000001 2024-09-21T10:30:09.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:30:09.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.446755+0000", 2024-09-21T10:30:09.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0050815249999999999 2024-09-21T10:30:09.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.453610+0000", 2024-09-21T10:30:09.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0068551269999999999 2024-09-21T10:30:09.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.453962+0000", 2024-09-21T10:30:09.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00035222299999999998 2024-09-21T10:30:09.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.455371+0000", 2024-09-21T10:30:09.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001408802 2024-09-21T10:30:09.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:30:09.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.455415+0000", 2024-09-21T10:30:09.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3591000000000003e-05 2024-09-21T10:30:09.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.371 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:30:09.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.455508+0000", 2024-09-21T10:30:09.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3774e-05 2024-09-21T10:30:09.371 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.371 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:30:09.371 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.371 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46025.0:223250 9.2 e332/300 9:4a754f52:::benchmark_data_smithi098_254407_object90519:head v 332'14254, mlcod=332'14251)", 2024-09-21T10:30:09.371 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:30.409377+0000", 2024-09-21T10:30:09.371 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 278.90427325500002, 2024-09-21T10:30:09.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.115180911, 2024-09-21T10:30:09.372 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:30:09.372 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:30:09.372 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:30:09.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:09.373 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:09.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:09.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:30:09.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:30.409377+0000", 2024-09-21T10:30:09.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:30:09.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:30.409377+0000", 2024-09-21T10:30:09.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:30:09.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:30.409377+0000", 2024-09-21T10:30:09.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:30:09.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:30:09.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:30.409382+0000", 2024-09-21T10:30:09.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7950000000000004e-06 2024-09-21T10:30:09.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:30:09.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:30.514704+0000", 2024-09-21T10:30:09.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.105321291 2024-09-21T10:30:09.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:30:09.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:30.514710+0000", 2024-09-21T10:30:09.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2110000000000003e-06 2024-09-21T10:30:09.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:30.514753+0000", 2024-09-21T10:30:09.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.295e-05 2024-09-21T10:30:09.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:30:09.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:30.514775+0000", 2024-09-21T10:30:09.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1732000000000001e-05 2024-09-21T10:30:09.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:30:09.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:30.524479+0000", 2024-09-21T10:30:09.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0097043349999999997 2024-09-21T10:30:09.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:30:09.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:30.524558+0000", 2024-09-21T10:30:09.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9096000000000003e-05 2024-09-21T10:30:09.377 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.377 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:30:09.377 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.377 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:265403 9.d 9:b0362428:::benchmark_data_smithi098_254407_object132672:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-09-21T10:30:09.377 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:16.862158+0000", 2024-09-21T10:30:09.377 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 232.45149263100001, 2024-09-21T10:30:09.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.187899645, 2024-09-21T10:30:09.378 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:30:09.378 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:30:09.378 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:30:09.378 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:30:09.378 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:30:09.378 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 265403 2024-09-21T10:30:09.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.378 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:30:09.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:30:09.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862158+0000", 2024-09-21T10:30:09.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:30:09.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862158+0000", 2024-09-21T10:30:09.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:30:09.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862156+0000", 2024-09-21T10:30:09.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:30:09.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:30:09.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862163+0000", 2024-09-21T10:30:09.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9249999999999998e-06 2024-09-21T10:30:09.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:30:09.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862176+0000", 2024-09-21T10:30:09.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3096e-05 2024-09-21T10:30:09.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:30:09.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862182+0000", 2024-09-21T10:30:09.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5219999999999998e-06 2024-09-21T10:30:09.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862418+0000", 2024-09-21T10:30:09.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00023681599999999999 2024-09-21T10:30:09.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:30:09.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862530+0000", 2024-09-21T10:30:09.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000111387 2024-09-21T10:30:09.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:30:09.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862638+0000", 2024-09-21T10:30:09.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010864 2024-09-21T10:30:09.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.874856+0000", 2024-09-21T10:30:09.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012217356 2024-09-21T10:30:09.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:30:09.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.874862+0000", 2024-09-21T10:30:09.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7790000000000001e-06 2024-09-21T10:30:09.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.049954+0000", 2024-09-21T10:30:09.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17509197300000001 2024-09-21T10:30:09.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:30:09.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.050012+0000", 2024-09-21T10:30:09.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8332999999999999e-05 2024-09-21T10:30:09.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:30:09.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.050057+0000", 2024-09-21T10:30:09.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5529999999999999e-05 2024-09-21T10:30:09.383 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.383 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:30:09.383 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.383 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:265404 9.d 9:be84246b:::benchmark_data_smithi098_254407_object132673:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-09-21T10:30:09.383 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:16.862588+0000", 2024-09-21T10:30:09.383 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 232.45106193800001, 2024-09-21T10:30:09.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.20599171599999999, 2024-09-21T10:30:09.384 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:30:09.384 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:30:09.384 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:30:09.384 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:30:09.384 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:30:09.384 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 265404 2024-09-21T10:30:09.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.384 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:30:09.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:30:09.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862588+0000", 2024-09-21T10:30:09.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:30:09.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862588+0000", 2024-09-21T10:30:09.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:30:09.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862588+0000", 2024-09-21T10:30:09.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:30:09.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:30:09.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862594+0000", 2024-09-21T10:30:09.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7509999999999997e-06 2024-09-21T10:30:09.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:30:09.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862602+0000", 2024-09-21T10:30:09.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4489999999999999e-06 2024-09-21T10:30:09.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:30:09.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862607+0000", 2024-09-21T10:30:09.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3240000000000002e-06 2024-09-21T10:30:09.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.871168+0000", 2024-09-21T10:30:09.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0085602720000000007 2024-09-21T10:30:09.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:30:09.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.871284+0000", 2024-09-21T10:30:09.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000116607 2024-09-21T10:30:09.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:30:09.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.871398+0000", 2024-09-21T10:30:09.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000113458 2024-09-21T10:30:09.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.878846+0000", 2024-09-21T10:30:09.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0074478829999999998 2024-09-21T10:30:09.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:30:09.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.878898+0000", 2024-09-21T10:30:09.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1996000000000002e-05 2024-09-21T10:30:09.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.068450+0000", 2024-09-21T10:30:09.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.189552424 2024-09-21T10:30:09.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:30:09.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.068524+0000", 2024-09-21T10:30:09.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3747999999999999e-05 2024-09-21T10:30:09.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:30:09.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.068580+0000", 2024-09-21T10:30:09.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6515999999999999e-05 2024-09-21T10:30:09.389 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.389 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:30:09.389 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.389 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:265406 9.7 9:e704644c:::benchmark_data_smithi098_254407_object132675:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-09-21T10:30:09.390 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:16.862782+0000", 2024-09-21T10:30:09.390 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 232.450868095, 2024-09-21T10:30:09.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.20006702200000001, 2024-09-21T10:30:09.390 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:30:09.390 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:30:09.390 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:30:09.390 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:30:09.390 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:30:09.390 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 265406 2024-09-21T10:30:09.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.390 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:30:09.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:30:09.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862782+0000", 2024-09-21T10:30:09.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:30:09.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862782+0000", 2024-09-21T10:30:09.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:30:09.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862782+0000", 2024-09-21T10:30:09.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:30:09.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:30:09.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862785+0000", 2024-09-21T10:30:09.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4759999999999998e-06 2024-09-21T10:30:09.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:30:09.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862795+0000", 2024-09-21T10:30:09.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5950000000000005e-06 2024-09-21T10:30:09.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:30:09.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862799+0000", 2024-09-21T10:30:09.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2150000000000003e-06 2024-09-21T10:30:09.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862842+0000", 2024-09-21T10:30:09.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2454999999999999e-05 2024-09-21T10:30:09.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:30:09.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862944+0000", 2024-09-21T10:30:09.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010283100000000001 2024-09-21T10:30:09.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:30:09.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.863048+0000", 2024-09-21T10:30:09.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104096 2024-09-21T10:30:09.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:30:09.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.873531+0000", 2024-09-21T10:30:09.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010482742 2024-09-21T10:30:09.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.873574+0000", 2024-09-21T10:30:09.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3056000000000002e-05 2024-09-21T10:30:09.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.062724+0000", 2024-09-21T10:30:09.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18914968600000001 2024-09-21T10:30:09.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:30:09.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.062805+0000", 2024-09-21T10:30:09.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0932000000000004e-05 2024-09-21T10:30:09.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:30:09.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.062849+0000", 2024-09-21T10:30:09.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4487999999999998e-05 2024-09-21T10:30:09.395 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.395 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:30:09.395 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.395 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:265407 9.7 9:eb620111:::benchmark_data_smithi098_254407_object132676:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-09-21T10:30:09.395 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:16.864645+0000", 2024-09-21T10:30:09.396 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 232.449005575, 2024-09-21T10:30:09.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.19854269299999999, 2024-09-21T10:30:09.396 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:30:09.396 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:30:09.396 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:30:09.396 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:30:09.396 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:30:09.396 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 265407 2024-09-21T10:30:09.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.396 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:30:09.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:30:09.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864645+0000", 2024-09-21T10:30:09.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:30:09.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864645+0000", 2024-09-21T10:30:09.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:30:09.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864643+0000", 2024-09-21T10:30:09.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:30:09.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:30:09.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864649+0000", 2024-09-21T10:30:09.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9560000000000002e-06 2024-09-21T10:30:09.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:30:09.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864667+0000", 2024-09-21T10:30:09.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7668000000000001e-05 2024-09-21T10:30:09.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:30:09.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864673+0000", 2024-09-21T10:30:09.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3319999999999997e-06 2024-09-21T10:30:09.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.866735+0000", 2024-09-21T10:30:09.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020622549999999998 2024-09-21T10:30:09.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:30:09.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.866851+0000", 2024-09-21T10:30:09.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000115458 2024-09-21T10:30:09.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:30:09.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.866969+0000", 2024-09-21T10:30:09.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000118417 2024-09-21T10:30:09.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:30:09.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.874132+0000", 2024-09-21T10:30:09.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0071623679999999997 2024-09-21T10:30:09.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.876314+0000", 2024-09-21T10:30:09.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021818670000000001 2024-09-21T10:30:09.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.063081+0000", 2024-09-21T10:30:09.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18676781200000001 2024-09-21T10:30:09.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:30:09.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.063151+0000", 2024-09-21T10:30:09.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9803999999999998e-05 2024-09-21T10:30:09.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:30:09.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.063188+0000", 2024-09-21T10:30:09.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6402999999999997e-05 2024-09-21T10:30:09.401 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.401 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:30:09.401 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.401 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:265415 9.d 9:bc500964:::benchmark_data_smithi098_254407_object132684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-09-21T10:30:09.401 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:16.872895+0000", 2024-09-21T10:30:09.401 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 232.440755368, 2024-09-21T10:30:09.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.19584159200000001, 2024-09-21T10:30:09.401 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:30:09.401 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:30:09.402 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:30:09.402 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:30:09.402 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:30:09.402 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 265415 2024-09-21T10:30:09.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.402 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:30:09.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:30:09.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.872895+0000", 2024-09-21T10:30:09.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:30:09.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.872895+0000", 2024-09-21T10:30:09.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:30:09.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.872894+0000", 2024-09-21T10:30:09.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:30:09.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:30:09.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.872900+0000", 2024-09-21T10:30:09.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5719999999999999e-06 2024-09-21T10:30:09.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:30:09.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.872909+0000", 2024-09-21T10:30:09.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.064e-06 2024-09-21T10:30:09.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:30:09.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.872914+0000", 2024-09-21T10:30:09.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1810000000000003e-06 2024-09-21T10:30:09.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.874897+0000", 2024-09-21T10:30:09.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001983369 2024-09-21T10:30:09.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:30:09.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.875001+0000", 2024-09-21T10:30:09.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104219 2024-09-21T10:30:09.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:30:09.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.875102+0000", 2024-09-21T10:30:09.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010056300000000001 2024-09-21T10:30:09.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.878884+0000", 2024-09-21T10:30:09.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0037825889999999998 2024-09-21T10:30:09.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:30:09.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.878921+0000", 2024-09-21T10:30:09.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6433999999999997e-05 2024-09-21T10:30:09.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.068610+0000", 2024-09-21T10:30:09.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.189689575 2024-09-21T10:30:09.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:30:09.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.068704+0000", 2024-09-21T10:30:09.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3576e-05 2024-09-21T10:30:09.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:30:09.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.068737+0000", 2024-09-21T10:30:09.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2762000000000002e-05 2024-09-21T10:30:09.407 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.407 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:30:09.407 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.407 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:265428 9.7 9:e73ca6e3:::benchmark_data_smithi098_254407_object132697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-09-21T10:30:09.407 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:16.889473+0000", 2024-09-21T10:30:09.407 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 232.42417773899999, 2024-09-21T10:30:09.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.173842887, 2024-09-21T10:30:09.407 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:30:09.407 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:30:09.407 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:30:09.407 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:30:09.407 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:30:09.407 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 265428 2024-09-21T10:30:09.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.408 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:30:09.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:30:09.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.889473+0000", 2024-09-21T10:30:09.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:30:09.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.889473+0000", 2024-09-21T10:30:09.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:30:09.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.889471+0000", 2024-09-21T10:30:09.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:30:09.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:30:09.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.889479+0000", 2024-09-21T10:30:09.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4619999999999999e-06 2024-09-21T10:30:09.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:30:09.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.889493+0000", 2024-09-21T10:30:09.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4270999999999999e-05 2024-09-21T10:30:09.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:30:09.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.889499+0000", 2024-09-21T10:30:09.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4760000000000003e-06 2024-09-21T10:30:09.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.889535+0000", 2024-09-21T10:30:09.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5768e-05 2024-09-21T10:30:09.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:30:09.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.889613+0000", 2024-09-21T10:30:09.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7492999999999998e-05 2024-09-21T10:30:09.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:30:09.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.889679+0000", 2024-09-21T10:30:09.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6655000000000005e-05 2024-09-21T10:30:09.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:30:09.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.892647+0000", 2024-09-21T10:30:09.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029680240000000001 2024-09-21T10:30:09.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.892973+0000", 2024-09-21T10:30:09.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00032576099999999999 2024-09-21T10:30:09.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.063219+0000", 2024-09-21T10:30:09.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17024564 2024-09-21T10:30:09.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:30:09.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.063284+0000", 2024-09-21T10:30:09.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4943000000000005e-05 2024-09-21T10:30:09.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:30:09.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.063316+0000", 2024-09-21T10:30:09.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1937000000000003e-05 2024-09-21T10:30:09.413 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.413 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:30:09.413 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.413 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:265429 9.d 9:b5f6dec9:::benchmark_data_smithi098_254407_object132698:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-09-21T10:30:09.413 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:16.890079+0000", 2024-09-21T10:30:09.413 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 232.42357161000001, 2024-09-21T10:30:09.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18602902499999999, 2024-09-21T10:30:09.413 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:30:09.413 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:30:09.413 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:30:09.413 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:30:09.414 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:30:09.414 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 265429 2024-09-21T10:30:09.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.414 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:30:09.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:30:09.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.890079+0000", 2024-09-21T10:30:09.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:30:09.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.890079+0000", 2024-09-21T10:30:09.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:30:09.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.890078+0000", 2024-09-21T10:30:09.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:30:09.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:30:09.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.890083+0000", 2024-09-21T10:30:09.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9799999999999998e-06 2024-09-21T10:30:09.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:30:09.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.890096+0000", 2024-09-21T10:30:09.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2683e-05 2024-09-21T10:30:09.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:30:09.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.890101+0000", 2024-09-21T10:30:09.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7089999999999999e-06 2024-09-21T10:30:09.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.890144+0000", 2024-09-21T10:30:09.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2836000000000001e-05 2024-09-21T10:30:09.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:30:09.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.890229+0000", 2024-09-21T10:30:09.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5389999999999994e-05 2024-09-21T10:30:09.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:30:09.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.890294+0000", 2024-09-21T10:30:09.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4559000000000003e-05 2024-09-21T10:30:09.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:30:09.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.892970+0000", 2024-09-21T10:30:09.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026759370000000002 2024-09-21T10:30:09.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.893239+0000", 2024-09-21T10:30:09.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00026952699999999998 2024-09-21T10:30:09.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.075955+0000", 2024-09-21T10:30:09.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18271579900000001 2024-09-21T10:30:09.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:30:09.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.076065+0000", 2024-09-21T10:30:09.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109869 2024-09-21T10:30:09.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:30:09.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.076108+0000", 2024-09-21T10:30:09.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2698999999999999e-05 2024-09-21T10:30:09.419 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.419 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:30:09.419 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.419 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46322.0:115193 10.c 10:3afafc88:::benchmark_data_smithi098_281896_object115192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-21T10:30:09.420 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:28:03.255737+0000", 2024-09-21T10:30:09.420 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 126.05791303300001, 2024-09-21T10:30:09.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.11935325500000001, 2024-09-21T10:30:09.420 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:30:09.420 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:30:09.420 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:30:09.420 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46322", 2024-09-21T10:30:09.420 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:30:09.420 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 115193 2024-09-21T10:30:09.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.420 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:30:09.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:30:09.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.255737+0000", 2024-09-21T10:30:09.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:30:09.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.255737+0000", 2024-09-21T10:30:09.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:30:09.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.255736+0000", 2024-09-21T10:30:09.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:30:09.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:30:09.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.255828+0000", 2024-09-21T10:30:09.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2367999999999994e-05 2024-09-21T10:30:09.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:30:09.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.255842+0000", 2024-09-21T10:30:09.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3935e-05 2024-09-21T10:30:09.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:30:09.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.255847+0000", 2024-09-21T10:30:09.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8250000000000004e-06 2024-09-21T10:30:09.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.255888+0000", 2024-09-21T10:30:09.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0448999999999998e-05 2024-09-21T10:30:09.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:30:09.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.255980+0000", 2024-09-21T10:30:09.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2737000000000003e-05 2024-09-21T10:30:09.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:30:09.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.256074+0000", 2024-09-21T10:30:09.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3426999999999998e-05 2024-09-21T10:30:09.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:30:09.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.261505+0000", 2024-09-21T10:30:09.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.005431015 2024-09-21T10:30:09.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.261537+0000", 2024-09-21T10:30:09.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2450000000000003e-05 2024-09-21T10:30:09.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.374986+0000", 2024-09-21T10:30:09.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.11344886899999999 2024-09-21T10:30:09.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:30:09.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.375046+0000", 2024-09-21T10:30:09.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9871000000000003e-05 2024-09-21T10:30:09.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:30:09.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.375091+0000", 2024-09-21T10:30:09.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.464e-05 2024-09-21T10:30:09.425 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.425 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:30:09.425 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.426 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46322.0:115194 10.c 10:36de5723:::benchmark_data_smithi098_281896_object115193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-21T10:30:09.426 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:28:03.256882+0000", 2024-09-21T10:30:09.426 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 126.056768359, 2024-09-21T10:30:09.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.118306798, 2024-09-21T10:30:09.426 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:30:09.426 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:30:09.426 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:30:09.426 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46322", 2024-09-21T10:30:09.426 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:30:09.426 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 115194 2024-09-21T10:30:09.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.426 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:30:09.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:30:09.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.256882+0000", 2024-09-21T10:30:09.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:30:09.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.256882+0000", 2024-09-21T10:30:09.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:30:09.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.256881+0000", 2024-09-21T10:30:09.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:30:09.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:30:09.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.256949+0000", 2024-09-21T10:30:09.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8158999999999996e-05 2024-09-21T10:30:09.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:30:09.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.256960+0000", 2024-09-21T10:30:09.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0475000000000001e-05 2024-09-21T10:30:09.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:30:09.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.256964+0000", 2024-09-21T10:30:09.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7500000000000001e-06 2024-09-21T10:30:09.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.258751+0000", 2024-09-21T10:30:09.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0017875580000000001 2024-09-21T10:30:09.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:30:09.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.258833+0000", 2024-09-21T10:30:09.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1308000000000001e-05 2024-09-21T10:30:09.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:30:09.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.258931+0000", 2024-09-21T10:30:09.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8045000000000001e-05 2024-09-21T10:30:09.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:30:09.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.261642+0000", 2024-09-21T10:30:09.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027117729999999998 2024-09-21T10:30:09.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.264241+0000", 2024-09-21T10:30:09.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025984179999999999 2024-09-21T10:30:09.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.375123+0000", 2024-09-21T10:30:09.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.110881718 2024-09-21T10:30:09.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:30:09.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.375160+0000", 2024-09-21T10:30:09.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7440000000000001e-05 2024-09-21T10:30:09.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:30:09.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.375189+0000", 2024-09-21T10:30:09.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8898999999999999e-05 2024-09-21T10:30:09.432 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.432 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:30:09.432 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.432 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46322.0:141177 10.c 10:35bed12b:::benchmark_data_smithi098_281896_object16152:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-21T10:30:09.432 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:28:40.807186+0000", 2024-09-21T10:30:09.432 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 88.506464805999997, 2024-09-21T10:30:09.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.136343505, 2024-09-21T10:30:09.432 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:30:09.432 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:30:09.432 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:30:09.432 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46322", 2024-09-21T10:30:09.433 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:30:09.433 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 141177 2024-09-21T10:30:09.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.433 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:30:09.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:30:09.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.807186+0000", 2024-09-21T10:30:09.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:30:09.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.807186+0000", 2024-09-21T10:30:09.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:30:09.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.807184+0000", 2024-09-21T10:30:09.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:30:09.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:30:09.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.807193+0000", 2024-09-21T10:30:09.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9940000000000006e-06 2024-09-21T10:30:09.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:30:09.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.807205+0000", 2024-09-21T10:30:09.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1576000000000001e-05 2024-09-21T10:30:09.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:30:09.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.807210+0000", 2024-09-21T10:30:09.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7829999999999998e-06 2024-09-21T10:30:09.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.807252+0000", 2024-09-21T10:30:09.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2020000000000001e-05 2024-09-21T10:30:09.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:30:09.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.807332+0000", 2024-09-21T10:30:09.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0029000000000002e-05 2024-09-21T10:30:09.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:30:09.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.807400+0000", 2024-09-21T10:30:09.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8283999999999999e-05 2024-09-21T10:30:09.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:30:09.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.810860+0000", 2024-09-21T10:30:09.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034603360000000001 2024-09-21T10:30:09.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.811430+0000", 2024-09-21T10:30:09.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00056996800000000002 2024-09-21T10:30:09.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.943442+0000", 2024-09-21T10:30:09.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13201190500000001 2024-09-21T10:30:09.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:30:09.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.943490+0000", 2024-09-21T10:30:09.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8007999999999998e-05 2024-09-21T10:30:09.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:30:09.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.943529+0000", 2024-09-21T10:30:09.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8911000000000001e-05 2024-09-21T10:30:09.437 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.438 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:30:09.438 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.438 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46322.0:141180 10.c 10:3d593fe7:::benchmark_data_smithi098_281896_object16155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-21T10:30:09.438 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:28:40.813316+0000", 2024-09-21T10:30:09.438 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 88.500334014999993, 2024-09-21T10:30:09.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17292048700000001, 2024-09-21T10:30:09.438 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:30:09.438 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:30:09.438 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:30:09.438 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46322", 2024-09-21T10:30:09.438 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:30:09.438 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 141180 2024-09-21T10:30:09.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.439 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:30:09.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:30:09.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.813316+0000", 2024-09-21T10:30:09.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:30:09.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.813316+0000", 2024-09-21T10:30:09.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:30:09.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.813315+0000", 2024-09-21T10:30:09.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:30:09.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:30:09.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.813324+0000", 2024-09-21T10:30:09.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2269999999999999e-06 2024-09-21T10:30:09.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:30:09.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.813338+0000", 2024-09-21T10:30:09.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3951000000000001e-05 2024-09-21T10:30:09.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:30:09.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.813344+0000", 2024-09-21T10:30:09.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6409999999999999e-06 2024-09-21T10:30:09.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.813382+0000", 2024-09-21T10:30:09.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8318999999999997e-05 2024-09-21T10:30:09.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:30:09.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.813497+0000", 2024-09-21T10:30:09.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000115387 2024-09-21T10:30:09.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:30:09.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.813607+0000", 2024-09-21T10:30:09.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011003800000000001 2024-09-21T10:30:09.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:30:09.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.817642+0000", 2024-09-21T10:30:09.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0040343280000000002 2024-09-21T10:30:09.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.817863+0000", 2024-09-21T10:30:09.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00022094000000000001 2024-09-21T10:30:09.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.986127+0000", 2024-09-21T10:30:09.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.168264527 2024-09-21T10:30:09.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:30:09.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.986196+0000", 2024-09-21T10:30:09.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8460000000000005e-05 2024-09-21T10:30:09.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:30:09.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.986237+0000", 2024-09-21T10:30:09.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1356000000000002e-05 2024-09-21T10:30:09.443 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.443 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:30:09.443 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.444 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46322.0:141196 10.c 10:36a81655:::benchmark_data_smithi098_281896_object16171:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-21T10:30:09.444 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:28:40.833528+0000", 2024-09-21T10:30:09.444 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 88.480122050999995, 2024-09-21T10:30:09.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16630146700000001, 2024-09-21T10:30:09.444 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:30:09.444 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:30:09.444 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:30:09.444 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46322", 2024-09-21T10:30:09.444 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:30:09.444 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 141196 2024-09-21T10:30:09.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.444 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:30:09.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:30:09.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.833528+0000", 2024-09-21T10:30:09.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:30:09.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.833528+0000", 2024-09-21T10:30:09.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:30:09.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:30:09.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.833527+0000", 2024-09-21T10:30:09.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:30:09.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:30:09.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.833534+0000", 2024-09-21T10:30:09.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1620000000000002e-06 2024-09-21T10:30:09.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.446 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:30:09.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.833549+0000", 2024-09-21T10:30:09.446 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5021000000000001e-05 2024-09-21T10:30:09.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.446 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:30:09.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.833555+0000", 2024-09-21T10:30:09.446 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6000000000000003e-06 2024-09-21T10:30:09.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.446 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:30:09.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.833603+0000", 2024-09-21T10:30:09.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7586999999999997e-05 2024-09-21T10:30:09.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:30:09.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.833722+0000", 2024-09-21T10:30:09.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119306 2024-09-21T10:30:09.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:30:09.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.833840+0000", 2024-09-21T10:30:09.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000117861 2024-09-21T10:30:09.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:30:09.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.837869+0000", 2024-09-21T10:30:09.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0040292380000000001 2024-09-21T10:30:09.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.838114+0000", 2024-09-21T10:30:09.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00024487000000000001 2024-09-21T10:30:09.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:09.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.999732+0000", 2024-09-21T10:30:09.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.161618176 2024-09-21T10:30:09.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:30:09.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.999787+0000", 2024-09-21T10:30:09.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4877000000000003e-05 2024-09-21T10:30:09.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:30:09.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:30:09.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:30:09.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.999830+0000", 2024-09-21T10:30:09.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2453000000000002e-05 2024-09-21T10:30:09.449 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.449 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:30:09.449 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.449 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:30:09.449 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:30:09.449 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:30:09.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:09.503 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:09.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:09.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:09.604 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:09.632 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:09.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:09.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:09.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:09 smithi196 ceph-mon[84784]: pgmap v1306: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 668 op/s; 16843/156735 objects degraded (10.746%); 9119/156735 objects misplaced (5.818%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:30:09.734 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:09.739 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:09 smithi098 ceph-mon[97580]: pgmap v1306: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 668 op/s; 16843/156735 objects degraded (10.746%); 9119/156735 objects misplaced (5.818%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:30:09.740 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:09 smithi098 ceph-mon[100914]: pgmap v1306: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 668 op/s; 16843/156735 objects degraded (10.746%); 9119/156735 objects misplaced (5.818%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:30:09.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:09.755 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:09.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:09.856 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:09.892 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:09.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:09.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:09.994 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:10.026 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:10.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:10.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:10.128 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:10.155 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:10.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:10.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:10.256 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:30:10.256 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:10.312 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:10.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:10.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:10.413 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:10.463 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:30:10.464 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:30:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:10.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:10.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:10.473 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:30:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:30:10.510 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:30:10.510 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:30:10.551 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:30:10.552 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:30:10.569 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:10.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:10 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/981015312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:30:10.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:10 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/981015312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:30:10.582 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:30:10.582 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:30:10.595 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:10.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:10.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:10.629 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:30:10.629 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:30:10.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:10 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/981015312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:30:10.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:30:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:30:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:30:10.670 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:30:10.670 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:30:10.697 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:10.713 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:30:10.713 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:30:10.720 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:10.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:10.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:10.822 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:10.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:30:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:30:10] "GET /metrics HTTP/1.1" 200 36615 "" "Prometheus/2.51.0" 2024-09-21T10:30:10.848 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:10.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:10.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:10.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:10.975 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:10.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:10.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:11.061 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:30:11.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:30:11.096 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:30:11.156 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:11.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:11.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:11.258 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:11.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:11.327 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:11.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:11.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:11.457 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:11.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:11.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:11.559 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:11.600 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:11.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:11.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:11.701 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:11.727 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:11.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:11.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:11.802 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:11 smithi098 ceph-mon[97580]: pgmap v1307: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 16952/148347 objects degraded (11.427%); 8463/148347 objects misplaced (5.705%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:30:11.802 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:11 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 16952/148347 objects degraded (11.427%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:11.806 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:11 smithi098 ceph-mon[100914]: pgmap v1307: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 16952/148347 objects degraded (11.427%); 8463/148347 objects misplaced (5.705%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:30:11.806 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:11 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 16952/148347 objects degraded (11.427%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:11.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:11.843 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:11.857 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:11.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:11.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:11.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:11 smithi196 ceph-mon[84784]: pgmap v1307: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 16952/148347 objects degraded (11.427%); 8463/148347 objects misplaced (5.705%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:30:11.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:11 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 16952/148347 objects degraded (11.427%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:11.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:12.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:12.003 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:12.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:12.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:12.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:12.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:12.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:12.278 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:12.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:12.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:12.380 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:12.404 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:12.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:12.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:12.418 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1307,"stamp":"2024-09-21T10:30:09.610414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3243370891,"num_objects":49449,"num_object_clones":0,"num_object_copies":148347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16952,"num_objects_misplaced":8463,"num_objects_unfound":0,"num_objects_dirty":49449,"num_whiteouts":0,"num_read":6012,"num_read_kb":13076,"num_write":201545,"num_write_kb":8014180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2081,"num_bytes_recovered":129301458,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129056,"ondisk_log_size":129056,"up":297,"acting":294,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":302,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":16983756,"kb_used_data":10513224,"kb_used_omap":35,"kb_used_meta":6470172,"kb_avail":732977460,"statfs":{"total":767960285184,"available":750568919040,"internally_reserved":0,"allocated":10765541376,"data_stored":10748192987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36353,"internal_metadata":6625456639},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,8,8,10,6,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-436862976,"num_objects":-6666,"num_object_clones":0,"num_object_copies":-19998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":221,"num_objects_misplaced":-1308,"num_objects_unfound":0,"num_objects_dirty":-6666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":13762560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002941"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":50,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.452998+0000","last_change":"2024-09-21T10:28:00.289895+0000","last_active":"2024-09-21T10:30:05.452998+0000","last_peered":"2024-09-21T10:30:05.452998+0000","last_clean":"2024-09-21T10:30:05.452998+0000","last_became_active":"2024-09-21T10:28:00.289622+0000","last_became_peered":"2024-09-21T10:28:00.289622+0000","last_unstale":"2024-09-21T10:30:05.452998+0000","last_undegraded":"2024-09-21T10:30:05.452998+0000","last_fullsized":"2024-09-21T10:30:05.452998+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:35:31.514813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":410,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.367026+0000","last_change":"2024-09-21T10:24:44.305198+0000","last_active":"2024-09-21T10:29:52.367026+0000","last_peered":"2024-09-21T10:29:52.367026+0000","last_clean":"2024-09-21T10:29:52.367026+0000","last_became_active":"2024-09-21T10:24:44.297653+0000","last_became_peered":"2024-09-21T10:24:44.297653+0000","last_unstale":"2024-09-21T10:29:52.367026+0000","last_undegraded":"2024-09-21T10:29:52.367026+0000","last_fullsized":"2024-09-21T10:29:52.367026+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:19.320833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":247,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.371576+0000","last_change":"2024-09-21T10:28:00.294917+0000","last_active":"2024-09-21T10:29:52.371576+0000","last_peered":"2024-09-21T10:29:52.371576+0000","last_clean":"2024-09-21T10:29:52.371576+0000","last_became_active":"2024-09-21T10:28:00.294134+0000","last_became_peered":"2024-09-21T10:28:00.294134+0000","last_unstale":"2024-09-21T10:29:52.371576+0000","last_undegraded":"2024-09-21T10:29:52.371576+0000","last_fullsized":"2024-09-21T10:29:52.371576+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:29:05.780212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":196,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.366732+0000","last_change":"2024-09-21T10:28:00.287636+0000","last_active":"2024-09-21T10:29:52.366732+0000","last_peered":"2024-09-21T10:29:52.366732+0000","last_clean":"2024-09-21T10:29:52.366732+0000","last_became_active":"2024-09-21T10:28:00.287308+0000","last_became_peered":"2024-09-21T10:28:00.287308+0000","last_unstale":"2024-09-21T10:29:52.366732+0000","last_undegraded":"2024-09-21T10:29:52.366732+0000","last_fullsized":"2024-09-21T10:29:52.366732+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:41:46.132365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":646,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.441843+0000","last_change":"2024-09-21T10:28:00.297783+0000","last_active":"2024-09-21T10:30:05.441843+0000","last_peered":"2024-09-21T10:30:05.441843+0000","last_clean":"2024-09-21T10:30:05.441843+0000","last_became_active":"2024-09-21T10:28:00.297024+0000","last_became_peered":"2024-09-21T10:28:00.297024+0000","last_unstale":"2024-09-21T10:30:05.441843+0000","last_undegraded":"2024-09-21T10:30:05.441843+0000","last_fullsized":"2024-09-21T10:30:05.441843+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:24.382523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":69,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.369632+0000","last_change":"2024-09-21T10:24:44.304399+0000","last_active":"2024-09-21T10:29:52.369632+0000","last_peered":"2024-09-21T10:29:52.369632+0000","last_clean":"2024-09-21T10:29:52.369632+0000","last_became_active":"2024-09-21T10:24:44.303437+0000","last_became_peered":"2024-09-21T10:24:44.303437+0000","last_unstale":"2024-09-21T10:29:52.369632+0000","last_undegraded":"2024-09-21T10:29:52.369632+0000","last_fullsized":"2024-09-21T10:29:52.369632+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:32:56.638481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":50,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.439986+0000","last_change":"2024-09-21T10:28:00.293505+0000","last_active":"2024-09-21T10:30:05.439986+0000","last_peered":"2024-09-21T10:30:05.439986+0000","last_clean":"2024-09-21T10:30:05.439986+0000","last_became_active":"2024-09-21T10:28:00.293173+0000","last_became_peered":"2024-09-21T10:28:00.293173+0000","last_unstale":"2024-09-21T10:30:05.439986+0000","last_undegraded":"2024-09-21T10:30:05.439986+0000","last_fullsized":"2024-09-21T10:30:05.439986+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:16.878847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":352,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.376735+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:29:52.376735+0000","last_peered":"2024-09-21T10:29:52.376735+0000","last_clean":"2024-09-21T10:29:52.376735+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:29:52.376735+0000","last_undegraded":"2024-09-21T10:29:52.376735+0000","last_fullsized":"2024-09-21T10:29:52.376735+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"380'12640","reported_seq":12717,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:09.518822+0000","last_change":"2024-09-21T10:26:36.989699+0000","last_active":"2024-09-21T10:30:09.518822+0000","last_peered":"2024-09-21T10:30:09.518822+0000","last_clean":"2024-09-21T10:30:09.518822+0000","last_became_active":"2024-09-21T10:26:36.989318+0000","last_became_peered":"2024-09-21T10:26:36.989318+0000","last_unstale":"2024-09-21T10:30:09.518822+0000","last_undegraded":"2024-09-21T10:30:09.518822+0000","last_fullsized":"2024-09-21T10:30:09.518822+0000","mapping_epoch":348,"log_start":"357'5300","ondisk_log_start":"357'5300","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7340,"log_dups_size":0,"ondisk_log_size":7340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:54:53.567344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197787648,"num_objects":3018,"num_object_clones":0,"num_object_copies":9054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":501056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":401,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.367276+0000","last_change":"2024-09-21T10:24:44.325196+0000","last_active":"2024-09-21T10:29:52.367276+0000","last_peered":"2024-09-21T10:29:52.367276+0000","last_clean":"2024-09-21T10:29:52.367276+0000","last_became_active":"2024-09-21T10:24:44.299614+0000","last_became_peered":"2024-09-21T10:24:44.299614+0000","last_unstale":"2024-09-21T10:29:52.367276+0000","last_undegraded":"2024-09-21T10:29:52.367276+0000","last_fullsized":"2024-09-21T10:29:52.367276+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:14.558663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":133,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.371441+0000","last_change":"2024-09-21T10:26:41.305571+0000","last_active":"2024-09-21T10:29:52.371441+0000","last_peered":"2024-09-21T10:29:52.371441+0000","last_clean":"2024-09-21T10:29:52.371441+0000","last_became_active":"2024-09-21T10:26:41.305135+0000","last_became_peered":"2024-09-21T10:26:41.305135+0000","last_unstale":"2024-09-21T10:29:52.371441+0000","last_undegraded":"2024-09-21T10:29:52.371441+0000","last_fullsized":"2024-09-21T10:29:52.371441+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:49:55.452467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"380'12356","reported_seq":12407,"reported_epoch":380,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:30:08.039553+0000","last_change":"2024-09-21T10:28:01.438839+0000","last_active":"2024-09-21T10:30:08.039553+0000","last_peered":"2024-09-21T10:30:08.039553+0000","last_clean":"2024-09-21T10:27:59.232048+0000","last_became_active":"2024-09-21T10:28:01.433549+0000","last_became_peered":"2024-09-21T10:28:01.433549+0000","last_unstale":"2024-09-21T10:30:08.039553+0000","last_undegraded":"2024-09-21T10:30:08.039553+0000","last_fullsized":"2024-09-21T10:30:08.039553+0000","mapping_epoch":361,"log_start":"355'4600","ondisk_log_start":"355'4600","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7756,"log_dups_size":0,"ondisk_log_size":7756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:58.936666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201981952,"num_objects":3082,"num_object_clones":0,"num_object_copies":9246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3082,"num_objects_unfound":0,"num_objects_dirty":3082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12356,"num_write_kb":494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,5,2],"avail_no_missing":["6","2","5"],"object_location_counts":[{"shards":"2,5,6","objects":3082}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":402,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.367008+0000","last_change":"2024-09-21T10:24:44.323182+0000","last_active":"2024-09-21T10:29:52.367008+0000","last_peered":"2024-09-21T10:29:52.367008+0000","last_clean":"2024-09-21T10:29:52.367008+0000","last_became_active":"2024-09-21T10:24:44.299698+0000","last_became_peered":"2024-09-21T10:24:44.299698+0000","last_unstale":"2024-09-21T10:29:52.367008+0000","last_undegraded":"2024-09-21T10:29:52.367008+0000","last_fullsized":"2024-09-21T10:29:52.367008+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:56.571057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":79,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.369846+0000","last_change":"2024-09-21T10:28:00.281253+0000","last_active":"2024-09-21T10:29:52.369846+0000","last_peered":"2024-09-21T10:29:52.369846+0000","last_clean":"2024-09-21T10:29:52.369846+0000","last_became_active":"2024-09-21T10:28:00.280773+0000","last_became_peered":"2024-09-21T10:28:00.280773+0000","last_unstale":"2024-09-21T10:29:52.369846+0000","last_undegraded":"2024-09-21T10:29:52.369846+0000","last_fullsized":"2024-09-21T10:29:52.369846+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:32:48.557279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"380'12445","reported_seq":16360,"reported_epoch":380,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T10:30:09.516032+0000","last_change":"2024-09-21T10:28:25.623040+0000","last_active":"2024-09-21T10:30:09.516032+0000","last_peered":"2024-09-21T10:30:09.516032+0000","last_clean":"2024-09-21T10:27:59.294389+0000","last_became_active":"2024-09-21T10:28:01.608572+0000","last_became_peered":"2024-09-21T10:28:01.608572+0000","last_unstale":"2024-09-21T10:30:09.516032+0000","last_undegraded":"2024-09-21T10:28:25.614349+0000","last_fullsized":"2024-09-21T10:28:01.272643+0000","mapping_epoch":361,"log_start":"357'5100","ondisk_log_start":"357'5100","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7345,"log_dups_size":0,"ondisk_log_size":7345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198770688,"num_objects":3033,"num_object_clones":0,"num_object_copies":9099,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2789,"num_objects_misplaced":2321,"num_objects_unfound":0,"num_objects_dirty":3033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":495296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1918,"num_bytes_recovered":125698048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":2321},{"shards":"0,4","objects":-2077}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":328,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.366926+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:29:52.366926+0000","last_peered":"2024-09-21T10:29:52.366926+0000","last_clean":"2024-09-21T10:29:52.366926+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:29:52.366926+0000","last_undegraded":"2024-09-21T10:29:52.366926+0000","last_fullsized":"2024-09-21T10:29:52.366926+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":343,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.375471+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:29:52.375471+0000","last_peered":"2024-09-21T10:29:52.375471+0000","last_clean":"2024-09-21T10:29:52.375471+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:29:52.375471+0000","last_undegraded":"2024-09-21T10:29:52.375471+0000","last_fullsized":"2024-09-21T10:29:52.375471+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"379'12221","reported_seq":12299,"reported_epoch":379,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:30:05.380880+0000","last_change":"2024-09-21T10:28:01.644360+0000","last_active":"2024-09-21T10:30:05.380880+0000","last_peered":"2024-09-21T10:30:05.380880+0000","last_clean":"2024-09-21T10:27:59.276032+0000","last_became_active":"2024-09-21T10:28:01.624219+0000","last_became_peered":"2024-09-21T10:28:01.624219+0000","last_unstale":"2024-09-21T10:30:05.380880+0000","last_undegraded":"2024-09-21T10:28:01.350374+0000","last_fullsized":"2024-09-21T10:28:01.295088+0000","mapping_epoch":361,"log_start":"355'2800","ondisk_log_start":"355'2800","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":9421,"log_dups_size":0,"ondisk_log_size":9421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206897152,"num_objects":3157,"num_object_clones":0,"num_object_copies":9471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12221,"num_write_kb":492096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0],"avail_no_missing":["2","0","7"],"object_location_counts":[{"shards":"0,2","objects":-3857},{"shards":"0,2,7","objects":2482}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":50,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.452903+0000","last_change":"2024-09-21T10:28:00.288968+0000","last_active":"2024-09-21T10:30:05.452903+0000","last_peered":"2024-09-21T10:30:05.452903+0000","last_clean":"2024-09-21T10:30:05.452903+0000","last_became_active":"2024-09-21T10:28:00.288244+0000","last_became_peered":"2024-09-21T10:28:00.288244+0000","last_unstale":"2024-09-21T10:30:05.452903+0000","last_undegraded":"2024-09-21T10:30:05.452903+0000","last_fullsized":"2024-09-21T10:30:05.452903+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:42.089701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":668,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.441747+0000","last_change":"2024-09-21T10:24:44.299417+0000","last_active":"2024-09-21T10:30:05.441747+0000","last_peered":"2024-09-21T10:30:05.441747+0000","last_clean":"2024-09-21T10:30:05.441747+0000","last_became_active":"2024-09-21T10:24:44.299116+0000","last_became_peered":"2024-09-21T10:24:44.299116+0000","last_unstale":"2024-09-21T10:30:05.441747+0000","last_undegraded":"2024-09-21T10:30:05.441747+0000","last_fullsized":"2024-09-21T10:30:05.441747+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:44.524841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"380'12693","reported_seq":12770,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:07.395012+0000","last_change":"2024-09-21T10:26:36.196642+0000","last_active":"2024-09-21T10:30:07.395012+0000","last_peered":"2024-09-21T10:30:07.395012+0000","last_clean":"2024-09-21T10:30:07.395012+0000","last_became_active":"2024-09-21T10:26:36.196357+0000","last_became_peered":"2024-09-21T10:26:36.196357+0000","last_unstale":"2024-09-21T10:30:07.395012+0000","last_undegraded":"2024-09-21T10:30:07.395012+0000","last_fullsized":"2024-09-21T10:30:07.395012+0000","mapping_epoch":348,"log_start":"355'4100","ondisk_log_start":"355'4100","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":8593,"log_dups_size":0,"ondisk_log_size":8593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:54:19.391031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207814656,"num_objects":3171,"num_object_clones":0,"num_object_copies":9513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12693,"num_write_kb":507648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"82'100","reported_seq":501,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.366587+0000","last_change":"2024-09-21T10:28:00.285919+0000","last_active":"2024-09-21T10:29:52.366587+0000","last_peered":"2024-09-21T10:29:52.366587+0000","last_clean":"2024-09-21T10:29:52.366587+0000","last_became_active":"2024-09-21T10:28:00.285324+0000","last_became_peered":"2024-09-21T10:28:00.285324+0000","last_unstale":"2024-09-21T10:29:52.366587+0000","last_undegraded":"2024-09-21T10:29:52.366587+0000","last_fullsized":"2024-09-21T10:29:52.366587+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.728195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.728195+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:00.841610+0000","scrub_duration":0.0069698140000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":54,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.438126+0000","last_change":"2024-09-21T10:28:00.303125+0000","last_active":"2024-09-21T10:30:05.438126+0000","last_peered":"2024-09-21T10:30:05.438126+0000","last_clean":"2024-09-21T10:30:05.438126+0000","last_became_active":"2024-09-21T10:28:00.293684+0000","last_became_peered":"2024-09-21T10:28:00.293684+0000","last_unstale":"2024-09-21T10:30:05.438126+0000","last_undegraded":"2024-09-21T10:30:05.438126+0000","last_fullsized":"2024-09-21T10:30:05.438126+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:04.495913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":79,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.370687+0000","last_change":"2024-09-21T10:28:00.283456+0000","last_active":"2024-09-21T10:29:52.370687+0000","last_peered":"2024-09-21T10:29:52.370687+0000","last_clean":"2024-09-21T10:29:52.370687+0000","last_became_active":"2024-09-21T10:28:00.282881+0000","last_became_peered":"2024-09-21T10:28:00.282881+0000","last_unstale":"2024-09-21T10:29:52.370687+0000","last_undegraded":"2024-09-21T10:29:52.370687+0000","last_fullsized":"2024-09-21T10:29:52.370687+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:43.681481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"379'12534","reported_seq":12612,"reported_epoch":379,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:30:05.300579+0000","last_change":"2024-09-21T10:28:01.680872+0000","last_active":"2024-09-21T10:30:05.300579+0000","last_peered":"2024-09-21T10:30:05.300579+0000","last_clean":"2024-09-21T10:27:59.257182+0000","last_became_active":"2024-09-21T10:28:01.651875+0000","last_became_peered":"2024-09-21T10:28:01.651875+0000","last_unstale":"2024-09-21T10:30:05.300579+0000","last_undegraded":"2024-09-21T10:28:01.349133+0000","last_fullsized":"2024-09-21T10:28:01.272587+0000","mapping_epoch":361,"log_start":"355'3100","ondisk_log_start":"355'3100","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":9434,"log_dups_size":0,"ondisk_log_size":9434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210370560,"num_objects":3210,"num_object_clones":0,"num_object_copies":9630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12534,"num_write_kb":503808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5],"avail_no_missing":["2","4","5"],"object_location_counts":[{"shards":"2,4,5","objects":2487},{"shards":"2,5","objects":-3939}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'100","reported_seq":169,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.373920+0000","last_change":"2024-09-21T10:28:00.301695+0000","last_active":"2024-09-21T10:29:52.373920+0000","last_peered":"2024-09-21T10:29:52.373920+0000","last_clean":"2024-09-21T10:29:52.373920+0000","last_became_active":"2024-09-21T10:28:00.289856+0000","last_became_peered":"2024-09-21T10:28:00.289856+0000","last_unstale":"2024-09-21T10:29:52.373920+0000","last_undegraded":"2024-09-21T10:29:52.373920+0000","last_fullsized":"2024-09-21T10:29:52.373920+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:27.234001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:27.234001+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:02.389036+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.375725+0000","last_change":"2024-09-21T10:24:44.309470+0000","last_active":"2024-09-21T10:29:52.375725+0000","last_peered":"2024-09-21T10:29:52.375725+0000","last_clean":"2024-09-21T10:29:52.375725+0000","last_became_active":"2024-09-21T10:24:44.308836+0000","last_became_peered":"2024-09-21T10:24:44.308836+0000","last_unstale":"2024-09-21T10:29:52.375725+0000","last_undegraded":"2024-09-21T10:29:52.375725+0000","last_fullsized":"2024-09-21T10:29:52.375725+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:09:44.622418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":69,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.370374+0000","last_change":"2024-09-21T10:24:44.305135+0000","last_active":"2024-09-21T10:29:52.370374+0000","last_peered":"2024-09-21T10:29:52.370374+0000","last_clean":"2024-09-21T10:29:52.370374+0000","last_became_active":"2024-09-21T10:24:44.304839+0000","last_became_peered":"2024-09-21T10:24:44.304839+0000","last_unstale":"2024-09-21T10:29:52.370374+0000","last_undegraded":"2024-09-21T10:29:52.370374+0000","last_fullsized":"2024-09-21T10:29:52.370374+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:02.264404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"380'12517","reported_seq":12557,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:08.017256+0000","last_change":"2024-09-21T10:26:36.194994+0000","last_active":"2024-09-21T10:30:08.017256+0000","last_peered":"2024-09-21T10:30:08.017256+0000","last_clean":"2024-09-21T10:30:08.017256+0000","last_became_active":"2024-09-21T10:26:36.194404+0000","last_became_peered":"2024-09-21T10:26:36.194404+0000","last_unstale":"2024-09-21T10:30:08.017256+0000","last_undegraded":"2024-09-21T10:30:08.017256+0000","last_fullsized":"2024-09-21T10:30:08.017256+0000","mapping_epoch":348,"log_start":"355'4800","ondisk_log_start":"355'4800","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7717,"log_dups_size":0,"ondisk_log_size":7717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:10:51.551059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196542464,"num_objects":2999,"num_object_clones":0,"num_object_copies":8997,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12517,"num_write_kb":496512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":100,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.443775+0000","last_change":"2024-09-21T10:25:39.872680+0000","last_active":"2024-09-21T10:30:05.443775+0000","last_peered":"2024-09-21T10:30:05.443775+0000","last_clean":"2024-09-21T10:30:05.443775+0000","last_became_active":"2024-09-21T10:25:39.872312+0000","last_became_peered":"2024-09-21T10:25:39.872312+0000","last_unstale":"2024-09-21T10:30:05.443775+0000","last_undegraded":"2024-09-21T10:30:05.443775+0000","last_fullsized":"2024-09-21T10:30:05.443775+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:54.942046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":69,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.373329+0000","last_change":"2024-09-21T10:24:44.310484+0000","last_active":"2024-09-21T10:29:52.373329+0000","last_peered":"2024-09-21T10:29:52.373329+0000","last_clean":"2024-09-21T10:29:52.373329+0000","last_became_active":"2024-09-21T10:24:44.307209+0000","last_became_peered":"2024-09-21T10:24:44.307209+0000","last_unstale":"2024-09-21T10:29:52.373329+0000","last_undegraded":"2024-09-21T10:29:52.373329+0000","last_fullsized":"2024-09-21T10:29:52.373329+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:48:36.767539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"380'12617","reported_seq":12657,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:08.124055+0000","last_change":"2024-09-21T10:26:36.196826+0000","last_active":"2024-09-21T10:30:08.124055+0000","last_peered":"2024-09-21T10:30:08.124055+0000","last_clean":"2024-09-21T10:30:08.124055+0000","last_became_active":"2024-09-21T10:26:36.196355+0000","last_became_peered":"2024-09-21T10:26:36.196355+0000","last_unstale":"2024-09-21T10:30:08.124055+0000","last_undegraded":"2024-09-21T10:30:08.124055+0000","last_fullsized":"2024-09-21T10:30:08.124055+0000","mapping_epoch":348,"log_start":"358'5900","ondisk_log_start":"358'5900","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":6717,"log_dups_size":0,"ondisk_log_size":6717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:58.869309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198115328,"num_objects":3023,"num_object_clones":0,"num_object_copies":9069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12617,"num_write_kb":500480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'100","reported_seq":153,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.374033+0000","last_change":"2024-09-21T10:23:26.197358+0000","last_active":"2024-09-21T10:29:52.374033+0000","last_peered":"2024-09-21T10:29:52.374033+0000","last_clean":"2024-09-21T10:29:52.374033+0000","last_became_active":"2024-09-21T10:23:20.695174+0000","last_became_peered":"2024-09-21T10:23:20.695174+0000","last_unstale":"2024-09-21T10:29:52.374033+0000","last_undegraded":"2024-09-21T10:29:52.374033+0000","last_fullsized":"2024-09-21T10:29:52.374033+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:26.197291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:26.197291+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:33.961031+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":50,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.434463+0000","last_change":"2024-09-21T10:28:00.287560+0000","last_active":"2024-09-21T10:30:05.434463+0000","last_peered":"2024-09-21T10:30:05.434463+0000","last_clean":"2024-09-21T10:30:05.434463+0000","last_became_active":"2024-09-21T10:28:00.285076+0000","last_became_peered":"2024-09-21T10:28:00.285076+0000","last_unstale":"2024-09-21T10:30:05.434463+0000","last_undegraded":"2024-09-21T10:30:05.434463+0000","last_fullsized":"2024-09-21T10:30:05.434463+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:49.950523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":663,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.440281+0000","last_change":"2024-09-21T10:28:00.282062+0000","last_active":"2024-09-21T10:30:05.440281+0000","last_peered":"2024-09-21T10:30:05.440281+0000","last_clean":"2024-09-21T10:30:05.440281+0000","last_became_active":"2024-09-21T10:28:00.281593+0000","last_became_peered":"2024-09-21T10:28:00.281593+0000","last_unstale":"2024-09-21T10:30:05.440281+0000","last_undegraded":"2024-09-21T10:30:05.440281+0000","last_fullsized":"2024-09-21T10:30:05.440281+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:45.704840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"380'12547","reported_seq":12587,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:08.106227+0000","last_change":"2024-09-21T10:26:36.197465+0000","last_active":"2024-09-21T10:30:08.106227+0000","last_peered":"2024-09-21T10:30:08.106227+0000","last_clean":"2024-09-21T10:30:08.106227+0000","last_became_active":"2024-09-21T10:26:36.196105+0000","last_became_peered":"2024-09-21T10:26:36.196105+0000","last_unstale":"2024-09-21T10:30:08.106227+0000","last_undegraded":"2024-09-21T10:30:08.106227+0000","last_fullsized":"2024-09-21T10:30:08.106227+0000","mapping_epoch":348,"log_start":"355'4800","ondisk_log_start":"355'4800","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7747,"log_dups_size":0,"ondisk_log_size":7747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:16.586685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200605696,"num_objects":3061,"num_object_clones":0,"num_object_copies":9183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12547,"num_write_kb":499456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":482,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.366252+0000","last_change":"2024-09-21T10:23:18.790100+0000","last_active":"2024-09-21T10:29:52.366252+0000","last_peered":"2024-09-21T10:29:52.366252+0000","last_clean":"2024-09-21T10:29:52.366252+0000","last_became_active":"2024-09-21T10:23:14.677610+0000","last_became_peered":"2024-09-21T10:23:14.677610+0000","last_unstale":"2024-09-21T10:29:52.366252+0000","last_undegraded":"2024-09-21T10:29:52.366252+0000","last_fullsized":"2024-09-21T10:29:52.366252+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:18.790017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:18.790017+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:56.562241+0000","scrub_duration":0.0080400790000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":575,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.371469+0000","last_change":"2024-09-21T10:26:41.279374+0000","last_active":"2024-09-21T10:29:52.371469+0000","last_peered":"2024-09-21T10:29:52.371469+0000","last_clean":"2024-09-21T10:29:52.371469+0000","last_became_active":"2024-09-21T10:26:41.278994+0000","last_became_peered":"2024-09-21T10:26:41.278994+0000","last_unstale":"2024-09-21T10:29:52.371469+0000","last_undegraded":"2024-09-21T10:29:52.371469+0000","last_fullsized":"2024-09-21T10:29:52.371469+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:49:57.821175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":482,"num_read_kb":1581,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":51,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.434942+0000","last_change":"2024-09-21T10:28:00.295436+0000","last_active":"2024-09-21T10:30:05.434942+0000","last_peered":"2024-09-21T10:30:05.434942+0000","last_clean":"2024-09-21T10:30:05.434942+0000","last_became_active":"2024-09-21T10:28:00.295135+0000","last_became_peered":"2024-09-21T10:28:00.295135+0000","last_unstale":"2024-09-21T10:30:05.434942+0000","last_undegraded":"2024-09-21T10:30:05.434942+0000","last_fullsized":"2024-09-21T10:30:05.434942+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:14:03.027395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":366,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.375494+0000","last_change":"2024-09-21T10:24:44.308008+0000","last_active":"2024-09-21T10:29:52.375494+0000","last_peered":"2024-09-21T10:29:52.375494+0000","last_clean":"2024-09-21T10:29:52.375494+0000","last_became_active":"2024-09-21T10:24:44.307595+0000","last_became_peered":"2024-09-21T10:24:44.307595+0000","last_unstale":"2024-09-21T10:29:52.375494+0000","last_undegraded":"2024-09-21T10:29:52.375494+0000","last_fullsized":"2024-09-21T10:29:52.375494+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:33:49.448676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"380'12428","reported_seq":12503,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:09.505958+0000","last_change":"2024-09-21T10:26:36.197089+0000","last_active":"2024-09-21T10:30:09.505958+0000","last_peered":"2024-09-21T10:30:09.505958+0000","last_clean":"2024-09-21T10:30:09.505958+0000","last_became_active":"2024-09-21T10:26:36.196846+0000","last_became_peered":"2024-09-21T10:26:36.196846+0000","last_unstale":"2024-09-21T10:30:09.505958+0000","last_undegraded":"2024-09-21T10:30:09.505958+0000","last_fullsized":"2024-09-21T10:30:09.505958+0000","mapping_epoch":348,"log_start":"357'5100","ondisk_log_start":"357'5100","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7328,"log_dups_size":0,"ondisk_log_size":7328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:14:52.494647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199884800,"num_objects":3050,"num_object_clones":0,"num_object_copies":9150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":495296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"82'100","reported_seq":162,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.444165+0000","last_change":"2024-09-21T10:24:44.309154+0000","last_active":"2024-09-21T10:30:05.444165+0000","last_peered":"2024-09-21T10:30:05.444165+0000","last_clean":"2024-09-21T10:30:05.444165+0000","last_became_active":"2024-09-21T10:24:44.308330+0000","last_became_peered":"2024-09-21T10:24:44.308330+0000","last_unstale":"2024-09-21T10:30:05.444165+0000","last_undegraded":"2024-09-21T10:30:05.444165+0000","last_fullsized":"2024-09-21T10:30:05.444165+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.641718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.641718+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:10.158739+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":724,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.366105+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:29:52.366105+0000","last_peered":"2024-09-21T10:29:52.366105+0000","last_clean":"2024-09-21T10:29:52.366105+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:29:52.366105+0000","last_undegraded":"2024-09-21T10:29:52.366105+0000","last_fullsized":"2024-09-21T10:29:52.366105+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":676,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.443953+0000","last_change":"2024-09-21T10:28:00.296831+0000","last_active":"2024-09-21T10:30:05.443953+0000","last_peered":"2024-09-21T10:30:05.443953+0000","last_clean":"2024-09-21T10:30:05.443953+0000","last_became_active":"2024-09-21T10:28:00.296527+0000","last_became_peered":"2024-09-21T10:28:00.296527+0000","last_unstale":"2024-09-21T10:30:05.443953+0000","last_undegraded":"2024-09-21T10:30:05.443953+0000","last_fullsized":"2024-09-21T10:30:05.443953+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:18.024868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":644,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.442000+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:30:05.442000+0000","last_peered":"2024-09-21T10:30:05.442000+0000","last_clean":"2024-09-21T10:30:05.442000+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:30:05.442000+0000","last_undegraded":"2024-09-21T10:30:05.442000+0000","last_fullsized":"2024-09-21T10:30:05.442000+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"380'12946","reported_seq":12987,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:07.987218+0000","last_change":"2024-09-21T10:26:41.278294+0000","last_active":"2024-09-21T10:30:07.987218+0000","last_peered":"2024-09-21T10:30:07.987218+0000","last_clean":"2024-09-21T10:30:07.987218+0000","last_became_active":"2024-09-21T10:26:41.277965+0000","last_became_peered":"2024-09-21T10:26:41.277965+0000","last_unstale":"2024-09-21T10:30:07.987218+0000","last_undegraded":"2024-09-21T10:30:07.987218+0000","last_fullsized":"2024-09-21T10:30:07.987218+0000","mapping_epoch":351,"log_start":"355'3200","ondisk_log_start":"355'3200","created":348,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":9746,"log_dups_size":0,"ondisk_log_size":9746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:30.913837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204079104,"num_objects":3114,"num_object_clones":0,"num_object_copies":9342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12946,"num_write_kb":513920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"82'100","reported_seq":308,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.450556+0000","last_change":"2024-09-21T10:24:44.294460+0000","last_active":"2024-09-21T10:30:05.450556+0000","last_peered":"2024-09-21T10:30:05.450556+0000","last_clean":"2024-09-21T10:30:05.450556+0000","last_became_active":"2024-09-21T10:24:44.293972+0000","last_became_peered":"2024-09-21T10:24:44.293972+0000","last_unstale":"2024-09-21T10:30:05.450556+0000","last_undegraded":"2024-09-21T10:30:05.450556+0000","last_fullsized":"2024-09-21T10:30:05.450556+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:26.199522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:26.199522+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:26.135171+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":1452,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.376934+0000","last_change":"2024-09-21T10:28:03.346064+0000","last_active":"2024-09-21T10:29:52.376934+0000","last_peered":"2024-09-21T10:29:52.376934+0000","last_clean":"2024-09-21T10:29:52.376934+0000","last_became_active":"2024-09-21T10:28:03.345313+0000","last_became_peered":"2024-09-21T10:28:03.345313+0000","last_unstale":"2024-09-21T10:29:52.376934+0000","last_undegraded":"2024-09-21T10:29:52.376934+0000","last_fullsized":"2024-09-21T10:29:52.376934+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:32.097521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":516,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.440804+0000","last_change":"2024-09-21T10:08:18.470748+0000","last_active":"2024-09-21T10:30:05.440804+0000","last_peered":"2024-09-21T10:30:05.440804+0000","last_clean":"2024-09-21T10:30:05.440804+0000","last_became_active":"2024-09-21T10:08:18.470227+0000","last_became_peered":"2024-09-21T10:08:18.470227+0000","last_unstale":"2024-09-21T10:30:05.440804+0000","last_undegraded":"2024-09-21T10:30:05.440804+0000","last_fullsized":"2024-09-21T10:30:05.440804+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:16.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":637,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.378040+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:29:52.378040+0000","last_peered":"2024-09-21T10:29:52.378040+0000","last_clean":"2024-09-21T10:29:52.378040+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:29:52.378040+0000","last_undegraded":"2024-09-21T10:29:52.378040+0000","last_fullsized":"2024-09-21T10:29:52.378040+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"380'12729","reported_seq":12769,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:08.117005+0000","last_change":"2024-09-21T10:26:36.989641+0000","last_active":"2024-09-21T10:30:08.117005+0000","last_peered":"2024-09-21T10:30:08.117005+0000","last_clean":"2024-09-21T10:30:08.117005+0000","last_became_active":"2024-09-21T10:26:36.989340+0000","last_became_peered":"2024-09-21T10:26:36.989340+0000","last_unstale":"2024-09-21T10:30:08.117005+0000","last_undegraded":"2024-09-21T10:30:08.117005+0000","last_fullsized":"2024-09-21T10:30:08.117005+0000","mapping_epoch":348,"log_start":"356'5000","ondisk_log_start":"356'5000","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7729,"log_dups_size":0,"ondisk_log_size":7729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:50.213332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202440704,"num_objects":3089,"num_object_clones":0,"num_object_copies":9267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12729,"num_write_kb":506176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"82'100","reported_seq":301,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.443872+0000","last_change":"2024-09-21T10:23:21.277739+0000","last_active":"2024-09-21T10:30:05.443872+0000","last_peered":"2024-09-21T10:30:05.443872+0000","last_clean":"2024-09-21T10:30:05.443872+0000","last_became_active":"2024-09-21T10:23:20.689733+0000","last_became_peered":"2024-09-21T10:23:20.689733+0000","last_unstale":"2024-09-21T10:30:05.443872+0000","last_undegraded":"2024-09-21T10:30:05.443872+0000","last_fullsized":"2024-09-21T10:30:05.443872+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.277663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.277663+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:01:54.523189+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":1115,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.441591+0000","last_change":"2024-09-21T10:07:43.794767+0000","last_active":"2024-09-21T10:30:05.441591+0000","last_peered":"2024-09-21T10:30:05.441591+0000","last_clean":"2024-09-21T10:30:05.441591+0000","last_became_active":"2024-09-21T10:07:43.794062+0000","last_became_peered":"2024-09-21T10:07:43.794062+0000","last_unstale":"2024-09-21T10:30:05.441591+0000","last_undegraded":"2024-09-21T10:30:05.441591+0000","last_fullsized":"2024-09-21T10:30:05.441591+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:51.977955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":69,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.370141+0000","last_change":"2024-09-21T10:24:44.299492+0000","last_active":"2024-09-21T10:29:52.370141+0000","last_peered":"2024-09-21T10:29:52.370141+0000","last_clean":"2024-09-21T10:29:52.370141+0000","last_became_active":"2024-09-21T10:24:44.299187+0000","last_became_peered":"2024-09-21T10:24:44.299187+0000","last_unstale":"2024-09-21T10:29:52.370141+0000","last_undegraded":"2024-09-21T10:29:52.370141+0000","last_fullsized":"2024-09-21T10:29:52.370141+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:58.500386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":567,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.374334+0000","last_change":"2024-09-21T10:28:00.289453+0000","last_active":"2024-09-21T10:29:52.374334+0000","last_peered":"2024-09-21T10:29:52.374334+0000","last_clean":"2024-09-21T10:29:52.374334+0000","last_became_active":"2024-09-21T10:28:00.289030+0000","last_became_peered":"2024-09-21T10:28:00.289030+0000","last_unstale":"2024-09-21T10:29:52.374334+0000","last_undegraded":"2024-09-21T10:29:52.374334+0000","last_fullsized":"2024-09-21T10:29:52.374334+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:21.959998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"380'12395","reported_seq":12435,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:07.935839+0000","last_change":"2024-09-21T10:26:36.196471+0000","last_active":"2024-09-21T10:30:07.935839+0000","last_peered":"2024-09-21T10:30:07.935839+0000","last_clean":"2024-09-21T10:30:07.935839+0000","last_became_active":"2024-09-21T10:26:36.196167+0000","last_became_peered":"2024-09-21T10:26:36.196167+0000","last_unstale":"2024-09-21T10:30:07.935839+0000","last_undegraded":"2024-09-21T10:30:07.935839+0000","last_fullsized":"2024-09-21T10:30:07.935839+0000","mapping_epoch":348,"log_start":"355'4500","ondisk_log_start":"355'4500","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7895,"log_dups_size":0,"ondisk_log_size":7895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:16.488795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202178560,"num_objects":3085,"num_object_clones":0,"num_object_copies":9255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12395,"num_write_kb":495360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"82'100","reported_seq":244,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.376773+0000","last_change":"2024-09-21T10:28:00.294997+0000","last_active":"2024-09-21T10:29:52.376773+0000","last_peered":"2024-09-21T10:29:52.376773+0000","last_clean":"2024-09-21T10:29:52.376773+0000","last_became_active":"2024-09-21T10:28:00.294595+0000","last_became_peered":"2024-09-21T10:28:00.294595+0000","last_unstale":"2024-09-21T10:29:52.376773+0000","last_undegraded":"2024-09-21T10:29:52.376773+0000","last_fullsized":"2024-09-21T10:29:52.376773+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:30.569982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:30.569982+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:01.422939+0000","scrub_duration":0.0051309349999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":178,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.371699+0000","last_change":"2024-09-21T10:28:00.294966+0000","last_active":"2024-09-21T10:29:52.371699+0000","last_peered":"2024-09-21T10:29:52.371699+0000","last_clean":"2024-09-21T10:29:52.371699+0000","last_became_active":"2024-09-21T10:28:00.294552+0000","last_became_peered":"2024-09-21T10:28:00.294552+0000","last_unstale":"2024-09-21T10:29:52.371699+0000","last_undegraded":"2024-09-21T10:29:52.371699+0000","last_fullsized":"2024-09-21T10:29:52.371699+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:47.292125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":1222,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.439617+0000","last_change":"2024-09-21T10:08:18.468995+0000","last_active":"2024-09-21T10:30:05.439617+0000","last_peered":"2024-09-21T10:30:05.439617+0000","last_clean":"2024-09-21T10:30:05.439617+0000","last_became_active":"2024-09-21T10:08:18.468243+0000","last_became_peered":"2024-09-21T10:08:18.468243+0000","last_unstale":"2024-09-21T10:30:05.439617+0000","last_undegraded":"2024-09-21T10:30:05.439617+0000","last_fullsized":"2024-09-21T10:30:05.439617+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:45.520968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":548,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.442577+0000","last_change":"2024-09-21T10:24:44.309614+0000","last_active":"2024-09-21T10:30:05.442577+0000","last_peered":"2024-09-21T10:30:05.442577+0000","last_clean":"2024-09-21T10:30:05.442577+0000","last_became_active":"2024-09-21T10:24:44.309187+0000","last_became_peered":"2024-09-21T10:24:44.309187+0000","last_unstale":"2024-09-21T10:30:05.442577+0000","last_undegraded":"2024-09-21T10:30:05.442577+0000","last_fullsized":"2024-09-21T10:30:05.442577+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:01.074803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"380'12593","reported_seq":12634,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:08.092340+0000","last_change":"2024-09-21T10:26:36.196357+0000","last_active":"2024-09-21T10:30:08.092340+0000","last_peered":"2024-09-21T10:30:08.092340+0000","last_clean":"2024-09-21T10:30:08.092340+0000","last_became_active":"2024-09-21T10:26:36.196060+0000","last_became_peered":"2024-09-21T10:26:36.196060+0000","last_unstale":"2024-09-21T10:30:08.092340+0000","last_undegraded":"2024-09-21T10:30:08.092340+0000","last_fullsized":"2024-09-21T10:30:08.092340+0000","mapping_epoch":348,"log_start":"356'4900","ondisk_log_start":"356'4900","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7693,"log_dups_size":0,"ondisk_log_size":7693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:46:34.643144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201588760,"num_objects":3077,"num_object_clones":0,"num_object_copies":9231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3077,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12593,"num_write_kb":501377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"82'100","reported_seq":254,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.374896+0000","last_change":"2024-09-21T10:28:00.295277+0000","last_active":"2024-09-21T10:29:52.374896+0000","last_peered":"2024-09-21T10:29:52.374896+0000","last_clean":"2024-09-21T10:29:52.374896+0000","last_became_active":"2024-09-21T10:28:00.294016+0000","last_became_peered":"2024-09-21T10:28:00.294016+0000","last_unstale":"2024-09-21T10:29:52.374896+0000","last_undegraded":"2024-09-21T10:29:52.374896+0000","last_fullsized":"2024-09-21T10:29:52.374896+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:24.536711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:24.536711+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:28:49.984470+0000","scrub_duration":0.0047525650000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":517,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.439507+0000","last_change":"2024-09-21T10:08:18.469907+0000","last_active":"2024-09-21T10:30:05.439507+0000","last_peered":"2024-09-21T10:30:05.439507+0000","last_clean":"2024-09-21T10:30:05.439507+0000","last_became_active":"2024-09-21T10:08:18.469443+0000","last_became_peered":"2024-09-21T10:08:18.469443+0000","last_unstale":"2024-09-21T10:30:05.439507+0000","last_undegraded":"2024-09-21T10:30:05.439507+0000","last_fullsized":"2024-09-21T10:30:05.439507+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:15.645803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":802,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.367000+0000","last_change":"2024-09-21T10:24:44.334847+0000","last_active":"2024-09-21T10:29:52.367000+0000","last_peered":"2024-09-21T10:29:52.367000+0000","last_clean":"2024-09-21T10:29:52.367000+0000","last_became_active":"2024-09-21T10:24:44.301321+0000","last_became_peered":"2024-09-21T10:24:44.301321+0000","last_unstale":"2024-09-21T10:29:52.367000+0000","last_undegraded":"2024-09-21T10:29:52.367000+0000","last_fullsized":"2024-09-21T10:29:52.367000+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:37.628940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":54,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.371826+0000","last_change":"2024-09-21T10:26:41.278937+0000","last_active":"2024-09-21T10:29:52.371826+0000","last_peered":"2024-09-21T10:29:52.371826+0000","last_clean":"2024-09-21T10:29:52.371826+0000","last_became_active":"2024-09-21T10:26:41.277726+0000","last_became_peered":"2024-09-21T10:26:41.277726+0000","last_unstale":"2024-09-21T10:29:52.371826+0000","last_undegraded":"2024-09-21T10:29:52.371826+0000","last_fullsized":"2024-09-21T10:29:52.371826+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:04.741057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"380'12581","reported_seq":12621,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:08.087701+0000","last_change":"2024-09-21T10:26:36.196641+0000","last_active":"2024-09-21T10:30:08.087701+0000","last_peered":"2024-09-21T10:30:08.087701+0000","last_clean":"2024-09-21T10:30:08.087701+0000","last_became_active":"2024-09-21T10:26:36.196351+0000","last_became_peered":"2024-09-21T10:26:36.196351+0000","last_unstale":"2024-09-21T10:30:08.087701+0000","last_undegraded":"2024-09-21T10:30:08.087701+0000","last_fullsized":"2024-09-21T10:30:08.087701+0000","mapping_epoch":348,"log_start":"358'5900","ondisk_log_start":"358'5900","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":6681,"log_dups_size":0,"ondisk_log_size":6681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:41:54.280247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208207872,"num_objects":3177,"num_object_clones":0,"num_object_copies":9531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12581,"num_write_kb":504256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'100","reported_seq":69,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.370186+0000","last_change":"2024-09-21T10:24:44.296758+0000","last_active":"2024-09-21T10:29:52.370186+0000","last_peered":"2024-09-21T10:29:52.370186+0000","last_clean":"2024-09-21T10:29:52.370186+0000","last_became_active":"2024-09-21T10:24:44.296403+0000","last_became_peered":"2024-09-21T10:24:44.296403+0000","last_unstale":"2024-09-21T10:29:52.370186+0000","last_undegraded":"2024-09-21T10:29:52.370186+0000","last_fullsized":"2024-09-21T10:29:52.370186+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.105746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.105746+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:20:12.502081+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":54,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.439845+0000","last_change":"2024-09-21T10:28:00.300952+0000","last_active":"2024-09-21T10:30:05.439845+0000","last_peered":"2024-09-21T10:30:05.439845+0000","last_clean":"2024-09-21T10:30:05.439845+0000","last_became_active":"2024-09-21T10:28:00.293337+0000","last_became_peered":"2024-09-21T10:28:00.293337+0000","last_unstale":"2024-09-21T10:30:05.439845+0000","last_undegraded":"2024-09-21T10:30:05.439845+0000","last_fullsized":"2024-09-21T10:30:05.439845+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:58.409761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":942,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.440057+0000","last_change":"2024-09-21T10:24:46.844278+0000","last_active":"2024-09-21T10:30:05.440057+0000","last_peered":"2024-09-21T10:30:05.440057+0000","last_clean":"2024-09-21T10:30:05.440057+0000","last_became_active":"2024-09-21T10:24:46.840961+0000","last_became_peered":"2024-09-21T10:24:46.840961+0000","last_unstale":"2024-09-21T10:30:05.440057+0000","last_undegraded":"2024-09-21T10:30:05.440057+0000","last_fullsized":"2024-09-21T10:30:05.440057+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:49:48.823524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":51,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.434229+0000","last_change":"2024-09-21T10:28:00.293864+0000","last_active":"2024-09-21T10:30:05.434229+0000","last_peered":"2024-09-21T10:30:05.434229+0000","last_clean":"2024-09-21T10:30:05.434229+0000","last_became_active":"2024-09-21T10:28:00.293494+0000","last_became_peered":"2024-09-21T10:28:00.293494+0000","last_unstale":"2024-09-21T10:30:05.434229+0000","last_undegraded":"2024-09-21T10:30:05.434229+0000","last_fullsized":"2024-09-21T10:30:05.434229+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:37.098068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"380'12382","reported_seq":12526,"reported_epoch":380,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:30:08.107322+0000","last_change":"2024-09-21T10:28:01.631803+0000","last_active":"2024-09-21T10:30:08.107322+0000","last_peered":"2024-09-21T10:30:08.107322+0000","last_clean":"2024-09-21T10:27:59.267411+0000","last_became_active":"2024-09-21T10:28:01.425385+0000","last_became_peered":"2024-09-21T10:28:01.425385+0000","last_unstale":"2024-09-21T10:30:08.107322+0000","last_undegraded":"2024-09-21T10:30:08.107322+0000","last_fullsized":"2024-09-21T10:30:08.107322+0000","mapping_epoch":361,"log_start":"355'4600","ondisk_log_start":"355'4600","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7782,"log_dups_size":0,"ondisk_log_size":7782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:51:26.681334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201588736,"num_objects":3076,"num_object_clones":0,"num_object_copies":9228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3060,"num_objects_unfound":0,"num_objects_dirty":3076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12382,"num_write_kb":494656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":2949120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,2,4],"avail_no_missing":["6","2","4"],"object_location_counts":[{"shards":"2,4,6","objects":3076}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"82'100","reported_seq":58,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.453366+0000","last_change":"2024-09-21T10:28:00.305255+0000","last_active":"2024-09-21T10:30:05.453366+0000","last_peered":"2024-09-21T10:30:05.453366+0000","last_clean":"2024-09-21T10:30:05.453366+0000","last_became_active":"2024-09-21T10:28:00.290981+0000","last_became_peered":"2024-09-21T10:28:00.290981+0000","last_unstale":"2024-09-21T10:30:05.453366+0000","last_undegraded":"2024-09-21T10:30:05.453366+0000","last_fullsized":"2024-09-21T10:30:05.453366+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.759355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.759355+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":69,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":681,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.441022+0000","last_change":"2024-09-21T10:24:44.307344+0000","last_active":"2024-09-21T10:30:05.441022+0000","last_peered":"2024-09-21T10:30:05.441022+0000","last_clean":"2024-09-21T10:30:05.441022+0000","last_became_active":"2024-09-21T10:24:44.298936+0000","last_became_peered":"2024-09-21T10:24:44.298936+0000","last_unstale":"2024-09-21T10:30:05.441022+0000","last_undegraded":"2024-09-21T10:30:05.441022+0000","last_fullsized":"2024-09-21T10:30:05.441022+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:58:26.025877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":606,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.376991+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:29:52.376991+0000","last_peered":"2024-09-21T10:29:52.376991+0000","last_clean":"2024-09-21T10:29:52.376991+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:29:52.376991+0000","last_undegraded":"2024-09-21T10:29:52.376991+0000","last_fullsized":"2024-09-21T10:29:52.376991+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":69,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.369709+0000","last_change":"2024-09-21T10:24:44.303781+0000","last_active":"2024-09-21T10:29:52.369709+0000","last_peered":"2024-09-21T10:29:52.369709+0000","last_clean":"2024-09-21T10:29:52.369709+0000","last_became_active":"2024-09-21T10:24:44.302058+0000","last_became_peered":"2024-09-21T10:24:44.302058+0000","last_unstale":"2024-09-21T10:29:52.369709+0000","last_undegraded":"2024-09-21T10:29:52.369709+0000","last_fullsized":"2024-09-21T10:29:52.369709+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:30.371229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":69,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.370204+0000","last_change":"2024-09-21T10:24:44.303417+0000","last_active":"2024-09-21T10:29:52.370204+0000","last_peered":"2024-09-21T10:29:52.370204+0000","last_clean":"2024-09-21T10:29:52.370204+0000","last_became_active":"2024-09-21T10:24:44.303069+0000","last_became_peered":"2024-09-21T10:24:44.303069+0000","last_unstale":"2024-09-21T10:29:52.370204+0000","last_undegraded":"2024-09-21T10:29:52.370204+0000","last_fullsized":"2024-09-21T10:29:52.370204+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:30:30.966228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":79,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.370624+0000","last_change":"2024-09-21T10:24:44.321421+0000","last_active":"2024-09-21T10:29:52.370624+0000","last_peered":"2024-09-21T10:29:52.370624+0000","last_clean":"2024-09-21T10:29:52.370624+0000","last_became_active":"2024-09-21T10:24:44.297082+0000","last_became_peered":"2024-09-21T10:24:44.297082+0000","last_unstale":"2024-09-21T10:29:52.370624+0000","last_undegraded":"2024-09-21T10:29:52.370624+0000","last_fullsized":"2024-09-21T10:29:52.370624+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:35:37.704881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":516,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.441962+0000","last_change":"2024-09-21T10:08:18.470378+0000","last_active":"2024-09-21T10:30:05.441962+0000","last_peered":"2024-09-21T10:30:05.441962+0000","last_clean":"2024-09-21T10:30:05.441962+0000","last_became_active":"2024-09-21T10:08:18.469482+0000","last_became_peered":"2024-09-21T10:08:18.469482+0000","last_unstale":"2024-09-21T10:30:05.441962+0000","last_undegraded":"2024-09-21T10:30:05.441962+0000","last_fullsized":"2024-09-21T10:30:05.441962+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:35.628724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":443,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.366775+0000","last_change":"2024-09-21T10:28:00.296771+0000","last_active":"2024-09-21T10:29:52.366775+0000","last_peered":"2024-09-21T10:29:52.366775+0000","last_clean":"2024-09-21T10:29:52.366775+0000","last_became_active":"2024-09-21T10:28:00.285336+0000","last_became_peered":"2024-09-21T10:28:00.285336+0000","last_unstale":"2024-09-21T10:29:52.366775+0000","last_undegraded":"2024-09-21T10:29:52.366775+0000","last_fullsized":"2024-09-21T10:29:52.366775+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:06:50.848492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":151,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.371968+0000","last_change":"2024-09-21T10:26:41.644952+0000","last_active":"2024-09-21T10:29:52.371968+0000","last_peered":"2024-09-21T10:29:52.371968+0000","last_clean":"2024-09-21T10:29:52.371968+0000","last_became_active":"2024-09-21T10:26:41.644592+0000","last_became_peered":"2024-09-21T10:26:41.644592+0000","last_unstale":"2024-09-21T10:29:52.371968+0000","last_undegraded":"2024-09-21T10:29:52.371968+0000","last_fullsized":"2024-09-21T10:29:52.371968+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:44:44.183877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":50,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.435478+0000","last_change":"2024-09-21T10:28:00.294338+0000","last_active":"2024-09-21T10:30:05.435478+0000","last_peered":"2024-09-21T10:30:05.435478+0000","last_clean":"2024-09-21T10:30:05.435478+0000","last_became_active":"2024-09-21T10:28:00.294044+0000","last_became_peered":"2024-09-21T10:28:00.294044+0000","last_unstale":"2024-09-21T10:30:05.435478+0000","last_undegraded":"2024-09-21T10:30:05.435478+0000","last_fullsized":"2024-09-21T10:30:05.435478+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:43:30.999988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":646,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.441142+0000","last_change":"2024-09-21T10:28:00.296935+0000","last_active":"2024-09-21T10:30:05.441142+0000","last_peered":"2024-09-21T10:30:05.441142+0000","last_clean":"2024-09-21T10:30:05.441142+0000","last_became_active":"2024-09-21T10:28:00.296562+0000","last_became_peered":"2024-09-21T10:28:00.296562+0000","last_unstale":"2024-09-21T10:30:05.441142+0000","last_undegraded":"2024-09-21T10:30:05.441142+0000","last_fullsized":"2024-09-21T10:30:05.441142+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:06.863310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":50,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.434392+0000","last_change":"2024-09-21T10:28:00.294718+0000","last_active":"2024-09-21T10:30:05.434392+0000","last_peered":"2024-09-21T10:30:05.434392+0000","last_clean":"2024-09-21T10:30:05.434392+0000","last_became_active":"2024-09-21T10:28:00.294299+0000","last_became_peered":"2024-09-21T10:28:00.294299+0000","last_unstale":"2024-09-21T10:30:05.434392+0000","last_undegraded":"2024-09-21T10:30:05.434392+0000","last_fullsized":"2024-09-21T10:30:05.434392+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:41:46.881336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":343,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.376847+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:29:52.376847+0000","last_peered":"2024-09-21T10:29:52.376847+0000","last_clean":"2024-09-21T10:29:52.376847+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:29:52.376847+0000","last_undegraded":"2024-09-21T10:29:52.376847+0000","last_fullsized":"2024-09-21T10:29:52.376847+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":50,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.434313+0000","last_change":"2024-09-21T10:28:00.293872+0000","last_active":"2024-09-21T10:30:05.434313+0000","last_peered":"2024-09-21T10:30:05.434313+0000","last_clean":"2024-09-21T10:30:05.434313+0000","last_became_active":"2024-09-21T10:28:00.293320+0000","last_became_peered":"2024-09-21T10:28:00.293320+0000","last_unstale":"2024-09-21T10:30:05.434313+0000","last_undegraded":"2024-09-21T10:30:05.434313+0000","last_fullsized":"2024-09-21T10:30:05.434313+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:45:43.909307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":663,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.441919+0000","last_change":"2024-09-21T10:28:00.297304+0000","last_active":"2024-09-21T10:30:05.441919+0000","last_peered":"2024-09-21T10:30:05.441919+0000","last_clean":"2024-09-21T10:30:05.441919+0000","last_became_active":"2024-09-21T10:28:00.296631+0000","last_became_peered":"2024-09-21T10:28:00.296631+0000","last_unstale":"2024-09-21T10:30:05.441919+0000","last_undegraded":"2024-09-21T10:30:05.441919+0000","last_fullsized":"2024-09-21T10:30:05.441919+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:22:16.633581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":662,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.439345+0000","last_change":"2024-09-21T10:28:00.284924+0000","last_active":"2024-09-21T10:30:05.439345+0000","last_peered":"2024-09-21T10:30:05.439345+0000","last_clean":"2024-09-21T10:30:05.439345+0000","last_became_active":"2024-09-21T10:28:00.284633+0000","last_became_peered":"2024-09-21T10:28:00.284633+0000","last_unstale":"2024-09-21T10:30:05.439345+0000","last_undegraded":"2024-09-21T10:30:05.439345+0000","last_fullsized":"2024-09-21T10:30:05.439345+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:23:09.808151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":50,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.435241+0000","last_change":"2024-09-21T10:28:00.283403+0000","last_active":"2024-09-21T10:30:05.435241+0000","last_peered":"2024-09-21T10:30:05.435241+0000","last_clean":"2024-09-21T10:30:05.435241+0000","last_became_active":"2024-09-21T10:28:00.283089+0000","last_became_peered":"2024-09-21T10:28:00.283089+0000","last_unstale":"2024-09-21T10:30:05.435241+0000","last_undegraded":"2024-09-21T10:30:05.435241+0000","last_fullsized":"2024-09-21T10:30:05.435241+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:42.213644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":69,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.369979+0000","last_change":"2024-09-21T10:24:44.305124+0000","last_active":"2024-09-21T10:29:52.369979+0000","last_peered":"2024-09-21T10:29:52.369979+0000","last_clean":"2024-09-21T10:29:52.369979+0000","last_became_active":"2024-09-21T10:24:44.304468+0000","last_became_peered":"2024-09-21T10:24:44.304468+0000","last_unstale":"2024-09-21T10:29:52.369979+0000","last_undegraded":"2024-09-21T10:29:52.369979+0000","last_fullsized":"2024-09-21T10:29:52.369979+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:37.904764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":299,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.374210+0000","last_change":"2024-09-21T10:28:00.286646+0000","last_active":"2024-09-21T10:29:52.374210+0000","last_peered":"2024-09-21T10:29:52.374210+0000","last_clean":"2024-09-21T10:29:52.374210+0000","last_became_active":"2024-09-21T10:28:00.286300+0000","last_became_peered":"2024-09-21T10:28:00.286300+0000","last_unstale":"2024-09-21T10:29:52.374210+0000","last_undegraded":"2024-09-21T10:29:52.374210+0000","last_fullsized":"2024-09-21T10:29:52.374210+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:58.215385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":653,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.441600+0000","last_change":"2024-09-21T10:24:44.299989+0000","last_active":"2024-09-21T10:30:05.441600+0000","last_peered":"2024-09-21T10:30:05.441600+0000","last_clean":"2024-09-21T10:30:05.441600+0000","last_became_active":"2024-09-21T10:24:44.299356+0000","last_became_peered":"2024-09-21T10:24:44.299356+0000","last_unstale":"2024-09-21T10:30:05.441600+0000","last_undegraded":"2024-09-21T10:30:05.441600+0000","last_fullsized":"2024-09-21T10:30:05.441600+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:58.867008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":79,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.370041+0000","last_change":"2024-09-21T10:28:00.281942+0000","last_active":"2024-09-21T10:29:52.370041+0000","last_peered":"2024-09-21T10:29:52.370041+0000","last_clean":"2024-09-21T10:29:52.370041+0000","last_became_active":"2024-09-21T10:28:00.281600+0000","last_became_peered":"2024-09-21T10:28:00.281600+0000","last_unstale":"2024-09-21T10:29:52.370041+0000","last_undegraded":"2024-09-21T10:29:52.370041+0000","last_fullsized":"2024-09-21T10:29:52.370041+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:21.144244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":73,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.370752+0000","last_change":"2024-09-21T10:24:44.329150+0000","last_active":"2024-09-21T10:29:52.370752+0000","last_peered":"2024-09-21T10:29:52.370752+0000","last_clean":"2024-09-21T10:29:52.370752+0000","last_became_active":"2024-09-21T10:24:44.306752+0000","last_became_peered":"2024-09-21T10:24:44.306752+0000","last_unstale":"2024-09-21T10:29:52.370752+0000","last_undegraded":"2024-09-21T10:29:52.370752+0000","last_fullsized":"2024-09-21T10:29:52.370752+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:16.592859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":79,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.370186+0000","last_change":"2024-09-21T10:28:00.286009+0000","last_active":"2024-09-21T10:29:52.370186+0000","last_peered":"2024-09-21T10:29:52.370186+0000","last_clean":"2024-09-21T10:29:52.370186+0000","last_became_active":"2024-09-21T10:28:00.285327+0000","last_became_peered":"2024-09-21T10:28:00.285327+0000","last_unstale":"2024-09-21T10:29:52.370186+0000","last_undegraded":"2024-09-21T10:29:52.370186+0000","last_fullsized":"2024-09-21T10:29:52.370186+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:34.985354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":73,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.370104+0000","last_change":"2024-09-21T10:24:44.324087+0000","last_active":"2024-09-21T10:29:52.370104+0000","last_peered":"2024-09-21T10:29:52.370104+0000","last_clean":"2024-09-21T10:29:52.370104+0000","last_became_active":"2024-09-21T10:24:44.298665+0000","last_became_peered":"2024-09-21T10:24:44.298665+0000","last_unstale":"2024-09-21T10:29:52.370104+0000","last_undegraded":"2024-09-21T10:29:52.370104+0000","last_fullsized":"2024-09-21T10:29:52.370104+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:59.819340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":50,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.435408+0000","last_change":"2024-09-21T10:28:00.293054+0000","last_active":"2024-09-21T10:30:05.435408+0000","last_peered":"2024-09-21T10:30:05.435408+0000","last_clean":"2024-09-21T10:30:05.435408+0000","last_became_active":"2024-09-21T10:28:00.292723+0000","last_became_peered":"2024-09-21T10:28:00.292723+0000","last_unstale":"2024-09-21T10:30:05.435408+0000","last_undegraded":"2024-09-21T10:30:05.435408+0000","last_fullsized":"2024-09-21T10:30:05.435408+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:01:45.933570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":51,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.435160+0000","last_change":"2024-09-21T10:28:00.285515+0000","last_active":"2024-09-21T10:30:05.435160+0000","last_peered":"2024-09-21T10:30:05.435160+0000","last_clean":"2024-09-21T10:30:05.435160+0000","last_became_active":"2024-09-21T10:28:00.285203+0000","last_became_peered":"2024-09-21T10:28:00.285203+0000","last_unstale":"2024-09-21T10:30:05.435160+0000","last_undegraded":"2024-09-21T10:30:05.435160+0000","last_fullsized":"2024-09-21T10:30:05.435160+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:41.982228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":69,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T10:29:52.372412+0000","last_change":"2024-09-21T10:24:44.298820+0000","last_active":"2024-09-21T10:29:52.372412+0000","last_peered":"2024-09-21T10:29:52.372412+0000","last_clean":"2024-09-21T10:29:52.372412+0000","last_became_active":"2024-09-21T10:24:44.298478+0000","last_became_peered":"2024-09-21T10:24:44.298478+0000","last_unstale":"2024-09-21T10:29:52.372412+0000","last_undegraded":"2024-09-21T10:29:52.372412+0000","last_fullsized":"2024-09-21T10:29:52.372412+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:59.567078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":638,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.441022+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:30:05.441022+0000","last_peered":"2024-09-21T10:30:05.441022+0000","last_clean":"2024-09-21T10:30:05.441022+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:30:05.441022+0000","last_undegraded":"2024-09-21T10:30:05.441022+0000","last_fullsized":"2024-09-21T10:30:05.441022+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":3238854680,"num_objects":49422,"num_object_clones":0,"num_object_copies":148266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16952,"num_objects_misplaced":8463,"num_objects_unfound":0,"num_objects_dirty":49422,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200624,"num_write_kb":8001409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1963,"num_bytes_recovered":128647168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10645811200,"data_stored":10645798984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126924,"ondisk_log_size":126924,"up":48,"acting":45,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":10252,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5983,"num_read_kb":13006,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":316358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":69,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":644038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1100,"ondisk_log_size":1100,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397984,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2149988,"kb_used_data":1252760,"kb_used_omap":1,"kb_used_meta":897214,"kb_avail":91595164,"statfs":{"total":95995035648,"available":93793447936,"internally_reserved":0,"allocated":1282826240,"data_stored":1280646183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918747480},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594212,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2288348,"kb_used_data":1391188,"kb_used_omap":1,"kb_used_meta":897150,"kb_avail":91456804,"statfs":{"total":95995035648,"available":93651767296,"internally_reserved":0,"allocated":1424576512,"data_stored":1422400448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918681944},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790442,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1917264,"kb_used_data":1013128,"kb_used_omap":1,"kb_used_meta":904062,"kb_avail":91827888,"statfs":{"total":95995035648,"available":94031757312,"internally_reserved":0,"allocated":1037443072,"data_stored":1035275327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":925759832},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986671,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2563808,"kb_used_data":2258072,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91181344,"statfs":{"total":95995035648,"available":93369696256,"internally_reserved":0,"allocated":2312265728,"data_stored":2310085617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":101,"seq":433791697233,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":812428,"kb_used_data":141892,"kb_used_omap":5,"kb_used_meta":670522,"kb_avail":92932724,"statfs":{"total":95995035648,"available":95163109376,"internally_reserved":0,"allocated":145297408,"data_stored":143133517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":686615312},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860759,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2417344,"kb_used_data":1457848,"kb_used_omap":8,"kb_used_meta":959479,"kb_avail":91327808,"statfs":{"total":95995035648,"available":93519675392,"internally_reserved":0,"allocated":1492836352,"data_stored":1490676668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":982507372},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024285,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1862528,"kb_used_data":1049784,"kb_used_omap":7,"kb_used_meta":812664,"kb_avail":91882624,"statfs":{"total":95995035648,"available":94087806976,"internally_reserved":0,"allocated":1074978816,"data_stored":1072825976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":832168498},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187821,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2972048,"kb_used_data":1948552,"kb_used_omap":8,"kb_used_meta":1023415,"kb_avail":90773104,"statfs":{"total":95995035648,"available":92951658496,"internally_reserved":0,"allocated":1995317248,"data_stored":1993149251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":1047977969},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1977094144,"data_stored":1977090072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1068761088,"data_stored":1068761088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1477967872,"data_stored":1477967872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":139329536,"data_stored":139329536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2286882816,"data_stored":2286878744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1014038528,"data_stored":1014038528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1412763648,"data_stored":1412759576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1268973568,"data_stored":1268973568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:30:12.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T10:30:12.506 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:12.543 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:12.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:12.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:12.645 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:12.650 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:12 smithi098 ceph-mon[97580]: from='client.36588 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:30:12.650 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:12 smithi098 ceph-mon[100914]: from='client.36588 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:30:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:12.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:12.795 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:12.823 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:12.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:12.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:12 smithi196 ceph-mon[84784]: from='client.36588 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:30:12.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:12.957 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:12.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:12.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:13.059 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:13.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:13.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:13.169 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:13.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:13.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:13.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:13.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:13.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:13.395 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:13.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:13.495 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:13.527 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:13.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:13.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:13.629 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:13.659 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:13.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:13.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:13.761 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:13.788 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:13.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:13.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:13.847 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:30:13.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:13 smithi098 ceph-mon[97580]: pgmap v1308: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 664 op/s; 16952/147177 objects degraded (11.518%); 8463/147177 objects misplaced (5.750%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:30:13.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:13 smithi098 ceph-mon[100914]: pgmap v1308: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 664 op/s; 16952/147177 objects degraded (11.518%); 8463/147177 objects misplaced (5.750%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:30:13.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:13 smithi196 ceph-mon[84784]: pgmap v1308: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 664 op/s; 16952/147177 objects degraded (11.518%); 8463/147177 objects misplaced (5.750%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:30:13.889 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:13.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:13.932 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:13.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:14.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:14.064 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:14.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:14.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:14.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:14.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:14.200 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:14.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:14.301 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:14.324 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:14.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:14.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:14.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:14.466 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:14.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:14.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:14.568 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:14.612 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:14.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:14.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:14.714 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:14.739 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:14.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:14.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:14.840 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:14.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:14 smithi098 ceph-mon[97580]: pgmap v1309: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 672 op/s; 16952/140229 objects degraded (12.089%); 8081/140229 objects misplaced (5.763%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:30:14.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:14 smithi098 ceph-mon[100914]: pgmap v1309: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 672 op/s; 16952/140229 objects degraded (12.089%); 8081/140229 objects misplaced (5.763%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:30:14.877 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:14.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:14.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:14 smithi196 ceph-mon[84784]: pgmap v1309: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 672 op/s; 16952/140229 objects degraded (12.089%); 8081/140229 objects misplaced (5.763%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:30:14.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:15.016 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:15.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:15.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:15.118 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:15.151 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:15.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:15.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:15.253 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:15.282 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:15.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:15.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:15.384 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:15.419 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:15.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:15.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:15.521 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:15.547 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:15.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:15.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:15.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:15.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:15.700 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:15.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:15.801 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:15.869 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:15.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:15.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:15.908 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:15.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:15.999 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:16.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:16.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:16.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:16.127 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:16.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:16.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:16.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:16.265 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:16.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:16.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:16.366 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:16.393 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:16.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:16.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:16.494 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:16.536 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:16.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:16.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:16.637 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:16.689 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:16.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:16.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:16.790 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:16.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:16.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:16 smithi098 ceph-mon[97580]: pgmap v1310: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 665 op/s; 17054/137064 objects degraded (12.442%); 7882/137064 objects misplaced (5.751%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:30:16.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:16 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 17054/137064 objects degraded (12.442%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:16.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:16 smithi098 ceph-mon[100914]: pgmap v1310: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 665 op/s; 17054/137064 objects degraded (12.442%); 7882/137064 objects misplaced (5.751%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:30:16.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:16 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 17054/137064 objects degraded (12.442%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:16.926 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:16.971 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:16.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:16.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:17.073 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:17.111 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:17.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:17.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:17.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:16 smithi196 ceph-mon[84784]: pgmap v1310: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 665 op/s; 17054/137064 objects degraded (12.442%); 7882/137064 objects misplaced (5.751%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:30:17.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:16 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 17054/137064 objects degraded (12.442%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:17.213 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:17.246 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:17.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:17.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:17.347 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:17.374 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:17.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:17.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:17.476 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:17.508 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:17.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:17.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:17.610 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:17.638 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:17.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:17.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:17.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:17.772 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:17.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:17.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:17.874 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:17.904 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:17.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:17.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:18.005 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:18.032 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:18.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:18.133 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:18.168 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:18.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:18.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:18.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:18.307 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:18.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:18.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:18.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:18.453 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:18.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:18.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:18.554 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:18.580 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:18.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:18.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:18.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:18.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:18.736 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:18.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:18.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:18 smithi098 ceph-mon[97580]: pgmap v1311: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 585 op/s; 17054/136338 objects degraded (12.509%); 7882/136338 objects misplaced (5.781%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:30:18.746 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:18 smithi098 ceph-mon[100914]: pgmap v1311: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 585 op/s; 17054/136338 objects degraded (12.509%); 7882/136338 objects misplaced (5.781%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:30:18.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:18.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:18.885 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:18.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:18.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:19.030 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:19.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:19.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:19.132 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:19.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:18 smithi196 ceph-mon[84784]: pgmap v1311: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 585 op/s; 17054/136338 objects degraded (12.509%); 7882/136338 objects misplaced (5.781%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:30:19.166 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:19.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:19.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:19.267 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:19.307 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:19.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:19.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:19.409 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:19.439 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:19.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:19.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:19.540 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:19.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:19.579 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:19.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:19.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:19.709 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:30:19.715 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:19.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:19.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:19.817 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:19.818 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:30:19.818 INFO:teuthology.orchestra.run.smithi098.stdout:{"fsid":"26a15512-77fe-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 17054/129741 objects degraded (13.145%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1917,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":381,"num_osds":8,"num_up_osds":8,"osd_up_since":1726912933,"num_in_osds":8,"osd_in_since":1726914479,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":94},{"state_name":"active+remapped+backfill_wait","count":2},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":99,"num_pools":5,"num_objects":43247,"data_bytes":2836916619,"bytes_used":16280936448,"bytes_avail":751679348736,"bytes_total":767960285184,"degraded_objects":17054,"degraded_total":129741,"degraded_ratio":0.13144649725221788,"misplaced_objects":7462,"misplaced_total":129741,"misplaced_ratio":0.057514586753609119,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":1080944,"recovering_keys_per_sec":0,"num_objects_recovered":198,"num_bytes_recovered":12976128,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":749},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi098.stcwsu","status":"up:active","gid":14640}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.98:8443/","prometheus":"http://172.21.15.98:9283/"}},"servicemap":{"epoch":30,"modified":"2024-09-21T10:29:21.594654+0000","services":{}},"progress_events":{"0d290b6d-ee2f-4f68-8c75-373d103609de":{"message":"Global Recovery Event (0s)\n [............................] ","progress":0,"add_to_ceph_s":true},"6de3e7dc-7aed-4a0c-b700-51217030d18b":{"message":"Upgrade to 19.3.0-5092-g62da54b4 (4m)\n [============................] (remaining: 5m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-09-21T10:30:19.852 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:19.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:19.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:19.957 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:19.980 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:19.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:19.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:20.067 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:19 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/141987925' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:30:20.067 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:19 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:30:20.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:20.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:19 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/141987925' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:30:20.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:19 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:30:20.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:19 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/141987925' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:30:20.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:19 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:30:20.112 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:20.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:20.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:20.213 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:20.240 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:20.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:20.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:20.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:20.383 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:20.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:20.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:20.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:30:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:30:20.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:30:20.486 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:20.525 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:20.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:20.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:20.627 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:20.651 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:20.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:20.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:20.753 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:20.779 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:20.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:20.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:20.880 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:20.905 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:20.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:20.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:21.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:21.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:21.044 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:21.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:21.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:20 smithi098 ceph-mon[97580]: pgmap v1312: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 749 op/s; 17054/129741 objects degraded (13.145%); 7462/129741 objects misplaced (5.751%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:30:21.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:20 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:30:21.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:20 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:30:21.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:20 smithi098 ceph-mon[97580]: osdmap e381: 8 total, 8 up, 8 in 2024-09-21T10:30:21.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:20 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4190338179' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:30:21.051 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:30:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:30:20] "GET /metrics HTTP/1.1" 200 36615 "" "Prometheus/2.51.0" 2024-09-21T10:30:21.051 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:20 smithi098 ceph-mon[100914]: pgmap v1312: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 749 op/s; 17054/129741 objects degraded (13.145%); 7462/129741 objects misplaced (5.751%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:30:21.051 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:20 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:30:21.052 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:20 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:30:21.052 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:20 smithi098 ceph-mon[100914]: osdmap e381: 8 total, 8 up, 8 in 2024-09-21T10:30:21.052 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:20 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4190338179' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:30:21.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:20 smithi196 ceph-mon[84784]: pgmap v1312: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 749 op/s; 17054/129741 objects degraded (13.145%); 7462/129741 objects misplaced (5.751%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:30:21.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:20 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:30:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:20 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:30:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:20 smithi196 ceph-mon[84784]: osdmap e381: 8 total, 8 up, 8 in 2024-09-21T10:30:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:20 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4190338179' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:30:21.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:21.178 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:21.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:21.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:21.209 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-21T10:30:21.209 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:30:21.280 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:21.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:21.328 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:21.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:21.429 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:21.454 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:21.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:21.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:21.555 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:21.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:21.612 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:21.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:21.713 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:21.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:21.761 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:21.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:21.862 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:21.908 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:21.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:21.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:22.011 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:22.041 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:22.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:22.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:22.081 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:21 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 17054/129741 objects degraded (13.145%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:22.082 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:21 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 17054/129741 objects degraded (13.145%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:22.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:21 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 17054/129741 objects degraded (13.145%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:22.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:22.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:22.199 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:22.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:22.299 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:22.325 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:22.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:22.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:22.426 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:22.454 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:22.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:22.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:22.556 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:22.582 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:22.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:22.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:22.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:22.720 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:22.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:22.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:22.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:22.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:22.858 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:22.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:22.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:22.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:22 smithi098 ceph-mon[97580]: pgmap v1314: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 696 op/s; 17209/126267 objects degraded (13.629%); 7224/126267 objects misplaced (5.721%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:30:22.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:22.975 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:22 smithi098 ceph-mon[100914]: pgmap v1314: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 696 op/s; 17209/126267 objects degraded (13.629%); 7224/126267 objects misplaced (5.721%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:30:22.975 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:23.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:23.016 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:23.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:23.116 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:23.126 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:23.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:22 smithi196 ceph-mon[84784]: pgmap v1314: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 696 op/s; 17209/126267 objects degraded (13.629%); 7224/126267 objects misplaced (5.721%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:30:23.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:23.140 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:30:23.140 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:30:23.159 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:23.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:23.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:23.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:23.307 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:23.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:23.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:23.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:23.445 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:23.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:23.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:23.547 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:23.572 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:23.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:23.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:23.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:23.712 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:23.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:23.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:23.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:23.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:23.850 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:23.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:23.855 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:30:23.906 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:30:23.906 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:30:23.949 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:30:23.949 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:30:23.951 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:23.996 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:30:24.187 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:30:24.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:24.188 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:24.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:24.188 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:30:24.188 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:30:24.189 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:30:24.189 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:30:24.189 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:30:24.189 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:30:24.190 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:30:24.190 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:30:24.190 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:30:24.191 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:30:24.289 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:24.315 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:24.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:24.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:24.417 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:24.449 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:24.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:24.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:24.551 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:24.579 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:24.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:24.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:24.680 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:24.707 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:24.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:24.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:24.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:24.841 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:24.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:24.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:24.943 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:24.966 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:24.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:24.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:24.987 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:24 smithi098 ceph-mon[100914]: pgmap v1315: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 17209/119151 objects degraded (14.443%); 6799/119151 objects misplaced (5.706%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:30:24.987 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:24 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/860445335' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:30:24.987 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:24 smithi098 ceph-mon[97580]: pgmap v1315: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 17209/119151 objects degraded (14.443%); 6799/119151 objects misplaced (5.706%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:30:24.987 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:24 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/860445335' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:30:25.067 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:25.094 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:25.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:25.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:25.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:24 smithi196 ceph-mon[84784]: pgmap v1315: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 17209/119151 objects degraded (14.443%); 6799/119151 objects misplaced (5.706%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:30:25.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:24 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/860445335' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:30:25.197 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:25.221 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:25.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:25.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:25.323 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:25.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:30:25.349 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:25.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:25.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:25.450 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:25.492 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:25.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:25.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:25.593 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:25.624 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:25.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:25.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:25.726 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:25.759 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:25.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:25.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:25.861 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:25.889 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:25.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:25.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:25.991 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:26.031 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:26.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:26.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:26.133 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:26.166 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:26.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:26.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:26.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:26.310 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:26.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:26.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:26.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:26.438 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:26.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:26.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:26.539 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:26.579 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:26.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:26.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:26.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:26.712 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:26.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:26.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:26.768 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:26.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:26.865 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:26.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:26.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:26.933 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:26 smithi098 ceph-mon[100914]: pgmap v1316: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 17317/115857 objects degraded (14.947%); 6591/115857 objects misplaced (5.689%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:30:26.933 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:26 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 17317/115857 objects degraded (14.947%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:26.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:26 smithi098 ceph-mon[97580]: pgmap v1316: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 17317/115857 objects degraded (14.947%); 6591/115857 objects misplaced (5.689%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:30:26.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:26 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 17317/115857 objects degraded (14.947%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:26.967 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:26.995 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:26.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:26.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:27.037 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:30:27.097 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:27.124 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:27.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:27.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:27.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:26 smithi196 ceph-mon[84784]: pgmap v1316: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 17317/115857 objects degraded (14.947%); 6591/115857 objects misplaced (5.689%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:30:27.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:26 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 17317/115857 objects degraded (14.947%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:27.225 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:27.267 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:27.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:27.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:27.370 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:27.445 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:27.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:27.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:27.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:27.593 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:27.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:27.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:27.695 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:27.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:27.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:27.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:27.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:27.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:27.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:27.958 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:27.984 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:27.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:27.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:28.085 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:28.111 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:28.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:28.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:28.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:28.254 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:28.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:28.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:28.313 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:30:28.314 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:30:28.356 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:28.389 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:28.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:28.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:28.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:28.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:28.526 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:28.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:28.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:28.670 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:28.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:28.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:28.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:28.814 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:28.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:28.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:28.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:28.953 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:28.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:28.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:29.029 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:28 smithi098 ceph-mon[97580]: pgmap v1317: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 705 op/s; 17317/115158 objects degraded (15.038%); 6591/115158 objects misplaced (5.723%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:30:29.029 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:28 smithi098 ceph-mon[97580]: from='client.55819 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:30:29.030 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:28 smithi098 ceph-mon[100914]: pgmap v1317: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 705 op/s; 17317/115158 objects degraded (15.038%); 6591/115158 objects misplaced (5.723%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:30:29.030 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:28 smithi098 ceph-mon[100914]: from='client.55819 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:30:29.054 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:29.082 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:29.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:29.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:29.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:28 smithi196 ceph-mon[84784]: pgmap v1317: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 705 op/s; 17317/115158 objects degraded (15.038%); 6591/115158 objects misplaced (5.723%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:30:29.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:28 smithi196 ceph-mon[84784]: from='client.55819 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:30:29.184 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:29.216 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:29.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:29.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:29.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:29.355 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:29.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:29.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:29.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:29.490 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:29.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:29.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:29.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:29.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:29.625 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:29.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:29.726 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:29.755 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:29.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:29.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:29.844 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1317,"stamp":"2024-09-21T10:30:27.617992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2518346123,"num_objects":38386,"num_object_clones":0,"num_object_copies":115158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17317,"num_objects_misplaced":6591,"num_objects_unfound":0,"num_objects_dirty":38386,"num_whiteouts":0,"num_read":6012,"num_read_kb":13076,"num_write":212608,"num_write_kb":8014180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2361,"num_bytes_recovered":147651538,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129519,"ondisk_log_size":129519,"up":297,"acting":294,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":302,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15011416,"kb_used_data":8465880,"kb_used_omap":35,"kb_used_meta":6545116,"kb_avail":734949800,"statfs":{"total":767960285184,"available":752588595200,"internally_reserved":0,"allocated":8669061120,"data_stored":8651642702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36353,"internal_metadata":6702199295},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-462684160,"num_objects":-7060,"num_object_clones":0,"num_object_copies":-21180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":263,"num_objects_misplaced":-1291,"num_objects_unfound":0,"num_objects_dirty":-7060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7060,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":11993088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004384"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":53,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.739376+0000","last_change":"2024-09-21T10:28:00.289895+0000","last_active":"2024-09-21T10:30:19.739376+0000","last_peered":"2024-09-21T10:30:19.739376+0000","last_clean":"2024-09-21T10:30:19.739376+0000","last_became_active":"2024-09-21T10:28:00.289622+0000","last_became_peered":"2024-09-21T10:28:00.289622+0000","last_unstale":"2024-09-21T10:30:19.739376+0000","last_undegraded":"2024-09-21T10:30:19.739376+0000","last_fullsized":"2024-09-21T10:30:19.739376+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:35:31.514813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":411,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.436355+0000","last_change":"2024-09-21T10:24:44.305198+0000","last_active":"2024-09-21T10:30:05.436355+0000","last_peered":"2024-09-21T10:30:05.436355+0000","last_clean":"2024-09-21T10:30:05.436355+0000","last_became_active":"2024-09-21T10:24:44.297653+0000","last_became_peered":"2024-09-21T10:24:44.297653+0000","last_unstale":"2024-09-21T10:30:05.436355+0000","last_undegraded":"2024-09-21T10:30:05.436355+0000","last_fullsized":"2024-09-21T10:30:05.436355+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:19.320833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":248,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.429577+0000","last_change":"2024-09-21T10:28:00.294917+0000","last_active":"2024-09-21T10:30:05.429577+0000","last_peered":"2024-09-21T10:30:05.429577+0000","last_clean":"2024-09-21T10:30:05.429577+0000","last_became_active":"2024-09-21T10:28:00.294134+0000","last_became_peered":"2024-09-21T10:28:00.294134+0000","last_unstale":"2024-09-21T10:30:05.429577+0000","last_undegraded":"2024-09-21T10:30:05.429577+0000","last_fullsized":"2024-09-21T10:30:05.429577+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:29:05.780212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":197,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.436198+0000","last_change":"2024-09-21T10:28:00.287636+0000","last_active":"2024-09-21T10:30:05.436198+0000","last_peered":"2024-09-21T10:30:05.436198+0000","last_clean":"2024-09-21T10:30:05.436198+0000","last_became_active":"2024-09-21T10:28:00.287308+0000","last_became_peered":"2024-09-21T10:28:00.287308+0000","last_unstale":"2024-09-21T10:30:05.436198+0000","last_undegraded":"2024-09-21T10:30:05.436198+0000","last_fullsized":"2024-09-21T10:30:05.436198+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:41:46.132365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":648,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.710606+0000","last_change":"2024-09-21T10:28:00.297783+0000","last_active":"2024-09-21T10:30:19.710606+0000","last_peered":"2024-09-21T10:30:19.710606+0000","last_clean":"2024-09-21T10:30:19.710606+0000","last_became_active":"2024-09-21T10:28:00.297024+0000","last_became_peered":"2024-09-21T10:28:00.297024+0000","last_unstale":"2024-09-21T10:30:19.710606+0000","last_undegraded":"2024-09-21T10:30:19.710606+0000","last_fullsized":"2024-09-21T10:30:19.710606+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:24.382523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":70,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.434836+0000","last_change":"2024-09-21T10:24:44.304399+0000","last_active":"2024-09-21T10:30:05.434836+0000","last_peered":"2024-09-21T10:30:05.434836+0000","last_clean":"2024-09-21T10:30:05.434836+0000","last_became_active":"2024-09-21T10:24:44.303437+0000","last_became_peered":"2024-09-21T10:24:44.303437+0000","last_unstale":"2024-09-21T10:30:05.434836+0000","last_undegraded":"2024-09-21T10:30:05.434836+0000","last_fullsized":"2024-09-21T10:30:05.434836+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:32:56.638481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":52,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.740272+0000","last_change":"2024-09-21T10:28:00.293505+0000","last_active":"2024-09-21T10:30:19.740272+0000","last_peered":"2024-09-21T10:30:19.740272+0000","last_clean":"2024-09-21T10:30:19.740272+0000","last_became_active":"2024-09-21T10:28:00.293173+0000","last_became_peered":"2024-09-21T10:28:00.293173+0000","last_unstale":"2024-09-21T10:30:19.740272+0000","last_undegraded":"2024-09-21T10:30:19.740272+0000","last_fullsized":"2024-09-21T10:30:19.740272+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:16.878847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":353,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.436728+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:30:05.436728+0000","last_peered":"2024-09-21T10:30:05.436728+0000","last_clean":"2024-09-21T10:30:05.436728+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:30:05.436728+0000","last_undegraded":"2024-09-21T10:30:05.436728+0000","last_fullsized":"2024-09-21T10:30:05.436728+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"381'13273","reported_seq":13352,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:24.664827+0000","last_change":"2024-09-21T10:26:36.989699+0000","last_active":"2024-09-21T10:30:24.664827+0000","last_peered":"2024-09-21T10:30:24.664827+0000","last_clean":"2024-09-21T10:30:24.664827+0000","last_became_active":"2024-09-21T10:26:36.989318+0000","last_became_peered":"2024-09-21T10:26:36.989318+0000","last_unstale":"2024-09-21T10:30:24.664827+0000","last_undegraded":"2024-09-21T10:30:24.664827+0000","last_fullsized":"2024-09-21T10:30:24.664827+0000","mapping_epoch":348,"log_start":"358'5900","ondisk_log_start":"358'5900","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7373,"log_dups_size":0,"ondisk_log_size":7373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:54:53.567344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156303360,"num_objects":2385,"num_object_clones":0,"num_object_copies":7155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13273,"num_write_kb":501056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":402,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.437298+0000","last_change":"2024-09-21T10:24:44.325196+0000","last_active":"2024-09-21T10:30:05.437298+0000","last_peered":"2024-09-21T10:30:05.437298+0000","last_clean":"2024-09-21T10:30:05.437298+0000","last_became_active":"2024-09-21T10:24:44.299614+0000","last_became_peered":"2024-09-21T10:24:44.299614+0000","last_unstale":"2024-09-21T10:30:05.437298+0000","last_undegraded":"2024-09-21T10:30:05.437298+0000","last_fullsized":"2024-09-21T10:30:05.437298+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:14.558663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":134,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.429423+0000","last_change":"2024-09-21T10:26:41.305571+0000","last_active":"2024-09-21T10:30:05.429423+0000","last_peered":"2024-09-21T10:30:05.429423+0000","last_clean":"2024-09-21T10:30:05.429423+0000","last_became_active":"2024-09-21T10:26:41.305135+0000","last_became_peered":"2024-09-21T10:26:41.305135+0000","last_unstale":"2024-09-21T10:30:05.429423+0000","last_undegraded":"2024-09-21T10:30:05.429423+0000","last_fullsized":"2024-09-21T10:30:05.429423+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:49:55.452467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"381'12974","reported_seq":13026,"reported_epoch":381,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:30:23.059283+0000","last_change":"2024-09-21T10:28:01.438839+0000","last_active":"2024-09-21T10:30:23.059283+0000","last_peered":"2024-09-21T10:30:23.059283+0000","last_clean":"2024-09-21T10:27:59.232048+0000","last_became_active":"2024-09-21T10:28:01.433549+0000","last_became_peered":"2024-09-21T10:28:01.433549+0000","last_unstale":"2024-09-21T10:30:23.059283+0000","last_undegraded":"2024-09-21T10:30:23.059283+0000","last_fullsized":"2024-09-21T10:30:23.059283+0000","mapping_epoch":361,"log_start":"357'5200","ondisk_log_start":"357'5200","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7774,"log_dups_size":0,"ondisk_log_size":7774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:58.936666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161480704,"num_objects":2464,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2464,"num_objects_unfound":0,"num_objects_dirty":2464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12974,"num_write_kb":494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,5,2],"avail_no_missing":["6","2","5"],"object_location_counts":[{"shards":"2,5,6","objects":2464}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":403,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.437070+0000","last_change":"2024-09-21T10:24:44.323182+0000","last_active":"2024-09-21T10:30:05.437070+0000","last_peered":"2024-09-21T10:30:05.437070+0000","last_clean":"2024-09-21T10:30:05.437070+0000","last_became_active":"2024-09-21T10:24:44.299698+0000","last_became_peered":"2024-09-21T10:24:44.299698+0000","last_unstale":"2024-09-21T10:30:05.437070+0000","last_undegraded":"2024-09-21T10:30:05.437070+0000","last_fullsized":"2024-09-21T10:30:05.437070+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:56.571057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":80,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.434926+0000","last_change":"2024-09-21T10:28:00.281253+0000","last_active":"2024-09-21T10:30:05.434926+0000","last_peered":"2024-09-21T10:30:05.434926+0000","last_clean":"2024-09-21T10:30:05.434926+0000","last_became_active":"2024-09-21T10:28:00.280773+0000","last_became_peered":"2024-09-21T10:28:00.280773+0000","last_unstale":"2024-09-21T10:30:05.434926+0000","last_undegraded":"2024-09-21T10:30:05.434926+0000","last_fullsized":"2024-09-21T10:30:05.434926+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:32:48.557279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"381'13089","reported_seq":17567,"reported_epoch":381,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T10:30:24.660660+0000","last_change":"2024-09-21T10:28:25.623040+0000","last_active":"2024-09-21T10:30:24.660660+0000","last_peered":"2024-09-21T10:30:24.660660+0000","last_clean":"2024-09-21T10:27:59.294389+0000","last_became_active":"2024-09-21T10:28:01.608572+0000","last_became_peered":"2024-09-21T10:28:01.608572+0000","last_unstale":"2024-09-21T10:30:24.660660+0000","last_undegraded":"2024-09-21T10:28:25.614349+0000","last_fullsized":"2024-09-21T10:28:01.272643+0000","mapping_epoch":361,"log_start":"358'5700","ondisk_log_start":"358'5700","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7389,"log_dups_size":0,"ondisk_log_size":7389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156565504,"num_objects":2389,"num_object_clones":0,"num_object_copies":7167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3154,"num_objects_misplaced":1676,"num_objects_unfound":0,"num_objects_dirty":2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13089,"num_write_kb":495296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2198,"num_bytes_recovered":144048128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":1676},{"shards":"0,4","objects":-2441}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":329,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.436822+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:30:05.436822+0000","last_peered":"2024-09-21T10:30:05.436822+0000","last_clean":"2024-09-21T10:30:05.436822+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:30:05.436822+0000","last_undegraded":"2024-09-21T10:30:05.436822+0000","last_fullsized":"2024-09-21T10:30:05.436822+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":344,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.433062+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:30:05.433062+0000","last_peered":"2024-09-21T10:30:05.433062+0000","last_clean":"2024-09-21T10:30:05.433062+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:30:05.433062+0000","last_undegraded":"2024-09-21T10:30:05.433062+0000","last_fullsized":"2024-09-21T10:30:05.433062+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"381'13042","reported_seq":13124,"reported_epoch":381,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:30:25.402608+0000","last_change":"2024-09-21T10:28:01.644360+0000","last_active":"2024-09-21T10:30:25.402608+0000","last_peered":"2024-09-21T10:30:25.402608+0000","last_clean":"2024-09-21T10:27:59.276032+0000","last_became_active":"2024-09-21T10:28:01.624219+0000","last_became_peered":"2024-09-21T10:28:01.624219+0000","last_unstale":"2024-09-21T10:30:25.402608+0000","last_undegraded":"2024-09-21T10:28:01.350374+0000","last_fullsized":"2024-09-21T10:28:01.295088+0000","mapping_epoch":361,"log_start":"355'3600","ondisk_log_start":"355'3600","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":9442,"log_dups_size":0,"ondisk_log_size":9442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153092096,"num_objects":2336,"num_object_clones":0,"num_object_copies":7008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13042,"num_write_kb":492096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0],"avail_no_missing":["2","0","7"],"object_location_counts":[{"shards":"0,2","objects":-4678},{"shards":"0,2,7","objects":1661}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":53,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.739244+0000","last_change":"2024-09-21T10:28:00.288968+0000","last_active":"2024-09-21T10:30:19.739244+0000","last_peered":"2024-09-21T10:30:19.739244+0000","last_clean":"2024-09-21T10:30:19.739244+0000","last_became_active":"2024-09-21T10:28:00.288244+0000","last_became_peered":"2024-09-21T10:28:00.288244+0000","last_unstale":"2024-09-21T10:30:19.739244+0000","last_undegraded":"2024-09-21T10:30:19.739244+0000","last_fullsized":"2024-09-21T10:30:19.739244+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:42.089701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":670,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.720553+0000","last_change":"2024-09-21T10:24:44.299417+0000","last_active":"2024-09-21T10:30:19.720553+0000","last_peered":"2024-09-21T10:30:19.720553+0000","last_clean":"2024-09-21T10:30:19.720553+0000","last_became_active":"2024-09-21T10:24:44.299116+0000","last_became_peered":"2024-09-21T10:24:44.299116+0000","last_unstale":"2024-09-21T10:30:19.720553+0000","last_undegraded":"2024-09-21T10:30:19.720553+0000","last_fullsized":"2024-09-21T10:30:19.720553+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:44.524841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"381'13609","reported_seq":13688,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:27.401220+0000","last_change":"2024-09-21T10:26:36.196642+0000","last_active":"2024-09-21T10:30:27.401220+0000","last_peered":"2024-09-21T10:30:27.401220+0000","last_clean":"2024-09-21T10:30:27.401220+0000","last_became_active":"2024-09-21T10:26:36.196357+0000","last_became_peered":"2024-09-21T10:26:36.196357+0000","last_unstale":"2024-09-21T10:30:27.401220+0000","last_undegraded":"2024-09-21T10:30:27.401220+0000","last_fullsized":"2024-09-21T10:30:27.401220+0000","mapping_epoch":348,"log_start":"356'5000","ondisk_log_start":"356'5000","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":8609,"log_dups_size":0,"ondisk_log_size":8609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:54:19.391031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147783680,"num_objects":2255,"num_object_clones":0,"num_object_copies":6765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13609,"num_write_kb":507648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"82'100","reported_seq":502,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.436216+0000","last_change":"2024-09-21T10:28:00.285919+0000","last_active":"2024-09-21T10:30:05.436216+0000","last_peered":"2024-09-21T10:30:05.436216+0000","last_clean":"2024-09-21T10:30:05.436216+0000","last_became_active":"2024-09-21T10:28:00.285324+0000","last_became_peered":"2024-09-21T10:28:00.285324+0000","last_unstale":"2024-09-21T10:30:05.436216+0000","last_undegraded":"2024-09-21T10:30:05.436216+0000","last_fullsized":"2024-09-21T10:30:05.436216+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.728195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.728195+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:00.841610+0000","scrub_duration":0.0069698140000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":57,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.738916+0000","last_change":"2024-09-21T10:28:00.303125+0000","last_active":"2024-09-21T10:30:19.738916+0000","last_peered":"2024-09-21T10:30:19.738916+0000","last_clean":"2024-09-21T10:30:19.738916+0000","last_became_active":"2024-09-21T10:28:00.293684+0000","last_became_peered":"2024-09-21T10:28:00.293684+0000","last_unstale":"2024-09-21T10:30:19.738916+0000","last_undegraded":"2024-09-21T10:30:19.738916+0000","last_fullsized":"2024-09-21T10:30:19.738916+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:04.495913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":80,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.435787+0000","last_change":"2024-09-21T10:28:00.283456+0000","last_active":"2024-09-21T10:30:05.435787+0000","last_peered":"2024-09-21T10:30:05.435787+0000","last_clean":"2024-09-21T10:30:05.435787+0000","last_became_active":"2024-09-21T10:28:00.282881+0000","last_became_peered":"2024-09-21T10:28:00.282881+0000","last_unstale":"2024-09-21T10:30:05.435787+0000","last_undegraded":"2024-09-21T10:30:05.435787+0000","last_fullsized":"2024-09-21T10:30:05.435787+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:43.681481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"381'13430","reported_seq":13512,"reported_epoch":381,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:30:25.451616+0000","last_change":"2024-09-21T10:28:01.680872+0000","last_active":"2024-09-21T10:30:25.451616+0000","last_peered":"2024-09-21T10:30:25.451616+0000","last_clean":"2024-09-21T10:27:59.257182+0000","last_became_active":"2024-09-21T10:28:01.651875+0000","last_became_peered":"2024-09-21T10:28:01.651875+0000","last_unstale":"2024-09-21T10:30:25.451616+0000","last_undegraded":"2024-09-21T10:28:01.349133+0000","last_fullsized":"2024-09-21T10:28:01.272587+0000","mapping_epoch":361,"log_start":"355'4000","ondisk_log_start":"355'4000","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":9430,"log_dups_size":0,"ondisk_log_size":9430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151650304,"num_objects":2314,"num_object_clones":0,"num_object_copies":6942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13430,"num_write_kb":503808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5],"avail_no_missing":["2","4","5"],"object_location_counts":[{"shards":"2,4,5","objects":1591},{"shards":"2,5","objects":-4835}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'100","reported_seq":173,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.717913+0000","last_change":"2024-09-21T10:28:00.301695+0000","last_active":"2024-09-21T10:30:19.717913+0000","last_peered":"2024-09-21T10:30:19.717913+0000","last_clean":"2024-09-21T10:30:19.717913+0000","last_became_active":"2024-09-21T10:28:00.289856+0000","last_became_peered":"2024-09-21T10:28:00.289856+0000","last_unstale":"2024-09-21T10:30:19.717913+0000","last_undegraded":"2024-09-21T10:30:19.717913+0000","last_fullsized":"2024-09-21T10:30:19.717913+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:27.234001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:27.234001+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:02.389036+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.433167+0000","last_change":"2024-09-21T10:24:44.309470+0000","last_active":"2024-09-21T10:30:05.433167+0000","last_peered":"2024-09-21T10:30:05.433167+0000","last_clean":"2024-09-21T10:30:05.433167+0000","last_became_active":"2024-09-21T10:24:44.308836+0000","last_became_peered":"2024-09-21T10:24:44.308836+0000","last_unstale":"2024-09-21T10:30:05.433167+0000","last_undegraded":"2024-09-21T10:30:05.433167+0000","last_fullsized":"2024-09-21T10:30:05.433167+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:09:44.622418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":70,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.435953+0000","last_change":"2024-09-21T10:24:44.305135+0000","last_active":"2024-09-21T10:30:05.435953+0000","last_peered":"2024-09-21T10:30:05.435953+0000","last_clean":"2024-09-21T10:30:05.435953+0000","last_became_active":"2024-09-21T10:24:44.304839+0000","last_became_peered":"2024-09-21T10:24:44.304839+0000","last_unstale":"2024-09-21T10:30:05.435953+0000","last_undegraded":"2024-09-21T10:30:05.435953+0000","last_fullsized":"2024-09-21T10:30:05.435953+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:02.264404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"381'13164","reported_seq":13205,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:23.043507+0000","last_change":"2024-09-21T10:26:36.194994+0000","last_active":"2024-09-21T10:30:23.043507+0000","last_peered":"2024-09-21T10:30:23.043507+0000","last_clean":"2024-09-21T10:30:23.043507+0000","last_became_active":"2024-09-21T10:26:36.194404+0000","last_became_peered":"2024-09-21T10:26:36.194404+0000","last_unstale":"2024-09-21T10:30:23.043507+0000","last_undegraded":"2024-09-21T10:30:23.043507+0000","last_fullsized":"2024-09-21T10:30:23.043507+0000","mapping_epoch":348,"log_start":"357'5400","ondisk_log_start":"357'5400","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7764,"log_dups_size":0,"ondisk_log_size":7764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:10:51.551059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154140672,"num_objects":2352,"num_object_clones":0,"num_object_copies":7056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13164,"num_write_kb":496512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":102,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.713077+0000","last_change":"2024-09-21T10:25:39.872680+0000","last_active":"2024-09-21T10:30:19.713077+0000","last_peered":"2024-09-21T10:30:19.713077+0000","last_clean":"2024-09-21T10:30:19.713077+0000","last_became_active":"2024-09-21T10:25:39.872312+0000","last_became_peered":"2024-09-21T10:25:39.872312+0000","last_unstale":"2024-09-21T10:30:19.713077+0000","last_undegraded":"2024-09-21T10:30:19.713077+0000","last_fullsized":"2024-09-21T10:30:19.713077+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:54.942046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":70,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.436847+0000","last_change":"2024-09-21T10:24:44.310484+0000","last_active":"2024-09-21T10:30:05.436847+0000","last_peered":"2024-09-21T10:30:05.436847+0000","last_clean":"2024-09-21T10:30:05.436847+0000","last_became_active":"2024-09-21T10:24:44.307209+0000","last_became_peered":"2024-09-21T10:24:44.307209+0000","last_unstale":"2024-09-21T10:30:05.436847+0000","last_undegraded":"2024-09-21T10:30:05.436847+0000","last_fullsized":"2024-09-21T10:30:05.436847+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:48:36.767539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"381'13264","reported_seq":13305,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:23.094629+0000","last_change":"2024-09-21T10:26:36.196826+0000","last_active":"2024-09-21T10:30:23.094629+0000","last_peered":"2024-09-21T10:30:23.094629+0000","last_clean":"2024-09-21T10:30:23.094629+0000","last_became_active":"2024-09-21T10:26:36.196355+0000","last_became_peered":"2024-09-21T10:26:36.196355+0000","last_unstale":"2024-09-21T10:30:23.094629+0000","last_undegraded":"2024-09-21T10:30:23.094629+0000","last_fullsized":"2024-09-21T10:30:23.094629+0000","mapping_epoch":348,"log_start":"358'6500","ondisk_log_start":"358'6500","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":6764,"log_dups_size":0,"ondisk_log_size":6764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:58.869309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155713536,"num_objects":2376,"num_object_clones":0,"num_object_copies":7128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13264,"num_write_kb":500480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'100","reported_seq":157,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.719953+0000","last_change":"2024-09-21T10:23:26.197358+0000","last_active":"2024-09-21T10:30:19.719953+0000","last_peered":"2024-09-21T10:30:19.719953+0000","last_clean":"2024-09-21T10:30:19.719953+0000","last_became_active":"2024-09-21T10:23:20.695174+0000","last_became_peered":"2024-09-21T10:23:20.695174+0000","last_unstale":"2024-09-21T10:30:19.719953+0000","last_undegraded":"2024-09-21T10:30:19.719953+0000","last_fullsized":"2024-09-21T10:30:19.719953+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:26.197291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:26.197291+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:33.961031+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":53,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.737994+0000","last_change":"2024-09-21T10:28:00.287560+0000","last_active":"2024-09-21T10:30:19.737994+0000","last_peered":"2024-09-21T10:30:19.737994+0000","last_clean":"2024-09-21T10:30:19.737994+0000","last_became_active":"2024-09-21T10:28:00.285076+0000","last_became_peered":"2024-09-21T10:28:00.285076+0000","last_unstale":"2024-09-21T10:30:19.737994+0000","last_undegraded":"2024-09-21T10:30:19.737994+0000","last_fullsized":"2024-09-21T10:30:19.737994+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:49.950523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":666,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.721466+0000","last_change":"2024-09-21T10:28:00.282062+0000","last_active":"2024-09-21T10:30:19.721466+0000","last_peered":"2024-09-21T10:30:19.721466+0000","last_clean":"2024-09-21T10:30:19.721466+0000","last_became_active":"2024-09-21T10:28:00.281593+0000","last_became_peered":"2024-09-21T10:28:00.281593+0000","last_unstale":"2024-09-21T10:30:19.721466+0000","last_undegraded":"2024-09-21T10:30:19.721466+0000","last_fullsized":"2024-09-21T10:30:19.721466+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:45.704840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"381'13191","reported_seq":13232,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:23.094393+0000","last_change":"2024-09-21T10:26:36.197465+0000","last_active":"2024-09-21T10:30:23.094393+0000","last_peered":"2024-09-21T10:30:23.094393+0000","last_clean":"2024-09-21T10:30:23.094393+0000","last_became_active":"2024-09-21T10:26:36.196105+0000","last_became_peered":"2024-09-21T10:26:36.196105+0000","last_unstale":"2024-09-21T10:30:23.094393+0000","last_undegraded":"2024-09-21T10:30:23.094393+0000","last_fullsized":"2024-09-21T10:30:23.094393+0000","mapping_epoch":348,"log_start":"357'5400","ondisk_log_start":"357'5400","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7791,"log_dups_size":0,"ondisk_log_size":7791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:16.586685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158400512,"num_objects":2417,"num_object_clones":0,"num_object_copies":7251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13191,"num_write_kb":499456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":483,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.436590+0000","last_change":"2024-09-21T10:23:18.790100+0000","last_active":"2024-09-21T10:30:05.436590+0000","last_peered":"2024-09-21T10:30:05.436590+0000","last_clean":"2024-09-21T10:30:05.436590+0000","last_became_active":"2024-09-21T10:23:14.677610+0000","last_became_peered":"2024-09-21T10:23:14.677610+0000","last_unstale":"2024-09-21T10:30:05.436590+0000","last_undegraded":"2024-09-21T10:30:05.436590+0000","last_fullsized":"2024-09-21T10:30:05.436590+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:18.790017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:18.790017+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:56.562241+0000","scrub_duration":0.0080400790000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":576,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.429473+0000","last_change":"2024-09-21T10:26:41.279374+0000","last_active":"2024-09-21T10:30:05.429473+0000","last_peered":"2024-09-21T10:30:05.429473+0000","last_clean":"2024-09-21T10:30:05.429473+0000","last_became_active":"2024-09-21T10:26:41.278994+0000","last_became_peered":"2024-09-21T10:26:41.278994+0000","last_unstale":"2024-09-21T10:30:05.429473+0000","last_undegraded":"2024-09-21T10:30:05.429473+0000","last_fullsized":"2024-09-21T10:30:05.429473+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:49:57.821175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":482,"num_read_kb":1581,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":53,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.735714+0000","last_change":"2024-09-21T10:28:00.295436+0000","last_active":"2024-09-21T10:30:19.735714+0000","last_peered":"2024-09-21T10:30:19.735714+0000","last_clean":"2024-09-21T10:30:19.735714+0000","last_became_active":"2024-09-21T10:28:00.295135+0000","last_became_peered":"2024-09-21T10:28:00.295135+0000","last_unstale":"2024-09-21T10:30:19.735714+0000","last_undegraded":"2024-09-21T10:30:19.735714+0000","last_fullsized":"2024-09-21T10:30:19.735714+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:14:03.027395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":367,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.432843+0000","last_change":"2024-09-21T10:24:44.308008+0000","last_active":"2024-09-21T10:30:05.432843+0000","last_peered":"2024-09-21T10:30:05.432843+0000","last_clean":"2024-09-21T10:30:05.432843+0000","last_became_active":"2024-09-21T10:24:44.307595+0000","last_became_peered":"2024-09-21T10:24:44.307595+0000","last_unstale":"2024-09-21T10:30:05.432843+0000","last_undegraded":"2024-09-21T10:30:05.432843+0000","last_fullsized":"2024-09-21T10:30:05.432843+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:33:49.448676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"381'13135","reported_seq":13214,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:24.594231+0000","last_change":"2024-09-21T10:26:36.197089+0000","last_active":"2024-09-21T10:30:24.594231+0000","last_peered":"2024-09-21T10:30:24.594231+0000","last_clean":"2024-09-21T10:30:24.594231+0000","last_became_active":"2024-09-21T10:26:36.196846+0000","last_became_peered":"2024-09-21T10:26:36.196846+0000","last_unstale":"2024-09-21T10:30:24.594231+0000","last_undegraded":"2024-09-21T10:30:24.594231+0000","last_fullsized":"2024-09-21T10:30:24.594231+0000","mapping_epoch":348,"log_start":"358'5800","ondisk_log_start":"358'5800","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7335,"log_dups_size":0,"ondisk_log_size":7335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:14:52.494647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153550848,"num_objects":2343,"num_object_clones":0,"num_object_copies":7029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13135,"num_write_kb":495296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"82'100","reported_seq":164,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.723581+0000","last_change":"2024-09-21T10:24:44.309154+0000","last_active":"2024-09-21T10:30:19.723581+0000","last_peered":"2024-09-21T10:30:19.723581+0000","last_clean":"2024-09-21T10:30:19.723581+0000","last_became_active":"2024-09-21T10:24:44.308330+0000","last_became_peered":"2024-09-21T10:24:44.308330+0000","last_unstale":"2024-09-21T10:30:19.723581+0000","last_undegraded":"2024-09-21T10:30:19.723581+0000","last_fullsized":"2024-09-21T10:30:19.723581+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.641718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.641718+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:10.158739+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":725,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.436511+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:30:05.436511+0000","last_peered":"2024-09-21T10:30:05.436511+0000","last_clean":"2024-09-21T10:30:05.436511+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:30:05.436511+0000","last_undegraded":"2024-09-21T10:30:05.436511+0000","last_fullsized":"2024-09-21T10:30:05.436511+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":678,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.713299+0000","last_change":"2024-09-21T10:28:00.296831+0000","last_active":"2024-09-21T10:30:19.713299+0000","last_peered":"2024-09-21T10:30:19.713299+0000","last_clean":"2024-09-21T10:30:19.713299+0000","last_became_active":"2024-09-21T10:28:00.296527+0000","last_became_peered":"2024-09-21T10:28:00.296527+0000","last_unstale":"2024-09-21T10:30:19.713299+0000","last_undegraded":"2024-09-21T10:30:19.713299+0000","last_fullsized":"2024-09-21T10:30:19.713299+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:18.024868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":647,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.710756+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:30:19.710756+0000","last_peered":"2024-09-21T10:30:19.710756+0000","last_clean":"2024-09-21T10:30:19.710756+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:30:19.710756+0000","last_undegraded":"2024-09-21T10:30:19.710756+0000","last_fullsized":"2024-09-21T10:30:19.710756+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"381'13569","reported_seq":13611,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:22.971263+0000","last_change":"2024-09-21T10:26:41.278294+0000","last_active":"2024-09-21T10:30:22.971263+0000","last_peered":"2024-09-21T10:30:22.971263+0000","last_clean":"2024-09-21T10:30:22.971263+0000","last_became_active":"2024-09-21T10:26:41.277965+0000","last_became_peered":"2024-09-21T10:26:41.277965+0000","last_unstale":"2024-09-21T10:30:22.971263+0000","last_undegraded":"2024-09-21T10:30:22.971263+0000","last_fullsized":"2024-09-21T10:30:22.971263+0000","mapping_epoch":351,"log_start":"355'3800","ondisk_log_start":"355'3800","created":348,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":9769,"log_dups_size":0,"ondisk_log_size":9769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:30.913837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163250176,"num_objects":2491,"num_object_clones":0,"num_object_copies":7473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13569,"num_write_kb":513920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"82'100","reported_seq":310,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.719984+0000","last_change":"2024-09-21T10:24:44.294460+0000","last_active":"2024-09-21T10:30:19.719984+0000","last_peered":"2024-09-21T10:30:19.719984+0000","last_clean":"2024-09-21T10:30:19.719984+0000","last_became_active":"2024-09-21T10:24:44.293972+0000","last_became_peered":"2024-09-21T10:24:44.293972+0000","last_unstale":"2024-09-21T10:30:19.719984+0000","last_undegraded":"2024-09-21T10:30:19.719984+0000","last_fullsized":"2024-09-21T10:30:19.719984+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:26.199522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:26.199522+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:26.135171+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":1456,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.718377+0000","last_change":"2024-09-21T10:28:03.346064+0000","last_active":"2024-09-21T10:30:19.718377+0000","last_peered":"2024-09-21T10:30:19.718377+0000","last_clean":"2024-09-21T10:30:19.718377+0000","last_became_active":"2024-09-21T10:28:03.345313+0000","last_became_peered":"2024-09-21T10:28:03.345313+0000","last_unstale":"2024-09-21T10:30:19.718377+0000","last_undegraded":"2024-09-21T10:30:19.718377+0000","last_fullsized":"2024-09-21T10:30:19.718377+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:32.097521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":519,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.723464+0000","last_change":"2024-09-21T10:08:18.470748+0000","last_active":"2024-09-21T10:30:19.723464+0000","last_peered":"2024-09-21T10:30:19.723464+0000","last_clean":"2024-09-21T10:30:19.723464+0000","last_became_active":"2024-09-21T10:08:18.470227+0000","last_became_peered":"2024-09-21T10:08:18.470227+0000","last_unstale":"2024-09-21T10:30:19.723464+0000","last_undegraded":"2024-09-21T10:30:19.723464+0000","last_fullsized":"2024-09-21T10:30:19.723464+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:16.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":641,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.720485+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:30:19.720485+0000","last_peered":"2024-09-21T10:30:19.720485+0000","last_clean":"2024-09-21T10:30:19.720485+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:30:19.720485+0000","last_undegraded":"2024-09-21T10:30:19.720485+0000","last_fullsized":"2024-09-21T10:30:19.720485+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"381'13377","reported_seq":13418,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:23.059480+0000","last_change":"2024-09-21T10:26:36.989641+0000","last_active":"2024-09-21T10:30:23.059480+0000","last_peered":"2024-09-21T10:30:23.059480+0000","last_clean":"2024-09-21T10:30:23.059480+0000","last_became_active":"2024-09-21T10:26:36.989340+0000","last_became_peered":"2024-09-21T10:26:36.989340+0000","last_unstale":"2024-09-21T10:30:23.059480+0000","last_undegraded":"2024-09-21T10:30:23.059480+0000","last_fullsized":"2024-09-21T10:30:23.059480+0000","mapping_epoch":348,"log_start":"357'5600","ondisk_log_start":"357'5600","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7777,"log_dups_size":0,"ondisk_log_size":7777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:50.213332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159973376,"num_objects":2441,"num_object_clones":0,"num_object_copies":7323,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13377,"num_write_kb":506176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"82'100","reported_seq":303,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.713176+0000","last_change":"2024-09-21T10:23:21.277739+0000","last_active":"2024-09-21T10:30:19.713176+0000","last_peered":"2024-09-21T10:30:19.713176+0000","last_clean":"2024-09-21T10:30:19.713176+0000","last_became_active":"2024-09-21T10:23:20.689733+0000","last_became_peered":"2024-09-21T10:23:20.689733+0000","last_unstale":"2024-09-21T10:30:19.713176+0000","last_undegraded":"2024-09-21T10:30:19.713176+0000","last_fullsized":"2024-09-21T10:30:19.713176+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.277663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.277663+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:01:54.523189+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":1117,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.720704+0000","last_change":"2024-09-21T10:07:43.794767+0000","last_active":"2024-09-21T10:30:19.720704+0000","last_peered":"2024-09-21T10:30:19.720704+0000","last_clean":"2024-09-21T10:30:19.720704+0000","last_became_active":"2024-09-21T10:07:43.794062+0000","last_became_peered":"2024-09-21T10:07:43.794062+0000","last_unstale":"2024-09-21T10:30:19.720704+0000","last_undegraded":"2024-09-21T10:30:19.720704+0000","last_fullsized":"2024-09-21T10:30:19.720704+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:51.977955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":70,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.435283+0000","last_change":"2024-09-21T10:24:44.299492+0000","last_active":"2024-09-21T10:30:05.435283+0000","last_peered":"2024-09-21T10:30:05.435283+0000","last_clean":"2024-09-21T10:30:05.435283+0000","last_became_active":"2024-09-21T10:24:44.299187+0000","last_became_peered":"2024-09-21T10:24:44.299187+0000","last_unstale":"2024-09-21T10:30:05.435283+0000","last_undegraded":"2024-09-21T10:30:05.435283+0000","last_fullsized":"2024-09-21T10:30:05.435283+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:58.500386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":571,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.718292+0000","last_change":"2024-09-21T10:28:00.289453+0000","last_active":"2024-09-21T10:30:19.718292+0000","last_peered":"2024-09-21T10:30:19.718292+0000","last_clean":"2024-09-21T10:30:19.718292+0000","last_became_active":"2024-09-21T10:28:00.289030+0000","last_became_peered":"2024-09-21T10:28:00.289030+0000","last_unstale":"2024-09-21T10:30:19.718292+0000","last_undegraded":"2024-09-21T10:30:19.718292+0000","last_fullsized":"2024-09-21T10:30:19.718292+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:21.959998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"381'13080","reported_seq":13121,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:22.998379+0000","last_change":"2024-09-21T10:26:36.196471+0000","last_active":"2024-09-21T10:30:22.998379+0000","last_peered":"2024-09-21T10:30:22.998379+0000","last_clean":"2024-09-21T10:30:22.998379+0000","last_became_active":"2024-09-21T10:26:36.196167+0000","last_became_peered":"2024-09-21T10:26:36.196167+0000","last_unstale":"2024-09-21T10:30:22.998379+0000","last_undegraded":"2024-09-21T10:30:22.998379+0000","last_fullsized":"2024-09-21T10:30:22.998379+0000","mapping_epoch":348,"log_start":"357'5100","ondisk_log_start":"357'5100","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7980,"log_dups_size":0,"ondisk_log_size":7980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:16.488795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157286400,"num_objects":2400,"num_object_clones":0,"num_object_copies":7200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13080,"num_write_kb":495360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"82'100","reported_seq":245,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.436190+0000","last_change":"2024-09-21T10:28:00.294997+0000","last_active":"2024-09-21T10:30:05.436190+0000","last_peered":"2024-09-21T10:30:05.436190+0000","last_clean":"2024-09-21T10:30:05.436190+0000","last_became_active":"2024-09-21T10:28:00.294595+0000","last_became_peered":"2024-09-21T10:28:00.294595+0000","last_unstale":"2024-09-21T10:30:05.436190+0000","last_undegraded":"2024-09-21T10:30:05.436190+0000","last_fullsized":"2024-09-21T10:30:05.436190+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:30.569982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:30.569982+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:01.422939+0000","scrub_duration":0.0051309349999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":179,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.429701+0000","last_change":"2024-09-21T10:28:00.294966+0000","last_active":"2024-09-21T10:30:05.429701+0000","last_peered":"2024-09-21T10:30:05.429701+0000","last_clean":"2024-09-21T10:30:05.429701+0000","last_became_active":"2024-09-21T10:28:00.294552+0000","last_became_peered":"2024-09-21T10:28:00.294552+0000","last_unstale":"2024-09-21T10:30:05.429701+0000","last_undegraded":"2024-09-21T10:30:05.429701+0000","last_fullsized":"2024-09-21T10:30:05.429701+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:47.292125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":1224,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.720731+0000","last_change":"2024-09-21T10:08:18.468995+0000","last_active":"2024-09-21T10:30:19.720731+0000","last_peered":"2024-09-21T10:30:19.720731+0000","last_clean":"2024-09-21T10:30:19.720731+0000","last_became_active":"2024-09-21T10:08:18.468243+0000","last_became_peered":"2024-09-21T10:08:18.468243+0000","last_unstale":"2024-09-21T10:30:19.720731+0000","last_undegraded":"2024-09-21T10:30:19.720731+0000","last_fullsized":"2024-09-21T10:30:19.720731+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:45.520968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":550,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.720944+0000","last_change":"2024-09-21T10:24:44.309614+0000","last_active":"2024-09-21T10:30:19.720944+0000","last_peered":"2024-09-21T10:30:19.720944+0000","last_clean":"2024-09-21T10:30:19.720944+0000","last_became_active":"2024-09-21T10:24:44.309187+0000","last_became_peered":"2024-09-21T10:24:44.309187+0000","last_unstale":"2024-09-21T10:30:19.720944+0000","last_undegraded":"2024-09-21T10:30:19.720944+0000","last_fullsized":"2024-09-21T10:30:19.720944+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:01.074803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"381'13229","reported_seq":13271,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:23.079827+0000","last_change":"2024-09-21T10:26:36.196357+0000","last_active":"2024-09-21T10:30:23.079827+0000","last_peered":"2024-09-21T10:30:23.079827+0000","last_clean":"2024-09-21T10:30:23.079827+0000","last_became_active":"2024-09-21T10:26:36.196060+0000","last_became_peered":"2024-09-21T10:26:36.196060+0000","last_unstale":"2024-09-21T10:30:23.079827+0000","last_undegraded":"2024-09-21T10:30:23.079827+0000","last_fullsized":"2024-09-21T10:30:23.079827+0000","mapping_epoch":348,"log_start":"357'5500","ondisk_log_start":"357'5500","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7729,"log_dups_size":0,"ondisk_log_size":7729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:46:34.643144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159907864,"num_objects":2441,"num_object_clones":0,"num_object_copies":7323,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2441,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13229,"num_write_kb":501377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"82'100","reported_seq":255,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.432502+0000","last_change":"2024-09-21T10:28:00.295277+0000","last_active":"2024-09-21T10:30:05.432502+0000","last_peered":"2024-09-21T10:30:05.432502+0000","last_clean":"2024-09-21T10:30:05.432502+0000","last_became_active":"2024-09-21T10:28:00.294016+0000","last_became_peered":"2024-09-21T10:28:00.294016+0000","last_unstale":"2024-09-21T10:30:05.432502+0000","last_undegraded":"2024-09-21T10:30:05.432502+0000","last_fullsized":"2024-09-21T10:30:05.432502+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:24.536711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:24.536711+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:28:49.984470+0000","scrub_duration":0.0047525650000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":519,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.720642+0000","last_change":"2024-09-21T10:08:18.469907+0000","last_active":"2024-09-21T10:30:19.720642+0000","last_peered":"2024-09-21T10:30:19.720642+0000","last_clean":"2024-09-21T10:30:19.720642+0000","last_became_active":"2024-09-21T10:08:18.469443+0000","last_became_peered":"2024-09-21T10:08:18.469443+0000","last_unstale":"2024-09-21T10:30:19.720642+0000","last_undegraded":"2024-09-21T10:30:19.720642+0000","last_fullsized":"2024-09-21T10:30:19.720642+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:15.645803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":803,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.436896+0000","last_change":"2024-09-21T10:24:44.334847+0000","last_active":"2024-09-21T10:30:05.436896+0000","last_peered":"2024-09-21T10:30:05.436896+0000","last_clean":"2024-09-21T10:30:05.436896+0000","last_became_active":"2024-09-21T10:24:44.301321+0000","last_became_peered":"2024-09-21T10:24:44.301321+0000","last_unstale":"2024-09-21T10:30:05.436896+0000","last_undegraded":"2024-09-21T10:30:05.436896+0000","last_fullsized":"2024-09-21T10:30:05.436896+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:37.628940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":55,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.429377+0000","last_change":"2024-09-21T10:26:41.278937+0000","last_active":"2024-09-21T10:30:05.429377+0000","last_peered":"2024-09-21T10:30:05.429377+0000","last_clean":"2024-09-21T10:30:05.429377+0000","last_became_active":"2024-09-21T10:26:41.277726+0000","last_became_peered":"2024-09-21T10:26:41.277726+0000","last_unstale":"2024-09-21T10:30:05.429377+0000","last_undegraded":"2024-09-21T10:30:05.429377+0000","last_fullsized":"2024-09-21T10:30:05.429377+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:04.741057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"381'13266","reported_seq":13307,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:23.130938+0000","last_change":"2024-09-21T10:26:36.196641+0000","last_active":"2024-09-21T10:30:23.130938+0000","last_peered":"2024-09-21T10:30:23.130938+0000","last_clean":"2024-09-21T10:30:23.130938+0000","last_became_active":"2024-09-21T10:26:36.196351+0000","last_became_peered":"2024-09-21T10:26:36.196351+0000","last_unstale":"2024-09-21T10:30:23.130938+0000","last_undegraded":"2024-09-21T10:30:23.130938+0000","last_fullsized":"2024-09-21T10:30:23.130938+0000","mapping_epoch":348,"log_start":"358'6500","ondisk_log_start":"358'6500","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":6766,"log_dups_size":0,"ondisk_log_size":6766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:41:54.280247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163315712,"num_objects":2492,"num_object_clones":0,"num_object_copies":7476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13266,"num_write_kb":504256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'100","reported_seq":70,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.435831+0000","last_change":"2024-09-21T10:24:44.296758+0000","last_active":"2024-09-21T10:30:05.435831+0000","last_peered":"2024-09-21T10:30:05.435831+0000","last_clean":"2024-09-21T10:30:05.435831+0000","last_became_active":"2024-09-21T10:24:44.296403+0000","last_became_peered":"2024-09-21T10:24:44.296403+0000","last_unstale":"2024-09-21T10:30:05.435831+0000","last_undegraded":"2024-09-21T10:30:05.435831+0000","last_fullsized":"2024-09-21T10:30:05.435831+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.105746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.105746+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:20:12.502081+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":57,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.740140+0000","last_change":"2024-09-21T10:28:00.300952+0000","last_active":"2024-09-21T10:30:19.740140+0000","last_peered":"2024-09-21T10:30:19.740140+0000","last_clean":"2024-09-21T10:30:19.740140+0000","last_became_active":"2024-09-21T10:28:00.293337+0000","last_became_peered":"2024-09-21T10:28:00.293337+0000","last_unstale":"2024-09-21T10:30:19.740140+0000","last_undegraded":"2024-09-21T10:30:19.740140+0000","last_fullsized":"2024-09-21T10:30:19.740140+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:58.409761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":944,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.721241+0000","last_change":"2024-09-21T10:24:46.844278+0000","last_active":"2024-09-21T10:30:19.721241+0000","last_peered":"2024-09-21T10:30:19.721241+0000","last_clean":"2024-09-21T10:30:19.721241+0000","last_became_active":"2024-09-21T10:24:46.840961+0000","last_became_peered":"2024-09-21T10:24:46.840961+0000","last_unstale":"2024-09-21T10:30:19.721241+0000","last_undegraded":"2024-09-21T10:30:19.721241+0000","last_fullsized":"2024-09-21T10:30:19.721241+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:49:48.823524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":53,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.737753+0000","last_change":"2024-09-21T10:28:00.293864+0000","last_active":"2024-09-21T10:30:19.737753+0000","last_peered":"2024-09-21T10:30:19.737753+0000","last_clean":"2024-09-21T10:30:19.737753+0000","last_became_active":"2024-09-21T10:28:00.293494+0000","last_became_peered":"2024-09-21T10:28:00.293494+0000","last_unstale":"2024-09-21T10:30:19.737753+0000","last_undegraded":"2024-09-21T10:30:19.737753+0000","last_fullsized":"2024-09-21T10:30:19.737753+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:37.098068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"381'12995","reported_seq":13140,"reported_epoch":381,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:30:23.105525+0000","last_change":"2024-09-21T10:28:01.631803+0000","last_active":"2024-09-21T10:30:23.105525+0000","last_peered":"2024-09-21T10:30:23.105525+0000","last_clean":"2024-09-21T10:27:59.267411+0000","last_became_active":"2024-09-21T10:28:01.425385+0000","last_became_peered":"2024-09-21T10:28:01.425385+0000","last_unstale":"2024-09-21T10:30:23.105525+0000","last_undegraded":"2024-09-21T10:30:23.105525+0000","last_fullsized":"2024-09-21T10:30:23.105525+0000","mapping_epoch":361,"log_start":"357'5300","ondisk_log_start":"357'5300","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7695,"log_dups_size":0,"ondisk_log_size":7695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:51:26.681334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161415168,"num_objects":2463,"num_object_clones":0,"num_object_copies":7389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2451,"num_objects_unfound":0,"num_objects_dirty":2463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12995,"num_write_kb":494656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":2949120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,2,4],"avail_no_missing":["6","2","4"],"object_location_counts":[{"shards":"2,4,6","objects":2463}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"82'100","reported_seq":60,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.739428+0000","last_change":"2024-09-21T10:28:00.305255+0000","last_active":"2024-09-21T10:30:19.739428+0000","last_peered":"2024-09-21T10:30:19.739428+0000","last_clean":"2024-09-21T10:30:19.739428+0000","last_became_active":"2024-09-21T10:28:00.290981+0000","last_became_peered":"2024-09-21T10:28:00.290981+0000","last_unstale":"2024-09-21T10:30:19.739428+0000","last_undegraded":"2024-09-21T10:30:19.739428+0000","last_fullsized":"2024-09-21T10:30:19.739428+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.759355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.759355+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":69,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":684,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.710506+0000","last_change":"2024-09-21T10:24:44.307344+0000","last_active":"2024-09-21T10:30:19.710506+0000","last_peered":"2024-09-21T10:30:19.710506+0000","last_clean":"2024-09-21T10:30:19.710506+0000","last_became_active":"2024-09-21T10:24:44.298936+0000","last_became_peered":"2024-09-21T10:24:44.298936+0000","last_unstale":"2024-09-21T10:30:19.710506+0000","last_undegraded":"2024-09-21T10:30:19.710506+0000","last_fullsized":"2024-09-21T10:30:19.710506+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:58:26.025877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":607,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.436917+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:30:05.436917+0000","last_peered":"2024-09-21T10:30:05.436917+0000","last_clean":"2024-09-21T10:30:05.436917+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:30:05.436917+0000","last_undegraded":"2024-09-21T10:30:05.436917+0000","last_fullsized":"2024-09-21T10:30:05.436917+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":70,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.434838+0000","last_change":"2024-09-21T10:24:44.303781+0000","last_active":"2024-09-21T10:30:05.434838+0000","last_peered":"2024-09-21T10:30:05.434838+0000","last_clean":"2024-09-21T10:30:05.434838+0000","last_became_active":"2024-09-21T10:24:44.302058+0000","last_became_peered":"2024-09-21T10:24:44.302058+0000","last_unstale":"2024-09-21T10:30:05.434838+0000","last_undegraded":"2024-09-21T10:30:05.434838+0000","last_fullsized":"2024-09-21T10:30:05.434838+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:30.371229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":70,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.435344+0000","last_change":"2024-09-21T10:24:44.303417+0000","last_active":"2024-09-21T10:30:05.435344+0000","last_peered":"2024-09-21T10:30:05.435344+0000","last_clean":"2024-09-21T10:30:05.435344+0000","last_became_active":"2024-09-21T10:24:44.303069+0000","last_became_peered":"2024-09-21T10:24:44.303069+0000","last_unstale":"2024-09-21T10:30:05.435344+0000","last_undegraded":"2024-09-21T10:30:05.435344+0000","last_fullsized":"2024-09-21T10:30:05.435344+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:30:30.966228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":80,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.435108+0000","last_change":"2024-09-21T10:24:44.321421+0000","last_active":"2024-09-21T10:30:05.435108+0000","last_peered":"2024-09-21T10:30:05.435108+0000","last_clean":"2024-09-21T10:30:05.435108+0000","last_became_active":"2024-09-21T10:24:44.297082+0000","last_became_peered":"2024-09-21T10:24:44.297082+0000","last_unstale":"2024-09-21T10:30:05.435108+0000","last_undegraded":"2024-09-21T10:30:05.435108+0000","last_fullsized":"2024-09-21T10:30:05.435108+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:35:37.704881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":518,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.720803+0000","last_change":"2024-09-21T10:08:18.470378+0000","last_active":"2024-09-21T10:30:19.720803+0000","last_peered":"2024-09-21T10:30:19.720803+0000","last_clean":"2024-09-21T10:30:19.720803+0000","last_became_active":"2024-09-21T10:08:18.469482+0000","last_became_peered":"2024-09-21T10:08:18.469482+0000","last_unstale":"2024-09-21T10:30:19.720803+0000","last_undegraded":"2024-09-21T10:30:19.720803+0000","last_fullsized":"2024-09-21T10:30:19.720803+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:35.628724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":444,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.436710+0000","last_change":"2024-09-21T10:28:00.296771+0000","last_active":"2024-09-21T10:30:05.436710+0000","last_peered":"2024-09-21T10:30:05.436710+0000","last_clean":"2024-09-21T10:30:05.436710+0000","last_became_active":"2024-09-21T10:28:00.285336+0000","last_became_peered":"2024-09-21T10:28:00.285336+0000","last_unstale":"2024-09-21T10:30:05.436710+0000","last_undegraded":"2024-09-21T10:30:05.436710+0000","last_fullsized":"2024-09-21T10:30:05.436710+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:06:50.848492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":152,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.429910+0000","last_change":"2024-09-21T10:26:41.644952+0000","last_active":"2024-09-21T10:30:05.429910+0000","last_peered":"2024-09-21T10:30:05.429910+0000","last_clean":"2024-09-21T10:30:05.429910+0000","last_became_active":"2024-09-21T10:26:41.644592+0000","last_became_peered":"2024-09-21T10:26:41.644592+0000","last_unstale":"2024-09-21T10:30:05.429910+0000","last_undegraded":"2024-09-21T10:30:05.429910+0000","last_fullsized":"2024-09-21T10:30:05.429910+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:44:44.183877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":53,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.735861+0000","last_change":"2024-09-21T10:28:00.294338+0000","last_active":"2024-09-21T10:30:19.735861+0000","last_peered":"2024-09-21T10:30:19.735861+0000","last_clean":"2024-09-21T10:30:19.735861+0000","last_became_active":"2024-09-21T10:28:00.294044+0000","last_became_peered":"2024-09-21T10:28:00.294044+0000","last_unstale":"2024-09-21T10:30:19.735861+0000","last_undegraded":"2024-09-21T10:30:19.735861+0000","last_fullsized":"2024-09-21T10:30:19.735861+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:43:30.999988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":648,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.713417+0000","last_change":"2024-09-21T10:28:00.296935+0000","last_active":"2024-09-21T10:30:19.713417+0000","last_peered":"2024-09-21T10:30:19.713417+0000","last_clean":"2024-09-21T10:30:19.713417+0000","last_became_active":"2024-09-21T10:28:00.296562+0000","last_became_peered":"2024-09-21T10:28:00.296562+0000","last_unstale":"2024-09-21T10:30:19.713417+0000","last_undegraded":"2024-09-21T10:30:19.713417+0000","last_fullsized":"2024-09-21T10:30:19.713417+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:06.863310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":53,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.737953+0000","last_change":"2024-09-21T10:28:00.294718+0000","last_active":"2024-09-21T10:30:19.737953+0000","last_peered":"2024-09-21T10:30:19.737953+0000","last_clean":"2024-09-21T10:30:19.737953+0000","last_became_active":"2024-09-21T10:28:00.294299+0000","last_became_peered":"2024-09-21T10:28:00.294299+0000","last_unstale":"2024-09-21T10:30:19.737953+0000","last_undegraded":"2024-09-21T10:30:19.737953+0000","last_fullsized":"2024-09-21T10:30:19.737953+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:41:46.881336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":344,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.436365+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:30:05.436365+0000","last_peered":"2024-09-21T10:30:05.436365+0000","last_clean":"2024-09-21T10:30:05.436365+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:30:05.436365+0000","last_undegraded":"2024-09-21T10:30:05.436365+0000","last_fullsized":"2024-09-21T10:30:05.436365+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":52,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.737834+0000","last_change":"2024-09-21T10:28:00.293872+0000","last_active":"2024-09-21T10:30:19.737834+0000","last_peered":"2024-09-21T10:30:19.737834+0000","last_clean":"2024-09-21T10:30:19.737834+0000","last_became_active":"2024-09-21T10:28:00.293320+0000","last_became_peered":"2024-09-21T10:28:00.293320+0000","last_unstale":"2024-09-21T10:30:19.737834+0000","last_undegraded":"2024-09-21T10:30:19.737834+0000","last_fullsized":"2024-09-21T10:30:19.737834+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:45:43.909307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":665,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.710681+0000","last_change":"2024-09-21T10:28:00.297304+0000","last_active":"2024-09-21T10:30:19.710681+0000","last_peered":"2024-09-21T10:30:19.710681+0000","last_clean":"2024-09-21T10:30:19.710681+0000","last_became_active":"2024-09-21T10:28:00.296631+0000","last_became_peered":"2024-09-21T10:28:00.296631+0000","last_unstale":"2024-09-21T10:30:19.710681+0000","last_undegraded":"2024-09-21T10:30:19.710681+0000","last_fullsized":"2024-09-21T10:30:19.710681+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:22:16.633581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":665,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.720195+0000","last_change":"2024-09-21T10:28:00.284924+0000","last_active":"2024-09-21T10:30:19.720195+0000","last_peered":"2024-09-21T10:30:19.720195+0000","last_clean":"2024-09-21T10:30:19.720195+0000","last_became_active":"2024-09-21T10:28:00.284633+0000","last_became_peered":"2024-09-21T10:28:00.284633+0000","last_unstale":"2024-09-21T10:30:19.720195+0000","last_undegraded":"2024-09-21T10:30:19.720195+0000","last_fullsized":"2024-09-21T10:30:19.720195+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:23:09.808151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":52,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.739827+0000","last_change":"2024-09-21T10:28:00.283403+0000","last_active":"2024-09-21T10:30:19.739827+0000","last_peered":"2024-09-21T10:30:19.739827+0000","last_clean":"2024-09-21T10:30:19.739827+0000","last_became_active":"2024-09-21T10:28:00.283089+0000","last_became_peered":"2024-09-21T10:28:00.283089+0000","last_unstale":"2024-09-21T10:30:19.739827+0000","last_undegraded":"2024-09-21T10:30:19.739827+0000","last_fullsized":"2024-09-21T10:30:19.739827+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:42.213644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":70,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.438968+0000","last_change":"2024-09-21T10:24:44.305124+0000","last_active":"2024-09-21T10:30:05.438968+0000","last_peered":"2024-09-21T10:30:05.438968+0000","last_clean":"2024-09-21T10:30:05.438968+0000","last_became_active":"2024-09-21T10:24:44.304468+0000","last_became_peered":"2024-09-21T10:24:44.304468+0000","last_unstale":"2024-09-21T10:30:05.438968+0000","last_undegraded":"2024-09-21T10:30:05.438968+0000","last_fullsized":"2024-09-21T10:30:05.438968+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:37.904764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":303,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.718146+0000","last_change":"2024-09-21T10:28:00.286646+0000","last_active":"2024-09-21T10:30:19.718146+0000","last_peered":"2024-09-21T10:30:19.718146+0000","last_clean":"2024-09-21T10:30:19.718146+0000","last_became_active":"2024-09-21T10:28:00.286300+0000","last_became_peered":"2024-09-21T10:28:00.286300+0000","last_unstale":"2024-09-21T10:30:19.718146+0000","last_undegraded":"2024-09-21T10:30:19.718146+0000","last_fullsized":"2024-09-21T10:30:19.718146+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:58.215385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":655,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.720421+0000","last_change":"2024-09-21T10:24:44.299989+0000","last_active":"2024-09-21T10:30:19.720421+0000","last_peered":"2024-09-21T10:30:19.720421+0000","last_clean":"2024-09-21T10:30:19.720421+0000","last_became_active":"2024-09-21T10:24:44.299356+0000","last_became_peered":"2024-09-21T10:24:44.299356+0000","last_unstale":"2024-09-21T10:30:19.720421+0000","last_undegraded":"2024-09-21T10:30:19.720421+0000","last_fullsized":"2024-09-21T10:30:19.720421+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:58.867008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":80,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.435166+0000","last_change":"2024-09-21T10:28:00.281942+0000","last_active":"2024-09-21T10:30:05.435166+0000","last_peered":"2024-09-21T10:30:05.435166+0000","last_clean":"2024-09-21T10:30:05.435166+0000","last_became_active":"2024-09-21T10:28:00.281600+0000","last_became_peered":"2024-09-21T10:28:00.281600+0000","last_unstale":"2024-09-21T10:30:05.435166+0000","last_undegraded":"2024-09-21T10:30:05.435166+0000","last_fullsized":"2024-09-21T10:30:05.435166+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:21.144244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":74,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.435883+0000","last_change":"2024-09-21T10:24:44.329150+0000","last_active":"2024-09-21T10:30:05.435883+0000","last_peered":"2024-09-21T10:30:05.435883+0000","last_clean":"2024-09-21T10:30:05.435883+0000","last_became_active":"2024-09-21T10:24:44.306752+0000","last_became_peered":"2024-09-21T10:24:44.306752+0000","last_unstale":"2024-09-21T10:30:05.435883+0000","last_undegraded":"2024-09-21T10:30:05.435883+0000","last_fullsized":"2024-09-21T10:30:05.435883+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:16.592859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":80,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.438974+0000","last_change":"2024-09-21T10:28:00.286009+0000","last_active":"2024-09-21T10:30:05.438974+0000","last_peered":"2024-09-21T10:30:05.438974+0000","last_clean":"2024-09-21T10:30:05.438974+0000","last_became_active":"2024-09-21T10:28:00.285327+0000","last_became_peered":"2024-09-21T10:28:00.285327+0000","last_unstale":"2024-09-21T10:30:05.438974+0000","last_undegraded":"2024-09-21T10:30:05.438974+0000","last_fullsized":"2024-09-21T10:30:05.438974+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:34.985354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":74,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.435757+0000","last_change":"2024-09-21T10:24:44.324087+0000","last_active":"2024-09-21T10:30:05.435757+0000","last_peered":"2024-09-21T10:30:05.435757+0000","last_clean":"2024-09-21T10:30:05.435757+0000","last_became_active":"2024-09-21T10:24:44.298665+0000","last_became_peered":"2024-09-21T10:24:44.298665+0000","last_unstale":"2024-09-21T10:30:05.435757+0000","last_undegraded":"2024-09-21T10:30:05.435757+0000","last_fullsized":"2024-09-21T10:30:05.435757+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:59.819340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":52,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.735748+0000","last_change":"2024-09-21T10:28:00.293054+0000","last_active":"2024-09-21T10:30:19.735748+0000","last_peered":"2024-09-21T10:30:19.735748+0000","last_clean":"2024-09-21T10:30:19.735748+0000","last_became_active":"2024-09-21T10:28:00.292723+0000","last_became_peered":"2024-09-21T10:28:00.292723+0000","last_unstale":"2024-09-21T10:30:19.735748+0000","last_undegraded":"2024-09-21T10:30:19.735748+0000","last_fullsized":"2024-09-21T10:30:19.735748+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:01:45.933570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":53,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.739675+0000","last_change":"2024-09-21T10:28:00.285515+0000","last_active":"2024-09-21T10:30:19.739675+0000","last_peered":"2024-09-21T10:30:19.739675+0000","last_clean":"2024-09-21T10:30:19.739675+0000","last_became_active":"2024-09-21T10:28:00.285203+0000","last_became_peered":"2024-09-21T10:28:00.285203+0000","last_unstale":"2024-09-21T10:30:19.739675+0000","last_undegraded":"2024-09-21T10:30:19.739675+0000","last_fullsized":"2024-09-21T10:30:19.739675+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:41.982228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":70,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T10:30:05.438651+0000","last_change":"2024-09-21T10:24:44.298820+0000","last_active":"2024-09-21T10:30:05.438651+0000","last_peered":"2024-09-21T10:30:05.438651+0000","last_clean":"2024-09-21T10:30:05.438651+0000","last_became_active":"2024-09-21T10:24:44.298478+0000","last_became_peered":"2024-09-21T10:24:44.298478+0000","last_unstale":"2024-09-21T10:30:05.438651+0000","last_undegraded":"2024-09-21T10:30:05.438651+0000","last_fullsized":"2024-09-21T10:30:05.438651+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:59.567078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":640,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.713305+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:30:19.713305+0000","last_peered":"2024-09-21T10:30:19.713305+0000","last_clean":"2024-09-21T10:30:19.713305+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:30:19.713305+0000","last_undegraded":"2024-09-21T10:30:19.713305+0000","last_fullsized":"2024-09-21T10:30:19.713305+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":2513829912,"num_objects":38359,"num_object_clones":0,"num_object_copies":115077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17317,"num_objects_misplaced":6591,"num_objects_unfound":0,"num_objects_dirty":38359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211687,"num_write_kb":8001409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":146997248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8583393280,"data_stored":8583381064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127387,"ondisk_log_size":127387,"up":48,"acting":45,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":10252,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5983,"num_read_kb":13006,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":316358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":69,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":644038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1100,"ondisk_log_size":1100,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397987,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1985904,"kb_used_data":1080360,"kb_used_omap":1,"kb_used_meta":905534,"kb_avail":91759248,"statfs":{"total":95995035648,"available":93961469952,"internally_reserved":0,"allocated":1106288640,"data_stored":1104100802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":927267160},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594215,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2001708,"kb_used_data":1104548,"kb_used_omap":1,"kb_used_meta":897150,"kb_avail":91743444,"statfs":{"total":95995035648,"available":93945286656,"internally_reserved":0,"allocated":1131057152,"data_stored":1128873307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918681944},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790445,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1701472,"kb_used_data":797400,"kb_used_omap":1,"kb_used_meta":904062,"kb_avail":92043680,"statfs":{"total":95995035648,"available":94252728320,"internally_reserved":0,"allocated":816537600,"data_stored":814362074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":925759832},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986674,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2176688,"kb_used_data":1862696,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":91568464,"statfs":{"total":95995035648,"available":93766107136,"internally_reserved":0,"allocated":1907400704,"data_stored":1905212812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":101,"seq":433791697236,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":825820,"kb_used_data":141396,"kb_used_omap":5,"kb_used_meta":684410,"kb_avail":92919332,"statfs":{"total":95995035648,"available":95149395968,"internally_reserved":0,"allocated":144789504,"data_stored":142617832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":700836624},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860763,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2051552,"kb_used_data":1065624,"kb_used_omap":8,"kb_used_meta":985847,"kb_avail":91693600,"statfs":{"total":95995035648,"available":93894246400,"internally_reserved":0,"allocated":1091198976,"data_stored":1089023730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":1009508204},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024289,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1709648,"kb_used_data":884488,"kb_used_omap":7,"kb_used_meta":825144,"kb_avail":92035504,"statfs":{"total":95995035648,"available":94244356096,"internally_reserved":0,"allocated":905715712,"data_stored":903555091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":844948018},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187824,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2558624,"kb_used_data":1529368,"kb_used_omap":8,"kb_used_meta":1028983,"kb_avail":91186528,"statfs":{"total":95995035648,"available":93375004672,"internally_reserved":0,"allocated":1566072832,"data_stored":1563897054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":1053679601},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1547177984,"data_stored":1547173912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":898695168,"data_stored":898695168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1078919168,"data_stored":1078919168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":139198464,"data_stored":139198464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1901334528,"data_stored":1901330456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":805175296,"data_stored":805175296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1118507008,"data_stored":1118502936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1094385664,"data_stored":1094385664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:30:29.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:29.898 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:29.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:30.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:30.023 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:30.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:30.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:30.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:30.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:30.164 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:30.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:30.265 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:30.293 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:30.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:30.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:30.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:30.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:30.462 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:30.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:30.563 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:30.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:30.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:30.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:30:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:30:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:30:30.704 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:30.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:30.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:30.847 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:30.885 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:30.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:30.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:30.986 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:31.019 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:31.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:31.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:31.086 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:30:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:30:30] "GET /metrics HTTP/1.1" 200 36613 "" "Prometheus/2.51.0" 2024-09-21T10:30:31.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:30 smithi098 ceph-mon[97580]: pgmap v1318: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 17317/108708 objects degraded (15.930%); 6144/108708 objects misplaced (5.652%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:30:31.087 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:30 smithi098 ceph-mon[100914]: pgmap v1318: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 17317/108708 objects degraded (15.930%); 6144/108708 objects misplaced (5.652%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:30:31.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:31.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:30 smithi196 ceph-mon[84784]: pgmap v1318: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 17317/108708 objects degraded (15.930%); 6144/108708 objects misplaced (5.652%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:30:31.182 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:31.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:31.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:31.283 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:31.309 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:31.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:31.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:31.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:31.452 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:31.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:31.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:31.554 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:31.582 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:31.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:31.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:31.684 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:31.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:31.712 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:31.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:31.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:31.844 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:31.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:31.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:31.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:31.982 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:31.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:31.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:32.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:32.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:31 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 17317/108708 objects degraded (15.930%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:32.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:31 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 17317/108708 objects degraded (15.930%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:32.127 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:32.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:32.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:32.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:31 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 17317/108708 objects degraded (15.930%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:32.229 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:32.254 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:32.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:32.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:32.356 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:32.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:32.387 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:32.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:32.453 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:32.462 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:30:32.462 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:30:32.462 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-21T10:30:32.463 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-21T10:30:32.463 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:30:32.487 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:32.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:32.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:32.634 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:32.661 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:32.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:32.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:32.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:32.779 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:30:32.791 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:32.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:32.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:32.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:30:32.893 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:32.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:32.922 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:32.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:33.023 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:33.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:33.063 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:33.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:33.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:32 smithi098 ceph-mon[97580]: pgmap v1319: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 17442/105378 objects degraded (16.552%); 5919/105378 objects misplaced (5.617%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:30:33.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:32 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3094647152' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:30:33.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:32 smithi098 ceph-mon[100914]: pgmap v1319: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 17442/105378 objects degraded (16.552%); 5919/105378 objects misplaced (5.617%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:30:33.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:32 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3094647152' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:30:33.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:32 smithi196 ceph-mon[84784]: pgmap v1319: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 17442/105378 objects degraded (16.552%); 5919/105378 objects misplaced (5.617%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:30:33.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:32 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3094647152' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:30:33.164 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:33.215 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:33.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:33.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:33.317 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:33.340 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:33.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:33.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:33.442 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:33.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:33.485 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:33.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:33.586 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:33.620 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:33.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:33.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:33.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:33.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:33.769 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:33.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:33.870 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:33.897 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:33.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:33.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:33.902 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:33 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:30:33.902 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:33 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3094647152' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:30:33.902 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:33 smithi098 ceph-mon[97580]: osdmap e382: 8 total, 8 up, 8 in 2024-09-21T10:30:33.903 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:33 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:30:33.903 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:33 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3094647152' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:30:33.903 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:33 smithi098 ceph-mon[100914]: osdmap e382: 8 total, 8 up, 8 in 2024-09-21T10:30:33.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:34.032 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:34.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:34.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:34.134 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:34.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:33 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:30:34.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:33 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3094647152' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:30:34.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:33 smithi196 ceph-mon[84784]: osdmap e382: 8 total, 8 up, 8 in 2024-09-21T10:30:34.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:34.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:34.274 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:34.312 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:34.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:34.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:34.402 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:30:34.414 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:34.439 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:34.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:34.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:34.540 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:34.566 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:34.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:34.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:34.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:34.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:34.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:34.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:34.856 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:34.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:34.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:34.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:34.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:34 smithi098 ceph-mon[97580]: pgmap v1321: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 699 op/s; 17442/98163 objects degraded (17.768%); 5445/98163 objects misplaced (5.547%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:30:34.971 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:34 smithi098 ceph-mon[100914]: pgmap v1321: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 699 op/s; 17442/98163 objects degraded (17.768%); 5445/98163 objects misplaced (5.547%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:30:35.006 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:35.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:35.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:35.108 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:35.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:34 smithi196 ceph-mon[84784]: pgmap v1321: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 2.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 699 op/s; 17442/98163 objects degraded (17.768%); 5445/98163 objects misplaced (5.547%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:30:35.151 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:35.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:35.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:35.253 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:35.291 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:35.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:35.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:35.393 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:35.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:35.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:35.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:35.575 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:35.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:35.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:35.677 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:35.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:35.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:35.818 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:35.856 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:35.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:35.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:35.953 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:35.957 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:36.048 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:36.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:36.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:36.151 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:36.186 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:36.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:36.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:36.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:36.340 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:36.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:36.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:36.441 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:36.469 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:36.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:36.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:36.571 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:36.599 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:36.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:36.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:36.700 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:36.727 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:36.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:36.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:36.828 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:36.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:36.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:36.978 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:37.005 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:37.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:37.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:37.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:36 smithi098 ceph-mon[97580]: pgmap v1322: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 17567/94920 objects degraded (18.507%); 5219/94920 objects misplaced (5.498%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:30:37.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:36 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 17567/94920 objects degraded (18.507%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:37.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:36 smithi098 ceph-mon[100914]: pgmap v1322: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 17567/94920 objects degraded (18.507%); 5219/94920 objects misplaced (5.498%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:30:37.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:36 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 17567/94920 objects degraded (18.507%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:37.107 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:37.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:36 smithi196 ceph-mon[84784]: pgmap v1322: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 17567/94920 objects degraded (18.507%); 5219/94920 objects misplaced (5.498%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:30:37.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:36 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 17567/94920 objects degraded (18.507%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:37.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:37.149 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:37.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:37.250 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:37.284 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:37.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:37.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:37.385 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:37.417 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:37.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:37.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:37.519 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:37.565 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:37.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:37.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:37.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:37.704 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:37.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:37.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:37.805 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:37.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:37.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:37.933 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:37.961 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:37.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:37.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:38.006 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:30:38.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:38.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:38.097 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:38.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:38.098 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:38.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:38.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:38.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:38.223 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:38.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:38.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:38.325 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:38.355 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:38.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:38.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:38.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:38.506 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:38.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:38.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:38.607 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:38.658 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:38.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:38.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:38.759 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:38.784 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:38.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:38.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:38.886 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:38.914 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:38.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:38.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:39.016 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:39.042 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:39.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:39.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:39.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:38 smithi098 ceph-mon[97580]: pgmap v1323: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 17567/94233 objects degraded (18.642%); 5219/94233 objects misplaced (5.538%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:30:39.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:38 smithi098 ceph-mon[100914]: pgmap v1323: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 17567/94233 objects degraded (18.642%); 5219/94233 objects misplaced (5.538%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:30:39.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:38 smithi196 ceph-mon[84784]: pgmap v1323: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 17567/94233 objects degraded (18.642%); 5219/94233 objects misplaced (5.538%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:30:39.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:39.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:39.178 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:39.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:39.279 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:39.311 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:39.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:39.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:39.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:39.462 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:39.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:39.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:39.564 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:39.596 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:39.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:39.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:39.698 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:39.722 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:39.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:39.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:39.824 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:39.859 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:39.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:39.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:39.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:39.995 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:39.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:39.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:40.096 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:40.130 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:40.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:40.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:40.232 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:40.260 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:40.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:40.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:40.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:40.410 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:40.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:40.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:40.512 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:40.554 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:40.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:40.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:40.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:30:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:30:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:30:40.656 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:40.697 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:40.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:40.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:40.799 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:40.825 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:40.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:40.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:40.926 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:40.964 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:40.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:40.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:41.066 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:41.091 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:41.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:41.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:41.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:30:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:30:40] "GET /metrics HTTP/1.1" 200 36610 "" "Prometheus/2.51.0" 2024-09-21T10:30:41.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:40 smithi098 ceph-mon[97580]: pgmap v1324: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 704 op/s; 17567/87561 objects degraded (20.063%); 4802/87561 objects misplaced (5.484%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:30:41.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:40 smithi098 ceph-mon[100914]: pgmap v1324: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 704 op/s; 17567/87561 objects degraded (20.063%); 4802/87561 objects misplaced (5.484%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:30:41.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:40 smithi196 ceph-mon[84784]: pgmap v1324: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 704 op/s; 17567/87561 objects degraded (20.063%); 4802/87561 objects misplaced (5.484%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:30:41.193 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:41.222 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:41.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:41.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:41.324 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:41.363 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:41.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:41.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:41.465 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:41.496 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:41.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:41.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:41.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:41.635 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:41.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:41.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:41.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:41.781 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:41.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:41.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:41.883 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:41.908 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:41.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:41.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:41.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:41 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 17567/87561 objects degraded (20.063%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:41.975 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:41 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 17567/87561 objects degraded (20.063%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:42.010 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:42.035 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:42.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:42.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:42.137 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:41 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 17567/87561 objects degraded (20.063%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:42.164 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:42.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:42.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:42.266 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:42.290 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:42.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:42.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:42.392 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:42.418 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:42.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:42.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:42.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:42.545 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:42.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:42.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:42.647 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:42.676 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:42.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:42.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:42.777 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:42.805 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:42.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:42.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:42.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:42.943 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:42.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:42.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:43.015 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:30:43.015 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:30:43.015 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:30:43.015 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:30:43.015 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:30:43.015 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:30:43.015 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:30:43.015 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:30:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:30:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:30:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:30:43.016 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:30:43.016 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:30:43.016 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:30:43.016 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:30:43.016 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:30:43.045 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:43.071 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:43.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:43.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:43.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:42 smithi098 ceph-mon[97580]: pgmap v1325: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 17673/84303 objects degraded (20.964%); 4594/84303 objects misplaced (5.449%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:30:43.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:42 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3649527824' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:30:43.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:42 smithi098 ceph-mon[100914]: pgmap v1325: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 17673/84303 objects degraded (20.964%); 4594/84303 objects misplaced (5.449%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:30:43.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:42 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3649527824' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:30:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:42 smithi196 ceph-mon[84784]: pgmap v1325: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 17673/84303 objects degraded (20.964%); 4594/84303 objects misplaced (5.449%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:30:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:42 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3649527824' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:30:43.173 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:43.203 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:43.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:43.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:43.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:43.343 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:43.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:43.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:43.445 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:43.469 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:43.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:43.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:43.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:43.601 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:43.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:43.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:43.703 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:43.734 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:43.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:43.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:43.835 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:43.866 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:43.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:43.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:43.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:44.005 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:44.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:44.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:44.107 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:44.153 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:44.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:44.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:44.255 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:44.287 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:44.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:44.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:44.306 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:44.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:44.431 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:44.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:44.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:44.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:44.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:44.572 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:44.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:44.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:44.701 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:44.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:44.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:44.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:44.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:44 smithi098 ceph-mon[97580]: pgmap v1326: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 849 op/s; 17673/77034 objects degraded (22.942%); 4132/77034 objects misplaced (5.364%); 1.2 MiB/s, 18 objects/s recovering 2024-09-21T10:30:44.827 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:44.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:44.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:44.928 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:44.959 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:44.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:44.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:45.061 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:45.087 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:45.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:45.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:45.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:44 smithi098 ceph-mon[100914]: pgmap v1326: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 849 op/s; 17673/77034 objects degraded (22.942%); 4132/77034 objects misplaced (5.364%); 1.2 MiB/s, 18 objects/s recovering 2024-09-21T10:30:45.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:44 smithi196 ceph-mon[84784]: pgmap v1326: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 849 op/s; 17673/77034 objects degraded (22.942%); 4132/77034 objects misplaced (5.364%); 1.2 MiB/s, 18 objects/s recovering 2024-09-21T10:30:45.189 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:45.227 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:45.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:45.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:45.328 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:45.364 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:45.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:45.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:45.466 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:45.500 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:45.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:45.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:45.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:45.638 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:45.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:45.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:45.739 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:45.769 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:45.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:45.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:45.870 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:45.913 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:45.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:45.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:46.014 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:46.053 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:46.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:46.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:46.155 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:46.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:46.194 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:46.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:46.238 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:30:46.241 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:30:46.295 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:46.319 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:46.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:46.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:46.421 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:46.466 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:46.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:46.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:46.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:46.600 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:46.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:46.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:46.702 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:46.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:46.730 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:46.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:46.831 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:46.855 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:46.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:46.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:46.942 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:46.957 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:46.981 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:46.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:46.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:47.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:30:47.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:46 smithi098 ceph-mon[97580]: pgmap v1327: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 681 op/s; 17827/73599 objects degraded (24.222%); 3880/73599 objects misplaced (5.272%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:30:47.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:46 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 17827/73599 objects degraded (24.222%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:47.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:46 smithi098 ceph-mon[97580]: from='client.36624 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:30:47.061 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:46 smithi098 ceph-mon[100914]: pgmap v1327: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 681 op/s; 17827/73599 objects degraded (24.222%); 3880/73599 objects misplaced (5.272%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:30:47.061 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:46 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 17827/73599 objects degraded (24.222%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:47.061 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:46 smithi098 ceph-mon[100914]: from='client.36624 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:30:47.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:47.114 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:47.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:47.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:47.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:46 smithi196 ceph-mon[84784]: pgmap v1327: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 681 op/s; 17827/73599 objects degraded (24.222%); 3880/73599 objects misplaced (5.272%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:30:47.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:46 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 17827/73599 objects degraded (24.222%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:47.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:46 smithi196 ceph-mon[84784]: from='client.36624 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:30:47.216 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:47.257 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:47.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:47.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:47.358 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:47.383 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:47.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:47.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:47.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:47.523 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:47.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:47.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:47.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:47.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:47.695 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:47.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:47.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:47.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:47.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:47.957 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:48.007 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:48.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:48.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:48.109 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:48.132 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:48.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:48.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:48.233 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:48.266 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:48.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:48.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:48.368 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:48.400 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:48.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:48.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:48.501 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:48.536 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:48.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:48.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:48.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:48.672 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:48.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:48.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:48.774 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:48.800 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:48.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:48.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:48.901 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:48.948 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:48.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:48.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:49.050 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:49.083 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:49.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:49.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:49.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:48 smithi098 ceph-mon[97580]: pgmap v1328: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 614 op/s; 17827/72804 objects degraded (24.486%); 3880/72804 objects misplaced (5.329%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:30:49.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:48 smithi098 ceph-mon[100914]: pgmap v1328: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 614 op/s; 17827/72804 objects degraded (24.486%); 3880/72804 objects misplaced (5.329%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:30:49.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:48 smithi196 ceph-mon[84784]: pgmap v1328: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 614 op/s; 17827/72804 objects degraded (24.486%); 3880/72804 objects misplaced (5.329%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:30:49.184 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:49.209 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:49.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:49.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:49.310 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:49.347 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:49.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:49.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:49.449 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:49.472 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:49.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:49.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:49.496 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1327,"stamp":"2024-09-21T10:30:45.629149+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1610475915,"num_objects":24533,"num_object_clones":0,"num_object_copies":73599,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17827,"num_objects_misplaced":3880,"num_objects_unfound":0,"num_objects_dirty":24533,"num_whiteouts":0,"num_read":6012,"num_read_kb":13076,"num_write":226461,"num_write_kb":8014180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2762,"num_bytes_recovered":173931474,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129372,"ondisk_log_size":129372,"up":297,"acting":294,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":302,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":12802648,"kb_used_data":5985432,"kb_used_omap":35,"kb_used_meta":6816860,"kb_avail":737158568,"statfs":{"total":767960285184,"available":754850373632,"internally_reserved":0,"allocated":6129082368,"data_stored":6111601702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36353,"internal_metadata":6980465151},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536608768,"num_objects":-8188,"num_object_clones":0,"num_object_copies":-24564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":385,"num_objects_misplaced":-1565,"num_objects_unfound":0,"num_objects_dirty":-8188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8188,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":19726336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006506"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":55,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.800686+0000","last_change":"2024-09-21T10:28:00.289895+0000","last_active":"2024-09-21T10:30:32.800686+0000","last_peered":"2024-09-21T10:30:32.800686+0000","last_clean":"2024-09-21T10:30:32.800686+0000","last_became_active":"2024-09-21T10:28:00.289622+0000","last_became_peered":"2024-09-21T10:28:00.289622+0000","last_unstale":"2024-09-21T10:30:32.800686+0000","last_undegraded":"2024-09-21T10:30:32.800686+0000","last_fullsized":"2024-09-21T10:30:32.800686+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:35:31.514813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":412,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.705669+0000","last_change":"2024-09-21T10:24:44.305198+0000","last_active":"2024-09-21T10:30:19.705669+0000","last_peered":"2024-09-21T10:30:19.705669+0000","last_clean":"2024-09-21T10:30:19.705669+0000","last_became_active":"2024-09-21T10:24:44.297653+0000","last_became_peered":"2024-09-21T10:24:44.297653+0000","last_unstale":"2024-09-21T10:30:19.705669+0000","last_undegraded":"2024-09-21T10:30:19.705669+0000","last_fullsized":"2024-09-21T10:30:19.705669+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:19.320833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":249,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.727313+0000","last_change":"2024-09-21T10:28:00.294917+0000","last_active":"2024-09-21T10:30:19.727313+0000","last_peered":"2024-09-21T10:30:19.727313+0000","last_clean":"2024-09-21T10:30:19.727313+0000","last_became_active":"2024-09-21T10:28:00.294134+0000","last_became_peered":"2024-09-21T10:28:00.294134+0000","last_unstale":"2024-09-21T10:30:19.727313+0000","last_undegraded":"2024-09-21T10:30:19.727313+0000","last_fullsized":"2024-09-21T10:30:19.727313+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:29:05.780212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":198,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.705561+0000","last_change":"2024-09-21T10:28:00.287636+0000","last_active":"2024-09-21T10:30:19.705561+0000","last_peered":"2024-09-21T10:30:19.705561+0000","last_clean":"2024-09-21T10:30:19.705561+0000","last_became_active":"2024-09-21T10:28:00.287308+0000","last_became_peered":"2024-09-21T10:28:00.287308+0000","last_unstale":"2024-09-21T10:30:19.705561+0000","last_undegraded":"2024-09-21T10:30:19.705561+0000","last_fullsized":"2024-09-21T10:30:19.705561+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:41:46.132365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":650,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.783002+0000","last_change":"2024-09-21T10:28:00.297783+0000","last_active":"2024-09-21T10:30:32.783002+0000","last_peered":"2024-09-21T10:30:32.783002+0000","last_clean":"2024-09-21T10:30:32.783002+0000","last_became_active":"2024-09-21T10:28:00.297024+0000","last_became_peered":"2024-09-21T10:28:00.297024+0000","last_unstale":"2024-09-21T10:30:32.783002+0000","last_undegraded":"2024-09-21T10:30:32.783002+0000","last_fullsized":"2024-09-21T10:30:32.783002+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:24.382523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":71,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.710200+0000","last_change":"2024-09-21T10:24:44.304399+0000","last_active":"2024-09-21T10:30:19.710200+0000","last_peered":"2024-09-21T10:30:19.710200+0000","last_clean":"2024-09-21T10:30:19.710200+0000","last_became_active":"2024-09-21T10:24:44.303437+0000","last_became_peered":"2024-09-21T10:24:44.303437+0000","last_unstale":"2024-09-21T10:30:19.710200+0000","last_undegraded":"2024-09-21T10:30:19.710200+0000","last_fullsized":"2024-09-21T10:30:19.710200+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:32:56.638481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":55,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.804282+0000","last_change":"2024-09-21T10:28:00.293505+0000","last_active":"2024-09-21T10:30:32.804282+0000","last_peered":"2024-09-21T10:30:32.804282+0000","last_clean":"2024-09-21T10:30:32.804282+0000","last_became_active":"2024-09-21T10:28:00.293173+0000","last_became_peered":"2024-09-21T10:28:00.293173+0000","last_unstale":"2024-09-21T10:30:32.804282+0000","last_undegraded":"2024-09-21T10:30:32.804282+0000","last_fullsized":"2024-09-21T10:30:32.804282+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:16.878847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":354,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.716393+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:30:19.716393+0000","last_peered":"2024-09-21T10:30:19.716393+0000","last_clean":"2024-09-21T10:30:19.716393+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:30:19.716393+0000","last_undegraded":"2024-09-21T10:30:19.716393+0000","last_fullsized":"2024-09-21T10:30:19.716393+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"382'14164","reported_seq":14245,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:44.630045+0000","last_change":"2024-09-21T10:26:36.989699+0000","last_active":"2024-09-21T10:30:44.630045+0000","last_peered":"2024-09-21T10:30:44.630045+0000","last_clean":"2024-09-21T10:30:44.630045+0000","last_became_active":"2024-09-21T10:26:36.989318+0000","last_became_peered":"2024-09-21T10:26:36.989318+0000","last_unstale":"2024-09-21T10:30:44.630045+0000","last_undegraded":"2024-09-21T10:30:44.630045+0000","last_fullsized":"2024-09-21T10:30:44.630045+0000","mapping_epoch":348,"log_start":"358'6800","ondisk_log_start":"358'6800","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7364,"log_dups_size":0,"ondisk_log_size":7364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:54:53.567344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97910784,"num_objects":1494,"num_object_clones":0,"num_object_copies":4482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14164,"num_write_kb":501056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":403,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.705944+0000","last_change":"2024-09-21T10:24:44.325196+0000","last_active":"2024-09-21T10:30:19.705944+0000","last_peered":"2024-09-21T10:30:19.705944+0000","last_clean":"2024-09-21T10:30:19.705944+0000","last_became_active":"2024-09-21T10:24:44.299614+0000","last_became_peered":"2024-09-21T10:24:44.299614+0000","last_unstale":"2024-09-21T10:30:19.705944+0000","last_undegraded":"2024-09-21T10:30:19.705944+0000","last_fullsized":"2024-09-21T10:30:19.705944+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:14.558663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":135,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.726998+0000","last_change":"2024-09-21T10:26:41.305571+0000","last_active":"2024-09-21T10:30:19.726998+0000","last_peered":"2024-09-21T10:30:19.726998+0000","last_clean":"2024-09-21T10:30:19.726998+0000","last_became_active":"2024-09-21T10:26:41.305135+0000","last_became_peered":"2024-09-21T10:26:41.305135+0000","last_unstale":"2024-09-21T10:30:19.726998+0000","last_undegraded":"2024-09-21T10:30:19.726998+0000","last_fullsized":"2024-09-21T10:30:19.726998+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:49:55.452467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"382'13900","reported_seq":13951,"reported_epoch":382,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:30:43.120629+0000","last_change":"2024-09-21T10:28:01.438839+0000","last_active":"2024-09-21T10:30:43.120629+0000","last_peered":"2024-09-21T10:30:43.120629+0000","last_clean":"2024-09-21T10:27:59.232048+0000","last_became_active":"2024-09-21T10:28:01.433549+0000","last_became_peered":"2024-09-21T10:28:01.433549+0000","last_unstale":"2024-09-21T10:30:43.120629+0000","last_undegraded":"2024-09-21T10:30:43.120629+0000","last_fullsized":"2024-09-21T10:30:43.120629+0000","mapping_epoch":361,"log_start":"358'6200","ondisk_log_start":"358'6200","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7700,"log_dups_size":0,"ondisk_log_size":7700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:58.936666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100794368,"num_objects":1538,"num_object_clones":0,"num_object_copies":4614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1538,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13900,"num_write_kb":494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,5,2],"avail_no_missing":["6","2","5"],"object_location_counts":[{"shards":"2,5,6","objects":1538}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":404,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.705793+0000","last_change":"2024-09-21T10:24:44.323182+0000","last_active":"2024-09-21T10:30:19.705793+0000","last_peered":"2024-09-21T10:30:19.705793+0000","last_clean":"2024-09-21T10:30:19.705793+0000","last_became_active":"2024-09-21T10:24:44.299698+0000","last_became_peered":"2024-09-21T10:24:44.299698+0000","last_unstale":"2024-09-21T10:30:19.705793+0000","last_undegraded":"2024-09-21T10:30:19.705793+0000","last_fullsized":"2024-09-21T10:30:19.705793+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:56.571057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":81,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.710280+0000","last_change":"2024-09-21T10:28:00.281253+0000","last_active":"2024-09-21T10:30:19.710280+0000","last_peered":"2024-09-21T10:30:19.710280+0000","last_clean":"2024-09-21T10:30:19.710280+0000","last_became_active":"2024-09-21T10:28:00.280773+0000","last_became_peered":"2024-09-21T10:28:00.280773+0000","last_unstale":"2024-09-21T10:30:19.710280+0000","last_undegraded":"2024-09-21T10:30:19.710280+0000","last_fullsized":"2024-09-21T10:30:19.710280+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:32:48.557279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"382'14001","reported_seq":19283,"reported_epoch":382,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T10:30:44.756179+0000","last_change":"2024-09-21T10:28:25.623040+0000","last_active":"2024-09-21T10:30:44.756179+0000","last_peered":"2024-09-21T10:30:44.756179+0000","last_clean":"2024-09-21T10:27:59.294389+0000","last_became_active":"2024-09-21T10:28:01.608572+0000","last_became_peered":"2024-09-21T10:28:01.608572+0000","last_unstale":"2024-09-21T10:30:44.756179+0000","last_undegraded":"2024-09-21T10:28:25.614349+0000","last_fullsized":"2024-09-21T10:28:01.272643+0000","mapping_epoch":361,"log_start":"358'6600","ondisk_log_start":"358'6600","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7401,"log_dups_size":0,"ondisk_log_size":7401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96796672,"num_objects":1477,"num_object_clones":0,"num_object_copies":4431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3664,"num_objects_misplaced":765,"num_objects_unfound":0,"num_objects_dirty":1477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14001,"num_write_kb":495296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2599,"num_bytes_recovered":170328064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":765},{"shards":"0,4","objects":-2952}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":330,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.705866+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:30:19.705866+0000","last_peered":"2024-09-21T10:30:19.705866+0000","last_clean":"2024-09-21T10:30:19.705866+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:30:19.705866+0000","last_undegraded":"2024-09-21T10:30:19.705866+0000","last_fullsized":"2024-09-21T10:30:19.705866+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":345,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.716149+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:30:19.716149+0000","last_peered":"2024-09-21T10:30:19.716149+0000","last_clean":"2024-09-21T10:30:19.716149+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:30:19.716149+0000","last_undegraded":"2024-09-21T10:30:19.716149+0000","last_fullsized":"2024-09-21T10:30:19.716149+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"382'13912","reported_seq":13996,"reported_epoch":382,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:30:45.476388+0000","last_change":"2024-09-21T10:28:01.644360+0000","last_active":"2024-09-21T10:30:45.476388+0000","last_peered":"2024-09-21T10:30:45.476388+0000","last_clean":"2024-09-21T10:27:59.276032+0000","last_became_active":"2024-09-21T10:28:01.624219+0000","last_became_peered":"2024-09-21T10:28:01.624219+0000","last_unstale":"2024-09-21T10:30:45.476388+0000","last_undegraded":"2024-09-21T10:28:01.350374+0000","last_fullsized":"2024-09-21T10:28:01.295088+0000","mapping_epoch":361,"log_start":"355'4500","ondisk_log_start":"355'4500","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":9412,"log_dups_size":0,"ondisk_log_size":9412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96075776,"num_objects":1466,"num_object_clones":0,"num_object_copies":4398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13912,"num_write_kb":492096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0],"avail_no_missing":["2","0","7"],"object_location_counts":[{"shards":"0,2","objects":-5548},{"shards":"0,2,7","objects":791}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":55,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.800598+0000","last_change":"2024-09-21T10:28:00.288968+0000","last_active":"2024-09-21T10:30:32.800598+0000","last_peered":"2024-09-21T10:30:32.800598+0000","last_clean":"2024-09-21T10:30:32.800598+0000","last_became_active":"2024-09-21T10:28:00.288244+0000","last_became_peered":"2024-09-21T10:28:00.288244+0000","last_unstale":"2024-09-21T10:30:32.800598+0000","last_undegraded":"2024-09-21T10:30:32.800598+0000","last_fullsized":"2024-09-21T10:30:32.800598+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:42.089701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":672,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.784322+0000","last_change":"2024-09-21T10:24:44.299417+0000","last_active":"2024-09-21T10:30:32.784322+0000","last_peered":"2024-09-21T10:30:32.784322+0000","last_clean":"2024-09-21T10:30:32.784322+0000","last_became_active":"2024-09-21T10:24:44.299116+0000","last_became_peered":"2024-09-21T10:24:44.299116+0000","last_unstale":"2024-09-21T10:30:32.784322+0000","last_undegraded":"2024-09-21T10:30:32.784322+0000","last_fullsized":"2024-09-21T10:30:32.784322+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:44.524841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"382'14284","reported_seq":14365,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:42.435714+0000","last_change":"2024-09-21T10:26:36.196642+0000","last_active":"2024-09-21T10:30:42.435714+0000","last_peered":"2024-09-21T10:30:42.435714+0000","last_clean":"2024-09-21T10:30:42.435714+0000","last_became_active":"2024-09-21T10:26:36.196357+0000","last_became_peered":"2024-09-21T10:26:36.196357+0000","last_unstale":"2024-09-21T10:30:42.435714+0000","last_undegraded":"2024-09-21T10:30:42.435714+0000","last_fullsized":"2024-09-21T10:30:42.435714+0000","mapping_epoch":348,"log_start":"357'5700","ondisk_log_start":"357'5700","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":8584,"log_dups_size":0,"ondisk_log_size":8584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:54:19.391031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103546880,"num_objects":1580,"num_object_clones":0,"num_object_copies":4740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14284,"num_write_kb":507648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"82'100","reported_seq":503,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.705449+0000","last_change":"2024-09-21T10:28:00.285919+0000","last_active":"2024-09-21T10:30:19.705449+0000","last_peered":"2024-09-21T10:30:19.705449+0000","last_clean":"2024-09-21T10:30:19.705449+0000","last_became_active":"2024-09-21T10:28:00.285324+0000","last_became_peered":"2024-09-21T10:28:00.285324+0000","last_unstale":"2024-09-21T10:30:19.705449+0000","last_undegraded":"2024-09-21T10:30:19.705449+0000","last_fullsized":"2024-09-21T10:30:19.705449+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.728195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.728195+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:00.841610+0000","scrub_duration":0.0069698140000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":59,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.800233+0000","last_change":"2024-09-21T10:28:00.303125+0000","last_active":"2024-09-21T10:30:32.800233+0000","last_peered":"2024-09-21T10:30:32.800233+0000","last_clean":"2024-09-21T10:30:32.800233+0000","last_became_active":"2024-09-21T10:28:00.293684+0000","last_became_peered":"2024-09-21T10:28:00.293684+0000","last_unstale":"2024-09-21T10:30:32.800233+0000","last_undegraded":"2024-09-21T10:30:32.800233+0000","last_fullsized":"2024-09-21T10:30:32.800233+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:04.495913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":81,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.708666+0000","last_change":"2024-09-21T10:28:00.283456+0000","last_active":"2024-09-21T10:30:19.708666+0000","last_peered":"2024-09-21T10:30:19.708666+0000","last_clean":"2024-09-21T10:30:19.708666+0000","last_became_active":"2024-09-21T10:28:00.282881+0000","last_became_peered":"2024-09-21T10:28:00.282881+0000","last_unstale":"2024-09-21T10:30:19.708666+0000","last_undegraded":"2024-09-21T10:30:19.708666+0000","last_fullsized":"2024-09-21T10:30:19.708666+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:43.681481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"382'14314","reported_seq":14397,"reported_epoch":382,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:30:45.513658+0000","last_change":"2024-09-21T10:28:01.680872+0000","last_active":"2024-09-21T10:30:45.513658+0000","last_peered":"2024-09-21T10:30:45.513658+0000","last_clean":"2024-09-21T10:27:59.257182+0000","last_became_active":"2024-09-21T10:28:01.651875+0000","last_became_peered":"2024-09-21T10:28:01.651875+0000","last_unstale":"2024-09-21T10:30:45.513658+0000","last_undegraded":"2024-09-21T10:28:01.349133+0000","last_fullsized":"2024-09-21T10:28:01.272587+0000","mapping_epoch":361,"log_start":"356'4900","ondisk_log_start":"356'4900","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":9414,"log_dups_size":0,"ondisk_log_size":9414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93716480,"num_objects":1430,"num_object_clones":0,"num_object_copies":4290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14314,"num_write_kb":503808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5],"avail_no_missing":["2","4","5"],"object_location_counts":[{"shards":"2,4,5","objects":707},{"shards":"2,5","objects":-5719}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'100","reported_seq":175,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.801149+0000","last_change":"2024-09-21T10:28:00.301695+0000","last_active":"2024-09-21T10:30:32.801149+0000","last_peered":"2024-09-21T10:30:32.801149+0000","last_clean":"2024-09-21T10:30:32.801149+0000","last_became_active":"2024-09-21T10:28:00.289856+0000","last_became_peered":"2024-09-21T10:28:00.289856+0000","last_unstale":"2024-09-21T10:30:32.801149+0000","last_undegraded":"2024-09-21T10:30:32.801149+0000","last_fullsized":"2024-09-21T10:30:32.801149+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:27.234001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:27.234001+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:02.389036+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":396,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.716266+0000","last_change":"2024-09-21T10:24:44.309470+0000","last_active":"2024-09-21T10:30:19.716266+0000","last_peered":"2024-09-21T10:30:19.716266+0000","last_clean":"2024-09-21T10:30:19.716266+0000","last_became_active":"2024-09-21T10:24:44.308836+0000","last_became_peered":"2024-09-21T10:24:44.308836+0000","last_unstale":"2024-09-21T10:30:19.716266+0000","last_undegraded":"2024-09-21T10:30:19.716266+0000","last_fullsized":"2024-09-21T10:30:19.716266+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:09:44.622418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":71,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.708939+0000","last_change":"2024-09-21T10:24:44.305135+0000","last_active":"2024-09-21T10:30:19.708939+0000","last_peered":"2024-09-21T10:30:19.708939+0000","last_clean":"2024-09-21T10:30:19.708939+0000","last_became_active":"2024-09-21T10:24:44.304839+0000","last_became_peered":"2024-09-21T10:24:44.304839+0000","last_unstale":"2024-09-21T10:30:19.708939+0000","last_undegraded":"2024-09-21T10:30:19.708939+0000","last_fullsized":"2024-09-21T10:30:19.708939+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:02.264404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"382'13973","reported_seq":14015,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:43.093857+0000","last_change":"2024-09-21T10:26:36.194994+0000","last_active":"2024-09-21T10:30:43.093857+0000","last_peered":"2024-09-21T10:30:43.093857+0000","last_clean":"2024-09-21T10:30:43.093857+0000","last_became_active":"2024-09-21T10:26:36.194404+0000","last_became_peered":"2024-09-21T10:26:36.194404+0000","last_unstale":"2024-09-21T10:30:43.093857+0000","last_undegraded":"2024-09-21T10:30:43.093857+0000","last_fullsized":"2024-09-21T10:30:43.093857+0000","mapping_epoch":348,"log_start":"358'6200","ondisk_log_start":"358'6200","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7773,"log_dups_size":0,"ondisk_log_size":7773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:10:51.551059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101122048,"num_objects":1543,"num_object_clones":0,"num_object_copies":4629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13973,"num_write_kb":496512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":104,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.782478+0000","last_change":"2024-09-21T10:25:39.872680+0000","last_active":"2024-09-21T10:30:32.782478+0000","last_peered":"2024-09-21T10:30:32.782478+0000","last_clean":"2024-09-21T10:30:32.782478+0000","last_became_active":"2024-09-21T10:25:39.872312+0000","last_became_peered":"2024-09-21T10:25:39.872312+0000","last_unstale":"2024-09-21T10:30:32.782478+0000","last_undegraded":"2024-09-21T10:30:32.782478+0000","last_fullsized":"2024-09-21T10:30:32.782478+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:54.942046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":71,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.708785+0000","last_change":"2024-09-21T10:24:44.310484+0000","last_active":"2024-09-21T10:30:19.708785+0000","last_peered":"2024-09-21T10:30:19.708785+0000","last_clean":"2024-09-21T10:30:19.708785+0000","last_became_active":"2024-09-21T10:24:44.307209+0000","last_became_peered":"2024-09-21T10:24:44.307209+0000","last_unstale":"2024-09-21T10:30:19.708785+0000","last_undegraded":"2024-09-21T10:30:19.708785+0000","last_fullsized":"2024-09-21T10:30:19.708785+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:48:36.767539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"382'14117","reported_seq":14159,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:43.129381+0000","last_change":"2024-09-21T10:26:36.196826+0000","last_active":"2024-09-21T10:30:43.129381+0000","last_peered":"2024-09-21T10:30:43.129381+0000","last_clean":"2024-09-21T10:30:43.129381+0000","last_became_active":"2024-09-21T10:26:36.196355+0000","last_became_peered":"2024-09-21T10:26:36.196355+0000","last_unstale":"2024-09-21T10:30:43.129381+0000","last_undegraded":"2024-09-21T10:30:43.129381+0000","last_fullsized":"2024-09-21T10:30:43.129381+0000","mapping_epoch":348,"log_start":"364'7400","ondisk_log_start":"364'7400","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":6717,"log_dups_size":0,"ondisk_log_size":6717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:58.869309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99811328,"num_objects":1523,"num_object_clones":0,"num_object_copies":4569,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14117,"num_write_kb":500480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'100","reported_seq":159,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.801711+0000","last_change":"2024-09-21T10:23:26.197358+0000","last_active":"2024-09-21T10:30:32.801711+0000","last_peered":"2024-09-21T10:30:32.801711+0000","last_clean":"2024-09-21T10:30:32.801711+0000","last_became_active":"2024-09-21T10:23:20.695174+0000","last_became_peered":"2024-09-21T10:23:20.695174+0000","last_unstale":"2024-09-21T10:30:32.801711+0000","last_undegraded":"2024-09-21T10:30:32.801711+0000","last_fullsized":"2024-09-21T10:30:32.801711+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:26.197291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:26.197291+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:33.961031+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":55,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.800352+0000","last_change":"2024-09-21T10:28:00.287560+0000","last_active":"2024-09-21T10:30:32.800352+0000","last_peered":"2024-09-21T10:30:32.800352+0000","last_clean":"2024-09-21T10:30:32.800352+0000","last_became_active":"2024-09-21T10:28:00.285076+0000","last_became_peered":"2024-09-21T10:28:00.285076+0000","last_unstale":"2024-09-21T10:30:32.800352+0000","last_undegraded":"2024-09-21T10:30:32.800352+0000","last_fullsized":"2024-09-21T10:30:32.800352+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:49.950523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":668,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.905622+0000","last_change":"2024-09-21T10:28:00.282062+0000","last_active":"2024-09-21T10:30:32.905622+0000","last_peered":"2024-09-21T10:30:32.905622+0000","last_clean":"2024-09-21T10:30:32.905622+0000","last_became_active":"2024-09-21T10:28:00.281593+0000","last_became_peered":"2024-09-21T10:28:00.281593+0000","last_unstale":"2024-09-21T10:30:32.905622+0000","last_undegraded":"2024-09-21T10:30:32.905622+0000","last_fullsized":"2024-09-21T10:30:32.905622+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:45.704840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"382'14084","reported_seq":14126,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:43.098457+0000","last_change":"2024-09-21T10:26:36.197465+0000","last_active":"2024-09-21T10:30:43.098457+0000","last_peered":"2024-09-21T10:30:43.098457+0000","last_clean":"2024-09-21T10:30:43.098457+0000","last_became_active":"2024-09-21T10:26:36.196105+0000","last_became_peered":"2024-09-21T10:26:36.196105+0000","last_unstale":"2024-09-21T10:30:43.098457+0000","last_undegraded":"2024-09-21T10:30:43.098457+0000","last_fullsized":"2024-09-21T10:30:43.098457+0000","mapping_epoch":348,"log_start":"358'6300","ondisk_log_start":"358'6300","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7784,"log_dups_size":0,"ondisk_log_size":7784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:16.586685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99876864,"num_objects":1524,"num_object_clones":0,"num_object_copies":4572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14084,"num_write_kb":499456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":484,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.705459+0000","last_change":"2024-09-21T10:23:18.790100+0000","last_active":"2024-09-21T10:30:19.705459+0000","last_peered":"2024-09-21T10:30:19.705459+0000","last_clean":"2024-09-21T10:30:19.705459+0000","last_became_active":"2024-09-21T10:23:14.677610+0000","last_became_peered":"2024-09-21T10:23:14.677610+0000","last_unstale":"2024-09-21T10:30:19.705459+0000","last_undegraded":"2024-09-21T10:30:19.705459+0000","last_fullsized":"2024-09-21T10:30:19.705459+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:18.790017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:18.790017+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:56.562241+0000","scrub_duration":0.0080400790000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":577,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.727213+0000","last_change":"2024-09-21T10:26:41.279374+0000","last_active":"2024-09-21T10:30:19.727213+0000","last_peered":"2024-09-21T10:30:19.727213+0000","last_clean":"2024-09-21T10:30:19.727213+0000","last_became_active":"2024-09-21T10:26:41.278994+0000","last_became_peered":"2024-09-21T10:26:41.278994+0000","last_unstale":"2024-09-21T10:30:19.727213+0000","last_undegraded":"2024-09-21T10:30:19.727213+0000","last_fullsized":"2024-09-21T10:30:19.727213+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:49:57.821175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":482,"num_read_kb":1581,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":55,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.800946+0000","last_change":"2024-09-21T10:28:00.295436+0000","last_active":"2024-09-21T10:30:32.800946+0000","last_peered":"2024-09-21T10:30:32.800946+0000","last_clean":"2024-09-21T10:30:32.800946+0000","last_became_active":"2024-09-21T10:28:00.295135+0000","last_became_peered":"2024-09-21T10:28:00.295135+0000","last_unstale":"2024-09-21T10:30:32.800946+0000","last_undegraded":"2024-09-21T10:30:32.800946+0000","last_fullsized":"2024-09-21T10:30:32.800946+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:14:03.027395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":368,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.719170+0000","last_change":"2024-09-21T10:24:44.308008+0000","last_active":"2024-09-21T10:30:19.719170+0000","last_peered":"2024-09-21T10:30:19.719170+0000","last_clean":"2024-09-21T10:30:19.719170+0000","last_became_active":"2024-09-21T10:24:44.307595+0000","last_became_peered":"2024-09-21T10:24:44.307595+0000","last_unstale":"2024-09-21T10:30:19.719170+0000","last_undegraded":"2024-09-21T10:30:19.719170+0000","last_fullsized":"2024-09-21T10:30:19.719170+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:33:49.448676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"382'14000","reported_seq":14081,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:44.734574+0000","last_change":"2024-09-21T10:26:36.197089+0000","last_active":"2024-09-21T10:30:44.734574+0000","last_peered":"2024-09-21T10:30:44.734574+0000","last_clean":"2024-09-21T10:30:44.734574+0000","last_became_active":"2024-09-21T10:26:36.196846+0000","last_became_peered":"2024-09-21T10:26:36.196846+0000","last_unstale":"2024-09-21T10:30:44.734574+0000","last_undegraded":"2024-09-21T10:30:44.734574+0000","last_fullsized":"2024-09-21T10:30:44.734574+0000","mapping_epoch":348,"log_start":"358'6600","ondisk_log_start":"358'6600","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7400,"log_dups_size":0,"ondisk_log_size":7400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:14:52.494647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96862208,"num_objects":1478,"num_object_clones":0,"num_object_copies":4434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14000,"num_write_kb":495296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"82'100","reported_seq":166,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.907345+0000","last_change":"2024-09-21T10:24:44.309154+0000","last_active":"2024-09-21T10:30:32.907345+0000","last_peered":"2024-09-21T10:30:32.907345+0000","last_clean":"2024-09-21T10:30:32.907345+0000","last_became_active":"2024-09-21T10:24:44.308330+0000","last_became_peered":"2024-09-21T10:24:44.308330+0000","last_unstale":"2024-09-21T10:30:32.907345+0000","last_undegraded":"2024-09-21T10:30:32.907345+0000","last_fullsized":"2024-09-21T10:30:32.907345+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.641718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.641718+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:10.158739+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":726,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.705449+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:30:19.705449+0000","last_peered":"2024-09-21T10:30:19.705449+0000","last_clean":"2024-09-21T10:30:19.705449+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:30:19.705449+0000","last_undegraded":"2024-09-21T10:30:19.705449+0000","last_fullsized":"2024-09-21T10:30:19.705449+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":680,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.782650+0000","last_change":"2024-09-21T10:28:00.296831+0000","last_active":"2024-09-21T10:30:32.782650+0000","last_peered":"2024-09-21T10:30:32.782650+0000","last_clean":"2024-09-21T10:30:32.782650+0000","last_became_active":"2024-09-21T10:28:00.296527+0000","last_became_peered":"2024-09-21T10:28:00.296527+0000","last_unstale":"2024-09-21T10:30:32.782650+0000","last_undegraded":"2024-09-21T10:30:32.782650+0000","last_fullsized":"2024-09-21T10:30:32.782650+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:18.024868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":649,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.783086+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:30:32.783086+0000","last_peered":"2024-09-21T10:30:32.783086+0000","last_clean":"2024-09-21T10:30:32.783086+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:30:32.783086+0000","last_undegraded":"2024-09-21T10:30:32.783086+0000","last_fullsized":"2024-09-21T10:30:32.783086+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"382'14429","reported_seq":14472,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:43.026936+0000","last_change":"2024-09-21T10:26:41.278294+0000","last_active":"2024-09-21T10:30:43.026936+0000","last_peered":"2024-09-21T10:30:43.026936+0000","last_clean":"2024-09-21T10:30:43.026936+0000","last_became_active":"2024-09-21T10:26:41.277965+0000","last_became_peered":"2024-09-21T10:26:41.277965+0000","last_unstale":"2024-09-21T10:30:43.026936+0000","last_undegraded":"2024-09-21T10:30:43.026936+0000","last_fullsized":"2024-09-21T10:30:43.026936+0000","mapping_epoch":351,"log_start":"355'4700","ondisk_log_start":"355'4700","created":348,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":9729,"log_dups_size":0,"ondisk_log_size":9729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:30.913837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106889216,"num_objects":1631,"num_object_clones":0,"num_object_copies":4893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14429,"num_write_kb":513920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"82'100","reported_seq":312,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.786630+0000","last_change":"2024-09-21T10:24:44.294460+0000","last_active":"2024-09-21T10:30:32.786630+0000","last_peered":"2024-09-21T10:30:32.786630+0000","last_clean":"2024-09-21T10:30:32.786630+0000","last_became_active":"2024-09-21T10:24:44.293972+0000","last_became_peered":"2024-09-21T10:24:44.293972+0000","last_unstale":"2024-09-21T10:30:32.786630+0000","last_undegraded":"2024-09-21T10:30:32.786630+0000","last_fullsized":"2024-09-21T10:30:32.786630+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:26.199522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:26.199522+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:26.135171+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":1458,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.802440+0000","last_change":"2024-09-21T10:28:03.346064+0000","last_active":"2024-09-21T10:30:32.802440+0000","last_peered":"2024-09-21T10:30:32.802440+0000","last_clean":"2024-09-21T10:30:32.802440+0000","last_became_active":"2024-09-21T10:28:03.345313+0000","last_became_peered":"2024-09-21T10:28:03.345313+0000","last_unstale":"2024-09-21T10:30:32.802440+0000","last_undegraded":"2024-09-21T10:30:32.802440+0000","last_fullsized":"2024-09-21T10:30:32.802440+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:32.097521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":521,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.912981+0000","last_change":"2024-09-21T10:08:18.470748+0000","last_active":"2024-09-21T10:30:32.912981+0000","last_peered":"2024-09-21T10:30:32.912981+0000","last_clean":"2024-09-21T10:30:32.912981+0000","last_became_active":"2024-09-21T10:08:18.470227+0000","last_became_peered":"2024-09-21T10:08:18.470227+0000","last_unstale":"2024-09-21T10:30:32.912981+0000","last_undegraded":"2024-09-21T10:30:32.912981+0000","last_fullsized":"2024-09-21T10:30:32.912981+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:16.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":643,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.804580+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:30:32.804580+0000","last_peered":"2024-09-21T10:30:32.804580+0000","last_clean":"2024-09-21T10:30:32.804580+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:30:32.804580+0000","last_undegraded":"2024-09-21T10:30:32.804580+0000","last_fullsized":"2024-09-21T10:30:32.804580+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"382'14261","reported_seq":14303,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:43.098484+0000","last_change":"2024-09-21T10:26:36.989641+0000","last_active":"2024-09-21T10:30:43.098484+0000","last_peered":"2024-09-21T10:30:43.098484+0000","last_clean":"2024-09-21T10:30:43.098484+0000","last_became_active":"2024-09-21T10:26:36.989340+0000","last_became_peered":"2024-09-21T10:26:36.989340+0000","last_unstale":"2024-09-21T10:30:43.098484+0000","last_undegraded":"2024-09-21T10:30:43.098484+0000","last_fullsized":"2024-09-21T10:30:43.098484+0000","mapping_epoch":348,"log_start":"358'6500","ondisk_log_start":"358'6500","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7761,"log_dups_size":0,"ondisk_log_size":7761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:50.213332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102039552,"num_objects":1557,"num_object_clones":0,"num_object_copies":4671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14261,"num_write_kb":506176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"82'100","reported_seq":305,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.782566+0000","last_change":"2024-09-21T10:23:21.277739+0000","last_active":"2024-09-21T10:30:32.782566+0000","last_peered":"2024-09-21T10:30:32.782566+0000","last_clean":"2024-09-21T10:30:32.782566+0000","last_became_active":"2024-09-21T10:23:20.689733+0000","last_became_peered":"2024-09-21T10:23:20.689733+0000","last_unstale":"2024-09-21T10:30:32.782566+0000","last_undegraded":"2024-09-21T10:30:32.782566+0000","last_fullsized":"2024-09-21T10:30:32.782566+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.277663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.277663+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:01:54.523189+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":1120,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.907613+0000","last_change":"2024-09-21T10:07:43.794767+0000","last_active":"2024-09-21T10:30:32.907613+0000","last_peered":"2024-09-21T10:30:32.907613+0000","last_clean":"2024-09-21T10:30:32.907613+0000","last_became_active":"2024-09-21T10:07:43.794062+0000","last_became_peered":"2024-09-21T10:07:43.794062+0000","last_unstale":"2024-09-21T10:30:32.907613+0000","last_undegraded":"2024-09-21T10:30:32.907613+0000","last_fullsized":"2024-09-21T10:30:32.907613+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:51.977955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":71,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.708590+0000","last_change":"2024-09-21T10:24:44.299492+0000","last_active":"2024-09-21T10:30:19.708590+0000","last_peered":"2024-09-21T10:30:19.708590+0000","last_clean":"2024-09-21T10:30:19.708590+0000","last_became_active":"2024-09-21T10:24:44.299187+0000","last_became_peered":"2024-09-21T10:24:44.299187+0000","last_unstale":"2024-09-21T10:30:19.708590+0000","last_undegraded":"2024-09-21T10:30:19.708590+0000","last_fullsized":"2024-09-21T10:30:19.708590+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:58.500386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":574,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.802044+0000","last_change":"2024-09-21T10:28:00.289453+0000","last_active":"2024-09-21T10:30:32.802044+0000","last_peered":"2024-09-21T10:30:32.802044+0000","last_clean":"2024-09-21T10:30:32.802044+0000","last_became_active":"2024-09-21T10:28:00.289030+0000","last_became_peered":"2024-09-21T10:28:00.289030+0000","last_unstale":"2024-09-21T10:30:32.802044+0000","last_undegraded":"2024-09-21T10:30:32.802044+0000","last_fullsized":"2024-09-21T10:30:32.802044+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:21.959998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"382'13956","reported_seq":13997,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:43.004030+0000","last_change":"2024-09-21T10:26:36.196471+0000","last_active":"2024-09-21T10:30:43.004030+0000","last_peered":"2024-09-21T10:30:43.004030+0000","last_clean":"2024-09-21T10:30:43.004030+0000","last_became_active":"2024-09-21T10:26:36.196167+0000","last_became_peered":"2024-09-21T10:26:36.196167+0000","last_unstale":"2024-09-21T10:30:43.004030+0000","last_undegraded":"2024-09-21T10:30:43.004030+0000","last_fullsized":"2024-09-21T10:30:43.004030+0000","mapping_epoch":348,"log_start":"358'6000","ondisk_log_start":"358'6000","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7956,"log_dups_size":0,"ondisk_log_size":7956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:16.488795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99876864,"num_objects":1524,"num_object_clones":0,"num_object_copies":4572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13956,"num_write_kb":495360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"82'100","reported_seq":246,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.716807+0000","last_change":"2024-09-21T10:28:00.294997+0000","last_active":"2024-09-21T10:30:19.716807+0000","last_peered":"2024-09-21T10:30:19.716807+0000","last_clean":"2024-09-21T10:30:19.716807+0000","last_became_active":"2024-09-21T10:28:00.294595+0000","last_became_peered":"2024-09-21T10:28:00.294595+0000","last_unstale":"2024-09-21T10:30:19.716807+0000","last_undegraded":"2024-09-21T10:30:19.716807+0000","last_fullsized":"2024-09-21T10:30:19.716807+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:30.569982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:30.569982+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:01.422939+0000","scrub_duration":0.0051309349999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":180,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.727398+0000","last_change":"2024-09-21T10:28:00.294966+0000","last_active":"2024-09-21T10:30:19.727398+0000","last_peered":"2024-09-21T10:30:19.727398+0000","last_clean":"2024-09-21T10:30:19.727398+0000","last_became_active":"2024-09-21T10:28:00.294552+0000","last_became_peered":"2024-09-21T10:28:00.294552+0000","last_unstale":"2024-09-21T10:30:19.727398+0000","last_undegraded":"2024-09-21T10:30:19.727398+0000","last_fullsized":"2024-09-21T10:30:19.727398+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:47.292125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":1226,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.905056+0000","last_change":"2024-09-21T10:08:18.468995+0000","last_active":"2024-09-21T10:30:32.905056+0000","last_peered":"2024-09-21T10:30:32.905056+0000","last_clean":"2024-09-21T10:30:32.905056+0000","last_became_active":"2024-09-21T10:08:18.468243+0000","last_became_peered":"2024-09-21T10:08:18.468243+0000","last_unstale":"2024-09-21T10:30:32.905056+0000","last_undegraded":"2024-09-21T10:30:32.905056+0000","last_fullsized":"2024-09-21T10:30:32.905056+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:45.520968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":552,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.907912+0000","last_change":"2024-09-21T10:24:44.309614+0000","last_active":"2024-09-21T10:30:32.907912+0000","last_peered":"2024-09-21T10:30:32.907912+0000","last_clean":"2024-09-21T10:30:32.907912+0000","last_became_active":"2024-09-21T10:24:44.309187+0000","last_became_peered":"2024-09-21T10:24:44.309187+0000","last_unstale":"2024-09-21T10:30:32.907912+0000","last_undegraded":"2024-09-21T10:30:32.907912+0000","last_fullsized":"2024-09-21T10:30:32.907912+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:01.074803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"382'14123","reported_seq":14166,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:43.112765+0000","last_change":"2024-09-21T10:26:36.196357+0000","last_active":"2024-09-21T10:30:43.112765+0000","last_peered":"2024-09-21T10:30:43.112765+0000","last_clean":"2024-09-21T10:30:43.112765+0000","last_became_active":"2024-09-21T10:26:36.196060+0000","last_became_peered":"2024-09-21T10:26:36.196060+0000","last_unstale":"2024-09-21T10:30:43.112765+0000","last_undegraded":"2024-09-21T10:30:43.112765+0000","last_fullsized":"2024-09-21T10:30:43.112765+0000","mapping_epoch":348,"log_start":"358'6400","ondisk_log_start":"358'6400","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7723,"log_dups_size":0,"ondisk_log_size":7723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:46:34.643144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101318680,"num_objects":1547,"num_object_clones":0,"num_object_copies":4641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1547,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14123,"num_write_kb":501377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"82'100","reported_seq":256,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.716336+0000","last_change":"2024-09-21T10:28:00.295277+0000","last_active":"2024-09-21T10:30:19.716336+0000","last_peered":"2024-09-21T10:30:19.716336+0000","last_clean":"2024-09-21T10:30:19.716336+0000","last_became_active":"2024-09-21T10:28:00.294016+0000","last_became_peered":"2024-09-21T10:28:00.294016+0000","last_unstale":"2024-09-21T10:30:19.716336+0000","last_undegraded":"2024-09-21T10:30:19.716336+0000","last_fullsized":"2024-09-21T10:30:19.716336+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:24.536711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:24.536711+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:28:49.984470+0000","scrub_duration":0.0047525650000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":521,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.904978+0000","last_change":"2024-09-21T10:08:18.469907+0000","last_active":"2024-09-21T10:30:32.904978+0000","last_peered":"2024-09-21T10:30:32.904978+0000","last_clean":"2024-09-21T10:30:32.904978+0000","last_became_active":"2024-09-21T10:08:18.469443+0000","last_became_peered":"2024-09-21T10:08:18.469443+0000","last_unstale":"2024-09-21T10:30:32.904978+0000","last_undegraded":"2024-09-21T10:30:32.904978+0000","last_fullsized":"2024-09-21T10:30:32.904978+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:15.645803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":804,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.705948+0000","last_change":"2024-09-21T10:24:44.334847+0000","last_active":"2024-09-21T10:30:19.705948+0000","last_peered":"2024-09-21T10:30:19.705948+0000","last_clean":"2024-09-21T10:30:19.705948+0000","last_became_active":"2024-09-21T10:24:44.301321+0000","last_became_peered":"2024-09-21T10:24:44.301321+0000","last_unstale":"2024-09-21T10:30:19.705948+0000","last_undegraded":"2024-09-21T10:30:19.705948+0000","last_fullsized":"2024-09-21T10:30:19.705948+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:37.628940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":56,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.727125+0000","last_change":"2024-09-21T10:26:41.278937+0000","last_active":"2024-09-21T10:30:19.727125+0000","last_peered":"2024-09-21T10:30:19.727125+0000","last_clean":"2024-09-21T10:30:19.727125+0000","last_became_active":"2024-09-21T10:26:41.277726+0000","last_became_peered":"2024-09-21T10:26:41.277726+0000","last_unstale":"2024-09-21T10:30:19.727125+0000","last_undegraded":"2024-09-21T10:30:19.727125+0000","last_fullsized":"2024-09-21T10:30:19.727125+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:04.741057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"382'14146","reported_seq":14188,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:43.118618+0000","last_change":"2024-09-21T10:26:36.196641+0000","last_active":"2024-09-21T10:30:43.118618+0000","last_peered":"2024-09-21T10:30:43.118618+0000","last_clean":"2024-09-21T10:30:43.118618+0000","last_became_active":"2024-09-21T10:26:36.196351+0000","last_became_peered":"2024-09-21T10:26:36.196351+0000","last_unstale":"2024-09-21T10:30:43.118618+0000","last_undegraded":"2024-09-21T10:30:43.118618+0000","last_fullsized":"2024-09-21T10:30:43.118618+0000","mapping_epoch":348,"log_start":"364'7400","ondisk_log_start":"364'7400","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":6746,"log_dups_size":0,"ondisk_log_size":6746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:41:54.280247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105644032,"num_objects":1612,"num_object_clones":0,"num_object_copies":4836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14146,"num_write_kb":504256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'100","reported_seq":71,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.708855+0000","last_change":"2024-09-21T10:24:44.296758+0000","last_active":"2024-09-21T10:30:19.708855+0000","last_peered":"2024-09-21T10:30:19.708855+0000","last_clean":"2024-09-21T10:30:19.708855+0000","last_became_active":"2024-09-21T10:24:44.296403+0000","last_became_peered":"2024-09-21T10:24:44.296403+0000","last_unstale":"2024-09-21T10:30:19.708855+0000","last_undegraded":"2024-09-21T10:30:19.708855+0000","last_fullsized":"2024-09-21T10:30:19.708855+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.105746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.105746+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:20:12.502081+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":59,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.804067+0000","last_change":"2024-09-21T10:28:00.300952+0000","last_active":"2024-09-21T10:30:32.804067+0000","last_peered":"2024-09-21T10:30:32.804067+0000","last_clean":"2024-09-21T10:30:32.804067+0000","last_became_active":"2024-09-21T10:28:00.293337+0000","last_became_peered":"2024-09-21T10:28:00.293337+0000","last_unstale":"2024-09-21T10:30:32.804067+0000","last_undegraded":"2024-09-21T10:30:32.804067+0000","last_fullsized":"2024-09-21T10:30:32.804067+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:58.409761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":946,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.905432+0000","last_change":"2024-09-21T10:24:46.844278+0000","last_active":"2024-09-21T10:30:32.905432+0000","last_peered":"2024-09-21T10:30:32.905432+0000","last_clean":"2024-09-21T10:30:32.905432+0000","last_became_active":"2024-09-21T10:24:46.840961+0000","last_became_peered":"2024-09-21T10:24:46.840961+0000","last_unstale":"2024-09-21T10:30:32.905432+0000","last_undegraded":"2024-09-21T10:30:32.905432+0000","last_fullsized":"2024-09-21T10:30:32.905432+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:49:48.823524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":55,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.800080+0000","last_change":"2024-09-21T10:28:00.293864+0000","last_active":"2024-09-21T10:30:32.800080+0000","last_peered":"2024-09-21T10:30:32.800080+0000","last_clean":"2024-09-21T10:30:32.800080+0000","last_became_active":"2024-09-21T10:28:00.293494+0000","last_became_peered":"2024-09-21T10:28:00.293494+0000","last_unstale":"2024-09-21T10:30:32.800080+0000","last_undegraded":"2024-09-21T10:30:32.800080+0000","last_fullsized":"2024-09-21T10:30:32.800080+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:37.098068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"382'13876","reported_seq":14022,"reported_epoch":382,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:30:43.101671+0000","last_change":"2024-09-21T10:28:01.631803+0000","last_active":"2024-09-21T10:30:43.101671+0000","last_peered":"2024-09-21T10:30:43.101671+0000","last_clean":"2024-09-21T10:27:59.267411+0000","last_became_active":"2024-09-21T10:28:01.425385+0000","last_became_peered":"2024-09-21T10:28:01.425385+0000","last_unstale":"2024-09-21T10:30:43.101671+0000","last_undegraded":"2024-09-21T10:30:43.101671+0000","last_fullsized":"2024-09-21T10:30:43.101671+0000","mapping_epoch":361,"log_start":"358'6100","ondisk_log_start":"358'6100","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7776,"log_dups_size":0,"ondisk_log_size":7776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:51:26.681334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103677952,"num_objects":1582,"num_object_clones":0,"num_object_copies":4746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1577,"num_objects_unfound":0,"num_objects_dirty":1582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13876,"num_write_kb":494656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":2949120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,2,4],"avail_no_missing":["6","2","4"],"object_location_counts":[{"shards":"2,4,6","objects":1582}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"82'100","reported_seq":62,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.800765+0000","last_change":"2024-09-21T10:28:00.305255+0000","last_active":"2024-09-21T10:30:32.800765+0000","last_peered":"2024-09-21T10:30:32.800765+0000","last_clean":"2024-09-21T10:30:32.800765+0000","last_became_active":"2024-09-21T10:28:00.290981+0000","last_became_peered":"2024-09-21T10:28:00.290981+0000","last_unstale":"2024-09-21T10:30:32.800765+0000","last_undegraded":"2024-09-21T10:30:32.800765+0000","last_fullsized":"2024-09-21T10:30:32.800765+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.759355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.759355+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":69,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":686,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.782933+0000","last_change":"2024-09-21T10:24:44.307344+0000","last_active":"2024-09-21T10:30:32.782933+0000","last_peered":"2024-09-21T10:30:32.782933+0000","last_clean":"2024-09-21T10:30:32.782933+0000","last_became_active":"2024-09-21T10:24:44.298936+0000","last_became_peered":"2024-09-21T10:24:44.298936+0000","last_unstale":"2024-09-21T10:30:32.782933+0000","last_undegraded":"2024-09-21T10:30:32.782933+0000","last_fullsized":"2024-09-21T10:30:32.782933+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:58:26.025877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":608,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.717103+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:30:19.717103+0000","last_peered":"2024-09-21T10:30:19.717103+0000","last_clean":"2024-09-21T10:30:19.717103+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:30:19.717103+0000","last_undegraded":"2024-09-21T10:30:19.717103+0000","last_fullsized":"2024-09-21T10:30:19.717103+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":71,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.710241+0000","last_change":"2024-09-21T10:24:44.303781+0000","last_active":"2024-09-21T10:30:19.710241+0000","last_peered":"2024-09-21T10:30:19.710241+0000","last_clean":"2024-09-21T10:30:19.710241+0000","last_became_active":"2024-09-21T10:24:44.302058+0000","last_became_peered":"2024-09-21T10:24:44.302058+0000","last_unstale":"2024-09-21T10:30:19.710241+0000","last_undegraded":"2024-09-21T10:30:19.710241+0000","last_fullsized":"2024-09-21T10:30:19.710241+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:30.371229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":71,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.708610+0000","last_change":"2024-09-21T10:24:44.303417+0000","last_active":"2024-09-21T10:30:19.708610+0000","last_peered":"2024-09-21T10:30:19.708610+0000","last_clean":"2024-09-21T10:30:19.708610+0000","last_became_active":"2024-09-21T10:24:44.303069+0000","last_became_peered":"2024-09-21T10:24:44.303069+0000","last_unstale":"2024-09-21T10:30:19.708610+0000","last_undegraded":"2024-09-21T10:30:19.708610+0000","last_fullsized":"2024-09-21T10:30:19.708610+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:30:30.966228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":81,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.708458+0000","last_change":"2024-09-21T10:24:44.321421+0000","last_active":"2024-09-21T10:30:19.708458+0000","last_peered":"2024-09-21T10:30:19.708458+0000","last_clean":"2024-09-21T10:30:19.708458+0000","last_became_active":"2024-09-21T10:24:44.297082+0000","last_became_peered":"2024-09-21T10:24:44.297082+0000","last_unstale":"2024-09-21T10:30:19.708458+0000","last_undegraded":"2024-09-21T10:30:19.708458+0000","last_fullsized":"2024-09-21T10:30:19.708458+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:35:37.704881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":520,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.907790+0000","last_change":"2024-09-21T10:08:18.470378+0000","last_active":"2024-09-21T10:30:32.907790+0000","last_peered":"2024-09-21T10:30:32.907790+0000","last_clean":"2024-09-21T10:30:32.907790+0000","last_became_active":"2024-09-21T10:08:18.469482+0000","last_became_peered":"2024-09-21T10:08:18.469482+0000","last_unstale":"2024-09-21T10:30:32.907790+0000","last_undegraded":"2024-09-21T10:30:32.907790+0000","last_fullsized":"2024-09-21T10:30:32.907790+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:35.628724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":445,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.705738+0000","last_change":"2024-09-21T10:28:00.296771+0000","last_active":"2024-09-21T10:30:19.705738+0000","last_peered":"2024-09-21T10:30:19.705738+0000","last_clean":"2024-09-21T10:30:19.705738+0000","last_became_active":"2024-09-21T10:28:00.285336+0000","last_became_peered":"2024-09-21T10:28:00.285336+0000","last_unstale":"2024-09-21T10:30:19.705738+0000","last_undegraded":"2024-09-21T10:30:19.705738+0000","last_fullsized":"2024-09-21T10:30:19.705738+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:06:50.848492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":153,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.727341+0000","last_change":"2024-09-21T10:26:41.644952+0000","last_active":"2024-09-21T10:30:19.727341+0000","last_peered":"2024-09-21T10:30:19.727341+0000","last_clean":"2024-09-21T10:30:19.727341+0000","last_became_active":"2024-09-21T10:26:41.644592+0000","last_became_peered":"2024-09-21T10:26:41.644592+0000","last_unstale":"2024-09-21T10:30:19.727341+0000","last_undegraded":"2024-09-21T10:30:19.727341+0000","last_fullsized":"2024-09-21T10:30:19.727341+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:44:44.183877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":55,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.800715+0000","last_change":"2024-09-21T10:28:00.294338+0000","last_active":"2024-09-21T10:30:32.800715+0000","last_peered":"2024-09-21T10:30:32.800715+0000","last_clean":"2024-09-21T10:30:32.800715+0000","last_became_active":"2024-09-21T10:28:00.294044+0000","last_became_peered":"2024-09-21T10:28:00.294044+0000","last_unstale":"2024-09-21T10:30:32.800715+0000","last_undegraded":"2024-09-21T10:30:32.800715+0000","last_fullsized":"2024-09-21T10:30:32.800715+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:43:30.999988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":650,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.782964+0000","last_change":"2024-09-21T10:28:00.296935+0000","last_active":"2024-09-21T10:30:32.782964+0000","last_peered":"2024-09-21T10:30:32.782964+0000","last_clean":"2024-09-21T10:30:32.782964+0000","last_became_active":"2024-09-21T10:28:00.296562+0000","last_became_peered":"2024-09-21T10:28:00.296562+0000","last_unstale":"2024-09-21T10:30:32.782964+0000","last_undegraded":"2024-09-21T10:30:32.782964+0000","last_fullsized":"2024-09-21T10:30:32.782964+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:06.863310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":55,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.800275+0000","last_change":"2024-09-21T10:28:00.294718+0000","last_active":"2024-09-21T10:30:32.800275+0000","last_peered":"2024-09-21T10:30:32.800275+0000","last_clean":"2024-09-21T10:30:32.800275+0000","last_became_active":"2024-09-21T10:28:00.294299+0000","last_became_peered":"2024-09-21T10:28:00.294299+0000","last_unstale":"2024-09-21T10:30:32.800275+0000","last_undegraded":"2024-09-21T10:30:32.800275+0000","last_fullsized":"2024-09-21T10:30:32.800275+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:41:46.881336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":345,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.716870+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:30:19.716870+0000","last_peered":"2024-09-21T10:30:19.716870+0000","last_clean":"2024-09-21T10:30:19.716870+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:30:19.716870+0000","last_undegraded":"2024-09-21T10:30:19.716870+0000","last_fullsized":"2024-09-21T10:30:19.716870+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":55,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.800186+0000","last_change":"2024-09-21T10:28:00.293872+0000","last_active":"2024-09-21T10:30:32.800186+0000","last_peered":"2024-09-21T10:30:32.800186+0000","last_clean":"2024-09-21T10:30:32.800186+0000","last_became_active":"2024-09-21T10:28:00.293320+0000","last_became_peered":"2024-09-21T10:28:00.293320+0000","last_unstale":"2024-09-21T10:30:32.800186+0000","last_undegraded":"2024-09-21T10:30:32.800186+0000","last_fullsized":"2024-09-21T10:30:32.800186+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:45:43.909307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":667,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.783035+0000","last_change":"2024-09-21T10:28:00.297304+0000","last_active":"2024-09-21T10:30:32.783035+0000","last_peered":"2024-09-21T10:30:32.783035+0000","last_clean":"2024-09-21T10:30:32.783035+0000","last_became_active":"2024-09-21T10:28:00.296631+0000","last_became_peered":"2024-09-21T10:28:00.296631+0000","last_unstale":"2024-09-21T10:30:32.783035+0000","last_undegraded":"2024-09-21T10:30:32.783035+0000","last_fullsized":"2024-09-21T10:30:32.783035+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:22:16.633581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":667,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.904805+0000","last_change":"2024-09-21T10:28:00.284924+0000","last_active":"2024-09-21T10:30:32.904805+0000","last_peered":"2024-09-21T10:30:32.904805+0000","last_clean":"2024-09-21T10:30:32.904805+0000","last_became_active":"2024-09-21T10:28:00.284633+0000","last_became_peered":"2024-09-21T10:28:00.284633+0000","last_unstale":"2024-09-21T10:30:32.904805+0000","last_undegraded":"2024-09-21T10:30:32.904805+0000","last_fullsized":"2024-09-21T10:30:32.904805+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:23:09.808151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":55,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.801585+0000","last_change":"2024-09-21T10:28:00.283403+0000","last_active":"2024-09-21T10:30:32.801585+0000","last_peered":"2024-09-21T10:30:32.801585+0000","last_clean":"2024-09-21T10:30:32.801585+0000","last_became_active":"2024-09-21T10:28:00.283089+0000","last_became_peered":"2024-09-21T10:28:00.283089+0000","last_unstale":"2024-09-21T10:30:32.801585+0000","last_undegraded":"2024-09-21T10:30:32.801585+0000","last_fullsized":"2024-09-21T10:30:32.801585+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:42.213644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":71,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.708333+0000","last_change":"2024-09-21T10:24:44.305124+0000","last_active":"2024-09-21T10:30:19.708333+0000","last_peered":"2024-09-21T10:30:19.708333+0000","last_clean":"2024-09-21T10:30:19.708333+0000","last_became_active":"2024-09-21T10:24:44.304468+0000","last_became_peered":"2024-09-21T10:24:44.304468+0000","last_unstale":"2024-09-21T10:30:19.708333+0000","last_undegraded":"2024-09-21T10:30:19.708333+0000","last_fullsized":"2024-09-21T10:30:19.708333+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:37.904764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":305,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.801877+0000","last_change":"2024-09-21T10:28:00.286646+0000","last_active":"2024-09-21T10:30:32.801877+0000","last_peered":"2024-09-21T10:30:32.801877+0000","last_clean":"2024-09-21T10:30:32.801877+0000","last_became_active":"2024-09-21T10:28:00.286300+0000","last_became_peered":"2024-09-21T10:28:00.286300+0000","last_unstale":"2024-09-21T10:30:32.801877+0000","last_undegraded":"2024-09-21T10:30:32.801877+0000","last_fullsized":"2024-09-21T10:30:32.801877+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:58.215385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":657,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.782657+0000","last_change":"2024-09-21T10:24:44.299989+0000","last_active":"2024-09-21T10:30:32.782657+0000","last_peered":"2024-09-21T10:30:32.782657+0000","last_clean":"2024-09-21T10:30:32.782657+0000","last_became_active":"2024-09-21T10:24:44.299356+0000","last_became_peered":"2024-09-21T10:24:44.299356+0000","last_unstale":"2024-09-21T10:30:32.782657+0000","last_undegraded":"2024-09-21T10:30:32.782657+0000","last_fullsized":"2024-09-21T10:30:32.782657+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:58.867008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":81,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.708514+0000","last_change":"2024-09-21T10:28:00.281942+0000","last_active":"2024-09-21T10:30:19.708514+0000","last_peered":"2024-09-21T10:30:19.708514+0000","last_clean":"2024-09-21T10:30:19.708514+0000","last_became_active":"2024-09-21T10:28:00.281600+0000","last_became_peered":"2024-09-21T10:28:00.281600+0000","last_unstale":"2024-09-21T10:30:19.708514+0000","last_undegraded":"2024-09-21T10:30:19.708514+0000","last_fullsized":"2024-09-21T10:30:19.708514+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:21.144244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":75,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.708713+0000","last_change":"2024-09-21T10:24:44.329150+0000","last_active":"2024-09-21T10:30:19.708713+0000","last_peered":"2024-09-21T10:30:19.708713+0000","last_clean":"2024-09-21T10:30:19.708713+0000","last_became_active":"2024-09-21T10:24:44.306752+0000","last_became_peered":"2024-09-21T10:24:44.306752+0000","last_unstale":"2024-09-21T10:30:19.708713+0000","last_undegraded":"2024-09-21T10:30:19.708713+0000","last_fullsized":"2024-09-21T10:30:19.708713+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:16.592859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":81,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.708391+0000","last_change":"2024-09-21T10:28:00.286009+0000","last_active":"2024-09-21T10:30:19.708391+0000","last_peered":"2024-09-21T10:30:19.708391+0000","last_clean":"2024-09-21T10:30:19.708391+0000","last_became_active":"2024-09-21T10:28:00.285327+0000","last_became_peered":"2024-09-21T10:28:00.285327+0000","last_unstale":"2024-09-21T10:30:19.708391+0000","last_undegraded":"2024-09-21T10:30:19.708391+0000","last_fullsized":"2024-09-21T10:30:19.708391+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:34.985354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":75,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.708814+0000","last_change":"2024-09-21T10:24:44.324087+0000","last_active":"2024-09-21T10:30:19.708814+0000","last_peered":"2024-09-21T10:30:19.708814+0000","last_clean":"2024-09-21T10:30:19.708814+0000","last_became_active":"2024-09-21T10:24:44.298665+0000","last_became_peered":"2024-09-21T10:24:44.298665+0000","last_unstale":"2024-09-21T10:30:19.708814+0000","last_undegraded":"2024-09-21T10:30:19.708814+0000","last_fullsized":"2024-09-21T10:30:19.708814+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:59.819340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":55,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.800637+0000","last_change":"2024-09-21T10:28:00.293054+0000","last_active":"2024-09-21T10:30:32.800637+0000","last_peered":"2024-09-21T10:30:32.800637+0000","last_clean":"2024-09-21T10:30:32.800637+0000","last_became_active":"2024-09-21T10:28:00.292723+0000","last_became_peered":"2024-09-21T10:28:00.292723+0000","last_unstale":"2024-09-21T10:30:32.800637+0000","last_undegraded":"2024-09-21T10:30:32.800637+0000","last_fullsized":"2024-09-21T10:30:32.800637+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:01:45.933570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":55,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.801198+0000","last_change":"2024-09-21T10:28:00.285515+0000","last_active":"2024-09-21T10:30:32.801198+0000","last_peered":"2024-09-21T10:30:32.801198+0000","last_clean":"2024-09-21T10:30:32.801198+0000","last_became_active":"2024-09-21T10:28:00.285203+0000","last_became_peered":"2024-09-21T10:28:00.285203+0000","last_unstale":"2024-09-21T10:30:32.801198+0000","last_undegraded":"2024-09-21T10:30:32.801198+0000","last_fullsized":"2024-09-21T10:30:32.801198+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:41.982228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":71,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T10:30:19.708203+0000","last_change":"2024-09-21T10:24:44.298820+0000","last_active":"2024-09-21T10:30:19.708203+0000","last_peered":"2024-09-21T10:30:19.708203+0000","last_clean":"2024-09-21T10:30:19.708203+0000","last_became_active":"2024-09-21T10:24:44.298478+0000","last_became_peered":"2024-09-21T10:24:44.298478+0000","last_unstale":"2024-09-21T10:30:19.708203+0000","last_undegraded":"2024-09-21T10:30:19.708203+0000","last_fullsized":"2024-09-21T10:30:19.708203+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:59.567078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":642,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-21T10:30:32.782877+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:30:32.782877+0000","last_peered":"2024-09-21T10:30:32.782877+0000","last_clean":"2024-09-21T10:30:32.782877+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:30:32.782877+0000","last_undegraded":"2024-09-21T10:30:32.782877+0000","last_fullsized":"2024-09-21T10:30:32.782877+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":1605959704,"num_objects":24506,"num_object_clones":0,"num_object_copies":73518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17827,"num_objects_misplaced":3880,"num_objects_unfound":0,"num_objects_dirty":24506,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":225540,"num_write_kb":8001409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2644,"num_bytes_recovered":173277184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6022508544,"data_stored":6022496328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127240,"ondisk_log_size":127240,"up":48,"acting":45,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":10252,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5983,"num_read_kb":13006,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":316358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":69,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":644038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1100,"ondisk_log_size":1100,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397991,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1766720,"kb_used_data":861176,"kb_used_omap":1,"kb_used_meta":905534,"kb_avail":91978432,"statfs":{"total":95995035648,"available":94185914368,"internally_reserved":0,"allocated":881844224,"data_stored":879648605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":927267160},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594219,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1614652,"kb_used_data":709108,"kb_used_omap":1,"kb_used_meta":905470,"kb_avail":92130500,"statfs":{"total":95995035648,"available":94341632000,"internally_reserved":0,"allocated":726126592,"data_stored":723934966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":927201624},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790449,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1430448,"kb_used_data":526376,"kb_used_omap":1,"kb_used_meta":904062,"kb_avail":92314704,"statfs":{"total":95995035648,"available":94530256896,"internally_reserved":0,"allocated":539009024,"data_stored":536825717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":925759832},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986678,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1961856,"kb_used_data":1363640,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":91783296,"statfs":{"total":95995035648,"available":93986095104,"internally_reserved":0,"allocated":1396367360,"data_stored":1394171687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":101,"seq":433791697240,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":825452,"kb_used_data":141028,"kb_used_omap":5,"kb_used_meta":684410,"kb_avail":92919700,"statfs":{"total":95995035648,"available":95149772800,"internally_reserved":0,"allocated":144412672,"data_stored":142233219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":700836624},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860767,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1640368,"kb_used_data":666984,"kb_used_omap":8,"kb_used_meta":973367,"kb_avail":92104784,"statfs":{"total":95995035648,"available":94315298816,"internally_reserved":0,"allocated":682991616,"data_stored":680808589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":996728684},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024292,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1575584,"kb_used_data":758744,"kb_used_omap":7,"kb_used_meta":816824,"kb_avail":92169568,"statfs":{"total":95995035648,"available":94381637632,"internally_reserved":0,"allocated":776953856,"data_stored":774785454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":836428338},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187828,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1987568,"kb_used_data":958376,"kb_used_omap":8,"kb_used_meta":1028983,"kb_avail":91757584,"statfs":{"total":95995035648,"available":93959766016,"internally_reserved":0,"allocated":981377024,"data_stored":979193465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":1053679601},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":962007040,"data_stored":962002968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":768933888,"data_stored":768933888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":669712384,"data_stored":669712384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":138674176,"data_stored":138674176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1385172992,"data_stored":1385168920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":514850816,"data_stored":514850816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":714346496,"data_stored":714342424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":868810752,"data_stored":868810752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:30:49.500 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T10:30:49.574 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:49.672 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:49.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:49.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:49.774 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:49.810 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:49.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:49.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:49.912 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:49.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:49.972 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:49.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:49.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:49 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:30:49.999 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:49 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:30:50.074 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:50.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:50.109 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:50.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:50.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:49 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:30:50.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:50.246 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:50.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:50.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:50.348 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:50.375 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:50.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:50.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:50.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:50.515 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:50.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:50.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:50.616 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:50.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:30:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:30:50.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:30:50.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:50.654 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:50.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:50.755 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:30:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:30:50] "GET /metrics HTTP/1.1" 200 36610 "" "Prometheus/2.51.0" 2024-09-21T10:30:50.755 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:50.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:50.784 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:50.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:50.801 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:30:50.801 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:30:50.885 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:50.915 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:50.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:50.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:51.018 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:51.039 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:51.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:51.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:51.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:50 smithi098 ceph-mon[97580]: pgmap v1329: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 779 op/s; 17827/66144 objects degraded (26.952%); 3429/66144 objects misplaced (5.184%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:30:51.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:50 smithi098 ceph-mon[100914]: pgmap v1329: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 779 op/s; 17827/66144 objects degraded (26.952%); 3429/66144 objects misplaced (5.184%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:30:51.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:50 smithi196 ceph-mon[84784]: pgmap v1329: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 779 op/s; 17827/66144 objects degraded (26.952%); 3429/66144 objects misplaced (5.184%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:30:51.141 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:51.180 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:51.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:51.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:51.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:51.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:51.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:51.417 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:51.469 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:51.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:51.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:51.571 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:51.597 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:51.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:51.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:51.699 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:51.738 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:51.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:51.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:51.840 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:51.868 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:51.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:51.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:51.970 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:52.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:52.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:52.109 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:52.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:51 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 17827/66144 objects degraded (26.952%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:52.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:52.140 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:52.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:52.194 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:51 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 17827/66144 objects degraded (26.952%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:52.194 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:51 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 17827/66144 objects degraded (26.952%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:52.235 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:30:52.240 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:52.277 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:52.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:52.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:52.379 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:52.406 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:52.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:52.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:52.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:52.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:52.549 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:52.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:52.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:52.691 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:52.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:52.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:52.723 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:52.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:52.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:52.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:52.947 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:52.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:52.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:53.023 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:52 smithi098 ceph-mon[100914]: pgmap v1330: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 17952/62664 objects degraded (28.648%); 3200/62664 objects misplaced (5.107%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:30:53.023 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:53.023 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:52 smithi098 ceph-mon[97580]: pgmap v1330: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 17952/62664 objects degraded (28.648%); 3200/62664 objects misplaced (5.107%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:30:53.023 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:53.089 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:53.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:53.133 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:53.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:52 smithi196 ceph-mon[84784]: pgmap v1330: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 17952/62664 objects degraded (28.648%); 3200/62664 objects misplaced (5.107%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:30:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:53.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:53.259 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:53.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:53.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:53.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:53.391 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:53.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:53.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:53.492 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:53.534 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:53.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:53.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:53.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:53.675 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:53.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:53.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:53.777 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:53.804 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:53.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:53.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:53.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:53.944 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:53.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:53.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:54.046 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:54.076 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:54.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:54.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:54.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:54.210 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:54.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:54.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:54.311 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:54.346 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:54.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:54.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:54.447 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:54.487 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:54.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:54.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:54.588 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:54.616 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:54.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:54.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:54.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:54.795 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:54.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:54.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:54.897 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:54.933 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:54.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:54.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:55.035 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:55.062 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:55.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:55.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:55.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:54 smithi196 ceph-mon[84784]: pgmap v1331: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 805 op/s; 17952/55284 objects degraded (32.472%); 2747/55284 objects misplaced (4.969%); 1.1 MiB/s, 16 objects/s recovering 2024-09-21T10:30:55.167 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:55.193 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:55.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:55.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:55.209 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:54 smithi098 ceph-mon[97580]: pgmap v1331: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 805 op/s; 17952/55284 objects degraded (32.472%); 2747/55284 objects misplaced (4.969%); 1.1 MiB/s, 16 objects/s recovering 2024-09-21T10:30:55.210 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:54 smithi098 ceph-mon[100914]: pgmap v1331: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 805 op/s; 17952/55284 objects degraded (32.472%); 2747/55284 objects misplaced (4.969%); 1.1 MiB/s, 16 objects/s recovering 2024-09-21T10:30:55.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:55.322 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:55.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:55.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:55.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:55.462 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:55.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:55.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:55.565 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:55.598 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:55.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:55.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:55.699 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:55.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:55.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:55.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:55.828 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:55.866 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:55.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:55.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:55.967 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:55.995 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:55.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:55.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:56.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:56.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:56.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:56.238 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:56.270 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:56.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:56.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:56.396 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:56.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:56.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:56.531 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:56.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:56.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:56.583 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:56 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:30:56.583 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:56 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:30:56.584 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:56 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 17952/55284 objects degraded (32.472%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:56.584 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:56 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:30:56.584 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:56 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:30:56.584 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:56 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 17952/55284 objects degraded (32.472%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:56.633 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:56.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:56 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:30:56.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:56 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:30:56.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:56 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 17952/55284 objects degraded (32.472%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:56.668 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:56.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:56.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:56.770 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:56.807 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:56.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:56.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:56.909 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:56.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:56.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:57.050 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:57.090 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:57.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:57.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:57.191 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:57.220 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:57.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:57.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:57.321 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:57.346 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:57.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:57.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:57.448 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:57.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:57.483 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:57.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:57.583 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:57.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:57 smithi098 ceph-mon[97580]: pgmap v1332: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 698 op/s; 18055/51873 objects degraded (34.806%); 2525/51873 objects misplaced (4.868%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:30:57.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:57 smithi098 ceph-mon[100914]: pgmap v1332: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 698 op/s; 18055/51873 objects degraded (34.806%); 2525/51873 objects misplaced (4.868%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:30:57.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:57.615 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:57.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:57.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:57 smithi196 ceph-mon[84784]: pgmap v1332: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 698 op/s; 18055/51873 objects degraded (34.806%); 2525/51873 objects misplaced (4.868%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:30:57.716 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:57.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:57.761 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:57.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:57.862 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:57.889 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:57.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:57.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:57.990 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:58.018 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:58.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:58.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:58.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:58.154 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:58.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:58.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:58.256 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:58.282 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:58.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:58.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:30:58.384 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:58.412 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:58.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:58.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:58.434 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:58 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3912907656' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:30:58.434 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:58 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:30:58.434 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:58 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3912907656' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:30:58.434 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:58 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:30:58.514 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:58.541 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:58.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:58.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:58.622 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:30:58.622 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:30:58.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:58 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3912907656' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:30:58.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:58 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:30:58.643 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:58.665 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:30:58.665 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:30:58.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:58.674 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:58.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:58.698 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:30:58.698 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:30:58.735 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:30:58.735 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:30:58.763 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:30:58.763 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:30:58.775 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:58.797 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:30:58.797 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:30:58.801 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:58.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:58.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:58.826 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:30:58.826 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:30:58.853 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:30:58.854 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:30:58.903 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:58.929 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:58.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:58.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:59.031 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:59.056 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:59.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:59.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:59.158 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:59.186 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:59.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:59.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:59.287 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:59.325 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:59.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:59.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:59.427 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:59.458 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:59.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:59.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:59.510 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:30:59.510 INFO:teuthology.orchestra.run.smithi098.stdout:{"fsid":"26a15512-77fe-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18055/51219 objects degraded (35.251%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1957,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":383,"num_osds":8,"num_up_osds":8,"osd_up_since":1726912933,"num_in_osds":8,"osd_in_since":1726914479,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":94},{"state_name":"active+remapped+backfill_wait","count":2},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":99,"num_pools":5,"num_objects":17073,"data_bytes":1121577355,"bytes_used":11731820544,"bytes_avail":756228464640,"bytes_total":767960285184,"degraded_objects":18055,"degraded_total":51219,"degraded_ratio":0.35250590601144105,"misplaced_objects":2525,"misplaced_total":51219,"misplaced_ratio":0.049298112028739335,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":1217473,"recovering_keys_per_sec":0,"num_objects_recovered":223,"num_bytes_recovered":14614528,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":621},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi098.stcwsu","status":"up:active","gid":14640}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.98:8443/","prometheus":"http://172.21.15.98:9283/"}},"servicemap":{"epoch":30,"modified":"2024-09-21T10:29:21.594654+0000","services":{}},"progress_events":{"0d290b6d-ee2f-4f68-8c75-373d103609de":{"message":"Global Recovery Event (0s)\n [............................] ","progress":0,"add_to_ceph_s":true},"6de3e7dc-7aed-4a0c-b700-51217030d18b":{"message":"Upgrade to 19.3.0-5092-g62da54b4 (4m)\n [============................] (remaining: 5m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-09-21T10:30:59.560 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:59.592 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:59.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:59.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:59.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:59 smithi098 ceph-mon[97580]: pgmap v1333: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 621 op/s; 18055/51219 objects degraded (35.251%); 2525/51219 objects misplaced (4.930%); 1.2 MiB/s, 18 objects/s recovering 2024-09-21T10:30:59.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:59 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:30:59.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:59 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:30:59.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:59 smithi098 ceph-mon[97580]: osdmap e383: 8 total, 8 up, 8 in 2024-09-21T10:30:59.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:30:59 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3612722496' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:30:59.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:59 smithi098 ceph-mon[100914]: pgmap v1333: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 621 op/s; 18055/51219 objects degraded (35.251%); 2525/51219 objects misplaced (4.930%); 1.2 MiB/s, 18 objects/s recovering 2024-09-21T10:30:59.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:59 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:30:59.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:59 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:30:59.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:59 smithi098 ceph-mon[100914]: osdmap e383: 8 total, 8 up, 8 in 2024-09-21T10:30:59.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:30:59 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3612722496' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:30:59.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:59 smithi196 ceph-mon[84784]: pgmap v1333: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 1.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 621 op/s; 18055/51219 objects degraded (35.251%); 2525/51219 objects misplaced (4.930%); 1.2 MiB/s, 18 objects/s recovering 2024-09-21T10:30:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:59 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:30:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:59 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:30:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:59 smithi196 ceph-mon[84784]: osdmap e383: 8 total, 8 up, 8 in 2024-09-21T10:30:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:30:59 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3612722496' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:30:59.694 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:59.723 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:59.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:59.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:59.825 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:59.851 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:30:59.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:59.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:59.953 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:59.985 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:30:59.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:59.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:00.087 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:00.133 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:00.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:00.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:00.235 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:00.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:00.282 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:00.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:00.383 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:00.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:00.417 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:00.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:00.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:00 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/938386273' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:31:00.504 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:00 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/938386273' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:31:00.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:00.551 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:00.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:00.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:00.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:00 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/938386273' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:31:00.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:31:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:31:00.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:31:00.653 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:00.693 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:00.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:00.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:00.795 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:00.820 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:00.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:00.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:00.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:31:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:31:00] "GET /metrics HTTP/1.1" 200 36611 "" "Prometheus/2.51.0" 2024-09-21T10:31:00.921 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:00.950 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:00.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:00.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:00.986 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-21T10:31:00.986 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:31:00.992 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:01.052 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:01.084 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:01.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:01.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:01.186 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:01.217 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:01.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:01.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:01.319 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:01.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:01 smithi098 ceph-mon[97580]: pgmap v1335: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 926 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 726 op/s; 18055/44340 objects degraded (40.719%); 2043/44340 objects misplaced (4.608%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:31:01.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:01 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 18055/44340 objects degraded (40.719%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:01.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:01.363 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:01.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:01.464 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:01.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:01.501 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:01.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:01.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:01 smithi098 ceph-mon[100914]: pgmap v1335: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 926 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 726 op/s; 18055/44340 objects degraded (40.719%); 2043/44340 objects misplaced (4.608%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:31:01.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:01 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 18055/44340 objects degraded (40.719%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:01.602 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:01.634 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:01.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:01.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:01.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:01 smithi196 ceph-mon[84784]: pgmap v1335: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 926 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 726 op/s; 18055/44340 objects degraded (40.719%); 2043/44340 objects misplaced (4.608%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:31:01.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:01 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 18055/44340 objects degraded (40.719%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:01.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:01.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:01.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:01.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:31:01.874 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:01.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:01.920 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:01.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:02.021 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:02.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:02.071 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:02.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:02.172 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:02.195 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:02.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:02.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:02.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:02.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:02.330 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:02.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:02.431 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:02.461 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:02.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:02.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:02.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:02.597 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:02.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:02.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:02.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:02.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:02.737 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:02.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:02.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:02.864 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:02.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:02.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:02.966 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:02.991 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:02.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:02.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:03.092 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:03.120 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:03.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:03.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:03.221 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:03.251 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:03.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:03.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:03.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:03.387 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:03.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:03.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:03.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:03.523 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:03.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:03.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:03.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:03 smithi098 ceph-mon[97580]: pgmap v1336: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 858 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 18154/41046 objects degraded (44.228%); 1823/41046 objects misplaced (4.441%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:31:03.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:03 smithi098 ceph-mon[100914]: pgmap v1336: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 858 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 18154/41046 objects degraded (44.228%); 1823/41046 objects misplaced (4.441%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:31:03.626 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:03 smithi196 ceph-mon[84784]: pgmap v1336: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 858 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 18154/41046 objects degraded (44.228%); 1823/41046 objects misplaced (4.441%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:31:03.668 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:03.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:03.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:03.769 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:03.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:03.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:03.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:03.953 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:03.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:03.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:04.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:04.112 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:04.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:04.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:04.215 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:04.242 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:04.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:04.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:04.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:04.375 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:04.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:04.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:04.477 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:04.504 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:04.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:04.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:04.512 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:04.605 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:04.659 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:04.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:04.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:04.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:31:04.761 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:04.838 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:04.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:04.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:04.940 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:05.001 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:05.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:05.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:05.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:05.146 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:05.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:05.214 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:05.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:05.315 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:05.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:05.343 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:05.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:05.444 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:05.472 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:05.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:05.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:05.574 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:05.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:05 smithi098 ceph-mon[97580]: pgmap v1337: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 701 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 725 op/s; 18154/33516 objects degraded (54.165%); 1339/33516 objects misplaced (3.995%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:31:05.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:05 smithi098 ceph-mon[100914]: pgmap v1337: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 701 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 725 op/s; 18154/33516 objects degraded (54.165%); 1339/33516 objects misplaced (3.995%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:31:05.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:05.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:05.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:05 smithi196 ceph-mon[84784]: pgmap v1337: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 701 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 725 op/s; 18154/33516 objects degraded (54.165%); 1339/33516 objects misplaced (3.995%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:31:05.733 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:05.767 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:05.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:05.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:05.868 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:05.902 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:05.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:05.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:06.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:06.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:06.044 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:06.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:06.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:06.182 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:06.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:06.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:06.284 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:06.312 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:06.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:06.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:06.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:06.447 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:06.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:06.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:06.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:06.594 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:06.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:06.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:06.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:06 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 18154/33516 objects degraded (54.165%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:06.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:06 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 18154/33516 objects degraded (54.165%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:06.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:06 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 18154/33516 objects degraded (54.165%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:06.696 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:06.737 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:06.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:06.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:06.839 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:06.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:06.889 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:06.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:06.990 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:07.015 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:07.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:07.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:07.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:07.154 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:07.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:07.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:07.255 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:07.256 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:07.300 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:07.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:07.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:07.402 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:07.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:07.442 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:07.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:07.543 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:07.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:07.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:07.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:07.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:07 smithi196 ceph-mon[84784]: pgmap v1338: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 621 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 18126/29667 objects degraded (61.098%); 1245/29667 objects misplaced (4.197%); 1.2 MiB/s, 24 objects/s recovering 2024-09-21T10:31:07.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:31:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:07.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:07.715 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:07.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:07.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:07.817 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:07.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:07 smithi098 ceph-mon[97580]: pgmap v1338: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 621 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 18126/29667 objects degraded (61.098%); 1245/29667 objects misplaced (4.197%); 1.2 MiB/s, 24 objects/s recovering 2024-09-21T10:31:07.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:31:07.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:07.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:07 smithi098 ceph-mon[100914]: pgmap v1338: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 621 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 18126/29667 objects degraded (61.098%); 1245/29667 objects misplaced (4.197%); 1.2 MiB/s, 24 objects/s recovering 2024-09-21T10:31:07.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:31:07.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:07.854 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:07.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:07.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:07.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:07.987 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:07.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:07.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:08.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:08.143 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:08.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:08.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:08.244 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:08.302 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:08.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:08.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:08.403 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:08.454 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:08.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:08.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:08.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:08.597 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:08.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:08.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:08.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:08.733 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:08.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:08.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:08.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:08.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:08.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:08.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:09.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:09.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:09.119 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:09.148 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:09.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:09.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:09.250 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:09.290 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:09.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:09.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:09.392 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:09.430 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:09.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:09.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:09.531 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:09.569 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:09.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:09.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:09 smithi196 ceph-mon[84784]: pgmap v1339: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 621 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 18126/29667 objects degraded (61.098%); 1245/29667 objects misplaced (4.197%); 1.2 MiB/s, 24 objects/s recovering 2024-09-21T10:31:09.671 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:09.705 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:09.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:09.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:09.807 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:09.836 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:09.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:09.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:09.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:09 smithi098 ceph-mon[97580]: pgmap v1339: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 621 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 18126/29667 objects degraded (61.098%); 1245/29667 objects misplaced (4.197%); 1.2 MiB/s, 24 objects/s recovering 2024-09-21T10:31:09.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:09 smithi098 ceph-mon[100914]: pgmap v1339: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 621 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 18126/29667 objects degraded (61.098%); 1245/29667 objects misplaced (4.197%); 1.2 MiB/s, 24 objects/s recovering 2024-09-21T10:31:09.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:09.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:09.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:10.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:10.139 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:10.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:10.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:10.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:10.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:10.284 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:10.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:10.385 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:10.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:31:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:31:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:31:10.426 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:10.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:10.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:10.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:10.572 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:10.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:10.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:10.674 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:10.738 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:10.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:10.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:10.840 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:10.866 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:10.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:10.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:10.968 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:11.006 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:11.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:11.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:11.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:31:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:31:10] "GET /metrics HTTP/1.1" 200 36613 "" "Prometheus/2.51.0" 2024-09-21T10:31:11.108 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:11.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:11.144 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:11.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:11.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:11.277 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:11.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:11.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:11.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:11.403 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:11.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:11.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:11.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:11.552 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:11.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:11.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:11 smithi196 ceph-mon[84784]: pgmap v1340: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 481 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 18126/22974 objects degraded (78.898%); 857/22974 objects misplaced (3.730%); 1.0 MiB/s, 21 objects/s recovering 2024-09-21T10:31:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:11 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 18126/22974 objects degraded (78.898%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:11.653 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:11.700 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:11.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:11.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:11.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:11.805 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:11 smithi098 ceph-mon[97580]: pgmap v1340: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 481 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 18126/22974 objects degraded (78.898%); 857/22974 objects misplaced (3.730%); 1.0 MiB/s, 21 objects/s recovering 2024-09-21T10:31:11.805 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:11 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 18126/22974 objects degraded (78.898%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:11.805 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:11 smithi098 ceph-mon[100914]: pgmap v1340: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 481 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 18126/22974 objects degraded (78.898%); 857/22974 objects misplaced (3.730%); 1.0 MiB/s, 21 objects/s recovering 2024-09-21T10:31:11.805 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:11 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 18126/22974 objects degraded (78.898%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:11.832 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:11.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:11.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:11.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:11.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:11.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:12.073 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:12.085 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:31:12.085 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:31:12.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:12.086 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:245802 10.0 e383/348 10:0d6f18cb:::benchmark_data_smithi098_281896_object120777:head v 383'15257, mlcod=383'15255)", 2024-09-21T10:31:12.086 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:31:12.080311+0000", 2024-09-21T10:31:12.086 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0024959779999999998, 2024-09-21T10:31:12.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025263360000000001, 2024-09-21T10:31:12.086 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:31:12.086 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:31:12.086 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:31:12.086 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:31:12.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:12.086 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:31:12.086 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.080311+0000", 2024-09-21T10:31:12.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:12.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:12.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:12.087 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:31:12.087 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.080311+0000", 2024-09-21T10:31:12.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:12.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:12.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:12.087 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:31:12.087 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.080312+0000", 2024-09-21T10:31:12.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.95e-07 2024-09-21T10:31:12.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:12.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:12.087 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:31:12.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.080319+0000", 2024-09-21T10:31:12.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0849999999999999e-06 2024-09-21T10:31:12.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:12.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:12.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:31:12.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.080333+0000", 2024-09-21T10:31:12.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4751e-05 2024-09-21T10:31:12.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:12.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:12.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:31:12.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.080341+0000", 2024-09-21T10:31:12.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3420000000000002e-06 2024-09-21T10:31:12.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:12.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:12.089 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:12.089 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.080498+0000", 2024-09-21T10:31:12.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000157267 2024-09-21T10:31:12.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:12.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:12.089 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:31:12.089 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.080544+0000", 2024-09-21T10:31:12.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5847000000000003e-05 2024-09-21T10:31:12.089 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:12.089 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:31:12.089 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:12.090 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:12.090 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-21T10:31:12.090 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 1 2024-09-21T10:31:12.090 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:31:12.098 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:12.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:12.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:12.200 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:12.241 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:12.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:12.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:12.343 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:12.375 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-21T10:31:12.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:12.387 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:12.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:12.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:12.561 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:12.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:12.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:12.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:12 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/268165206' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:31:12.663 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:12.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:12.698 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:12.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:12.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:12 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/268165206' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:31:12.753 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:12 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/268165206' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:31:12.798 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:12.834 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:12.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:12.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:12.936 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:12.972 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:12.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:12.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:13.073 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:13.105 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:13.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:13.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:13.207 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:13.241 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:13.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:13.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:13.343 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:13.367 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:13.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:13.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:13.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:13 smithi098 ceph-mon[97580]: pgmap v1341: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 399 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 18001/19029 objects degraded (94.598%); 857/19029 objects misplaced (4.504%); 992 KiB/s, 30 objects/s recovering 2024-09-21T10:31:13.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:13 smithi098 ceph-mon[97580]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:31:13.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:13 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/268165206' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:31:13.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:13 smithi098 ceph-mon[97580]: osdmap e384: 8 total, 8 up, 8 in 2024-09-21T10:31:13.467 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:13 smithi098 ceph-mon[100914]: pgmap v1341: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 399 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 18001/19029 objects degraded (94.598%); 857/19029 objects misplaced (4.504%); 992 KiB/s, 30 objects/s recovering 2024-09-21T10:31:13.467 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:13 smithi098 ceph-mon[100914]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:31:13.467 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:13 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/268165206' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:31:13.467 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:13 smithi098 ceph-mon[100914]: osdmap e384: 8 total, 8 up, 8 in 2024-09-21T10:31:13.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:13.496 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:13.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:13.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:13.597 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:13.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:13.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:13.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:13 smithi196 ceph-mon[84784]: pgmap v1341: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 399 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 18001/19029 objects degraded (94.598%); 857/19029 objects misplaced (4.504%); 992 KiB/s, 30 objects/s recovering 2024-09-21T10:31:13.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:13 smithi196 ceph-mon[84784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:31:13.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:13 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/268165206' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:31:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:13 smithi196 ceph-mon[84784]: osdmap e384: 8 total, 8 up, 8 in 2024-09-21T10:31:13.694 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:31:13.695 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:31:13.737 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:13.782 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:13.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:13.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:13.884 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:13.910 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:13.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:13.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:13.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:31:14.011 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:14.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:14.060 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:14.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:14.161 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:14.216 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:14.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:14.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:14.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:14.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:14.358 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:14.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:14.458 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:14.487 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:14.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:14.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:14.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:14.623 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:14.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:14.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:14.660 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1343,"stamp":"2024-09-21T10:31:13.645877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280553867,"num_objects":4240,"num_object_clones":0,"num_object_copies":12720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18001,"num_objects_misplaced":416,"num_objects_unfound":0,"num_objects_dirty":4240,"num_whiteouts":0,"num_read":6012,"num_read_kb":13076,"num_write":246754,"num_write_kb":8014180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3353,"num_bytes_recovered":200735698,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129265,"ondisk_log_size":129265,"up":297,"acting":294,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":302,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9148128,"kb_used_data":2310360,"kb_used_omap":35,"kb_used_meta":6837468,"kb_avail":740813088,"statfs":{"total":767960285184,"available":758592602112,"internally_reserved":0,"allocated":2365808640,"data_stored":2348234602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36353,"internal_metadata":7001567743},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,4,6,6,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-454295552,"num_objects":-6932,"num_object_clones":0,"num_object_copies":-20796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":-923,"num_objects_unfound":0,"num_objects_dirty":-6932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6932,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":4259840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.011140"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":57,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.353640+0000","last_change":"2024-09-21T10:28:00.289895+0000","last_active":"2024-09-21T10:30:58.353640+0000","last_peered":"2024-09-21T10:30:58.353640+0000","last_clean":"2024-09-21T10:30:58.353640+0000","last_became_active":"2024-09-21T10:28:00.289622+0000","last_became_peered":"2024-09-21T10:28:00.289622+0000","last_unstale":"2024-09-21T10:30:58.353640+0000","last_undegraded":"2024-09-21T10:30:58.353640+0000","last_fullsized":"2024-09-21T10:30:58.353640+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:35:31.514813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":414,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.349137+0000","last_change":"2024-09-21T10:24:44.305198+0000","last_active":"2024-09-21T10:30:58.349137+0000","last_peered":"2024-09-21T10:30:58.349137+0000","last_clean":"2024-09-21T10:30:58.349137+0000","last_became_active":"2024-09-21T10:24:44.297653+0000","last_became_peered":"2024-09-21T10:24:44.297653+0000","last_unstale":"2024-09-21T10:30:58.349137+0000","last_undegraded":"2024-09-21T10:30:58.349137+0000","last_fullsized":"2024-09-21T10:30:58.349137+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:19.320833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":251,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.332541+0000","last_change":"2024-09-21T10:28:00.294917+0000","last_active":"2024-09-21T10:30:58.332541+0000","last_peered":"2024-09-21T10:30:58.332541+0000","last_clean":"2024-09-21T10:30:58.332541+0000","last_became_active":"2024-09-21T10:28:00.294134+0000","last_became_peered":"2024-09-21T10:28:00.294134+0000","last_unstale":"2024-09-21T10:30:58.332541+0000","last_undegraded":"2024-09-21T10:30:58.332541+0000","last_fullsized":"2024-09-21T10:30:58.332541+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:29:05.780212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":200,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.349027+0000","last_change":"2024-09-21T10:28:00.287636+0000","last_active":"2024-09-21T10:30:58.349027+0000","last_peered":"2024-09-21T10:30:58.349027+0000","last_clean":"2024-09-21T10:30:58.349027+0000","last_became_active":"2024-09-21T10:28:00.287308+0000","last_became_peered":"2024-09-21T10:28:00.287308+0000","last_unstale":"2024-09-21T10:30:58.349027+0000","last_undegraded":"2024-09-21T10:30:58.349027+0000","last_fullsized":"2024-09-21T10:30:58.349027+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:41:46.132365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":652,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.343925+0000","last_change":"2024-09-21T10:28:00.297783+0000","last_active":"2024-09-21T10:30:58.343925+0000","last_peered":"2024-09-21T10:30:58.343925+0000","last_clean":"2024-09-21T10:30:58.343925+0000","last_became_active":"2024-09-21T10:28:00.297024+0000","last_became_peered":"2024-09-21T10:28:00.297024+0000","last_unstale":"2024-09-21T10:30:58.343925+0000","last_undegraded":"2024-09-21T10:30:58.343925+0000","last_fullsized":"2024-09-21T10:30:58.343925+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:24.382523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":73,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347441+0000","last_change":"2024-09-21T10:24:44.304399+0000","last_active":"2024-09-21T10:30:58.347441+0000","last_peered":"2024-09-21T10:30:58.347441+0000","last_clean":"2024-09-21T10:30:58.347441+0000","last_became_active":"2024-09-21T10:24:44.303437+0000","last_became_peered":"2024-09-21T10:24:44.303437+0000","last_unstale":"2024-09-21T10:30:58.347441+0000","last_undegraded":"2024-09-21T10:30:58.347441+0000","last_fullsized":"2024-09-21T10:30:58.347441+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:32:56.638481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":57,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.346007+0000","last_change":"2024-09-21T10:28:00.293505+0000","last_active":"2024-09-21T10:30:58.346007+0000","last_peered":"2024-09-21T10:30:58.346007+0000","last_clean":"2024-09-21T10:30:58.346007+0000","last_became_active":"2024-09-21T10:28:00.293173+0000","last_became_peered":"2024-09-21T10:28:00.293173+0000","last_unstale":"2024-09-21T10:30:58.346007+0000","last_undegraded":"2024-09-21T10:30:58.346007+0000","last_fullsized":"2024-09-21T10:30:58.346007+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:16.878847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":356,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.345680+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:30:58.345680+0000","last_peered":"2024-09-21T10:30:58.345680+0000","last_clean":"2024-09-21T10:30:58.345680+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:30:58.345680+0000","last_undegraded":"2024-09-21T10:30:58.345680+0000","last_fullsized":"2024-09-21T10:30:58.345680+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"383'15301","reported_seq":15384,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:31:09.878319+0000","last_change":"2024-09-21T10:26:36.989699+0000","last_active":"2024-09-21T10:31:09.878319+0000","last_peered":"2024-09-21T10:31:09.878319+0000","last_clean":"2024-09-21T10:31:09.878319+0000","last_became_active":"2024-09-21T10:26:36.989318+0000","last_became_peered":"2024-09-21T10:26:36.989318+0000","last_unstale":"2024-09-21T10:31:09.878319+0000","last_undegraded":"2024-09-21T10:31:09.878319+0000","last_fullsized":"2024-09-21T10:31:09.878319+0000","mapping_epoch":348,"log_start":"366'7900","ondisk_log_start":"366'7900","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7401,"log_dups_size":0,"ondisk_log_size":7401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:54:53.567344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23396352,"num_objects":357,"num_object_clones":0,"num_object_copies":1071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15301,"num_write_kb":501056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":405,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.349927+0000","last_change":"2024-09-21T10:24:44.325196+0000","last_active":"2024-09-21T10:30:58.349927+0000","last_peered":"2024-09-21T10:30:58.349927+0000","last_clean":"2024-09-21T10:30:58.349927+0000","last_became_active":"2024-09-21T10:24:44.299614+0000","last_became_peered":"2024-09-21T10:24:44.299614+0000","last_unstale":"2024-09-21T10:30:58.349927+0000","last_undegraded":"2024-09-21T10:30:58.349927+0000","last_fullsized":"2024-09-21T10:30:58.349927+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:14.558663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":137,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.332428+0000","last_change":"2024-09-21T10:26:41.305571+0000","last_active":"2024-09-21T10:30:58.332428+0000","last_peered":"2024-09-21T10:30:58.332428+0000","last_clean":"2024-09-21T10:30:58.332428+0000","last_became_active":"2024-09-21T10:26:41.305135+0000","last_became_peered":"2024-09-21T10:26:41.305135+0000","last_unstale":"2024-09-21T10:30:58.332428+0000","last_undegraded":"2024-09-21T10:30:58.332428+0000","last_fullsized":"2024-09-21T10:30:58.332428+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:49:55.452467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"384'15231","reported_seq":15286,"reported_epoch":384,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:31:13.133732+0000","last_change":"2024-09-21T10:28:01.438839+0000","last_active":"2024-09-21T10:31:13.133732+0000","last_peered":"2024-09-21T10:31:13.133732+0000","last_clean":"2024-09-21T10:27:59.232048+0000","last_became_active":"2024-09-21T10:28:01.433549+0000","last_became_peered":"2024-09-21T10:28:01.433549+0000","last_unstale":"2024-09-21T10:31:13.133732+0000","last_undegraded":"2024-09-21T10:31:13.133732+0000","last_fullsized":"2024-09-21T10:31:13.133732+0000","mapping_epoch":361,"log_start":"365'7500","ondisk_log_start":"365'7500","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7731,"log_dups_size":0,"ondisk_log_size":7731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:58.936666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13565952,"num_objects":207,"num_object_clones":0,"num_object_copies":621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":207,"num_objects_unfound":0,"num_objects_dirty":207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15231,"num_write_kb":494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,5,2],"avail_no_missing":["6","2","5"],"object_location_counts":[{"shards":"2,5,6","objects":207}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":406,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.349580+0000","last_change":"2024-09-21T10:24:44.323182+0000","last_active":"2024-09-21T10:30:58.349580+0000","last_peered":"2024-09-21T10:30:58.349580+0000","last_clean":"2024-09-21T10:30:58.349580+0000","last_became_active":"2024-09-21T10:24:44.299698+0000","last_became_peered":"2024-09-21T10:24:44.299698+0000","last_unstale":"2024-09-21T10:30:58.349580+0000","last_undegraded":"2024-09-21T10:30:58.349580+0000","last_fullsized":"2024-09-21T10:30:58.349580+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:56.571057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":83,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347651+0000","last_change":"2024-09-21T10:28:00.281253+0000","last_active":"2024-09-21T10:30:58.347651+0000","last_peered":"2024-09-21T10:30:58.347651+0000","last_clean":"2024-09-21T10:30:58.347651+0000","last_became_active":"2024-09-21T10:28:00.280773+0000","last_became_peered":"2024-09-21T10:28:00.280773+0000","last_unstale":"2024-09-21T10:30:58.347651+0000","last_undegraded":"2024-09-21T10:30:58.347651+0000","last_fullsized":"2024-09-21T10:30:58.347651+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:32:48.557279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"383'15125","reported_seq":21592,"reported_epoch":383,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T10:31:09.886143+0000","last_change":"2024-09-21T10:28:25.623040+0000","last_active":"2024-09-21T10:31:09.886143+0000","last_peered":"2024-09-21T10:31:09.886143+0000","last_clean":"2024-09-21T10:27:59.294389+0000","last_became_active":"2024-09-21T10:28:01.608572+0000","last_became_peered":"2024-09-21T10:28:01.608572+0000","last_unstale":"2024-09-21T10:31:09.886143+0000","last_undegraded":"2024-09-21T10:28:25.614349+0000","last_fullsized":"2024-09-21T10:28:01.272643+0000","mapping_epoch":361,"log_start":"366'7800","ondisk_log_start":"366'7800","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7325,"log_dups_size":0,"ondisk_log_size":7325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23134208,"num_objects":353,"num_object_clones":0,"num_object_copies":1059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15125,"num_write_kb":495296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3190,"num_bytes_recovered":197132288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,4","objects":-3485}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":332,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.349430+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:30:58.349430+0000","last_peered":"2024-09-21T10:30:58.349430+0000","last_clean":"2024-09-21T10:30:58.349430+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:30:58.349430+0000","last_undegraded":"2024-09-21T10:30:58.349430+0000","last_fullsized":"2024-09-21T10:30:58.349430+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":347,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.345535+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:30:58.345535+0000","last_peered":"2024-09-21T10:30:58.345535+0000","last_clean":"2024-09-21T10:30:58.345535+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:30:58.345535+0000","last_undegraded":"2024-09-21T10:30:58.345535+0000","last_fullsized":"2024-09-21T10:30:58.345535+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"383'15070","reported_seq":15157,"reported_epoch":383,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:31:10.583137+0000","last_change":"2024-09-21T10:28:01.644360+0000","last_active":"2024-09-21T10:31:10.583137+0000","last_peered":"2024-09-21T10:31:10.583137+0000","last_clean":"2024-09-21T10:27:59.276032+0000","last_became_active":"2024-09-21T10:28:01.624219+0000","last_became_peered":"2024-09-21T10:28:01.624219+0000","last_unstale":"2024-09-21T10:31:10.583137+0000","last_undegraded":"2024-09-21T10:28:01.350374+0000","last_fullsized":"2024-09-21T10:28:01.295088+0000","mapping_epoch":361,"log_start":"358'5600","ondisk_log_start":"358'5600","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":9470,"log_dups_size":0,"ondisk_log_size":9470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20185088,"num_objects":308,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15070,"num_write_kb":492096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0],"avail_no_missing":["2","0","7"],"object_location_counts":[{"shards":"0,2","objects":-6706}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":57,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.353542+0000","last_change":"2024-09-21T10:28:00.288968+0000","last_active":"2024-09-21T10:30:58.353542+0000","last_peered":"2024-09-21T10:30:58.353542+0000","last_clean":"2024-09-21T10:30:58.353542+0000","last_became_active":"2024-09-21T10:28:00.288244+0000","last_became_peered":"2024-09-21T10:28:00.288244+0000","last_unstale":"2024-09-21T10:30:58.353542+0000","last_undegraded":"2024-09-21T10:30:58.353542+0000","last_fullsized":"2024-09-21T10:30:58.353542+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:42.089701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":674,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.343029+0000","last_change":"2024-09-21T10:24:44.299417+0000","last_active":"2024-09-21T10:30:58.343029+0000","last_peered":"2024-09-21T10:30:58.343029+0000","last_clean":"2024-09-21T10:30:58.343029+0000","last_became_active":"2024-09-21T10:24:44.299116+0000","last_became_peered":"2024-09-21T10:24:44.299116+0000","last_unstale":"2024-09-21T10:30:58.343029+0000","last_undegraded":"2024-09-21T10:30:58.343029+0000","last_fullsized":"2024-09-21T10:30:58.343029+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:44.524841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"383'15556","reported_seq":15640,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:31:10.394608+0000","last_change":"2024-09-21T10:26:36.196642+0000","last_active":"2024-09-21T10:31:10.394608+0000","last_peered":"2024-09-21T10:31:10.394608+0000","last_clean":"2024-09-21T10:31:10.394608+0000","last_became_active":"2024-09-21T10:26:36.196357+0000","last_became_peered":"2024-09-21T10:26:36.196357+0000","last_unstale":"2024-09-21T10:31:10.394608+0000","last_undegraded":"2024-09-21T10:31:10.394608+0000","last_fullsized":"2024-09-21T10:31:10.394608+0000","mapping_epoch":348,"log_start":"358'6900","ondisk_log_start":"358'6900","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":8656,"log_dups_size":0,"ondisk_log_size":8656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:54:19.391031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20185088,"num_objects":308,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15556,"num_write_kb":507648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"82'100","reported_seq":505,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.351242+0000","last_change":"2024-09-21T10:28:00.285919+0000","last_active":"2024-09-21T10:30:58.351242+0000","last_peered":"2024-09-21T10:30:58.351242+0000","last_clean":"2024-09-21T10:30:58.351242+0000","last_became_active":"2024-09-21T10:28:00.285324+0000","last_became_peered":"2024-09-21T10:28:00.285324+0000","last_unstale":"2024-09-21T10:30:58.351242+0000","last_undegraded":"2024-09-21T10:30:58.351242+0000","last_fullsized":"2024-09-21T10:30:58.351242+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.728195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.728195+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:00.841610+0000","scrub_duration":0.0069698140000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":61,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.345652+0000","last_change":"2024-09-21T10:28:00.303125+0000","last_active":"2024-09-21T10:30:58.345652+0000","last_peered":"2024-09-21T10:30:58.345652+0000","last_clean":"2024-09-21T10:30:58.345652+0000","last_became_active":"2024-09-21T10:28:00.293684+0000","last_became_peered":"2024-09-21T10:28:00.293684+0000","last_unstale":"2024-09-21T10:30:58.345652+0000","last_undegraded":"2024-09-21T10:30:58.345652+0000","last_fullsized":"2024-09-21T10:30:58.345652+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:04.495913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":83,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347910+0000","last_change":"2024-09-21T10:28:00.283456+0000","last_active":"2024-09-21T10:30:58.347910+0000","last_peered":"2024-09-21T10:30:58.347910+0000","last_clean":"2024-09-21T10:30:58.347910+0000","last_became_active":"2024-09-21T10:28:00.282881+0000","last_became_peered":"2024-09-21T10:28:00.282881+0000","last_unstale":"2024-09-21T10:30:58.347910+0000","last_undegraded":"2024-09-21T10:30:58.347910+0000","last_fullsized":"2024-09-21T10:30:58.347910+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:43.681481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"383'15394","reported_seq":15480,"reported_epoch":383,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:31:10.631277+0000","last_change":"2024-09-21T10:28:01.680872+0000","last_active":"2024-09-21T10:31:10.631277+0000","last_peered":"2024-09-21T10:31:10.631277+0000","last_clean":"2024-09-21T10:27:59.257182+0000","last_became_active":"2024-09-21T10:28:01.651875+0000","last_became_peered":"2024-09-21T10:28:01.651875+0000","last_unstale":"2024-09-21T10:31:10.631277+0000","last_undegraded":"2024-09-21T10:28:01.349133+0000","last_fullsized":"2024-09-21T10:28:01.272587+0000","mapping_epoch":361,"log_start":"358'6000","ondisk_log_start":"358'6000","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":9394,"log_dups_size":0,"ondisk_log_size":9394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22937600,"num_objects":350,"num_object_clones":0,"num_object_copies":1050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15394,"num_write_kb":503808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5],"avail_no_missing":["2","4","5"],"object_location_counts":[{"shards":"2,5","objects":-6799}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'100","reported_seq":177,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.328970+0000","last_change":"2024-09-21T10:28:00.301695+0000","last_active":"2024-09-21T10:30:58.328970+0000","last_peered":"2024-09-21T10:30:58.328970+0000","last_clean":"2024-09-21T10:30:58.328970+0000","last_became_active":"2024-09-21T10:28:00.289856+0000","last_became_peered":"2024-09-21T10:28:00.289856+0000","last_unstale":"2024-09-21T10:30:58.328970+0000","last_undegraded":"2024-09-21T10:30:58.328970+0000","last_fullsized":"2024-09-21T10:30:58.328970+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:27.234001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:27.234001+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:02.389036+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.345736+0000","last_change":"2024-09-21T10:24:44.309470+0000","last_active":"2024-09-21T10:30:58.345736+0000","last_peered":"2024-09-21T10:30:58.345736+0000","last_clean":"2024-09-21T10:30:58.345736+0000","last_became_active":"2024-09-21T10:24:44.308836+0000","last_became_peered":"2024-09-21T10:24:44.308836+0000","last_unstale":"2024-09-21T10:30:58.345736+0000","last_undegraded":"2024-09-21T10:30:58.345736+0000","last_fullsized":"2024-09-21T10:30:58.345736+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:09:44.622418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":73,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.348273+0000","last_change":"2024-09-21T10:24:44.305135+0000","last_active":"2024-09-21T10:30:58.348273+0000","last_peered":"2024-09-21T10:30:58.348273+0000","last_clean":"2024-09-21T10:30:58.348273+0000","last_became_active":"2024-09-21T10:24:44.304839+0000","last_became_peered":"2024-09-21T10:24:44.304839+0000","last_unstale":"2024-09-21T10:30:58.348273+0000","last_undegraded":"2024-09-21T10:30:58.348273+0000","last_fullsized":"2024-09-21T10:30:58.348273+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:02.264404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"384'15297","reported_seq":15341,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:13.100985+0000","last_change":"2024-09-21T10:26:36.194994+0000","last_active":"2024-09-21T10:31:13.100985+0000","last_peered":"2024-09-21T10:31:13.100985+0000","last_clean":"2024-09-21T10:31:13.100985+0000","last_became_active":"2024-09-21T10:26:36.194404+0000","last_became_peered":"2024-09-21T10:26:36.194404+0000","last_unstale":"2024-09-21T10:31:13.100985+0000","last_undegraded":"2024-09-21T10:31:13.100985+0000","last_fullsized":"2024-09-21T10:31:13.100985+0000","mapping_epoch":348,"log_start":"365'7600","ondisk_log_start":"365'7600","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7697,"log_dups_size":0,"ondisk_log_size":7697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:10:51.551059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14352384,"num_objects":219,"num_object_clones":0,"num_object_copies":657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15297,"num_write_kb":496512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":106,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.344050+0000","last_change":"2024-09-21T10:25:39.872680+0000","last_active":"2024-09-21T10:30:58.344050+0000","last_peered":"2024-09-21T10:30:58.344050+0000","last_clean":"2024-09-21T10:30:58.344050+0000","last_became_active":"2024-09-21T10:25:39.872312+0000","last_became_peered":"2024-09-21T10:25:39.872312+0000","last_unstale":"2024-09-21T10:30:58.344050+0000","last_undegraded":"2024-09-21T10:30:58.344050+0000","last_fullsized":"2024-09-21T10:30:58.344050+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:54.942046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":73,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.348168+0000","last_change":"2024-09-21T10:24:44.310484+0000","last_active":"2024-09-21T10:30:58.348168+0000","last_peered":"2024-09-21T10:30:58.348168+0000","last_clean":"2024-09-21T10:30:58.348168+0000","last_became_active":"2024-09-21T10:24:44.307209+0000","last_became_peered":"2024-09-21T10:24:44.307209+0000","last_unstale":"2024-09-21T10:30:58.348168+0000","last_undegraded":"2024-09-21T10:30:58.348168+0000","last_fullsized":"2024-09-21T10:30:58.348168+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:48:36.767539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"384'15410","reported_seq":15454,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:13.113632+0000","last_change":"2024-09-21T10:26:36.196826+0000","last_active":"2024-09-21T10:31:13.113632+0000","last_peered":"2024-09-21T10:31:13.113632+0000","last_clean":"2024-09-21T10:31:13.113632+0000","last_became_active":"2024-09-21T10:26:36.196355+0000","last_became_peered":"2024-09-21T10:26:36.196355+0000","last_unstale":"2024-09-21T10:31:13.113632+0000","last_undegraded":"2024-09-21T10:31:13.113632+0000","last_fullsized":"2024-09-21T10:31:13.113632+0000","mapping_epoch":348,"log_start":"367'8700","ondisk_log_start":"367'8700","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":6710,"log_dups_size":0,"ondisk_log_size":6710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:58.869309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15073280,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15410,"num_write_kb":500480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'100","reported_seq":161,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.329689+0000","last_change":"2024-09-21T10:23:26.197358+0000","last_active":"2024-09-21T10:30:58.329689+0000","last_peered":"2024-09-21T10:30:58.329689+0000","last_clean":"2024-09-21T10:30:58.329689+0000","last_became_active":"2024-09-21T10:23:20.695174+0000","last_became_peered":"2024-09-21T10:23:20.695174+0000","last_unstale":"2024-09-21T10:30:58.329689+0000","last_undegraded":"2024-09-21T10:30:58.329689+0000","last_fullsized":"2024-09-21T10:30:58.329689+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:26.197291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:26.197291+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:33.961031+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":57,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.345225+0000","last_change":"2024-09-21T10:28:00.287560+0000","last_active":"2024-09-21T10:30:58.345225+0000","last_peered":"2024-09-21T10:30:58.345225+0000","last_clean":"2024-09-21T10:30:58.345225+0000","last_became_active":"2024-09-21T10:28:00.285076+0000","last_became_peered":"2024-09-21T10:28:00.285076+0000","last_unstale":"2024-09-21T10:30:58.345225+0000","last_undegraded":"2024-09-21T10:30:58.345225+0000","last_fullsized":"2024-09-21T10:30:58.345225+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:49.950523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":670,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.348954+0000","last_change":"2024-09-21T10:28:00.282062+0000","last_active":"2024-09-21T10:30:58.348954+0000","last_peered":"2024-09-21T10:30:58.348954+0000","last_clean":"2024-09-21T10:30:58.348954+0000","last_became_active":"2024-09-21T10:28:00.281593+0000","last_became_peered":"2024-09-21T10:28:00.281593+0000","last_unstale":"2024-09-21T10:30:58.348954+0000","last_undegraded":"2024-09-21T10:30:58.348954+0000","last_fullsized":"2024-09-21T10:30:58.348954+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:45.704840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"384'15386","reported_seq":15430,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:13.152056+0000","last_change":"2024-09-21T10:26:36.197465+0000","last_active":"2024-09-21T10:31:13.152056+0000","last_peered":"2024-09-21T10:31:13.152056+0000","last_clean":"2024-09-21T10:31:13.152056+0000","last_became_active":"2024-09-21T10:26:36.196105+0000","last_became_peered":"2024-09-21T10:26:36.196105+0000","last_unstale":"2024-09-21T10:31:13.152056+0000","last_undegraded":"2024-09-21T10:31:13.152056+0000","last_fullsized":"2024-09-21T10:31:13.152056+0000","mapping_epoch":348,"log_start":"365'7600","ondisk_log_start":"365'7600","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7786,"log_dups_size":0,"ondisk_log_size":7786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:16.586685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14548992,"num_objects":222,"num_object_clones":0,"num_object_copies":666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15386,"num_write_kb":499456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":486,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.349478+0000","last_change":"2024-09-21T10:23:18.790100+0000","last_active":"2024-09-21T10:30:58.349478+0000","last_peered":"2024-09-21T10:30:58.349478+0000","last_clean":"2024-09-21T10:30:58.349478+0000","last_became_active":"2024-09-21T10:23:14.677610+0000","last_became_peered":"2024-09-21T10:23:14.677610+0000","last_unstale":"2024-09-21T10:30:58.349478+0000","last_undegraded":"2024-09-21T10:30:58.349478+0000","last_fullsized":"2024-09-21T10:30:58.349478+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:18.790017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:18.790017+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:56.562241+0000","scrub_duration":0.0080400790000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":579,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.332792+0000","last_change":"2024-09-21T10:26:41.279374+0000","last_active":"2024-09-21T10:30:58.332792+0000","last_peered":"2024-09-21T10:30:58.332792+0000","last_clean":"2024-09-21T10:30:58.332792+0000","last_became_active":"2024-09-21T10:26:41.278994+0000","last_became_peered":"2024-09-21T10:26:41.278994+0000","last_unstale":"2024-09-21T10:30:58.332792+0000","last_undegraded":"2024-09-21T10:30:58.332792+0000","last_fullsized":"2024-09-21T10:30:58.332792+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:49:57.821175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":482,"num_read_kb":1581,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":57,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.345658+0000","last_change":"2024-09-21T10:28:00.295436+0000","last_active":"2024-09-21T10:30:58.345658+0000","last_peered":"2024-09-21T10:30:58.345658+0000","last_clean":"2024-09-21T10:30:58.345658+0000","last_became_active":"2024-09-21T10:28:00.295135+0000","last_became_peered":"2024-09-21T10:28:00.295135+0000","last_unstale":"2024-09-21T10:30:58.345658+0000","last_undegraded":"2024-09-21T10:30:58.345658+0000","last_fullsized":"2024-09-21T10:30:58.345658+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:14:03.027395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":370,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.345753+0000","last_change":"2024-09-21T10:24:44.308008+0000","last_active":"2024-09-21T10:30:58.345753+0000","last_peered":"2024-09-21T10:30:58.345753+0000","last_clean":"2024-09-21T10:30:58.345753+0000","last_became_active":"2024-09-21T10:24:44.307595+0000","last_became_peered":"2024-09-21T10:24:44.307595+0000","last_unstale":"2024-09-21T10:30:58.345753+0000","last_undegraded":"2024-09-21T10:30:58.345753+0000","last_fullsized":"2024-09-21T10:30:58.345753+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:33:49.448676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"383'15146","reported_seq":15230,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:31:09.845420+0000","last_change":"2024-09-21T10:26:36.197089+0000","last_active":"2024-09-21T10:31:09.845420+0000","last_peered":"2024-09-21T10:31:09.845420+0000","last_clean":"2024-09-21T10:31:09.845420+0000","last_became_active":"2024-09-21T10:26:36.196846+0000","last_became_peered":"2024-09-21T10:26:36.196846+0000","last_unstale":"2024-09-21T10:31:09.845420+0000","last_undegraded":"2024-09-21T10:31:09.845420+0000","last_fullsized":"2024-09-21T10:31:09.845420+0000","mapping_epoch":348,"log_start":"366'7800","ondisk_log_start":"366'7800","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7346,"log_dups_size":0,"ondisk_log_size":7346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:14:52.494647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21757952,"num_objects":332,"num_object_clones":0,"num_object_copies":996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15146,"num_write_kb":495296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"82'100","reported_seq":168,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.357603+0000","last_change":"2024-09-21T10:24:44.309154+0000","last_active":"2024-09-21T10:30:58.357603+0000","last_peered":"2024-09-21T10:30:58.357603+0000","last_clean":"2024-09-21T10:30:58.357603+0000","last_became_active":"2024-09-21T10:24:44.308330+0000","last_became_peered":"2024-09-21T10:24:44.308330+0000","last_unstale":"2024-09-21T10:30:58.357603+0000","last_undegraded":"2024-09-21T10:30:58.357603+0000","last_fullsized":"2024-09-21T10:30:58.357603+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.641718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.641718+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:10.158739+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":728,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.349397+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:30:58.349397+0000","last_peered":"2024-09-21T10:30:58.349397+0000","last_clean":"2024-09-21T10:30:58.349397+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:30:58.349397+0000","last_undegraded":"2024-09-21T10:30:58.349397+0000","last_fullsized":"2024-09-21T10:30:58.349397+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":683,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.344291+0000","last_change":"2024-09-21T10:28:00.296831+0000","last_active":"2024-09-21T10:30:58.344291+0000","last_peered":"2024-09-21T10:30:58.344291+0000","last_clean":"2024-09-21T10:30:58.344291+0000","last_became_active":"2024-09-21T10:28:00.296527+0000","last_became_peered":"2024-09-21T10:28:00.296527+0000","last_unstale":"2024-09-21T10:30:58.344291+0000","last_undegraded":"2024-09-21T10:30:58.344291+0000","last_fullsized":"2024-09-21T10:30:58.344291+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:18.024868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":651,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.344186+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:30:58.344186+0000","last_peered":"2024-09-21T10:30:58.344186+0000","last_clean":"2024-09-21T10:30:58.344186+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:30:58.344186+0000","last_undegraded":"2024-09-21T10:30:58.344186+0000","last_fullsized":"2024-09-21T10:30:58.344186+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"384'15816","reported_seq":15861,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.984024+0000","last_change":"2024-09-21T10:26:41.278294+0000","last_active":"2024-09-21T10:31:12.984024+0000","last_peered":"2024-09-21T10:31:12.984024+0000","last_clean":"2024-09-21T10:31:12.984024+0000","last_became_active":"2024-09-21T10:26:41.277965+0000","last_became_peered":"2024-09-21T10:26:41.277965+0000","last_unstale":"2024-09-21T10:31:12.984024+0000","last_undegraded":"2024-09-21T10:31:12.984024+0000","last_fullsized":"2024-09-21T10:31:12.984024+0000","mapping_epoch":351,"log_start":"358'6100","ondisk_log_start":"358'6100","created":348,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":9716,"log_dups_size":0,"ondisk_log_size":9716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:30.913837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15990784,"num_objects":244,"num_object_clones":0,"num_object_copies":732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15816,"num_write_kb":513920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"82'100","reported_seq":314,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.343920+0000","last_change":"2024-09-21T10:24:44.294460+0000","last_active":"2024-09-21T10:30:58.343920+0000","last_peered":"2024-09-21T10:30:58.343920+0000","last_clean":"2024-09-21T10:30:58.343920+0000","last_became_active":"2024-09-21T10:24:44.293972+0000","last_became_peered":"2024-09-21T10:24:44.293972+0000","last_unstale":"2024-09-21T10:30:58.343920+0000","last_undegraded":"2024-09-21T10:30:58.343920+0000","last_fullsized":"2024-09-21T10:30:58.343920+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:26.199522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:26.199522+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:26.135171+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":1461,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.330273+0000","last_change":"2024-09-21T10:28:03.346064+0000","last_active":"2024-09-21T10:30:58.330273+0000","last_peered":"2024-09-21T10:30:58.330273+0000","last_clean":"2024-09-21T10:30:58.330273+0000","last_became_active":"2024-09-21T10:28:03.345313+0000","last_became_peered":"2024-09-21T10:28:03.345313+0000","last_unstale":"2024-09-21T10:30:58.330273+0000","last_undegraded":"2024-09-21T10:30:58.330273+0000","last_fullsized":"2024-09-21T10:30:58.330273+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:32.097521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":523,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.349848+0000","last_change":"2024-09-21T10:08:18.470748+0000","last_active":"2024-09-21T10:30:58.349848+0000","last_peered":"2024-09-21T10:30:58.349848+0000","last_clean":"2024-09-21T10:30:58.349848+0000","last_became_active":"2024-09-21T10:08:18.470227+0000","last_became_peered":"2024-09-21T10:08:18.470227+0000","last_unstale":"2024-09-21T10:30:58.349848+0000","last_undegraded":"2024-09-21T10:30:58.349848+0000","last_fullsized":"2024-09-21T10:30:58.349848+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:16.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":645,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.329694+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:30:58.329694+0000","last_peered":"2024-09-21T10:30:58.329694+0000","last_clean":"2024-09-21T10:30:58.329694+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:30:58.329694+0000","last_undegraded":"2024-09-21T10:30:58.329694+0000","last_fullsized":"2024-09-21T10:30:58.329694+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"384'15606","reported_seq":15650,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:13.157089+0000","last_change":"2024-09-21T10:26:36.989641+0000","last_active":"2024-09-21T10:31:13.157089+0000","last_peered":"2024-09-21T10:31:13.157089+0000","last_clean":"2024-09-21T10:31:13.157089+0000","last_became_active":"2024-09-21T10:26:36.989340+0000","last_became_peered":"2024-09-21T10:26:36.989340+0000","last_unstale":"2024-09-21T10:31:13.157089+0000","last_undegraded":"2024-09-21T10:31:13.157089+0000","last_fullsized":"2024-09-21T10:31:13.157089+0000","mapping_epoch":348,"log_start":"365'7900","ondisk_log_start":"365'7900","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7706,"log_dups_size":0,"ondisk_log_size":7706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:50.213332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13893632,"num_objects":212,"num_object_clones":0,"num_object_copies":636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15606,"num_write_kb":506176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"82'100","reported_seq":307,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.344155+0000","last_change":"2024-09-21T10:23:21.277739+0000","last_active":"2024-09-21T10:30:58.344155+0000","last_peered":"2024-09-21T10:30:58.344155+0000","last_clean":"2024-09-21T10:30:58.344155+0000","last_became_active":"2024-09-21T10:23:20.689733+0000","last_became_peered":"2024-09-21T10:23:20.689733+0000","last_unstale":"2024-09-21T10:30:58.344155+0000","last_undegraded":"2024-09-21T10:30:58.344155+0000","last_fullsized":"2024-09-21T10:30:58.344155+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.277663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.277663+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:01:54.523189+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":1122,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.348630+0000","last_change":"2024-09-21T10:07:43.794767+0000","last_active":"2024-09-21T10:30:58.348630+0000","last_peered":"2024-09-21T10:30:58.348630+0000","last_clean":"2024-09-21T10:30:58.348630+0000","last_became_active":"2024-09-21T10:07:43.794062+0000","last_became_peered":"2024-09-21T10:07:43.794062+0000","last_unstale":"2024-09-21T10:30:58.348630+0000","last_undegraded":"2024-09-21T10:30:58.348630+0000","last_fullsized":"2024-09-21T10:30:58.348630+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:51.977955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":73,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347866+0000","last_change":"2024-09-21T10:24:44.299492+0000","last_active":"2024-09-21T10:30:58.347866+0000","last_peered":"2024-09-21T10:30:58.347866+0000","last_clean":"2024-09-21T10:30:58.347866+0000","last_became_active":"2024-09-21T10:24:44.299187+0000","last_became_peered":"2024-09-21T10:24:44.299187+0000","last_unstale":"2024-09-21T10:30:58.347866+0000","last_undegraded":"2024-09-21T10:30:58.347866+0000","last_fullsized":"2024-09-21T10:30:58.347866+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:58.500386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":576,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.330300+0000","last_change":"2024-09-21T10:28:00.289453+0000","last_active":"2024-09-21T10:30:58.330300+0000","last_peered":"2024-09-21T10:30:58.330300+0000","last_clean":"2024-09-21T10:30:58.330300+0000","last_became_active":"2024-09-21T10:28:00.289030+0000","last_became_peered":"2024-09-21T10:28:00.289030+0000","last_unstale":"2024-09-21T10:30:58.330300+0000","last_undegraded":"2024-09-21T10:30:58.330300+0000","last_fullsized":"2024-09-21T10:30:58.330300+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:21.959998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"384'15265","reported_seq":15309,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.973522+0000","last_change":"2024-09-21T10:26:36.196471+0000","last_active":"2024-09-21T10:31:12.973522+0000","last_peered":"2024-09-21T10:31:12.973522+0000","last_clean":"2024-09-21T10:31:12.973522+0000","last_became_active":"2024-09-21T10:26:36.196167+0000","last_became_peered":"2024-09-21T10:26:36.196167+0000","last_unstale":"2024-09-21T10:31:12.973522+0000","last_undegraded":"2024-09-21T10:31:12.973522+0000","last_fullsized":"2024-09-21T10:31:12.973522+0000","mapping_epoch":348,"log_start":"364'7300","ondisk_log_start":"364'7300","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7965,"log_dups_size":0,"ondisk_log_size":7965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:16.488795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14090240,"num_objects":215,"num_object_clones":0,"num_object_copies":645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15265,"num_write_kb":495360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"82'100","reported_seq":248,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.346001+0000","last_change":"2024-09-21T10:28:00.294997+0000","last_active":"2024-09-21T10:30:58.346001+0000","last_peered":"2024-09-21T10:30:58.346001+0000","last_clean":"2024-09-21T10:30:58.346001+0000","last_became_active":"2024-09-21T10:28:00.294595+0000","last_became_peered":"2024-09-21T10:28:00.294595+0000","last_unstale":"2024-09-21T10:30:58.346001+0000","last_undegraded":"2024-09-21T10:30:58.346001+0000","last_fullsized":"2024-09-21T10:30:58.346001+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:30.569982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:30.569982+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:01.422939+0000","scrub_duration":0.0051309349999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":182,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.332681+0000","last_change":"2024-09-21T10:28:00.294966+0000","last_active":"2024-09-21T10:30:58.332681+0000","last_peered":"2024-09-21T10:30:58.332681+0000","last_clean":"2024-09-21T10:30:58.332681+0000","last_became_active":"2024-09-21T10:28:00.294552+0000","last_became_peered":"2024-09-21T10:28:00.294552+0000","last_unstale":"2024-09-21T10:30:58.332681+0000","last_undegraded":"2024-09-21T10:30:58.332681+0000","last_fullsized":"2024-09-21T10:30:58.332681+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:47.292125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":1228,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.348748+0000","last_change":"2024-09-21T10:08:18.468995+0000","last_active":"2024-09-21T10:30:58.348748+0000","last_peered":"2024-09-21T10:30:58.348748+0000","last_clean":"2024-09-21T10:30:58.348748+0000","last_became_active":"2024-09-21T10:08:18.468243+0000","last_became_peered":"2024-09-21T10:08:18.468243+0000","last_unstale":"2024-09-21T10:30:58.348748+0000","last_undegraded":"2024-09-21T10:30:58.348748+0000","last_fullsized":"2024-09-21T10:30:58.348748+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:45.520968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":554,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.348990+0000","last_change":"2024-09-21T10:24:44.309614+0000","last_active":"2024-09-21T10:30:58.348990+0000","last_peered":"2024-09-21T10:30:58.348990+0000","last_clean":"2024-09-21T10:30:58.348990+0000","last_became_active":"2024-09-21T10:24:44.309187+0000","last_became_peered":"2024-09-21T10:24:44.309187+0000","last_unstale":"2024-09-21T10:30:58.348990+0000","last_undegraded":"2024-09-21T10:30:58.348990+0000","last_fullsized":"2024-09-21T10:30:58.348990+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:01.074803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"384'15441","reported_seq":15486,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:13.118070+0000","last_change":"2024-09-21T10:26:36.196357+0000","last_active":"2024-09-21T10:31:13.118070+0000","last_peered":"2024-09-21T10:31:13.118070+0000","last_clean":"2024-09-21T10:31:13.118070+0000","last_became_active":"2024-09-21T10:26:36.196060+0000","last_became_peered":"2024-09-21T10:26:36.196060+0000","last_unstale":"2024-09-21T10:31:13.118070+0000","last_undegraded":"2024-09-21T10:31:13.118070+0000","last_fullsized":"2024-09-21T10:31:13.118070+0000","mapping_epoch":348,"log_start":"365'7700","ondisk_log_start":"365'7700","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7741,"log_dups_size":0,"ondisk_log_size":7741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:46:34.643144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14942232,"num_objects":229,"num_object_clones":0,"num_object_copies":687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":229,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15441,"num_write_kb":501377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"82'100","reported_seq":258,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.345705+0000","last_change":"2024-09-21T10:28:00.295277+0000","last_active":"2024-09-21T10:30:58.345705+0000","last_peered":"2024-09-21T10:30:58.345705+0000","last_clean":"2024-09-21T10:30:58.345705+0000","last_became_active":"2024-09-21T10:28:00.294016+0000","last_became_peered":"2024-09-21T10:28:00.294016+0000","last_unstale":"2024-09-21T10:30:58.345705+0000","last_undegraded":"2024-09-21T10:30:58.345705+0000","last_fullsized":"2024-09-21T10:30:58.345705+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:24.536711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:24.536711+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:28:49.984470+0000","scrub_duration":0.0047525650000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":523,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.348646+0000","last_change":"2024-09-21T10:08:18.469907+0000","last_active":"2024-09-21T10:30:58.348646+0000","last_peered":"2024-09-21T10:30:58.348646+0000","last_clean":"2024-09-21T10:30:58.348646+0000","last_became_active":"2024-09-21T10:08:18.469443+0000","last_became_peered":"2024-09-21T10:08:18.469443+0000","last_unstale":"2024-09-21T10:30:58.348646+0000","last_undegraded":"2024-09-21T10:30:58.348646+0000","last_fullsized":"2024-09-21T10:30:58.348646+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:15.645803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":806,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.349463+0000","last_change":"2024-09-21T10:24:44.334847+0000","last_active":"2024-09-21T10:30:58.349463+0000","last_peered":"2024-09-21T10:30:58.349463+0000","last_clean":"2024-09-21T10:30:58.349463+0000","last_became_active":"2024-09-21T10:24:44.301321+0000","last_became_peered":"2024-09-21T10:24:44.301321+0000","last_unstale":"2024-09-21T10:30:58.349463+0000","last_undegraded":"2024-09-21T10:30:58.349463+0000","last_fullsized":"2024-09-21T10:30:58.349463+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:37.628940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":58,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.332727+0000","last_change":"2024-09-21T10:26:41.278937+0000","last_active":"2024-09-21T10:30:58.332727+0000","last_peered":"2024-09-21T10:30:58.332727+0000","last_clean":"2024-09-21T10:30:58.332727+0000","last_became_active":"2024-09-21T10:26:41.277726+0000","last_became_peered":"2024-09-21T10:26:41.277726+0000","last_unstale":"2024-09-21T10:30:58.332727+0000","last_undegraded":"2024-09-21T10:30:58.332727+0000","last_fullsized":"2024-09-21T10:30:58.332727+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:04.741057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"384'15540","reported_seq":15584,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:13.163936+0000","last_change":"2024-09-21T10:26:36.196641+0000","last_active":"2024-09-21T10:31:13.163936+0000","last_peered":"2024-09-21T10:31:13.163936+0000","last_clean":"2024-09-21T10:31:13.163936+0000","last_became_active":"2024-09-21T10:26:36.196351+0000","last_became_peered":"2024-09-21T10:26:36.196351+0000","last_unstale":"2024-09-21T10:31:13.163936+0000","last_undegraded":"2024-09-21T10:31:13.163936+0000","last_fullsized":"2024-09-21T10:31:13.163936+0000","mapping_epoch":348,"log_start":"367'8800","ondisk_log_start":"367'8800","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":6740,"log_dups_size":0,"ondisk_log_size":6740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:41:54.280247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14286848,"num_objects":218,"num_object_clones":0,"num_object_copies":654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15540,"num_write_kb":504256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'100","reported_seq":73,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.348183+0000","last_change":"2024-09-21T10:24:44.296758+0000","last_active":"2024-09-21T10:30:58.348183+0000","last_peered":"2024-09-21T10:30:58.348183+0000","last_clean":"2024-09-21T10:30:58.348183+0000","last_became_active":"2024-09-21T10:24:44.296403+0000","last_became_peered":"2024-09-21T10:24:44.296403+0000","last_unstale":"2024-09-21T10:30:58.348183+0000","last_undegraded":"2024-09-21T10:30:58.348183+0000","last_fullsized":"2024-09-21T10:30:58.348183+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.105746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.105746+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:20:12.502081+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":61,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.345871+0000","last_change":"2024-09-21T10:28:00.300952+0000","last_active":"2024-09-21T10:30:58.345871+0000","last_peered":"2024-09-21T10:30:58.345871+0000","last_clean":"2024-09-21T10:30:58.345871+0000","last_became_active":"2024-09-21T10:28:00.293337+0000","last_became_peered":"2024-09-21T10:28:00.293337+0000","last_unstale":"2024-09-21T10:30:58.345871+0000","last_undegraded":"2024-09-21T10:30:58.345871+0000","last_fullsized":"2024-09-21T10:30:58.345871+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:58.409761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":948,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.349119+0000","last_change":"2024-09-21T10:24:46.844278+0000","last_active":"2024-09-21T10:30:58.349119+0000","last_peered":"2024-09-21T10:30:58.349119+0000","last_clean":"2024-09-21T10:30:58.349119+0000","last_became_active":"2024-09-21T10:24:46.840961+0000","last_became_peered":"2024-09-21T10:24:46.840961+0000","last_unstale":"2024-09-21T10:30:58.349119+0000","last_undegraded":"2024-09-21T10:30:58.349119+0000","last_fullsized":"2024-09-21T10:30:58.349119+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:49:48.823524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":57,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.345099+0000","last_change":"2024-09-21T10:28:00.293864+0000","last_active":"2024-09-21T10:30:58.345099+0000","last_peered":"2024-09-21T10:30:58.345099+0000","last_clean":"2024-09-21T10:30:58.345099+0000","last_became_active":"2024-09-21T10:28:00.293494+0000","last_became_peered":"2024-09-21T10:28:00.293494+0000","last_unstale":"2024-09-21T10:30:58.345099+0000","last_undegraded":"2024-09-21T10:30:58.345099+0000","last_fullsized":"2024-09-21T10:30:58.345099+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:37.098068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"384'15249","reported_seq":15397,"reported_epoch":384,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:31:13.095152+0000","last_change":"2024-09-21T10:28:01.631803+0000","last_active":"2024-09-21T10:31:13.095152+0000","last_peered":"2024-09-21T10:31:13.095152+0000","last_clean":"2024-09-21T10:27:59.267411+0000","last_became_active":"2024-09-21T10:28:01.425385+0000","last_became_peered":"2024-09-21T10:28:01.425385+0000","last_unstale":"2024-09-21T10:31:13.095152+0000","last_undegraded":"2024-09-21T10:31:13.095152+0000","last_fullsized":"2024-09-21T10:31:13.095152+0000","mapping_epoch":361,"log_start":"365'7500","ondisk_log_start":"365'7500","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7749,"log_dups_size":0,"ondisk_log_size":7749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:51:26.681334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13697024,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":209,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15249,"num_write_kb":494656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":2949120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,2,4],"avail_no_missing":["6","2","4"],"object_location_counts":[{"shards":"2,4,6","objects":209}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"82'100","reported_seq":64,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.353719+0000","last_change":"2024-09-21T10:28:00.305255+0000","last_active":"2024-09-21T10:30:58.353719+0000","last_peered":"2024-09-21T10:30:58.353719+0000","last_clean":"2024-09-21T10:30:58.353719+0000","last_became_active":"2024-09-21T10:28:00.290981+0000","last_became_peered":"2024-09-21T10:28:00.290981+0000","last_unstale":"2024-09-21T10:30:58.353719+0000","last_undegraded":"2024-09-21T10:30:58.353719+0000","last_fullsized":"2024-09-21T10:30:58.353719+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.759355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.759355+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":69,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":688,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.343778+0000","last_change":"2024-09-21T10:24:44.307344+0000","last_active":"2024-09-21T10:30:58.343778+0000","last_peered":"2024-09-21T10:30:58.343778+0000","last_clean":"2024-09-21T10:30:58.343778+0000","last_became_active":"2024-09-21T10:24:44.298936+0000","last_became_peered":"2024-09-21T10:24:44.298936+0000","last_unstale":"2024-09-21T10:30:58.343778+0000","last_undegraded":"2024-09-21T10:30:58.343778+0000","last_fullsized":"2024-09-21T10:30:58.343778+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:58:26.025877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":610,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.346282+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:30:58.346282+0000","last_peered":"2024-09-21T10:30:58.346282+0000","last_clean":"2024-09-21T10:30:58.346282+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:30:58.346282+0000","last_undegraded":"2024-09-21T10:30:58.346282+0000","last_fullsized":"2024-09-21T10:30:58.346282+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":73,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347525+0000","last_change":"2024-09-21T10:24:44.303781+0000","last_active":"2024-09-21T10:30:58.347525+0000","last_peered":"2024-09-21T10:30:58.347525+0000","last_clean":"2024-09-21T10:30:58.347525+0000","last_became_active":"2024-09-21T10:24:44.302058+0000","last_became_peered":"2024-09-21T10:24:44.302058+0000","last_unstale":"2024-09-21T10:30:58.347525+0000","last_undegraded":"2024-09-21T10:30:58.347525+0000","last_fullsized":"2024-09-21T10:30:58.347525+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:30.371229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":73,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347868+0000","last_change":"2024-09-21T10:24:44.303417+0000","last_active":"2024-09-21T10:30:58.347868+0000","last_peered":"2024-09-21T10:30:58.347868+0000","last_clean":"2024-09-21T10:30:58.347868+0000","last_became_active":"2024-09-21T10:24:44.303069+0000","last_became_peered":"2024-09-21T10:24:44.303069+0000","last_unstale":"2024-09-21T10:30:58.347868+0000","last_undegraded":"2024-09-21T10:30:58.347868+0000","last_fullsized":"2024-09-21T10:30:58.347868+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:30:30.966228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":83,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347766+0000","last_change":"2024-09-21T10:24:44.321421+0000","last_active":"2024-09-21T10:30:58.347766+0000","last_peered":"2024-09-21T10:30:58.347766+0000","last_clean":"2024-09-21T10:30:58.347766+0000","last_became_active":"2024-09-21T10:24:44.297082+0000","last_became_peered":"2024-09-21T10:24:44.297082+0000","last_unstale":"2024-09-21T10:30:58.347766+0000","last_undegraded":"2024-09-21T10:30:58.347766+0000","last_fullsized":"2024-09-21T10:30:58.347766+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:35:37.704881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":522,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.348727+0000","last_change":"2024-09-21T10:08:18.470378+0000","last_active":"2024-09-21T10:30:58.348727+0000","last_peered":"2024-09-21T10:30:58.348727+0000","last_clean":"2024-09-21T10:30:58.348727+0000","last_became_active":"2024-09-21T10:08:18.469482+0000","last_became_peered":"2024-09-21T10:08:18.469482+0000","last_unstale":"2024-09-21T10:30:58.348727+0000","last_undegraded":"2024-09-21T10:30:58.348727+0000","last_fullsized":"2024-09-21T10:30:58.348727+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:35.628724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":447,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.349303+0000","last_change":"2024-09-21T10:28:00.296771+0000","last_active":"2024-09-21T10:30:58.349303+0000","last_peered":"2024-09-21T10:30:58.349303+0000","last_clean":"2024-09-21T10:30:58.349303+0000","last_became_active":"2024-09-21T10:28:00.285336+0000","last_became_peered":"2024-09-21T10:28:00.285336+0000","last_unstale":"2024-09-21T10:30:58.349303+0000","last_undegraded":"2024-09-21T10:30:58.349303+0000","last_fullsized":"2024-09-21T10:30:58.349303+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:06:50.848492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":155,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.332686+0000","last_change":"2024-09-21T10:26:41.644952+0000","last_active":"2024-09-21T10:30:58.332686+0000","last_peered":"2024-09-21T10:30:58.332686+0000","last_clean":"2024-09-21T10:30:58.332686+0000","last_became_active":"2024-09-21T10:26:41.644592+0000","last_became_peered":"2024-09-21T10:26:41.644592+0000","last_unstale":"2024-09-21T10:30:58.332686+0000","last_undegraded":"2024-09-21T10:30:58.332686+0000","last_fullsized":"2024-09-21T10:30:58.332686+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:44:44.183877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":57,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.345475+0000","last_change":"2024-09-21T10:28:00.294338+0000","last_active":"2024-09-21T10:30:58.345475+0000","last_peered":"2024-09-21T10:30:58.345475+0000","last_clean":"2024-09-21T10:30:58.345475+0000","last_became_active":"2024-09-21T10:28:00.294044+0000","last_became_peered":"2024-09-21T10:28:00.294044+0000","last_unstale":"2024-09-21T10:30:58.345475+0000","last_undegraded":"2024-09-21T10:30:58.345475+0000","last_fullsized":"2024-09-21T10:30:58.345475+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:43:30.999988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":652,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.342502+0000","last_change":"2024-09-21T10:28:00.296935+0000","last_active":"2024-09-21T10:30:58.342502+0000","last_peered":"2024-09-21T10:30:58.342502+0000","last_clean":"2024-09-21T10:30:58.342502+0000","last_became_active":"2024-09-21T10:28:00.296562+0000","last_became_peered":"2024-09-21T10:28:00.296562+0000","last_unstale":"2024-09-21T10:30:58.342502+0000","last_undegraded":"2024-09-21T10:30:58.342502+0000","last_fullsized":"2024-09-21T10:30:58.342502+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:06.863310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":57,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.345203+0000","last_change":"2024-09-21T10:28:00.294718+0000","last_active":"2024-09-21T10:30:58.345203+0000","last_peered":"2024-09-21T10:30:58.345203+0000","last_clean":"2024-09-21T10:30:58.345203+0000","last_became_active":"2024-09-21T10:28:00.294299+0000","last_became_peered":"2024-09-21T10:28:00.294299+0000","last_unstale":"2024-09-21T10:30:58.345203+0000","last_undegraded":"2024-09-21T10:30:58.345203+0000","last_fullsized":"2024-09-21T10:30:58.345203+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:41:46.881336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":347,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.346076+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:30:58.346076+0000","last_peered":"2024-09-21T10:30:58.346076+0000","last_clean":"2024-09-21T10:30:58.346076+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:30:58.346076+0000","last_undegraded":"2024-09-21T10:30:58.346076+0000","last_fullsized":"2024-09-21T10:30:58.346076+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":57,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.345137+0000","last_change":"2024-09-21T10:28:00.293872+0000","last_active":"2024-09-21T10:30:58.345137+0000","last_peered":"2024-09-21T10:30:58.345137+0000","last_clean":"2024-09-21T10:30:58.345137+0000","last_became_active":"2024-09-21T10:28:00.293320+0000","last_became_peered":"2024-09-21T10:28:00.293320+0000","last_unstale":"2024-09-21T10:30:58.345137+0000","last_undegraded":"2024-09-21T10:30:58.345137+0000","last_fullsized":"2024-09-21T10:30:58.345137+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:45:43.909307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":669,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.344004+0000","last_change":"2024-09-21T10:28:00.297304+0000","last_active":"2024-09-21T10:30:58.344004+0000","last_peered":"2024-09-21T10:30:58.344004+0000","last_clean":"2024-09-21T10:30:58.344004+0000","last_became_active":"2024-09-21T10:28:00.296631+0000","last_became_peered":"2024-09-21T10:28:00.296631+0000","last_unstale":"2024-09-21T10:30:58.344004+0000","last_undegraded":"2024-09-21T10:30:58.344004+0000","last_fullsized":"2024-09-21T10:30:58.344004+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:22:16.633581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":669,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.348493+0000","last_change":"2024-09-21T10:28:00.284924+0000","last_active":"2024-09-21T10:30:58.348493+0000","last_peered":"2024-09-21T10:30:58.348493+0000","last_clean":"2024-09-21T10:30:58.348493+0000","last_became_active":"2024-09-21T10:28:00.284633+0000","last_became_peered":"2024-09-21T10:28:00.284633+0000","last_unstale":"2024-09-21T10:30:58.348493+0000","last_undegraded":"2024-09-21T10:30:58.348493+0000","last_fullsized":"2024-09-21T10:30:58.348493+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:23:09.808151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":57,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.388345+0000","last_change":"2024-09-21T10:28:00.283403+0000","last_active":"2024-09-21T10:30:58.388345+0000","last_peered":"2024-09-21T10:30:58.388345+0000","last_clean":"2024-09-21T10:30:58.388345+0000","last_became_active":"2024-09-21T10:28:00.283089+0000","last_became_peered":"2024-09-21T10:28:00.283089+0000","last_unstale":"2024-09-21T10:30:58.388345+0000","last_undegraded":"2024-09-21T10:30:58.388345+0000","last_fullsized":"2024-09-21T10:30:58.388345+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:42.213644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":73,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347561+0000","last_change":"2024-09-21T10:24:44.305124+0000","last_active":"2024-09-21T10:30:58.347561+0000","last_peered":"2024-09-21T10:30:58.347561+0000","last_clean":"2024-09-21T10:30:58.347561+0000","last_became_active":"2024-09-21T10:24:44.304468+0000","last_became_peered":"2024-09-21T10:24:44.304468+0000","last_unstale":"2024-09-21T10:30:58.347561+0000","last_undegraded":"2024-09-21T10:30:58.347561+0000","last_fullsized":"2024-09-21T10:30:58.347561+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:37.904764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":308,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.330156+0000","last_change":"2024-09-21T10:28:00.286646+0000","last_active":"2024-09-21T10:30:58.330156+0000","last_peered":"2024-09-21T10:30:58.330156+0000","last_clean":"2024-09-21T10:30:58.330156+0000","last_became_active":"2024-09-21T10:28:00.286300+0000","last_became_peered":"2024-09-21T10:28:00.286300+0000","last_unstale":"2024-09-21T10:30:58.330156+0000","last_undegraded":"2024-09-21T10:30:58.330156+0000","last_fullsized":"2024-09-21T10:30:58.330156+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:58.215385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":659,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.342905+0000","last_change":"2024-09-21T10:24:44.299989+0000","last_active":"2024-09-21T10:30:58.342905+0000","last_peered":"2024-09-21T10:30:58.342905+0000","last_clean":"2024-09-21T10:30:58.342905+0000","last_became_active":"2024-09-21T10:24:44.299356+0000","last_became_peered":"2024-09-21T10:24:44.299356+0000","last_unstale":"2024-09-21T10:30:58.342905+0000","last_undegraded":"2024-09-21T10:30:58.342905+0000","last_fullsized":"2024-09-21T10:30:58.342905+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:58.867008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":83,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347766+0000","last_change":"2024-09-21T10:28:00.281942+0000","last_active":"2024-09-21T10:30:58.347766+0000","last_peered":"2024-09-21T10:30:58.347766+0000","last_clean":"2024-09-21T10:30:58.347766+0000","last_became_active":"2024-09-21T10:28:00.281600+0000","last_became_peered":"2024-09-21T10:28:00.281600+0000","last_unstale":"2024-09-21T10:30:58.347766+0000","last_undegraded":"2024-09-21T10:30:58.347766+0000","last_fullsized":"2024-09-21T10:30:58.347766+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:21.144244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":77,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347992+0000","last_change":"2024-09-21T10:24:44.329150+0000","last_active":"2024-09-21T10:30:58.347992+0000","last_peered":"2024-09-21T10:30:58.347992+0000","last_clean":"2024-09-21T10:30:58.347992+0000","last_became_active":"2024-09-21T10:24:44.306752+0000","last_became_peered":"2024-09-21T10:24:44.306752+0000","last_unstale":"2024-09-21T10:30:58.347992+0000","last_undegraded":"2024-09-21T10:30:58.347992+0000","last_fullsized":"2024-09-21T10:30:58.347992+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:16.592859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":83,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347556+0000","last_change":"2024-09-21T10:28:00.286009+0000","last_active":"2024-09-21T10:30:58.347556+0000","last_peered":"2024-09-21T10:30:58.347556+0000","last_clean":"2024-09-21T10:30:58.347556+0000","last_became_active":"2024-09-21T10:28:00.285327+0000","last_became_peered":"2024-09-21T10:28:00.285327+0000","last_unstale":"2024-09-21T10:30:58.347556+0000","last_undegraded":"2024-09-21T10:30:58.347556+0000","last_fullsized":"2024-09-21T10:30:58.347556+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:34.985354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":77,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.348105+0000","last_change":"2024-09-21T10:24:44.324087+0000","last_active":"2024-09-21T10:30:58.348105+0000","last_peered":"2024-09-21T10:30:58.348105+0000","last_clean":"2024-09-21T10:30:58.348105+0000","last_became_active":"2024-09-21T10:24:44.298665+0000","last_became_peered":"2024-09-21T10:24:44.298665+0000","last_unstale":"2024-09-21T10:30:58.348105+0000","last_undegraded":"2024-09-21T10:30:58.348105+0000","last_fullsized":"2024-09-21T10:30:58.348105+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:59.819340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":57,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.345397+0000","last_change":"2024-09-21T10:28:00.293054+0000","last_active":"2024-09-21T10:30:58.345397+0000","last_peered":"2024-09-21T10:30:58.345397+0000","last_clean":"2024-09-21T10:30:58.345397+0000","last_became_active":"2024-09-21T10:28:00.292723+0000","last_became_peered":"2024-09-21T10:28:00.292723+0000","last_unstale":"2024-09-21T10:30:58.345397+0000","last_undegraded":"2024-09-21T10:30:58.345397+0000","last_fullsized":"2024-09-21T10:30:58.345397+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:01:45.933570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":57,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.354036+0000","last_change":"2024-09-21T10:28:00.285515+0000","last_active":"2024-09-21T10:30:58.354036+0000","last_peered":"2024-09-21T10:30:58.354036+0000","last_clean":"2024-09-21T10:30:58.354036+0000","last_became_active":"2024-09-21T10:28:00.285203+0000","last_became_peered":"2024-09-21T10:28:00.285203+0000","last_unstale":"2024-09-21T10:30:58.354036+0000","last_undegraded":"2024-09-21T10:30:58.354036+0000","last_fullsized":"2024-09-21T10:30:58.354036+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:41.982228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":73,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347349+0000","last_change":"2024-09-21T10:24:44.298820+0000","last_active":"2024-09-21T10:30:58.347349+0000","last_peered":"2024-09-21T10:30:58.347349+0000","last_clean":"2024-09-21T10:30:58.347349+0000","last_became_active":"2024-09-21T10:24:44.298478+0000","last_became_peered":"2024-09-21T10:24:44.298478+0000","last_unstale":"2024-09-21T10:30:58.347349+0000","last_undegraded":"2024-09-21T10:30:58.347349+0000","last_fullsized":"2024-09-21T10:30:58.347349+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:59.567078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":645,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.342456+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:30:58.342456+0000","last_peered":"2024-09-21T10:30:58.342456+0000","last_clean":"2024-09-21T10:30:58.342456+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:30:58.342456+0000","last_undegraded":"2024-09-21T10:30:58.342456+0000","last_fullsized":"2024-09-21T10:30:58.342456+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":276037656,"num_objects":4213,"num_object_clones":0,"num_object_copies":12639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18001,"num_objects_misplaced":416,"num_objects_unfound":0,"num_objects_dirty":4213,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":245833,"num_write_kb":8001409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3235,"num_bytes_recovered":200081408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2265198592,"data_stored":2265186376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127133,"ondisk_log_size":127133,"up":48,"acting":45,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":10252,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5983,"num_read_kb":13006,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":316358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":69,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":644038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1100,"ondisk_log_size":1100,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397997,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1410596,"kb_used_data":517528,"kb_used_omap":1,"kb_used_meta":893054,"kb_avail":92334556,"statfs":{"total":95995035648,"available":94550585344,"internally_reserved":0,"allocated":529948672,"data_stored":527737491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":914487640},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594225,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1022496,"kb_used_data":112852,"kb_used_omap":1,"kb_used_meta":909630,"kb_avail":92722656,"statfs":{"total":95995035648,"available":94947999744,"internally_reserved":0,"allocated":115560448,"data_stored":113353260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931461464},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790455,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":980176,"kb_used_data":84424,"kb_used_omap":1,"kb_used_meta":895742,"kb_avail":92764976,"statfs":{"total":95995035648,"available":94991335424,"internally_reserved":0,"allocated":86450176,"data_stored":84251307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":917240152},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986684,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1184864,"kb_used_data":590808,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":92560288,"statfs":{"total":95995035648,"available":94781734912,"internally_reserved":0,"allocated":604987392,"data_stored":602776157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":101,"seq":433791697245,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":761980,"kb_used_data":85748,"kb_used_omap":5,"kb_used_meta":676090,"kb_avail":92983172,"statfs":{"total":95995035648,"available":95214768128,"internally_reserved":0,"allocated":87805952,"data_stored":85618718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":692316944},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860772,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1160704,"kb_used_data":156728,"kb_used_omap":8,"kb_used_meta":1003895,"kb_avail":92584448,"statfs":{"total":95995035648,"available":94806474752,"internally_reserved":0,"allocated":160489472,"data_stored":158298664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":1027989356},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024298,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1331568,"kb_used_data":518888,"kb_used_omap":7,"kb_used_meta":812664,"kb_avail":92413584,"statfs":{"total":95995035648,"available":94631510016,"internally_reserved":0,"allocated":531341312,"data_stored":529165129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":832168498},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187833,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1295744,"kb_used_data":243384,"kb_used_omap":8,"kb_used_meta":1052343,"kb_avail":92449408,"statfs":{"total":95995035648,"available":94668193792,"internally_reserved":0,"allocated":249225216,"data_stored":247033876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":1077600241},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":228200448,"data_stored":228196376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":523304960,"data_stored":523304960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":147783680,"data_stored":147783680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":74186752,"data_stored":74186752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":598937600,"data_stored":598933528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":76152832,"data_stored":76152832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":99225600,"data_stored":99221528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":517406720,"data_stored":517406720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:31:14.726 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:14.751 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:14.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:14.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:14.853 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:14.879 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:14.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:14.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:14.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:15.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:15.023 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:15.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:15.123 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:15.174 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:15.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:15.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:15.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:15.311 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:15.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:15.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:15.412 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:15.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:15.449 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:15.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:15.550 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:15.582 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:15.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:15.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:15 smithi196 ceph-mon[84784]: pgmap v1343: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 268 MiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 18001/12720 objects degraded (141.517%); 416/12720 objects misplaced (3.270%); 416 KiB/s, 24 objects/s recovering 2024-09-21T10:31:15.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:15 smithi196 ceph-mon[84784]: from='client.46520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:31:15.684 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:15.709 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:15.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:15.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:15.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:15.844 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:15.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:15.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:15.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:15 smithi098 ceph-mon[97580]: pgmap v1343: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 268 MiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 18001/12720 objects degraded (141.517%); 416/12720 objects misplaced (3.270%); 416 KiB/s, 24 objects/s recovering 2024-09-21T10:31:15.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:15 smithi098 ceph-mon[97580]: from='client.46520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:31:15.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:15 smithi098 ceph-mon[100914]: pgmap v1343: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 268 MiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 18001/12720 objects degraded (141.517%); 416/12720 objects misplaced (3.270%); 416 KiB/s, 24 objects/s recovering 2024-09-21T10:31:15.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:15 smithi098 ceph-mon[100914]: from='client.46520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:31:15.946 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:15.972 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:15.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:15.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:16.074 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:16.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:16.117 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:16.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:16.218 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:16.250 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:16.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:16.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:16.352 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:16.375 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:16.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:16.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:16.477 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:16.505 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:16.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:16.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:16.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:16 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 18001/12720 objects degraded (141.517%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:16.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:16 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 18001/12720 objects degraded (141.517%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:16.606 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:16.635 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:16.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:16.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:16.737 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:16.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:16.771 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:16.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:16.872 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:16 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 18001/12720 objects degraded (141.517%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:16.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:16.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:16.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:17.034 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:17.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:17.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:17.136 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:17.159 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:17.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:17.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:17.260 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:17.297 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:17.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:17.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:17.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:17.425 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:17.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:17.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:17.527 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:17.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:17.581 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:17.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:17.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:31:17.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:17.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:17 smithi098 ceph-mon[97580]: pgmap v1344: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 217 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 644 op/s; 17870/10317 objects degraded (173.209%); 416/10317 objects misplaced (4.032%); 0 B/s, 25 objects/s recovering 2024-09-21T10:31:17.706 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:17 smithi098 ceph-mon[100914]: pgmap v1344: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 217 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 644 op/s; 17870/10317 objects degraded (173.209%); 416/10317 objects misplaced (4.032%); 0 B/s, 25 objects/s recovering 2024-09-21T10:31:17.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:17.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:17.847 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:17.877 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:17.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:17.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:17.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:17 smithi196 ceph-mon[84784]: pgmap v1344: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 217 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 644 op/s; 17870/10317 objects degraded (173.209%); 416/10317 objects misplaced (4.032%); 0 B/s, 25 objects/s recovering 2024-09-21T10:31:17.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:17.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:31:18.005 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:18.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:18.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:18.071 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Removed 125022 objects 2024-09-21T10:31:18.072 INFO:tasks.radosbench.radosbench.0.smithi098.stdout:Clean up completed and total clean up time :179.865 2024-09-21T10:31:18.107 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:18.128 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:18.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:18.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:18.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:18.252 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:18.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:18.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:18.353 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:18.377 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:18.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:18.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:18.478 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:18.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:18.503 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:18.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:18.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:18.626 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:18.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:18.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:18.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:18.753 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:18.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:18.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:18.854 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:18.878 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:18.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:18.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:18.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:19.001 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:19.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:19.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:19.103 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:19.127 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:19.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:19.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:19.229 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:19.250 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:19.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:19.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:19.352 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:19.374 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:19.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:19.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:19.476 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:19.502 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:19.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:19.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:19.603 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:19.625 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:19.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:19.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:19.727 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:19.752 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:19.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:19.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:19.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:19 smithi098 ceph-mon[97580]: pgmap v1345: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 191 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 687 op/s; 17870/9027 objects degraded (197.962%); 416/9027 objects misplaced (4.608%); 0 B/s, 25 objects/s recovering 2024-09-21T10:31:19.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:19 smithi098 ceph-mon[100914]: pgmap v1345: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 191 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 687 op/s; 17870/9027 objects degraded (197.962%); 416/9027 objects misplaced (4.608%); 0 B/s, 25 objects/s recovering 2024-09-21T10:31:19.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:19.879 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:19.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:19.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:19.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:19 smithi196 ceph-mon[84784]: pgmap v1345: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 191 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 687 op/s; 17870/9027 objects degraded (197.962%); 416/9027 objects misplaced (4.608%); 0 B/s, 25 objects/s recovering 2024-09-21T10:31:19.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:20.004 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:20.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:20.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:20.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:20.128 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:20.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:20.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:20.188 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:31:20.188 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:31:20.199 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:31:20.199 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:31:20.217 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:31:20.217 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:31:20.230 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:20.234 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:31:20.234 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:31:20.251 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:31:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:31:20.255 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:20.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:20.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:20.268 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:31:20.268 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:31:20.286 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:31:20.287 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:31:20.303 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:31:20.303 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:31:20.357 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:20.381 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:20.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:20.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:20.483 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:20.506 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:20.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:20.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:20.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:20 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2267820680' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:31:20.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:20 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2267820680' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:31:20.608 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:20.630 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:20.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:20.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:20.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:20 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2267820680' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:31:20.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:31:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:31:20.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:31:20.710 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:20.732 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:20.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:20.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:20.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:20.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:20.884 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:20.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:20.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:20.978 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:31:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:31:20] "GET /metrics HTTP/1.1" 200 36613 "" "Prometheus/2.51.0" 2024-09-21T10:31:20.986 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:21.011 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:21.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:21.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:21.113 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:21.135 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:21.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:21.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:21.237 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:21.260 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:21.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:21.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:21.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:21.383 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:21.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:21.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:21.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:21.507 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:21.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:21.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:21.608 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:21.634 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:21.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:21.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:21.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:21 smithi196 ceph-mon[84784]: pgmap v1346: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 53 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 684 op/s; 17870/2412 objects degraded (740.879%); 0 B/s, 25 objects/s recovering 2024-09-21T10:31:21.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:21 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 17870/2412 objects degraded (740.879%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:21.658 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-09-21T10:31:21.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-09-21T10:31:21.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:21 smithi098 ceph-mon[97580]: pgmap v1346: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 53 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 684 op/s; 17870/2412 objects degraded (740.879%); 0 B/s, 25 objects/s recovering 2024-09-21T10:31:21.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:21 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 17870/2412 objects degraded (740.879%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:21.669 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:21 smithi098 ceph-mon[100914]: pgmap v1346: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 53 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 684 op/s; 17870/2412 objects degraded (740.879%); 0 B/s, 25 objects/s recovering 2024-09-21T10:31:21.670 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:21 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 17870/2412 objects degraded (740.879%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:21.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:21.758 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:21.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:21.859 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:21.882 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:21.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:21.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:21.984 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:22.010 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:22.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:22.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:22.112 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:22.134 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:22.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:22.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:22.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:22.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:22.262 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:22.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:22.363 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:22.385 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:22.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:22.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:22.487 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:22.512 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:22.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:22.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:22.585 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:22.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:22.632 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:22.640 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:22.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:22.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:22.742 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:22.768 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:22.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:22.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:22.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:31:22.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:31:22.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:22 smithi098 ceph-mon[100914]: pgmap v1347: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 630 op/s; 17548/81 objects degraded (21664.198%); 0 B/s, 45 objects/s recovering 2024-09-21T10:31:22.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:31:22.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:22.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:31:22.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:31:22.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:22 smithi098 ceph-mon[97580]: pgmap v1347: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 630 op/s; 17548/81 objects degraded (21664.198%); 0 B/s, 45 objects/s recovering 2024-09-21T10:31:22.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:31:22.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:22.869 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:31:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:31:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:22 smithi196 ceph-mon[84784]: pgmap v1347: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 630 op/s; 17548/81 objects degraded (21664.198%); 0 B/s, 45 objects/s recovering 2024-09-21T10:31:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:31:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:22.891 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:22.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:22.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:22.992 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:23.013 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:23.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:23.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:23.115 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:23.137 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:23.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:23.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:23.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:23.260 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:23.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:23.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:23.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:23.383 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:23.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:23.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:23.485 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:23.510 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:23.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:23.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:23.611 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:23.633 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:23.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:23.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:23.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:23.756 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:23.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:23.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:23.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:23.879 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:23.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:23.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:23.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:24.006 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:24.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:24.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:24.108 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:24.129 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:24.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:24.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:24.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:31:24.231 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:24.254 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:24.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:24.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:24.356 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:24.380 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:24.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:24.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:24.482 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:24.505 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:24.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:24.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:24.607 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:24.628 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:24.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:24.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:24.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:24.754 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:24.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:24.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:24.856 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:24.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:24.882 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:24.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:24.983 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:25.004 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:25.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:25.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:25.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:24 smithi098 ceph-mon[97580]: pgmap v1348: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 560 op/s; 17548/81 objects degraded (21664.198%); 0 B/s, 40 objects/s recovering 2024-09-21T10:31:25.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:24 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:31:25.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:24 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:31:25.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:24 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:31:25.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:24 smithi098 ceph-mon[100914]: pgmap v1348: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 560 op/s; 17548/81 objects degraded (21664.198%); 0 B/s, 40 objects/s recovering 2024-09-21T10:31:25.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:24 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:31:25.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:24 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:31:25.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:24 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:31:25.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:25.128 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:25.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:25.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:25.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:24 smithi196 ceph-mon[84784]: pgmap v1348: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 560 op/s; 17548/81 objects degraded (21664.198%); 0 B/s, 40 objects/s recovering 2024-09-21T10:31:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:24 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:31:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:24 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:31:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:24 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:31:25.229 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:25.252 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:25.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:25.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:25.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:25.380 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:25.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:25.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:25.481 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:25.506 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:25.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:25.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:25.608 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:25.629 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:25.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:25.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:25.731 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:25.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:25.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:25.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:25.859 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:25.880 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:25.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:25.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:25.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:26.003 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:26.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:26.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:26.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:26.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:26.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:26.229 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:26.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:26.255 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:26.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:26.356 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:26.378 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:26.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:26.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:26.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:26.501 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:26.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:26.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:26.603 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:26.628 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:26.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:26.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:26.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:26.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:26.758 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:26.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:26.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:26.884 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:26.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:26.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:26.986 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:27.011 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:27.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:27.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:26 smithi098 ceph-mon[97580]: pgmap v1349: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 350 op/s; 16898/81 objects degraded (20861.728%); 0 B/s, 91 objects/s recovering 2024-09-21T10:31:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:26 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 16898/81 objects degraded (20861.728%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:27.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:26 smithi098 ceph-mon[100914]: pgmap v1349: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 350 op/s; 16898/81 objects degraded (20861.728%); 0 B/s, 91 objects/s recovering 2024-09-21T10:31:27.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:26 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 16898/81 objects degraded (20861.728%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:27.112 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:27.137 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:27.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:27.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:26 smithi196 ceph-mon[84784]: pgmap v1349: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 350 op/s; 16898/81 objects degraded (20861.728%); 0 B/s, 91 objects/s recovering 2024-09-21T10:31:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:26 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 16898/81 objects degraded (20861.728%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:27.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:27.265 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:27.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:27.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:27.367 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:27.388 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:27.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:27.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:27.490 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:27.511 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:27.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:27.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:27.613 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:27.638 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:27.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:27.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:27.740 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:27.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:27.766 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:27.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:27.867 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:27.892 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:27.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:27.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:27.994 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:28.019 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:28.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:28.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:28.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:28.143 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:28.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:28.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:28.244 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:28.265 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:28.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:28.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:28.367 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:28.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:28.393 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:28.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:28.494 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:28.518 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:28.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:28.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:28.619 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:28.641 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:28.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:28.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:28.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:28.764 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:28.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:28.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:28.865 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:28.891 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:28.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:28.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:28.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:29.018 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:29.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:29.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:29.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:28 smithi098 ceph-mon[97580]: pgmap v1350: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 284 op/s; 16898/81 objects degraded (20861.728%); 0 B/s, 80 objects/s recovering 2024-09-21T10:31:29.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:28 smithi098 ceph-mon[100914]: pgmap v1350: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 284 op/s; 16898/81 objects degraded (20861.728%); 0 B/s, 80 objects/s recovering 2024-09-21T10:31:29.120 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:29.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:28 smithi196 ceph-mon[84784]: pgmap v1350: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 284 op/s; 16898/81 objects degraded (20861.728%); 0 B/s, 80 objects/s recovering 2024-09-21T10:31:29.146 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:29.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:29.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:29.248 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:29.270 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:29.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:29.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:29.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:29.393 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:29.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:29.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:29.495 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:29.521 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:29.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:29.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:29.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:29.644 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:29.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:29.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:29.746 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:29.766 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:29.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:29.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:29.868 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:29.889 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:29.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:29.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:29.991 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:30.017 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:30.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:30.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:30.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:30.142 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:30.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:30.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:30.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:30.269 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:30.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:30.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:30.371 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:30.397 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:30.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:30.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:30.499 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:30.524 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:30.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:30.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:30.626 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:30.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:31:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:31:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:31:30.648 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:30.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:30.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:30.751 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:30.774 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:30.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:30.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:30.875 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:30.901 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:30.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:30.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:31.002 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:31.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:31.028 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:31.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:31.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:30 smithi098 ceph-mon[97580]: pgmap v1351: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 248 op/s; 16898/81 objects degraded (20861.728%); 0 B/s, 80 objects/s recovering 2024-09-21T10:31:31.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:31:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:31:30] "GET /metrics HTTP/1.1" 200 36589 "" "Prometheus/2.51.0" 2024-09-21T10:31:31.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:30 smithi098 ceph-mon[100914]: pgmap v1351: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 248 op/s; 16898/81 objects degraded (20861.728%); 0 B/s, 80 objects/s recovering 2024-09-21T10:31:31.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:31.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:30 smithi196 ceph-mon[84784]: pgmap v1351: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 248 op/s; 16898/81 objects degraded (20861.728%); 0 B/s, 80 objects/s recovering 2024-09-21T10:31:31.153 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:31.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:31.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:31.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:31.276 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:31.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:31.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:31.308 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:31:31.308 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:31:31.378 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:31.403 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:31.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:31.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:31.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:31.527 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:31.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:31.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:31.628 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:31.654 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:31.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:31.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:31.755 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:31.778 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:31.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:31.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:31.880 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:31.905 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:31.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:31.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:32.007 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:32.032 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:32.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:32.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:32.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:31 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 16038/81 objects degraded (19800.000%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:32.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:31 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 16038/81 objects degraded (19800.000%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:32.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:31:32.133 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:31 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 16038/81 objects degraded (19800.000%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:32.159 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:32.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:32.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:32.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:32.283 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:32.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:32.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:32.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:32.409 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:32.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:32.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:32.510 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:32.532 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:32.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:32.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:32.633 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:32.655 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:32.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:32.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:32.756 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:32.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:32.779 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:32.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:32.880 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:32.901 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:32.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:32.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:33.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:33.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:33.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:33.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:32 smithi098 ceph-mon[97580]: pgmap v1352: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 64 op/s; 16038/81 objects degraded (19800.000%); 0 B/s, 152 objects/s recovering 2024-09-21T10:31:33.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:32 smithi098 ceph-mon[100914]: pgmap v1352: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 64 op/s; 16038/81 objects degraded (19800.000%); 0 B/s, 152 objects/s recovering 2024-09-21T10:31:33.126 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:33.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:32 smithi196 ceph-mon[84784]: pgmap v1352: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 64 op/s; 16038/81 objects degraded (19800.000%); 0 B/s, 152 objects/s recovering 2024-09-21T10:31:33.148 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:33.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:33.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:33.250 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:33.275 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:33.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:33.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:33.377 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:33.402 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:33.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:33.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:33.504 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:33.529 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:33.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:33.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:33.631 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:33.654 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:33.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:33.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:33.756 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:33.781 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:33.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:33.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:33.883 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:33.908 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:33.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:33.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:34.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:34.033 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:34.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:34.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:34.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:34.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:34.157 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:34.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:34.258 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:34.279 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:34.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:34.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:34.381 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:34.402 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:34.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:34.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:34.454 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:34.500 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:34.504 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:34.529 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:34.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:34.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:34.624 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:31:34.625 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:31:34.631 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:34.654 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:34.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:34.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:34.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:34.774 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-21T10:31:34.782 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:34.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:34.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:34.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:34 smithi098 ceph-mon[97580]: pgmap v1353: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 16038/81 objects degraded (19800.000%); 0 B/s, 125 objects/s recovering 2024-09-21T10:31:34.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:34 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2614629810' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:31:34.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:34 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:31:34.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:34 smithi098 ceph-mon[100914]: pgmap v1353: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 16038/81 objects degraded (19800.000%); 0 B/s, 125 objects/s recovering 2024-09-21T10:31:34.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:34 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2614629810' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:31:34.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:34 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:31:34.884 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:34.905 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:34.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:34.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:35.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:35.029 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:35.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:35.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:35.131 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:35.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:34 smithi196 ceph-mon[84784]: pgmap v1353: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 16038/81 objects degraded (19800.000%); 0 B/s, 125 objects/s recovering 2024-09-21T10:31:35.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:34 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2614629810' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:31:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:34 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:31:35.154 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:35.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:35.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:35.255 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:35.281 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:35.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:35.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:35.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:35.405 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:35.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:35.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:35.507 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:35.515 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1353,"stamp":"2024-09-21T10:31:33.652028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4516211,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":6012,"num_read_kb":13076,"num_write":250967,"num_write_kb":8014180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5316,"num_bytes_recovered":200735698,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129378,"ondisk_log_size":129378,"up":297,"acting":294,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":302,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8267788,"kb_used_data":1410764,"kb_used_omap":35,"kb_used_meta":6856924,"kb_avail":741693428,"statfs":{"total":767960285184,"available":759494070272,"internally_reserved":0,"allocated":1444622336,"data_stored":1427029390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36353,"internal_metadata":7021490687},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1510,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1510,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002727"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":59,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.395586+0000","last_change":"2024-09-21T10:28:00.289895+0000","last_active":"2024-09-21T10:31:12.395586+0000","last_peered":"2024-09-21T10:31:12.395586+0000","last_clean":"2024-09-21T10:31:12.395586+0000","last_became_active":"2024-09-21T10:28:00.289622+0000","last_became_peered":"2024-09-21T10:28:00.289622+0000","last_unstale":"2024-09-21T10:31:12.395586+0000","last_undegraded":"2024-09-21T10:31:12.395586+0000","last_fullsized":"2024-09-21T10:31:12.395586+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:35:31.514813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":414,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.349137+0000","last_change":"2024-09-21T10:24:44.305198+0000","last_active":"2024-09-21T10:30:58.349137+0000","last_peered":"2024-09-21T10:30:58.349137+0000","last_clean":"2024-09-21T10:30:58.349137+0000","last_became_active":"2024-09-21T10:24:44.297653+0000","last_became_peered":"2024-09-21T10:24:44.297653+0000","last_unstale":"2024-09-21T10:30:58.349137+0000","last_undegraded":"2024-09-21T10:30:58.349137+0000","last_fullsized":"2024-09-21T10:30:58.349137+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:19.320833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":251,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.332541+0000","last_change":"2024-09-21T10:28:00.294917+0000","last_active":"2024-09-21T10:30:58.332541+0000","last_peered":"2024-09-21T10:30:58.332541+0000","last_clean":"2024-09-21T10:30:58.332541+0000","last_became_active":"2024-09-21T10:28:00.294134+0000","last_became_peered":"2024-09-21T10:28:00.294134+0000","last_unstale":"2024-09-21T10:30:58.332541+0000","last_undegraded":"2024-09-21T10:30:58.332541+0000","last_fullsized":"2024-09-21T10:30:58.332541+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:29:05.780212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":200,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.349027+0000","last_change":"2024-09-21T10:28:00.287636+0000","last_active":"2024-09-21T10:30:58.349027+0000","last_peered":"2024-09-21T10:30:58.349027+0000","last_clean":"2024-09-21T10:30:58.349027+0000","last_became_active":"2024-09-21T10:28:00.287308+0000","last_became_peered":"2024-09-21T10:28:00.287308+0000","last_unstale":"2024-09-21T10:30:58.349027+0000","last_undegraded":"2024-09-21T10:30:58.349027+0000","last_fullsized":"2024-09-21T10:30:58.349027+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:41:46.132365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":655,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.398782+0000","last_change":"2024-09-21T10:28:00.297783+0000","last_active":"2024-09-21T10:31:12.398782+0000","last_peered":"2024-09-21T10:31:12.398782+0000","last_clean":"2024-09-21T10:31:12.398782+0000","last_became_active":"2024-09-21T10:28:00.297024+0000","last_became_peered":"2024-09-21T10:28:00.297024+0000","last_unstale":"2024-09-21T10:31:12.398782+0000","last_undegraded":"2024-09-21T10:31:12.398782+0000","last_fullsized":"2024-09-21T10:31:12.398782+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:24.382523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":73,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347441+0000","last_change":"2024-09-21T10:24:44.304399+0000","last_active":"2024-09-21T10:30:58.347441+0000","last_peered":"2024-09-21T10:30:58.347441+0000","last_clean":"2024-09-21T10:30:58.347441+0000","last_became_active":"2024-09-21T10:24:44.303437+0000","last_became_peered":"2024-09-21T10:24:44.303437+0000","last_unstale":"2024-09-21T10:30:58.347441+0000","last_undegraded":"2024-09-21T10:30:58.347441+0000","last_fullsized":"2024-09-21T10:30:58.347441+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:32:56.638481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":59,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.397534+0000","last_change":"2024-09-21T10:28:00.293505+0000","last_active":"2024-09-21T10:31:12.397534+0000","last_peered":"2024-09-21T10:31:12.397534+0000","last_clean":"2024-09-21T10:31:12.397534+0000","last_became_active":"2024-09-21T10:28:00.293173+0000","last_became_peered":"2024-09-21T10:28:00.293173+0000","last_unstale":"2024-09-21T10:31:12.397534+0000","last_undegraded":"2024-09-21T10:31:12.397534+0000","last_fullsized":"2024-09-21T10:31:12.397534+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:16.878847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":356,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.345680+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:30:58.345680+0000","last_peered":"2024-09-21T10:30:58.345680+0000","last_clean":"2024-09-21T10:30:58.345680+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:30:58.345680+0000","last_undegraded":"2024-09-21T10:30:58.345680+0000","last_fullsized":"2024-09-21T10:30:58.345680+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"384'15658","reported_seq":15744,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:18.047479+0000","last_change":"2024-09-21T10:26:36.989699+0000","last_active":"2024-09-21T10:31:18.047479+0000","last_peered":"2024-09-21T10:31:18.047479+0000","last_clean":"2024-09-21T10:31:18.047479+0000","last_became_active":"2024-09-21T10:26:36.989318+0000","last_became_peered":"2024-09-21T10:26:36.989318+0000","last_unstale":"2024-09-21T10:31:18.047479+0000","last_undegraded":"2024-09-21T10:31:18.047479+0000","last_fullsized":"2024-09-21T10:31:18.047479+0000","mapping_epoch":348,"log_start":"366'8300","ondisk_log_start":"366'8300","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7358,"log_dups_size":0,"ondisk_log_size":7358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:54:53.567344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15658,"num_write_kb":501056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":405,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.349927+0000","last_change":"2024-09-21T10:24:44.325196+0000","last_active":"2024-09-21T10:30:58.349927+0000","last_peered":"2024-09-21T10:30:58.349927+0000","last_clean":"2024-09-21T10:30:58.349927+0000","last_became_active":"2024-09-21T10:24:44.299614+0000","last_became_peered":"2024-09-21T10:24:44.299614+0000","last_unstale":"2024-09-21T10:30:58.349927+0000","last_undegraded":"2024-09-21T10:30:58.349927+0000","last_fullsized":"2024-09-21T10:30:58.349927+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:14.558663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":137,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.332428+0000","last_change":"2024-09-21T10:26:41.305571+0000","last_active":"2024-09-21T10:30:58.332428+0000","last_peered":"2024-09-21T10:30:58.332428+0000","last_clean":"2024-09-21T10:30:58.332428+0000","last_became_active":"2024-09-21T10:26:41.305135+0000","last_became_peered":"2024-09-21T10:26:41.305135+0000","last_unstale":"2024-09-21T10:30:58.332428+0000","last_undegraded":"2024-09-21T10:30:58.332428+0000","last_fullsized":"2024-09-21T10:30:58.332428+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:49:55.452467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"384'15438","reported_seq":15493,"reported_epoch":384,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:31:17.969575+0000","last_change":"2024-09-21T10:28:01.438839+0000","last_active":"2024-09-21T10:31:17.969575+0000","last_peered":"2024-09-21T10:31:17.969575+0000","last_clean":"2024-09-21T10:27:59.232048+0000","last_became_active":"2024-09-21T10:28:01.433549+0000","last_became_peered":"2024-09-21T10:28:01.433549+0000","last_unstale":"2024-09-21T10:31:17.969575+0000","last_undegraded":"2024-09-21T10:31:17.969575+0000","last_fullsized":"2024-09-21T10:31:17.969575+0000","mapping_epoch":361,"log_start":"365'7700","ondisk_log_start":"365'7700","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7738,"log_dups_size":0,"ondisk_log_size":7738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:58.936666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15438,"num_write_kb":494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,5,2],"avail_no_missing":["6","2","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":406,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.349580+0000","last_change":"2024-09-21T10:24:44.323182+0000","last_active":"2024-09-21T10:30:58.349580+0000","last_peered":"2024-09-21T10:30:58.349580+0000","last_clean":"2024-09-21T10:30:58.349580+0000","last_became_active":"2024-09-21T10:24:44.299698+0000","last_became_peered":"2024-09-21T10:24:44.299698+0000","last_unstale":"2024-09-21T10:30:58.349580+0000","last_undegraded":"2024-09-21T10:30:58.349580+0000","last_fullsized":"2024-09-21T10:30:58.349580+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:56.571057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":83,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347651+0000","last_change":"2024-09-21T10:28:00.281253+0000","last_active":"2024-09-21T10:30:58.347651+0000","last_peered":"2024-09-21T10:30:58.347651+0000","last_clean":"2024-09-21T10:30:58.347651+0000","last_became_active":"2024-09-21T10:28:00.280773+0000","last_became_peered":"2024-09-21T10:28:00.280773+0000","last_unstale":"2024-09-21T10:30:58.347651+0000","last_undegraded":"2024-09-21T10:30:58.347651+0000","last_fullsized":"2024-09-21T10:30:58.347651+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:32:48.557279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"384'15478","reported_seq":25873,"reported_epoch":384,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T10:31:29.914759+0000","last_change":"2024-09-21T10:28:25.623040+0000","last_active":"2024-09-21T10:31:29.914759+0000","last_peered":"2024-09-21T10:31:29.914759+0000","last_clean":"2024-09-21T10:27:59.294389+0000","last_became_active":"2024-09-21T10:28:01.608572+0000","last_became_peered":"2024-09-21T10:28:01.608572+0000","last_unstale":"2024-09-21T10:31:29.914759+0000","last_undegraded":"2024-09-21T10:28:25.614349+0000","last_fullsized":"2024-09-21T10:28:01.272643+0000","mapping_epoch":361,"log_start":"366'8100","ondisk_log_start":"366'8100","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7378,"log_dups_size":0,"ondisk_log_size":7378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1875,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15478,"num_write_kb":495296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5153,"num_bytes_recovered":197132288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,4","objects":-1875}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":332,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.349430+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:30:58.349430+0000","last_peered":"2024-09-21T10:30:58.349430+0000","last_clean":"2024-09-21T10:30:58.349430+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:30:58.349430+0000","last_undegraded":"2024-09-21T10:30:58.349430+0000","last_fullsized":"2024-09-21T10:30:58.349430+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":347,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.345535+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:30:58.345535+0000","last_peered":"2024-09-21T10:30:58.345535+0000","last_clean":"2024-09-21T10:30:58.345535+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:30:58.345535+0000","last_undegraded":"2024-09-21T10:30:58.345535+0000","last_fullsized":"2024-09-21T10:30:58.345535+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"384'15378","reported_seq":15467,"reported_epoch":384,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:31:18.061827+0000","last_change":"2024-09-21T10:28:01.644360+0000","last_active":"2024-09-21T10:31:18.061827+0000","last_peered":"2024-09-21T10:31:18.061827+0000","last_clean":"2024-09-21T10:27:59.276032+0000","last_became_active":"2024-09-21T10:28:01.624219+0000","last_became_peered":"2024-09-21T10:28:01.624219+0000","last_unstale":"2024-09-21T10:31:18.061827+0000","last_undegraded":"2024-09-21T10:28:01.350374+0000","last_fullsized":"2024-09-21T10:28:01.295088+0000","mapping_epoch":361,"log_start":"358'6000","ondisk_log_start":"358'6000","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":9378,"log_dups_size":0,"ondisk_log_size":9378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15378,"num_write_kb":492096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0],"avail_no_missing":["2","0","7"],"object_location_counts":[{"shards":"0,2","objects":-7014}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":59,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.395737+0000","last_change":"2024-09-21T10:28:00.288968+0000","last_active":"2024-09-21T10:31:12.395737+0000","last_peered":"2024-09-21T10:31:12.395737+0000","last_clean":"2024-09-21T10:31:12.395737+0000","last_became_active":"2024-09-21T10:28:00.288244+0000","last_became_peered":"2024-09-21T10:28:00.288244+0000","last_unstale":"2024-09-21T10:31:12.395737+0000","last_undegraded":"2024-09-21T10:31:12.395737+0000","last_fullsized":"2024-09-21T10:31:12.395737+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:42.089701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":677,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.398707+0000","last_change":"2024-09-21T10:24:44.299417+0000","last_active":"2024-09-21T10:31:12.398707+0000","last_peered":"2024-09-21T10:31:12.398707+0000","last_clean":"2024-09-21T10:31:12.398707+0000","last_became_active":"2024-09-21T10:24:44.299116+0000","last_became_peered":"2024-09-21T10:24:44.299116+0000","last_unstale":"2024-09-21T10:31:12.398707+0000","last_undegraded":"2024-09-21T10:31:12.398707+0000","last_fullsized":"2024-09-21T10:31:12.398707+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:44.524841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"384'15864","reported_seq":15950,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:18.006194+0000","last_change":"2024-09-21T10:26:36.196642+0000","last_active":"2024-09-21T10:31:18.006194+0000","last_peered":"2024-09-21T10:31:18.006194+0000","last_clean":"2024-09-21T10:31:18.006194+0000","last_became_active":"2024-09-21T10:26:36.196357+0000","last_became_peered":"2024-09-21T10:26:36.196357+0000","last_unstale":"2024-09-21T10:31:18.006194+0000","last_undegraded":"2024-09-21T10:31:18.006194+0000","last_fullsized":"2024-09-21T10:31:18.006194+0000","mapping_epoch":348,"log_start":"359'7200","ondisk_log_start":"359'7200","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":8664,"log_dups_size":0,"ondisk_log_size":8664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:54:19.391031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15864,"num_write_kb":507648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"82'100","reported_seq":505,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.351242+0000","last_change":"2024-09-21T10:28:00.285919+0000","last_active":"2024-09-21T10:30:58.351242+0000","last_peered":"2024-09-21T10:30:58.351242+0000","last_clean":"2024-09-21T10:30:58.351242+0000","last_became_active":"2024-09-21T10:28:00.285324+0000","last_became_peered":"2024-09-21T10:28:00.285324+0000","last_unstale":"2024-09-21T10:30:58.351242+0000","last_undegraded":"2024-09-21T10:30:58.351242+0000","last_fullsized":"2024-09-21T10:30:58.351242+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.728195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.728195+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:00.841610+0000","scrub_duration":0.0069698140000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":63,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.396500+0000","last_change":"2024-09-21T10:28:00.303125+0000","last_active":"2024-09-21T10:31:12.396500+0000","last_peered":"2024-09-21T10:31:12.396500+0000","last_clean":"2024-09-21T10:31:12.396500+0000","last_became_active":"2024-09-21T10:28:00.293684+0000","last_became_peered":"2024-09-21T10:28:00.293684+0000","last_unstale":"2024-09-21T10:31:12.396500+0000","last_undegraded":"2024-09-21T10:31:12.396500+0000","last_fullsized":"2024-09-21T10:31:12.396500+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:04.495913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":83,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347910+0000","last_change":"2024-09-21T10:28:00.283456+0000","last_active":"2024-09-21T10:30:58.347910+0000","last_peered":"2024-09-21T10:30:58.347910+0000","last_clean":"2024-09-21T10:30:58.347910+0000","last_became_active":"2024-09-21T10:28:00.282881+0000","last_became_peered":"2024-09-21T10:28:00.282881+0000","last_unstale":"2024-09-21T10:30:58.347910+0000","last_undegraded":"2024-09-21T10:30:58.347910+0000","last_fullsized":"2024-09-21T10:30:58.347910+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:43.681481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"384'15744","reported_seq":15833,"reported_epoch":384,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:31:18.063589+0000","last_change":"2024-09-21T10:28:01.680872+0000","last_active":"2024-09-21T10:31:18.063589+0000","last_peered":"2024-09-21T10:31:18.063589+0000","last_clean":"2024-09-21T10:27:59.257182+0000","last_became_active":"2024-09-21T10:28:01.651875+0000","last_became_peered":"2024-09-21T10:28:01.651875+0000","last_unstale":"2024-09-21T10:31:18.063589+0000","last_undegraded":"2024-09-21T10:28:01.349133+0000","last_fullsized":"2024-09-21T10:28:01.272587+0000","mapping_epoch":361,"log_start":"358'6300","ondisk_log_start":"358'6300","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":9444,"log_dups_size":0,"ondisk_log_size":9444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15744,"num_write_kb":503808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5],"avail_no_missing":["2","4","5"],"object_location_counts":[{"shards":"2,5","objects":-7149}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'100","reported_seq":179,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.391161+0000","last_change":"2024-09-21T10:28:00.301695+0000","last_active":"2024-09-21T10:31:12.391161+0000","last_peered":"2024-09-21T10:31:12.391161+0000","last_clean":"2024-09-21T10:31:12.391161+0000","last_became_active":"2024-09-21T10:28:00.289856+0000","last_became_peered":"2024-09-21T10:28:00.289856+0000","last_unstale":"2024-09-21T10:31:12.391161+0000","last_undegraded":"2024-09-21T10:31:12.391161+0000","last_fullsized":"2024-09-21T10:31:12.391161+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:27.234001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:27.234001+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:02.389036+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.345736+0000","last_change":"2024-09-21T10:24:44.309470+0000","last_active":"2024-09-21T10:30:58.345736+0000","last_peered":"2024-09-21T10:30:58.345736+0000","last_clean":"2024-09-21T10:30:58.345736+0000","last_became_active":"2024-09-21T10:24:44.308836+0000","last_became_peered":"2024-09-21T10:24:44.308836+0000","last_unstale":"2024-09-21T10:30:58.345736+0000","last_undegraded":"2024-09-21T10:30:58.345736+0000","last_fullsized":"2024-09-21T10:30:58.345736+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:09:44.622418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":73,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.348273+0000","last_change":"2024-09-21T10:24:44.305135+0000","last_active":"2024-09-21T10:30:58.348273+0000","last_peered":"2024-09-21T10:30:58.348273+0000","last_clean":"2024-09-21T10:30:58.348273+0000","last_became_active":"2024-09-21T10:24:44.304839+0000","last_became_peered":"2024-09-21T10:24:44.304839+0000","last_unstale":"2024-09-21T10:30:58.348273+0000","last_undegraded":"2024-09-21T10:30:58.348273+0000","last_fullsized":"2024-09-21T10:30:58.348273+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:02.264404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"384'15516","reported_seq":15560,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:18.013018+0000","last_change":"2024-09-21T10:26:36.194994+0000","last_active":"2024-09-21T10:31:18.013018+0000","last_peered":"2024-09-21T10:31:18.013018+0000","last_clean":"2024-09-21T10:31:18.013018+0000","last_became_active":"2024-09-21T10:26:36.194404+0000","last_became_peered":"2024-09-21T10:26:36.194404+0000","last_unstale":"2024-09-21T10:31:18.013018+0000","last_undegraded":"2024-09-21T10:31:18.013018+0000","last_fullsized":"2024-09-21T10:31:18.013018+0000","mapping_epoch":348,"log_start":"366'7800","ondisk_log_start":"366'7800","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7716,"log_dups_size":0,"ondisk_log_size":7716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:10:51.551059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15516,"num_write_kb":496512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":108,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.412880+0000","last_change":"2024-09-21T10:25:39.872680+0000","last_active":"2024-09-21T10:31:12.412880+0000","last_peered":"2024-09-21T10:31:12.412880+0000","last_clean":"2024-09-21T10:31:12.412880+0000","last_became_active":"2024-09-21T10:25:39.872312+0000","last_became_peered":"2024-09-21T10:25:39.872312+0000","last_unstale":"2024-09-21T10:31:12.412880+0000","last_undegraded":"2024-09-21T10:31:12.412880+0000","last_fullsized":"2024-09-21T10:31:12.412880+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:54.942046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":73,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.348168+0000","last_change":"2024-09-21T10:24:44.310484+0000","last_active":"2024-09-21T10:30:58.348168+0000","last_peered":"2024-09-21T10:30:58.348168+0000","last_clean":"2024-09-21T10:30:58.348168+0000","last_became_active":"2024-09-21T10:24:44.307209+0000","last_became_peered":"2024-09-21T10:24:44.307209+0000","last_unstale":"2024-09-21T10:30:58.348168+0000","last_undegraded":"2024-09-21T10:30:58.348168+0000","last_fullsized":"2024-09-21T10:30:58.348168+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:48:36.767539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"384'15640","reported_seq":15684,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:18.028585+0000","last_change":"2024-09-21T10:26:36.196826+0000","last_active":"2024-09-21T10:31:18.028585+0000","last_peered":"2024-09-21T10:31:18.028585+0000","last_clean":"2024-09-21T10:31:18.028585+0000","last_became_active":"2024-09-21T10:26:36.196355+0000","last_became_peered":"2024-09-21T10:26:36.196355+0000","last_unstale":"2024-09-21T10:31:18.028585+0000","last_undegraded":"2024-09-21T10:31:18.028585+0000","last_fullsized":"2024-09-21T10:31:18.028585+0000","mapping_epoch":348,"log_start":"368'8900","ondisk_log_start":"368'8900","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":6740,"log_dups_size":0,"ondisk_log_size":6740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:58.869309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15640,"num_write_kb":500480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'100","reported_seq":163,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.391017+0000","last_change":"2024-09-21T10:23:26.197358+0000","last_active":"2024-09-21T10:31:12.391017+0000","last_peered":"2024-09-21T10:31:12.391017+0000","last_clean":"2024-09-21T10:31:12.391017+0000","last_became_active":"2024-09-21T10:23:20.695174+0000","last_became_peered":"2024-09-21T10:23:20.695174+0000","last_unstale":"2024-09-21T10:31:12.391017+0000","last_undegraded":"2024-09-21T10:31:12.391017+0000","last_fullsized":"2024-09-21T10:31:12.391017+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:26.197291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:26.197291+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:33.961031+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":59,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.395951+0000","last_change":"2024-09-21T10:28:00.287560+0000","last_active":"2024-09-21T10:31:12.395951+0000","last_peered":"2024-09-21T10:31:12.395951+0000","last_clean":"2024-09-21T10:31:12.395951+0000","last_became_active":"2024-09-21T10:28:00.285076+0000","last_became_peered":"2024-09-21T10:28:00.285076+0000","last_unstale":"2024-09-21T10:31:12.395951+0000","last_undegraded":"2024-09-21T10:31:12.395951+0000","last_fullsized":"2024-09-21T10:31:12.395951+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:49.950523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":672,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.393489+0000","last_change":"2024-09-21T10:28:00.282062+0000","last_active":"2024-09-21T10:31:12.393489+0000","last_peered":"2024-09-21T10:31:12.393489+0000","last_clean":"2024-09-21T10:31:12.393489+0000","last_became_active":"2024-09-21T10:28:00.281593+0000","last_became_peered":"2024-09-21T10:28:00.281593+0000","last_unstale":"2024-09-21T10:31:12.393489+0000","last_undegraded":"2024-09-21T10:31:12.393489+0000","last_fullsized":"2024-09-21T10:31:12.393489+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:45.704840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"384'15608","reported_seq":15652,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:18.048770+0000","last_change":"2024-09-21T10:26:36.197465+0000","last_active":"2024-09-21T10:31:18.048770+0000","last_peered":"2024-09-21T10:31:18.048770+0000","last_clean":"2024-09-21T10:31:18.048770+0000","last_became_active":"2024-09-21T10:26:36.196105+0000","last_became_peered":"2024-09-21T10:26:36.196105+0000","last_unstale":"2024-09-21T10:31:18.048770+0000","last_undegraded":"2024-09-21T10:31:18.048770+0000","last_fullsized":"2024-09-21T10:31:18.048770+0000","mapping_epoch":348,"log_start":"366'7900","ondisk_log_start":"366'7900","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7708,"log_dups_size":0,"ondisk_log_size":7708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:16.586685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15608,"num_write_kb":499456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":486,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.349478+0000","last_change":"2024-09-21T10:23:18.790100+0000","last_active":"2024-09-21T10:30:58.349478+0000","last_peered":"2024-09-21T10:30:58.349478+0000","last_clean":"2024-09-21T10:30:58.349478+0000","last_became_active":"2024-09-21T10:23:14.677610+0000","last_became_peered":"2024-09-21T10:23:14.677610+0000","last_unstale":"2024-09-21T10:30:58.349478+0000","last_undegraded":"2024-09-21T10:30:58.349478+0000","last_fullsized":"2024-09-21T10:30:58.349478+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:18.790017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:18.790017+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:56.562241+0000","scrub_duration":0.0080400790000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":579,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.332792+0000","last_change":"2024-09-21T10:26:41.279374+0000","last_active":"2024-09-21T10:30:58.332792+0000","last_peered":"2024-09-21T10:30:58.332792+0000","last_clean":"2024-09-21T10:30:58.332792+0000","last_became_active":"2024-09-21T10:26:41.278994+0000","last_became_peered":"2024-09-21T10:26:41.278994+0000","last_unstale":"2024-09-21T10:30:58.332792+0000","last_undegraded":"2024-09-21T10:30:58.332792+0000","last_fullsized":"2024-09-21T10:30:58.332792+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:49:57.821175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":482,"num_read_kb":1581,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":59,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.396519+0000","last_change":"2024-09-21T10:28:00.295436+0000","last_active":"2024-09-21T10:31:12.396519+0000","last_peered":"2024-09-21T10:31:12.396519+0000","last_clean":"2024-09-21T10:31:12.396519+0000","last_became_active":"2024-09-21T10:28:00.295135+0000","last_became_peered":"2024-09-21T10:28:00.295135+0000","last_unstale":"2024-09-21T10:31:12.396519+0000","last_undegraded":"2024-09-21T10:31:12.396519+0000","last_fullsized":"2024-09-21T10:31:12.396519+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:14:03.027395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":370,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.345753+0000","last_change":"2024-09-21T10:24:44.308008+0000","last_active":"2024-09-21T10:30:58.345753+0000","last_peered":"2024-09-21T10:30:58.345753+0000","last_clean":"2024-09-21T10:30:58.345753+0000","last_became_active":"2024-09-21T10:24:44.307595+0000","last_became_peered":"2024-09-21T10:24:44.307595+0000","last_unstale":"2024-09-21T10:30:58.345753+0000","last_undegraded":"2024-09-21T10:30:58.345753+0000","last_fullsized":"2024-09-21T10:30:58.345753+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:33:49.448676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"384'15478","reported_seq":15564,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:18.060227+0000","last_change":"2024-09-21T10:26:36.197089+0000","last_active":"2024-09-21T10:31:18.060227+0000","last_peered":"2024-09-21T10:31:18.060227+0000","last_clean":"2024-09-21T10:31:18.060227+0000","last_became_active":"2024-09-21T10:26:36.196846+0000","last_became_peered":"2024-09-21T10:26:36.196846+0000","last_unstale":"2024-09-21T10:31:18.060227+0000","last_undegraded":"2024-09-21T10:31:18.060227+0000","last_fullsized":"2024-09-21T10:31:18.060227+0000","mapping_epoch":348,"log_start":"366'8100","ondisk_log_start":"366'8100","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7378,"log_dups_size":0,"ondisk_log_size":7378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:14:52.494647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15478,"num_write_kb":495296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"82'100","reported_seq":170,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.393718+0000","last_change":"2024-09-21T10:24:44.309154+0000","last_active":"2024-09-21T10:31:12.393718+0000","last_peered":"2024-09-21T10:31:12.393718+0000","last_clean":"2024-09-21T10:31:12.393718+0000","last_became_active":"2024-09-21T10:24:44.308330+0000","last_became_peered":"2024-09-21T10:24:44.308330+0000","last_unstale":"2024-09-21T10:31:12.393718+0000","last_undegraded":"2024-09-21T10:31:12.393718+0000","last_fullsized":"2024-09-21T10:31:12.393718+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.641718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.641718+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:10.158739+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":728,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.349397+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:30:58.349397+0000","last_peered":"2024-09-21T10:30:58.349397+0000","last_clean":"2024-09-21T10:30:58.349397+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:30:58.349397+0000","last_undegraded":"2024-09-21T10:30:58.349397+0000","last_fullsized":"2024-09-21T10:30:58.349397+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":685,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.413104+0000","last_change":"2024-09-21T10:28:00.296831+0000","last_active":"2024-09-21T10:31:12.413104+0000","last_peered":"2024-09-21T10:31:12.413104+0000","last_clean":"2024-09-21T10:31:12.413104+0000","last_became_active":"2024-09-21T10:28:00.296527+0000","last_became_peered":"2024-09-21T10:28:00.296527+0000","last_unstale":"2024-09-21T10:31:12.413104+0000","last_undegraded":"2024-09-21T10:31:12.413104+0000","last_fullsized":"2024-09-21T10:31:12.413104+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:18.024868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":653,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.398907+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:31:12.398907+0000","last_peered":"2024-09-21T10:31:12.398907+0000","last_clean":"2024-09-21T10:31:12.398907+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:31:12.398907+0000","last_undegraded":"2024-09-21T10:31:12.398907+0000","last_fullsized":"2024-09-21T10:31:12.398907+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"384'16060","reported_seq":16105,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:18.043731+0000","last_change":"2024-09-21T10:26:41.278294+0000","last_active":"2024-09-21T10:31:18.043731+0000","last_peered":"2024-09-21T10:31:18.043731+0000","last_clean":"2024-09-21T10:31:18.043731+0000","last_became_active":"2024-09-21T10:26:41.277965+0000","last_became_peered":"2024-09-21T10:26:41.277965+0000","last_unstale":"2024-09-21T10:31:18.043731+0000","last_undegraded":"2024-09-21T10:31:18.043731+0000","last_fullsized":"2024-09-21T10:31:18.043731+0000","mapping_epoch":351,"log_start":"358'6300","ondisk_log_start":"358'6300","created":348,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":9760,"log_dups_size":0,"ondisk_log_size":9760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:30.913837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16060,"num_write_kb":513920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"82'100","reported_seq":316,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.400899+0000","last_change":"2024-09-21T10:24:44.294460+0000","last_active":"2024-09-21T10:31:12.400899+0000","last_peered":"2024-09-21T10:31:12.400899+0000","last_clean":"2024-09-21T10:31:12.400899+0000","last_became_active":"2024-09-21T10:24:44.293972+0000","last_became_peered":"2024-09-21T10:24:44.293972+0000","last_unstale":"2024-09-21T10:31:12.400899+0000","last_undegraded":"2024-09-21T10:31:12.400899+0000","last_fullsized":"2024-09-21T10:31:12.400899+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:26.199522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:26.199522+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:26.135171+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":1463,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.391851+0000","last_change":"2024-09-21T10:28:03.346064+0000","last_active":"2024-09-21T10:31:12.391851+0000","last_peered":"2024-09-21T10:31:12.391851+0000","last_clean":"2024-09-21T10:31:12.391851+0000","last_became_active":"2024-09-21T10:28:03.345313+0000","last_became_peered":"2024-09-21T10:28:03.345313+0000","last_unstale":"2024-09-21T10:31:12.391851+0000","last_undegraded":"2024-09-21T10:31:12.391851+0000","last_fullsized":"2024-09-21T10:31:12.391851+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:32.097521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":525,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.393005+0000","last_change":"2024-09-21T10:08:18.470748+0000","last_active":"2024-09-21T10:31:12.393005+0000","last_peered":"2024-09-21T10:31:12.393005+0000","last_clean":"2024-09-21T10:31:12.393005+0000","last_became_active":"2024-09-21T10:08:18.470227+0000","last_became_peered":"2024-09-21T10:08:18.470227+0000","last_unstale":"2024-09-21T10:31:12.393005+0000","last_undegraded":"2024-09-21T10:31:12.393005+0000","last_fullsized":"2024-09-21T10:31:12.393005+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:16.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":647,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.400354+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:31:12.400354+0000","last_peered":"2024-09-21T10:31:12.400354+0000","last_clean":"2024-09-21T10:31:12.400354+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:31:12.400354+0000","last_undegraded":"2024-09-21T10:31:12.400354+0000","last_fullsized":"2024-09-21T10:31:12.400354+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"384'15818","reported_seq":15862,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:18.033375+0000","last_change":"2024-09-21T10:26:36.989641+0000","last_active":"2024-09-21T10:31:18.033375+0000","last_peered":"2024-09-21T10:31:18.033375+0000","last_clean":"2024-09-21T10:31:18.033375+0000","last_became_active":"2024-09-21T10:26:36.989340+0000","last_became_peered":"2024-09-21T10:26:36.989340+0000","last_unstale":"2024-09-21T10:31:18.033375+0000","last_undegraded":"2024-09-21T10:31:18.033375+0000","last_fullsized":"2024-09-21T10:31:18.033375+0000","mapping_epoch":348,"log_start":"366'8100","ondisk_log_start":"366'8100","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7718,"log_dups_size":0,"ondisk_log_size":7718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:50.213332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15818,"num_write_kb":506176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"82'100","reported_seq":309,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.413008+0000","last_change":"2024-09-21T10:23:21.277739+0000","last_active":"2024-09-21T10:31:12.413008+0000","last_peered":"2024-09-21T10:31:12.413008+0000","last_clean":"2024-09-21T10:31:12.413008+0000","last_became_active":"2024-09-21T10:23:20.689733+0000","last_became_peered":"2024-09-21T10:23:20.689733+0000","last_unstale":"2024-09-21T10:31:12.413008+0000","last_undegraded":"2024-09-21T10:31:12.413008+0000","last_fullsized":"2024-09-21T10:31:12.413008+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.277663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.277663+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:01:54.523189+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":1124,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.393278+0000","last_change":"2024-09-21T10:07:43.794767+0000","last_active":"2024-09-21T10:31:12.393278+0000","last_peered":"2024-09-21T10:31:12.393278+0000","last_clean":"2024-09-21T10:31:12.393278+0000","last_became_active":"2024-09-21T10:07:43.794062+0000","last_became_peered":"2024-09-21T10:07:43.794062+0000","last_unstale":"2024-09-21T10:31:12.393278+0000","last_undegraded":"2024-09-21T10:31:12.393278+0000","last_fullsized":"2024-09-21T10:31:12.393278+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:51.977955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":73,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347866+0000","last_change":"2024-09-21T10:24:44.299492+0000","last_active":"2024-09-21T10:30:58.347866+0000","last_peered":"2024-09-21T10:30:58.347866+0000","last_clean":"2024-09-21T10:30:58.347866+0000","last_became_active":"2024-09-21T10:24:44.299187+0000","last_became_peered":"2024-09-21T10:24:44.299187+0000","last_unstale":"2024-09-21T10:30:58.347866+0000","last_undegraded":"2024-09-21T10:30:58.347866+0000","last_fullsized":"2024-09-21T10:30:58.347866+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:58.500386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":578,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.391349+0000","last_change":"2024-09-21T10:28:00.289453+0000","last_active":"2024-09-21T10:31:12.391349+0000","last_peered":"2024-09-21T10:31:12.391349+0000","last_clean":"2024-09-21T10:31:12.391349+0000","last_became_active":"2024-09-21T10:28:00.289030+0000","last_became_peered":"2024-09-21T10:28:00.289030+0000","last_unstale":"2024-09-21T10:31:12.391349+0000","last_undegraded":"2024-09-21T10:31:12.391349+0000","last_fullsized":"2024-09-21T10:31:12.391349+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:21.959998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"384'15480","reported_seq":15524,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:18.042131+0000","last_change":"2024-09-21T10:26:36.196471+0000","last_active":"2024-09-21T10:31:18.042131+0000","last_peered":"2024-09-21T10:31:18.042131+0000","last_clean":"2024-09-21T10:31:18.042131+0000","last_became_active":"2024-09-21T10:26:36.196167+0000","last_became_peered":"2024-09-21T10:26:36.196167+0000","last_unstale":"2024-09-21T10:31:18.042131+0000","last_undegraded":"2024-09-21T10:31:18.042131+0000","last_fullsized":"2024-09-21T10:31:18.042131+0000","mapping_epoch":348,"log_start":"365'7500","ondisk_log_start":"365'7500","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7980,"log_dups_size":0,"ondisk_log_size":7980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:16.488795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15480,"num_write_kb":495360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"82'100","reported_seq":248,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.346001+0000","last_change":"2024-09-21T10:28:00.294997+0000","last_active":"2024-09-21T10:30:58.346001+0000","last_peered":"2024-09-21T10:30:58.346001+0000","last_clean":"2024-09-21T10:30:58.346001+0000","last_became_active":"2024-09-21T10:28:00.294595+0000","last_became_peered":"2024-09-21T10:28:00.294595+0000","last_unstale":"2024-09-21T10:30:58.346001+0000","last_undegraded":"2024-09-21T10:30:58.346001+0000","last_fullsized":"2024-09-21T10:30:58.346001+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:30.569982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:30.569982+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:01.422939+0000","scrub_duration":0.0051309349999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":182,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.332681+0000","last_change":"2024-09-21T10:28:00.294966+0000","last_active":"2024-09-21T10:30:58.332681+0000","last_peered":"2024-09-21T10:30:58.332681+0000","last_clean":"2024-09-21T10:30:58.332681+0000","last_became_active":"2024-09-21T10:28:00.294552+0000","last_became_peered":"2024-09-21T10:28:00.294552+0000","last_unstale":"2024-09-21T10:30:58.332681+0000","last_undegraded":"2024-09-21T10:30:58.332681+0000","last_fullsized":"2024-09-21T10:30:58.332681+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:47.292125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":1231,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.392824+0000","last_change":"2024-09-21T10:08:18.468995+0000","last_active":"2024-09-21T10:31:12.392824+0000","last_peered":"2024-09-21T10:31:12.392824+0000","last_clean":"2024-09-21T10:31:12.392824+0000","last_became_active":"2024-09-21T10:08:18.468243+0000","last_became_peered":"2024-09-21T10:08:18.468243+0000","last_unstale":"2024-09-21T10:31:12.392824+0000","last_undegraded":"2024-09-21T10:31:12.392824+0000","last_fullsized":"2024-09-21T10:31:12.392824+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:45.520968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":557,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.393533+0000","last_change":"2024-09-21T10:24:44.309614+0000","last_active":"2024-09-21T10:31:12.393533+0000","last_peered":"2024-09-21T10:31:12.393533+0000","last_clean":"2024-09-21T10:31:12.393533+0000","last_became_active":"2024-09-21T10:24:44.309187+0000","last_became_peered":"2024-09-21T10:24:44.309187+0000","last_unstale":"2024-09-21T10:31:12.393533+0000","last_undegraded":"2024-09-21T10:31:12.393533+0000","last_fullsized":"2024-09-21T10:31:12.393533+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:01.074803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"384'15670","reported_seq":15715,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:18.021088+0000","last_change":"2024-09-21T10:26:36.196357+0000","last_active":"2024-09-21T10:31:18.021088+0000","last_peered":"2024-09-21T10:31:18.021088+0000","last_clean":"2024-09-21T10:31:18.021088+0000","last_became_active":"2024-09-21T10:26:36.196060+0000","last_became_peered":"2024-09-21T10:26:36.196060+0000","last_unstale":"2024-09-21T10:31:18.021088+0000","last_undegraded":"2024-09-21T10:31:18.021088+0000","last_fullsized":"2024-09-21T10:31:18.021088+0000","mapping_epoch":348,"log_start":"366'7900","ondisk_log_start":"366'7900","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7770,"log_dups_size":0,"ondisk_log_size":7770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:46:34.643144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15670,"num_write_kb":501377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"82'100","reported_seq":258,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.345705+0000","last_change":"2024-09-21T10:28:00.295277+0000","last_active":"2024-09-21T10:30:58.345705+0000","last_peered":"2024-09-21T10:30:58.345705+0000","last_clean":"2024-09-21T10:30:58.345705+0000","last_became_active":"2024-09-21T10:28:00.294016+0000","last_became_peered":"2024-09-21T10:28:00.294016+0000","last_unstale":"2024-09-21T10:30:58.345705+0000","last_undegraded":"2024-09-21T10:30:58.345705+0000","last_fullsized":"2024-09-21T10:30:58.345705+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:24.536711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:24.536711+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:28:49.984470+0000","scrub_duration":0.0047525650000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":526,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.392726+0000","last_change":"2024-09-21T10:08:18.469907+0000","last_active":"2024-09-21T10:31:12.392726+0000","last_peered":"2024-09-21T10:31:12.392726+0000","last_clean":"2024-09-21T10:31:12.392726+0000","last_became_active":"2024-09-21T10:08:18.469443+0000","last_became_peered":"2024-09-21T10:08:18.469443+0000","last_unstale":"2024-09-21T10:31:12.392726+0000","last_undegraded":"2024-09-21T10:31:12.392726+0000","last_fullsized":"2024-09-21T10:31:12.392726+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:15.645803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":806,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.349463+0000","last_change":"2024-09-21T10:24:44.334847+0000","last_active":"2024-09-21T10:30:58.349463+0000","last_peered":"2024-09-21T10:30:58.349463+0000","last_clean":"2024-09-21T10:30:58.349463+0000","last_became_active":"2024-09-21T10:24:44.301321+0000","last_became_peered":"2024-09-21T10:24:44.301321+0000","last_unstale":"2024-09-21T10:30:58.349463+0000","last_undegraded":"2024-09-21T10:30:58.349463+0000","last_fullsized":"2024-09-21T10:30:58.349463+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:37.628940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":58,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.332727+0000","last_change":"2024-09-21T10:26:41.278937+0000","last_active":"2024-09-21T10:30:58.332727+0000","last_peered":"2024-09-21T10:30:58.332727+0000","last_clean":"2024-09-21T10:30:58.332727+0000","last_became_active":"2024-09-21T10:26:41.277726+0000","last_became_peered":"2024-09-21T10:26:41.277726+0000","last_unstale":"2024-09-21T10:30:58.332727+0000","last_undegraded":"2024-09-21T10:30:58.332727+0000","last_fullsized":"2024-09-21T10:30:58.332727+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:04.741057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"384'15758","reported_seq":15802,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:17.976836+0000","last_change":"2024-09-21T10:26:36.196641+0000","last_active":"2024-09-21T10:31:17.976836+0000","last_peered":"2024-09-21T10:31:17.976836+0000","last_clean":"2024-09-21T10:31:17.976836+0000","last_became_active":"2024-09-21T10:26:36.196351+0000","last_became_peered":"2024-09-21T10:26:36.196351+0000","last_unstale":"2024-09-21T10:31:17.976836+0000","last_undegraded":"2024-09-21T10:31:17.976836+0000","last_fullsized":"2024-09-21T10:31:17.976836+0000","mapping_epoch":348,"log_start":"369'9000","ondisk_log_start":"369'9000","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":6758,"log_dups_size":0,"ondisk_log_size":6758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:41:54.280247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15758,"num_write_kb":504256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'100","reported_seq":73,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.348183+0000","last_change":"2024-09-21T10:24:44.296758+0000","last_active":"2024-09-21T10:30:58.348183+0000","last_peered":"2024-09-21T10:30:58.348183+0000","last_clean":"2024-09-21T10:30:58.348183+0000","last_became_active":"2024-09-21T10:24:44.296403+0000","last_became_peered":"2024-09-21T10:24:44.296403+0000","last_unstale":"2024-09-21T10:30:58.348183+0000","last_undegraded":"2024-09-21T10:30:58.348183+0000","last_fullsized":"2024-09-21T10:30:58.348183+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.105746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.105746+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:20:12.502081+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":63,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.397373+0000","last_change":"2024-09-21T10:28:00.300952+0000","last_active":"2024-09-21T10:31:12.397373+0000","last_peered":"2024-09-21T10:31:12.397373+0000","last_clean":"2024-09-21T10:31:12.397373+0000","last_became_active":"2024-09-21T10:28:00.293337+0000","last_became_peered":"2024-09-21T10:28:00.293337+0000","last_unstale":"2024-09-21T10:31:12.397373+0000","last_undegraded":"2024-09-21T10:31:12.397373+0000","last_fullsized":"2024-09-21T10:31:12.397373+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:58.409761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":950,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.393686+0000","last_change":"2024-09-21T10:24:46.844278+0000","last_active":"2024-09-21T10:31:12.393686+0000","last_peered":"2024-09-21T10:31:12.393686+0000","last_clean":"2024-09-21T10:31:12.393686+0000","last_became_active":"2024-09-21T10:24:46.840961+0000","last_became_peered":"2024-09-21T10:24:46.840961+0000","last_unstale":"2024-09-21T10:31:12.393686+0000","last_undegraded":"2024-09-21T10:31:12.393686+0000","last_fullsized":"2024-09-21T10:31:12.393686+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:49:48.823524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":59,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.395698+0000","last_change":"2024-09-21T10:28:00.293864+0000","last_active":"2024-09-21T10:31:12.395698+0000","last_peered":"2024-09-21T10:31:12.395698+0000","last_clean":"2024-09-21T10:31:12.395698+0000","last_became_active":"2024-09-21T10:28:00.293494+0000","last_became_peered":"2024-09-21T10:28:00.293494+0000","last_unstale":"2024-09-21T10:31:12.395698+0000","last_undegraded":"2024-09-21T10:31:12.395698+0000","last_fullsized":"2024-09-21T10:31:12.395698+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:37.098068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"384'15458","reported_seq":15606,"reported_epoch":384,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:31:18.041439+0000","last_change":"2024-09-21T10:28:01.631803+0000","last_active":"2024-09-21T10:31:18.041439+0000","last_peered":"2024-09-21T10:31:18.041439+0000","last_clean":"2024-09-21T10:27:59.267411+0000","last_became_active":"2024-09-21T10:28:01.425385+0000","last_became_peered":"2024-09-21T10:28:01.425385+0000","last_unstale":"2024-09-21T10:31:18.041439+0000","last_undegraded":"2024-09-21T10:31:18.041439+0000","last_fullsized":"2024-09-21T10:31:18.041439+0000","mapping_epoch":361,"log_start":"365'7700","ondisk_log_start":"365'7700","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:26:35.170220+0000","last_clean_scrub_stamp":"2024-09-21T10:26:35.170220+0000","objects_scrubbed":0,"log_size":7758,"log_dups_size":0,"ondisk_log_size":7758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:51:26.681334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15458,"num_write_kb":494656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":2949120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,2,4],"avail_no_missing":["6","2","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"82'100","reported_seq":66,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.395380+0000","last_change":"2024-09-21T10:28:00.305255+0000","last_active":"2024-09-21T10:31:12.395380+0000","last_peered":"2024-09-21T10:31:12.395380+0000","last_clean":"2024-09-21T10:31:12.395380+0000","last_became_active":"2024-09-21T10:28:00.290981+0000","last_became_peered":"2024-09-21T10:28:00.290981+0000","last_unstale":"2024-09-21T10:31:12.395380+0000","last_undegraded":"2024-09-21T10:31:12.395380+0000","last_fullsized":"2024-09-21T10:31:12.395380+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.759355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.759355+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":69,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":690,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.398637+0000","last_change":"2024-09-21T10:24:44.307344+0000","last_active":"2024-09-21T10:31:12.398637+0000","last_peered":"2024-09-21T10:31:12.398637+0000","last_clean":"2024-09-21T10:31:12.398637+0000","last_became_active":"2024-09-21T10:24:44.298936+0000","last_became_peered":"2024-09-21T10:24:44.298936+0000","last_unstale":"2024-09-21T10:31:12.398637+0000","last_undegraded":"2024-09-21T10:31:12.398637+0000","last_fullsized":"2024-09-21T10:31:12.398637+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:58:26.025877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":610,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.346282+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:30:58.346282+0000","last_peered":"2024-09-21T10:30:58.346282+0000","last_clean":"2024-09-21T10:30:58.346282+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:30:58.346282+0000","last_undegraded":"2024-09-21T10:30:58.346282+0000","last_fullsized":"2024-09-21T10:30:58.346282+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":73,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347525+0000","last_change":"2024-09-21T10:24:44.303781+0000","last_active":"2024-09-21T10:30:58.347525+0000","last_peered":"2024-09-21T10:30:58.347525+0000","last_clean":"2024-09-21T10:30:58.347525+0000","last_became_active":"2024-09-21T10:24:44.302058+0000","last_became_peered":"2024-09-21T10:24:44.302058+0000","last_unstale":"2024-09-21T10:30:58.347525+0000","last_undegraded":"2024-09-21T10:30:58.347525+0000","last_fullsized":"2024-09-21T10:30:58.347525+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:30.371229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":73,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347868+0000","last_change":"2024-09-21T10:24:44.303417+0000","last_active":"2024-09-21T10:30:58.347868+0000","last_peered":"2024-09-21T10:30:58.347868+0000","last_clean":"2024-09-21T10:30:58.347868+0000","last_became_active":"2024-09-21T10:24:44.303069+0000","last_became_peered":"2024-09-21T10:24:44.303069+0000","last_unstale":"2024-09-21T10:30:58.347868+0000","last_undegraded":"2024-09-21T10:30:58.347868+0000","last_fullsized":"2024-09-21T10:30:58.347868+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:30:30.966228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":83,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347766+0000","last_change":"2024-09-21T10:24:44.321421+0000","last_active":"2024-09-21T10:30:58.347766+0000","last_peered":"2024-09-21T10:30:58.347766+0000","last_clean":"2024-09-21T10:30:58.347766+0000","last_became_active":"2024-09-21T10:24:44.297082+0000","last_became_peered":"2024-09-21T10:24:44.297082+0000","last_unstale":"2024-09-21T10:30:58.347766+0000","last_undegraded":"2024-09-21T10:30:58.347766+0000","last_fullsized":"2024-09-21T10:30:58.347766+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:35:37.704881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":524,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.393366+0000","last_change":"2024-09-21T10:08:18.470378+0000","last_active":"2024-09-21T10:31:12.393366+0000","last_peered":"2024-09-21T10:31:12.393366+0000","last_clean":"2024-09-21T10:31:12.393366+0000","last_became_active":"2024-09-21T10:08:18.469482+0000","last_became_peered":"2024-09-21T10:08:18.469482+0000","last_unstale":"2024-09-21T10:31:12.393366+0000","last_undegraded":"2024-09-21T10:31:12.393366+0000","last_fullsized":"2024-09-21T10:31:12.393366+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:35.628724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":447,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.349303+0000","last_change":"2024-09-21T10:28:00.296771+0000","last_active":"2024-09-21T10:30:58.349303+0000","last_peered":"2024-09-21T10:30:58.349303+0000","last_clean":"2024-09-21T10:30:58.349303+0000","last_became_active":"2024-09-21T10:28:00.285336+0000","last_became_peered":"2024-09-21T10:28:00.285336+0000","last_unstale":"2024-09-21T10:30:58.349303+0000","last_undegraded":"2024-09-21T10:30:58.349303+0000","last_fullsized":"2024-09-21T10:30:58.349303+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:06:50.848492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":155,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.332686+0000","last_change":"2024-09-21T10:26:41.644952+0000","last_active":"2024-09-21T10:30:58.332686+0000","last_peered":"2024-09-21T10:30:58.332686+0000","last_clean":"2024-09-21T10:30:58.332686+0000","last_became_active":"2024-09-21T10:26:41.644592+0000","last_became_peered":"2024-09-21T10:26:41.644592+0000","last_unstale":"2024-09-21T10:30:58.332686+0000","last_undegraded":"2024-09-21T10:30:58.332686+0000","last_fullsized":"2024-09-21T10:30:58.332686+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:44:44.183877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":59,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.396954+0000","last_change":"2024-09-21T10:28:00.294338+0000","last_active":"2024-09-21T10:31:12.396954+0000","last_peered":"2024-09-21T10:31:12.396954+0000","last_clean":"2024-09-21T10:31:12.396954+0000","last_became_active":"2024-09-21T10:28:00.294044+0000","last_became_peered":"2024-09-21T10:28:00.294044+0000","last_unstale":"2024-09-21T10:31:12.396954+0000","last_undegraded":"2024-09-21T10:31:12.396954+0000","last_fullsized":"2024-09-21T10:31:12.396954+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:43:30.999988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":655,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.398730+0000","last_change":"2024-09-21T10:28:00.296935+0000","last_active":"2024-09-21T10:31:12.398730+0000","last_peered":"2024-09-21T10:31:12.398730+0000","last_clean":"2024-09-21T10:31:12.398730+0000","last_became_active":"2024-09-21T10:28:00.296562+0000","last_became_peered":"2024-09-21T10:28:00.296562+0000","last_unstale":"2024-09-21T10:31:12.398730+0000","last_undegraded":"2024-09-21T10:31:12.398730+0000","last_fullsized":"2024-09-21T10:31:12.398730+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:06.863310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":59,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.395872+0000","last_change":"2024-09-21T10:28:00.294718+0000","last_active":"2024-09-21T10:31:12.395872+0000","last_peered":"2024-09-21T10:31:12.395872+0000","last_clean":"2024-09-21T10:31:12.395872+0000","last_became_active":"2024-09-21T10:28:00.294299+0000","last_became_peered":"2024-09-21T10:28:00.294299+0000","last_unstale":"2024-09-21T10:31:12.395872+0000","last_undegraded":"2024-09-21T10:31:12.395872+0000","last_fullsized":"2024-09-21T10:31:12.395872+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:41:46.881336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":347,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.346076+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:30:58.346076+0000","last_peered":"2024-09-21T10:30:58.346076+0000","last_clean":"2024-09-21T10:30:58.346076+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:30:58.346076+0000","last_undegraded":"2024-09-21T10:30:58.346076+0000","last_fullsized":"2024-09-21T10:30:58.346076+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":59,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.395781+0000","last_change":"2024-09-21T10:28:00.293872+0000","last_active":"2024-09-21T10:31:12.395781+0000","last_peered":"2024-09-21T10:31:12.395781+0000","last_clean":"2024-09-21T10:31:12.395781+0000","last_became_active":"2024-09-21T10:28:00.293320+0000","last_became_peered":"2024-09-21T10:28:00.293320+0000","last_unstale":"2024-09-21T10:31:12.395781+0000","last_undegraded":"2024-09-21T10:31:12.395781+0000","last_fullsized":"2024-09-21T10:31:12.395781+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:45:43.909307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":671,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.398821+0000","last_change":"2024-09-21T10:28:00.297304+0000","last_active":"2024-09-21T10:31:12.398821+0000","last_peered":"2024-09-21T10:31:12.398821+0000","last_clean":"2024-09-21T10:31:12.398821+0000","last_became_active":"2024-09-21T10:28:00.296631+0000","last_became_peered":"2024-09-21T10:28:00.296631+0000","last_unstale":"2024-09-21T10:31:12.398821+0000","last_undegraded":"2024-09-21T10:31:12.398821+0000","last_fullsized":"2024-09-21T10:31:12.398821+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:22:16.633581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":671,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.392651+0000","last_change":"2024-09-21T10:28:00.284924+0000","last_active":"2024-09-21T10:31:12.392651+0000","last_peered":"2024-09-21T10:31:12.392651+0000","last_clean":"2024-09-21T10:31:12.392651+0000","last_became_active":"2024-09-21T10:28:00.284633+0000","last_became_peered":"2024-09-21T10:28:00.284633+0000","last_unstale":"2024-09-21T10:31:12.392651+0000","last_undegraded":"2024-09-21T10:31:12.392651+0000","last_fullsized":"2024-09-21T10:31:12.392651+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:23:09.808151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":59,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.397128+0000","last_change":"2024-09-21T10:28:00.283403+0000","last_active":"2024-09-21T10:31:12.397128+0000","last_peered":"2024-09-21T10:31:12.397128+0000","last_clean":"2024-09-21T10:31:12.397128+0000","last_became_active":"2024-09-21T10:28:00.283089+0000","last_became_peered":"2024-09-21T10:28:00.283089+0000","last_unstale":"2024-09-21T10:31:12.397128+0000","last_undegraded":"2024-09-21T10:31:12.397128+0000","last_fullsized":"2024-09-21T10:31:12.397128+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:42.213644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":73,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347561+0000","last_change":"2024-09-21T10:24:44.305124+0000","last_active":"2024-09-21T10:30:58.347561+0000","last_peered":"2024-09-21T10:30:58.347561+0000","last_clean":"2024-09-21T10:30:58.347561+0000","last_became_active":"2024-09-21T10:24:44.304468+0000","last_became_peered":"2024-09-21T10:24:44.304468+0000","last_unstale":"2024-09-21T10:30:58.347561+0000","last_undegraded":"2024-09-21T10:30:58.347561+0000","last_fullsized":"2024-09-21T10:30:58.347561+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:37.904764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":310,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.391196+0000","last_change":"2024-09-21T10:28:00.286646+0000","last_active":"2024-09-21T10:31:12.391196+0000","last_peered":"2024-09-21T10:31:12.391196+0000","last_clean":"2024-09-21T10:31:12.391196+0000","last_became_active":"2024-09-21T10:28:00.286300+0000","last_became_peered":"2024-09-21T10:28:00.286300+0000","last_unstale":"2024-09-21T10:31:12.391196+0000","last_undegraded":"2024-09-21T10:31:12.391196+0000","last_fullsized":"2024-09-21T10:31:12.391196+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:58.215385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":661,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.398523+0000","last_change":"2024-09-21T10:24:44.299989+0000","last_active":"2024-09-21T10:31:12.398523+0000","last_peered":"2024-09-21T10:31:12.398523+0000","last_clean":"2024-09-21T10:31:12.398523+0000","last_became_active":"2024-09-21T10:24:44.299356+0000","last_became_peered":"2024-09-21T10:24:44.299356+0000","last_unstale":"2024-09-21T10:31:12.398523+0000","last_undegraded":"2024-09-21T10:31:12.398523+0000","last_fullsized":"2024-09-21T10:31:12.398523+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:58.867008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":83,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347766+0000","last_change":"2024-09-21T10:28:00.281942+0000","last_active":"2024-09-21T10:30:58.347766+0000","last_peered":"2024-09-21T10:30:58.347766+0000","last_clean":"2024-09-21T10:30:58.347766+0000","last_became_active":"2024-09-21T10:28:00.281600+0000","last_became_peered":"2024-09-21T10:28:00.281600+0000","last_unstale":"2024-09-21T10:30:58.347766+0000","last_undegraded":"2024-09-21T10:30:58.347766+0000","last_fullsized":"2024-09-21T10:30:58.347766+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:21.144244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":77,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347992+0000","last_change":"2024-09-21T10:24:44.329150+0000","last_active":"2024-09-21T10:30:58.347992+0000","last_peered":"2024-09-21T10:30:58.347992+0000","last_clean":"2024-09-21T10:30:58.347992+0000","last_became_active":"2024-09-21T10:24:44.306752+0000","last_became_peered":"2024-09-21T10:24:44.306752+0000","last_unstale":"2024-09-21T10:30:58.347992+0000","last_undegraded":"2024-09-21T10:30:58.347992+0000","last_fullsized":"2024-09-21T10:30:58.347992+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:16.592859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":83,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347556+0000","last_change":"2024-09-21T10:28:00.286009+0000","last_active":"2024-09-21T10:30:58.347556+0000","last_peered":"2024-09-21T10:30:58.347556+0000","last_clean":"2024-09-21T10:30:58.347556+0000","last_became_active":"2024-09-21T10:28:00.285327+0000","last_became_peered":"2024-09-21T10:28:00.285327+0000","last_unstale":"2024-09-21T10:30:58.347556+0000","last_undegraded":"2024-09-21T10:30:58.347556+0000","last_fullsized":"2024-09-21T10:30:58.347556+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:34.985354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":77,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.348105+0000","last_change":"2024-09-21T10:24:44.324087+0000","last_active":"2024-09-21T10:30:58.348105+0000","last_peered":"2024-09-21T10:30:58.348105+0000","last_clean":"2024-09-21T10:30:58.348105+0000","last_became_active":"2024-09-21T10:24:44.298665+0000","last_became_peered":"2024-09-21T10:24:44.298665+0000","last_unstale":"2024-09-21T10:30:58.348105+0000","last_undegraded":"2024-09-21T10:30:58.348105+0000","last_fullsized":"2024-09-21T10:30:58.348105+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:59.819340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":59,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.396319+0000","last_change":"2024-09-21T10:28:00.293054+0000","last_active":"2024-09-21T10:31:12.396319+0000","last_peered":"2024-09-21T10:31:12.396319+0000","last_clean":"2024-09-21T10:31:12.396319+0000","last_became_active":"2024-09-21T10:28:00.292723+0000","last_became_peered":"2024-09-21T10:28:00.292723+0000","last_unstale":"2024-09-21T10:31:12.396319+0000","last_undegraded":"2024-09-21T10:31:12.396319+0000","last_fullsized":"2024-09-21T10:31:12.396319+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:01:45.933570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":59,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.396773+0000","last_change":"2024-09-21T10:28:00.285515+0000","last_active":"2024-09-21T10:31:12.396773+0000","last_peered":"2024-09-21T10:31:12.396773+0000","last_clean":"2024-09-21T10:31:12.396773+0000","last_became_active":"2024-09-21T10:28:00.285203+0000","last_became_peered":"2024-09-21T10:28:00.285203+0000","last_unstale":"2024-09-21T10:31:12.396773+0000","last_undegraded":"2024-09-21T10:31:12.396773+0000","last_fullsized":"2024-09-21T10:31:12.396773+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:41.982228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":73,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T10:30:58.347349+0000","last_change":"2024-09-21T10:24:44.298820+0000","last_active":"2024-09-21T10:30:58.347349+0000","last_peered":"2024-09-21T10:30:58.347349+0000","last_clean":"2024-09-21T10:30:58.347349+0000","last_became_active":"2024-09-21T10:24:44.298478+0000","last_became_peered":"2024-09-21T10:24:44.298478+0000","last_unstale":"2024-09-21T10:30:58.347349+0000","last_undegraded":"2024-09-21T10:30:58.347349+0000","last_fullsized":"2024-09-21T10:30:58.347349+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:59.567078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":647,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-21T10:31:12.398531+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:31:12.398531+0000","last_peered":"2024-09-21T10:31:12.398531+0000","last_clean":"2024-09-21T10:31:12.398531+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:31:12.398531+0000","last_undegraded":"2024-09-21T10:31:12.398531+0000","last_fullsized":"2024-09-21T10:31:12.398531+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":250046,"num_write_kb":8001409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5198,"num_bytes_recovered":200081408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1388773376,"data_stored":1388773376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127246,"ondisk_log_size":127246,"up":48,"acting":45,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":10252,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5983,"num_read_kb":13006,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":316358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":69,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":644038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1100,"ondisk_log_size":1100,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453398001,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1355616,"kb_used_data":458392,"kb_used_omap":1,"kb_used_meta":897214,"kb_avail":92389536,"statfs":{"total":95995035648,"available":94606884864,"internally_reserved":0,"allocated":469393408,"data_stored":467182227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918747480},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594229,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":902232,"kb_used_data":5072,"kb_used_omap":1,"kb_used_meta":897150,"kb_avail":92842920,"statfs":{"total":95995035648,"available":95071150080,"internally_reserved":0,"allocated":5193728,"data_stored":2990612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918681944},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790459,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":909648,"kb_used_data":9736,"kb_used_omap":1,"kb_used_meta":899902,"kb_avail":92835504,"statfs":{"total":95995035648,"available":95063556096,"internally_reserved":0,"allocated":9969664,"data_stored":7770795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":921499992},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986688,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1048348,"kb_used_data":462612,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":92696804,"statfs":{"total":95995035648,"available":94921527296,"internally_reserved":0,"allocated":473714688,"data_stored":471507525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":101,"seq":433791697249,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":703948,"kb_used_data":5636,"kb_used_omap":5,"kb_used_meta":698298,"kb_avail":93041204,"statfs":{"total":95995035648,"available":95274192896,"internally_reserved":0,"allocated":5771264,"data_stored":3576249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":715057936},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860776,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1023056,"kb_used_data":9416,"kb_used_omap":8,"kb_used_meta":1013623,"kb_avail":92722096,"statfs":{"total":95995035648,"available":94947426304,"internally_reserved":0,"allocated":9641984,"data_stored":7443395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":1037950828},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024302,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1271680,"kb_used_data":454840,"kb_used_omap":7,"kb_used_meta":816824,"kb_avail":92473472,"statfs":{"total":95995035648,"available":94692835328,"internally_reserved":0,"allocated":465756160,"data_stored":463572196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":836428338},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187837,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1053260,"kb_used_data":5060,"kb_used_omap":8,"kb_used_meta":1048183,"kb_avail":92691892,"statfs":{"total":95995035648,"available":94916497408,"internally_reserved":0,"allocated":5181440,"data_stored":2986391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":1073340401},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":460587008,"data_stored":460587008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":468516864,"data_stored":468516864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":459669504,"data_stored":459669504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:31:35.519 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T10:31:35.533 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:35.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:35.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:35.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:35.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:35.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:35.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:35.784 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:35.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:35.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:35.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:35.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:35.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:36.012 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:36.038 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:36.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:36.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:36.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:35 smithi098 ceph-mon[97580]: from='client.46538 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:31:36.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:35 smithi098 ceph-mon[97580]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:31:36.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:35 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:31:36.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:35 smithi098 ceph-mon[97580]: osdmap e385: 8 total, 8 up, 8 in 2024-09-21T10:31:36.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:35 smithi098 ceph-mon[100914]: from='client.46538 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:31:36.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:35 smithi098 ceph-mon[100914]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:31:36.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:35 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:31:36.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:35 smithi098 ceph-mon[100914]: osdmap e385: 8 total, 8 up, 8 in 2024-09-21T10:31:36.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:35 smithi196 ceph-mon[84784]: from='client.46538 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:31:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:35 smithi196 ceph-mon[84784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:31:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:35 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:31:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:35 smithi196 ceph-mon[84784]: osdmap e385: 8 total, 8 up, 8 in 2024-09-21T10:31:36.140 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:36.165 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:36.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:36.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:36.267 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:36.292 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:36.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:36.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:36.393 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:36.416 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:36.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:36.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:36.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:36.541 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:36.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:36.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:36.643 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:36.668 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:36.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:36.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:36.770 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:36.794 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:36.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:36.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:36.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:36.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:36.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:36.985 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:36 smithi098 ceph-mon[97580]: pgmap v1355: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 14874/81 objects degraded (18362.963%); 0 B/s, 202 objects/s recovering 2024-09-21T10:31:36.985 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:36 smithi098 ceph-mon[100914]: pgmap v1355: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 14874/81 objects degraded (18362.963%); 0 B/s, 202 objects/s recovering 2024-09-21T10:31:37.019 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:37.041 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:37.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:37.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:37.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:36 smithi196 ceph-mon[84784]: pgmap v1355: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 14874/81 objects degraded (18362.963%); 0 B/s, 202 objects/s recovering 2024-09-21T10:31:37.143 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:37.168 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:37.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:37.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:37.270 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:37.291 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:37.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:37.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:37.369 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:37.394 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:37.419 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:37.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:37.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:37.521 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:37.543 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:37.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:37.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:37.645 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:37.667 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:37.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:37.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:37.768 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:37.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:37.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:37.895 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:37.918 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:37.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:37.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:37.983 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:37 smithi098 ceph-mon[97580]: osdmap e386: 8 total, 8 up, 8 in 2024-09-21T10:31:37.983 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:37.984 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:37 smithi098 ceph-mon[100914]: osdmap e386: 8 total, 8 up, 8 in 2024-09-21T10:31:37.984 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:38.020 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:38.046 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:38.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:38.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:38.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:37 smithi196 ceph-mon[84784]: osdmap e386: 8 total, 8 up, 8 in 2024-09-21T10:31:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:38.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:38.170 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:38.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:38.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:38.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:38.295 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:38.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:38.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:38.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:31:38.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:38.451 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:38.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:38.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:38.552 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:38.574 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:38.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:38.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:38.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:38.698 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:38.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:38.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:38.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:38.825 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:38.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:38.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:38.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:38 smithi098 ceph-mon[97580]: pgmap v1357: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 14874/81 objects degraded (18362.963%); 0 B/s, 252 objects/s recovering 2024-09-21T10:31:38.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:38 smithi098 ceph-mon[97580]: osdmap e387: 8 total, 8 up, 8 in 2024-09-21T10:31:38.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:38 smithi098 ceph-mon[100914]: pgmap v1357: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 14874/81 objects degraded (18362.963%); 0 B/s, 252 objects/s recovering 2024-09-21T10:31:38.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:38 smithi098 ceph-mon[100914]: osdmap e387: 8 total, 8 up, 8 in 2024-09-21T10:31:38.927 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:38.948 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:38.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:38.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:39.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:38 smithi196 ceph-mon[84784]: pgmap v1357: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 14874/81 objects degraded (18362.963%); 0 B/s, 252 objects/s recovering 2024-09-21T10:31:39.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:38 smithi196 ceph-mon[84784]: osdmap e387: 8 total, 8 up, 8 in 2024-09-21T10:31:39.050 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:39.075 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:39.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:39.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:39.177 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:39.199 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:39.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:39.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:39.301 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:39.322 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:39.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:39.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:39.424 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:39.445 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:39.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:39.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:39.547 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:39.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:39.569 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:39.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:39.670 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:39.693 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:39.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:39.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:39.794 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:39.815 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:39.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:39.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:39.917 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:39.939 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:39.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:39.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:40.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:40.063 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:40.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:40.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:40.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:39 smithi098 ceph-mon[97580]: Health check update: Degraded data redundancy: 14646/81 objects degraded (18081.481%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:40.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:39 smithi098 ceph-mon[100914]: Health check update: Degraded data redundancy: 14646/81 objects degraded (18081.481%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:40.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:39 smithi196 ceph-mon[84784]: Health check update: Degraded data redundancy: 14646/81 objects degraded (18081.481%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:40.165 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:40.186 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:40.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:40.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:40.287 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:40.308 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:40.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:40.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:40.410 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:40.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:40.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:40.534 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:40.555 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:40.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:40.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:40.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:31:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:31:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:31:40.657 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:40.678 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:40.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:40.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:40.779 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:40.801 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:40.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:40.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:40.903 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:40.924 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:40.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:40.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:41.025 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:41.047 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:41.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:41.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:41.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:40 smithi098 ceph-mon[97580]: pgmap v1359: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 14646/81 objects degraded (18081.481%); 0 B/s, 231 objects/s recovering 2024-09-21T10:31:41.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:31:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:31:40] "GET /metrics HTTP/1.1" 200 36592 "" "Prometheus/2.51.0" 2024-09-21T10:31:41.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:40 smithi098 ceph-mon[100914]: pgmap v1359: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 14646/81 objects degraded (18081.481%); 0 B/s, 231 objects/s recovering 2024-09-21T10:31:41.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:40 smithi196 ceph-mon[84784]: pgmap v1359: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 94 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 14646/81 objects degraded (18081.481%); 0 B/s, 231 objects/s recovering 2024-09-21T10:31:41.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:41.171 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:41.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:41.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:41.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:41.295 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:41.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:41.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:41.396 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:41.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:41.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:41.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:41.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:41.541 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:41.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:41.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:41.643 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:41.664 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:41.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:41.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:41.766 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:41.788 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:41.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:41.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:41.890 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:41.911 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:41.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:41.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:42.013 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:42.034 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:42.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:42.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:42.136 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:42.157 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:42.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:42.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:42.259 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:42.280 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:42.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:42.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:42.381 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:42.403 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:42.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:42.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:42.504 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:42.526 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:42.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:42.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:42.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:42.651 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:42.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:42.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:42.753 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:42.775 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:42.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:42.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:42.876 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:42.898 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:42.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:42.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:43.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:43.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:43.026 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:43.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:43.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:42 smithi098 ceph-mon[97580]: pgmap v1360: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 95 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 13935/81 objects degraded (17203.704%); 0 B/s, 305 objects/s recovering 2024-09-21T10:31:43.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:42 smithi098 ceph-mon[100914]: pgmap v1360: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 95 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 13935/81 objects degraded (17203.704%); 0 B/s, 305 objects/s recovering 2024-09-21T10:31:43.128 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:42 smithi196 ceph-mon[84784]: pgmap v1360: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 95 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 13935/81 objects degraded (17203.704%); 0 B/s, 305 objects/s recovering 2024-09-21T10:31:43.150 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:43.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:43.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:43.252 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:43.275 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:43.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:43.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:43.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:43.398 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:43.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:43.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:43.500 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:43.526 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:43.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:43.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:43.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:43.649 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:43.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:43.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:43.725 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:43.751 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:43.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:43.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:43.861 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'unique_pool_5' removed 2024-09-21T10:31:43.874 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:43.908 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:43.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:43.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:44.010 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:44.037 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:44.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:44.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:44.082 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:43 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2071405752' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:31:44.082 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:43 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:31:44.082 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:43 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2071405752' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:31:44.082 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:43 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:31:44.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:43 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2071405752' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:31:44.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:43 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:31:44.139 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:44.160 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:44.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:44.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:44.262 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:44.284 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:44.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:44.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:44.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:44.407 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:44.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:44.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:44.509 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:44.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:44.531 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:44.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:44.632 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:44.653 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:44.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:44.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:44.754 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:44.775 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:44.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:44.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:44.877 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:44.898 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:44.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:44.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:44.989 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:44 smithi098 ceph-mon[100914]: pgmap v1361: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 95 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 13935/81 objects degraded (17203.704%); 0 B/s, 117 objects/s recovering 2024-09-21T10:31:44.989 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:44 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:31:44.989 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:44 smithi098 ceph-mon[100914]: osdmap e388: 8 total, 8 up, 8 in 2024-09-21T10:31:44.989 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:44 smithi098 ceph-mon[97580]: pgmap v1361: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 95 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 13935/81 objects degraded (17203.704%); 0 B/s, 117 objects/s recovering 2024-09-21T10:31:44.989 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:44 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:31:44.990 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:44 smithi098 ceph-mon[97580]: osdmap e388: 8 total, 8 up, 8 in 2024-09-21T10:31:45.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:45.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:45.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:45.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:45.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:44 smithi196 ceph-mon[84784]: pgmap v1361: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 95 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 13935/81 objects degraded (17203.704%); 0 B/s, 117 objects/s recovering 2024-09-21T10:31:45.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:44 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:31:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:44 smithi196 ceph-mon[84784]: osdmap e388: 8 total, 8 up, 8 in 2024-09-21T10:31:45.146 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:45.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:45.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:45.247 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:45.269 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:45.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:45.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:45.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:45.394 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:45.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:45.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:45.495 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:45.518 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:45.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:45.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:45.619 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:45.645 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:45.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:45.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:45.747 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:45.772 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:45.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:45.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:45.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:45.898 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:45.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:45.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:45.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:46.026 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:46.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:46.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:46.127 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:46.150 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:46.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:46.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:46.252 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:46.277 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:46.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:46.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:46.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:45 smithi098 ceph-mon[97580]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13935/81 objects degraded (17203.704%), 2 pgs degraded, 2 pgs undersized) 2024-09-21T10:31:46.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:45 smithi098 ceph-mon[100914]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13935/81 objects degraded (17203.704%), 2 pgs degraded, 2 pgs undersized) 2024-09-21T10:31:46.380 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:46.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:45 smithi196 ceph-mon[84784]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13935/81 objects degraded (17203.704%), 2 pgs degraded, 2 pgs undersized) 2024-09-21T10:31:46.405 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:46.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:46.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:46.506 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:46.531 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:46.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:46.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:46.633 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:46.658 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:46.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:46.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:46.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:46.785 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:46.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:46.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:46.887 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:46.913 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:46.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:46.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:47.015 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:47.040 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:47.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:47.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:47.141 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:47.166 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:47.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:47.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:47.267 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:47.293 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:47.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:47.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:47.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:46 smithi098 ceph-mon[97580]: pgmap v1363: 83 pgs: 83 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:31:47.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:46 smithi098 ceph-mon[100914]: pgmap v1363: 83 pgs: 83 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:31:47.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:46 smithi196 ceph-mon[84784]: pgmap v1363: 83 pgs: 83 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:31:47.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:47.421 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:47.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:47.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:47.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:47.548 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:47.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:47.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:47.650 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:47.676 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:47.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:47.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:47.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:47.803 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:47.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:47.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:47.905 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:47.931 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:47.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:47.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:48.033 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:48.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:48.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:48.161 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:48.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:48.188 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:48.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:48.289 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:48.314 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:48.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:48.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:48.416 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:48.440 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:48.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:48.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:48.541 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:48.565 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:48.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:48.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:48.667 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:48.689 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:48.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:48.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:48.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:48.817 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:48.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:48.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:48.919 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:48.944 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:48.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:48.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:49.045 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:49.070 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:49.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:49.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:49.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:49.198 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:49.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:49.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:49.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:48 smithi098 ceph-mon[97580]: pgmap v1364: 83 pgs: 83 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:31:49.252 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:48 smithi098 ceph-mon[100914]: pgmap v1364: 83 pgs: 83 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:31:49.300 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:49.323 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:49.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:49.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:49.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:48 smithi196 ceph-mon[84784]: pgmap v1364: 83 pgs: 83 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:31:49.425 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:49.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:49.449 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:49.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:49.550 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:49.573 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:49.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:49.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:49.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:49.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:49.698 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:49.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:49.799 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:49.820 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:49.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:49.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:49.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:49.944 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:49.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:49.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:50.046 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:50.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:50.070 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:50.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:49 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4104301293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:31:50.170 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:50.195 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:50.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:50.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:50.244 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:49 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4104301293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:31:50.244 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:49 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4104301293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:31:50.244 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:31:50.245 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:31:50.245 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:31:50.245 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:31:50.245 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:31:50.245 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:31:50.245 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:31:50.245 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:31:50.245 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:31:50.245 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:31:50.246 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:31:50.246 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:31:50.246 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:31:50.246 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:31:50.246 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:31:50.246 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:31:50.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:50.321 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:50.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:50.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:50.423 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:50.445 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:50.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:50.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:50.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:50.571 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:50.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:50.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:50.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:31:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:31:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:31:50.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:50.695 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:50.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:50.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:50.796 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:50.819 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:50.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:50.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:50.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:31:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:31:50] "GET /metrics HTTP/1.1" 200 36592 "" "Prometheus/2.51.0" 2024-09-21T10:31:50.920 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:50.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:50.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:51.047 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:51.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:51.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:51.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:51.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:51.198 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:51.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:51.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:51.299 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:51.309 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:50 smithi098 ceph-mon[97580]: pgmap v1365: 83 pgs: 83 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:31:51.310 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:50 smithi098 ceph-mon[100914]: pgmap v1365: 83 pgs: 83 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:31:51.323 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:51.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:51.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:51.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:50 smithi196 ceph-mon[84784]: pgmap v1365: 83 pgs: 83 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:31:51.424 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:51.448 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:51.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:51.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:51.550 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:51.574 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:51.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:51.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:51.675 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:51.679 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:31:51.680 INFO:teuthology.orchestra.run.smithi098.stdout:{"fsid":"26a15512-77fe-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":2009,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":388,"num_osds":8,"num_up_osds":8,"osd_up_since":1726912933,"num_in_osds":8,"osd_in_since":1726914479,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":83}],"num_pgs":83,"num_pools":4,"num_objects":27,"data_bytes":4516211,"bytes_used":8444088320,"bytes_avail":759516196864,"bytes_total":767960285184},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi098.stcwsu","status":"up:active","gid":14640}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.98:8443/","prometheus":"http://172.21.15.98:9283/"}},"servicemap":{"epoch":30,"modified":"2024-09-21T10:29:21.594654+0000","services":{}},"progress_events":{"6de3e7dc-7aed-4a0c-b700-51217030d18b":{"message":"Upgrade to 19.3.0-5092-g62da54b4 (4m)\n [============................] (remaining: 5m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-09-21T10:31:51.698 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:51.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:51.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:51.800 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:51.824 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:51.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:51.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:51.904 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-21T10:31:51.904 INFO:teuthology.task.print:**** done end radosbench.yaml 2024-09-21T10:31:51.904 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-09-21T10:31:51.904 INFO:tasks.workunit:Pulling workunits from ref quincy 2024-09-21T10:31:51.905 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-21T10:31:51.905 DEBUG:teuthology.orchestra.run.smithi098:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-21T10:31:51.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:51.923 INFO:teuthology.orchestra.run.smithi098.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-21T10:31:51.923 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-21T10:31:51.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:51.949 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:51.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:51.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:51.988 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-21T10:31:51.988 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-21T10:31:52.043 INFO:tasks.workunit:timeout=3h 2024-09-21T10:31:52.043 INFO:tasks.workunit:cleanup=True 2024-09-21T10:31:52.044 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch quincy https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2024-09-21T10:31:52.050 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:52.073 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:52.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:52.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:52.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:51 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3663379657' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:31:52.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:51 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3663379657' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:31:52.140 INFO:tasks.workunit.client.0.smithi098.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-21T10:31:52.175 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:52.196 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:52.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:52.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:52.297 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:52.318 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:52.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:52.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:52.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:51 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3663379657' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:31:52.420 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:52.443 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:52.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:52.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:52.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:52.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:52.568 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:52.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:52.667 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-21T10:31:52.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:31:52.669 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:52.689 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:52.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:52.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:52.791 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:52.816 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:52.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:52.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:52.917 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:52.941 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:52.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:52.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:53.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:53.064 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:53.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:53.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:53.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:53.187 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:53.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:53.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:53.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:53.310 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:53.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:53.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:53.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:53 smithi098 ceph-mon[100914]: pgmap v1366: 83 pgs: 83 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:31:53.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:31:53.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:53.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:53 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:31:53.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:53 smithi098 ceph-mon[97580]: pgmap v1366: 83 pgs: 83 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:31:53.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:31:53.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:53.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:53 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:31:53.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:53.438 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:53.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:53.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:53.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:31:53.540 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:53.599 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:53.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:53.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:53.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:53 smithi196 ceph-mon[84784]: pgmap v1366: 83 pgs: 83 active+clean; 4.3 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:31:53.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:31:53.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:53 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:31:53.701 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:53.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:53.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:53.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:53.752 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:53.768 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:31:53.768 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-21T10:31:53.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-21T10:31:53.768 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:31:53.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.768 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46025.0:181819 9.4 9:2b711ebb:::benchmark_data_smithi098_254407_object49088:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:31:53.768 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:24:43.276727+0000", 2024-09-21T10:31:53.768 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 430.48717862299998, 2024-09-21T10:31:53.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.159685471, 2024-09-21T10:31:53.768 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:31:53.768 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:31:53.769 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:31:53.769 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:31:53.769 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46025", 2024-09-21T10:31:53.769 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:31:53.769 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 181819 2024-09-21T10:31:53.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.769 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:31:53.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:31:53.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.276727+0000", 2024-09-21T10:31:53.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:31:53.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.276727+0000", 2024-09-21T10:31:53.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:31:53.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.276729+0000", 2024-09-21T10:31:53.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.872e-06 2024-09-21T10:31:53.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:31:53.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.276734+0000", 2024-09-21T10:31:53.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1390000000000003e-06 2024-09-21T10:31:53.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.771 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:31:53.771 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.276747+0000", 2024-09-21T10:31:53.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2677000000000001e-05 2024-09-21T10:31:53.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.771 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:31:53.771 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.276756+0000", 2024-09-21T10:31:53.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3440000000000007e-06 2024-09-21T10:31:53.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.771 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.771 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.277997+0000", 2024-09-21T10:31:53.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0012409680000000001 2024-09-21T10:31:53.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.772 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:31:53.772 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.278009+0000", 2024-09-21T10:31:53.772 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2337000000000001e-05 2024-09-21T10:31:53.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.772 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.772 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:44.292360+0000", 2024-09-21T10:31:53.772 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.014350326 2024-09-21T10:31:53.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:31:53.773 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:44.292372+0000", 2024-09-21T10:31:53.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2415999999999999e-05 2024-09-21T10:31:53.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.773 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.417715+0000", 2024-09-21T10:31:53.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1253431780000001 2024-09-21T10:31:53.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:31:53.773 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.417859+0000", 2024-09-21T10:31:53.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014354400000000001 2024-09-21T10:31:53.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4,6", 2024-09-21T10:31:53.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.417946+0000", 2024-09-21T10:31:53.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6582999999999994e-05 2024-09-21T10:31:53.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:31:53.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.425356+0000", 2024-09-21T10:31:53.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0074104160000000004 2024-09-21T10:31:53.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.426024+0000", 2024-09-21T10:31:53.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00066778200000000001 2024-09-21T10:31:53.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.426930+0000", 2024-09-21T10:31:53.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00090659699999999998 2024-09-21T10:31:53.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.436223+0000", 2024-09-21T10:31:53.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.009292926 2024-09-21T10:31:53.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:31:53.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.436298+0000", 2024-09-21T10:31:53.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5230999999999999e-05 2024-09-21T10:31:53.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:31:53.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.436413+0000", 2024-09-21T10:31:53.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114135 2024-09-21T10:31:53.776 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.776 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:31:53.776 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.776 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46025.0:181864 9.4 9:2897bfcd:::benchmark_data_smithi098_254407_object49133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:31:53.777 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:24:43.337373+0000", 2024-09-21T10:31:53.777 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 430.42653233499999, 2024-09-21T10:31:53.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0993005579999999, 2024-09-21T10:31:53.777 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:31:53.777 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:31:53.777 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:31:53.777 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:31:53.777 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46025", 2024-09-21T10:31:53.777 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:31:53.777 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 181864 2024-09-21T10:31:53.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.777 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:31:53.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:31:53.778 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.337373+0000", 2024-09-21T10:31:53.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.778 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:31:53.778 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.337373+0000", 2024-09-21T10:31:53.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.778 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:31:53.778 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.337375+0000", 2024-09-21T10:31:53.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.203e-06 2024-09-21T10:31:53.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:31:53.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.337380+0000", 2024-09-21T10:31:53.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0170000000000004e-06 2024-09-21T10:31:53.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:31:53.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.337389+0000", 2024-09-21T10:31:53.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6260000000000007e-06 2024-09-21T10:31:53.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:31:53.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.337396+0000", 2024-09-21T10:31:53.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9460000000000001e-06 2024-09-21T10:31:53.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.337450+0000", 2024-09-21T10:31:53.780 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4228999999999999e-05 2024-09-21T10:31:53.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:31:53.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.337460+0000", 2024-09-21T10:31:53.780 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3190000000000001e-06 2024-09-21T10:31:53.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:44.293375+0000", 2024-09-21T10:31:53.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.95591494499999996 2024-09-21T10:31:53.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:31:53.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:44.293384+0000", 2024-09-21T10:31:53.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4350000000000003e-06 2024-09-21T10:31:53.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.420182+0000", 2024-09-21T10:31:53.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.126798322 2024-09-21T10:31:53.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:31:53.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.420285+0000", 2024-09-21T10:31:53.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000102815 2024-09-21T10:31:53.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4,6", 2024-09-21T10:31:53.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.420362+0000", 2024-09-21T10:31:53.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6526999999999994e-05 2024-09-21T10:31:53.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:31:53.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.425365+0000", 2024-09-21T10:31:53.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050036380000000004 2024-09-21T10:31:53.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.426081+0000", 2024-09-21T10:31:53.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00071544499999999999 2024-09-21T10:31:53.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.426962+0000", 2024-09-21T10:31:53.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088092499999999996 2024-09-21T10:31:53.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.436497+0000", 2024-09-21T10:31:53.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0095347499999999998 2024-09-21T10:31:53.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:31:53.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.436558+0000", 2024-09-21T10:31:53.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1910999999999997e-05 2024-09-21T10:31:53.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:31:53.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.436674+0000", 2024-09-21T10:31:53.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000115505 2024-09-21T10:31:53.785 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.785 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:31:53.785 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.785 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46025.0:181867 9.4 9:2a10222f:::benchmark_data_smithi098_254407_object49136:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:31:53.785 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:24:43.338016+0000", 2024-09-21T10:31:53.785 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 430.42588968400003, 2024-09-21T10:31:53.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0988506149999999, 2024-09-21T10:31:53.786 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:31:53.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:31:53.786 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:31:53.786 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:31:53.786 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46025", 2024-09-21T10:31:53.786 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:31:53.786 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 181867 2024-09-21T10:31:53.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.786 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:31:53.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:31:53.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.338016+0000", 2024-09-21T10:31:53.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:31:53.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.338016+0000", 2024-09-21T10:31:53.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:31:53.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.338017+0000", 2024-09-21T10:31:53.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9299999999999998e-07 2024-09-21T10:31:53.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:31:53.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.338020+0000", 2024-09-21T10:31:53.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4750000000000002e-06 2024-09-21T10:31:53.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:31:53.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.338027+0000", 2024-09-21T10:31:53.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2099999999999996e-06 2024-09-21T10:31:53.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:31:53.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.338032+0000", 2024-09-21T10:31:53.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4390000000000002e-06 2024-09-21T10:31:53.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.338083+0000", 2024-09-21T10:31:53.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1712000000000002e-05 2024-09-21T10:31:53.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:31:53.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.338094+0000", 2024-09-21T10:31:53.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0597e-05 2024-09-21T10:31:53.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:44.293416+0000", 2024-09-21T10:31:53.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.95532220800000001 2024-09-21T10:31:53.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:31:53.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:44.293424+0000", 2024-09-21T10:31:53.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4039999999999998e-06 2024-09-21T10:31:53.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.422249+0000", 2024-09-21T10:31:53.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.128825116 2024-09-21T10:31:53.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:31:53.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.422368+0000", 2024-09-21T10:31:53.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000119345 2024-09-21T10:31:53.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4,6", 2024-09-21T10:31:53.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.422439+0000", 2024-09-21T10:31:53.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0875000000000002e-05 2024-09-21T10:31:53.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:31:53.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.425371+0000", 2024-09-21T10:31:53.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002931964 2024-09-21T10:31:53.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.426991+0000", 2024-09-21T10:31:53.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001619611 2024-09-21T10:31:53.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.436165+0000", 2024-09-21T10:31:53.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0091741280000000001 2024-09-21T10:31:53.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.436716+0000", 2024-09-21T10:31:53.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00055105300000000002 2024-09-21T10:31:53.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:31:53.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.436766+0000", 2024-09-21T10:31:53.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0241999999999998e-05 2024-09-21T10:31:53.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:31:53.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.436867+0000", 2024-09-21T10:31:53.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010064300000000001 2024-09-21T10:31:53.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.793 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:31:53.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.793 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46025.0:181870 9.4 9:2afb0b70:::benchmark_data_smithi098_254407_object49139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:31:53.793 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:24:43.342681+0000", 2024-09-21T10:31:53.793 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 430.42122443300002, 2024-09-21T10:31:53.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0943812749999999, 2024-09-21T10:31:53.793 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:31:53.794 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:31:53.794 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:31:53.794 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:31:53.794 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46025", 2024-09-21T10:31:53.794 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:31:53.794 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 181870 2024-09-21T10:31:53.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.794 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:31:53.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:31:53.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.342681+0000", 2024-09-21T10:31:53.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:31:53.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.342681+0000", 2024-09-21T10:31:53.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:31:53.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.342682+0000", 2024-09-21T10:31:53.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4300000000000003e-07 2024-09-21T10:31:53.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:31:53.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.342685+0000", 2024-09-21T10:31:53.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.331e-06 2024-09-21T10:31:53.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:31:53.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.342693+0000", 2024-09-21T10:31:53.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8839999999999994e-06 2024-09-21T10:31:53.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:31:53.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.342698+0000", 2024-09-21T10:31:53.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3610000000000003e-06 2024-09-21T10:31:53.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.342748+0000", 2024-09-21T10:31:53.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0404000000000001e-05 2024-09-21T10:31:53.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:31:53.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:43.342758+0000", 2024-09-21T10:31:53.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0382e-05 2024-09-21T10:31:53.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:44.293449+0000", 2024-09-21T10:31:53.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.95069105799999998 2024-09-21T10:31:53.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:31:53.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:44.293457+0000", 2024-09-21T10:31:53.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6599999999999995e-06 2024-09-21T10:31:53.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.423848+0000", 2024-09-21T10:31:53.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1303906690000001 2024-09-21T10:31:53.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:31:53.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.423954+0000", 2024-09-21T10:31:53.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010610199999999999 2024-09-21T10:31:53.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4,6", 2024-09-21T10:31:53.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.424025+0000", 2024-09-21T10:31:53.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1006000000000004e-05 2024-09-21T10:31:53.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:31:53.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.425386+0000", 2024-09-21T10:31:53.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00136112 2024-09-21T10:31:53.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.436112+0000", 2024-09-21T10:31:53.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010726375 2024-09-21T10:31:53.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.436460+0000", 2024-09-21T10:31:53.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000347425 2024-09-21T10:31:53.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.436909+0000", 2024-09-21T10:31:53.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000449164 2024-09-21T10:31:53.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:31:53.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.436958+0000", 2024-09-21T10:31:53.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8653999999999999e-05 2024-09-21T10:31:53.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:31:53.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:24:45.437063+0000", 2024-09-21T10:31:53.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105037 2024-09-21T10:31:53.801 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.801 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:31:53.801 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.801 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46025.0:244312 9.4 9:2584593b:::benchmark_data_smithi098_254407_object111581:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-09-21T10:31:53.801 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:25:55.007891+0000", 2024-09-21T10:31:53.801 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 358.75601480400002, 2024-09-21T10:31:53.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0455959020000001, 2024-09-21T10:31:53.801 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:31:53.801 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:31:53.802 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:31:53.802 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:31:53.802 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46025", 2024-09-21T10:31:53.802 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:31:53.802 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 244312 2024-09-21T10:31:53.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.802 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:31:53.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:31:53.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.007891+0000", 2024-09-21T10:31:53.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:31:53.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.007891+0000", 2024-09-21T10:31:53.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:31:53.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.007893+0000", 2024-09-21T10:31:53.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.68e-06 2024-09-21T10:31:53.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:31:53.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.007899+0000", 2024-09-21T10:31:53.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3069999999999999e-06 2024-09-21T10:31:53.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:31:53.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.007918+0000", 2024-09-21T10:31:53.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9485999999999999e-05 2024-09-21T10:31:53.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:31:53.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.007925+0000", 2024-09-21T10:31:53.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.046e-06 2024-09-21T10:31:53.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.008815+0000", 2024-09-21T10:31:53.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088950200000000002 2024-09-21T10:31:53.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:31:53.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.008821+0000", 2024-09-21T10:31:53.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2230000000000004e-06 2024-09-21T10:31:53.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.046963+0000", 2024-09-21T10:31:53.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.038141413 2024-09-21T10:31:53.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:31:53.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.047117+0000", 2024-09-21T10:31:53.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015440899999999999 2024-09-21T10:31:53.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:31:53.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.047258+0000", 2024-09-21T10:31:53.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000141268 2024-09-21T10:31:53.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.050613+0000", 2024-09-21T10:31:53.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033546729999999999 2024-09-21T10:31:53.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.050655+0000", 2024-09-21T10:31:53.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2379000000000002e-05 2024-09-21T10:31:53.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:31:53.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.053402+0000", 2024-09-21T10:31:53.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027461690000000001 2024-09-21T10:31:53.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:31:53.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.053442+0000", 2024-09-21T10:31:53.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0191999999999997e-05 2024-09-21T10:31:53.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:31:53.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.053487+0000", 2024-09-21T10:31:53.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5154999999999997e-05 2024-09-21T10:31:53.808 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.808 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:31:53.808 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.808 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46025.0:244319 9.4 9:203c25b2:::benchmark_data_smithi098_254407_object111588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-21T10:31:53.808 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:25:55.013626+0000", 2024-09-21T10:31:53.808 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 358.750279374, 2024-09-21T10:31:53.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0433725680000001, 2024-09-21T10:31:53.808 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:31:53.808 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:31:53.808 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:31:53.808 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:31:53.808 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46025", 2024-09-21T10:31:53.809 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:31:53.809 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 244319 2024-09-21T10:31:53.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.809 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:31:53.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:31:53.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.013626+0000", 2024-09-21T10:31:53.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:31:53.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.013626+0000", 2024-09-21T10:31:53.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:31:53.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.013627+0000", 2024-09-21T10:31:53.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1900000000000001e-07 2024-09-21T10:31:53.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:31:53.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.013631+0000", 2024-09-21T10:31:53.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.568e-06 2024-09-21T10:31:53.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:31:53.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.013640+0000", 2024-09-21T10:31:53.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5249999999999994e-06 2024-09-21T10:31:53.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:31:53.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.013645+0000", 2024-09-21T10:31:53.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6650000000000002e-06 2024-09-21T10:31:53.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.013694+0000", 2024-09-21T10:31:53.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8733000000000002e-05 2024-09-21T10:31:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:31:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.013702+0000", 2024-09-21T10:31:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.422e-06 2024-09-21T10:31:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.050685+0000", 2024-09-21T10:31:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0369826900000001 2024-09-21T10:31:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:31:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.050781+0000", 2024-09-21T10:31:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5721000000000001e-05 2024-09-21T10:31:53.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:31:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.050861+0000", 2024-09-21T10:31:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0552999999999999e-05 2024-09-21T10:31:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.056661+0000", 2024-09-21T10:31:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057997860000000004 2024-09-21T10:31:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.056817+0000", 2024-09-21T10:31:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015557099999999999 2024-09-21T10:31:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:31:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.056897+0000", 2024-09-21T10:31:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0323999999999999e-05 2024-09-21T10:31:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:31:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.056955+0000", 2024-09-21T10:31:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8548000000000003e-05 2024-09-21T10:31:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:31:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.056999+0000", 2024-09-21T10:31:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3543e-05 2024-09-21T10:31:53.814 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:31:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46025.0:244321 9.4 9:2c449319:::benchmark_data_smithi098_254407_object111590:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-21T10:31:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:25:55.014088+0000", 2024-09-21T10:31:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 358.749817441, 2024-09-21T10:31:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.049725244, 2024-09-21T10:31:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:31:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:31:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:31:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:31:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46025", 2024-09-21T10:31:53.815 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:31:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 244321 2024-09-21T10:31:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:31:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:31:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.014088+0000", 2024-09-21T10:31:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:31:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.014088+0000", 2024-09-21T10:31:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:31:53.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.014089+0000", 2024-09-21T10:31:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3200000000000004e-07 2024-09-21T10:31:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:31:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.014093+0000", 2024-09-21T10:31:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6689999999999999e-06 2024-09-21T10:31:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:31:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.014101+0000", 2024-09-21T10:31:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9789999999999993e-06 2024-09-21T10:31:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:31:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.014113+0000", 2024-09-21T10:31:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2362e-05 2024-09-21T10:31:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.014159+0000", 2024-09-21T10:31:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5881000000000003e-05 2024-09-21T10:31:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:31:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.014164+0000", 2024-09-21T10:31:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6439999999999999e-06 2024-09-21T10:31:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.053524+0000", 2024-09-21T10:31:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.039360032 2024-09-21T10:31:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:31:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.053601+0000", 2024-09-21T10:31:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7733999999999998e-05 2024-09-21T10:31:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:31:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.053674+0000", 2024-09-21T10:31:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2638999999999994e-05 2024-09-21T10:31:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.056878+0000", 2024-09-21T10:31:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032042260000000001 2024-09-21T10:31:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:31:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.060544+0000", 2024-09-21T10:31:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036657920000000002 2024-09-21T10:31:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.063720+0000", 2024-09-21T10:31:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003176075 2024-09-21T10:31:53.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:31:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.063772+0000", 2024-09-21T10:31:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2157999999999998e-05 2024-09-21T10:31:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:31:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.063814+0000", 2024-09-21T10:31:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1321e-05 2024-09-21T10:31:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:31:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46025.0:244364 9.4 9:2a54db6c:::benchmark_data_smithi098_254407_object111633:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-21T10:31:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:25:55.060954+0000", 2024-09-21T10:31:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 358.702952057, 2024-09-21T10:31:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0096707469999999, 2024-09-21T10:31:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:31:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:31:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:31:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:31:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46025", 2024-09-21T10:31:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:31:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 244364 2024-09-21T10:31:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:31:53.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:31:53.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.060954+0000", 2024-09-21T10:31:53.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:31:53.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.060954+0000", 2024-09-21T10:31:53.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:31:53.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.060955+0000", 2024-09-21T10:31:53.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2190000000000001e-06 2024-09-21T10:31:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:31:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.060960+0000", 2024-09-21T10:31:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8500000000000002e-06 2024-09-21T10:31:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:31:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.060967+0000", 2024-09-21T10:31:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5610000000000001e-06 2024-09-21T10:31:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:31:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.060972+0000", 2024-09-21T10:31:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6240000000000001e-06 2024-09-21T10:31:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.061029+0000", 2024-09-21T10:31:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6733e-05 2024-09-21T10:31:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:31:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.061037+0000", 2024-09-21T10:31:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7940000000000008e-06 2024-09-21T10:31:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.057036+0000", 2024-09-21T10:31:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.99599844299999996 2024-09-21T10:31:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:31:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.057175+0000", 2024-09-21T10:31:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000138928 2024-09-21T10:31:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:31:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.057308+0000", 2024-09-21T10:31:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013285799999999999 2024-09-21T10:31:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.063852+0000", 2024-09-21T10:31:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0065446489999999996 2024-09-21T10:31:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:31:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.070418+0000", 2024-09-21T10:31:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0065653129999999997 2024-09-21T10:31:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.070524+0000", 2024-09-21T10:31:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106807 2024-09-21T10:31:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:31:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.070578+0000", 2024-09-21T10:31:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3758000000000003e-05 2024-09-21T10:31:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:31:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.070624+0000", 2024-09-21T10:31:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.621e-05 2024-09-21T10:31:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:31:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.829 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46025.0:244365 9.4 9:2bbc545c:::benchmark_data_smithi098_254407_object111634:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-21T10:31:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:25:55.061055+0000", 2024-09-21T10:31:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 358.70285104499999, 2024-09-21T10:31:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.009761761, 2024-09-21T10:31:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:31:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:31:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:31:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:31:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46025", 2024-09-21T10:31:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:31:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 244365 2024-09-21T10:31:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:31:53.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:31:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.061055+0000", 2024-09-21T10:31:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:31:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.061055+0000", 2024-09-21T10:31:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:31:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.061056+0000", 2024-09-21T10:31:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2999999999999999e-07 2024-09-21T10:31:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:31:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.061060+0000", 2024-09-21T10:31:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1849999999999997e-06 2024-09-21T10:31:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:31:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.061069+0000", 2024-09-21T10:31:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4690000000000003e-06 2024-09-21T10:31:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:31:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.061076+0000", 2024-09-21T10:31:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3260000000000001e-06 2024-09-21T10:31:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.061127+0000", 2024-09-21T10:31:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1885999999999998e-05 2024-09-21T10:31:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:31:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.061135+0000", 2024-09-21T10:31:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9710000000000004e-06 2024-09-21T10:31:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.060590+0000", 2024-09-21T10:31:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.99945498200000005 2024-09-21T10:31:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:31:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.060714+0000", 2024-09-21T10:31:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012353900000000001 2024-09-21T10:31:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:31:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.060813+0000", 2024-09-21T10:31:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9488999999999996e-05 2024-09-21T10:31:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.070407+0000", 2024-09-21T10:31:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0095940890000000001 2024-09-21T10:31:53.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:31:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.070422+0000", 2024-09-21T10:31:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4472e-05 2024-09-21T10:31:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.070731+0000", 2024-09-21T10:31:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000308919 2024-09-21T10:31:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:31:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.070780+0000", 2024-09-21T10:31:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9438000000000003e-05 2024-09-21T10:31:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:31:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.070816+0000", 2024-09-21T10:31:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6165999999999999e-05 2024-09-21T10:31:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:31:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46025.0:244371 9.4 9:2b995aef:::benchmark_data_smithi098_254407_object111640:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-21T10:31:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:25:55.068149+0000", 2024-09-21T10:31:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 358.69575672000002, 2024-09-21T10:31:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0027908290000001, 2024-09-21T10:31:53.836 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:31:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:31:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:31:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:31:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46025", 2024-09-21T10:31:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:31:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 244371 2024-09-21T10:31:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:31:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:31:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.068149+0000", 2024-09-21T10:31:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:31:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.068149+0000", 2024-09-21T10:31:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:31:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.068150+0000", 2024-09-21T10:31:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0100000000000001e-06 2024-09-21T10:31:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:31:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.068154+0000", 2024-09-21T10:31:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0980000000000004e-06 2024-09-21T10:31:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:31:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.068166+0000", 2024-09-21T10:31:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2002e-05 2024-09-21T10:31:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:31:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.068173+0000", 2024-09-21T10:31:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8290000000000002e-06 2024-09-21T10:31:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.068236+0000", 2024-09-21T10:31:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3270999999999998e-05 2024-09-21T10:31:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:31:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.068244+0000", 2024-09-21T10:31:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.554e-06 2024-09-21T10:31:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.063879+0000", 2024-09-21T10:31:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.99563529900000003 2024-09-21T10:31:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:31:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.063973+0000", 2024-09-21T10:31:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3706e-05 2024-09-21T10:31:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:31:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.064102+0000", 2024-09-21T10:31:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012894600000000001 2024-09-21T10:31:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:31:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.070426+0000", 2024-09-21T10:31:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0063238729999999998 2024-09-21T10:31:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.070674+0000", 2024-09-21T10:31:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00024851300000000001 2024-09-21T10:31:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.070861+0000", 2024-09-21T10:31:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000187123 2024-09-21T10:31:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:31:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.070905+0000", 2024-09-21T10:31:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3269000000000002e-05 2024-09-21T10:31:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:31:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.070940+0000", 2024-09-21T10:31:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5336000000000002e-05 2024-09-21T10:31:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:31:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46025.0:244402 9.4 9:24064c04:::benchmark_data_smithi098_254407_object111671:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-21T10:31:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:25:55.096352+0000", 2024-09-21T10:31:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 358.667553613, 2024-09-21T10:31:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.99703707900000005, 2024-09-21T10:31:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:31:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:31:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:31:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:31:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46025", 2024-09-21T10:31:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:31:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 244402 2024-09-21T10:31:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:31:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:31:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.096352+0000", 2024-09-21T10:31:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:31:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.096352+0000", 2024-09-21T10:31:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:31:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.096353+0000", 2024-09-21T10:31:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3200000000000004e-07 2024-09-21T10:31:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:31:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.096357+0000", 2024-09-21T10:31:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8039999999999999e-06 2024-09-21T10:31:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:31:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.096367+0000", 2024-09-21T10:31:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0147e-05 2024-09-21T10:31:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:31:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.096374+0000", 2024-09-21T10:31:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8399999999999997e-06 2024-09-21T10:31:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.096425+0000", 2024-09-21T10:31:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0915999999999998e-05 2024-09-21T10:31:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:31:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:55.096430+0000", 2024-09-21T10:31:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8610000000000003e-06 2024-09-21T10:31:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.071132+0000", 2024-09-21T10:31:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.97470204800000004 2024-09-21T10:31:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:31:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.071225+0000", 2024-09-21T10:31:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2707000000000005e-05 2024-09-21T10:31:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:31:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.071331+0000", 2024-09-21T10:31:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010558800000000001 2024-09-21T10:31:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.081296+0000", 2024-09-21T10:31:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0099651529999999992 2024-09-21T10:31:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.081336+0000", 2024-09-21T10:31:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0101000000000001e-05 2024-09-21T10:31:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:31:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.093293+0000", 2024-09-21T10:31:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011957008999999999 2024-09-21T10:31:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:31:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.093343+0000", 2024-09-21T10:31:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0176999999999997e-05 2024-09-21T10:31:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:31:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:25:56.093389+0000", 2024-09-21T10:31:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5995999999999998e-05 2024-09-21T10:31:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:31:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46025.0:265405 9.f 9:fbed789a:::benchmark_data_smithi098_254407_object132674:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-09-21T10:31:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:26:16.862839+0000", 2024-09-21T10:31:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 336.90106626599999, 2024-09-21T10:31:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.19322964000000001, 2024-09-21T10:31:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:31:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:31:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:31:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:31:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46025", 2024-09-21T10:31:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:31:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 265405 2024-09-21T10:31:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:31:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:31:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.862839+0000", 2024-09-21T10:31:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:31:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.862839+0000", 2024-09-21T10:31:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:31:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.862841+0000", 2024-09-21T10:31:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.066e-06 2024-09-21T10:31:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:31:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.862844+0000", 2024-09-21T10:31:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8229999999999997e-06 2024-09-21T10:31:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:31:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.862864+0000", 2024-09-21T10:31:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9893000000000001e-05 2024-09-21T10:31:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:31:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.862870+0000", 2024-09-21T10:31:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.207e-06 2024-09-21T10:31:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.862935+0000", 2024-09-21T10:31:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4212000000000001e-05 2024-09-21T10:31:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:31:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.863074+0000", 2024-09-21T10:31:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000138956 2024-09-21T10:31:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:31:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.863207+0000", 2024-09-21T10:31:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000133015 2024-09-21T10:31:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:31:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.866534+0000", 2024-09-21T10:31:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033273 2024-09-21T10:31:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.868190+0000", 2024-09-21T10:31:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001656289 2024-09-21T10:31:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:17.055982+0000", 2024-09-21T10:31:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.187791765 2024-09-21T10:31:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:31:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:17.056031+0000", 2024-09-21T10:31:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9001000000000001e-05 2024-09-21T10:31:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:31:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:17.056069+0000", 2024-09-21T10:31:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8112999999999999e-05 2024-09-21T10:31:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:31:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46025.0:265413 9.f 9:fc236b48:::benchmark_data_smithi098_254407_object132682:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-09-21T10:31:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:26:16.869922+0000", 2024-09-21T10:31:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 336.89398382299999, 2024-09-21T10:31:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.196931415, 2024-09-21T10:31:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:31:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:31:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:31:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:31:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46025", 2024-09-21T10:31:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:31:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 265413 2024-09-21T10:31:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:31:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:31:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.869922+0000", 2024-09-21T10:31:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:31:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.869922+0000", 2024-09-21T10:31:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:31:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.869923+0000", 2024-09-21T10:31:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1349999999999999e-06 2024-09-21T10:31:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:31:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.869927+0000", 2024-09-21T10:31:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4259999999999996e-06 2024-09-21T10:31:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:31:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.869938+0000", 2024-09-21T10:31:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0671e-05 2024-09-21T10:31:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:31:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.869944+0000", 2024-09-21T10:31:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3519999999999997e-06 2024-09-21T10:31:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.869997+0000", 2024-09-21T10:31:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3544000000000001e-05 2024-09-21T10:31:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:31:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.870145+0000", 2024-09-21T10:31:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014824200000000001 2024-09-21T10:31:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:31:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.870296+0000", 2024-09-21T10:31:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000150989 2024-09-21T10:31:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:31:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.873259+0000", 2024-09-21T10:31:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029623240000000001 2024-09-21T10:31:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:16.876220+0000", 2024-09-21T10:31:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029611709999999999 2024-09-21T10:31:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:17.066778+0000", 2024-09-21T10:31:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.190557852 2024-09-21T10:31:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:31:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:17.066819+0000", 2024-09-21T10:31:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1443999999999998e-05 2024-09-21T10:31:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:31:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:26:17.066853+0000", 2024-09-21T10:31:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4264999999999998e-05 2024-09-21T10:31:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:31:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:217835 10.2 10:421a0aa3:::benchmark_data_smithi098_281896_object92810:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-21T10:31:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:30:32.714866+0000", 2024-09-21T10:31:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 81.049039629000006, 2024-09-21T10:31:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.19259701100000001, 2024-09-21T10:31:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:31:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:31:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:31:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:31:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:31:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:31:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 217835 2024-09-21T10:31:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:31:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:31:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.714866+0000", 2024-09-21T10:31:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:31:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.714866+0000", 2024-09-21T10:31:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:31:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.714868+0000", 2024-09-21T10:31:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0820000000000001e-06 2024-09-21T10:31:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:31:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.714876+0000", 2024-09-21T10:31:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9699999999999999e-06 2024-09-21T10:31:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:31:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.714894+0000", 2024-09-21T10:31:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8289999999999999e-05 2024-09-21T10:31:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:31:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.714903+0000", 2024-09-21T10:31:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8149999999999995e-06 2024-09-21T10:31:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.714966+0000", 2024-09-21T10:31:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2595999999999995e-05 2024-09-21T10:31:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:31:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.715084+0000", 2024-09-21T10:31:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000118164 2024-09-21T10:31:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:31:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.715204+0000", 2024-09-21T10:31:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000120388 2024-09-21T10:31:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:31:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.720123+0000", 2024-09-21T10:31:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004918983 2024-09-21T10:31:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.721487+0000", 2024-09-21T10:31:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001363866 2024-09-21T10:31:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.907366+0000", 2024-09-21T10:31:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.18587822200000001 2024-09-21T10:31:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:31:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.907422+0000", 2024-09-21T10:31:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6728000000000003e-05 2024-09-21T10:31:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:31:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.907463+0000", 2024-09-21T10:31:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0907e-05 2024-09-21T10:31:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:31:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:217840 10.0 e381/348 10:032abaef:::benchmark_data_smithi098_281896_object92815:head v 381'13536, mlcod=381'13534)", 2024-09-21T10:31:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:30:32.721536+0000", 2024-09-21T10:31:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 81.042369977000007, 2024-09-21T10:31:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.19087325899999999, 2024-09-21T10:31:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:31:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:31:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:31:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:31:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:31:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.721536+0000", 2024-09-21T10:31:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:31:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.721536+0000", 2024-09-21T10:31:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:31:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.721536+0000", 2024-09-21T10:31:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7399999999999999e-07 2024-09-21T10:31:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:31:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.721540+0000", 2024-09-21T10:31:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8539999999999999e-06 2024-09-21T10:31:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:31:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.721547+0000", 2024-09-21T10:31:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0310000000000001e-06 2024-09-21T10:31:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:31:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.721551+0000", 2024-09-21T10:31:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2610000000000004e-06 2024-09-21T10:31:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.721620+0000", 2024-09-21T10:31:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8998999999999994e-05 2024-09-21T10:31:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:31:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.721637+0000", 2024-09-21T10:31:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6895e-05 2024-09-21T10:31:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:31:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.912256+0000", 2024-09-21T10:31:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.19061876799999999 2024-09-21T10:31:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:31:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.912409+0000", 2024-09-21T10:31:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000153077 2024-09-21T10:31:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:31:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:217842 10.0 e381/348 10:02920e10:::benchmark_data_smithi098_281896_object92817:head v 381'13537, mlcod=381'13534)", 2024-09-21T10:31:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:30:32.724497+0000", 2024-09-21T10:31:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 81.039408502000001, 2024-09-21T10:31:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.188628675, 2024-09-21T10:31:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:31:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:31:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:31:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:31:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:31:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.724497+0000", 2024-09-21T10:31:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:31:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.724497+0000", 2024-09-21T10:31:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:31:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.724498+0000", 2024-09-21T10:31:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.133e-06 2024-09-21T10:31:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:31:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.724509+0000", 2024-09-21T10:31:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0156999999999999e-05 2024-09-21T10:31:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:31:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.724523+0000", 2024-09-21T10:31:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4187e-05 2024-09-21T10:31:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:31:53.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.724530+0000", 2024-09-21T10:31:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4120000000000004e-06 2024-09-21T10:31:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.904037+0000", 2024-09-21T10:31:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.179506374 2024-09-21T10:31:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:31:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.904057+0000", 2024-09-21T10:31:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0421999999999999e-05 2024-09-21T10:31:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:31:53.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.912679+0000", 2024-09-21T10:31:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0086218639999999999 2024-09-21T10:31:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:31:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.913126+0000", 2024-09-21T10:31:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000447126 2024-09-21T10:31:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:31:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:217843 10.2 10:4ec7e3cf:::benchmark_data_smithi098_281896_object92818:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-21T10:31:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:30:32.724537+0000", 2024-09-21T10:31:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 81.039368823000004, 2024-09-21T10:31:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.183933818, 2024-09-21T10:31:53.877 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:31:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:31:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:31:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:31:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:31:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:31:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 217843 2024-09-21T10:31:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:31:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:31:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.724537+0000", 2024-09-21T10:31:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:31:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.724537+0000", 2024-09-21T10:31:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:31:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.724539+0000", 2024-09-21T10:31:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.699e-06 2024-09-21T10:31:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:31:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.724545+0000", 2024-09-21T10:31:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4239999999999998e-06 2024-09-21T10:31:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:31:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.724558+0000", 2024-09-21T10:31:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3409e-05 2024-09-21T10:31:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:31:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.724566+0000", 2024-09-21T10:31:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7810000000000005e-06 2024-09-21T10:31:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.724623+0000", 2024-09-21T10:31:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6266e-05 2024-09-21T10:31:53.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:31:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.724729+0000", 2024-09-21T10:31:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106512 2024-09-21T10:31:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:31:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.724941+0000", 2024-09-21T10:31:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00021148299999999999 2024-09-21T10:31:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:31:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.907836+0000", 2024-09-21T10:31:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.18289599000000001 2024-09-21T10:31:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.908164+0000", 2024-09-21T10:31:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00032793699999999999 2024-09-21T10:31:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.908331+0000", 2024-09-21T10:31:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016687599999999999 2024-09-21T10:31:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:31:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.908428+0000", 2024-09-21T10:31:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6784e-05 2024-09-21T10:31:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:31:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.908471+0000", 2024-09-21T10:31:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2657000000000002e-05 2024-09-21T10:31:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:31:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.46322.0:217835 10.2 e381/348 ondisk, result = 0)", 2024-09-21T10:31:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:30:32.724714+0000", 2024-09-21T10:31:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 81.039192055000001, 2024-09-21T10:31:53.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.18275556300000001, 2024-09-21T10:31:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:31:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:31:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-21T10:31:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:31:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:31:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.724714+0000", 2024-09-21T10:31:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:31:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.724714+0000", 2024-09-21T10:31:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:31:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.724714+0000", 2024-09-21T10:31:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7400000000000002e-07 2024-09-21T10:31:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:31:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.724719+0000", 2024-09-21T10:31:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6970000000000001e-06 2024-09-21T10:31:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:31:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.724729+0000", 2024-09-21T10:31:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6309999999999998e-06 2024-09-21T10:31:53.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:31:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.724735+0000", 2024-09-21T10:31:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6889999999999998e-06 2024-09-21T10:31:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.907351+0000", 2024-09-21T10:31:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.18261542 2024-09-21T10:31:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:31:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.907359+0000", 2024-09-21T10:31:53.886 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4789999999999996e-06 2024-09-21T10:31:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:31:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.907469+0000", 2024-09-21T10:31:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000109873 2024-09-21T10:31:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:31:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:217849 10.2 10:4b039756:::benchmark_data_smithi098_281896_object92824:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-21T10:31:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:30:32.736267+0000", 2024-09-21T10:31:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 81.027638289999999, 2024-09-21T10:31:53.887 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.217335049, 2024-09-21T10:31:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:31:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:31:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:31:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:31:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:31:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:31:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 217849 2024-09-21T10:31:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:31:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:31:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.736267+0000", 2024-09-21T10:31:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:31:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.736267+0000", 2024-09-21T10:31:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:31:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.736269+0000", 2024-09-21T10:31:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.903e-06 2024-09-21T10:31:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:31:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.736277+0000", 2024-09-21T10:31:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7800000000000001e-06 2024-09-21T10:31:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:31:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.903152+0000", 2024-09-21T10:31:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.16687505899999999 2024-09-21T10:31:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:31:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.903169+0000", 2024-09-21T10:31:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6866000000000001e-05 2024-09-21T10:31:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.908533+0000", 2024-09-21T10:31:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053634479999999998 2024-09-21T10:31:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:31:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.908691+0000", 2024-09-21T10:31:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015804299999999999 2024-09-21T10:31:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:31:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.908818+0000", 2024-09-21T10:31:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012755899999999999 2024-09-21T10:31:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.924103+0000", 2024-09-21T10:31:53.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015284496 2024-09-21T10:31:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:31:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.928919+0000", 2024-09-21T10:31:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048161539999999996 2024-09-21T10:31:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.953389+0000", 2024-09-21T10:31:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.024469897000000001 2024-09-21T10:31:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:31:53.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.953553+0000", 2024-09-21T10:31:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000164585 2024-09-21T10:31:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:31:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.953603+0000", 2024-09-21T10:31:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9258999999999997e-05 2024-09-21T10:31:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:31:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:217852 10.0 e381/348 10:0c6ac168:::benchmark_data_smithi098_281896_object92827:head v 381'13538, mlcod=381'13534)", 2024-09-21T10:31:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:30:32.744600+0000", 2024-09-21T10:31:53.893 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 81.019305978999995, 2024-09-21T10:31:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.181899012, 2024-09-21T10:31:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:31:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:31:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:31:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:31:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:31:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.744600+0000", 2024-09-21T10:31:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:31:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.744600+0000", 2024-09-21T10:31:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:31:53.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:31:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.744601+0000", 2024-09-21T10:31:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.054e-06 2024-09-21T10:31:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:31:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.744615+0000", 2024-09-21T10:31:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3949e-05 2024-09-21T10:31:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:31:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.903245+0000", 2024-09-21T10:31:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.15863033500000001 2024-09-21T10:31:53.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:31:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.903253+0000", 2024-09-21T10:31:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2640000000000008e-06 2024-09-21T10:31:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:31:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.907636+0000", 2024-09-21T10:31:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043822879999999998 2024-09-21T10:31:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:31:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.907659+0000", 2024-09-21T10:31:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2901e-05 2024-09-21T10:31:53.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:31:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.926357+0000", 2024-09-21T10:31:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018698172999999998 2024-09-21T10:31:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:31:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:31:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:31:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:32.926499+0000", 2024-09-21T10:31:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014204800000000001 2024-09-21T10:31:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:31:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:31:53.897 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:31:53.898 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:31:53.898 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:53.923 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:53.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:53.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:54.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:54.047 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:54.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:54.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:54.148 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:54.174 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:54.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:54.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:54.263 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:31:54.276 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:54.297 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:54.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:54.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:54.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:54.425 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:54.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:54.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:54.527 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:54.553 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:54.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:54.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:54.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:54 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1534604766' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:31:54.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:54 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1534604766' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:31:54.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:54 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1534604766' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:31:54.655 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:54.675 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:54.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:54.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:54.777 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:54.802 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:54.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:54.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:54.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:54.924 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:54.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:54.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:55.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:55.049 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:55.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:55.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:55.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:55.170 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:55.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:55.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:55.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:55.294 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:55.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:55.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:55.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:55.417 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:55.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:55.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:55.519 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:55.540 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:55.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:55.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:55.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:55 smithi098 ceph-mon[97580]: pgmap v1367: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:31:55.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:55 smithi098 ceph-mon[97580]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:31:55.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:55 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1534604766' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:31:55.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:55 smithi098 ceph-mon[97580]: osdmap e389: 8 total, 8 up, 8 in 2024-09-21T10:31:55.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:55 smithi098 ceph-mon[100914]: pgmap v1367: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:31:55.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:55 smithi098 ceph-mon[100914]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:31:55.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:55 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1534604766' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:31:55.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:55 smithi098 ceph-mon[100914]: osdmap e389: 8 total, 8 up, 8 in 2024-09-21T10:31:55.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:55 smithi196 ceph-mon[84784]: pgmap v1367: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:31:55.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:55 smithi196 ceph-mon[84784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:31:55.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:55 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1534604766' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:31:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:55 smithi196 ceph-mon[84784]: osdmap e389: 8 total, 8 up, 8 in 2024-09-21T10:31:55.641 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:55.666 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:55.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:55.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:55.768 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:55.793 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:55.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:55.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:55.895 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:55.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:31:55.920 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:55.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:55.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:56.022 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:56.048 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:56.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:56.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:56.149 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:56.266 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:56.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:56.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:56.368 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:56.393 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:56.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:56.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:56.495 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:56.518 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:56.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:56.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:56.620 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:56.626 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:56.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:56.647 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:56.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:56.747 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:56.769 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:56.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:56.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:56.871 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:56.891 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:56.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:56.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:56.992 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:57.018 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:57.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:57.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:57.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:57.140 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:57.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:57.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:57.205 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:57.242 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:57.267 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:57.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:57.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:57.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:57.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:31:57.394 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:57.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:57.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:57.496 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:57.520 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:57.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:57.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:57.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:57 smithi098 ceph-mon[97580]: pgmap v1369: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:31:57.532 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:57 smithi098 ceph-mon[100914]: pgmap v1369: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:31:57.622 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:57.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:57 smithi196 ceph-mon[84784]: pgmap v1369: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:31:57.646 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:57.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:57.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:57.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:57.772 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:57.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:57.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:57.874 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:57.897 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:57.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:57.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:57.999 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:58.025 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:58.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:58.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:58.127 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:58.153 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:58.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:58.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:58.254 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:58.280 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:58.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:58.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:58.382 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:58.406 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:58.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:58.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:58.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:58.529 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:58.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:58.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:58.631 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:58.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:58.654 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:58.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:58.755 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:58.781 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:58.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:58.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:58.883 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:58.908 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:58.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:58.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:59.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:59.031 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:59.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:59.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:59.060 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-21T10:31:59.133 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:59.184 INFO:tasks.workunit.client.0.smithi098.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-21T10:31:59.186 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T10:31:59.186 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-21T10:31:59.193 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:59.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:59.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:59.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:59.321 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:59.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:59.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:59.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:59.428 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-21T10:31:59.437 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:31:59 smithi098 ceph-mon[100914]: pgmap v1370: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:31:59.437 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:31:59 smithi098 ceph-mon[97580]: pgmap v1370: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:31:59.445 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:31:59.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:59.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:59.478 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-21T10:31:59.519 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T10:31:59.520 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T10:31:59.520 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-21T10:31:59.547 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:59.561 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T10:31:59.573 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:59.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:59.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:59.580 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-21T10:31:59.580 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-21T10:31:59.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:31:59 smithi196 ceph-mon[84784]: pgmap v1370: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:31:59.645 INFO:tasks.workunit:Running workunits matching cls/test_cls_rbd.sh on client.0... 2024-09-21T10:31:59.646 INFO:tasks.workunit:Running workunit cls/test_cls_rbd.sh... 2024-09-21T10:31:59.646 DEBUG:teuthology.orchestra.run.smithi098:workunit test cls/test_cls_rbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2024-09-21T10:31:59.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:59.699 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:59.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:59.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:59.801 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:59.826 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:59.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:59.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:59.928 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:59.946 INFO:tasks.workunit.client.0.smithi098.stdout:Running main() from gmock_main.cc 2024-09-21T10:31:59.946 INFO:tasks.workunit.client.0.smithi098.stdout:[==========] Running 70 tests from 1 test suite. 2024-09-21T10:31:59.946 INFO:tasks.workunit.client.0.smithi098.stdout:[----------] Global test environment set-up. 2024-09-21T10:31:59.946 INFO:tasks.workunit.client.0.smithi098.stdout:[----------] 70 tests from TestClsRbd 2024-09-21T10:31:59.954 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:31:59.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:59.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:00.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:00.077 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:00.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:00.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:00.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:00.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:00.219 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:32:00.220 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:32:00.302 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:00.326 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:32:00.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:00.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:00.336 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:32:00.336 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:32:00.347 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:32:00.348 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:32:00.364 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:32:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:32:00.380 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:32:00.380 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:32:00.397 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:32:00.397 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:32:00.412 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:32:00.413 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:32:00.428 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:00.429 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:32:00.429 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:32:00.444 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:32:00.444 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:32:00.451 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:00.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:00.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:00.553 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:00.574 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:00.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:00.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:00.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:32:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:32:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:32:00.676 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:00.702 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:32:00.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:00.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:00.803 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:00.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:00.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:00.927 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:00.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:32:00.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:00.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:01.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:01.076 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:01.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:01.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:01.097 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:32:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:32:00] "GET /metrics HTTP/1.1" 200 34183 "" "Prometheus/2.51.0" 2024-09-21T10:32:01.177 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:01.203 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:32:01.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:01.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:01.282 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.get_all_features 2024-09-21T10:32:01.304 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:01.327 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:32:01.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:01.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:01.328 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.get_all_features (47 ms) 2024-09-21T10:32:01.329 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.copyup 2024-09-21T10:32:01.429 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:01.451 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:01.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:01.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:01.512 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:01 smithi098 ceph-mon[100914]: pgmap v1371: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:01.512 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:01 smithi098 ceph-mon[100914]: from='client.46574 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:32:01.512 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:01 smithi098 ceph-mon[100914]: osdmap e390: 8 total, 8 up, 8 in 2024-09-21T10:32:01.512 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:01 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2072282554' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi098-315023-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:32:01.512 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:01 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi098-315023-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:32:01.512 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:01 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1386648657' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:32:01.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:01 smithi098 ceph-mon[97580]: pgmap v1371: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:01.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:01 smithi098 ceph-mon[97580]: from='client.46574 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:32:01.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:01 smithi098 ceph-mon[97580]: osdmap e390: 8 total, 8 up, 8 in 2024-09-21T10:32:01.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:01 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2072282554' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi098-315023-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:32:01.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:01 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi098-315023-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:32:01.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:01 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1386648657' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:32:01.553 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:01.554 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.copyup (225 ms) 2024-09-21T10:32:01.554 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.sparse_copyup 2024-09-21T10:32:01.571 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.sparse_copyup (18 ms) 2024-09-21T10:32:01.571 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.get_and_set_id 2024-09-21T10:32:01.577 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:32:01.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:01.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:01.581 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.get_and_set_id (10 ms) 2024-09-21T10:32:01.581 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.add_remove_child 2024-09-21T10:32:01.593 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.add_remove_child (12 ms) 2024-09-21T10:32:01.593 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.directory_methods 2024-09-21T10:32:01.619 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.directory_methods (26 ms) 2024-09-21T10:32:01.619 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.create 2024-09-21T10:32:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:01 smithi196 ceph-mon[84784]: pgmap v1371: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:01 smithi196 ceph-mon[84784]: from='client.46574 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:32:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:01 smithi196 ceph-mon[84784]: osdmap e390: 8 total, 8 up, 8 in 2024-09-21T10:32:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:01 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2072282554' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi098-315023-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:32:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:01 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi098-315023-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:32:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:01 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1386648657' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:32:01.653 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.create (34 ms) 2024-09-21T10:32:01.654 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.get_features 2024-09-21T10:32:01.655 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.get_features (2 ms) 2024-09-21T10:32:01.655 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.get_object_prefix 2024-09-21T10:32:01.657 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.get_object_prefix (2 ms) 2024-09-21T10:32:01.657 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.get_create_timestamp 2024-09-21T10:32:01.658 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.get_create_timestamp (1 ms) 2024-09-21T10:32:01.658 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.get_access_timestamp 2024-09-21T10:32:01.660 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.get_access_timestamp (2 ms) 2024-09-21T10:32:01.660 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.get_modify_timestamp 2024-09-21T10:32:01.661 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.get_modify_timestamp (1 ms) 2024-09-21T10:32:01.661 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.get_data_pool 2024-09-21T10:32:01.664 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.get_data_pool (3 ms) 2024-09-21T10:32:01.664 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.get_size 2024-09-21T10:32:01.667 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.get_size (3 ms) 2024-09-21T10:32:01.667 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.set_size 2024-09-21T10:32:01.671 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.set_size (4 ms) 2024-09-21T10:32:01.672 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.protection_status 2024-09-21T10:32:01.679 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:01.687 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.protection_status (16 ms) 2024-09-21T10:32:01.687 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.snapshot_limits 2024-09-21T10:32:01.694 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.snapshot_limits (7 ms) 2024-09-21T10:32:01.694 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.parents_v1 2024-09-21T10:32:01.700 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:01.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:01.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:01.760 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.parents_v1 (66 ms) 2024-09-21T10:32:01.760 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.parents_v2 2024-09-21T10:32:01.789 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.parents_v2 (29 ms) 2024-09-21T10:32:01.789 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.snapshots 2024-09-21T10:32:01.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:01.810 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.snapshots (21 ms) 2024-09-21T10:32:01.810 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.snapid_race 2024-09-21T10:32:01.817 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.snapid_race (7 ms) 2024-09-21T10:32:01.817 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.stripingv2 2024-09-21T10:32:01.825 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:01.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:01.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:01.826 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.stripingv2 (9 ms) 2024-09-21T10:32:01.827 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.object_map_save 2024-09-21T10:32:01.830 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.object_map_save (4 ms) 2024-09-21T10:32:01.830 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.object_map_resize 2024-09-21T10:32:01.837 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.object_map_resize (7 ms) 2024-09-21T10:32:01.837 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.object_map_update 2024-09-21T10:32:01.847 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.object_map_update (9 ms) 2024-09-21T10:32:01.847 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.object_map_load_enoent 2024-09-21T10:32:01.847 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.object_map_load_enoent (0 ms) 2024-09-21T10:32:01.847 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.object_map_snap_add 2024-09-21T10:32:01.853 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.object_map_snap_add (7 ms) 2024-09-21T10:32:01.853 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.object_map_snap_remove 2024-09-21T10:32:01.860 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.object_map_snap_remove (7 ms) 2024-09-21T10:32:01.860 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.flags 2024-09-21T10:32:01.865 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.flags (5 ms) 2024-09-21T10:32:01.865 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.metadata 2024-09-21T10:32:01.889 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.metadata (24 ms) 2024-09-21T10:32:01.889 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.set_features 2024-09-21T10:32:01.895 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.set_features (6 ms) 2024-09-21T10:32:01.895 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.mirror 2024-09-21T10:32:01.927 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:01.929 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.mirror (34 ms) 2024-09-21T10:32:01.929 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.mirror_image 2024-09-21T10:32:01.944 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.mirror_image (15 ms) 2024-09-21T10:32:01.944 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.mirror_image_status 2024-09-21T10:32:01.948 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:32:01.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:01.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:01.992 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:02.050 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:02.071 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:32:02.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:02.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:02.172 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:02.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:02.198 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:02.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:02.299 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:02.323 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:02.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:02.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:02.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:02.451 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:02.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:02.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:02.477 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1371,"stamp":"2024-09-21T10:31:59.658571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4516211,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":6011,"num_read_kb":13075,"num_write":921,"num_write_kb":12771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":654290,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2132,"ondisk_log_size":2132,"up":249,"acting":249,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":252,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8190588,"kb_used_data":1325216,"kb_used_omap":35,"kb_used_meta":6865244,"kb_avail":741770628,"statfs":{"total":767960285184,"available":759573123072,"internally_reserved":0,"allocated":1357021184,"data_stored":1339149169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36353,"internal_metadata":7030010367},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002221"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":69,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.349421+0000","last_change":"2024-09-21T10:28:00.289895+0000","last_active":"2024-09-21T10:31:54.349421+0000","last_peered":"2024-09-21T10:31:54.349421+0000","last_clean":"2024-09-21T10:31:54.349421+0000","last_became_active":"2024-09-21T10:28:00.289622+0000","last_became_peered":"2024-09-21T10:28:00.289622+0000","last_unstale":"2024-09-21T10:31:54.349421+0000","last_undegraded":"2024-09-21T10:31:54.349421+0000","last_fullsized":"2024-09-21T10:31:54.349421+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:35:31.514813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":419,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.915847+0000","last_change":"2024-09-21T10:24:44.305198+0000","last_active":"2024-09-21T10:31:43.915847+0000","last_peered":"2024-09-21T10:31:43.915847+0000","last_clean":"2024-09-21T10:31:43.915847+0000","last_became_active":"2024-09-21T10:24:44.297653+0000","last_became_peered":"2024-09-21T10:24:44.297653+0000","last_unstale":"2024-09-21T10:31:43.915847+0000","last_undegraded":"2024-09-21T10:31:43.915847+0000","last_fullsized":"2024-09-21T10:31:43.915847+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:19.320833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":256,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.919262+0000","last_change":"2024-09-21T10:28:00.294917+0000","last_active":"2024-09-21T10:31:43.919262+0000","last_peered":"2024-09-21T10:31:43.919262+0000","last_clean":"2024-09-21T10:31:43.919262+0000","last_became_active":"2024-09-21T10:28:00.294134+0000","last_became_peered":"2024-09-21T10:28:00.294134+0000","last_unstale":"2024-09-21T10:31:43.919262+0000","last_undegraded":"2024-09-21T10:31:43.919262+0000","last_fullsized":"2024-09-21T10:31:43.919262+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:29:05.780212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":205,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.915654+0000","last_change":"2024-09-21T10:28:00.287636+0000","last_active":"2024-09-21T10:31:43.915654+0000","last_peered":"2024-09-21T10:31:43.915654+0000","last_clean":"2024-09-21T10:31:43.915654+0000","last_became_active":"2024-09-21T10:28:00.287308+0000","last_became_peered":"2024-09-21T10:28:00.287308+0000","last_unstale":"2024-09-21T10:31:43.915654+0000","last_undegraded":"2024-09-21T10:31:43.915654+0000","last_fullsized":"2024-09-21T10:31:43.915654+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:41:46.132365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":665,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.349178+0000","last_change":"2024-09-21T10:28:00.297783+0000","last_active":"2024-09-21T10:31:54.349178+0000","last_peered":"2024-09-21T10:31:54.349178+0000","last_clean":"2024-09-21T10:31:54.349178+0000","last_became_active":"2024-09-21T10:28:00.297024+0000","last_became_peered":"2024-09-21T10:28:00.297024+0000","last_unstale":"2024-09-21T10:31:54.349178+0000","last_undegraded":"2024-09-21T10:31:54.349178+0000","last_fullsized":"2024-09-21T10:31:54.349178+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:24.382523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":78,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.917214+0000","last_change":"2024-09-21T10:24:44.304399+0000","last_active":"2024-09-21T10:31:43.917214+0000","last_peered":"2024-09-21T10:31:43.917214+0000","last_clean":"2024-09-21T10:31:43.917214+0000","last_became_active":"2024-09-21T10:24:44.303437+0000","last_became_peered":"2024-09-21T10:24:44.303437+0000","last_unstale":"2024-09-21T10:31:43.917214+0000","last_undegraded":"2024-09-21T10:31:43.917214+0000","last_fullsized":"2024-09-21T10:31:43.917214+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:32:56.638481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":69,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.349670+0000","last_change":"2024-09-21T10:28:00.293505+0000","last_active":"2024-09-21T10:31:54.349670+0000","last_peered":"2024-09-21T10:31:54.349670+0000","last_clean":"2024-09-21T10:31:54.349670+0000","last_became_active":"2024-09-21T10:28:00.293173+0000","last_became_peered":"2024-09-21T10:28:00.293173+0000","last_unstale":"2024-09-21T10:31:54.349670+0000","last_undegraded":"2024-09-21T10:31:54.349670+0000","last_fullsized":"2024-09-21T10:31:54.349670+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:16.878847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":361,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.858856+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:31:43.858856+0000","last_peered":"2024-09-21T10:31:43.858856+0000","last_clean":"2024-09-21T10:31:43.858856+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:31:43.858856+0000","last_undegraded":"2024-09-21T10:31:43.858856+0000","last_fullsized":"2024-09-21T10:31:43.858856+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":410,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.923119+0000","last_change":"2024-09-21T10:24:44.325196+0000","last_active":"2024-09-21T10:31:43.923119+0000","last_peered":"2024-09-21T10:31:43.923119+0000","last_clean":"2024-09-21T10:31:43.923119+0000","last_became_active":"2024-09-21T10:24:44.299614+0000","last_became_peered":"2024-09-21T10:24:44.299614+0000","last_unstale":"2024-09-21T10:31:43.923119+0000","last_undegraded":"2024-09-21T10:31:43.923119+0000","last_fullsized":"2024-09-21T10:31:43.923119+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:14.558663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":142,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.919418+0000","last_change":"2024-09-21T10:26:41.305571+0000","last_active":"2024-09-21T10:31:43.919418+0000","last_peered":"2024-09-21T10:31:43.919418+0000","last_clean":"2024-09-21T10:31:43.919418+0000","last_became_active":"2024-09-21T10:26:41.305135+0000","last_became_peered":"2024-09-21T10:26:41.305135+0000","last_unstale":"2024-09-21T10:31:43.919418+0000","last_undegraded":"2024-09-21T10:31:43.919418+0000","last_fullsized":"2024-09-21T10:31:43.919418+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:49:55.452467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":411,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.922832+0000","last_change":"2024-09-21T10:24:44.323182+0000","last_active":"2024-09-21T10:31:43.922832+0000","last_peered":"2024-09-21T10:31:43.922832+0000","last_clean":"2024-09-21T10:31:43.922832+0000","last_became_active":"2024-09-21T10:24:44.299698+0000","last_became_peered":"2024-09-21T10:24:44.299698+0000","last_unstale":"2024-09-21T10:31:43.922832+0000","last_undegraded":"2024-09-21T10:31:43.922832+0000","last_fullsized":"2024-09-21T10:31:43.922832+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:56.571057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":88,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.917400+0000","last_change":"2024-09-21T10:28:00.281253+0000","last_active":"2024-09-21T10:31:43.917400+0000","last_peered":"2024-09-21T10:31:43.917400+0000","last_clean":"2024-09-21T10:31:43.917400+0000","last_became_active":"2024-09-21T10:28:00.280773+0000","last_became_peered":"2024-09-21T10:28:00.280773+0000","last_unstale":"2024-09-21T10:31:43.917400+0000","last_undegraded":"2024-09-21T10:31:43.917400+0000","last_fullsized":"2024-09-21T10:31:43.917400+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:32:48.557279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":337,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.924946+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:31:43.924946+0000","last_peered":"2024-09-21T10:31:43.924946+0000","last_clean":"2024-09-21T10:31:43.924946+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:31:43.924946+0000","last_undegraded":"2024-09-21T10:31:43.924946+0000","last_fullsized":"2024-09-21T10:31:43.924946+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":352,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.858640+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:31:43.858640+0000","last_peered":"2024-09-21T10:31:43.858640+0000","last_clean":"2024-09-21T10:31:43.858640+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:31:43.858640+0000","last_undegraded":"2024-09-21T10:31:43.858640+0000","last_fullsized":"2024-09-21T10:31:43.858640+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":69,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.349416+0000","last_change":"2024-09-21T10:28:00.288968+0000","last_active":"2024-09-21T10:31:54.349416+0000","last_peered":"2024-09-21T10:31:54.349416+0000","last_clean":"2024-09-21T10:31:54.349416+0000","last_became_active":"2024-09-21T10:28:00.288244+0000","last_became_peered":"2024-09-21T10:28:00.288244+0000","last_unstale":"2024-09-21T10:31:54.349416+0000","last_undegraded":"2024-09-21T10:31:54.349416+0000","last_fullsized":"2024-09-21T10:31:54.349416+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:42.089701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":687,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348706+0000","last_change":"2024-09-21T10:24:44.299417+0000","last_active":"2024-09-21T10:31:54.348706+0000","last_peered":"2024-09-21T10:31:54.348706+0000","last_clean":"2024-09-21T10:31:54.348706+0000","last_became_active":"2024-09-21T10:24:44.299116+0000","last_became_peered":"2024-09-21T10:24:44.299116+0000","last_unstale":"2024-09-21T10:31:54.348706+0000","last_undegraded":"2024-09-21T10:31:54.348706+0000","last_fullsized":"2024-09-21T10:31:54.348706+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:44.524841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"82'100","reported_seq":510,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.916081+0000","last_change":"2024-09-21T10:28:00.285919+0000","last_active":"2024-09-21T10:31:43.916081+0000","last_peered":"2024-09-21T10:31:43.916081+0000","last_clean":"2024-09-21T10:31:43.916081+0000","last_became_active":"2024-09-21T10:28:00.285324+0000","last_became_peered":"2024-09-21T10:28:00.285324+0000","last_unstale":"2024-09-21T10:31:43.916081+0000","last_undegraded":"2024-09-21T10:31:43.916081+0000","last_fullsized":"2024-09-21T10:31:43.916081+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.728195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.728195+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:00.841610+0000","scrub_duration":0.0069698140000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":73,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348772+0000","last_change":"2024-09-21T10:28:00.303125+0000","last_active":"2024-09-21T10:31:54.348772+0000","last_peered":"2024-09-21T10:31:54.348772+0000","last_clean":"2024-09-21T10:31:54.348772+0000","last_became_active":"2024-09-21T10:28:00.293684+0000","last_became_peered":"2024-09-21T10:28:00.293684+0000","last_unstale":"2024-09-21T10:31:54.348772+0000","last_undegraded":"2024-09-21T10:31:54.348772+0000","last_fullsized":"2024-09-21T10:31:54.348772+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:04.495913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":88,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.919143+0000","last_change":"2024-09-21T10:28:00.283456+0000","last_active":"2024-09-21T10:31:43.919143+0000","last_peered":"2024-09-21T10:31:43.919143+0000","last_clean":"2024-09-21T10:31:43.919143+0000","last_became_active":"2024-09-21T10:28:00.282881+0000","last_became_peered":"2024-09-21T10:28:00.282881+0000","last_unstale":"2024-09-21T10:31:43.919143+0000","last_undegraded":"2024-09-21T10:31:43.919143+0000","last_fullsized":"2024-09-21T10:31:43.919143+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:43.681481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"82'100","reported_seq":189,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.316230+0000","last_change":"2024-09-21T10:28:00.301695+0000","last_active":"2024-09-21T10:31:54.316230+0000","last_peered":"2024-09-21T10:31:54.316230+0000","last_clean":"2024-09-21T10:31:54.316230+0000","last_became_active":"2024-09-21T10:28:00.289856+0000","last_became_peered":"2024-09-21T10:28:00.289856+0000","last_unstale":"2024-09-21T10:31:54.316230+0000","last_undegraded":"2024-09-21T10:31:54.316230+0000","last_fullsized":"2024-09-21T10:31:54.316230+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:27.234001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:27.234001+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:02.389036+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":403,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.858767+0000","last_change":"2024-09-21T10:24:44.309470+0000","last_active":"2024-09-21T10:31:43.858767+0000","last_peered":"2024-09-21T10:31:43.858767+0000","last_clean":"2024-09-21T10:31:43.858767+0000","last_became_active":"2024-09-21T10:24:44.308836+0000","last_became_peered":"2024-09-21T10:24:44.308836+0000","last_unstale":"2024-09-21T10:31:43.858767+0000","last_undegraded":"2024-09-21T10:31:43.858767+0000","last_fullsized":"2024-09-21T10:31:43.858767+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:09:44.622418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":78,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.920700+0000","last_change":"2024-09-21T10:24:44.305135+0000","last_active":"2024-09-21T10:31:43.920700+0000","last_peered":"2024-09-21T10:31:43.920700+0000","last_clean":"2024-09-21T10:31:43.920700+0000","last_became_active":"2024-09-21T10:24:44.304839+0000","last_became_peered":"2024-09-21T10:24:44.304839+0000","last_unstale":"2024-09-21T10:31:43.920700+0000","last_undegraded":"2024-09-21T10:31:43.920700+0000","last_fullsized":"2024-09-21T10:31:43.920700+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:02.264404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":118,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348284+0000","last_change":"2024-09-21T10:25:39.872680+0000","last_active":"2024-09-21T10:31:54.348284+0000","last_peered":"2024-09-21T10:31:54.348284+0000","last_clean":"2024-09-21T10:31:54.348284+0000","last_became_active":"2024-09-21T10:25:39.872312+0000","last_became_peered":"2024-09-21T10:25:39.872312+0000","last_unstale":"2024-09-21T10:31:54.348284+0000","last_undegraded":"2024-09-21T10:31:54.348284+0000","last_fullsized":"2024-09-21T10:31:54.348284+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:54.942046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":78,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.927242+0000","last_change":"2024-09-21T10:24:44.310484+0000","last_active":"2024-09-21T10:31:43.927242+0000","last_peered":"2024-09-21T10:31:43.927242+0000","last_clean":"2024-09-21T10:31:43.927242+0000","last_became_active":"2024-09-21T10:24:44.307209+0000","last_became_peered":"2024-09-21T10:24:44.307209+0000","last_unstale":"2024-09-21T10:31:43.927242+0000","last_undegraded":"2024-09-21T10:31:43.927242+0000","last_fullsized":"2024-09-21T10:31:43.927242+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:48:36.767539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'100","reported_seq":173,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.316368+0000","last_change":"2024-09-21T10:23:26.197358+0000","last_active":"2024-09-21T10:31:54.316368+0000","last_peered":"2024-09-21T10:31:54.316368+0000","last_clean":"2024-09-21T10:31:54.316368+0000","last_became_active":"2024-09-21T10:23:20.695174+0000","last_became_peered":"2024-09-21T10:23:20.695174+0000","last_unstale":"2024-09-21T10:31:54.316368+0000","last_undegraded":"2024-09-21T10:31:54.316368+0000","last_fullsized":"2024-09-21T10:31:54.316368+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:26.197291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:26.197291+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:33.961031+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":69,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348488+0000","last_change":"2024-09-21T10:28:00.287560+0000","last_active":"2024-09-21T10:31:54.348488+0000","last_peered":"2024-09-21T10:31:54.348488+0000","last_clean":"2024-09-21T10:31:54.348488+0000","last_became_active":"2024-09-21T10:28:00.285076+0000","last_became_peered":"2024-09-21T10:28:00.285076+0000","last_unstale":"2024-09-21T10:31:54.348488+0000","last_undegraded":"2024-09-21T10:31:54.348488+0000","last_fullsized":"2024-09-21T10:31:54.348488+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:49.950523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":682,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348325+0000","last_change":"2024-09-21T10:28:00.282062+0000","last_active":"2024-09-21T10:31:54.348325+0000","last_peered":"2024-09-21T10:31:54.348325+0000","last_clean":"2024-09-21T10:31:54.348325+0000","last_became_active":"2024-09-21T10:28:00.281593+0000","last_became_peered":"2024-09-21T10:28:00.281593+0000","last_unstale":"2024-09-21T10:31:54.348325+0000","last_undegraded":"2024-09-21T10:31:54.348325+0000","last_fullsized":"2024-09-21T10:31:54.348325+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:45.704840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":491,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.915650+0000","last_change":"2024-09-21T10:23:18.790100+0000","last_active":"2024-09-21T10:31:43.915650+0000","last_peered":"2024-09-21T10:31:43.915650+0000","last_clean":"2024-09-21T10:31:43.915650+0000","last_became_active":"2024-09-21T10:23:14.677610+0000","last_became_peered":"2024-09-21T10:23:14.677610+0000","last_unstale":"2024-09-21T10:31:43.915650+0000","last_undegraded":"2024-09-21T10:31:43.915650+0000","last_fullsized":"2024-09-21T10:31:43.915650+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:18.790017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:18.790017+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:56.562241+0000","scrub_duration":0.0080400790000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":584,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.920156+0000","last_change":"2024-09-21T10:26:41.279374+0000","last_active":"2024-09-21T10:31:43.920156+0000","last_peered":"2024-09-21T10:31:43.920156+0000","last_clean":"2024-09-21T10:31:43.920156+0000","last_became_active":"2024-09-21T10:26:41.278994+0000","last_became_peered":"2024-09-21T10:26:41.278994+0000","last_unstale":"2024-09-21T10:31:43.920156+0000","last_undegraded":"2024-09-21T10:31:43.920156+0000","last_fullsized":"2024-09-21T10:31:43.920156+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:49:57.821175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":482,"num_read_kb":1581,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":69,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348815+0000","last_change":"2024-09-21T10:28:00.295436+0000","last_active":"2024-09-21T10:31:54.348815+0000","last_peered":"2024-09-21T10:31:54.348815+0000","last_clean":"2024-09-21T10:31:54.348815+0000","last_became_active":"2024-09-21T10:28:00.295135+0000","last_became_peered":"2024-09-21T10:28:00.295135+0000","last_unstale":"2024-09-21T10:31:54.348815+0000","last_undegraded":"2024-09-21T10:31:54.348815+0000","last_fullsized":"2024-09-21T10:31:54.348815+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:14:03.027395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":375,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.859068+0000","last_change":"2024-09-21T10:24:44.308008+0000","last_active":"2024-09-21T10:31:43.859068+0000","last_peered":"2024-09-21T10:31:43.859068+0000","last_clean":"2024-09-21T10:31:43.859068+0000","last_became_active":"2024-09-21T10:24:44.307595+0000","last_became_peered":"2024-09-21T10:24:44.307595+0000","last_unstale":"2024-09-21T10:31:43.859068+0000","last_undegraded":"2024-09-21T10:31:43.859068+0000","last_fullsized":"2024-09-21T10:31:43.859068+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:33:49.448676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"82'100","reported_seq":181,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.349112+0000","last_change":"2024-09-21T10:24:44.309154+0000","last_active":"2024-09-21T10:31:54.349112+0000","last_peered":"2024-09-21T10:31:54.349112+0000","last_clean":"2024-09-21T10:31:54.349112+0000","last_became_active":"2024-09-21T10:24:44.308330+0000","last_became_peered":"2024-09-21T10:24:44.308330+0000","last_unstale":"2024-09-21T10:31:54.349112+0000","last_undegraded":"2024-09-21T10:31:54.349112+0000","last_fullsized":"2024-09-21T10:31:54.349112+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.641718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.641718+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:10.158739+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":733,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.915554+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:31:43.915554+0000","last_peered":"2024-09-21T10:31:43.915554+0000","last_clean":"2024-09-21T10:31:43.915554+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:31:43.915554+0000","last_undegraded":"2024-09-21T10:31:43.915554+0000","last_fullsized":"2024-09-21T10:31:43.915554+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":695,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348491+0000","last_change":"2024-09-21T10:28:00.296831+0000","last_active":"2024-09-21T10:31:54.348491+0000","last_peered":"2024-09-21T10:31:54.348491+0000","last_clean":"2024-09-21T10:31:54.348491+0000","last_became_active":"2024-09-21T10:28:00.296527+0000","last_became_peered":"2024-09-21T10:28:00.296527+0000","last_unstale":"2024-09-21T10:31:54.348491+0000","last_undegraded":"2024-09-21T10:31:54.348491+0000","last_fullsized":"2024-09-21T10:31:54.348491+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:18.024868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":663,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.349255+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:31:54.349255+0000","last_peered":"2024-09-21T10:31:54.349255+0000","last_clean":"2024-09-21T10:31:54.349255+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:31:54.349255+0000","last_undegraded":"2024-09-21T10:31:54.349255+0000","last_fullsized":"2024-09-21T10:31:54.349255+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"82'100","reported_seq":327,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348948+0000","last_change":"2024-09-21T10:24:44.294460+0000","last_active":"2024-09-21T10:31:54.348948+0000","last_peered":"2024-09-21T10:31:54.348948+0000","last_clean":"2024-09-21T10:31:54.348948+0000","last_became_active":"2024-09-21T10:24:44.293972+0000","last_became_peered":"2024-09-21T10:24:44.293972+0000","last_unstale":"2024-09-21T10:31:54.348948+0000","last_undegraded":"2024-09-21T10:31:54.348948+0000","last_fullsized":"2024-09-21T10:31:54.348948+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:26.199522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:26.199522+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:26.135171+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":1473,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.316431+0000","last_change":"2024-09-21T10:28:03.346064+0000","last_active":"2024-09-21T10:31:54.316431+0000","last_peered":"2024-09-21T10:31:54.316431+0000","last_clean":"2024-09-21T10:31:54.316431+0000","last_became_active":"2024-09-21T10:28:03.345313+0000","last_became_peered":"2024-09-21T10:28:03.345313+0000","last_unstale":"2024-09-21T10:31:54.316431+0000","last_undegraded":"2024-09-21T10:31:54.316431+0000","last_fullsized":"2024-09-21T10:31:54.316431+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:32.097521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":535,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348953+0000","last_change":"2024-09-21T10:08:18.470748+0000","last_active":"2024-09-21T10:31:54.348953+0000","last_peered":"2024-09-21T10:31:54.348953+0000","last_clean":"2024-09-21T10:31:54.348953+0000","last_became_active":"2024-09-21T10:08:18.470227+0000","last_became_peered":"2024-09-21T10:08:18.470227+0000","last_unstale":"2024-09-21T10:31:54.348953+0000","last_undegraded":"2024-09-21T10:31:54.348953+0000","last_fullsized":"2024-09-21T10:31:54.348953+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:16.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":658,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.316756+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:31:54.316756+0000","last_peered":"2024-09-21T10:31:54.316756+0000","last_clean":"2024-09-21T10:31:54.316756+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:31:54.316756+0000","last_undegraded":"2024-09-21T10:31:54.316756+0000","last_fullsized":"2024-09-21T10:31:54.316756+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"82'100","reported_seq":320,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348380+0000","last_change":"2024-09-21T10:23:21.277739+0000","last_active":"2024-09-21T10:31:54.348380+0000","last_peered":"2024-09-21T10:31:54.348380+0000","last_clean":"2024-09-21T10:31:54.348380+0000","last_became_active":"2024-09-21T10:23:20.689733+0000","last_became_peered":"2024-09-21T10:23:20.689733+0000","last_unstale":"2024-09-21T10:31:54.348380+0000","last_undegraded":"2024-09-21T10:31:54.348380+0000","last_fullsized":"2024-09-21T10:31:54.348380+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.277663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.277663+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:01:54.523189+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":1134,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348850+0000","last_change":"2024-09-21T10:07:43.794767+0000","last_active":"2024-09-21T10:31:54.348850+0000","last_peered":"2024-09-21T10:31:54.348850+0000","last_clean":"2024-09-21T10:31:54.348850+0000","last_became_active":"2024-09-21T10:07:43.794062+0000","last_became_peered":"2024-09-21T10:07:43.794062+0000","last_unstale":"2024-09-21T10:31:54.348850+0000","last_undegraded":"2024-09-21T10:31:54.348850+0000","last_fullsized":"2024-09-21T10:31:54.348850+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:51.977955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":78,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.918109+0000","last_change":"2024-09-21T10:24:44.299492+0000","last_active":"2024-09-21T10:31:43.918109+0000","last_peered":"2024-09-21T10:31:43.918109+0000","last_clean":"2024-09-21T10:31:43.918109+0000","last_became_active":"2024-09-21T10:24:44.299187+0000","last_became_peered":"2024-09-21T10:24:44.299187+0000","last_unstale":"2024-09-21T10:31:43.918109+0000","last_undegraded":"2024-09-21T10:31:43.918109+0000","last_fullsized":"2024-09-21T10:31:43.918109+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:58.500386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":588,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.316565+0000","last_change":"2024-09-21T10:28:00.289453+0000","last_active":"2024-09-21T10:31:54.316565+0000","last_peered":"2024-09-21T10:31:54.316565+0000","last_clean":"2024-09-21T10:31:54.316565+0000","last_became_active":"2024-09-21T10:28:00.289030+0000","last_became_peered":"2024-09-21T10:28:00.289030+0000","last_unstale":"2024-09-21T10:31:54.316565+0000","last_undegraded":"2024-09-21T10:31:54.316565+0000","last_fullsized":"2024-09-21T10:31:54.316565+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:21.959998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"82'100","reported_seq":253,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.859203+0000","last_change":"2024-09-21T10:28:00.294997+0000","last_active":"2024-09-21T10:31:43.859203+0000","last_peered":"2024-09-21T10:31:43.859203+0000","last_clean":"2024-09-21T10:31:43.859203+0000","last_became_active":"2024-09-21T10:28:00.294595+0000","last_became_peered":"2024-09-21T10:28:00.294595+0000","last_unstale":"2024-09-21T10:31:43.859203+0000","last_undegraded":"2024-09-21T10:31:43.859203+0000","last_fullsized":"2024-09-21T10:31:43.859203+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:30.569982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:30.569982+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:01.422939+0000","scrub_duration":0.0051309349999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":187,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.919415+0000","last_change":"2024-09-21T10:28:00.294966+0000","last_active":"2024-09-21T10:31:43.919415+0000","last_peered":"2024-09-21T10:31:43.919415+0000","last_clean":"2024-09-21T10:31:43.919415+0000","last_became_active":"2024-09-21T10:28:00.294552+0000","last_became_peered":"2024-09-21T10:28:00.294552+0000","last_unstale":"2024-09-21T10:31:43.919415+0000","last_undegraded":"2024-09-21T10:31:43.919415+0000","last_fullsized":"2024-09-21T10:31:43.919415+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:47.292125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":1241,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348467+0000","last_change":"2024-09-21T10:08:18.468995+0000","last_active":"2024-09-21T10:31:54.348467+0000","last_peered":"2024-09-21T10:31:54.348467+0000","last_clean":"2024-09-21T10:31:54.348467+0000","last_became_active":"2024-09-21T10:08:18.468243+0000","last_became_peered":"2024-09-21T10:08:18.468243+0000","last_unstale":"2024-09-21T10:31:54.348467+0000","last_undegraded":"2024-09-21T10:31:54.348467+0000","last_fullsized":"2024-09-21T10:31:54.348467+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:45.520968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":567,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.349167+0000","last_change":"2024-09-21T10:24:44.309614+0000","last_active":"2024-09-21T10:31:54.349167+0000","last_peered":"2024-09-21T10:31:54.349167+0000","last_clean":"2024-09-21T10:31:54.349167+0000","last_became_active":"2024-09-21T10:24:44.309187+0000","last_became_peered":"2024-09-21T10:24:44.309187+0000","last_unstale":"2024-09-21T10:31:54.349167+0000","last_undegraded":"2024-09-21T10:31:54.349167+0000","last_fullsized":"2024-09-21T10:31:54.349167+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:01.074803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'100","reported_seq":263,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.859079+0000","last_change":"2024-09-21T10:28:00.295277+0000","last_active":"2024-09-21T10:31:43.859079+0000","last_peered":"2024-09-21T10:31:43.859079+0000","last_clean":"2024-09-21T10:31:43.859079+0000","last_became_active":"2024-09-21T10:28:00.294016+0000","last_became_peered":"2024-09-21T10:28:00.294016+0000","last_unstale":"2024-09-21T10:31:43.859079+0000","last_undegraded":"2024-09-21T10:31:43.859079+0000","last_fullsized":"2024-09-21T10:31:43.859079+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:24.536711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:24.536711+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:28:49.984470+0000","scrub_duration":0.0047525650000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":536,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348381+0000","last_change":"2024-09-21T10:08:18.469907+0000","last_active":"2024-09-21T10:31:54.348381+0000","last_peered":"2024-09-21T10:31:54.348381+0000","last_clean":"2024-09-21T10:31:54.348381+0000","last_became_active":"2024-09-21T10:08:18.469443+0000","last_became_peered":"2024-09-21T10:08:18.469443+0000","last_unstale":"2024-09-21T10:31:54.348381+0000","last_undegraded":"2024-09-21T10:31:54.348381+0000","last_fullsized":"2024-09-21T10:31:54.348381+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:15.645803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":811,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.924810+0000","last_change":"2024-09-21T10:24:44.334847+0000","last_active":"2024-09-21T10:31:43.924810+0000","last_peered":"2024-09-21T10:31:43.924810+0000","last_clean":"2024-09-21T10:31:43.924810+0000","last_became_active":"2024-09-21T10:24:44.301321+0000","last_became_peered":"2024-09-21T10:24:44.301321+0000","last_unstale":"2024-09-21T10:31:43.924810+0000","last_undegraded":"2024-09-21T10:31:43.924810+0000","last_fullsized":"2024-09-21T10:31:43.924810+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:37.628940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":63,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.920079+0000","last_change":"2024-09-21T10:26:41.278937+0000","last_active":"2024-09-21T10:31:43.920079+0000","last_peered":"2024-09-21T10:31:43.920079+0000","last_clean":"2024-09-21T10:31:43.920079+0000","last_became_active":"2024-09-21T10:26:41.277726+0000","last_became_peered":"2024-09-21T10:26:41.277726+0000","last_unstale":"2024-09-21T10:31:43.920079+0000","last_undegraded":"2024-09-21T10:31:43.920079+0000","last_fullsized":"2024-09-21T10:31:43.920079+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:04.741057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"82'100","reported_seq":78,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.920543+0000","last_change":"2024-09-21T10:24:44.296758+0000","last_active":"2024-09-21T10:31:43.920543+0000","last_peered":"2024-09-21T10:31:43.920543+0000","last_clean":"2024-09-21T10:31:43.920543+0000","last_became_active":"2024-09-21T10:24:44.296403+0000","last_became_peered":"2024-09-21T10:24:44.296403+0000","last_unstale":"2024-09-21T10:31:43.920543+0000","last_undegraded":"2024-09-21T10:31:43.920543+0000","last_fullsized":"2024-09-21T10:31:43.920543+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.105746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.105746+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:20:12.502081+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":73,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.349617+0000","last_change":"2024-09-21T10:28:00.300952+0000","last_active":"2024-09-21T10:31:54.349617+0000","last_peered":"2024-09-21T10:31:54.349617+0000","last_clean":"2024-09-21T10:31:54.349617+0000","last_became_active":"2024-09-21T10:28:00.293337+0000","last_became_peered":"2024-09-21T10:28:00.293337+0000","last_unstale":"2024-09-21T10:31:54.349617+0000","last_undegraded":"2024-09-21T10:31:54.349617+0000","last_fullsized":"2024-09-21T10:31:54.349617+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:58.409761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":960,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348627+0000","last_change":"2024-09-21T10:24:46.844278+0000","last_active":"2024-09-21T10:31:54.348627+0000","last_peered":"2024-09-21T10:31:54.348627+0000","last_clean":"2024-09-21T10:31:54.348627+0000","last_became_active":"2024-09-21T10:24:46.840961+0000","last_became_peered":"2024-09-21T10:24:46.840961+0000","last_unstale":"2024-09-21T10:31:54.348627+0000","last_undegraded":"2024-09-21T10:31:54.348627+0000","last_fullsized":"2024-09-21T10:31:54.348627+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:49:48.823524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":69,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348215+0000","last_change":"2024-09-21T10:28:00.293864+0000","last_active":"2024-09-21T10:31:54.348215+0000","last_peered":"2024-09-21T10:31:54.348215+0000","last_clean":"2024-09-21T10:31:54.348215+0000","last_became_active":"2024-09-21T10:28:00.293494+0000","last_became_peered":"2024-09-21T10:28:00.293494+0000","last_unstale":"2024-09-21T10:31:54.348215+0000","last_undegraded":"2024-09-21T10:31:54.348215+0000","last_fullsized":"2024-09-21T10:31:54.348215+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:37.098068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"82'100","reported_seq":76,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.349503+0000","last_change":"2024-09-21T10:28:00.305255+0000","last_active":"2024-09-21T10:31:54.349503+0000","last_peered":"2024-09-21T10:31:54.349503+0000","last_clean":"2024-09-21T10:31:54.349503+0000","last_became_active":"2024-09-21T10:28:00.290981+0000","last_became_peered":"2024-09-21T10:28:00.290981+0000","last_unstale":"2024-09-21T10:31:54.349503+0000","last_undegraded":"2024-09-21T10:31:54.349503+0000","last_fullsized":"2024-09-21T10:31:54.349503+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.759355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.759355+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:32:44.600746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":69,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":700,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.349117+0000","last_change":"2024-09-21T10:24:44.307344+0000","last_active":"2024-09-21T10:31:54.349117+0000","last_peered":"2024-09-21T10:31:54.349117+0000","last_clean":"2024-09-21T10:31:54.349117+0000","last_became_active":"2024-09-21T10:24:44.298936+0000","last_became_peered":"2024-09-21T10:24:44.298936+0000","last_unstale":"2024-09-21T10:31:54.349117+0000","last_undegraded":"2024-09-21T10:31:54.349117+0000","last_fullsized":"2024-09-21T10:31:54.349117+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:58:26.025877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":615,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.866072+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:31:43.866072+0000","last_peered":"2024-09-21T10:31:43.866072+0000","last_clean":"2024-09-21T10:31:43.866072+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:31:43.866072+0000","last_undegraded":"2024-09-21T10:31:43.866072+0000","last_fullsized":"2024-09-21T10:31:43.866072+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":78,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.917283+0000","last_change":"2024-09-21T10:24:44.303781+0000","last_active":"2024-09-21T10:31:43.917283+0000","last_peered":"2024-09-21T10:31:43.917283+0000","last_clean":"2024-09-21T10:31:43.917283+0000","last_became_active":"2024-09-21T10:24:44.302058+0000","last_became_peered":"2024-09-21T10:24:44.302058+0000","last_unstale":"2024-09-21T10:31:43.917283+0000","last_undegraded":"2024-09-21T10:31:43.917283+0000","last_fullsized":"2024-09-21T10:31:43.917283+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:30.371229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":78,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.918184+0000","last_change":"2024-09-21T10:24:44.303417+0000","last_active":"2024-09-21T10:31:43.918184+0000","last_peered":"2024-09-21T10:31:43.918184+0000","last_clean":"2024-09-21T10:31:43.918184+0000","last_became_active":"2024-09-21T10:24:44.303069+0000","last_became_peered":"2024-09-21T10:24:44.303069+0000","last_unstale":"2024-09-21T10:31:43.918184+0000","last_undegraded":"2024-09-21T10:31:43.918184+0000","last_fullsized":"2024-09-21T10:31:43.918184+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:30:30.966228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":88,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.919075+0000","last_change":"2024-09-21T10:24:44.321421+0000","last_active":"2024-09-21T10:31:43.919075+0000","last_peered":"2024-09-21T10:31:43.919075+0000","last_clean":"2024-09-21T10:31:43.919075+0000","last_became_active":"2024-09-21T10:24:44.297082+0000","last_became_peered":"2024-09-21T10:24:44.297082+0000","last_unstale":"2024-09-21T10:31:43.919075+0000","last_undegraded":"2024-09-21T10:31:43.919075+0000","last_fullsized":"2024-09-21T10:31:43.919075+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:35:37.704881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":535,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348940+0000","last_change":"2024-09-21T10:08:18.470378+0000","last_active":"2024-09-21T10:31:54.348940+0000","last_peered":"2024-09-21T10:31:54.348940+0000","last_clean":"2024-09-21T10:31:54.348940+0000","last_became_active":"2024-09-21T10:08:18.469482+0000","last_became_peered":"2024-09-21T10:08:18.469482+0000","last_unstale":"2024-09-21T10:31:54.348940+0000","last_undegraded":"2024-09-21T10:31:54.348940+0000","last_fullsized":"2024-09-21T10:31:54.348940+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:35.628724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":452,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.924672+0000","last_change":"2024-09-21T10:28:00.296771+0000","last_active":"2024-09-21T10:31:43.924672+0000","last_peered":"2024-09-21T10:31:43.924672+0000","last_clean":"2024-09-21T10:31:43.924672+0000","last_became_active":"2024-09-21T10:28:00.285336+0000","last_became_peered":"2024-09-21T10:28:00.285336+0000","last_unstale":"2024-09-21T10:31:43.924672+0000","last_undegraded":"2024-09-21T10:31:43.924672+0000","last_fullsized":"2024-09-21T10:31:43.924672+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:06:50.848492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":160,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.920064+0000","last_change":"2024-09-21T10:26:41.644952+0000","last_active":"2024-09-21T10:31:43.920064+0000","last_peered":"2024-09-21T10:31:43.920064+0000","last_clean":"2024-09-21T10:31:43.920064+0000","last_became_active":"2024-09-21T10:26:41.644592+0000","last_became_peered":"2024-09-21T10:26:41.644592+0000","last_unstale":"2024-09-21T10:31:43.920064+0000","last_undegraded":"2024-09-21T10:31:43.920064+0000","last_fullsized":"2024-09-21T10:31:43.920064+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:44:44.183877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":69,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348560+0000","last_change":"2024-09-21T10:28:00.294338+0000","last_active":"2024-09-21T10:31:54.348560+0000","last_peered":"2024-09-21T10:31:54.348560+0000","last_clean":"2024-09-21T10:31:54.348560+0000","last_became_active":"2024-09-21T10:28:00.294044+0000","last_became_peered":"2024-09-21T10:28:00.294044+0000","last_unstale":"2024-09-21T10:31:54.348560+0000","last_undegraded":"2024-09-21T10:31:54.348560+0000","last_fullsized":"2024-09-21T10:31:54.348560+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:43:30.999988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":665,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.349083+0000","last_change":"2024-09-21T10:28:00.296935+0000","last_active":"2024-09-21T10:31:54.349083+0000","last_peered":"2024-09-21T10:31:54.349083+0000","last_clean":"2024-09-21T10:31:54.349083+0000","last_became_active":"2024-09-21T10:28:00.296562+0000","last_became_peered":"2024-09-21T10:28:00.296562+0000","last_unstale":"2024-09-21T10:31:54.349083+0000","last_undegraded":"2024-09-21T10:31:54.349083+0000","last_fullsized":"2024-09-21T10:31:54.349083+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:06.863310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":69,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348403+0000","last_change":"2024-09-21T10:28:00.294718+0000","last_active":"2024-09-21T10:31:54.348403+0000","last_peered":"2024-09-21T10:31:54.348403+0000","last_clean":"2024-09-21T10:31:54.348403+0000","last_became_active":"2024-09-21T10:28:00.294299+0000","last_became_peered":"2024-09-21T10:28:00.294299+0000","last_unstale":"2024-09-21T10:31:54.348403+0000","last_undegraded":"2024-09-21T10:31:54.348403+0000","last_fullsized":"2024-09-21T10:31:54.348403+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:41:46.881336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":352,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.859277+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:31:43.859277+0000","last_peered":"2024-09-21T10:31:43.859277+0000","last_clean":"2024-09-21T10:31:43.859277+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:31:43.859277+0000","last_undegraded":"2024-09-21T10:31:43.859277+0000","last_fullsized":"2024-09-21T10:31:43.859277+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":69,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348310+0000","last_change":"2024-09-21T10:28:00.293872+0000","last_active":"2024-09-21T10:31:54.348310+0000","last_peered":"2024-09-21T10:31:54.348310+0000","last_clean":"2024-09-21T10:31:54.348310+0000","last_became_active":"2024-09-21T10:28:00.293320+0000","last_became_peered":"2024-09-21T10:28:00.293320+0000","last_unstale":"2024-09-21T10:31:54.348310+0000","last_undegraded":"2024-09-21T10:31:54.348310+0000","last_fullsized":"2024-09-21T10:31:54.348310+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:45:43.909307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":681,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.349221+0000","last_change":"2024-09-21T10:28:00.297304+0000","last_active":"2024-09-21T10:31:54.349221+0000","last_peered":"2024-09-21T10:31:54.349221+0000","last_clean":"2024-09-21T10:31:54.349221+0000","last_became_active":"2024-09-21T10:28:00.296631+0000","last_became_peered":"2024-09-21T10:28:00.296631+0000","last_unstale":"2024-09-21T10:31:54.349221+0000","last_undegraded":"2024-09-21T10:31:54.349221+0000","last_fullsized":"2024-09-21T10:31:54.349221+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:22:16.633581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":681,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348209+0000","last_change":"2024-09-21T10:28:00.284924+0000","last_active":"2024-09-21T10:31:54.348209+0000","last_peered":"2024-09-21T10:31:54.348209+0000","last_clean":"2024-09-21T10:31:54.348209+0000","last_became_active":"2024-09-21T10:28:00.284633+0000","last_became_peered":"2024-09-21T10:28:00.284633+0000","last_unstale":"2024-09-21T10:31:54.348209+0000","last_undegraded":"2024-09-21T10:31:54.348209+0000","last_fullsized":"2024-09-21T10:31:54.348209+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:23:09.808151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":69,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.349515+0000","last_change":"2024-09-21T10:28:00.283403+0000","last_active":"2024-09-21T10:31:54.349515+0000","last_peered":"2024-09-21T10:31:54.349515+0000","last_clean":"2024-09-21T10:31:54.349515+0000","last_became_active":"2024-09-21T10:28:00.283089+0000","last_became_peered":"2024-09-21T10:28:00.283089+0000","last_unstale":"2024-09-21T10:31:54.349515+0000","last_undegraded":"2024-09-21T10:31:54.349515+0000","last_fullsized":"2024-09-21T10:31:54.349515+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:42.213644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":78,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.917373+0000","last_change":"2024-09-21T10:24:44.305124+0000","last_active":"2024-09-21T10:31:43.917373+0000","last_peered":"2024-09-21T10:31:43.917373+0000","last_clean":"2024-09-21T10:31:43.917373+0000","last_became_active":"2024-09-21T10:24:44.304468+0000","last_became_peered":"2024-09-21T10:24:44.304468+0000","last_unstale":"2024-09-21T10:31:43.917373+0000","last_undegraded":"2024-09-21T10:31:43.917373+0000","last_fullsized":"2024-09-21T10:31:43.917373+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:37.904764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":320,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.316386+0000","last_change":"2024-09-21T10:28:00.286646+0000","last_active":"2024-09-21T10:31:54.316386+0000","last_peered":"2024-09-21T10:31:54.316386+0000","last_clean":"2024-09-21T10:31:54.316386+0000","last_became_active":"2024-09-21T10:28:00.286300+0000","last_became_peered":"2024-09-21T10:28:00.286300+0000","last_unstale":"2024-09-21T10:31:54.316386+0000","last_undegraded":"2024-09-21T10:31:54.316386+0000","last_fullsized":"2024-09-21T10:31:54.316386+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:58.215385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":671,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348557+0000","last_change":"2024-09-21T10:24:44.299989+0000","last_active":"2024-09-21T10:31:54.348557+0000","last_peered":"2024-09-21T10:31:54.348557+0000","last_clean":"2024-09-21T10:31:54.348557+0000","last_became_active":"2024-09-21T10:24:44.299356+0000","last_became_peered":"2024-09-21T10:24:44.299356+0000","last_unstale":"2024-09-21T10:31:54.348557+0000","last_undegraded":"2024-09-21T10:31:54.348557+0000","last_fullsized":"2024-09-21T10:31:54.348557+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:58.867008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":88,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.917971+0000","last_change":"2024-09-21T10:28:00.281942+0000","last_active":"2024-09-21T10:31:43.917971+0000","last_peered":"2024-09-21T10:31:43.917971+0000","last_clean":"2024-09-21T10:31:43.917971+0000","last_became_active":"2024-09-21T10:28:00.281600+0000","last_became_peered":"2024-09-21T10:28:00.281600+0000","last_unstale":"2024-09-21T10:31:43.917971+0000","last_undegraded":"2024-09-21T10:31:43.917971+0000","last_fullsized":"2024-09-21T10:31:43.917971+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:21.144244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":82,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.919222+0000","last_change":"2024-09-21T10:24:44.329150+0000","last_active":"2024-09-21T10:31:43.919222+0000","last_peered":"2024-09-21T10:31:43.919222+0000","last_clean":"2024-09-21T10:31:43.919222+0000","last_became_active":"2024-09-21T10:24:44.306752+0000","last_became_peered":"2024-09-21T10:24:44.306752+0000","last_unstale":"2024-09-21T10:31:43.919222+0000","last_undegraded":"2024-09-21T10:31:43.919222+0000","last_fullsized":"2024-09-21T10:31:43.919222+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:16.592859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":88,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.917438+0000","last_change":"2024-09-21T10:28:00.286009+0000","last_active":"2024-09-21T10:31:43.917438+0000","last_peered":"2024-09-21T10:31:43.917438+0000","last_clean":"2024-09-21T10:31:43.917438+0000","last_became_active":"2024-09-21T10:28:00.285327+0000","last_became_peered":"2024-09-21T10:28:00.285327+0000","last_unstale":"2024-09-21T10:31:43.917438+0000","last_undegraded":"2024-09-21T10:31:43.917438+0000","last_fullsized":"2024-09-21T10:31:43.917438+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:34.985354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":82,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.920465+0000","last_change":"2024-09-21T10:24:44.324087+0000","last_active":"2024-09-21T10:31:43.920465+0000","last_peered":"2024-09-21T10:31:43.920465+0000","last_clean":"2024-09-21T10:31:43.920465+0000","last_became_active":"2024-09-21T10:24:44.298665+0000","last_became_peered":"2024-09-21T10:24:44.298665+0000","last_unstale":"2024-09-21T10:31:43.920465+0000","last_undegraded":"2024-09-21T10:31:43.920465+0000","last_fullsized":"2024-09-21T10:31:43.920465+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:59.819340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":69,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348474+0000","last_change":"2024-09-21T10:28:00.293054+0000","last_active":"2024-09-21T10:31:54.348474+0000","last_peered":"2024-09-21T10:31:54.348474+0000","last_clean":"2024-09-21T10:31:54.348474+0000","last_became_active":"2024-09-21T10:28:00.292723+0000","last_became_peered":"2024-09-21T10:28:00.292723+0000","last_unstale":"2024-09-21T10:31:54.348474+0000","last_undegraded":"2024-09-21T10:31:54.348474+0000","last_fullsized":"2024-09-21T10:31:54.348474+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:01:45.933570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":69,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.349606+0000","last_change":"2024-09-21T10:28:00.285515+0000","last_active":"2024-09-21T10:31:54.349606+0000","last_peered":"2024-09-21T10:31:54.349606+0000","last_clean":"2024-09-21T10:31:54.349606+0000","last_became_active":"2024-09-21T10:28:00.285203+0000","last_became_peered":"2024-09-21T10:28:00.285203+0000","last_unstale":"2024-09-21T10:31:54.349606+0000","last_undegraded":"2024-09-21T10:31:54.349606+0000","last_fullsized":"2024-09-21T10:31:54.349606+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:41.982228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":78,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T10:31:43.916165+0000","last_change":"2024-09-21T10:24:44.298820+0000","last_active":"2024-09-21T10:31:43.916165+0000","last_peered":"2024-09-21T10:31:43.916165+0000","last_clean":"2024-09-21T10:31:43.916165+0000","last_became_active":"2024-09-21T10:24:44.298478+0000","last_became_peered":"2024-09-21T10:24:44.298478+0000","last_unstale":"2024-09-21T10:31:43.916165+0000","last_undegraded":"2024-09-21T10:31:43.916165+0000","last_fullsized":"2024-09-21T10:31:43.916165+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:59.567078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":657,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-21T10:31:54.348988+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:31:54.348988+0000","last_peered":"2024-09-21T10:31:54.348988+0000","last_clean":"2024-09-21T10:31:54.348988+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:31:54.348988+0000","last_undegraded":"2024-09-21T10:31:54.348988+0000","last_fullsized":"2024-09-21T10:31:54.348988+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":10252,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5983,"num_read_kb":13006,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":316358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":69,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":644038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1100,"ondisk_log_size":1100,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453398006,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1328880,"kb_used_data":431652,"kb_used_omap":1,"kb_used_meta":897214,"kb_avail":92416272,"statfs":{"total":95995035648,"available":94634262528,"internally_reserved":0,"allocated":442011648,"data_stored":439765058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918747480},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594234,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":902312,"kb_used_data":5148,"kb_used_omap":1,"kb_used_meta":897150,"kb_avail":92842840,"statfs":{"total":95995035648,"available":95071068160,"internally_reserved":0,"allocated":5271552,"data_stored":3033027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918681944},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790464,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":909728,"kb_used_data":9812,"kb_used_omap":1,"kb_used_meta":899902,"kb_avail":92835424,"statfs":{"total":95995035648,"available":95063474176,"internally_reserved":0,"allocated":10047488,"data_stored":7813210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":921499992},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986693,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1019688,"kb_used_data":433952,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":92725464,"statfs":{"total":95995035648,"available":94950875136,"internally_reserved":0,"allocated":444366848,"data_stored":442124276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":101,"seq":433791697255,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":712348,"kb_used_data":5712,"kb_used_omap":5,"kb_used_meta":706618,"kb_avail":93032804,"statfs":{"total":95995035648,"available":95265591296,"internally_reserved":0,"allocated":5849088,"data_stored":3618664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":723577616},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860782,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1023136,"kb_used_data":9492,"kb_used_omap":8,"kb_used_meta":1013623,"kb_avail":92722016,"statfs":{"total":95995035648,"available":94947344384,"internally_reserved":0,"allocated":9719808,"data_stored":7485810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":1037950828},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024307,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1241168,"kb_used_data":424324,"kb_used_omap":7,"kb_used_meta":816824,"kb_avail":92503984,"statfs":{"total":95995035648,"available":94724079616,"internally_reserved":0,"allocated":434507776,"data_stored":432288403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":836428338},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187842,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1053328,"kb_used_data":5124,"kb_used_omap":8,"kb_used_meta":1048183,"kb_avail":92691824,"statfs":{"total":95995035648,"available":94916427776,"internally_reserved":0,"allocated":5246976,"data_stored":3020721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":1073340401},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:32:02.548 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:02 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi098-315023-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:32:02.548 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:02 smithi098 ceph-mon[97580]: osdmap e391: 8 total, 8 up, 8 in 2024-09-21T10:32:02.549 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:02 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi098-315023-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:32:02.549 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:02 smithi098 ceph-mon[100914]: osdmap e391: 8 total, 8 up, 8 in 2024-09-21T10:32:02.553 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:02.575 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:32:02.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:02.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:02.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:02 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi098-315023-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:32:02.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:02 smithi196 ceph-mon[84784]: osdmap e391: 8 total, 8 up, 8 in 2024-09-21T10:32:02.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:02.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:02.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:02.801 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:02.827 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:02.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:02.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:02.928 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:02.950 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:32:02.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:02.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:03.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:03.076 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:03.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:03.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:03.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:03.201 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:03.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:03.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:03.303 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:03.328 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:03.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:03.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:03.408 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-21T10:32:03.409 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:32:03.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:03 smithi098 ceph-mon[97580]: pgmap v1374: 115 pgs: 2 creating+peering, 30 unknown, 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:03.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:03 smithi098 ceph-mon[97580]: osdmap e392: 8 total, 8 up, 8 in 2024-09-21T10:32:03.409 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:03 smithi098 ceph-mon[100914]: pgmap v1374: 115 pgs: 2 creating+peering, 30 unknown, 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:03.409 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:03 smithi098 ceph-mon[100914]: osdmap e392: 8 total, 8 up, 8 in 2024-09-21T10:32:03.429 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:03.454 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:03.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:03.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:03.555 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:03.581 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:03.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:03.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:03.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:03 smithi196 ceph-mon[84784]: pgmap v1374: 115 pgs: 2 creating+peering, 30 unknown, 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:03 smithi196 ceph-mon[84784]: osdmap e392: 8 total, 8 up, 8 in 2024-09-21T10:32:03.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:03.708 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:03.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:03.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:03.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:03.838 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:03.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:03.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:03.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:03.964 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:32:03.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:03.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:04.066 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:04.091 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:32:04.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:04.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:04.192 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:04.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:04.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:04.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:04.344 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:32:04.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:04.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:04.397 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.mirror_image_status (2453 ms) 2024-09-21T10:32:04.397 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.mirror_image_map 2024-09-21T10:32:04.446 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:04.469 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:04.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:04.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:04.479 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.mirror_image_map (82 ms) 2024-09-21T10:32:04.479 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.mirror_instances 2024-09-21T10:32:04.486 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.mirror_instances (7 ms) 2024-09-21T10:32:04.486 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.mirror_snapshot 2024-09-21T10:32:04.528 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.mirror_snapshot (42 ms) 2024-09-21T10:32:04.529 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.group_dir_list 2024-09-21T10:32:04.532 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.group_dir_list (3 ms) 2024-09-21T10:32:04.532 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.group_dir_add 2024-09-21T10:32:04.535 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.group_dir_add (4 ms) 2024-09-21T10:32:04.535 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.dir_add_already_existing 2024-09-21T10:32:04.539 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.dir_add_already_existing (4 ms) 2024-09-21T10:32:04.539 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.group_dir_rename 2024-09-21T10:32:04.546 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.group_dir_rename (7 ms) 2024-09-21T10:32:04.546 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.group_dir_remove 2024-09-21T10:32:04.549 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.group_dir_remove (3 ms) 2024-09-21T10:32:04.549 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.group_dir_remove_missing 2024-09-21T10:32:04.553 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.group_dir_remove_missing (4 ms) 2024-09-21T10:32:04.553 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.group_image_add 2024-09-21T10:32:04.555 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.group_image_add (2 ms) 2024-09-21T10:32:04.555 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.group_image_remove 2024-09-21T10:32:04.558 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.group_image_remove (3 ms) 2024-09-21T10:32:04.558 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.group_image_list 2024-09-21T10:32:04.561 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.group_image_list (3 ms) 2024-09-21T10:32:04.561 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.group_image_clean 2024-09-21T10:32:04.565 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.group_image_clean (4 ms) 2024-09-21T10:32:04.565 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.image_group_add 2024-09-21T10:32:04.567 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.image_group_add (2 ms) 2024-09-21T10:32:04.567 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.image_group_remove 2024-09-21T10:32:04.570 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.image_group_remove (2 ms) 2024-09-21T10:32:04.570 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.image_group_get 2024-09-21T10:32:04.571 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:04.573 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.image_group_get (4 ms) 2024-09-21T10:32:04.573 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_name 2024-09-21T10:32:04.575 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.group_snap_set_empty_name (2 ms) 2024-09-21T10:32:04.575 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_id 2024-09-21T10:32:04.577 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.group_snap_set_empty_id (2 ms) 2024-09-21T10:32:04.577 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_id 2024-09-21T10:32:04.580 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_id (3 ms) 2024-09-21T10:32:04.580 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_name 2024-09-21T10:32:04.583 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_name (3 ms) 2024-09-21T10:32:04.583 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.group_snap_set 2024-09-21T10:32:04.585 INFO:tasks.workunit.client.0.smithi098.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.7-1649-gfbd1a6ab/rpm/el9/BUILD/ceph-17.2.7-1649-gfbd1a6ab/src/test/cls_rbd/test_cls_rbd.cc:2681: Failure 2024-09-21T10:32:04.585 INFO:tasks.workunit.client.0.smithi098.stdout:Expected equality of these values: 2024-09-21T10:32:04.585 INFO:tasks.workunit.client.0.smithi098.stdout: 1U 2024-09-21T10:32:04.585 INFO:tasks.workunit.client.0.smithi098.stdout: Which is: 1 2024-09-21T10:32:04.585 INFO:tasks.workunit.client.0.smithi098.stdout: keys.size() 2024-09-21T10:32:04.585 INFO:tasks.workunit.client.0.smithi098.stdout: Which is: 3 2024-09-21T10:32:04.585 INFO:tasks.workunit.client.0.smithi098.stdout:[ FAILED ] TestClsRbd.group_snap_set (2 ms) 2024-09-21T10:32:04.585 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.group_snap_list 2024-09-21T10:32:04.590 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.group_snap_list (5 ms) 2024-09-21T10:32:04.590 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.group_snap_list_max_return 2024-09-21T10:32:04.593 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:04.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:04.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:04.605 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.group_snap_list_max_return (15 ms) 2024-09-21T10:32:04.605 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.group_snap_list_max_read 2024-09-21T10:32:04.695 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:04.717 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:04.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:04.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:04.792 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.group_snap_list_max_read (187 ms) 2024-09-21T10:32:04.792 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.group_snap_remove 2024-09-21T10:32:04.795 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.group_snap_remove (3 ms) 2024-09-21T10:32:04.795 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.group_snap_get_by_id 2024-09-21T10:32:04.797 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.group_snap_get_by_id (2 ms) 2024-09-21T10:32:04.797 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.trash_methods 2024-09-21T10:32:04.804 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.trash_methods (6 ms) 2024-09-21T10:32:04.804 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.op_features 2024-09-21T10:32:04.811 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.op_features (8 ms) 2024-09-21T10:32:04.811 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.clone_parent 2024-09-21T10:32:04.819 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:04.831 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.clone_parent (20 ms) 2024-09-21T10:32:04.831 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.clone_parent_ns 2024-09-21T10:32:04.840 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:32:04.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:04.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:04.842 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.clone_parent_ns (11 ms) 2024-09-21T10:32:04.842 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.clone_child 2024-09-21T10:32:04.852 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.clone_child (10 ms) 2024-09-21T10:32:04.853 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.namespace_methods 2024-09-21T10:32:04.860 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.namespace_methods (7 ms) 2024-09-21T10:32:04.860 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.migration 2024-09-21T10:32:04.869 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.migration (10 ms) 2024-09-21T10:32:04.869 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.migration_v1 2024-09-21T10:32:04.878 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.migration_v1 (9 ms) 2024-09-21T10:32:04.878 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.assert_snapc_seq 2024-09-21T10:32:04.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:04.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:04.964 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:04.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:05.065 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:05.086 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:05.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:05.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:05.182 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:32:05.187 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:05.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:05.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:05.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:05.370 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:05.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:05.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:05.471 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:05.482 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:32:05.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:05.493 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:05.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:05.514 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:05 smithi098 ceph-mon[97580]: pgmap v1376: 115 pgs: 2 creating+peering, 14 unknown, 99 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.4 MiB/s rd, 695 KiB/s wr, 35 op/s 2024-09-21T10:32:05.514 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:05 smithi098 ceph-mon[100914]: pgmap v1376: 115 pgs: 2 creating+peering, 14 unknown, 99 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.4 MiB/s rd, 695 KiB/s wr, 35 op/s 2024-09-21T10:32:05.595 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:05.618 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:32:05.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:05.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:05.622 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:32:05.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:05 smithi196 ceph-mon[84784]: pgmap v1376: 115 pgs: 2 creating+peering, 14 unknown, 99 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.4 MiB/s rd, 695 KiB/s wr, 35 op/s 2024-09-21T10:32:05.721 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:05.742 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:05.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:05.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:05.843 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:05.866 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:05.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:05.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:05.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:05.990 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:05.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:05.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:06.092 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:06.117 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:32:06.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:06.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:06.219 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:06.244 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:32:06.245 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:06.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:06.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:06.257 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:06.260 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:06.315 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.assert_snapc_seq (1437 ms) 2024-09-21T10:32:06.315 INFO:tasks.workunit.client.0.smithi098.stdout:[ RUN ] TestClsRbd.sparsify 2024-09-21T10:32:06.346 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:06.349 INFO:tasks.workunit.client.0.smithi098.stdout:[ OK ] TestClsRbd.sparsify (34 ms) 2024-09-21T10:32:06.370 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:06.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:06.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:06.372 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:06 smithi098 ceph-mon[97580]: osdmap e393: 8 total, 8 up, 8 in 2024-09-21T10:32:06.372 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:06 smithi098 ceph-mon[100914]: osdmap e393: 8 total, 8 up, 8 in 2024-09-21T10:32:06.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:06.497 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:32:06.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:06.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:06.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:06.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:06.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:06.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:06 smithi196 ceph-mon[84784]: osdmap e393: 8 total, 8 up, 8 in 2024-09-21T10:32:06.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:06.744 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:06.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:06.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:06.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:06.867 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:06.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:06.968 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:06.994 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:32:06.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:06.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:07.096 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:07.121 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-21T10:32:07.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:07.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:07.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:07.244 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:07.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:07.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:07.316 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:32:07.317 INFO:tasks.workunit.client.0.smithi098.stdout:[----------] 70 tests from TestClsRbd (5068 ms total) 2024-09-21T10:32:07.318 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-21T10:32:07.318 INFO:tasks.workunit.client.0.smithi098.stdout:[----------] Global test environment tear-down 2024-09-21T10:32:07.318 INFO:tasks.workunit.client.0.smithi098.stdout:[==========] 70 tests from 1 test suite ran. (7371 ms total) 2024-09-21T10:32:07.318 INFO:tasks.workunit.client.0.smithi098.stdout:[ PASSED ] 69 tests. 2024-09-21T10:32:07.318 INFO:tasks.workunit.client.0.smithi098.stdout:[ FAILED ] 1 test, listed below: 2024-09-21T10:32:07.318 INFO:tasks.workunit.client.0.smithi098.stdout:[ FAILED ] TestClsRbd.group_snap_set 2024-09-21T10:32:07.318 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-09-21T10:32:07.318 INFO:tasks.workunit.client.0.smithi098.stdout: 1 FAILED TEST 2024-09-21T10:32:07.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:07.321 INFO:tasks.workunit:Stopping ['cls/test_cls_rbd.sh'] on client.0... 2024-09-21T10:32:07.321 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-21T10:32:07.346 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:07.385 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:07.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:07.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:07.487 DEBUG:teuthology.orchestra.run.smithi098:> sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:07.507 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: docker: command not found 2024-09-21T10:32:07.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:07.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi098 with status 1: 'sudo docker kill -s 1 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:07.523 INFO:tasks.thrashosds:joining thrashosds 2024-09-21T10:32:07.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:07 smithi098 ceph-mon[100914]: pgmap v1378: 115 pgs: 115 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.5 MiB/s wr, 2.54k op/s 2024-09-21T10:32:07.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:07 smithi098 ceph-mon[100914]: osdmap e394: 8 total, 8 up, 8 in 2024-09-21T10:32:07.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:07 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3914294693' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:32:07.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:07 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:32:07.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:07.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:07 smithi098 ceph-mon[97580]: pgmap v1378: 115 pgs: 115 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.5 MiB/s wr, 2.54k op/s 2024-09-21T10:32:07.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:07 smithi098 ceph-mon[97580]: osdmap e394: 8 total, 8 up, 8 in 2024-09-21T10:32:07.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:07 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3914294693' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:32:07.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:07 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:32:07.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:07.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:07 smithi196 ceph-mon[84784]: pgmap v1378: 115 pgs: 115 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.5 MiB/s wr, 2.54k op/s 2024-09-21T10:32:07.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:07 smithi196 ceph-mon[84784]: osdmap e394: 8 total, 8 up, 8 in 2024-09-21T10:32:07.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:07 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3914294693' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:32:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:07 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:32:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:08 smithi098 ceph-mon[97580]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:32:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:08 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:32:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:08 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:32:08.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:08 smithi098 ceph-mon[97580]: osdmap e395: 8 total, 8 up, 8 in 2024-09-21T10:32:08.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:08 smithi098 ceph-mon[100914]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:32:08.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:08 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:32:08.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:08 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:32:08.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:08 smithi098 ceph-mon[100914]: osdmap e395: 8 total, 8 up, 8 in 2024-09-21T10:32:08.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:08 smithi196 ceph-mon[84784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:32:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:08 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:32:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:08 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:32:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:08 smithi196 ceph-mon[84784]: osdmap e395: 8 total, 8 up, 8 in 2024-09-21T10:32:09.545 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:09 smithi098 ceph-mon[97580]: pgmap v1381: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:09.545 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:09 smithi098 ceph-mon[100914]: pgmap v1381: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:09.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:09 smithi196 ceph-mon[84784]: pgmap v1381: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:10.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:32:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:32:10.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:32:10.809 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:32:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:32:10] "GET /metrics HTTP/1.1" 200 36587 "" "Prometheus/2.51.0" 2024-09-21T10:32:11.588 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:11 smithi098 ceph-mon[97580]: pgmap v1382: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:11.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:11 smithi098 ceph-mon[100914]: pgmap v1382: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:11.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:11 smithi196 ceph-mon[84784]: pgmap v1382: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:11.650 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:32:11.651 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:32:12.722 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-21T10:32:12.723 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-21T10:32:12.723 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-21T10:32:12.723 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-21T10:32:12.723 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:32:12.723 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:32:12.723 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:32:12.723 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:32:12.723 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-21T10:32:12.724 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-21T10:32:12.724 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-21T10:32:12.724 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-21T10:32:12.724 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:32:12.724 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:32:12.724 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:32:12.724 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:32:12.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:32:13.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:13 smithi098 ceph-mon[97580]: from='client.46592 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:32:13.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:13 smithi098 ceph-mon[97580]: pgmap v1383: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:13.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:13 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3739089247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:32:13.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:13 smithi098 ceph-mon[100914]: from='client.46592 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:32:13.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:13 smithi098 ceph-mon[100914]: pgmap v1383: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:13.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:13 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3739089247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:32:13.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:13 smithi196 ceph-mon[84784]: from='client.46592 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:32:13.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:13 smithi196 ceph-mon[84784]: pgmap v1383: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:13 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3739089247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:32:14.123 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1382,"stamp":"2024-09-21T10:32:09.660655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4516211,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":6011,"num_read_kb":13075,"num_write":921,"num_write_kb":12771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":654290,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2132,"ondisk_log_size":2132,"up":249,"acting":249,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":348,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8164368,"kb_used_data":1268308,"kb_used_omap":35,"kb_used_meta":6895772,"kb_avail":741796848,"statfs":{"total":767960285184,"available":759599972352,"internally_reserved":0,"allocated":1298747392,"data_stored":1280527968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36353,"internal_metadata":7061271039},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-58,"num_objects":-11,"num_object_clones":0,"num_object_copies":-33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11,"num_whiteouts":0,"num_read":-150,"num_read_kb":-8367,"num_write":-63,"num_write_kb":-4170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001359"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":81,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T10:32:08.652139+0000","last_change":"2024-09-21T10:28:00.289895+0000","last_active":"2024-09-21T10:32:08.652139+0000","last_peered":"2024-09-21T10:32:08.652139+0000","last_clean":"2024-09-21T10:32:08.652139+0000","last_became_active":"2024-09-21T10:28:00.289622+0000","last_became_peered":"2024-09-21T10:28:00.289622+0000","last_unstale":"2024-09-21T10:32:08.652139+0000","last_undegraded":"2024-09-21T10:32:08.652139+0000","last_fullsized":"2024-09-21T10:32:08.652139+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:35:31.514813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":425,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.314152+0000","last_change":"2024-09-21T10:24:44.305198+0000","last_active":"2024-09-21T10:32:06.314152+0000","last_peered":"2024-09-21T10:32:06.314152+0000","last_clean":"2024-09-21T10:32:06.314152+0000","last_became_active":"2024-09-21T10:24:44.297653+0000","last_became_peered":"2024-09-21T10:24:44.297653+0000","last_unstale":"2024-09-21T10:32:06.314152+0000","last_undegraded":"2024-09-21T10:32:06.314152+0000","last_fullsized":"2024-09-21T10:32:06.314152+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:19.320833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":261,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.500688+0000","last_change":"2024-09-21T10:28:00.294917+0000","last_active":"2024-09-21T10:32:05.500688+0000","last_peered":"2024-09-21T10:32:05.500688+0000","last_clean":"2024-09-21T10:32:05.500688+0000","last_became_active":"2024-09-21T10:28:00.294134+0000","last_became_peered":"2024-09-21T10:28:00.294134+0000","last_unstale":"2024-09-21T10:32:05.500688+0000","last_undegraded":"2024-09-21T10:32:05.500688+0000","last_fullsized":"2024-09-21T10:32:05.500688+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:29:05.780212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":211,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.312927+0000","last_change":"2024-09-21T10:28:00.287636+0000","last_active":"2024-09-21T10:32:06.312927+0000","last_peered":"2024-09-21T10:32:06.312927+0000","last_clean":"2024-09-21T10:32:06.312927+0000","last_became_active":"2024-09-21T10:28:00.287308+0000","last_became_peered":"2024-09-21T10:28:00.287308+0000","last_unstale":"2024-09-21T10:32:06.312927+0000","last_undegraded":"2024-09-21T10:32:06.312927+0000","last_fullsized":"2024-09-21T10:32:06.312927+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:41:46.132365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":671,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-21T10:32:02.292650+0000","last_change":"2024-09-21T10:28:00.297783+0000","last_active":"2024-09-21T10:32:02.292650+0000","last_peered":"2024-09-21T10:32:02.292650+0000","last_clean":"2024-09-21T10:32:02.292650+0000","last_became_active":"2024-09-21T10:28:00.297024+0000","last_became_peered":"2024-09-21T10:28:00.297024+0000","last_unstale":"2024-09-21T10:32:02.292650+0000","last_undegraded":"2024-09-21T10:32:02.292650+0000","last_fullsized":"2024-09-21T10:32:02.292650+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:24.382523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":84,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.312482+0000","last_change":"2024-09-21T10:24:44.304399+0000","last_active":"2024-09-21T10:32:06.312482+0000","last_peered":"2024-09-21T10:32:06.312482+0000","last_clean":"2024-09-21T10:32:06.312482+0000","last_became_active":"2024-09-21T10:24:44.303437+0000","last_became_peered":"2024-09-21T10:24:44.303437+0000","last_unstale":"2024-09-21T10:32:06.312482+0000","last_undegraded":"2024-09-21T10:32:06.312482+0000","last_fullsized":"2024-09-21T10:32:06.312482+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:32:56.638481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":81,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T10:32:08.652382+0000","last_change":"2024-09-21T10:28:00.293505+0000","last_active":"2024-09-21T10:32:08.652382+0000","last_peered":"2024-09-21T10:32:08.652382+0000","last_clean":"2024-09-21T10:32:08.652382+0000","last_became_active":"2024-09-21T10:28:00.293173+0000","last_became_peered":"2024-09-21T10:28:00.293173+0000","last_unstale":"2024-09-21T10:32:08.652382+0000","last_undegraded":"2024-09-21T10:32:08.652382+0000","last_fullsized":"2024-09-21T10:32:08.652382+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:16.878847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":366,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.500311+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:32:05.500311+0000","last_peered":"2024-09-21T10:32:05.500311+0000","last_clean":"2024-09-21T10:32:05.500311+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:32:05.500311+0000","last_undegraded":"2024-09-21T10:32:05.500311+0000","last_fullsized":"2024-09-21T10:32:05.500311+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":416,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.314147+0000","last_change":"2024-09-21T10:24:44.325196+0000","last_active":"2024-09-21T10:32:06.314147+0000","last_peered":"2024-09-21T10:32:06.314147+0000","last_clean":"2024-09-21T10:32:06.314147+0000","last_became_active":"2024-09-21T10:24:44.299614+0000","last_became_peered":"2024-09-21T10:24:44.299614+0000","last_unstale":"2024-09-21T10:32:06.314147+0000","last_undegraded":"2024-09-21T10:32:06.314147+0000","last_fullsized":"2024-09-21T10:32:06.314147+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:14.558663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":147,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.500059+0000","last_change":"2024-09-21T10:26:41.305571+0000","last_active":"2024-09-21T10:32:05.500059+0000","last_peered":"2024-09-21T10:32:05.500059+0000","last_clean":"2024-09-21T10:32:05.500059+0000","last_became_active":"2024-09-21T10:26:41.305135+0000","last_became_peered":"2024-09-21T10:26:41.305135+0000","last_unstale":"2024-09-21T10:32:05.500059+0000","last_undegraded":"2024-09-21T10:32:05.500059+0000","last_fullsized":"2024-09-21T10:32:05.500059+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:49:55.452467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":417,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.313899+0000","last_change":"2024-09-21T10:24:44.323182+0000","last_active":"2024-09-21T10:32:06.313899+0000","last_peered":"2024-09-21T10:32:06.313899+0000","last_clean":"2024-09-21T10:32:06.313899+0000","last_became_active":"2024-09-21T10:24:44.299698+0000","last_became_peered":"2024-09-21T10:24:44.299698+0000","last_unstale":"2024-09-21T10:32:06.313899+0000","last_undegraded":"2024-09-21T10:32:06.313899+0000","last_fullsized":"2024-09-21T10:32:06.313899+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:56.571057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":94,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.312745+0000","last_change":"2024-09-21T10:28:00.281253+0000","last_active":"2024-09-21T10:32:06.312745+0000","last_peered":"2024-09-21T10:32:06.312745+0000","last_clean":"2024-09-21T10:32:06.312745+0000","last_became_active":"2024-09-21T10:28:00.280773+0000","last_became_peered":"2024-09-21T10:28:00.280773+0000","last_unstale":"2024-09-21T10:32:06.312745+0000","last_undegraded":"2024-09-21T10:32:06.312745+0000","last_fullsized":"2024-09-21T10:32:06.312745+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:32:48.557279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":343,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.314278+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:32:06.314278+0000","last_peered":"2024-09-21T10:32:06.314278+0000","last_clean":"2024-09-21T10:32:06.314278+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:32:06.314278+0000","last_undegraded":"2024-09-21T10:32:06.314278+0000","last_fullsized":"2024-09-21T10:32:06.314278+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":357,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.500475+0000","last_change":"2024-09-21T10:01:07.424769+0000","last_active":"2024-09-21T10:32:05.500475+0000","last_peered":"2024-09-21T10:32:05.500475+0000","last_clean":"2024-09-21T10:32:05.500475+0000","last_became_active":"2024-09-21T10:01:07.424413+0000","last_became_peered":"2024-09-21T10:01:07.424413+0000","last_unstale":"2024-09-21T10:32:05.500475+0000","last_undegraded":"2024-09-21T10:32:05.500475+0000","last_fullsized":"2024-09-21T10:32:05.500475+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:04.177453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":81,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T10:32:08.652304+0000","last_change":"2024-09-21T10:28:00.288968+0000","last_active":"2024-09-21T10:32:08.652304+0000","last_peered":"2024-09-21T10:32:08.652304+0000","last_clean":"2024-09-21T10:32:08.652304+0000","last_became_active":"2024-09-21T10:28:00.288244+0000","last_became_peered":"2024-09-21T10:28:00.288244+0000","last_unstale":"2024-09-21T10:32:08.652304+0000","last_undegraded":"2024-09-21T10:32:08.652304+0000","last_fullsized":"2024-09-21T10:32:08.652304+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:42.089701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":693,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-21T10:32:02.293161+0000","last_change":"2024-09-21T10:24:44.299417+0000","last_active":"2024-09-21T10:32:02.293161+0000","last_peered":"2024-09-21T10:32:02.293161+0000","last_clean":"2024-09-21T10:32:02.293161+0000","last_became_active":"2024-09-21T10:24:44.299116+0000","last_became_peered":"2024-09-21T10:24:44.299116+0000","last_unstale":"2024-09-21T10:32:02.293161+0000","last_undegraded":"2024-09-21T10:32:02.293161+0000","last_fullsized":"2024-09-21T10:32:02.293161+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:44.524841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"82'100","reported_seq":516,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.312606+0000","last_change":"2024-09-21T10:28:00.285919+0000","last_active":"2024-09-21T10:32:06.312606+0000","last_peered":"2024-09-21T10:32:06.312606+0000","last_clean":"2024-09-21T10:32:06.312606+0000","last_became_active":"2024-09-21T10:28:00.285324+0000","last_became_peered":"2024-09-21T10:28:00.285324+0000","last_unstale":"2024-09-21T10:32:06.312606+0000","last_undegraded":"2024-09-21T10:32:06.312606+0000","last_fullsized":"2024-09-21T10:32:06.312606+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.728195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.728195+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:00.841610+0000","scrub_duration":0.0069698140000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":85,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T10:32:08.651904+0000","last_change":"2024-09-21T10:28:00.303125+0000","last_active":"2024-09-21T10:32:08.651904+0000","last_peered":"2024-09-21T10:32:08.651904+0000","last_clean":"2024-09-21T10:32:08.651904+0000","last_became_active":"2024-09-21T10:28:00.293684+0000","last_became_peered":"2024-09-21T10:28:00.293684+0000","last_unstale":"2024-09-21T10:32:08.651904+0000","last_undegraded":"2024-09-21T10:32:08.651904+0000","last_fullsized":"2024-09-21T10:32:08.651904+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:04.495913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":94,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.313176+0000","last_change":"2024-09-21T10:28:00.283456+0000","last_active":"2024-09-21T10:32:06.313176+0000","last_peered":"2024-09-21T10:32:06.313176+0000","last_clean":"2024-09-21T10:32:06.313176+0000","last_became_active":"2024-09-21T10:28:00.282881+0000","last_became_peered":"2024-09-21T10:28:00.282881+0000","last_unstale":"2024-09-21T10:32:06.313176+0000","last_undegraded":"2024-09-21T10:32:06.313176+0000","last_fullsized":"2024-09-21T10:32:06.313176+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:43.681481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"82'100","reported_seq":202,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T10:32:07.618543+0000","last_change":"2024-09-21T10:28:00.301695+0000","last_active":"2024-09-21T10:32:07.618543+0000","last_peered":"2024-09-21T10:32:07.618543+0000","last_clean":"2024-09-21T10:32:07.618543+0000","last_became_active":"2024-09-21T10:28:00.289856+0000","last_became_peered":"2024-09-21T10:28:00.289856+0000","last_unstale":"2024-09-21T10:32:07.618543+0000","last_undegraded":"2024-09-21T10:32:07.618543+0000","last_fullsized":"2024-09-21T10:32:07.618543+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:27.234001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:27.234001+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:02.389036+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":408,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.500570+0000","last_change":"2024-09-21T10:24:44.309470+0000","last_active":"2024-09-21T10:32:05.500570+0000","last_peered":"2024-09-21T10:32:05.500570+0000","last_clean":"2024-09-21T10:32:05.500570+0000","last_became_active":"2024-09-21T10:24:44.308836+0000","last_became_peered":"2024-09-21T10:24:44.308836+0000","last_unstale":"2024-09-21T10:32:05.500570+0000","last_undegraded":"2024-09-21T10:32:05.500570+0000","last_fullsized":"2024-09-21T10:32:05.500570+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:09:44.622418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":84,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.313202+0000","last_change":"2024-09-21T10:24:44.305135+0000","last_active":"2024-09-21T10:32:06.313202+0000","last_peered":"2024-09-21T10:32:06.313202+0000","last_clean":"2024-09-21T10:32:06.313202+0000","last_became_active":"2024-09-21T10:24:44.304839+0000","last_became_peered":"2024-09-21T10:24:44.304839+0000","last_unstale":"2024-09-21T10:32:06.313202+0000","last_undegraded":"2024-09-21T10:32:06.313202+0000","last_fullsized":"2024-09-21T10:32:06.313202+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:02.264404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":124,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-21T10:32:02.292276+0000","last_change":"2024-09-21T10:25:39.872680+0000","last_active":"2024-09-21T10:32:02.292276+0000","last_peered":"2024-09-21T10:32:02.292276+0000","last_clean":"2024-09-21T10:32:02.292276+0000","last_became_active":"2024-09-21T10:25:39.872312+0000","last_became_peered":"2024-09-21T10:25:39.872312+0000","last_unstale":"2024-09-21T10:32:02.292276+0000","last_undegraded":"2024-09-21T10:32:02.292276+0000","last_fullsized":"2024-09-21T10:32:02.292276+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:54.942046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":84,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.313697+0000","last_change":"2024-09-21T10:24:44.310484+0000","last_active":"2024-09-21T10:32:06.313697+0000","last_peered":"2024-09-21T10:32:06.313697+0000","last_clean":"2024-09-21T10:32:06.313697+0000","last_became_active":"2024-09-21T10:24:44.307209+0000","last_became_peered":"2024-09-21T10:24:44.307209+0000","last_unstale":"2024-09-21T10:32:06.313697+0000","last_undegraded":"2024-09-21T10:32:06.313697+0000","last_fullsized":"2024-09-21T10:32:06.313697+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:48:36.767539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'100","reported_seq":186,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T10:32:07.618183+0000","last_change":"2024-09-21T10:23:26.197358+0000","last_active":"2024-09-21T10:32:07.618183+0000","last_peered":"2024-09-21T10:32:07.618183+0000","last_clean":"2024-09-21T10:32:07.618183+0000","last_became_active":"2024-09-21T10:23:20.695174+0000","last_became_peered":"2024-09-21T10:23:20.695174+0000","last_unstale":"2024-09-21T10:32:07.618183+0000","last_undegraded":"2024-09-21T10:32:07.618183+0000","last_fullsized":"2024-09-21T10:32:07.618183+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:26.197291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:26.197291+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:33.961031+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":81,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T10:32:08.651563+0000","last_change":"2024-09-21T10:28:00.287560+0000","last_active":"2024-09-21T10:32:08.651563+0000","last_peered":"2024-09-21T10:32:08.651563+0000","last_clean":"2024-09-21T10:32:08.651563+0000","last_became_active":"2024-09-21T10:28:00.285076+0000","last_became_peered":"2024-09-21T10:28:00.285076+0000","last_unstale":"2024-09-21T10:32:08.651563+0000","last_undegraded":"2024-09-21T10:32:08.651563+0000","last_fullsized":"2024-09-21T10:32:08.651563+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:49.950523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":690,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.299468+0000","last_change":"2024-09-21T10:28:00.282062+0000","last_active":"2024-09-21T10:32:05.299468+0000","last_peered":"2024-09-21T10:32:05.299468+0000","last_clean":"2024-09-21T10:32:05.299468+0000","last_became_active":"2024-09-21T10:28:00.281593+0000","last_became_peered":"2024-09-21T10:28:00.281593+0000","last_unstale":"2024-09-21T10:32:05.299468+0000","last_undegraded":"2024-09-21T10:32:05.299468+0000","last_fullsized":"2024-09-21T10:32:05.299468+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:45.704840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":497,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.313192+0000","last_change":"2024-09-21T10:23:18.790100+0000","last_active":"2024-09-21T10:32:06.313192+0000","last_peered":"2024-09-21T10:32:06.313192+0000","last_clean":"2024-09-21T10:32:06.313192+0000","last_became_active":"2024-09-21T10:23:14.677610+0000","last_became_peered":"2024-09-21T10:23:14.677610+0000","last_unstale":"2024-09-21T10:32:06.313192+0000","last_undegraded":"2024-09-21T10:32:06.313192+0000","last_fullsized":"2024-09-21T10:32:06.313192+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:18.790017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:18.790017+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:56.562241+0000","scrub_duration":0.0080400790000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":589,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.500101+0000","last_change":"2024-09-21T10:26:41.279374+0000","last_active":"2024-09-21T10:32:05.500101+0000","last_peered":"2024-09-21T10:32:05.500101+0000","last_clean":"2024-09-21T10:32:05.500101+0000","last_became_active":"2024-09-21T10:26:41.278994+0000","last_became_peered":"2024-09-21T10:26:41.278994+0000","last_unstale":"2024-09-21T10:32:05.500101+0000","last_undegraded":"2024-09-21T10:32:05.500101+0000","last_fullsized":"2024-09-21T10:32:05.500101+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:49:57.821175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":482,"num_read_kb":1581,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":81,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T10:32:08.651627+0000","last_change":"2024-09-21T10:28:00.295436+0000","last_active":"2024-09-21T10:32:08.651627+0000","last_peered":"2024-09-21T10:32:08.651627+0000","last_clean":"2024-09-21T10:32:08.651627+0000","last_became_active":"2024-09-21T10:28:00.295135+0000","last_became_peered":"2024-09-21T10:28:00.295135+0000","last_unstale":"2024-09-21T10:32:08.651627+0000","last_undegraded":"2024-09-21T10:32:08.651627+0000","last_fullsized":"2024-09-21T10:32:08.651627+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:14:03.027395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":380,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.501041+0000","last_change":"2024-09-21T10:24:44.308008+0000","last_active":"2024-09-21T10:32:05.501041+0000","last_peered":"2024-09-21T10:32:05.501041+0000","last_clean":"2024-09-21T10:32:05.501041+0000","last_became_active":"2024-09-21T10:24:44.307595+0000","last_became_peered":"2024-09-21T10:24:44.307595+0000","last_unstale":"2024-09-21T10:32:05.501041+0000","last_undegraded":"2024-09-21T10:32:05.501041+0000","last_fullsized":"2024-09-21T10:32:05.501041+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:33:49.448676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"82'100","reported_seq":189,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.298913+0000","last_change":"2024-09-21T10:24:44.309154+0000","last_active":"2024-09-21T10:32:05.298913+0000","last_peered":"2024-09-21T10:32:05.298913+0000","last_clean":"2024-09-21T10:32:05.298913+0000","last_became_active":"2024-09-21T10:24:44.308330+0000","last_became_peered":"2024-09-21T10:24:44.308330+0000","last_unstale":"2024-09-21T10:32:05.298913+0000","last_undegraded":"2024-09-21T10:32:05.298913+0000","last_fullsized":"2024-09-21T10:32:05.298913+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.641718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.641718+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:55:10.158739+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":739,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.312565+0000","last_change":"2024-09-21T10:01:07.426934+0000","last_active":"2024-09-21T10:32:06.312565+0000","last_peered":"2024-09-21T10:32:06.312565+0000","last_clean":"2024-09-21T10:32:06.312565+0000","last_became_active":"2024-09-21T10:01:07.426331+0000","last_became_peered":"2024-09-21T10:01:07.426331+0000","last_unstale":"2024-09-21T10:32:06.312565+0000","last_undegraded":"2024-09-21T10:32:06.312565+0000","last_fullsized":"2024-09-21T10:32:06.312565+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:19.070023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":701,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-21T10:32:02.292882+0000","last_change":"2024-09-21T10:28:00.296831+0000","last_active":"2024-09-21T10:32:02.292882+0000","last_peered":"2024-09-21T10:32:02.292882+0000","last_clean":"2024-09-21T10:32:02.292882+0000","last_became_active":"2024-09-21T10:28:00.296527+0000","last_became_peered":"2024-09-21T10:28:00.296527+0000","last_unstale":"2024-09-21T10:32:02.292882+0000","last_undegraded":"2024-09-21T10:32:02.292882+0000","last_fullsized":"2024-09-21T10:32:02.292882+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:18.024868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":669,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-21T10:32:02.293044+0000","last_change":"2024-09-21T10:01:41.360287+0000","last_active":"2024-09-21T10:32:02.293044+0000","last_peered":"2024-09-21T10:32:02.293044+0000","last_clean":"2024-09-21T10:32:02.293044+0000","last_became_active":"2024-09-21T10:01:41.359987+0000","last_became_peered":"2024-09-21T10:01:41.359987+0000","last_unstale":"2024-09-21T10:32:02.293044+0000","last_undegraded":"2024-09-21T10:32:02.293044+0000","last_fullsized":"2024-09-21T10:32:02.293044+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:18.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"82'100","reported_seq":333,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-21T10:32:02.292229+0000","last_change":"2024-09-21T10:24:44.294460+0000","last_active":"2024-09-21T10:32:02.292229+0000","last_peered":"2024-09-21T10:32:02.292229+0000","last_clean":"2024-09-21T10:32:02.292229+0000","last_became_active":"2024-09-21T10:24:44.293972+0000","last_became_peered":"2024-09-21T10:24:44.293972+0000","last_unstale":"2024-09-21T10:32:02.292229+0000","last_undegraded":"2024-09-21T10:32:02.292229+0000","last_fullsized":"2024-09-21T10:32:02.292229+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:26.199522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:26.199522+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:26.135171+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":1485,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T10:32:07.618764+0000","last_change":"2024-09-21T10:28:03.346064+0000","last_active":"2024-09-21T10:32:07.618764+0000","last_peered":"2024-09-21T10:32:07.618764+0000","last_clean":"2024-09-21T10:32:07.618764+0000","last_became_active":"2024-09-21T10:28:03.345313+0000","last_became_peered":"2024-09-21T10:28:03.345313+0000","last_unstale":"2024-09-21T10:32:07.618764+0000","last_undegraded":"2024-09-21T10:32:07.618764+0000","last_fullsized":"2024-09-21T10:32:07.618764+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:32.097521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":543,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.298788+0000","last_change":"2024-09-21T10:08:18.470748+0000","last_active":"2024-09-21T10:32:05.298788+0000","last_peered":"2024-09-21T10:32:05.298788+0000","last_clean":"2024-09-21T10:32:05.298788+0000","last_became_active":"2024-09-21T10:08:18.470227+0000","last_became_peered":"2024-09-21T10:08:18.470227+0000","last_unstale":"2024-09-21T10:32:05.298788+0000","last_undegraded":"2024-09-21T10:32:05.298788+0000","last_fullsized":"2024-09-21T10:32:05.298788+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:16.749175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":670,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T10:32:07.619521+0000","last_change":"2024-09-21T10:01:41.360104+0000","last_active":"2024-09-21T10:32:07.619521+0000","last_peered":"2024-09-21T10:32:07.619521+0000","last_clean":"2024-09-21T10:32:07.619521+0000","last_became_active":"2024-09-21T10:01:41.359502+0000","last_became_peered":"2024-09-21T10:01:41.359502+0000","last_unstale":"2024-09-21T10:32:07.619521+0000","last_undegraded":"2024-09-21T10:32:07.619521+0000","last_fullsized":"2024-09-21T10:32:07.619521+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:31.947424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"82'100","reported_seq":326,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-21T10:32:02.292785+0000","last_change":"2024-09-21T10:23:21.277739+0000","last_active":"2024-09-21T10:32:02.292785+0000","last_peered":"2024-09-21T10:32:02.292785+0000","last_clean":"2024-09-21T10:32:02.292785+0000","last_became_active":"2024-09-21T10:23:20.689733+0000","last_became_peered":"2024-09-21T10:23:20.689733+0000","last_unstale":"2024-09-21T10:32:02.292785+0000","last_undegraded":"2024-09-21T10:32:02.292785+0000","last_fullsized":"2024-09-21T10:32:02.292785+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.277663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.277663+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:01:54.523189+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":1142,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.298687+0000","last_change":"2024-09-21T10:07:43.794767+0000","last_active":"2024-09-21T10:32:05.298687+0000","last_peered":"2024-09-21T10:32:05.298687+0000","last_clean":"2024-09-21T10:32:05.298687+0000","last_became_active":"2024-09-21T10:07:43.794062+0000","last_became_peered":"2024-09-21T10:07:43.794062+0000","last_unstale":"2024-09-21T10:32:05.298687+0000","last_undegraded":"2024-09-21T10:32:05.298687+0000","last_fullsized":"2024-09-21T10:32:05.298687+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:51.977955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":84,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.313686+0000","last_change":"2024-09-21T10:24:44.299492+0000","last_active":"2024-09-21T10:32:06.313686+0000","last_peered":"2024-09-21T10:32:06.313686+0000","last_clean":"2024-09-21T10:32:06.313686+0000","last_became_active":"2024-09-21T10:24:44.299187+0000","last_became_peered":"2024-09-21T10:24:44.299187+0000","last_unstale":"2024-09-21T10:32:06.313686+0000","last_undegraded":"2024-09-21T10:32:06.313686+0000","last_fullsized":"2024-09-21T10:32:06.313686+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:58.500386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":600,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T10:32:07.619155+0000","last_change":"2024-09-21T10:28:00.289453+0000","last_active":"2024-09-21T10:32:07.619155+0000","last_peered":"2024-09-21T10:32:07.619155+0000","last_clean":"2024-09-21T10:32:07.619155+0000","last_became_active":"2024-09-21T10:28:00.289030+0000","last_became_peered":"2024-09-21T10:28:00.289030+0000","last_unstale":"2024-09-21T10:32:07.619155+0000","last_undegraded":"2024-09-21T10:32:07.619155+0000","last_fullsized":"2024-09-21T10:32:07.619155+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:21.959998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"82'100","reported_seq":258,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.501045+0000","last_change":"2024-09-21T10:28:00.294997+0000","last_active":"2024-09-21T10:32:05.501045+0000","last_peered":"2024-09-21T10:32:05.501045+0000","last_clean":"2024-09-21T10:32:05.501045+0000","last_became_active":"2024-09-21T10:28:00.294595+0000","last_became_peered":"2024-09-21T10:28:00.294595+0000","last_unstale":"2024-09-21T10:32:05.501045+0000","last_undegraded":"2024-09-21T10:32:05.501045+0000","last_fullsized":"2024-09-21T10:32:05.501045+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:30.569982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:30.569982+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:01.422939+0000","scrub_duration":0.0051309349999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":192,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.500823+0000","last_change":"2024-09-21T10:28:00.294966+0000","last_active":"2024-09-21T10:32:05.500823+0000","last_peered":"2024-09-21T10:32:05.500823+0000","last_clean":"2024-09-21T10:32:05.500823+0000","last_became_active":"2024-09-21T10:28:00.294552+0000","last_became_peered":"2024-09-21T10:28:00.294552+0000","last_unstale":"2024-09-21T10:32:05.500823+0000","last_undegraded":"2024-09-21T10:32:05.500823+0000","last_fullsized":"2024-09-21T10:32:05.500823+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:47.292125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":1249,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.299185+0000","last_change":"2024-09-21T10:08:18.468995+0000","last_active":"2024-09-21T10:32:05.299185+0000","last_peered":"2024-09-21T10:32:05.299185+0000","last_clean":"2024-09-21T10:32:05.299185+0000","last_became_active":"2024-09-21T10:08:18.468243+0000","last_became_peered":"2024-09-21T10:08:18.468243+0000","last_unstale":"2024-09-21T10:32:05.299185+0000","last_undegraded":"2024-09-21T10:32:05.299185+0000","last_fullsized":"2024-09-21T10:32:05.299185+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:45.520968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":575,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.298872+0000","last_change":"2024-09-21T10:24:44.309614+0000","last_active":"2024-09-21T10:32:05.298872+0000","last_peered":"2024-09-21T10:32:05.298872+0000","last_clean":"2024-09-21T10:32:05.298872+0000","last_became_active":"2024-09-21T10:24:44.309187+0000","last_became_peered":"2024-09-21T10:24:44.309187+0000","last_unstale":"2024-09-21T10:32:05.298872+0000","last_undegraded":"2024-09-21T10:32:05.298872+0000","last_fullsized":"2024-09-21T10:32:05.298872+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:01.074803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'100","reported_seq":268,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.501093+0000","last_change":"2024-09-21T10:28:00.295277+0000","last_active":"2024-09-21T10:32:05.501093+0000","last_peered":"2024-09-21T10:32:05.501093+0000","last_clean":"2024-09-21T10:32:05.501093+0000","last_became_active":"2024-09-21T10:28:00.294016+0000","last_became_peered":"2024-09-21T10:28:00.294016+0000","last_unstale":"2024-09-21T10:32:05.501093+0000","last_undegraded":"2024-09-21T10:32:05.501093+0000","last_fullsized":"2024-09-21T10:32:05.501093+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:24.536711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:24.536711+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:28:49.984470+0000","scrub_duration":0.0047525650000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":544,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.299085+0000","last_change":"2024-09-21T10:08:18.469907+0000","last_active":"2024-09-21T10:32:05.299085+0000","last_peered":"2024-09-21T10:32:05.299085+0000","last_clean":"2024-09-21T10:32:05.299085+0000","last_became_active":"2024-09-21T10:08:18.469443+0000","last_became_peered":"2024-09-21T10:08:18.469443+0000","last_unstale":"2024-09-21T10:32:05.299085+0000","last_undegraded":"2024-09-21T10:32:05.299085+0000","last_fullsized":"2024-09-21T10:32:05.299085+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:15.645803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":817,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.314364+0000","last_change":"2024-09-21T10:24:44.334847+0000","last_active":"2024-09-21T10:32:06.314364+0000","last_peered":"2024-09-21T10:32:06.314364+0000","last_clean":"2024-09-21T10:32:06.314364+0000","last_became_active":"2024-09-21T10:24:44.301321+0000","last_became_peered":"2024-09-21T10:24:44.301321+0000","last_unstale":"2024-09-21T10:32:06.314364+0000","last_undegraded":"2024-09-21T10:32:06.314364+0000","last_fullsized":"2024-09-21T10:32:06.314364+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:37.628940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":68,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.500002+0000","last_change":"2024-09-21T10:26:41.278937+0000","last_active":"2024-09-21T10:32:05.500002+0000","last_peered":"2024-09-21T10:32:05.500002+0000","last_clean":"2024-09-21T10:32:05.500002+0000","last_became_active":"2024-09-21T10:26:41.277726+0000","last_became_peered":"2024-09-21T10:26:41.277726+0000","last_unstale":"2024-09-21T10:32:05.500002+0000","last_undegraded":"2024-09-21T10:32:05.500002+0000","last_fullsized":"2024-09-21T10:32:05.500002+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:04.741057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"82'100","reported_seq":84,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.313110+0000","last_change":"2024-09-21T10:24:44.296758+0000","last_active":"2024-09-21T10:32:06.313110+0000","last_peered":"2024-09-21T10:32:06.313110+0000","last_clean":"2024-09-21T10:32:06.313110+0000","last_became_active":"2024-09-21T10:24:44.296403+0000","last_became_peered":"2024-09-21T10:24:44.296403+0000","last_unstale":"2024-09-21T10:32:06.313110+0000","last_undegraded":"2024-09-21T10:32:06.313110+0000","last_fullsized":"2024-09-21T10:32:06.313110+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.105746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.105746+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:20:12.502081+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":85,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T10:32:08.652510+0000","last_change":"2024-09-21T10:28:00.300952+0000","last_active":"2024-09-21T10:32:08.652510+0000","last_peered":"2024-09-21T10:32:08.652510+0000","last_clean":"2024-09-21T10:32:08.652510+0000","last_became_active":"2024-09-21T10:28:00.293337+0000","last_became_peered":"2024-09-21T10:28:00.293337+0000","last_unstale":"2024-09-21T10:32:08.652510+0000","last_undegraded":"2024-09-21T10:32:08.652510+0000","last_fullsized":"2024-09-21T10:32:08.652510+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:58.409761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":969,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.299284+0000","last_change":"2024-09-21T10:24:46.844278+0000","last_active":"2024-09-21T10:32:05.299284+0000","last_peered":"2024-09-21T10:32:05.299284+0000","last_clean":"2024-09-21T10:32:05.299284+0000","last_became_active":"2024-09-21T10:24:46.840961+0000","last_became_peered":"2024-09-21T10:24:46.840961+0000","last_unstale":"2024-09-21T10:32:05.299284+0000","last_undegraded":"2024-09-21T10:32:05.299284+0000","last_fullsized":"2024-09-21T10:32:05.299284+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:49:48.823524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":82,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T10:32:08.651742+0000","last_change":"2024-09-21T10:28:00.293864+0000","last_active":"2024-09-21T10:32:08.651742+0000","last_peered":"2024-09-21T10:32:08.651742+0000","last_clean":"2024-09-21T10:32:08.651742+0000","last_became_active":"2024-09-21T10:28:00.293494+0000","last_became_peered":"2024-09-21T10:28:00.293494+0000","last_unstale":"2024-09-21T10:32:08.651742+0000","last_undegraded":"2024-09-21T10:32:08.651742+0000","last_fullsized":"2024-09-21T10:32:08.651742+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:37.098068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"82'100","reported_seq":88,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T10:32:08.651988+0000","last_change":"2024-09-21T10:28:00.305255+0000","last_active":"2024-09-21T10:32:08.651988+0000","last_peered":"2024-09-21T10:32:08.651988+0000","last_clean":"2024-09-21T10:32:08.651988+0000","last_became_active":"2024-09-21T10:28:00.290981+0000","last_became_peered":"2024-09-21T10:28:00.290981+0000","last_unstale":"2024-09-21T10:32:08.651988+0000","last_undegraded":"2024-09-21T10:32:08.651988+0000","last_fullsized":"2024-09-21T10:32:08.651988+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.759355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.759355+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:32:44.600746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":69,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":706,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-21T10:32:02.292380+0000","last_change":"2024-09-21T10:24:44.307344+0000","last_active":"2024-09-21T10:32:02.292380+0000","last_peered":"2024-09-21T10:32:02.292380+0000","last_clean":"2024-09-21T10:32:02.292380+0000","last_became_active":"2024-09-21T10:24:44.298936+0000","last_became_peered":"2024-09-21T10:24:44.298936+0000","last_unstale":"2024-09-21T10:32:02.292380+0000","last_undegraded":"2024-09-21T10:32:02.292380+0000","last_fullsized":"2024-09-21T10:32:02.292380+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:58:26.025877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":620,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.499904+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:32:05.499904+0000","last_peered":"2024-09-21T10:32:05.499904+0000","last_clean":"2024-09-21T10:32:05.499904+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:32:05.499904+0000","last_undegraded":"2024-09-21T10:32:05.499904+0000","last_fullsized":"2024-09-21T10:32:05.499904+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":84,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.312601+0000","last_change":"2024-09-21T10:24:44.303781+0000","last_active":"2024-09-21T10:32:06.312601+0000","last_peered":"2024-09-21T10:32:06.312601+0000","last_clean":"2024-09-21T10:32:06.312601+0000","last_became_active":"2024-09-21T10:24:44.302058+0000","last_became_peered":"2024-09-21T10:24:44.302058+0000","last_unstale":"2024-09-21T10:32:06.312601+0000","last_undegraded":"2024-09-21T10:32:06.312601+0000","last_fullsized":"2024-09-21T10:32:06.312601+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:30.371229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":84,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.313769+0000","last_change":"2024-09-21T10:24:44.303417+0000","last_active":"2024-09-21T10:32:06.313769+0000","last_peered":"2024-09-21T10:32:06.313769+0000","last_clean":"2024-09-21T10:32:06.313769+0000","last_became_active":"2024-09-21T10:24:44.303069+0000","last_became_peered":"2024-09-21T10:24:44.303069+0000","last_unstale":"2024-09-21T10:32:06.313769+0000","last_undegraded":"2024-09-21T10:32:06.313769+0000","last_fullsized":"2024-09-21T10:32:06.313769+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:30:30.966228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":94,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.313087+0000","last_change":"2024-09-21T10:24:44.321421+0000","last_active":"2024-09-21T10:32:06.313087+0000","last_peered":"2024-09-21T10:32:06.313087+0000","last_clean":"2024-09-21T10:32:06.313087+0000","last_became_active":"2024-09-21T10:24:44.297082+0000","last_became_peered":"2024-09-21T10:24:44.297082+0000","last_unstale":"2024-09-21T10:32:06.313087+0000","last_undegraded":"2024-09-21T10:32:06.313087+0000","last_fullsized":"2024-09-21T10:32:06.313087+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:35:37.704881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":543,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.298891+0000","last_change":"2024-09-21T10:08:18.470378+0000","last_active":"2024-09-21T10:32:05.298891+0000","last_peered":"2024-09-21T10:32:05.298891+0000","last_clean":"2024-09-21T10:32:05.298891+0000","last_became_active":"2024-09-21T10:08:18.469482+0000","last_became_peered":"2024-09-21T10:08:18.469482+0000","last_unstale":"2024-09-21T10:32:05.298891+0000","last_undegraded":"2024-09-21T10:32:05.298891+0000","last_fullsized":"2024-09-21T10:32:05.298891+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:35.628724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":458,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.314001+0000","last_change":"2024-09-21T10:28:00.296771+0000","last_active":"2024-09-21T10:32:06.314001+0000","last_peered":"2024-09-21T10:32:06.314001+0000","last_clean":"2024-09-21T10:32:06.314001+0000","last_became_active":"2024-09-21T10:28:00.285336+0000","last_became_peered":"2024-09-21T10:28:00.285336+0000","last_unstale":"2024-09-21T10:32:06.314001+0000","last_undegraded":"2024-09-21T10:32:06.314001+0000","last_fullsized":"2024-09-21T10:32:06.314001+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:06:50.848492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":165,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.500722+0000","last_change":"2024-09-21T10:26:41.644952+0000","last_active":"2024-09-21T10:32:05.500722+0000","last_peered":"2024-09-21T10:32:05.500722+0000","last_clean":"2024-09-21T10:32:05.500722+0000","last_became_active":"2024-09-21T10:26:41.644592+0000","last_became_peered":"2024-09-21T10:26:41.644592+0000","last_unstale":"2024-09-21T10:32:05.500722+0000","last_undegraded":"2024-09-21T10:32:05.500722+0000","last_fullsized":"2024-09-21T10:32:05.500722+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:44:44.183877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":81,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T10:32:08.651604+0000","last_change":"2024-09-21T10:28:00.294338+0000","last_active":"2024-09-21T10:32:08.651604+0000","last_peered":"2024-09-21T10:32:08.651604+0000","last_clean":"2024-09-21T10:32:08.651604+0000","last_became_active":"2024-09-21T10:28:00.294044+0000","last_became_peered":"2024-09-21T10:28:00.294044+0000","last_unstale":"2024-09-21T10:32:08.651604+0000","last_undegraded":"2024-09-21T10:32:08.651604+0000","last_fullsized":"2024-09-21T10:32:08.651604+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:43:30.999988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":671,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-21T10:32:02.293176+0000","last_change":"2024-09-21T10:28:00.296935+0000","last_active":"2024-09-21T10:32:02.293176+0000","last_peered":"2024-09-21T10:32:02.293176+0000","last_clean":"2024-09-21T10:32:02.293176+0000","last_became_active":"2024-09-21T10:28:00.296562+0000","last_became_peered":"2024-09-21T10:28:00.296562+0000","last_unstale":"2024-09-21T10:32:02.293176+0000","last_undegraded":"2024-09-21T10:32:02.293176+0000","last_fullsized":"2024-09-21T10:32:02.293176+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:06.863310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":81,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T10:32:08.653493+0000","last_change":"2024-09-21T10:28:00.294718+0000","last_active":"2024-09-21T10:32:08.653493+0000","last_peered":"2024-09-21T10:32:08.653493+0000","last_clean":"2024-09-21T10:32:08.653493+0000","last_became_active":"2024-09-21T10:28:00.294299+0000","last_became_peered":"2024-09-21T10:28:00.294299+0000","last_unstale":"2024-09-21T10:32:08.653493+0000","last_undegraded":"2024-09-21T10:32:08.653493+0000","last_fullsized":"2024-09-21T10:32:08.653493+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:41:46.881336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":357,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.501039+0000","last_change":"2024-09-21T10:01:07.424312+0000","last_active":"2024-09-21T10:32:05.501039+0000","last_peered":"2024-09-21T10:32:05.501039+0000","last_clean":"2024-09-21T10:32:05.501039+0000","last_became_active":"2024-09-21T10:01:07.424008+0000","last_became_peered":"2024-09-21T10:01:07.424008+0000","last_unstale":"2024-09-21T10:32:05.501039+0000","last_undegraded":"2024-09-21T10:32:05.501039+0000","last_fullsized":"2024-09-21T10:32:05.501039+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:30.562442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":81,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T10:32:08.651570+0000","last_change":"2024-09-21T10:28:00.293872+0000","last_active":"2024-09-21T10:32:08.651570+0000","last_peered":"2024-09-21T10:32:08.651570+0000","last_clean":"2024-09-21T10:32:08.651570+0000","last_became_active":"2024-09-21T10:28:00.293320+0000","last_became_peered":"2024-09-21T10:28:00.293320+0000","last_unstale":"2024-09-21T10:32:08.651570+0000","last_undegraded":"2024-09-21T10:32:08.651570+0000","last_fullsized":"2024-09-21T10:32:08.651570+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:45:43.909307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":688,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-21T10:32:02.292959+0000","last_change":"2024-09-21T10:28:00.297304+0000","last_active":"2024-09-21T10:32:02.292959+0000","last_peered":"2024-09-21T10:32:02.292959+0000","last_clean":"2024-09-21T10:32:02.292959+0000","last_became_active":"2024-09-21T10:28:00.296631+0000","last_became_peered":"2024-09-21T10:28:00.296631+0000","last_unstale":"2024-09-21T10:32:02.292959+0000","last_undegraded":"2024-09-21T10:32:02.292959+0000","last_fullsized":"2024-09-21T10:32:02.292959+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:22:16.633581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":689,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:32:05.298495+0000","last_change":"2024-09-21T10:28:00.284924+0000","last_active":"2024-09-21T10:32:05.298495+0000","last_peered":"2024-09-21T10:32:05.298495+0000","last_clean":"2024-09-21T10:32:05.298495+0000","last_became_active":"2024-09-21T10:28:00.284633+0000","last_became_peered":"2024-09-21T10:28:00.284633+0000","last_unstale":"2024-09-21T10:32:05.298495+0000","last_undegraded":"2024-09-21T10:32:05.298495+0000","last_fullsized":"2024-09-21T10:32:05.298495+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:23:09.808151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":81,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T10:32:08.652024+0000","last_change":"2024-09-21T10:28:00.283403+0000","last_active":"2024-09-21T10:32:08.652024+0000","last_peered":"2024-09-21T10:32:08.652024+0000","last_clean":"2024-09-21T10:32:08.652024+0000","last_became_active":"2024-09-21T10:28:00.283089+0000","last_became_peered":"2024-09-21T10:28:00.283089+0000","last_unstale":"2024-09-21T10:32:08.652024+0000","last_undegraded":"2024-09-21T10:32:08.652024+0000","last_fullsized":"2024-09-21T10:32:08.652024+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:42.213644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":84,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.312985+0000","last_change":"2024-09-21T10:24:44.305124+0000","last_active":"2024-09-21T10:32:06.312985+0000","last_peered":"2024-09-21T10:32:06.312985+0000","last_clean":"2024-09-21T10:32:06.312985+0000","last_became_active":"2024-09-21T10:24:44.304468+0000","last_became_peered":"2024-09-21T10:24:44.304468+0000","last_unstale":"2024-09-21T10:32:06.312985+0000","last_undegraded":"2024-09-21T10:32:06.312985+0000","last_fullsized":"2024-09-21T10:32:06.312985+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:37.904764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":332,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T10:32:07.618768+0000","last_change":"2024-09-21T10:28:00.286646+0000","last_active":"2024-09-21T10:32:07.618768+0000","last_peered":"2024-09-21T10:32:07.618768+0000","last_clean":"2024-09-21T10:32:07.618768+0000","last_became_active":"2024-09-21T10:28:00.286300+0000","last_became_peered":"2024-09-21T10:28:00.286300+0000","last_unstale":"2024-09-21T10:32:07.618768+0000","last_undegraded":"2024-09-21T10:32:07.618768+0000","last_fullsized":"2024-09-21T10:32:07.618768+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:58.215385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":677,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-21T10:32:02.293015+0000","last_change":"2024-09-21T10:24:44.299989+0000","last_active":"2024-09-21T10:32:02.293015+0000","last_peered":"2024-09-21T10:32:02.293015+0000","last_clean":"2024-09-21T10:32:02.293015+0000","last_became_active":"2024-09-21T10:24:44.299356+0000","last_became_peered":"2024-09-21T10:24:44.299356+0000","last_unstale":"2024-09-21T10:32:02.293015+0000","last_undegraded":"2024-09-21T10:32:02.293015+0000","last_fullsized":"2024-09-21T10:32:02.293015+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:58.867008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":94,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.313507+0000","last_change":"2024-09-21T10:28:00.281942+0000","last_active":"2024-09-21T10:32:06.313507+0000","last_peered":"2024-09-21T10:32:06.313507+0000","last_clean":"2024-09-21T10:32:06.313507+0000","last_became_active":"2024-09-21T10:28:00.281600+0000","last_became_peered":"2024-09-21T10:28:00.281600+0000","last_unstale":"2024-09-21T10:32:06.313507+0000","last_undegraded":"2024-09-21T10:32:06.313507+0000","last_fullsized":"2024-09-21T10:32:06.313507+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:21.144244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":88,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.313319+0000","last_change":"2024-09-21T10:24:44.329150+0000","last_active":"2024-09-21T10:32:06.313319+0000","last_peered":"2024-09-21T10:32:06.313319+0000","last_clean":"2024-09-21T10:32:06.313319+0000","last_became_active":"2024-09-21T10:24:44.306752+0000","last_became_peered":"2024-09-21T10:24:44.306752+0000","last_unstale":"2024-09-21T10:32:06.313319+0000","last_undegraded":"2024-09-21T10:32:06.313319+0000","last_fullsized":"2024-09-21T10:32:06.313319+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:16.592859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":94,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.313082+0000","last_change":"2024-09-21T10:28:00.286009+0000","last_active":"2024-09-21T10:32:06.313082+0000","last_peered":"2024-09-21T10:32:06.313082+0000","last_clean":"2024-09-21T10:32:06.313082+0000","last_became_active":"2024-09-21T10:28:00.285327+0000","last_became_peered":"2024-09-21T10:28:00.285327+0000","last_unstale":"2024-09-21T10:32:06.313082+0000","last_undegraded":"2024-09-21T10:32:06.313082+0000","last_fullsized":"2024-09-21T10:32:06.313082+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:34.985354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":88,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.313064+0000","last_change":"2024-09-21T10:24:44.324087+0000","last_active":"2024-09-21T10:32:06.313064+0000","last_peered":"2024-09-21T10:32:06.313064+0000","last_clean":"2024-09-21T10:32:06.313064+0000","last_became_active":"2024-09-21T10:24:44.298665+0000","last_became_peered":"2024-09-21T10:24:44.298665+0000","last_unstale":"2024-09-21T10:32:06.313064+0000","last_undegraded":"2024-09-21T10:32:06.313064+0000","last_fullsized":"2024-09-21T10:32:06.313064+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:59.819340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":81,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T10:32:08.651757+0000","last_change":"2024-09-21T10:28:00.293054+0000","last_active":"2024-09-21T10:32:08.651757+0000","last_peered":"2024-09-21T10:32:08.651757+0000","last_clean":"2024-09-21T10:32:08.651757+0000","last_became_active":"2024-09-21T10:28:00.292723+0000","last_became_peered":"2024-09-21T10:28:00.292723+0000","last_unstale":"2024-09-21T10:32:08.651757+0000","last_undegraded":"2024-09-21T10:32:08.651757+0000","last_fullsized":"2024-09-21T10:32:08.651757+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:01:45.933570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":82,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T10:32:08.651899+0000","last_change":"2024-09-21T10:28:00.285515+0000","last_active":"2024-09-21T10:32:08.651899+0000","last_peered":"2024-09-21T10:32:08.651899+0000","last_clean":"2024-09-21T10:32:08.651899+0000","last_became_active":"2024-09-21T10:28:00.285203+0000","last_became_peered":"2024-09-21T10:28:00.285203+0000","last_unstale":"2024-09-21T10:32:08.651899+0000","last_undegraded":"2024-09-21T10:32:08.651899+0000","last_fullsized":"2024-09-21T10:32:08.651899+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:41.982228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":84,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:32:06.312544+0000","last_change":"2024-09-21T10:24:44.298820+0000","last_active":"2024-09-21T10:32:06.312544+0000","last_peered":"2024-09-21T10:32:06.312544+0000","last_clean":"2024-09-21T10:32:06.312544+0000","last_became_active":"2024-09-21T10:24:44.298478+0000","last_became_peered":"2024-09-21T10:24:44.298478+0000","last_unstale":"2024-09-21T10:32:06.312544+0000","last_undegraded":"2024-09-21T10:32:06.312544+0000","last_fullsized":"2024-09-21T10:32:06.312544+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:59.567078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":663,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-21T10:32:02.293063+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:32:02.293063+0000","last_peered":"2024-09-21T10:32:02.293063+0000","last_clean":"2024-09-21T10:32:02.293063+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:32:02.293063+0000","last_undegraded":"2024-09-21T10:32:02.293063+0000","last_fullsized":"2024-09-21T10:32:02.293063+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":10252,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5983,"num_read_kb":13006,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":316358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":69,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":644038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1100,"ondisk_log_size":1100,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453398008,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1309828,"kb_used_data":412548,"kb_used_omap":1,"kb_used_meta":897214,"kb_avail":92435324,"statfs":{"total":95995035648,"available":94653771776,"internally_reserved":0,"allocated":422449152,"data_stored":420157001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918747480},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594236,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":906616,"kb_used_data":5244,"kb_used_omap":1,"kb_used_meta":901310,"kb_avail":92838536,"statfs":{"total":95995035648,"available":95066660864,"internally_reserved":0,"allocated":5369856,"data_stored":3077520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922941784},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790466,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":914020,"kb_used_data":9892,"kb_used_omap":1,"kb_used_meta":904062,"kb_avail":92831132,"statfs":{"total":95995035648,"available":95059079168,"internally_reserved":0,"allocated":10129408,"data_stored":7857447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":925759832},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986695,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1006668,"kb_used_data":416768,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":92738484,"statfs":{"total":95995035648,"available":94964207616,"internally_reserved":0,"allocated":426770432,"data_stored":424482299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":101,"seq":433791697257,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":695804,"kb_used_data":5808,"kb_used_omap":5,"kb_used_meta":689978,"kb_avail":93049348,"statfs":{"total":95995035648,"available":95282532352,"internally_reserved":0,"allocated":5947392,"data_stored":3671407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":706538256},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860784,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1053768,"kb_used_data":9588,"kb_used_omap":8,"kb_used_meta":1044151,"kb_avail":92691384,"statfs":{"total":95995035648,"available":94915977216,"internally_reserved":0,"allocated":9818112,"data_stored":7538553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":1069211500},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024309,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1215948,"kb_used_data":403268,"kb_used_omap":7,"kb_used_meta":812664,"kb_avail":92529204,"statfs":{"total":95995035648,"available":94749904896,"internally_reserved":0,"allocated":412946432,"data_stored":410688208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":832168498},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187844,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1061716,"kb_used_data":5192,"kb_used_omap":8,"kb_used_meta":1056503,"kb_avail":92683436,"statfs":{"total":95995035648,"available":94907838464,"internally_reserved":0,"allocated":5316608,"data_stored":3055533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":1081860081},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:32:14.127 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T10:32:14.127 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:32:14.127 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 5, 6, 7, 4, 3] 2024-09-21T10:32:14.127 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd out 1 2024-09-21T10:32:14.772 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:32:14.772 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:32:14.772 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-21T10:32:14.773 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-21T10:32:14.773 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:32:15.044 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:15.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:15 smithi098 ceph-mon[97580]: pgmap v1384: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:15.449 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:15 smithi098 ceph-mon[100914]: pgmap v1384: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:15.454 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:15.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:32:15.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:15 smithi196 ceph-mon[84784]: pgmap v1384: 83 pgs: 83 active+clean; 4.3 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:17.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:17 smithi098 ceph-mon[97580]: pgmap v1385: 83 pgs: 83 active+clean; 4.3 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:17.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:17 smithi098 ceph-mon[100914]: pgmap v1385: 83 pgs: 83 active+clean; 4.3 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:17.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:17 smithi196 ceph-mon[84784]: pgmap v1385: 83 pgs: 83 active+clean; 4.3 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:18.002 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:18.333 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-21T10:32:18.386 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:18 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3607239609' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:32:18.386 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:18 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3607239609' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:32:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:18 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3607239609' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:32:19.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:19 smithi098 ceph-mon[97580]: pgmap v1386: 83 pgs: 83 active+clean; 4.3 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:19.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:19 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3607239609' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:32:19.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:19 smithi098 ceph-mon[97580]: osdmap e396: 8 total, 8 up, 8 in 2024-09-21T10:32:19.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:19 smithi098 ceph-mon[100914]: pgmap v1386: 83 pgs: 83 active+clean; 4.3 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:19.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:19 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3607239609' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:32:19.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:19 smithi098 ceph-mon[100914]: osdmap e396: 8 total, 8 up, 8 in 2024-09-21T10:32:19.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:19 smithi196 ceph-mon[84784]: pgmap v1386: 83 pgs: 83 active+clean; 4.3 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:19 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3607239609' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:32:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:19 smithi196 ceph-mon[84784]: osdmap e396: 8 total, 8 up, 8 in 2024-09-21T10:32:19.864 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:32:20.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:32:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:32:20.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:32:20.789 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:32:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:32:20] "GET /metrics HTTP/1.1" 200 36587 "" "Prometheus/2.51.0" 2024-09-21T10:32:21.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:21 smithi098 ceph-mon[97580]: pgmap v1388: 83 pgs: 83 active+clean; 4.3 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:21.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:21 smithi098 ceph-mon[100914]: pgmap v1388: 83 pgs: 83 active+clean; 4.3 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:21.637 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:21.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:21 smithi196 ceph-mon[84784]: pgmap v1388: 83 pgs: 83 active+clean; 4.3 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:21.645 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:32:21.646 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-21T10:32:21.646 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-21T10:32:21.646 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:32:21.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.646 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:114049 10.5 10:a91b5b21:::benchmark_data_smithi098_281896_object114048:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-09-21T10:32:21.646 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:01.455401+0000", 2024-09-21T10:32:21.646 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 260.186044756, 2024-09-21T10:32:21.646 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.223158365, 2024-09-21T10:32:21.646 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:21.646 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:21.646 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:21.646 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:32:21.647 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:32:21.647 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:21.647 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 114049 2024-09-21T10:32:21.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.647 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:21.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.647 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:21.647 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.455401+0000", 2024-09-21T10:32:21.647 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.647 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:21.648 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.455401+0000", 2024-09-21T10:32:21.648 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.648 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:21.648 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.455403+0000", 2024-09-21T10:32:21.648 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7290000000000001e-06 2024-09-21T10:32:21.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.648 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:21.648 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.455429+0000", 2024-09-21T10:32:21.648 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5422000000000002e-05 2024-09-21T10:32:21.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.649 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:21.649 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.455442+0000", 2024-09-21T10:32:21.649 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3037000000000001e-05 2024-09-21T10:32:21.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.649 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:21.649 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.455450+0000", 2024-09-21T10:32:21.649 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3219999999999993e-06 2024-09-21T10:32:21.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.650 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:21.650 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.497851+0000", 2024-09-21T10:32:21.650 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.042401094 2024-09-21T10:32:21.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.650 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.650 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:32:21.651 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.497857+0000", 2024-09-21T10:32:21.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4130000000000003e-06 2024-09-21T10:32:21.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.651 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:21.651 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.649683+0000", 2024-09-21T10:32:21.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.15182572799999999 2024-09-21T10:32:21.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.651 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:21.651 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.649865+0000", 2024-09-21T10:32:21.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018164200000000001 2024-09-21T10:32:21.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.652 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:32:21.652 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.650040+0000", 2024-09-21T10:32:21.652 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017542400000000001 2024-09-21T10:32:21.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.652 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.652 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.652 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.664461+0000", 2024-09-21T10:32:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014420448000000001 2024-09-21T10:32:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:32:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.664519+0000", 2024-09-21T10:32:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8797e-05 2024-09-21T10:32:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.673850+0000", 2024-09-21T10:32:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0093306540000000007 2024-09-21T10:32:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.673953+0000", 2024-09-21T10:32:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000103106 2024-09-21T10:32:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.678560+0000", 2024-09-21T10:32:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046065489999999997 2024-09-21T10:32:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:114077 10.3 10:c932c26e:::benchmark_data_smithi098_281896_object114076:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-09-21T10:32:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:01.525273+0000", 2024-09-21T10:32:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 260.11617298499999, 2024-09-21T10:32:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.19998084099999999, 2024-09-21T10:32:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:21.655 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:21.655 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:21.655 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:32:21.655 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:32:21.655 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:21.655 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 114077 2024-09-21T10:32:21.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.655 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:21.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.655 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:21.655 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.525273+0000", 2024-09-21T10:32:21.655 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.655 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:21.656 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.525273+0000", 2024-09-21T10:32:21.656 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.656 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:21.656 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.525275+0000", 2024-09-21T10:32:21.656 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6780000000000001e-06 2024-09-21T10:32:21.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.656 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:21.656 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.525310+0000", 2024-09-21T10:32:21.656 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5423999999999998e-05 2024-09-21T10:32:21.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.656 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:21.656 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.525328+0000", 2024-09-21T10:32:21.657 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8151e-05 2024-09-21T10:32:21.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.657 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.657 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:21.657 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.525337+0000", 2024-09-21T10:32:21.657 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0589999999999992e-06 2024-09-21T10:32:21.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.657 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.657 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:21.657 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.525402+0000", 2024-09-21T10:32:21.658 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4238000000000004e-05 2024-09-21T10:32:21.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.658 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-21T10:32:21.658 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.525412+0000", 2024-09-21T10:32:21.658 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8719999999999997e-06 2024-09-21T10:32:21.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.658 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:21.658 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.686377+0000", 2024-09-21T10:32:21.658 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.16096533099999999 2024-09-21T10:32:21.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.659 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:21.659 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.686539+0000", 2024-09-21T10:32:21.659 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016166600000000001 2024-09-21T10:32:21.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.659 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:32:21.659 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.686700+0000", 2024-09-21T10:32:21.659 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016121500000000001 2024-09-21T10:32:21.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.659 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.659 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.692189+0000", 2024-09-21T10:32:21.659 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054890980000000004 2024-09-21T10:32:21.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.660 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:32:21.660 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.701100+0000", 2024-09-21T10:32:21.660 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0089110319999999993 2024-09-21T10:32:21.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.660 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.660 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.719247+0000", 2024-09-21T10:32:21.660 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018147513000000001 2024-09-21T10:32:21.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.661 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:21.661 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.719457+0000", 2024-09-21T10:32:21.661 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020962300000000001 2024-09-21T10:32:21.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.662 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:21.662 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:01.725254+0000", 2024-09-21T10:32:21.662 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057969409999999999 2024-09-21T10:32:21.662 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.662 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:21.662 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.662 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:114728 10.5 10:a6448361:::benchmark_data_smithi098_281896_object114727:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-21T10:32:21.662 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:02.533408+0000", 2024-09-21T10:32:21.662 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 259.10803769, 2024-09-21T10:32:21.662 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.216344283, 2024-09-21T10:32:21.662 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:21.662 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:21.663 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:21.663 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:32:21.663 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:32:21.663 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:21.663 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 114728 2024-09-21T10:32:21.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.663 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:21.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.663 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:21.663 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.533408+0000", 2024-09-21T10:32:21.663 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.663 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:21.663 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.533408+0000", 2024-09-21T10:32:21.664 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:21.664 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.533410+0000", 2024-09-21T10:32:21.664 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4869999999999999e-06 2024-09-21T10:32:21.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:21.664 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.533437+0000", 2024-09-21T10:32:21.664 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7410999999999999e-05 2024-09-21T10:32:21.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:21.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.533448+0000", 2024-09-21T10:32:21.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0917000000000001e-05 2024-09-21T10:32:21.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:21.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.533459+0000", 2024-09-21T10:32:21.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0363999999999999e-05 2024-09-21T10:32:21.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:21.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.545536+0000", 2024-09-21T10:32:21.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012076988 2024-09-21T10:32:21.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.666 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:21.666 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.545705+0000", 2024-09-21T10:32:21.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016944200000000001 2024-09-21T10:32:21.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.666 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:32:21.666 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.546015+0000", 2024-09-21T10:32:21.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00030984300000000002 2024-09-21T10:32:21.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.666 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.666 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.557937+0000", 2024-09-21T10:32:21.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011921932 2024-09-21T10:32:21.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:32:21.667 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.557952+0000", 2024-09-21T10:32:21.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5170999999999999e-05 2024-09-21T10:32:21.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.667 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.744455+0000", 2024-09-21T10:32:21.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.18650330400000001 2024-09-21T10:32:21.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.668 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:21.668 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.744517+0000", 2024-09-21T10:32:21.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1334000000000006e-05 2024-09-21T10:32:21.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.668 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:21.668 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.749753+0000", 2024-09-21T10:32:21.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052360899999999997 2024-09-21T10:32:21.668 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.668 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:21.668 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.668 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:114735 10.5 10:a3b215c2:::benchmark_data_smithi098_281896_object114734:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-21T10:32:21.669 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:02.534475+0000", 2024-09-21T10:32:21.669 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 259.10697085200002, 2024-09-21T10:32:21.669 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.22054879599999999, 2024-09-21T10:32:21.669 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:21.669 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:21.669 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:21.669 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:32:21.669 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:32:21.669 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:21.669 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 114735 2024-09-21T10:32:21.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.669 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:21.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:21.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.534475+0000", 2024-09-21T10:32:21.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:21.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.534475+0000", 2024-09-21T10:32:21.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:21.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.534477+0000", 2024-09-21T10:32:21.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-21T10:32:21.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.671 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:21.671 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.534497+0000", 2024-09-21T10:32:21.671 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0078e-05 2024-09-21T10:32:21.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.671 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:21.671 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.534515+0000", 2024-09-21T10:32:21.671 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8036000000000001e-05 2024-09-21T10:32:21.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.671 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:21.671 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.534528+0000", 2024-09-21T10:32:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3369e-05 2024-09-21T10:32:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.552136+0000", 2024-09-21T10:32:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01760774 2024-09-21T10:32:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.552314+0000", 2024-09-21T10:32:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017825800000000001 2024-09-21T10:32:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:32:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.552471+0000", 2024-09-21T10:32:21.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000156532 2024-09-21T10:32:21.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.673 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:32:21.673 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.577378+0000", 2024-09-21T10:32:21.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.024906924 2024-09-21T10:32:21.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.673 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.673 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.577440+0000", 2024-09-21T10:32:21.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2334000000000003e-05 2024-09-21T10:32:21.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.673 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.674 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.749808+0000", 2024-09-21T10:32:21.674 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.17236832199999999 2024-09-21T10:32:21.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.674 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:21.674 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.749916+0000", 2024-09-21T10:32:21.674 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010746799999999999 2024-09-21T10:32:21.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.674 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:21.674 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.755024+0000", 2024-09-21T10:32:21.674 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051083179999999997 2024-09-21T10:32:21.674 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.675 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:21.675 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.675 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:114736 10.5 10:a153896d:::benchmark_data_smithi098_281896_object114735:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-21T10:32:21.675 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:02.535607+0000", 2024-09-21T10:32:21.675 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 259.10583876300001, 2024-09-21T10:32:21.675 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.22527894700000001, 2024-09-21T10:32:21.675 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:21.675 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:21.675 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:21.675 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:32:21.675 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:32:21.676 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:21.676 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 114736 2024-09-21T10:32:21.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.676 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:21.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.676 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:21.676 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.535607+0000", 2024-09-21T10:32:21.676 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.676 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:21.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.535607+0000", 2024-09-21T10:32:21.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:21.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.535609+0000", 2024-09-21T10:32:21.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3599999999999999e-06 2024-09-21T10:32:21.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:21.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.535654+0000", 2024-09-21T10:32:21.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5602999999999997e-05 2024-09-21T10:32:21.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:21.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.535666+0000", 2024-09-21T10:32:21.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2054e-05 2024-09-21T10:32:21.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:21.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.535674+0000", 2024-09-21T10:32:21.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7030000000000006e-06 2024-09-21T10:32:21.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:21.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.558050+0000", 2024-09-21T10:32:21.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.022376255000000001 2024-09-21T10:32:21.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:21.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.558251+0000", 2024-09-21T10:32:21.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020019100000000001 2024-09-21T10:32:21.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:32:21.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.558393+0000", 2024-09-21T10:32:21.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000142931 2024-09-21T10:32:21.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.680 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:32:21.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.577385+0000", 2024-09-21T10:32:21.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018991642999999999 2024-09-21T10:32:21.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.680 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.577498+0000", 2024-09-21T10:32:21.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000112521 2024-09-21T10:32:21.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.755073+0000", 2024-09-21T10:32:21.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.177575545 2024-09-21T10:32:21.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:21.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.755201+0000", 2024-09-21T10:32:21.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012760700000000001 2024-09-21T10:32:21.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:21.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.760886+0000", 2024-09-21T10:32:21.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056855339999999999 2024-09-21T10:32:21.682 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.682 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:21.682 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.683 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:114756 10.5 10:a9ea4135:::benchmark_data_smithi098_281896_object114755:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-21T10:32:21.683 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:02.561872+0000", 2024-09-21T10:32:21.683 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 259.07957375799998, 2024-09-21T10:32:21.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.20555221200000001, 2024-09-21T10:32:21.683 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:21.683 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:32:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:32:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 114756 2024-09-21T10:32:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.561872+0000", 2024-09-21T10:32:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.561872+0000", 2024-09-21T10:32:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.561874+0000", 2024-09-21T10:32:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7540000000000001e-06 2024-09-21T10:32:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.561903+0000", 2024-09-21T10:32:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8479999999999998e-05 2024-09-21T10:32:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:21.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.561917+0000", 2024-09-21T10:32:21.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4191e-05 2024-09-21T10:32:21.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:21.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.561925+0000", 2024-09-21T10:32:21.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5089999999999993e-06 2024-09-21T10:32:21.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:21.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.562239+0000", 2024-09-21T10:32:21.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031390200000000002 2024-09-21T10:32:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.562422+0000", 2024-09-21T10:32:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000183003 2024-09-21T10:32:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:32:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.562595+0000", 2024-09-21T10:32:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017252200000000001 2024-09-21T10:32:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:32:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.577512+0000", 2024-09-21T10:32:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014916992 2024-09-21T10:32:21.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.577547+0000", 2024-09-21T10:32:21.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5769000000000003e-05 2024-09-21T10:32:21.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.760953+0000", 2024-09-21T10:32:21.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.18340527400000001 2024-09-21T10:32:21.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.761221+0000", 2024-09-21T10:32:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00026867699999999999 2024-09-21T10:32:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.767425+0000", 2024-09-21T10:32:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062031389999999999 2024-09-21T10:32:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.690 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:114760 10.5 10:ab5b8f37:::benchmark_data_smithi098_281896_object114759:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-21T10:32:21.690 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:28:02.563166+0000", 2024-09-21T10:32:21.690 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 259.07827973299999, 2024-09-21T10:32:21.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.21826158900000001, 2024-09-21T10:32:21.690 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:21.690 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:21.690 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:21.690 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:32:21.690 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:32:21.690 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:21.690 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 114760 2024-09-21T10:32:21.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.690 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:21.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:21.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.563166+0000", 2024-09-21T10:32:21.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:21.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.563166+0000", 2024-09-21T10:32:21.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:21.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.563168+0000", 2024-09-21T10:32:21.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.201e-06 2024-09-21T10:32:21.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:21.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.563222+0000", 2024-09-21T10:32:21.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4141999999999998e-05 2024-09-21T10:32:21.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:21.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.563235+0000", 2024-09-21T10:32:21.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.33e-05 2024-09-21T10:32:21.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:21.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.563243+0000", 2024-09-21T10:32:21.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8869999999999992e-06 2024-09-21T10:32:21.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.693 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:21.693 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.569889+0000", 2024-09-21T10:32:21.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0066461630000000001 2024-09-21T10:32:21.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.693 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:21.693 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.570071+0000", 2024-09-21T10:32:21.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000182287 2024-09-21T10:32:21.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.694 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:32:21.694 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.570256+0000", 2024-09-21T10:32:21.694 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018436 2024-09-21T10:32:21.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.694 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.694 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.579264+0000", 2024-09-21T10:32:21.694 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0090080300000000002 2024-09-21T10:32:21.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.694 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:32:21.695 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.744409+0000", 2024-09-21T10:32:21.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.16514512200000001 2024-09-21T10:32:21.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.695 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.695 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.769251+0000", 2024-09-21T10:32:21.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.024842426000000001 2024-09-21T10:32:21.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.695 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:21.695 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.769351+0000", 2024-09-21T10:32:21.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9467000000000002e-05 2024-09-21T10:32:21.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.696 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:21.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:28:02.781428+0000", 2024-09-21T10:32:21.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012077203999999999 2024-09-21T10:32:21.696 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.696 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:21.696 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.696 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:194347 10.3 10:c983ca5d:::benchmark_data_smithi098_281896_object69322:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T10:32:21.696 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:58.408514+0000", 2024-09-21T10:32:21.696 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 143.23293240800001, 2024-09-21T10:32:21.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.21595936499999999, 2024-09-21T10:32:21.696 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:21.696 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:21.697 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:21.697 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:32:21.697 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:32:21.697 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:21.697 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 194347 2024-09-21T10:32:21.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.697 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:21.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.697 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:21.697 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.408514+0000", 2024-09-21T10:32:21.697 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.698 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:21.698 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.408514+0000", 2024-09-21T10:32:21.698 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.698 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:21.698 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.408515+0000", 2024-09-21T10:32:21.698 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.42e-06 2024-09-21T10:32:21.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.698 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:21.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.408521+0000", 2024-09-21T10:32:21.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6579999999999999e-06 2024-09-21T10:32:21.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:21.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.408538+0000", 2024-09-21T10:32:21.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7563000000000001e-05 2024-09-21T10:32:21.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:21.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.408546+0000", 2024-09-21T10:32:21.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5179999999999998e-06 2024-09-21T10:32:21.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.700 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:21.700 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.413619+0000", 2024-09-21T10:32:21.700 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050727869999999996 2024-09-21T10:32:21.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.700 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:21.700 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.413833+0000", 2024-09-21T10:32:21.700 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00021412699999999999 2024-09-21T10:32:21.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.700 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:32:21.701 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.414351+0000", 2024-09-21T10:32:21.701 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00051823199999999998 2024-09-21T10:32:21.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.701 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.701 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.420501+0000", 2024-09-21T10:32:21.701 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061496559999999999 2024-09-21T10:32:21.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.701 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:32:21.702 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.420573+0000", 2024-09-21T10:32:21.702 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1888e-05 2024-09-21T10:32:21.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.702 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.702 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.619870+0000", 2024-09-21T10:32:21.702 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.19929702299999999 2024-09-21T10:32:21.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:21.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.619985+0000", 2024-09-21T10:32:21.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011560700000000001 2024-09-21T10:32:21.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:21.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.624473+0000", 2024-09-21T10:32:21.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004487886 2024-09-21T10:32:21.704 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.704 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:21.704 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.704 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:194363 10.3 10:c55fd18a:::benchmark_data_smithi098_281896_object69338:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T10:32:21.704 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:58.428127+0000", 2024-09-21T10:32:21.704 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 143.213318824, 2024-09-21T10:32:21.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.20107881, 2024-09-21T10:32:21.705 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:21.705 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:21.705 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:21.705 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:32:21.705 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:32:21.705 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:21.705 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 194363 2024-09-21T10:32:21.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.705 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:21.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.706 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:21.706 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.428127+0000", 2024-09-21T10:32:21.706 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.706 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:21.706 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.428127+0000", 2024-09-21T10:32:21.707 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.707 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:21.707 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.428128+0000", 2024-09-21T10:32:21.707 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.034e-06 2024-09-21T10:32:21.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.708 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:21.708 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.428133+0000", 2024-09-21T10:32:21.708 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.493e-06 2024-09-21T10:32:21.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.710 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:21.710 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.428148+0000", 2024-09-21T10:32:21.710 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4778e-05 2024-09-21T10:32:21.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.710 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:21.710 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.428155+0000", 2024-09-21T10:32:21.710 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5440000000000001e-06 2024-09-21T10:32:21.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.711 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:21.711 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.428220+0000", 2024-09-21T10:32:21.711 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4445999999999999e-05 2024-09-21T10:32:21.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.711 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:21.711 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.428402+0000", 2024-09-21T10:32:21.712 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000182561 2024-09-21T10:32:21.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.712 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:32:21.712 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.428577+0000", 2024-09-21T10:32:21.712 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000174554 2024-09-21T10:32:21.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.713 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:32:21.713 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.433385+0000", 2024-09-21T10:32:21.713 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048079769999999997 2024-09-21T10:32:21.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.713 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.713 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.434036+0000", 2024-09-21T10:32:21.713 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00065165399999999999 2024-09-21T10:32:21.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.714 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.714 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.624566+0000", 2024-09-21T10:32:21.714 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.19052939099999999 2024-09-21T10:32:21.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.714 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:21.714 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.624748+0000", 2024-09-21T10:32:21.715 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000182146 2024-09-21T10:32:21.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.715 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:21.715 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.629206+0000", 2024-09-21T10:32:21.715 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044582320000000003 2024-09-21T10:32:21.715 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.715 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:21.715 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.715 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:194375 10.3 10:c3c497a9:::benchmark_data_smithi098_281896_object69350:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T10:32:21.716 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:58.448483+0000", 2024-09-21T10:32:21.716 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 143.19296275799999, 2024-09-21T10:32:21.716 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.20344902100000001, 2024-09-21T10:32:21.716 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:21.716 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:21.716 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:21.716 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:32:21.716 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:32:21.716 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:21.716 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 194375 2024-09-21T10:32:21.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.716 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:21.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.716 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:21.717 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.448483+0000", 2024-09-21T10:32:21.717 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.717 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:21.717 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.448483+0000", 2024-09-21T10:32:21.717 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.717 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:21.717 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.448486+0000", 2024-09-21T10:32:21.717 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6960000000000001e-06 2024-09-21T10:32:21.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.718 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:21.718 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.448496+0000", 2024-09-21T10:32:21.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.042e-05 2024-09-21T10:32:21.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.718 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:21.718 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.448530+0000", 2024-09-21T10:32:21.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3046999999999997e-05 2024-09-21T10:32:21.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.718 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:21.718 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.448543+0000", 2024-09-21T10:32:21.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3949e-05 2024-09-21T10:32:21.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.719 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:21.719 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.448610+0000", 2024-09-21T10:32:21.719 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6186999999999995e-05 2024-09-21T10:32:21.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.719 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:21.719 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.448718+0000", 2024-09-21T10:32:21.719 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108769 2024-09-21T10:32:21.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.719 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:32:21.719 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.448816+0000", 2024-09-21T10:32:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7490000000000003e-05 2024-09-21T10:32:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:32:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.453383+0000", 2024-09-21T10:32:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0045669420000000001 2024-09-21T10:32:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.460793+0000", 2024-09-21T10:32:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0074100600000000004 2024-09-21T10:32:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.721 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.647165+0000", 2024-09-21T10:32:21.721 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.18637221400000001 2024-09-21T10:32:21.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.721 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:21.721 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.647226+0000", 2024-09-21T10:32:21.721 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0843e-05 2024-09-21T10:32:21.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.721 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:21.721 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.651932+0000", 2024-09-21T10:32:21.721 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004706404 2024-09-21T10:32:21.721 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.721 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:21.722 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.722 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:194379 10.3 10:c4251f5b:::benchmark_data_smithi098_281896_object69354:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T10:32:21.722 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:58.453493+0000", 2024-09-21T10:32:21.722 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 143.18795276099999, 2024-09-21T10:32:21.722 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.20492676400000001, 2024-09-21T10:32:21.722 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:21.722 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:21.722 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:21.722 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:32:21.722 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:32:21.722 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:21.723 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 194379 2024-09-21T10:32:21.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.723 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:21.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.723 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:21.723 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.453493+0000", 2024-09-21T10:32:21.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.723 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:21.723 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.453493+0000", 2024-09-21T10:32:21.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.724 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:21.724 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.453495+0000", 2024-09-21T10:32:21.724 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2559999999999999e-06 2024-09-21T10:32:21.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.724 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:21.724 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.453499+0000", 2024-09-21T10:32:21.724 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6099999999999999e-06 2024-09-21T10:32:21.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.724 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:21.724 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.453511+0000", 2024-09-21T10:32:21.724 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2169e-05 2024-09-21T10:32:21.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.725 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:21.725 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.453517+0000", 2024-09-21T10:32:21.725 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0939999999999996e-06 2024-09-21T10:32:21.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.725 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:21.725 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.453572+0000", 2024-09-21T10:32:21.725 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5024e-05 2024-09-21T10:32:21.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.725 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:21.725 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.453714+0000", 2024-09-21T10:32:21.725 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014154 2024-09-21T10:32:21.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.726 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:32:21.726 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.453812+0000", 2024-09-21T10:32:21.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7578000000000007e-05 2024-09-21T10:32:21.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.726 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:32:21.726 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.460812+0000", 2024-09-21T10:32:21.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070002839999999998 2024-09-21T10:32:21.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.726 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.460856+0000", 2024-09-21T10:32:21.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4047e-05 2024-09-21T10:32:21.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.652157+0000", 2024-09-21T10:32:21.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.19130058699999999 2024-09-21T10:32:21.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:21.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.652484+0000", 2024-09-21T10:32:21.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00032727100000000002 2024-09-21T10:32:21.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:21.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.658420+0000", 2024-09-21T10:32:21.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0059363039999999999 2024-09-21T10:32:21.728 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.729 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:21.729 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.729 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:194381 10.3 10:cad332c4:::benchmark_data_smithi098_281896_object69356:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T10:32:21.729 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:58.453535+0000", 2024-09-21T10:32:21.729 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 143.18791119700001, 2024-09-21T10:32:21.729 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.21411211399999999, 2024-09-21T10:32:21.729 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:21.729 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:21.729 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:21.729 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:32:21.729 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:32:21.729 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:21.730 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 194381 2024-09-21T10:32:21.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.730 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:21.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:21.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.453535+0000", 2024-09-21T10:32:21.730 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:21.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.453535+0000", 2024-09-21T10:32:21.730 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.731 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:21.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.453535+0000", 2024-09-21T10:32:21.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1100000000000002e-07 2024-09-21T10:32:21.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.731 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:21.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.453537+0000", 2024-09-21T10:32:21.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9939999999999999e-06 2024-09-21T10:32:21.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.731 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:21.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.453547+0000", 2024-09-21T10:32:21.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9760000000000007e-06 2024-09-21T10:32:21.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:21.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.453552+0000", 2024-09-21T10:32:21.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6870000000000002e-06 2024-09-21T10:32:21.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:21.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.457364+0000", 2024-09-21T10:32:21.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038125110000000002 2024-09-21T10:32:21.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:21.733 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.457495+0000", 2024-09-21T10:32:21.733 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000130584 2024-09-21T10:32:21.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.733 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:32:21.733 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.457590+0000", 2024-09-21T10:32:21.733 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5341999999999996e-05 2024-09-21T10:32:21.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.733 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:32:21.733 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.460875+0000", 2024-09-21T10:32:21.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032845779999999998 2024-09-21T10:32:21.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.734 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.734 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.461799+0000", 2024-09-21T10:32:21.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00092424800000000002 2024-09-21T10:32:21.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.734 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.734 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.658628+0000", 2024-09-21T10:32:21.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.19682842 2024-09-21T10:32:21.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:21.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.658855+0000", 2024-09-21T10:32:21.735 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000227352 2024-09-21T10:32:21.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:21.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.667647+0000", 2024-09-21T10:32:21.735 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.008792111 2024-09-21T10:32:21.735 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.735 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:194918 10.3 10:cb5ee597:::benchmark_data_smithi098_281896_object69893:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T10:32:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:59.306407+0000", 2024-09-21T10:32:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 142.33503891699999, 2024-09-21T10:32:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.210726461, 2024-09-21T10:32:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:32:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:32:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 194918 2024-09-21T10:32:21.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.737 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:21.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:21.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:59.306407+0000", 2024-09-21T10:32:21.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:21.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:59.306407+0000", 2024-09-21T10:32:21.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:21.738 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:59.306409+0000", 2024-09-21T10:32:21.738 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.691e-06 2024-09-21T10:32:21.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.738 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:21.738 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:59.306413+0000", 2024-09-21T10:32:21.738 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6509999999999999e-06 2024-09-21T10:32:21.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.738 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:59.306421+0000", 2024-09-21T10:32:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5169999999999999e-06 2024-09-21T10:32:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:59.306427+0000", 2024-09-21T10:32:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5589999999999996e-06 2024-09-21T10:32:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:59.323924+0000", 2024-09-21T10:32:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017497803999999999 2024-09-21T10:32:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:21.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:59.324104+0000", 2024-09-21T10:32:21.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017936699999999999 2024-09-21T10:32:21.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:32:21.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:59.324280+0000", 2024-09-21T10:32:21.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017633899999999999 2024-09-21T10:32:21.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:59.336026+0000", 2024-09-21T10:32:21.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011745996 2024-09-21T10:32:21.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:32:21.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:59.336042+0000", 2024-09-21T10:32:21.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5971999999999999e-05 2024-09-21T10:32:21.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:59.512448+0000", 2024-09-21T10:32:21.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.17640571599999999 2024-09-21T10:32:21.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:21.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:59.512518+0000", 2024-09-21T10:32:21.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9849000000000002e-05 2024-09-21T10:32:21.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.742 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:21.742 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:59.517134+0000", 2024-09-21T10:32:21.742 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046160000000000003 2024-09-21T10:32:21.742 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.742 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:21.742 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.742 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:195637 10.3 10:cd271909:::benchmark_data_smithi098_281896_object70612:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T10:32:21.742 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:30:00.561243+0000", 2024-09-21T10:32:21.742 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 141.08020321800001, 2024-09-21T10:32:21.742 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.219495205, 2024-09-21T10:32:21.743 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:21.743 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:21.743 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:21.743 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:32:21.743 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:32:21.743 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:21.743 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 195637 2024-09-21T10:32:21.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.743 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:21.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.743 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:21.743 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.561243+0000", 2024-09-21T10:32:21.743 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:21.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.561243+0000", 2024-09-21T10:32:21.744 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:21.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.561245+0000", 2024-09-21T10:32:21.744 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.674e-06 2024-09-21T10:32:21.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:21.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.561250+0000", 2024-09-21T10:32:21.744 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1950000000000002e-06 2024-09-21T10:32:21.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:21.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.561267+0000", 2024-09-21T10:32:21.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7397e-05 2024-09-21T10:32:21.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:21.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.561275+0000", 2024-09-21T10:32:21.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2800000000000003e-06 2024-09-21T10:32:21.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:21.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.562098+0000", 2024-09-21T10:32:21.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0008229 2024-09-21T10:32:21.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:21.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.562269+0000", 2024-09-21T10:32:21.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017080799999999999 2024-09-21T10:32:21.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:32:21.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.562428+0000", 2024-09-21T10:32:21.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015912899999999999 2024-09-21T10:32:21.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.747 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:32:21.747 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.578778+0000", 2024-09-21T10:32:21.747 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016350111000000001 2024-09-21T10:32:21.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.747 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.747 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.588687+0000", 2024-09-21T10:32:21.747 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0099083699999999997 2024-09-21T10:32:21.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.747 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.747 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.776730+0000", 2024-09-21T10:32:21.747 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.18804307100000001 2024-09-21T10:32:21.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.748 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:21.748 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.776796+0000", 2024-09-21T10:32:21.748 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5747999999999995e-05 2024-09-21T10:32:21.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.748 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:21.748 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.780738+0000", 2024-09-21T10:32:21.748 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039425220000000004 2024-09-21T10:32:21.748 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.748 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:21.748 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.749 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:195639 10.3 10:c5c325f2:::benchmark_data_smithi098_281896_object70614:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T10:32:21.749 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:30:00.561295+0000", 2024-09-21T10:32:21.749 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 141.08015078299999, 2024-09-21T10:32:21.749 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.22352591299999999, 2024-09-21T10:32:21.749 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:21.749 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:21.749 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:21.749 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:32:21.749 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:32:21.749 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:21.749 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 195639 2024-09-21T10:32:21.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.750 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:21.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.750 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:21.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.561295+0000", 2024-09-21T10:32:21.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.750 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:21.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.561295+0000", 2024-09-21T10:32:21.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.750 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:21.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.561296+0000", 2024-09-21T10:32:21.751 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2099999999999997e-07 2024-09-21T10:32:21.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.751 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:21.751 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.561298+0000", 2024-09-21T10:32:21.751 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5890000000000001e-06 2024-09-21T10:32:21.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.751 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:21.751 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.561305+0000", 2024-09-21T10:32:21.751 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5300000000000002e-06 2024-09-21T10:32:21.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.751 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:21.752 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.561310+0000", 2024-09-21T10:32:21.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9130000000000002e-06 2024-09-21T10:32:21.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.752 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:21.752 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.567925+0000", 2024-09-21T10:32:21.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0066154400000000002 2024-09-21T10:32:21.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.752 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:21.752 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.568090+0000", 2024-09-21T10:32:21.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016445699999999999 2024-09-21T10:32:21.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:32:21.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.568280+0000", 2024-09-21T10:32:21.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000189829 2024-09-21T10:32:21.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:32:21.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.578792+0000", 2024-09-21T10:32:21.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010511924000000001 2024-09-21T10:32:21.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.602229+0000", 2024-09-21T10:32:21.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.023437360000000001 2024-09-21T10:32:21.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.780811+0000", 2024-09-21T10:32:21.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.17858190700000001 2024-09-21T10:32:21.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:21.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.780881+0000", 2024-09-21T10:32:21.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0179999999999996e-05 2024-09-21T10:32:21.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:21.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.784821+0000", 2024-09-21T10:32:21.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039402630000000003 2024-09-21T10:32:21.754 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.755 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:21.755 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.755 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:195660 10.3 10:c9558508:::benchmark_data_smithi098_281896_object70635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T10:32:21.755 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:30:00.588451+0000", 2024-09-21T10:32:21.755 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 141.052995259, 2024-09-21T10:32:21.755 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.21088068700000001, 2024-09-21T10:32:21.755 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:21.755 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:21.755 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:21.755 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:32:21.755 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:32:21.756 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:21.756 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 195660 2024-09-21T10:32:21.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.756 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:21.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:21.756 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.588451+0000", 2024-09-21T10:32:21.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:21.756 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.588451+0000", 2024-09-21T10:32:21.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.757 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:21.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.588452+0000", 2024-09-21T10:32:21.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5880000000000001e-06 2024-09-21T10:32:21.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.757 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:21.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.588459+0000", 2024-09-21T10:32:21.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3180000000000003e-06 2024-09-21T10:32:21.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.757 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:21.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.588475+0000", 2024-09-21T10:32:21.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6453999999999999e-05 2024-09-21T10:32:21.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:21.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.588483+0000", 2024-09-21T10:32:21.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5209999999999997e-06 2024-09-21T10:32:21.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:21.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.588771+0000", 2024-09-21T10:32:21.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00028863000000000001 2024-09-21T10:32:21.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:21.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.588938+0000", 2024-09-21T10:32:21.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016695300000000001 2024-09-21T10:32:21.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:32:21.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.589097+0000", 2024-09-21T10:32:21.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015886399999999999 2024-09-21T10:32:21.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:32:21.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.594256+0000", 2024-09-21T10:32:21.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051588709999999998 2024-09-21T10:32:21.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.602273+0000", 2024-09-21T10:32:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0080166490000000007 2024-09-21T10:32:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.795035+0000", 2024-09-21T10:32:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.19276238100000001 2024-09-21T10:32:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.795096+0000", 2024-09-21T10:32:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1155000000000007e-05 2024-09-21T10:32:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:00.799332+0000", 2024-09-21T10:32:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0042353030000000002 2024-09-21T10:32:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:201762 10.3 10:c0dfe06a:::benchmark_data_smithi098_281896_object76737:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T10:32:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:30:09.592773+0000", 2024-09-21T10:32:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 132.04867278899999, 2024-09-21T10:32:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.27284814299999999, 2024-09-21T10:32:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:32:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:32:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 201762 2024-09-21T10:32:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.592773+0000", 2024-09-21T10:32:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.592773+0000", 2024-09-21T10:32:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.592775+0000", 2024-09-21T10:32:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5770000000000001e-06 2024-09-21T10:32:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.592779+0000", 2024-09-21T10:32:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4120000000000003e-06 2024-09-21T10:32:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.592793+0000", 2024-09-21T10:32:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3667e-05 2024-09-21T10:32:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.592800+0000", 2024-09-21T10:32:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2760000000000003e-06 2024-09-21T10:32:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.595299+0000", 2024-09-21T10:32:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024991380000000001 2024-09-21T10:32:21.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:21.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.595511+0000", 2024-09-21T10:32:21.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00021163599999999999 2024-09-21T10:32:21.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:32:21.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.595698+0000", 2024-09-21T10:32:21.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000186971 2024-09-21T10:32:21.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:32:21.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.629612+0000", 2024-09-21T10:32:21.766 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.033913820999999997 2024-09-21T10:32:21.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.766 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.766 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.636144+0000", 2024-09-21T10:32:21.766 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0065320899999999999 2024-09-21T10:32:21.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.766 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.766 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.861616+0000", 2024-09-21T10:32:21.766 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.22547167600000001 2024-09-21T10:32:21.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.766 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:21.767 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.861659+0000", 2024-09-21T10:32:21.767 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3522000000000001e-05 2024-09-21T10:32:21.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.767 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:21.767 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.865621+0000", 2024-09-21T10:32:21.767 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039623569999999997 2024-09-21T10:32:21.767 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.767 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:21.767 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.767 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:201763 10.3 10:ccdaf4b6:::benchmark_data_smithi098_281896_object76738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T10:32:21.767 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:30:09.595575+0000", 2024-09-21T10:32:21.768 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 132.04587107699999, 2024-09-21T10:32:21.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.27446412799999997, 2024-09-21T10:32:21.768 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:21.768 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:21.768 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:21.768 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:32:21.768 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:32:21.768 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:21.768 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 201763 2024-09-21T10:32:21.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.768 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:21.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:21.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.595575+0000", 2024-09-21T10:32:21.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:21.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.595575+0000", 2024-09-21T10:32:21.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:21.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.595577+0000", 2024-09-21T10:32:21.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8050000000000001e-06 2024-09-21T10:32:21.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:21.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.595584+0000", 2024-09-21T10:32:21.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5429999999999996e-06 2024-09-21T10:32:21.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:21.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.595598+0000", 2024-09-21T10:32:21.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3988000000000001e-05 2024-09-21T10:32:21.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:21.771 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.595607+0000", 2024-09-21T10:32:21.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7569999999999993e-06 2024-09-21T10:32:21.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.771 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:21.771 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.602195+0000", 2024-09-21T10:32:21.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00658787 2024-09-21T10:32:21.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.771 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:21.771 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.602353+0000", 2024-09-21T10:32:21.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015812399999999999 2024-09-21T10:32:21.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.772 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:32:21.772 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.602511+0000", 2024-09-21T10:32:21.772 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000158183 2024-09-21T10:32:21.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.772 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:32:21.772 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.629621+0000", 2024-09-21T10:32:21.772 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027110074000000001 2024-09-21T10:32:21.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.772 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.772 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.636502+0000", 2024-09-21T10:32:21.772 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0068806570000000001 2024-09-21T10:32:21.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.773 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.865714+0000", 2024-09-21T10:32:21.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.229211943 2024-09-21T10:32:21.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:21.773 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.865917+0000", 2024-09-21T10:32:21.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000203055 2024-09-21T10:32:21.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:21.773 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:09.870039+0000", 2024-09-21T10:32:21.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0041221290000000004 2024-09-21T10:32:21.774 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.774 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:21.774 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.774 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:246055 10.3 10:c1c1b688:::benchmark_data_smithi098_281896_object121030:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-21T10:32:21.774 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:31:12.465680+0000", 2024-09-21T10:32:21.774 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 69.175766445999997, 2024-09-21T10:32:21.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.15064111899999999, 2024-09-21T10:32:21.774 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:21.774 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:21.774 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:21.774 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:32:21.775 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:32:21.775 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:21.775 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 246055 2024-09-21T10:32:21.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.775 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:21.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:21.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.465680+0000", 2024-09-21T10:32:21.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:21.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.465680+0000", 2024-09-21T10:32:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.465681+0000", 2024-09-21T10:32:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3960000000000001e-06 2024-09-21T10:32:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.465687+0000", 2024-09-21T10:32:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8420000000000001e-06 2024-09-21T10:32:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.465698+0000", 2024-09-21T10:32:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1474999999999999e-05 2024-09-21T10:32:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.465706+0000", 2024-09-21T10:32:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3209999999999999e-06 2024-09-21T10:32:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.486305+0000", 2024-09-21T10:32:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020599310999999999 2024-09-21T10:32:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:21.778 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.486475+0000", 2024-09-21T10:32:21.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000169567 2024-09-21T10:32:21.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.778 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:32:21.778 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.486639+0000", 2024-09-21T10:32:21.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000164797 2024-09-21T10:32:21.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.778 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.778 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.491804+0000", 2024-09-21T10:32:21.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051646920000000002 2024-09-21T10:32:21.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:32:21.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.542517+0000", 2024-09-21T10:32:21.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.050712977999999999 2024-09-21T10:32:21.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.613206+0000", 2024-09-21T10:32:21.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.070688619999999994 2024-09-21T10:32:21.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:21.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.613273+0000", 2024-09-21T10:32:21.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6859000000000005e-05 2024-09-21T10:32:21.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:21.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.616321+0000", 2024-09-21T10:32:21.780 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003048261 2024-09-21T10:32:21.780 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.780 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:21.780 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.780 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.46322.0:248372 10.3 10:c1549b8c:::benchmark_data_smithi098_281896_object123347:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-21T10:32:21.780 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:31:15.745987+0000", 2024-09-21T10:32:21.780 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.895459227000003, 2024-09-21T10:32:21.780 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.155598134, 2024-09-21T10:32:21.780 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:21.781 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:21.781 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:21.781 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-21T10:32:21.781 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.46322", 2024-09-21T10:32:21.781 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:21.781 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 248372 2024-09-21T10:32:21.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.781 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:21.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:21.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:15.745987+0000", 2024-09-21T10:32:21.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:21.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:15.745987+0000", 2024-09-21T10:32:21.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:21.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:21.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:15.745988+0000", 2024-09-21T10:32:21.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.533e-06 2024-09-21T10:32:21.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:21.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:15.745993+0000", 2024-09-21T10:32:21.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6140000000000002e-06 2024-09-21T10:32:21.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:21.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:15.746007+0000", 2024-09-21T10:32:21.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869999999999999e-05 2024-09-21T10:32:21.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:21.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:15.746014+0000", 2024-09-21T10:32:21.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1740000000000003e-06 2024-09-21T10:32:21.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:21.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:15.771547+0000", 2024-09-21T10:32:21.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.025532664 2024-09-21T10:32:21.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:21.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:15.771686+0000", 2024-09-21T10:32:21.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000139603 2024-09-21T10:32:21.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:32:21.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:15.771843+0000", 2024-09-21T10:32:21.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000156735 2024-09-21T10:32:21.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:15.786791+0000", 2024-09-21T10:32:21.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014948088 2024-09-21T10:32:21.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-21T10:32:21.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:15.793598+0000", 2024-09-21T10:32:21.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0068070730000000003 2024-09-21T10:32:21.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:21.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:15.898673+0000", 2024-09-21T10:32:21.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.105074374 2024-09-21T10:32:21.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:21.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:15.898744+0000", 2024-09-21T10:32:21.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1246000000000002e-05 2024-09-21T10:32:21.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:21.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:21.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:21.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:15.901585+0000", 2024-09-21T10:32:21.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028411600000000001 2024-09-21T10:32:21.787 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.787 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:21.787 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.787 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:21.787 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:21.787 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:32:21.899 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:21 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a[97576]: 2024-09-21T10:32:21.615+0000 7f65561f9640 -1 mon.a@0(leader).osd e396 definitely_dead 0 2024-09-21T10:32:22.341 INFO:teuthology.orchestra.run.smithi098.stderr:marked out osd.1. 2024-09-21T10:32:22.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:22 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3490850949' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-21T10:32:22.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:22 smithi098 ceph-mon[97580]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-21T10:32:22.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:22.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:22.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:22 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3490850949' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-21T10:32:22.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:22 smithi098 ceph-mon[100914]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-21T10:32:22.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:22.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:22.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:22 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3490850949' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-21T10:32:22.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:22 smithi196 ceph-mon[84784]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-21T10:32:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:23.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:23 smithi098 ceph-mon[97580]: pgmap v1389: 83 pgs: 83 active+clean; 4.3 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:23.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:23 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3490850949' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-21T10:32:23.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:23 smithi098 ceph-mon[97580]: osdmap e397: 8 total, 8 up, 7 in 2024-09-21T10:32:23.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:23 smithi098 ceph-mon[100914]: pgmap v1389: 83 pgs: 83 active+clean; 4.3 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:23.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:23 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3490850949' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-21T10:32:23.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:23 smithi098 ceph-mon[100914]: osdmap e397: 8 total, 8 up, 7 in 2024-09-21T10:32:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:23 smithi196 ceph-mon[84784]: pgmap v1389: 83 pgs: 83 active+clean; 4.3 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail 2024-09-21T10:32:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:23 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3490850949' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-21T10:32:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:23 smithi196 ceph-mon[84784]: osdmap e397: 8 total, 8 up, 7 in 2024-09-21T10:32:23.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:32:24.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:24 smithi098 ceph-mon[97580]: osdmap e398: 8 total, 8 up, 7 in 2024-09-21T10:32:24.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:24 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:32:24.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:24 smithi098 ceph-mon[100914]: osdmap e398: 8 total, 8 up, 7 in 2024-09-21T10:32:24.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:24 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:32:24.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:24 smithi196 ceph-mon[84784]: osdmap e398: 8 total, 8 up, 7 in 2024-09-21T10:32:24.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:24 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:32:24.674 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:25.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:25 smithi098 ceph-mon[100914]: pgmap v1392: 83 pgs: 21 peering, 62 active+clean; 4.4 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 11 KiB/s rd, 25 KiB/s wr, 6 op/s; 8.7 KiB/s, 1 objects/s recovering 2024-09-21T10:32:25.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:25 smithi098 ceph-mon[100914]: Health check failed: Reduced data availability: 3 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-09-21T10:32:25.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:25 smithi098 ceph-mon[100914]: osdmap e399: 8 total, 8 up, 7 in 2024-09-21T10:32:25.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:25 smithi098 ceph-mon[97580]: pgmap v1392: 83 pgs: 21 peering, 62 active+clean; 4.4 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 11 KiB/s rd, 25 KiB/s wr, 6 op/s; 8.7 KiB/s, 1 objects/s recovering 2024-09-21T10:32:25.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:25 smithi098 ceph-mon[97580]: Health check failed: Reduced data availability: 3 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-09-21T10:32:25.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:25 smithi098 ceph-mon[97580]: osdmap e399: 8 total, 8 up, 7 in 2024-09-21T10:32:25.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:25 smithi196 ceph-mon[84784]: pgmap v1392: 83 pgs: 21 peering, 62 active+clean; 4.4 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 11 KiB/s rd, 25 KiB/s wr, 6 op/s; 8.7 KiB/s, 1 objects/s recovering 2024-09-21T10:32:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:25 smithi196 ceph-mon[84784]: Health check failed: Reduced data availability: 3 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-09-21T10:32:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:25 smithi196 ceph-mon[84784]: osdmap e399: 8 total, 8 up, 7 in 2024-09-21T10:32:26.377 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-21T10:32:26.614 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:26 smithi196 ceph-mon[84784]: osdmap e400: 8 total, 8 up, 7 in 2024-09-21T10:32:26.615 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:26 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/108356094' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:32:26.615 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:26 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:32:26.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:26 smithi098 ceph-mon[97580]: osdmap e400: 8 total, 8 up, 7 in 2024-09-21T10:32:26.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:26 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/108356094' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:32:26.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:26 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:32:26.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:26 smithi098 ceph-mon[100914]: osdmap e400: 8 total, 8 up, 7 in 2024-09-21T10:32:26.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:26 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/108356094' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:32:26.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:26 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:32:27.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:27 smithi098 ceph-mon[97580]: pgmap v1395: 83 pgs: 21 peering, 62 active+clean; 4.4 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 17 KiB/s rd, 37 KiB/s wr, 10 op/s; 13 KiB/s, 2 objects/s recovering 2024-09-21T10:32:27.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:27 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:32:27.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:27 smithi098 ceph-mon[97580]: osdmap e401: 8 total, 8 up, 7 in 2024-09-21T10:32:27.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:27 smithi098 ceph-mon[97580]: 1.3 scrub starts 2024-09-21T10:32:27.598 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:27 smithi098 ceph-mon[97580]: 1.3 scrub ok 2024-09-21T10:32:27.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:27 smithi098 ceph-mon[100914]: pgmap v1395: 83 pgs: 21 peering, 62 active+clean; 4.4 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 17 KiB/s rd, 37 KiB/s wr, 10 op/s; 13 KiB/s, 2 objects/s recovering 2024-09-21T10:32:27.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:27 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:32:27.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:27 smithi098 ceph-mon[100914]: osdmap e401: 8 total, 8 up, 7 in 2024-09-21T10:32:27.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:27 smithi098 ceph-mon[100914]: 1.3 scrub starts 2024-09-21T10:32:27.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:27 smithi098 ceph-mon[100914]: 1.3 scrub ok 2024-09-21T10:32:27.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:27 smithi196 ceph-mon[84784]: pgmap v1395: 83 pgs: 21 peering, 62 active+clean; 4.4 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 17 KiB/s rd, 37 KiB/s wr, 10 op/s; 13 KiB/s, 2 objects/s recovering 2024-09-21T10:32:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:27 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:32:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:27 smithi196 ceph-mon[84784]: osdmap e401: 8 total, 8 up, 7 in 2024-09-21T10:32:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:27 smithi196 ceph-mon[84784]: 1.3 scrub starts 2024-09-21T10:32:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:27 smithi196 ceph-mon[84784]: 1.3 scrub ok 2024-09-21T10:32:28.344 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:32:28.344 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:32:28.344 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-21T10:32:28.344 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:32:29.051 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:32:29.293 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-21T10:32:29.293 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-09-21T10:32:29.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd in 1 2024-09-21T10:32:29.328 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:29.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:29 smithi098 ceph-mon[97580]: pgmap v1397: 83 pgs: 21 peering, 62 active+clean; 4.4 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T10:32:29.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:29 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:29.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:29 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:29.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:29 smithi098 ceph-mon[100914]: pgmap v1397: 83 pgs: 21 peering, 62 active+clean; 4.4 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T10:32:29.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:29 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:29.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:29 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:29 smithi196 ceph-mon[84784]: pgmap v1397: 83 pgs: 21 peering, 62 active+clean; 4.4 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T10:32:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:29 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:29 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:30.476 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:30 smithi098 ceph-mon[97580]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 7 pgs peering) 2024-09-21T10:32:30.476 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:30 smithi098 ceph-mon[97580]: Cluster is now healthy 2024-09-21T10:32:30.476 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:30 smithi098 ceph-mon[100914]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 7 pgs peering) 2024-09-21T10:32:30.476 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:30 smithi098 ceph-mon[100914]: Cluster is now healthy 2024-09-21T10:32:30.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:30 smithi196 ceph-mon[84784]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 7 pgs peering) 2024-09-21T10:32:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:30 smithi196 ceph-mon[84784]: Cluster is now healthy 2024-09-21T10:32:30.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:32:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:32:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:32:30.829 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:32:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:32:30] "GET /metrics HTTP/1.1" 200 34175 "" "Prometheus/2.51.0" 2024-09-21T10:32:31.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:31 smithi098 ceph-mon[97580]: pgmap v1398: 83 pgs: 83 active+clean; 4.3 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s, 7 objects/s recovering 2024-09-21T10:32:31.499 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:31 smithi098 ceph-mon[100914]: pgmap v1398: 83 pgs: 83 active+clean; 4.3 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s, 7 objects/s recovering 2024-09-21T10:32:31.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:31 smithi196 ceph-mon[84784]: pgmap v1398: 83 pgs: 83 active+clean; 4.3 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s, 7 objects/s recovering 2024-09-21T10:32:32.866 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:32:32.866 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-21T10:32:32.866 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-21T10:32:32.866 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-21T10:32:32.866 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:32:33.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:33 smithi098 ceph-mon[97580]: pgmap v1399: 83 pgs: 83 active+clean; 4.3 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s, 6 objects/s recovering 2024-09-21T10:32:33.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:33 smithi098 ceph-mon[100914]: pgmap v1399: 83 pgs: 83 active+clean; 4.3 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s, 6 objects/s recovering 2024-09-21T10:32:33.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:33 smithi196 ceph-mon[84784]: pgmap v1399: 83 pgs: 83 active+clean; 4.3 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s, 6 objects/s recovering 2024-09-21T10:32:33.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:32:34.011 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:35.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:35 smithi098 ceph-mon[97580]: pgmap v1400: 83 pgs: 83 active+clean; 4.3 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s, 5 objects/s recovering 2024-09-21T10:32:35.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:35 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:35.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:35 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:35.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:35 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:35.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:35 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:35.329 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:35 smithi098 ceph-mon[100914]: pgmap v1400: 83 pgs: 83 active+clean; 4.3 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s, 5 objects/s recovering 2024-09-21T10:32:35.329 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:35 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:35.329 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:35 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:35.329 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:35 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:35.329 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:35 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:35 smithi196 ceph-mon[84784]: pgmap v1400: 83 pgs: 83 active+clean; 4.3 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s, 5 objects/s recovering 2024-09-21T10:32:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:35 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:35 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:35 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:35 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:35.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:35 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a[97576]: 2024-09-21T10:32:35.841+0000 7f65561f9640 -1 mon.a@0(leader).osd e401 definitely_dead 0 2024-09-21T10:32:36.055 INFO:teuthology.orchestra.run.smithi098.stderr:marked in osd.1. 2024-09-21T10:32:36.215 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:36 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/862468445' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-21T10:32:36.216 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:36 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/862468445' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-21T10:32:36.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:36 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/862468445' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-21T10:32:37.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:37 smithi098 ceph-mon[97580]: pgmap v1401: 83 pgs: 83 active+clean; 4.3 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s, 4 objects/s recovering 2024-09-21T10:32:37.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:37 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/862468445' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-21T10:32:37.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:37 smithi098 ceph-mon[97580]: osdmap e402: 8 total, 8 up, 8 in 2024-09-21T10:32:37.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:37 smithi098 ceph-mon[100914]: pgmap v1401: 83 pgs: 83 active+clean; 4.3 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s, 4 objects/s recovering 2024-09-21T10:32:37.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:37 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/862468445' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-21T10:32:37.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:37 smithi098 ceph-mon[100914]: osdmap e402: 8 total, 8 up, 8 in 2024-09-21T10:32:37.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:37 smithi196 ceph-mon[84784]: pgmap v1401: 83 pgs: 83 active+clean; 4.3 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s, 4 objects/s recovering 2024-09-21T10:32:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:37 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/862468445' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-21T10:32:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:37 smithi196 ceph-mon[84784]: osdmap e402: 8 total, 8 up, 8 in 2024-09-21T10:32:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:38 smithi098 ceph-mon[97580]: osdmap e403: 8 total, 8 up, 8 in 2024-09-21T10:32:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:38 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:38.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:38 smithi098 ceph-mon[97580]: osdmap e404: 8 total, 8 up, 8 in 2024-09-21T10:32:38.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:38 smithi098 ceph-mon[100914]: osdmap e403: 8 total, 8 up, 8 in 2024-09-21T10:32:38.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:38 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:38.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:38 smithi098 ceph-mon[100914]: osdmap e404: 8 total, 8 up, 8 in 2024-09-21T10:32:38.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:38 smithi196 ceph-mon[84784]: osdmap e403: 8 total, 8 up, 8 in 2024-09-21T10:32:38.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:38 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:38.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:38 smithi196 ceph-mon[84784]: osdmap e404: 8 total, 8 up, 8 in 2024-09-21T10:32:38.557 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-09-21T10:32:38.557 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-21T10:32:38.557 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pgp_num 11 2024-09-21T10:32:38.785 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-21T10:32:38.785 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-21T10:32:38.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-21T10:32:38.785 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-21T10:32:38.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.785 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:160578 10.3 e371/361 10:c3ba3c86:::benchmark_data_smithi098_281896_object35553:head v 371'10111, mlcod=0'0)", 2024-09-21T10:32:38.785 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:09.321026+0000", 2024-09-21T10:32:38.786 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 209.460088273, 2024-09-21T10:32:38.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.123462081, 2024-09-21T10:32:38.786 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:38.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:38.786 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:38.786 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:38.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:38.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.321026+0000", 2024-09-21T10:32:38.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:38.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.321026+0000", 2024-09-21T10:32:38.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:38.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.321027+0000", 2024-09-21T10:32:38.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.277e-06 2024-09-21T10:32:38.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:38.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.321035+0000", 2024-09-21T10:32:38.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2530000000000004e-06 2024-09-21T10:32:38.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:38.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.321047+0000", 2024-09-21T10:32:38.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1411e-05 2024-09-21T10:32:38.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:38.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.321054+0000", 2024-09-21T10:32:38.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7950000000000008e-06 2024-09-21T10:32:38.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:38.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.370908+0000", 2024-09-21T10:32:38.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.049853498000000003 2024-09-21T10:32:38.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:38.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.370938+0000", 2024-09-21T10:32:38.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0080999999999999e-05 2024-09-21T10:32:38.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:38.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.444303+0000", 2024-09-21T10:32:38.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.073365185999999999 2024-09-21T10:32:38.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:38.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:09.444488+0000", 2024-09-21T10:32:38.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018458 2024-09-21T10:32:38.790 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.790 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:38.790 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.790 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:181057 10.5 e376/361 10:a9d1f123:::benchmark_data_smithi098_281896_object56032:head v 376'11080, mlcod=0'0)", 2024-09-21T10:32:38.790 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:39.098131+0000", 2024-09-21T10:32:38.790 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 179.68298277, 2024-09-21T10:32:38.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.14523421, 2024-09-21T10:32:38.790 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:38.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:38.791 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:38.791 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:38.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:38.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.098131+0000", 2024-09-21T10:32:38.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:38.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.098131+0000", 2024-09-21T10:32:38.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:38.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.098133+0000", 2024-09-21T10:32:38.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4950000000000001e-06 2024-09-21T10:32:38.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:38.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.098142+0000", 2024-09-21T10:32:38.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8740000000000001e-06 2024-09-21T10:32:38.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:38.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.098163+0000", 2024-09-21T10:32:38.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1416999999999998e-05 2024-09-21T10:32:38.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:38.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.098172+0000", 2024-09-21T10:32:38.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0659999999999993e-06 2024-09-21T10:32:38.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:38.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.117831+0000", 2024-09-21T10:32:38.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019659224999999999 2024-09-21T10:32:38.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:38.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.117890+0000", 2024-09-21T10:32:38.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.889e-05 2024-09-21T10:32:38.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:38.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.243295+0000", 2024-09-21T10:32:38.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.125404828 2024-09-21T10:32:38.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:38.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.243365+0000", 2024-09-21T10:32:38.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0414999999999996e-05 2024-09-21T10:32:38.794 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.794 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:38.794 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.795 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:181064 10.5 e376/361 10:a76245a6:::benchmark_data_smithi098_281896_object56039:head v 376'11081, mlcod=0'0)", 2024-09-21T10:32:38.795 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:39.101786+0000", 2024-09-21T10:32:38.795 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 179.67932829099999, 2024-09-21T10:32:38.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.1417136, 2024-09-21T10:32:38.795 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:38.795 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:38.795 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:38.795 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:38.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:38.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.101786+0000", 2024-09-21T10:32:38.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:38.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.101786+0000", 2024-09-21T10:32:38.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:38.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.101787+0000", 2024-09-21T10:32:38.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3820000000000001e-06 2024-09-21T10:32:38.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:38.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.101794+0000", 2024-09-21T10:32:38.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9949999999999999e-06 2024-09-21T10:32:38.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:38.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.101811+0000", 2024-09-21T10:32:38.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7376000000000001e-05 2024-09-21T10:32:38.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:38.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.101820+0000", 2024-09-21T10:32:38.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2549999999999997e-06 2024-09-21T10:32:38.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:38.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.131655+0000", 2024-09-21T10:32:38.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029835285999999999 2024-09-21T10:32:38.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:38.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.131680+0000", 2024-09-21T10:32:38.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5239999999999999e-05 2024-09-21T10:32:38.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:38.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.243371+0000", 2024-09-21T10:32:38.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.111690861 2024-09-21T10:32:38.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:38.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.243499+0000", 2024-09-21T10:32:38.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000128205 2024-09-21T10:32:38.799 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.799 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:38.799 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.799 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:181068 10.5 e376/361 10:afc370d2:::benchmark_data_smithi098_281896_object56043:head v 376'11082, mlcod=0'0)", 2024-09-21T10:32:38.799 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:39.107057+0000", 2024-09-21T10:32:38.799 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 179.67405730999999, 2024-09-21T10:32:38.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.136583387, 2024-09-21T10:32:38.800 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:38.800 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:38.801 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:38.801 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:38.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:38.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.107057+0000", 2024-09-21T10:32:38.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:38.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.107057+0000", 2024-09-21T10:32:38.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:38.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.107058+0000", 2024-09-21T10:32:38.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.187e-06 2024-09-21T10:32:38.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:38.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.107065+0000", 2024-09-21T10:32:38.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3849999999999996e-06 2024-09-21T10:32:38.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:38.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.107085+0000", 2024-09-21T10:32:38.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9743000000000001e-05 2024-09-21T10:32:38.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:38.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.107093+0000", 2024-09-21T10:32:38.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0010000000000001e-06 2024-09-21T10:32:38.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:38.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.142738+0000", 2024-09-21T10:32:38.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035645385000000002 2024-09-21T10:32:38.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:38.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.142774+0000", 2024-09-21T10:32:38.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5277999999999997e-05 2024-09-21T10:32:38.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:38.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.243503+0000", 2024-09-21T10:32:38.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.100729769 2024-09-21T10:32:38.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:38.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.243640+0000", 2024-09-21T10:32:38.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000136639 2024-09-21T10:32:38.805 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.805 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:38.805 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.805 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:181070 10.5 e376/361 10:ac6ed289:::benchmark_data_smithi098_281896_object56045:head v 376'11083, mlcod=0'0)", 2024-09-21T10:32:38.805 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:39.112609+0000", 2024-09-21T10:32:38.805 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 179.66850512299999, 2024-09-21T10:32:38.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.13115652899999999, 2024-09-21T10:32:38.805 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:38.805 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:38.805 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:38.805 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:38.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:38.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.112609+0000", 2024-09-21T10:32:38.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:38.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.112609+0000", 2024-09-21T10:32:38.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:38.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.112610+0000", 2024-09-21T10:32:38.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0500000000000002e-07 2024-09-21T10:32:38.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:38.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.112616+0000", 2024-09-21T10:32:38.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.691e-06 2024-09-21T10:32:38.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:38.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.112631+0000", 2024-09-21T10:32:38.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4739000000000001e-05 2024-09-21T10:32:38.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:38.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.112638+0000", 2024-09-21T10:32:38.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4880000000000004e-06 2024-09-21T10:32:38.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:38.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.156570+0000", 2024-09-21T10:32:38.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.043932643 2024-09-21T10:32:38.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:38.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.156594+0000", 2024-09-21T10:32:38.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3234000000000002e-05 2024-09-21T10:32:38.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:38.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.243646+0000", 2024-09-21T10:32:38.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.087052697999999998 2024-09-21T10:32:38.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:38.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.243765+0000", 2024-09-21T10:32:38.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011913099999999999 2024-09-21T10:32:38.809 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.809 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:38.809 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.809 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:181079 10.5 e376/361 10:ab19edef:::benchmark_data_smithi098_281896_object56054:head v 376'11084, mlcod=0'0)", 2024-09-21T10:32:38.809 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:39.118304+0000", 2024-09-21T10:32:38.809 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 179.66280999400001, 2024-09-21T10:32:38.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.12558893400000001, 2024-09-21T10:32:38.810 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:38.810 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:38.810 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:38.810 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:38.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:38.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.118304+0000", 2024-09-21T10:32:38.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:38.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.118304+0000", 2024-09-21T10:32:38.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:38.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.118305+0000", 2024-09-21T10:32:38.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4600000000000003e-07 2024-09-21T10:32:38.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:38.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.118310+0000", 2024-09-21T10:32:38.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2140000000000004e-06 2024-09-21T10:32:38.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:38.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.118322+0000", 2024-09-21T10:32:38.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1807999999999999e-05 2024-09-21T10:32:38.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:38.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.118328+0000", 2024-09-21T10:32:38.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4760000000000003e-06 2024-09-21T10:32:38.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:38.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.167314+0000", 2024-09-21T10:32:38.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.048985122999999998 2024-09-21T10:32:38.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:38.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.167362+0000", 2024-09-21T10:32:38.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8618000000000001e-05 2024-09-21T10:32:38.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:38.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.243769+0000", 2024-09-21T10:32:38.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.076407089999999997 2024-09-21T10:32:38.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:38.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:39.243893+0000", 2024-09-21T10:32:38.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123659 2024-09-21T10:32:38.813 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.814 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:38.814 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.814 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:194344 10.7 e379/361 10:ed426a0a:::benchmark_data_smithi098_281896_object69319:head v 379'11944, mlcod=379'11942)", 2024-09-21T10:32:38.814 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:58.404471+0000", 2024-09-21T10:32:38.814 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 160.37664302799999, 2024-09-21T10:32:38.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.15613059200000001, 2024-09-21T10:32:38.814 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:38.814 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:38.814 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:38.814 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:38.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:38.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.404471+0000", 2024-09-21T10:32:38.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:38.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.404471+0000", 2024-09-21T10:32:38.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:38.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.404472+0000", 2024-09-21T10:32:38.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2950000000000001e-06 2024-09-21T10:32:38.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:38.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.404480+0000", 2024-09-21T10:32:38.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1259999999999998e-06 2024-09-21T10:32:38.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:38.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.404496+0000", 2024-09-21T10:32:38.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5877999999999999e-05 2024-09-21T10:32:38.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:38.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.404504+0000", 2024-09-21T10:32:38.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8599999999999993e-06 2024-09-21T10:32:38.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:38.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.404572+0000", 2024-09-21T10:32:38.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7656000000000004e-05 2024-09-21T10:32:38.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:38.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.404593+0000", 2024-09-21T10:32:38.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0805000000000002e-05 2024-09-21T10:32:38.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:38.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.560550+0000", 2024-09-21T10:32:38.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.155957023 2024-09-21T10:32:38.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:38.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.560602+0000", 2024-09-21T10:32:38.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1949e-05 2024-09-21T10:32:38.818 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.818 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:38.818 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.818 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:194335 10.3 e379/361 10:c3d7206b:::benchmark_data_smithi098_281896_object69310:head v 379'12259, mlcod=0'0)", 2024-09-21T10:32:38.818 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:58.407507+0000", 2024-09-21T10:32:38.819 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 160.37360717000001, 2024-09-21T10:32:38.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.175384348, 2024-09-21T10:32:38.819 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:38.819 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:38.819 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:38.819 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:38.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:38.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.407507+0000", 2024-09-21T10:32:38.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:38.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.407507+0000", 2024-09-21T10:32:38.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:38.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.407508+0000", 2024-09-21T10:32:38.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2270000000000001e-06 2024-09-21T10:32:38.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:38.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.407516+0000", 2024-09-21T10:32:38.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4249999999999998e-06 2024-09-21T10:32:38.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:38.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.549487+0000", 2024-09-21T10:32:38.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.14197022400000001 2024-09-21T10:32:38.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:38.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.549499+0000", 2024-09-21T10:32:38.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2595999999999999e-05 2024-09-21T10:32:38.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:38.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.549566+0000", 2024-09-21T10:32:38.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7148000000000002e-05 2024-09-21T10:32:38.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:38.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.549592+0000", 2024-09-21T10:32:38.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5420000000000001e-05 2024-09-21T10:32:38.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:38.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.582838+0000", 2024-09-21T10:32:38.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.033246445999999999 2024-09-21T10:32:38.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:38.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.582891+0000", 2024-09-21T10:32:38.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2862000000000003e-05 2024-09-21T10:32:38.823 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.823 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:38.823 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.823 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:194324 10.4 e379/361 10:255193c3:::benchmark_data_smithi098_281896_object69299:head v 379'11982, mlcod=353'122)", 2024-09-21T10:32:38.823 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:29:58.411548+0000", 2024-09-21T10:32:38.823 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 160.36956616699999, 2024-09-21T10:32:38.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.145352131, 2024-09-21T10:32:38.823 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:38.824 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:38.824 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:38.824 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:38.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:38.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.411548+0000", 2024-09-21T10:32:38.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:38.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.411548+0000", 2024-09-21T10:32:38.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:38.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.411549+0000", 2024-09-21T10:32:38.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4449999999999999e-06 2024-09-21T10:32:38.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:38.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.411558+0000", 2024-09-21T10:32:38.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3690000000000007e-06 2024-09-21T10:32:38.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:38.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.549483+0000", 2024-09-21T10:32:38.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.13792543400000001 2024-09-21T10:32:38.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:38.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.549499+0000", 2024-09-21T10:32:38.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6203999999999999e-05 2024-09-21T10:32:38.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:38.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.549562+0000", 2024-09-21T10:32:38.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2445000000000003e-05 2024-09-21T10:32:38.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:38.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.549591+0000", 2024-09-21T10:32:38.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.904e-05 2024-09-21T10:32:38.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:38.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.556862+0000", 2024-09-21T10:32:38.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0072714930000000004 2024-09-21T10:32:38.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:38.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:29:58.556900+0000", 2024-09-21T10:32:38.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7700999999999997e-05 2024-09-21T10:32:38.827 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.827 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:38.828 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.828 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:220772 10.3 e382/361 10:cb2d97eb:::benchmark_data_smithi098_281896_object95747:head v 382'13931, mlcod=0'0)", 2024-09-21T10:32:38.828 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:30:36.942179+0000", 2024-09-21T10:32:38.828 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 121.83893487500001, 2024-09-21T10:32:38.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.128700076, 2024-09-21T10:32:38.828 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:38.828 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:38.828 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:38.828 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:38.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:38.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:36.942179+0000", 2024-09-21T10:32:38.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:38.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:36.942179+0000", 2024-09-21T10:32:38.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:38.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:36.942180+0000", 2024-09-21T10:32:38.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2759999999999999e-06 2024-09-21T10:32:38.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:38.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:36.942188+0000", 2024-09-21T10:32:38.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2250000000000003e-06 2024-09-21T10:32:38.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:38.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:36.942204+0000", 2024-09-21T10:32:38.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6231000000000001e-05 2024-09-21T10:32:38.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:38.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:36.942212+0000", 2024-09-21T10:32:38.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2660000000000001e-06 2024-09-21T10:32:38.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:38.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:36.948025+0000", 2024-09-21T10:32:38.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005812528 2024-09-21T10:32:38.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:38.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:36.948069+0000", 2024-09-21T10:32:38.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3918999999999997e-05 2024-09-21T10:32:38.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:38.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:37.066967+0000", 2024-09-21T10:32:38.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11889801699999999 2024-09-21T10:32:38.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:38.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:37.070879+0000", 2024-09-21T10:32:38.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003912614 2024-09-21T10:32:38.832 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.832 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:38.832 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.832 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:221610 10.3 e382/361 10:ca92ef80:::benchmark_data_smithi098_281896_object96585:head v 382'13986, mlcod=0'0)", 2024-09-21T10:32:38.832 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:30:38.157009+0000", 2024-09-21T10:32:38.832 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 120.624104906, 2024-09-21T10:32:38.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.151646803, 2024-09-21T10:32:38.833 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:38.833 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:38.833 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:38.833 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:38.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:38.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.157009+0000", 2024-09-21T10:32:38.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:38.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.157009+0000", 2024-09-21T10:32:38.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:38.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.157010+0000", 2024-09-21T10:32:38.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4249999999999999e-06 2024-09-21T10:32:38.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:38.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.157020+0000", 2024-09-21T10:32:38.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3500000000000003e-06 2024-09-21T10:32:38.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:38.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.157034+0000", 2024-09-21T10:32:38.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3978000000000001e-05 2024-09-21T10:32:38.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:38.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.157042+0000", 2024-09-21T10:32:38.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5900000000000008e-06 2024-09-21T10:32:38.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:38.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.157140+0000", 2024-09-21T10:32:38.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7518999999999999e-05 2024-09-21T10:32:38.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:38.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.157169+0000", 2024-09-21T10:32:38.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8640999999999999e-05 2024-09-21T10:32:38.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:38.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.308597+0000", 2024-09-21T10:32:38.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.15142810300000001 2024-09-21T10:32:38.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:38.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.308656+0000", 2024-09-21T10:32:38.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9197000000000003e-05 2024-09-21T10:32:38.837 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.837 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:38.837 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.837 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:221611 10.3 e382/361 10:c189741a:::benchmark_data_smithi098_281896_object96586:head v 382'13987, mlcod=0'0)", 2024-09-21T10:32:38.837 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:30:38.160869+0000", 2024-09-21T10:32:38.837 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 120.620245293, 2024-09-21T10:32:38.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.14781987099999999, 2024-09-21T10:32:38.837 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:38.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:38.837 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:38.837 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:38.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:38.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.160869+0000", 2024-09-21T10:32:38.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:38.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.160869+0000", 2024-09-21T10:32:38.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:38.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.160869+0000", 2024-09-21T10:32:38.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.96e-07 2024-09-21T10:32:38.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:38.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.160873+0000", 2024-09-21T10:32:38.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6509999999999999e-06 2024-09-21T10:32:38.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:38.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.160879+0000", 2024-09-21T10:32:38.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.015e-06 2024-09-21T10:32:38.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:38.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.160884+0000", 2024-09-21T10:32:38.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3639999999999998e-06 2024-09-21T10:32:38.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:38.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.171322+0000", 2024-09-21T10:32:38.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010438060000000001 2024-09-21T10:32:38.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:38.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.171346+0000", 2024-09-21T10:32:38.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3864999999999999e-05 2024-09-21T10:32:38.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:38.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.308660+0000", 2024-09-21T10:32:38.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.13731370700000001 2024-09-21T10:32:38.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:38.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.308689+0000", 2024-09-21T10:32:38.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8912999999999999e-05 2024-09-21T10:32:38.841 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.841 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:38.841 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.841 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:221617 10.7 e382/361 10:ea56006e:::benchmark_data_smithi098_281896_object96592:head v 382'13657, mlcod=382'13655)", 2024-09-21T10:32:38.842 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:30:38.168199+0000", 2024-09-21T10:32:38.842 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 120.61291500900001, 2024-09-21T10:32:38.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.14048289999999999, 2024-09-21T10:32:38.842 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:38.842 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:38.842 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:38.842 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:38.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:38.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.168199+0000", 2024-09-21T10:32:38.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:38.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.168199+0000", 2024-09-21T10:32:38.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:38.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.168199+0000", 2024-09-21T10:32:38.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7e-07 2024-09-21T10:32:38.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:38.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.168203+0000", 2024-09-21T10:32:38.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4000000000000001e-06 2024-09-21T10:32:38.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:38.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.168209+0000", 2024-09-21T10:32:38.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8769999999999998e-06 2024-09-21T10:32:38.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:38.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.168213+0000", 2024-09-21T10:32:38.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0419999999999996e-06 2024-09-21T10:32:38.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:38.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.171255+0000", 2024-09-21T10:32:38.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030426139999999999 2024-09-21T10:32:38.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:38.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.171276+0000", 2024-09-21T10:32:38.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1199000000000002e-05 2024-09-21T10:32:38.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:38.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.308630+0000", 2024-09-21T10:32:38.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.137353054 2024-09-21T10:32:38.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:38.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.308682+0000", 2024-09-21T10:32:38.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2343999999999999e-05 2024-09-21T10:32:38.847 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.847 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:38.847 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.847 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:221621 10.5 e382/361 10:a9f93a44:::benchmark_data_smithi098_281896_object96596:head v 382'13615, mlcod=0'0)", 2024-09-21T10:32:38.847 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:30:38.174585+0000", 2024-09-21T10:32:38.847 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 120.60652936, 2024-09-21T10:32:38.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.134110429, 2024-09-21T10:32:38.847 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:38.847 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:38.847 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:38.847 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:38.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:38.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.174585+0000", 2024-09-21T10:32:38.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:38.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.174585+0000", 2024-09-21T10:32:38.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:38.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.174586+0000", 2024-09-21T10:32:38.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.33e-07 2024-09-21T10:32:38.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:38.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.174592+0000", 2024-09-21T10:32:38.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1680000000000001e-06 2024-09-21T10:32:38.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:38.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.174604+0000", 2024-09-21T10:32:38.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2602000000000001e-05 2024-09-21T10:32:38.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:38.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.174611+0000", 2024-09-21T10:32:38.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0369999999999997e-06 2024-09-21T10:32:38.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:38.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.174681+0000", 2024-09-21T10:32:38.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9727000000000005e-05 2024-09-21T10:32:38.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:38.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.174703+0000", 2024-09-21T10:32:38.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1690000000000001e-05 2024-09-21T10:32:38.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:38.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.308626+0000", 2024-09-21T10:32:38.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.13392345999999999 2024-09-21T10:32:38.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:38.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.308695+0000", 2024-09-21T10:32:38.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8812000000000005e-05 2024-09-21T10:32:38.851 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.851 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:38.851 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.852 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:221607 10.4 e382/361 10:2a878f99:::benchmark_data_smithi098_281896_object96582:head v 382'13675, mlcod=353'122)", 2024-09-21T10:32:38.852 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:30:38.180130+0000", 2024-09-21T10:32:38.852 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 120.600984428, 2024-09-21T10:32:38.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.17173469699999999, 2024-09-21T10:32:38.852 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:38.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:38.852 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:38.852 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:38.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.852 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:38.852 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.180130+0000", 2024-09-21T10:32:38.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:38.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.180130+0000", 2024-09-21T10:32:38.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:38.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.180131+0000", 2024-09-21T10:32:38.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.13e-06 2024-09-21T10:32:38.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:38.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.180139+0000", 2024-09-21T10:32:38.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8380000000000002e-06 2024-09-21T10:32:38.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:38.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.180151+0000", 2024-09-21T10:32:38.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2840000000000001e-05 2024-09-21T10:32:38.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:38.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.180159+0000", 2024-09-21T10:32:38.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6690000000000006e-06 2024-09-21T10:32:38.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:38.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.182153+0000", 2024-09-21T10:32:38.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0019941070000000002 2024-09-21T10:32:38.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:38.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.182176+0000", 2024-09-21T10:32:38.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2901999999999999e-05 2024-09-21T10:32:38.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:38.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.351772+0000", 2024-09-21T10:32:38.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.16959639900000001 2024-09-21T10:32:38.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.856 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:38.856 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.351864+0000", 2024-09-21T10:32:38.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1811999999999994e-05 2024-09-21T10:32:38.856 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.856 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:38.856 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.856 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:221628 10.3 e382/361 10:c0e468e3:::benchmark_data_smithi098_281896_object96603:head v 382'13988, mlcod=0'0)", 2024-09-21T10:32:38.856 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:30:38.189134+0000", 2024-09-21T10:32:38.856 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 120.591980402, 2024-09-21T10:32:38.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.15721009499999999, 2024-09-21T10:32:38.856 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:38.856 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:38.857 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:38.857 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:38.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:38.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.189134+0000", 2024-09-21T10:32:38.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:38.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.189134+0000", 2024-09-21T10:32:38.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:38.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.189135+0000", 2024-09-21T10:32:38.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3239999999999999e-06 2024-09-21T10:32:38.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:38.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.189144+0000", 2024-09-21T10:32:38.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0950000000000002e-06 2024-09-21T10:32:38.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:38.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.308587+0000", 2024-09-21T10:32:38.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11944281299999999 2024-09-21T10:32:38.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:38.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.308609+0000", 2024-09-21T10:32:38.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1821999999999999e-05 2024-09-21T10:32:38.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:38.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.308716+0000", 2024-09-21T10:32:38.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107734 2024-09-21T10:32:38.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:38.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.308741+0000", 2024-09-21T10:32:38.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4939e-05 2024-09-21T10:32:38.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:38.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.346265+0000", 2024-09-21T10:32:38.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037523954999999998 2024-09-21T10:32:38.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:38.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.346344+0000", 2024-09-21T10:32:38.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8412999999999996e-05 2024-09-21T10:32:38.860 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.861 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:38.861 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.861 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:221631 10.7 e382/361 10:ea47006a:::benchmark_data_smithi098_281896_object96606:head v 382'13658, mlcod=382'13655)", 2024-09-21T10:32:38.861 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:30:38.190274+0000", 2024-09-21T10:32:38.861 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 120.590840078, 2024-09-21T10:32:38.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.13146644699999999, 2024-09-21T10:32:38.861 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:38.861 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:38.861 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:38.861 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:38.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:38.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.190274+0000", 2024-09-21T10:32:38.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:38.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.190274+0000", 2024-09-21T10:32:38.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:38.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.190275+0000", 2024-09-21T10:32:38.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.063e-06 2024-09-21T10:32:38.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:38.863 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.190283+0000", 2024-09-21T10:32:38.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2940000000000006e-06 2024-09-21T10:32:38.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:38.863 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.308608+0000", 2024-09-21T10:32:38.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.11832447 2024-09-21T10:32:38.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:38.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.308615+0000", 2024-09-21T10:32:38.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4669999999999998e-06 2024-09-21T10:32:38.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:38.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.308711+0000", 2024-09-21T10:32:38.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6069999999999993e-05 2024-09-21T10:32:38.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:38.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.308734+0000", 2024-09-21T10:32:38.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.249e-05 2024-09-21T10:32:38.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:38.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.321695+0000", 2024-09-21T10:32:38.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012961145 2024-09-21T10:32:38.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:38.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:30:38.321740+0000", 2024-09-21T10:32:38.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5448000000000003e-05 2024-09-21T10:32:38.865 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.865 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:38.865 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.866 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:239309 10.3 e383/361 10:c07f738a:::benchmark_data_smithi098_281896_object114284:head v 383'15035, mlcod=0'0)", 2024-09-21T10:32:38.866 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:31:02.750023+0000", 2024-09-21T10:32:38.866 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 96.031090480000003, 2024-09-21T10:32:38.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.12534630299999999, 2024-09-21T10:32:38.866 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:38.866 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:38.866 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:38.866 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:38.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.866 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:38.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:02.750023+0000", 2024-09-21T10:32:38.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:38.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:02.750023+0000", 2024-09-21T10:32:38.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:38.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:02.750025+0000", 2024-09-21T10:32:38.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.203e-06 2024-09-21T10:32:38.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:38.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:02.750032+0000", 2024-09-21T10:32:38.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.08e-06 2024-09-21T10:32:38.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:38.868 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:02.750045+0000", 2024-09-21T10:32:38.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3220999999999999e-05 2024-09-21T10:32:38.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:38.868 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:02.750051+0000", 2024-09-21T10:32:38.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4489999999999996e-06 2024-09-21T10:32:38.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:38.869 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:02.772031+0000", 2024-09-21T10:32:38.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.021979736 2024-09-21T10:32:38.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.869 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:38.869 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:02.772056+0000", 2024-09-21T10:32:38.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5069999999999999e-05 2024-09-21T10:32:38.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.869 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:38.869 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:02.875317+0000", 2024-09-21T10:32:38.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.103260702 2024-09-21T10:32:38.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:38.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:02.875370+0000", 2024-09-21T10:32:38.870 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2842e-05 2024-09-21T10:32:38.870 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.870 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:38.870 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.870 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:239314 10.3 e383/361 10:c707010a:::benchmark_data_smithi098_281896_object114289:head v 383'15036, mlcod=0'0)", 2024-09-21T10:32:38.870 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:31:02.752708+0000", 2024-09-21T10:32:38.871 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 96.028405613999993, 2024-09-21T10:32:38.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.122693283, 2024-09-21T10:32:38.871 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:38.871 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:38.871 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:38.871 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:38.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:38.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:02.752708+0000", 2024-09-21T10:32:38.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:38.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:02.752708+0000", 2024-09-21T10:32:38.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:38.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:02.752709+0000", 2024-09-21T10:32:38.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5799999999999998e-07 2024-09-21T10:32:38.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:38.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:02.752716+0000", 2024-09-21T10:32:38.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8260000000000004e-06 2024-09-21T10:32:38.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:38.873 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:02.752724+0000", 2024-09-21T10:32:38.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3000000000000002e-06 2024-09-21T10:32:38.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:38.873 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:02.752731+0000", 2024-09-21T10:32:38.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6839999999999999e-06 2024-09-21T10:32:38.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:38.873 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:02.784893+0000", 2024-09-21T10:32:38.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.032162063999999997 2024-09-21T10:32:38.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:38.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:02.784916+0000", 2024-09-21T10:32:38.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2787999999999999e-05 2024-09-21T10:32:38.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:38.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:02.875377+0000", 2024-09-21T10:32:38.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.090460755000000004 2024-09-21T10:32:38.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:38.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:02.875402+0000", 2024-09-21T10:32:38.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4907999999999999e-05 2024-09-21T10:32:38.875 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.875 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:38.875 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.875 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.46322.0:246054 10.3 e384/361 10:c7b20f48:::benchmark_data_smithi098_281896_object121029:head v 384'15461, mlcod=0'0)", 2024-09-21T10:32:38.875 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-21T10:31:12.481745+0000", 2024-09-21T10:32:38.875 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 86.299368998999995, 2024-09-21T10:32:38.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.12796384899999999, 2024-09-21T10:32:38.875 INFO:teuthology.orchestra.run.smithi098.stdout: "continuous": false, 2024-09-21T10:32:38.875 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-21T10:32:38.875 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:38.875 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-21T10:32:38.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-21T10:32:38.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.481745+0000", 2024-09-21T10:32:38.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-21T10:32:38.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.481745+0000", 2024-09-21T10:32:38.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-21T10:32:38.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-21T10:32:38.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.481746+0000", 2024-09-21T10:32:38.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.13e-06 2024-09-21T10:32:38.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.877 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-21T10:32:38.877 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.481753+0000", 2024-09-21T10:32:38.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0729999999999999e-06 2024-09-21T10:32:38.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.877 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-21T10:32:38.877 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.481875+0000", 2024-09-21T10:32:38.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012163900000000001 2024-09-21T10:32:38.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.877 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-21T10:32:38.877 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.481884+0000", 2024-09-21T10:32:38.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4590000000000004e-06 2024-09-21T10:32:38.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-21T10:32:38.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.481988+0000", 2024-09-21T10:32:38.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000103572 2024-09-21T10:32:38.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-21T10:32:38.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.482014+0000", 2024-09-21T10:32:38.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6180999999999999e-05 2024-09-21T10:32:38.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-21T10:32:38.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.609676+0000", 2024-09-21T10:32:38.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.12766175199999999 2024-09-21T10:32:38.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-21T10:32:38.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-21T10:32:38.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-21T10:32:38.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-21T10:31:12.609709+0000", 2024-09-21T10:32:38.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3043000000000002e-05 2024-09-21T10:32:38.879 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.879 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:38.879 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.879 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-21T10:32:38.879 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-21T10:32:38.879 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-21T10:32:39.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:39 smithi098 ceph-mon[97580]: pgmap v1404: 83 pgs: 6 peering, 77 active+clean; 4.3 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail 2024-09-21T10:32:39.340 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:39 smithi098 ceph-mon[100914]: pgmap v1404: 83 pgs: 6 peering, 77 active+clean; 4.3 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail 2024-09-21T10:32:39.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:39 smithi196 ceph-mon[84784]: pgmap v1404: 83 pgs: 6 peering, 77 active+clean; 4.3 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail 2024-09-21T10:32:39.680 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:40.184 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:40 smithi098 ceph-mon[97580]: osdmap e405: 8 total, 8 up, 8 in 2024-09-21T10:32:40.184 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:40 smithi098 ceph-mon[100914]: osdmap e405: 8 total, 8 up, 8 in 2024-09-21T10:32:40.207 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:32:40.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:40 smithi196 ceph-mon[84784]: osdmap e405: 8 total, 8 up, 8 in 2024-09-21T10:32:40.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:32:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:32:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:32:40.467 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:32:40.847 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:32:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:32:40] "GET /metrics HTTP/1.1" 200 34174 "" "Prometheus/2.51.0" 2024-09-21T10:32:41.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:41 smithi098 ceph-mon[97580]: pgmap v1407: 83 pgs: 1 remapped+peering, 6 peering, 76 active+clean; 4.3 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T10:32:41.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:41 smithi098 ceph-mon[97580]: osdmap e406: 8 total, 8 up, 8 in 2024-09-21T10:32:41.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:41 smithi098 ceph-mon[100914]: pgmap v1407: 83 pgs: 1 remapped+peering, 6 peering, 76 active+clean; 4.3 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T10:32:41.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:41 smithi098 ceph-mon[100914]: osdmap e406: 8 total, 8 up, 8 in 2024-09-21T10:32:41.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:41 smithi196 ceph-mon[84784]: pgmap v1407: 83 pgs: 1 remapped+peering, 6 peering, 76 active+clean; 4.3 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T10:32:41.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:41 smithi196 ceph-mon[84784]: osdmap e406: 8 total, 8 up, 8 in 2024-09-21T10:32:42.278 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:32:42.279 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:32:42.279 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-21T10:32:42.279 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:32:42.896 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:32:43.087 INFO:teuthology.orchestra.run.smithi098.stderr:set pool 1 pgp_num to 11 2024-09-21T10:32:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:43 smithi196 ceph-mon[84784]: pgmap v1409: 83 pgs: 1 remapped+peering, 82 active+clean; 4.3 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T10:32:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:43 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1508572139' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-21T10:32:43.141 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:32:43.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:43 smithi098 ceph-mon[97580]: pgmap v1409: 83 pgs: 1 remapped+peering, 82 active+clean; 4.3 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T10:32:43.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:43 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1508572139' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-21T10:32:43.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:43 smithi098 ceph-mon[100914]: pgmap v1409: 83 pgs: 1 remapped+peering, 82 active+clean; 4.3 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T10:32:43.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:43 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1508572139' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-21T10:32:44.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-21T10:32:44.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:44 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1508572139' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-21T10:32:44.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:44 smithi196 ceph-mon[84784]: osdmap e407: 8 total, 8 up, 8 in 2024-09-21T10:32:44.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:44 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:44.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:44 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:44.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:44 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:32:44.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:44 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:32:44.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:44 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:44.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:44 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1508572139' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-21T10:32:44.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:44 smithi098 ceph-mon[97580]: osdmap e407: 8 total, 8 up, 8 in 2024-09-21T10:32:44.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:44 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:44.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:44 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:44.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:44 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:32:44.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:44 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:32:44.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:44 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:44.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:44 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1508572139' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-21T10:32:44.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:44 smithi098 ceph-mon[100914]: osdmap e407: 8 total, 8 up, 8 in 2024-09-21T10:32:44.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:44 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:44.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:44 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:44.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:44 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:32:44.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:44 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:32:44.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:44 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:32:44.389 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:44.842 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:32:44.842 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:32:44.842 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-21T10:32:44.842 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-21T10:32:44.842 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:32:45.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:45 smithi196 ceph-mon[84784]: pgmap v1411: 83 pgs: 83 active+clean; 4.3 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 2 objects/s recovering 2024-09-21T10:32:45.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:45 smithi196 ceph-mon[84784]: 1.4 scrub starts 2024-09-21T10:32:45.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:45 smithi196 ceph-mon[84784]: 1.4 scrub ok 2024-09-21T10:32:45.494 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:32:45.738 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:32:45.772 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:45 smithi098 ceph-mon[97580]: pgmap v1411: 83 pgs: 83 active+clean; 4.3 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 2 objects/s recovering 2024-09-21T10:32:45.772 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:45 smithi098 ceph-mon[97580]: 1.4 scrub starts 2024-09-21T10:32:45.772 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:45 smithi098 ceph-mon[97580]: 1.4 scrub ok 2024-09-21T10:32:45.772 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:45 smithi098 ceph-mon[100914]: pgmap v1411: 83 pgs: 83 active+clean; 4.3 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 2 objects/s recovering 2024-09-21T10:32:45.772 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:45 smithi098 ceph-mon[100914]: 1.4 scrub starts 2024-09-21T10:32:45.772 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:45 smithi098 ceph-mon[100914]: 1.4 scrub ok 2024-09-21T10:32:46.173 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a: {} 2024-09-21T10:32:46.173 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b: {} 2024-09-21T10:32:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T10:32:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T10:32:46.178 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c: {} 2024-09-21T10:32:46.178 INFO:teuthology.orchestra.run.smithi098.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T10:32:46.593 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:46 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3392106064' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T10:32:46.652 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:46 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3392106064' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T10:32:46.652 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:46 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3392106064' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T10:32:46.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T10:32:47.212 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:47.401 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:32:47.401 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-21T10:32:47.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-21T10:32:47.401 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:32:47.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.402 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181808 9.c 9.d25df03c (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:32:47.402 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.277707+0000", 2024-09-21T10:32:47.402 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 484.12037652100003, 2024-09-21T10:32:47.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0082070830000001, 2024-09-21T10:32:47.402 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:47.402 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:32:47.402 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:47.402 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:32:47.402 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:32:47.402 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181808 2024-09-21T10:32:47.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.402 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:47.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:47.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277707+0000", 2024-09-21T10:32:47.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:47.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277707+0000", 2024-09-21T10:32:47.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:47.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277705+0000", 2024-09-21T10:32:47.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:32:47.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:47.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277710+0000", 2024-09-21T10:32:47.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9729999999999997e-06 2024-09-21T10:32:47.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:47.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277717+0000", 2024-09-21T10:32:47.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7460000000000003e-06 2024-09-21T10:32:47.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:47.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277722+0000", 2024-09-21T10:32:47.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4260000000000002e-06 2024-09-21T10:32:47.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277974+0000", 2024-09-21T10:32:47.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025160500000000001 2024-09-21T10:32:47.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:32:47.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277980+0000", 2024-09-21T10:32:47.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0730000000000001e-06 2024-09-21T10:32:47.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.285892+0000", 2024-09-21T10:32:47.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0079123059999999 2024-09-21T10:32:47.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:47.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.285914+0000", 2024-09-21T10:32:47.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1756999999999999e-05 2024-09-21T10:32:47.406 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.406 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:47.406 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.406 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181841 9.c 9.8718e16c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:32:47.406 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.307618+0000", 2024-09-21T10:32:47.406 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 484.09046512899999, 2024-09-21T10:32:47.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97832473499999995, 2024-09-21T10:32:47.407 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:47.407 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:32:47.407 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:47.407 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:32:47.407 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:32:47.407 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181841 2024-09-21T10:32:47.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.407 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:47.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:47.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307618+0000", 2024-09-21T10:32:47.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:47.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307618+0000", 2024-09-21T10:32:47.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:47.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307617+0000", 2024-09-21T10:32:47.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:32:47.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:47.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307625+0000", 2024-09-21T10:32:47.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.473e-06 2024-09-21T10:32:47.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:47.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307642+0000", 2024-09-21T10:32:47.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7408e-05 2024-09-21T10:32:47.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:47.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307648+0000", 2024-09-21T10:32:47.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6400000000000002e-06 2024-09-21T10:32:47.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307699+0000", 2024-09-21T10:32:47.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1359000000000001e-05 2024-09-21T10:32:47.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:32:47.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.307707+0000", 2024-09-21T10:32:47.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.328e-06 2024-09-21T10:32:47.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.285935+0000", 2024-09-21T10:32:47.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97822819699999997 2024-09-21T10:32:47.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:47.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.285943+0000", 2024-09-21T10:32:47.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0290000000000005e-06 2024-09-21T10:32:47.411 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.411 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:47.411 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.412 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181856 9.c 9.1a6bfdac (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:32:47.412 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.326694+0000", 2024-09-21T10:32:47.412 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 484.07138967600002, 2024-09-21T10:32:47.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.95929115300000001, 2024-09-21T10:32:47.412 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:47.412 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:32:47.412 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:47.412 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:32:47.412 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:32:47.412 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181856 2024-09-21T10:32:47.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.412 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:47.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:47.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326694+0000", 2024-09-21T10:32:47.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:47.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326694+0000", 2024-09-21T10:32:47.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:47.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326692+0000", 2024-09-21T10:32:47.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:32:47.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:47.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326704+0000", 2024-09-21T10:32:47.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1864e-05 2024-09-21T10:32:47.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:47.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326720+0000", 2024-09-21T10:32:47.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5603999999999998e-05 2024-09-21T10:32:47.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:47.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326727+0000", 2024-09-21T10:32:47.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.374e-06 2024-09-21T10:32:47.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326780+0000", 2024-09-21T10:32:47.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2669e-05 2024-09-21T10:32:47.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:32:47.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.326788+0000", 2024-09-21T10:32:47.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1280000000000008e-06 2024-09-21T10:32:47.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.285962+0000", 2024-09-21T10:32:47.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.95917384299999997 2024-09-21T10:32:47.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:47.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.285985+0000", 2024-09-21T10:32:47.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3300000000000001e-05 2024-09-21T10:32:47.417 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.417 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:47.417 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.417 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181866 9.c 9.d4fb720c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:32:47.417 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.337971+0000", 2024-09-21T10:32:47.417 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 484.06011264, 2024-09-21T10:32:47.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94809235000000003, 2024-09-21T10:32:47.417 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:47.417 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:32:47.418 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:47.418 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:32:47.418 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:32:47.418 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181866 2024-09-21T10:32:47.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.418 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:47.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:47.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337971+0000", 2024-09-21T10:32:47.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:47.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337971+0000", 2024-09-21T10:32:47.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:47.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337969+0000", 2024-09-21T10:32:47.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:32:47.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:47.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337976+0000", 2024-09-21T10:32:47.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0319999999999998e-06 2024-09-21T10:32:47.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:47.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337992+0000", 2024-09-21T10:32:47.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5277e-05 2024-09-21T10:32:47.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:47.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337999+0000", 2024-09-21T10:32:47.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1720000000000001e-06 2024-09-21T10:32:47.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.338041+0000", 2024-09-21T10:32:47.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1777999999999999e-05 2024-09-21T10:32:47.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:32:47.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.338046+0000", 2024-09-21T10:32:47.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6119999999999998e-06 2024-09-21T10:32:47.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.286056+0000", 2024-09-21T10:32:47.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94800968200000002 2024-09-21T10:32:47.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:47.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.286063+0000", 2024-09-21T10:32:47.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1940000000000001e-06 2024-09-21T10:32:47.421 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.421 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:47.422 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.422 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243459 9.c 9:3515c86a:::benchmark_data_smithi098_254407_object110728:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:32:47.422 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:52.997457+0000", 2024-09-21T10:32:47.422 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.40062648000003, 2024-09-21T10:32:47.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.328995454, 2024-09-21T10:32:47.422 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:47.422 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:32:47.422 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:47.422 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:32:47.422 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:32:47.422 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243459 2024-09-21T10:32:47.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.423 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:47.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:47.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997457+0000", 2024-09-21T10:32:47.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:47.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997457+0000", 2024-09-21T10:32:47.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:47.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997456+0000", 2024-09-21T10:32:47.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:32:47.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:47.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997461+0000", 2024-09-21T10:32:47.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1320000000000002e-06 2024-09-21T10:32:47.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:47.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997484+0000", 2024-09-21T10:32:47.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3192000000000001e-05 2024-09-21T10:32:47.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:47.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997494+0000", 2024-09-21T10:32:47.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0175e-05 2024-09-21T10:32:47.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003474+0000", 2024-09-21T10:32:47.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059798409999999996 2024-09-21T10:32:47.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:32:47.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003481+0000", 2024-09-21T10:32:47.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4039999999999998e-06 2024-09-21T10:32:47.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.326400+0000", 2024-09-21T10:32:47.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3229192140000001 2024-09-21T10:32:47.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:47.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.326452+0000", 2024-09-21T10:32:47.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1841000000000001e-05 2024-09-21T10:32:47.426 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.426 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:47.426 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.427 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243460 9.c 9:3d3ff595:::benchmark_data_smithi098_254407_object110729:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:32:47.427 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:52.997502+0000", 2024-09-21T10:32:47.427 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.40058124199999, 2024-09-21T10:32:47.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3586443749999999, 2024-09-21T10:32:47.427 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:47.427 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:32:47.427 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:47.427 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:32:47.427 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:32:47.427 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243460 2024-09-21T10:32:47.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.427 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:47.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:47.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997502+0000", 2024-09-21T10:32:47.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:47.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997502+0000", 2024-09-21T10:32:47.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:47.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997502+0000", 2024-09-21T10:32:47.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:32:47.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:47.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997505+0000", 2024-09-21T10:32:47.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1889999999999999e-06 2024-09-21T10:32:47.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:47.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997512+0000", 2024-09-21T10:32:47.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8360000000000003e-06 2024-09-21T10:32:47.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:47.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997516+0000", 2024-09-21T10:32:47.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3100000000000002e-06 2024-09-21T10:32:47.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003820+0000", 2024-09-21T10:32:47.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0063037989999999997 2024-09-21T10:32:47.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:32:47.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003825+0000", 2024-09-21T10:32:47.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3410000000000002e-06 2024-09-21T10:32:47.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.326483+0000", 2024-09-21T10:32:47.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.322658127 2024-09-21T10:32:47.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for missing object", 2024-09-21T10:32:47.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.326563+0000", 2024-09-21T10:32:47.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0270000000000002e-05 2024-09-21T10:32:47.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356089+0000", 2024-09-21T10:32:47.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.029525136 2024-09-21T10:32:47.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:47.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356146+0000", 2024-09-21T10:32:47.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7917000000000001e-05 2024-09-21T10:32:47.432 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.432 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:47.432 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.433 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243465 9.c 9:36d01026:::benchmark_data_smithi098_254407_object110734:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:32:47.433 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:52.997520+0000", 2024-09-21T10:32:47.433 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.40056300200001, 2024-09-21T10:32:47.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.359137246, 2024-09-21T10:32:47.433 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:47.433 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:47.433 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:47.433 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:32:47.433 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:32:47.433 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243465 2024-09-21T10:32:47.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.433 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:47.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:47.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997520+0000", 2024-09-21T10:32:47.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:47.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997520+0000", 2024-09-21T10:32:47.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:47.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997520+0000", 2024-09-21T10:32:47.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:32:47.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:47.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997522+0000", 2024-09-21T10:32:47.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2139999999999999e-06 2024-09-21T10:32:47.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:47.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997528+0000", 2024-09-21T10:32:47.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.049e-06 2024-09-21T10:32:47.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:47.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997533+0000", 2024-09-21T10:32:47.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.578e-06 2024-09-21T10:32:47.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003841+0000", 2024-09-21T10:32:47.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0063081760000000004 2024-09-21T10:32:47.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:32:47.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003845+0000", 2024-09-21T10:32:47.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2760000000000002e-06 2024-09-21T10:32:47.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.326590+0000", 2024-09-21T10:32:47.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3227443409999999 2024-09-21T10:32:47.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:47.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.326677+0000", 2024-09-21T10:32:47.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7176999999999996e-05 2024-09-21T10:32:47.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:32:47.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.327203+0000", 2024-09-21T10:32:47.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00052658000000000002 2024-09-21T10:32:47.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:32:47.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.337718+0000", 2024-09-21T10:32:47.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.01051439 2024-09-21T10:32:47.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356287+0000", 2024-09-21T10:32:47.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018568972999999999 2024-09-21T10:32:47.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356519+0000", 2024-09-21T10:32:47.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000231921 2024-09-21T10:32:47.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:47.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356565+0000", 2024-09-21T10:32:47.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5957000000000001e-05 2024-09-21T10:32:47.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:47.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356658+0000", 2024-09-21T10:32:47.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3066999999999995e-05 2024-09-21T10:32:47.440 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.440 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:47.440 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.441 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243471 9.c 9:3e8385fd:::benchmark_data_smithi098_254407_object110740:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:32:47.441 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:52.997537+0000", 2024-09-21T10:32:47.441 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.40054617499999, 2024-09-21T10:32:47.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3592854779999999, 2024-09-21T10:32:47.441 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:47.441 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:47.441 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:47.441 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:32:47.441 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:32:47.441 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243471 2024-09-21T10:32:47.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.442 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:47.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:47.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997537+0000", 2024-09-21T10:32:47.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:47.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997537+0000", 2024-09-21T10:32:47.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:47.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997537+0000", 2024-09-21T10:32:47.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:32:47.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:47.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997539+0000", 2024-09-21T10:32:47.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3750000000000001e-06 2024-09-21T10:32:47.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:47.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997548+0000", 2024-09-21T10:32:47.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.02e-06 2024-09-21T10:32:47.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:47.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:52.997552+0000", 2024-09-21T10:32:47.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9380000000000002e-06 2024-09-21T10:32:47.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003861+0000", 2024-09-21T10:32:47.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0063086239999999997 2024-09-21T10:32:47.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.446 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:32:47.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.003865+0000", 2024-09-21T10:32:47.446 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1239999999999998e-06 2024-09-21T10:32:47.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.446 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.332121+0000", 2024-09-21T10:32:47.446 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.328256586 2024-09-21T10:32:47.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.446 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:47.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.332255+0000", 2024-09-21T10:32:47.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013350999999999999 2024-09-21T10:32:47.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:32:47.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.332401+0000", 2024-09-21T10:32:47.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000146465 2024-09-21T10:32:47.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:32:47.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.341719+0000", 2024-09-21T10:32:47.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0093176160000000008 2024-09-21T10:32:47.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356328+0000", 2024-09-21T10:32:47.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014608612 2024-09-21T10:32:47.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356695+0000", 2024-09-21T10:32:47.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00036710999999999997 2024-09-21T10:32:47.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:47.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356736+0000", 2024-09-21T10:32:47.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1112000000000002e-05 2024-09-21T10:32:47.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:47.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356823+0000", 2024-09-21T10:32:47.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6779000000000004e-05 2024-09-21T10:32:47.449 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.449 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:47.449 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.450 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243517 9.c 9:3e65f1ad:::benchmark_data_smithi098_254407_object110786:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:32:47.450 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.050602+0000", 2024-09-21T10:32:47.450 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.34748172000002, 2024-09-21T10:32:47.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.306614151, 2024-09-21T10:32:47.450 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:47.450 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:47.450 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:47.450 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:32:47.450 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:32:47.450 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243517 2024-09-21T10:32:47.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.451 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:47.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:47.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050602+0000", 2024-09-21T10:32:47.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:47.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050602+0000", 2024-09-21T10:32:47.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:47.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050599+0000", 2024-09-21T10:32:47.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999976 2024-09-21T10:32:47.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:47.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050607+0000", 2024-09-21T10:32:47.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0690000000000002e-06 2024-09-21T10:32:47.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:47.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050627+0000", 2024-09-21T10:32:47.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9422000000000001e-05 2024-09-21T10:32:47.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:47.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050634+0000", 2024-09-21T10:32:47.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1160000000000001e-06 2024-09-21T10:32:47.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050699+0000", 2024-09-21T10:32:47.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5191999999999995e-05 2024-09-21T10:32:47.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:32:47.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.050709+0000", 2024-09-21T10:32:47.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3270000000000007e-06 2024-09-21T10:32:47.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.334376+0000", 2024-09-21T10:32:47.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.283667766 2024-09-21T10:32:47.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:47.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.334474+0000", 2024-09-21T10:32:47.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7481999999999999e-05 2024-09-21T10:32:47.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:32:47.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.334602+0000", 2024-09-21T10:32:47.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000128014 2024-09-21T10:32:47.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:32:47.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.341726+0000", 2024-09-21T10:32:47.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0071239490000000001 2024-09-21T10:32:47.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356357+0000", 2024-09-21T10:32:47.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014631183000000001 2024-09-21T10:32:47.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357084+0000", 2024-09-21T10:32:47.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00072674199999999995 2024-09-21T10:32:47.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:47.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357126+0000", 2024-09-21T10:32:47.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2732999999999999e-05 2024-09-21T10:32:47.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:47.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357216+0000", 2024-09-21T10:32:47.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9373e-05 2024-09-21T10:32:47.458 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.458 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:47.458 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.458 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243534 9.c 9:3cf10c44:::benchmark_data_smithi098_254407_object110803:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:32:47.458 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.063492+0000", 2024-09-21T10:32:47.458 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.33459109, 2024-09-21T10:32:47.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2940934849999999, 2024-09-21T10:32:47.458 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:47.458 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:47.459 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:47.459 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:32:47.459 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:32:47.459 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243534 2024-09-21T10:32:47.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.459 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:47.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:47.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063492+0000", 2024-09-21T10:32:47.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:47.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063492+0000", 2024-09-21T10:32:47.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:47.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063491+0000", 2024-09-21T10:32:47.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:32:47.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:47.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063498+0000", 2024-09-21T10:32:47.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9759999999999995e-06 2024-09-21T10:32:47.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:47.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063511+0000", 2024-09-21T10:32:47.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2238e-05 2024-09-21T10:32:47.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:47.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063516+0000", 2024-09-21T10:32:47.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6849999999999998e-06 2024-09-21T10:32:47.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063572+0000", 2024-09-21T10:32:47.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5648e-05 2024-09-21T10:32:47.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:32:47.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.063579+0000", 2024-09-21T10:32:47.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2169999999999997e-06 2024-09-21T10:32:47.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.336131+0000", 2024-09-21T10:32:47.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.27255148 2024-09-21T10:32:47.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:47.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.336203+0000", 2024-09-21T10:32:47.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2214999999999999e-05 2024-09-21T10:32:47.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:32:47.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.336278+0000", 2024-09-21T10:32:47.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4629000000000007e-05 2024-09-21T10:32:47.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:32:47.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.343957+0000", 2024-09-21T10:32:47.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0076795850000000001 2024-09-21T10:32:47.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356395+0000", 2024-09-21T10:32:47.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012437325000000001 2024-09-21T10:32:47.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357453+0000", 2024-09-21T10:32:47.465 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001058743 2024-09-21T10:32:47.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:47.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357495+0000", 2024-09-21T10:32:47.465 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1498999999999997e-05 2024-09-21T10:32:47.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:47.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357586+0000", 2024-09-21T10:32:47.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1014999999999995e-05 2024-09-21T10:32:47.466 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.466 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:47.466 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.466 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243535 9.c 9:37d3df54:::benchmark_data_smithi098_254407_object110804:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:32:47.466 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.068354+0000", 2024-09-21T10:32:47.466 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.32972901099998, 2024-09-21T10:32:47.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2894149260000001, 2024-09-21T10:32:47.467 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:47.467 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:47.467 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:47.467 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:32:47.467 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:32:47.467 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243535 2024-09-21T10:32:47.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.467 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:47.467 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.467 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:47.467 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068354+0000", 2024-09-21T10:32:47.467 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:47.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068354+0000", 2024-09-21T10:32:47.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:47.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068352+0000", 2024-09-21T10:32:47.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:32:47.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:47.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068360+0000", 2024-09-21T10:32:47.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7209999999999994e-06 2024-09-21T10:32:47.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:47.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068377+0000", 2024-09-21T10:32:47.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7198e-05 2024-09-21T10:32:47.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:47.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068383+0000", 2024-09-21T10:32:47.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1160000000000004e-06 2024-09-21T10:32:47.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068465+0000", 2024-09-21T10:32:47.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1681999999999995e-05 2024-09-21T10:32:47.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:32:47.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.068473+0000", 2024-09-21T10:32:47.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8310000000000001e-06 2024-09-21T10:32:47.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.337739+0000", 2024-09-21T10:32:47.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.269265758 2024-09-21T10:32:47.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:47.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.337850+0000", 2024-09-21T10:32:47.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000111327 2024-09-21T10:32:47.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:32:47.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.337980+0000", 2024-09-21T10:32:47.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000130149 2024-09-21T10:32:47.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:32:47.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.343962+0000", 2024-09-21T10:32:47.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059821479999999996 2024-09-21T10:32:47.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356421+0000", 2024-09-21T10:32:47.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012459328 2024-09-21T10:32:47.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.473 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.473 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.473 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357634+0000", 2024-09-21T10:32:47.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012120410000000001 2024-09-21T10:32:47.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.473 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.473 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:47.473 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357681+0000", 2024-09-21T10:32:47.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7024999999999998e-05 2024-09-21T10:32:47.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.473 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.474 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:47.474 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357769+0000", 2024-09-21T10:32:47.474 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8727999999999995e-05 2024-09-21T10:32:47.474 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.474 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:47.474 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.474 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.474 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243548 9.c 9:3c4fc234:::benchmark_data_smithi098_254407_object110817:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:32:47.474 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.078991+0000", 2024-09-21T10:32:47.474 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.31909243799998, 2024-09-21T10:32:47.475 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.278948075, 2024-09-21T10:32:47.475 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:47.475 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:47.475 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:47.475 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:32:47.475 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:32:47.475 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243548 2024-09-21T10:32:47.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.475 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:47.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:47.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.078991+0000", 2024-09-21T10:32:47.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:47.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.078991+0000", 2024-09-21T10:32:47.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:47.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.078989+0000", 2024-09-21T10:32:47.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:32:47.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.477 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.477 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:47.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.078996+0000", 2024-09-21T10:32:47.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.968e-06 2024-09-21T10:32:47.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.477 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.477 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:47.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.079013+0000", 2024-09-21T10:32:47.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6549e-05 2024-09-21T10:32:47.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.477 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:47.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.079019+0000", 2024-09-21T10:32:47.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6729999999999997e-06 2024-09-21T10:32:47.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.079063+0000", 2024-09-21T10:32:47.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4376000000000003e-05 2024-09-21T10:32:47.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.479 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:32:47.479 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.079070+0000", 2024-09-21T10:32:47.479 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6089999999999997e-06 2024-09-21T10:32:47.479 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.479 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.479 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.339499+0000", 2024-09-21T10:32:47.479 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.260429467 2024-09-21T10:32:47.479 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.479 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:47.480 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.339588+0000", 2024-09-21T10:32:47.480 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8708000000000005e-05 2024-09-21T10:32:47.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.480 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.480 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:32:47.480 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.339668+0000", 2024-09-21T10:32:47.480 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0549000000000004e-05 2024-09-21T10:32:47.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.480 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.480 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:32:47.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.350422+0000", 2024-09-21T10:32:47.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010753257 2024-09-21T10:32:47.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.482 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356457+0000", 2024-09-21T10:32:47.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0060354479999999997 2024-09-21T10:32:47.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.482 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357809+0000", 2024-09-21T10:32:47.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0013525049999999999 2024-09-21T10:32:47.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:47.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357848+0000", 2024-09-21T10:32:47.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8537000000000001e-05 2024-09-21T10:32:47.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:47.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357939+0000", 2024-09-21T10:32:47.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1001000000000005e-05 2024-09-21T10:32:47.484 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.484 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:47.484 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.484 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243570 9.c 9:342610f6:::benchmark_data_smithi098_254407_object110839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:32:47.484 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.101422+0000", 2024-09-21T10:32:47.484 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.29666144800001, 2024-09-21T10:32:47.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2566680909999999, 2024-09-21T10:32:47.484 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:47.484 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:47.485 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:47.485 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:32:47.485 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:32:47.485 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243570 2024-09-21T10:32:47.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.485 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:47.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.485 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:47.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101422+0000", 2024-09-21T10:32:47.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.486 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:47.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101422+0000", 2024-09-21T10:32:47.486 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.486 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.486 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:47.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101420+0000", 2024-09-21T10:32:47.486 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:32:47.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.486 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.486 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:47.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101428+0000", 2024-09-21T10:32:47.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8420000000000005e-06 2024-09-21T10:32:47.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:47.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101446+0000", 2024-09-21T10:32:47.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8026e-05 2024-09-21T10:32:47.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:47.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101452+0000", 2024-09-21T10:32:47.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4910000000000002e-06 2024-09-21T10:32:47.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101565+0000", 2024-09-21T10:32:47.488 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011296 2024-09-21T10:32:47.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:32:47.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.101587+0000", 2024-09-21T10:32:47.488 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2072000000000001e-05 2024-09-21T10:32:47.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.489 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.341753+0000", 2024-09-21T10:32:47.489 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2401660510000001 2024-09-21T10:32:47.489 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.489 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.489 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:47.489 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.341857+0000", 2024-09-21T10:32:47.489 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000103174 2024-09-21T10:32:47.489 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.489 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.489 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:32:47.489 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.341986+0000", 2024-09-21T10:32:47.490 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012943199999999999 2024-09-21T10:32:47.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.490 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.490 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:32:47.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.355716+0000", 2024-09-21T10:32:47.490 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013729961000000001 2024-09-21T10:32:47.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.490 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.490 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356484+0000", 2024-09-21T10:32:47.490 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00076824599999999999 2024-09-21T10:32:47.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.491 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.491 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357972+0000", 2024-09-21T10:32:47.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014878319999999999 2024-09-21T10:32:47.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.491 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.491 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:47.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.358008+0000", 2024-09-21T10:32:47.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6263999999999997e-05 2024-09-21T10:32:47.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.491 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.491 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:47.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.358090+0000", 2024-09-21T10:32:47.492 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1607999999999995e-05 2024-09-21T10:32:47.492 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.492 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:47.492 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.492 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.492 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.492 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243592 9.c 9:3d6001ff:::benchmark_data_smithi098_254407_object110861:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:32:47.492 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.125239+0000", 2024-09-21T10:32:47.492 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.27284458299999, 2024-09-21T10:32:47.492 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.242184476, 2024-09-21T10:32:47.492 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:47.492 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:47.493 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:47.493 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:32:47.493 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:32:47.493 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243592 2024-09-21T10:32:47.493 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.493 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:47.493 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.493 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:47.493 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125239+0000", 2024-09-21T10:32:47.493 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.493 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.493 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:47.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125239+0000", 2024-09-21T10:32:47.494 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.494 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:47.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125237+0000", 2024-09-21T10:32:47.494 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:32:47.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.494 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:47.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125244+0000", 2024-09-21T10:32:47.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9600000000000003e-06 2024-09-21T10:32:47.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.495 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:47.495 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125256+0000", 2024-09-21T10:32:47.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1759e-05 2024-09-21T10:32:47.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.495 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:47.495 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125262+0000", 2024-09-21T10:32:47.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5139999999999998e-06 2024-09-21T10:32:47.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.496 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.496 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125306+0000", 2024-09-21T10:32:47.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3724999999999999e-05 2024-09-21T10:32:47.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.496 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.496 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:32:47.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.125316+0000", 2024-09-21T10:32:47.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6500000000000008e-06 2024-09-21T10:32:47.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.496 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.496 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.344004+0000", 2024-09-21T10:32:47.497 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2186879720000001 2024-09-21T10:32:47.497 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.497 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.497 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:47.497 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.344092+0000", 2024-09-21T10:32:47.497 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8027000000000003e-05 2024-09-21T10:32:47.497 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.497 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.497 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:32:47.497 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.344168+0000", 2024-09-21T10:32:47.497 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6508999999999995e-05 2024-09-21T10:32:47.497 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.497 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.498 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:32:47.498 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.355720+0000", 2024-09-21T10:32:47.498 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011551532 2024-09-21T10:32:47.498 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.498 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.498 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.498 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356879+0000", 2024-09-21T10:32:47.498 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001159642 2024-09-21T10:32:47.498 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.498 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.499 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.499 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367261+0000", 2024-09-21T10:32:47.499 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010381081 2024-09-21T10:32:47.499 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.499 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.499 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:47.499 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367319+0000", 2024-09-21T10:32:47.499 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8953999999999998e-05 2024-09-21T10:32:47.499 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.499 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.499 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:47.499 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367423+0000", 2024-09-21T10:32:47.499 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000103762 2024-09-21T10:32:47.500 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.500 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:47.500 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.500 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.500 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.500 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243598 9.c 9:30869b3f:::benchmark_data_smithi098_254407_object110867:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:32:47.500 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.132539+0000", 2024-09-21T10:32:47.500 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.26554436800001, 2024-09-21T10:32:47.500 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2350737359999999, 2024-09-21T10:32:47.500 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:47.501 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:47.501 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:47.501 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:32:47.501 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:32:47.501 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243598 2024-09-21T10:32:47.501 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.501 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:47.501 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.501 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:47.501 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132539+0000", 2024-09-21T10:32:47.501 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.501 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.502 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.502 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:47.502 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132539+0000", 2024-09-21T10:32:47.502 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.502 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.502 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.502 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:47.502 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132537+0000", 2024-09-21T10:32:47.502 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:32:47.502 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.502 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.502 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:47.503 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132545+0000", 2024-09-21T10:32:47.503 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4660000000000002e-06 2024-09-21T10:32:47.503 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.503 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.503 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:47.503 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132564+0000", 2024-09-21T10:32:47.503 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8981999999999999e-05 2024-09-21T10:32:47.503 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.503 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.503 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:47.503 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132570+0000", 2024-09-21T10:32:47.503 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3400000000000003e-06 2024-09-21T10:32:47.503 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.504 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.504 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.504 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132754+0000", 2024-09-21T10:32:47.504 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000183821 2024-09-21T10:32:47.504 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.504 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.504 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:32:47.504 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.132782+0000", 2024-09-21T10:32:47.504 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8096e-05 2024-09-21T10:32:47.504 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.504 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.504 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.505 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.345661+0000", 2024-09-21T10:32:47.505 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2128794350000001 2024-09-21T10:32:47.505 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.505 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.505 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:47.505 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.345758+0000", 2024-09-21T10:32:47.505 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6637000000000003e-05 2024-09-21T10:32:47.505 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.505 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.505 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:32:47.505 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.345832+0000", 2024-09-21T10:32:47.506 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3719000000000003e-05 2024-09-21T10:32:47.506 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.506 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.506 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:32:47.506 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.355724+0000", 2024-09-21T10:32:47.506 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0098921329999999991 2024-09-21T10:32:47.506 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.506 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.506 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.506 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356929+0000", 2024-09-21T10:32:47.506 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012046579999999999 2024-09-21T10:32:47.507 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.507 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.507 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.507 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367459+0000", 2024-09-21T10:32:47.507 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010530358 2024-09-21T10:32:47.507 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.507 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.507 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:47.507 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367509+0000", 2024-09-21T10:32:47.507 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0309999999999998e-05 2024-09-21T10:32:47.507 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.508 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.508 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:47.508 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367613+0000", 2024-09-21T10:32:47.508 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000103531 2024-09-21T10:32:47.508 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.508 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:47.508 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.508 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.508 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.508 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243614 9.c 9:3cd37905:::benchmark_data_smithi098_254407_object110883:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:32:47.508 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.149686+0000", 2024-09-21T10:32:47.508 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.24839775100003, 2024-09-21T10:32:47.508 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2181167049999999, 2024-09-21T10:32:47.509 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:47.509 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:47.509 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:47.509 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:32:47.509 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:32:47.509 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243614 2024-09-21T10:32:47.509 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.509 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:47.509 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.509 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:47.509 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149686+0000", 2024-09-21T10:32:47.509 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.509 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.510 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.510 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:47.510 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149686+0000", 2024-09-21T10:32:47.510 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.510 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.510 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.510 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:47.510 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149684+0000", 2024-09-21T10:32:47.510 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:32:47.510 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.510 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.511 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:47.511 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149691+0000", 2024-09-21T10:32:47.511 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1439999999999997e-06 2024-09-21T10:32:47.511 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.511 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.511 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:47.511 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149711+0000", 2024-09-21T10:32:47.511 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0307999999999999e-05 2024-09-21T10:32:47.511 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.511 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.511 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:47.511 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149723+0000", 2024-09-21T10:32:47.512 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1687e-05 2024-09-21T10:32:47.512 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.512 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.512 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.512 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149834+0000", 2024-09-21T10:32:47.512 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000110895 2024-09-21T10:32:47.512 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.512 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.512 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:32:47.512 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.149870+0000", 2024-09-21T10:32:47.512 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.591e-05 2024-09-21T10:32:47.512 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.512 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.513 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.513 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.347247+0000", 2024-09-21T10:32:47.513 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1973775120000001 2024-09-21T10:32:47.513 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.513 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.513 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:47.513 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.347335+0000", 2024-09-21T10:32:47.513 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7427000000000002e-05 2024-09-21T10:32:47.513 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.513 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.513 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:32:47.513 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.347412+0000", 2024-09-21T10:32:47.514 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6972999999999996e-05 2024-09-21T10:32:47.514 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.514 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.514 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:32:47.514 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.355727+0000", 2024-09-21T10:32:47.514 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0083153749999999998 2024-09-21T10:32:47.514 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.514 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.514 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.514 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356967+0000", 2024-09-21T10:32:47.515 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001240285 2024-09-21T10:32:47.515 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.515 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.515 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.515 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367650+0000", 2024-09-21T10:32:47.515 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010682574 2024-09-21T10:32:47.515 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.515 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.515 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:47.515 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367693+0000", 2024-09-21T10:32:47.516 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3123e-05 2024-09-21T10:32:47.516 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.516 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.516 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:47.516 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367802+0000", 2024-09-21T10:32:47.516 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109219 2024-09-21T10:32:47.516 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.516 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:47.516 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.516 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.516 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.516 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243648 9.c 9:33fdd93d:::benchmark_data_smithi098_254407_object110917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:32:47.516 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.196470+0000", 2024-09-21T10:32:47.517 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.20161305599999, 2024-09-21T10:32:47.517 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1715113070000001, 2024-09-21T10:32:47.517 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:47.517 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:47.517 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:47.517 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:32:47.517 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:32:47.517 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243648 2024-09-21T10:32:47.517 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.517 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:47.517 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.517 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:47.518 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196470+0000", 2024-09-21T10:32:47.518 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.518 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.518 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.518 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:47.518 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196470+0000", 2024-09-21T10:32:47.518 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.518 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.518 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.518 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:47.518 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196469+0000", 2024-09-21T10:32:47.519 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:32:47.519 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.519 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.519 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:47.519 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196476+0000", 2024-09-21T10:32:47.519 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7000000000000008e-06 2024-09-21T10:32:47.519 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.519 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.519 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:47.519 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196492+0000", 2024-09-21T10:32:47.519 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5877000000000001e-05 2024-09-21T10:32:47.519 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.520 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.520 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:47.520 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196499+0000", 2024-09-21T10:32:47.520 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1550000000000001e-06 2024-09-21T10:32:47.520 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.520 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.520 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.520 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196545+0000", 2024-09-21T10:32:47.520 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5679000000000001e-05 2024-09-21T10:32:47.520 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.520 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.520 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:32:47.520 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.196560+0000", 2024-09-21T10:32:47.521 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4919000000000001e-05 2024-09-21T10:32:47.521 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.521 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.521 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.521 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.348867+0000", 2024-09-21T10:32:47.521 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1523073770000001 2024-09-21T10:32:47.521 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.521 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.521 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:47.521 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.348947+0000", 2024-09-21T10:32:47.521 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9827999999999995e-05 2024-09-21T10:32:47.521 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.522 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.522 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:32:47.522 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.349016+0000", 2024-09-21T10:32:47.522 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9091000000000006e-05 2024-09-21T10:32:47.522 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.522 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.522 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:32:47.522 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.355766+0000", 2024-09-21T10:32:47.522 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0067494419999999996 2024-09-21T10:32:47.522 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.522 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.522 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.523 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357002+0000", 2024-09-21T10:32:47.523 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012364780000000001 2024-09-21T10:32:47.523 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.523 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.523 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.523 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367840+0000", 2024-09-21T10:32:47.523 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010837704 2024-09-21T10:32:47.523 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.523 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.523 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:47.523 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367884+0000", 2024-09-21T10:32:47.524 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4353e-05 2024-09-21T10:32:47.524 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.524 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.524 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:47.524 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.367982+0000", 2024-09-21T10:32:47.524 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7347000000000002e-05 2024-09-21T10:32:47.524 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.524 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:47.524 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.524 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.524 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.524 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243665 9.c 9:3b9d021e:::benchmark_data_smithi098_254407_object110934:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:32:47.525 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.227151+0000", 2024-09-21T10:32:47.525 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.17093190000003, 2024-09-21T10:32:47.525 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.141037334, 2024-09-21T10:32:47.525 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:47.525 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:47.525 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:47.525 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:32:47.525 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:32:47.525 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243665 2024-09-21T10:32:47.525 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.525 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:47.526 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.526 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:47.526 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227151+0000", 2024-09-21T10:32:47.526 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.526 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.526 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.526 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:47.526 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227151+0000", 2024-09-21T10:32:47.526 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.526 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.526 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.526 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:47.527 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227150+0000", 2024-09-21T10:32:47.527 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:32:47.527 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.527 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.527 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:47.527 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227157+0000", 2024-09-21T10:32:47.527 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3030000000000002e-06 2024-09-21T10:32:47.527 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.527 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.527 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:47.527 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227169+0000", 2024-09-21T10:32:47.527 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1999e-05 2024-09-21T10:32:47.527 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.528 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.528 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:47.528 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227174+0000", 2024-09-21T10:32:47.528 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4580000000000001e-06 2024-09-21T10:32:47.528 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.528 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.528 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.528 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227202+0000", 2024-09-21T10:32:47.528 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8002999999999999e-05 2024-09-21T10:32:47.528 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.528 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.528 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:32:47.529 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.227208+0000", 2024-09-21T10:32:47.529 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4099999999999999e-06 2024-09-21T10:32:47.529 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.529 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.529 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.529 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.350461+0000", 2024-09-21T10:32:47.529 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.123253126 2024-09-21T10:32:47.529 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.529 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.529 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:47.529 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.350554+0000", 2024-09-21T10:32:47.529 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2917000000000005e-05 2024-09-21T10:32:47.529 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.530 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.530 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:32:47.530 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.350643+0000", 2024-09-21T10:32:47.530 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9167999999999997e-05 2024-09-21T10:32:47.530 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.530 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.530 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:32:47.530 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.355771+0000", 2024-09-21T10:32:47.530 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051275189999999997 2024-09-21T10:32:47.530 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.530 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.531 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.531 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357271+0000", 2024-09-21T10:32:47.531 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015008459999999999 2024-09-21T10:32:47.531 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.531 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.531 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.531 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.368035+0000", 2024-09-21T10:32:47.531 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010763679 2024-09-21T10:32:47.531 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.531 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.532 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:47.532 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.368089+0000", 2024-09-21T10:32:47.532 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3399000000000002e-05 2024-09-21T10:32:47.532 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.532 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.532 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:47.532 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.368189+0000", 2024-09-21T10:32:47.532 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100296 2024-09-21T10:32:47.532 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.532 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:47.532 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.532 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.533 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.533 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243683 9.c 9:3b325d23:::benchmark_data_smithi098_254407_object110952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:32:47.533 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.260563+0000", 2024-09-21T10:32:47.533 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.137520833, 2024-09-21T10:32:47.533 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1154280519999999, 2024-09-21T10:32:47.533 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:47.533 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:47.533 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:47.533 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:32:47.533 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:32:47.533 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243683 2024-09-21T10:32:47.533 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.533 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:47.534 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.534 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:47.534 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260563+0000", 2024-09-21T10:32:47.534 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.534 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.534 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.534 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:47.534 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260563+0000", 2024-09-21T10:32:47.534 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.534 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.534 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.534 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:47.534 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260561+0000", 2024-09-21T10:32:47.535 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:32:47.535 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.535 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.535 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:47.535 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260567+0000", 2024-09-21T10:32:47.535 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6849999999999998e-06 2024-09-21T10:32:47.535 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.535 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.535 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:47.535 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260577+0000", 2024-09-21T10:32:47.535 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0217000000000001e-05 2024-09-21T10:32:47.535 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.536 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.536 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:47.536 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260582+0000", 2024-09-21T10:32:47.536 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3880000000000002e-06 2024-09-21T10:32:47.536 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.536 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.536 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.536 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260623+0000", 2024-09-21T10:32:47.536 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0915999999999999e-05 2024-09-21T10:32:47.536 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.536 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.537 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:32:47.537 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.260630+0000", 2024-09-21T10:32:47.537 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5730000000000002e-06 2024-09-21T10:32:47.537 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.537 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.537 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.537 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.352085+0000", 2024-09-21T10:32:47.537 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0914547569999999 2024-09-21T10:32:47.537 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.537 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.537 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:47.538 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.352173+0000", 2024-09-21T10:32:47.538 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7819999999999996e-05 2024-09-21T10:32:47.538 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.538 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.538 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:32:47.538 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.352293+0000", 2024-09-21T10:32:47.538 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000120475 2024-09-21T10:32:47.538 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.538 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.538 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:32:47.538 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.355774+0000", 2024-09-21T10:32:47.538 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003481137 2024-09-21T10:32:47.539 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.539 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.539 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.539 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357314+0000", 2024-09-21T10:32:47.539 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001539285 2024-09-21T10:32:47.539 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.539 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.539 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.539 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.375813+0000", 2024-09-21T10:32:47.539 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018498905 2024-09-21T10:32:47.539 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.539 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.540 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:47.540 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.375865+0000", 2024-09-21T10:32:47.540 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2945000000000002e-05 2024-09-21T10:32:47.540 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.540 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.540 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:47.540 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.375991+0000", 2024-09-21T10:32:47.540 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012511 2024-09-21T10:32:47.540 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.540 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:47.540 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.540 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:243684 9.c 9:3d7d9148:::benchmark_data_smithi098_254407_object110953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-21T10:32:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:53.263461+0000", 2024-09-21T10:32:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.13462266400001, 2024-09-21T10:32:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.115013201, 2024-09-21T10:32:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:32:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:32:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 243684 2024-09-21T10:32:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263461+0000", 2024-09-21T10:32:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263461+0000", 2024-09-21T10:32:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263460+0000", 2024-09-21T10:32:47.543 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:32:47.543 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.543 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.543 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:47.543 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263465+0000", 2024-09-21T10:32:47.543 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6280000000000004e-06 2024-09-21T10:32:47.543 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.543 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.543 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:47.543 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263473+0000", 2024-09-21T10:32:47.543 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5290000000000008e-06 2024-09-21T10:32:47.543 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263477+0000", 2024-09-21T10:32:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9500000000000003e-06 2024-09-21T10:32:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263505+0000", 2024-09-21T10:32:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7491000000000002e-05 2024-09-21T10:32:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:32:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:53.263510+0000", 2024-09-21T10:32:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4090000000000002e-06 2024-09-21T10:32:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.353714+0000", 2024-09-21T10:32:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0902042750000001 2024-09-21T10:32:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.353823+0000", 2024-09-21T10:32:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108884 2024-09-21T10:32:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:32:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.353950+0000", 2024-09-21T10:32:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012683599999999999 2024-09-21T10:32:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:32:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.356144+0000", 2024-09-21T10:32:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021941030000000002 2024-09-21T10:32:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.357352+0000", 2024-09-21T10:32:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012075149999999999 2024-09-21T10:32:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.378332+0000", 2024-09-21T10:32:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.020980784999999998 2024-09-21T10:32:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.378381+0000", 2024-09-21T10:32:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8603000000000002e-05 2024-09-21T10:32:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:47.548 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:54.378474+0000", 2024-09-21T10:32:47.548 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2977000000000001e-05 2024-09-21T10:32:47.548 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.548 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:47.548 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.548 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:47.548 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:47.548 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:32:47.548 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:47 smithi196 ceph-mon[84784]: pgmap v1412: 83 pgs: 83 active+clean; 4.3 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-21T10:32:47.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:47 smithi098 ceph-mon[97580]: pgmap v1412: 83 pgs: 83 active+clean; 4.3 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-21T10:32:47.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:47 smithi098 ceph-mon[100914]: pgmap v1412: 83 pgs: 83 active+clean; 4.3 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-21T10:32:48.430 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:32:48.674 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:32:49.405 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T10:32:49.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:49 smithi098 ceph-mon[97580]: pgmap v1413: 83 pgs: 83 active+clean; 4.6 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-21T10:32:49.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:49 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/384029077' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:32:49.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:49 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:32:49.753 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:49 smithi098 ceph-mon[100914]: pgmap v1413: 83 pgs: 83 active+clean; 4.6 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-21T10:32:49.753 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:49 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/384029077' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:32:49.753 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:49 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:32:49.764 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:49 smithi196 ceph-mon[84784]: pgmap v1413: 83 pgs: 83 active+clean; 4.6 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-21T10:32:49.764 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:49 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/384029077' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:32:49.764 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:49 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:32:50.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 0 1 2024-09-21T10:32:50.243 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:32:50.243 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:32:50.243 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-21T10:32:50.243 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:32:50.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:32:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:32:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:32:50.391 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:50.681 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:32:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:32:50] "GET /metrics HTTP/1.1" 200 34174 "" "Prometheus/2.51.0" 2024-09-21T10:32:50.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:50 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:32:50.682 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:50 smithi098 ceph-mon[97580]: osdmap e408: 8 total, 8 up, 8 in 2024-09-21T10:32:50.682 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:50 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:32:50.682 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:50 smithi098 ceph-mon[100914]: osdmap e408: 8 total, 8 up, 8 in 2024-09-21T10:32:50.821 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:50 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:32:50.821 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:50 smithi196 ceph-mon[84784]: osdmap e408: 8 total, 8 up, 8 in 2024-09-21T10:32:50.851 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:32:51.109 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:32:51.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:51 smithi196 ceph-mon[84784]: pgmap v1415: 83 pgs: 83 active+clean; 4.6 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-21T10:32:51.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:51 smithi098 ceph-mon[97580]: pgmap v1415: 83 pgs: 83 active+clean; 4.6 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-21T10:32:51.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:51 smithi098 ceph-mon[100914]: pgmap v1415: 83 pgs: 83 active+clean; 4.6 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-21T10:32:52.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:52.458 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:52.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:52.825 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:32:52.826 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:32:52.826 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-21T10:32:52.826 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-21T10:32:52.826 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:32:53.419 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-21T10:32:53.476 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:32:53.721 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:32:53.795 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:53 smithi098 ceph-mon[97580]: pgmap v1416: 83 pgs: 83 active+clean; 4.6 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:32:53.796 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:53 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3800221429' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:32:53.796 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:53 smithi098 ceph-mon[100914]: pgmap v1416: 83 pgs: 83 active+clean; 4.6 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:32:53.796 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:53 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3800221429' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:32:53.842 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:53 smithi196 ceph-mon[84784]: pgmap v1416: 83 pgs: 83 active+clean; 4.6 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:32:53.842 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:53 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3800221429' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:32:54.219 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T10:32:54.473 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:54.732 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:54 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3800221429' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:32:54.732 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:54 smithi098 ceph-mon[97580]: osdmap e409: 8 total, 8 up, 8 in 2024-09-21T10:32:54.733 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:54 smithi098 ceph-mon[97580]: osdmap e410: 8 total, 8 up, 8 in 2024-09-21T10:32:54.733 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:54 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3800221429' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:32:54.733 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:54 smithi098 ceph-mon[100914]: osdmap e409: 8 total, 8 up, 8 in 2024-09-21T10:32:54.733 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:54 smithi098 ceph-mon[100914]: osdmap e410: 8 total, 8 up, 8 in 2024-09-21T10:32:54.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:54 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3800221429' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:32:54.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:54 smithi196 ceph-mon[84784]: osdmap e409: 8 total, 8 up, 8 in 2024-09-21T10:32:54.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:54 smithi196 ceph-mon[84784]: osdmap e410: 8 total, 8 up, 8 in 2024-09-21T10:32:55.329 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:32:55.329 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-21T10:32:55.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-21T10:32:55.329 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:32:55.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.329 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46025.0:265405 9.f e345/300 9:fbed789a:::benchmark_data_smithi098_254407_object132674:head v 345'16721, mlcod=345'16719)", 2024-09-21T10:32:55.329 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:16.864281+0000", 2024-09-21T10:32:55.329 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 398.45699219099998, 2024-09-21T10:32:55.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.1915328, 2024-09-21T10:32:55.330 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:55.330 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:55.330 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:55.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:55.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864281+0000", 2024-09-21T10:32:55.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:55.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864281+0000", 2024-09-21T10:32:55.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:55.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864280+0000", 2024-09-21T10:32:55.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:32:55.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:55.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864291+0000", 2024-09-21T10:32:55.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0149999999999999e-05 2024-09-21T10:32:55.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:55.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864302+0000", 2024-09-21T10:32:55.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1776e-05 2024-09-21T10:32:55.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:55.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864308+0000", 2024-09-21T10:32:55.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4360000000000001e-06 2024-09-21T10:32:55.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:55.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864476+0000", 2024-09-21T10:32:55.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016844599999999999 2024-09-21T10:32:55.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:55.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864494+0000", 2024-09-21T10:32:55.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.774e-05 2024-09-21T10:32:55.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:55.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.055776+0000", 2024-09-21T10:32:55.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.191282441 2024-09-21T10:32:55.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:55.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.055814+0000", 2024-09-21T10:32:55.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7656e-05 2024-09-21T10:32:55.333 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.333 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:55.333 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.334 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46025.0:265406 9.7 e345/300 9:e704644c:::benchmark_data_smithi098_254407_object132675:head v 345'16540, mlcod=345'16538)", 2024-09-21T10:32:55.334 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:16.864772+0000", 2024-09-21T10:32:55.334 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 398.45650167399998, 2024-09-21T10:32:55.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.197820207, 2024-09-21T10:32:55.334 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:55.334 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:55.334 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:55.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:55.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864772+0000", 2024-09-21T10:32:55.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:55.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864772+0000", 2024-09-21T10:32:55.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:55.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864771+0000", 2024-09-21T10:32:55.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:32:55.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:55.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864778+0000", 2024-09-21T10:32:55.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5679999999999996e-06 2024-09-21T10:32:55.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:55.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864789+0000", 2024-09-21T10:32:55.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0973e-05 2024-09-21T10:32:55.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:55.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864794+0000", 2024-09-21T10:32:55.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4299999999999997e-06 2024-09-21T10:32:55.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:55.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.868695+0000", 2024-09-21T10:32:55.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039012629999999999 2024-09-21T10:32:55.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:55.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.868714+0000", 2024-09-21T10:32:55.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8445999999999999e-05 2024-09-21T10:32:55.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:55.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.061488+0000", 2024-09-21T10:32:55.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.192774272 2024-09-21T10:32:55.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:55.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.062592+0000", 2024-09-21T10:32:55.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001103978 2024-09-21T10:32:55.338 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.338 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:55.338 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.338 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46025.0:265412 9.b e345/300 9:de773c0c:::benchmark_data_smithi098_254407_object132681:head v 345'16485, mlcod=345'16483)", 2024-09-21T10:32:55.338 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:16.868296+0000", 2024-09-21T10:32:55.338 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 398.45297799899998, 2024-09-21T10:32:55.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.19568555100000001, 2024-09-21T10:32:55.338 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:55.338 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:55.338 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:55.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:55.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.868296+0000", 2024-09-21T10:32:55.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:55.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.868296+0000", 2024-09-21T10:32:55.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:55.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.868295+0000", 2024-09-21T10:32:55.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:32:55.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:55.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.868302+0000", 2024-09-21T10:32:55.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1930000000000004e-06 2024-09-21T10:32:55.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:55.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.868314+0000", 2024-09-21T10:32:55.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2401e-05 2024-09-21T10:32:55.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:55.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.868320+0000", 2024-09-21T10:32:55.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.105e-06 2024-09-21T10:32:55.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:55.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.868908+0000", 2024-09-21T10:32:55.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00058771599999999995 2024-09-21T10:32:55.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:55.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.868927+0000", 2024-09-21T10:32:55.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8975999999999999e-05 2024-09-21T10:32:55.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:55.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.063951+0000", 2024-09-21T10:32:55.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.19502398000000001 2024-09-21T10:32:55.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:55.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.063981+0000", 2024-09-21T10:32:55.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0043999999999999e-05 2024-09-21T10:32:55.342 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.342 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:55.342 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.342 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46025.0:265413 9.f e345/300 9:fc236b48:::benchmark_data_smithi098_254407_object132682:head v 345'16722, mlcod=345'16719)", 2024-09-21T10:32:55.342 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:16.870528+0000", 2024-09-21T10:32:55.342 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 398.45074511000001, 2024-09-21T10:32:55.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.19215579699999999, 2024-09-21T10:32:55.343 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:55.343 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:55.343 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:55.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:55.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.870528+0000", 2024-09-21T10:32:55.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:55.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.870528+0000", 2024-09-21T10:32:55.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:55.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.870528+0000", 2024-09-21T10:32:55.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:32:55.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:55.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.870537+0000", 2024-09-21T10:32:55.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5440000000000004e-06 2024-09-21T10:32:55.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.345 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:55.345 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.870548+0000", 2024-09-21T10:32:55.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0698e-05 2024-09-21T10:32:55.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.345 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:55.345 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.870553+0000", 2024-09-21T10:32:55.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4639999999999997e-06 2024-09-21T10:32:55.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.345 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:55.345 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.871443+0000", 2024-09-21T10:32:55.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00089001499999999997 2024-09-21T10:32:55.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:55.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.871486+0000", 2024-09-21T10:32:55.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2635000000000001e-05 2024-09-21T10:32:55.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:55.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.062597+0000", 2024-09-21T10:32:55.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.19111130800000001 2024-09-21T10:32:55.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:55.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.062684+0000", 2024-09-21T10:32:55.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6970000000000002e-05 2024-09-21T10:32:55.347 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.347 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:55.347 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.347 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46025.0:265414 9.b e345/300 9:d9a3d866:::benchmark_data_smithi098_254407_object132683:head v 345'16486, mlcod=345'16483)", 2024-09-21T10:32:55.347 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:16.872315+0000", 2024-09-21T10:32:55.347 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 398.44895847700002, 2024-09-21T10:32:55.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.19168617199999999, 2024-09-21T10:32:55.347 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:55.347 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:55.347 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:55.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:55.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.872315+0000", 2024-09-21T10:32:55.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.348 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:55.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.872315+0000", 2024-09-21T10:32:55.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.348 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:55.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.872314+0000", 2024-09-21T10:32:55.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:32:55.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:55.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.872322+0000", 2024-09-21T10:32:55.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8599999999999993e-06 2024-09-21T10:32:55.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:55.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.049348+0000", 2024-09-21T10:32:55.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.177025666 2024-09-21T10:32:55.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:55.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.049356+0000", 2024-09-21T10:32:55.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7830000000000004e-06 2024-09-21T10:32:55.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:55.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.049395+0000", 2024-09-21T10:32:55.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8352000000000001e-05 2024-09-21T10:32:55.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:55.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.049423+0000", 2024-09-21T10:32:55.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7923999999999999e-05 2024-09-21T10:32:55.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:55.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.063985+0000", 2024-09-21T10:32:55.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014562235 2024-09-21T10:32:55.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:55.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.064001+0000", 2024-09-21T10:32:55.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6279000000000001e-05 2024-09-21T10:32:55.351 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.351 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:55.351 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.351 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46322.0:194939 10.9 10:9b661029:::benchmark_data_smithi098_281896_object69914:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T10:32:55.351 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:29:59.331976+0000", 2024-09-21T10:32:55.351 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 175.989297929, 2024-09-21T10:32:55.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.19444286599999999, 2024-09-21T10:32:55.351 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:55.352 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:55.352 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:55.352 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46322", 2024-09-21T10:32:55.352 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:55.352 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 194939 2024-09-21T10:32:55.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.352 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:55.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:55.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.331976+0000", 2024-09-21T10:32:55.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:55.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.331976+0000", 2024-09-21T10:32:55.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:55.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.331974+0000", 2024-09-21T10:32:55.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:32:55.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:55.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.331980+0000", 2024-09-21T10:32:55.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1970000000000004e-06 2024-09-21T10:32:55.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:55.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.331989+0000", 2024-09-21T10:32:55.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4429999999999992e-06 2024-09-21T10:32:55.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:55.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.331994+0000", 2024-09-21T10:32:55.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2549999999999997e-06 2024-09-21T10:32:55.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:55.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.338840+0000", 2024-09-21T10:32:55.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0068451099999999997 2024-09-21T10:32:55.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:55.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.338955+0000", 2024-09-21T10:32:55.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011561999999999999 2024-09-21T10:32:55.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:32:55.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.339078+0000", 2024-09-21T10:32:55.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012290599999999999 2024-09-21T10:32:55.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:32:55.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.353351+0000", 2024-09-21T10:32:55.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014273084 2024-09-21T10:32:55.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.356 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:55.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.353394+0000", 2024-09-21T10:32:55.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2914000000000003e-05 2024-09-21T10:32:55.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.356 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:55.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.526304+0000", 2024-09-21T10:32:55.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17290995200000001 2024-09-21T10:32:55.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.356 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:55.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.526379+0000", 2024-09-21T10:32:55.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5011000000000004e-05 2024-09-21T10:32:55.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:55.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.526418+0000", 2024-09-21T10:32:55.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9458000000000001e-05 2024-09-21T10:32:55.357 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.357 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:55.357 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.357 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46322.0:194943 10.9 10:9783ad8e:::benchmark_data_smithi098_281896_object69918:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T10:32:55.357 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:29:59.338985+0000", 2024-09-21T10:32:55.357 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 175.98228805799999, 2024-09-21T10:32:55.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.187547094, 2024-09-21T10:32:55.357 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:55.358 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:55.358 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:55.358 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46322", 2024-09-21T10:32:55.358 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:55.358 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 194943 2024-09-21T10:32:55.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.358 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:55.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:55.358 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.338985+0000", 2024-09-21T10:32:55.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:55.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.338985+0000", 2024-09-21T10:32:55.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:55.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.338984+0000", 2024-09-21T10:32:55.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:32:55.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:55.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.338991+0000", 2024-09-21T10:32:55.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.64e-06 2024-09-21T10:32:55.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:55.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.339002+0000", 2024-09-21T10:32:55.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1695999999999999e-05 2024-09-21T10:32:55.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:55.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.339011+0000", 2024-09-21T10:32:55.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0319999999999992e-06 2024-09-21T10:32:55.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:55.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.343315+0000", 2024-09-21T10:32:55.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043039319999999999 2024-09-21T10:32:55.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:55.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.343406+0000", 2024-09-21T10:32:55.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0449000000000001e-05 2024-09-21T10:32:55.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:32:55.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.343532+0000", 2024-09-21T10:32:55.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000126129 2024-09-21T10:32:55.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:32:55.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.353885+0000", 2024-09-21T10:32:55.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.01035296 2024-09-21T10:32:55.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:55.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.357745+0000", 2024-09-21T10:32:55.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00386031 2024-09-21T10:32:55.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:55.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.526467+0000", 2024-09-21T10:32:55.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16872199199999999 2024-09-21T10:32:55.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:55.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.526502+0000", 2024-09-21T10:32:55.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5256000000000003e-05 2024-09-21T10:32:55.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:55.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:29:59.526533+0000", 2024-09-21T10:32:55.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0057e-05 2024-09-21T10:32:55.363 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.363 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:55.363 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.363 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46322.0:195635 10.9 10:9a6c7f51:::benchmark_data_smithi098_281896_object70610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T10:32:55.363 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:30:00.554996+0000", 2024-09-21T10:32:55.363 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 174.766277301, 2024-09-21T10:32:55.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.22136946499999999, 2024-09-21T10:32:55.364 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:55.364 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:55.364 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:55.364 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46322", 2024-09-21T10:32:55.364 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:55.364 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 195635 2024-09-21T10:32:55.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.364 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:55.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:55.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.554996+0000", 2024-09-21T10:32:55.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:55.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.554996+0000", 2024-09-21T10:32:55.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:55.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.554995+0000", 2024-09-21T10:32:55.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:32:55.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:55.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.555000+0000", 2024-09-21T10:32:55.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7899999999999996e-06 2024-09-21T10:32:55.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:55.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.555010+0000", 2024-09-21T10:32:55.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9280000000000005e-06 2024-09-21T10:32:55.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:55.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.555015+0000", 2024-09-21T10:32:55.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1620000000000001e-06 2024-09-21T10:32:55.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:55.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.555048+0000", 2024-09-21T10:32:55.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3099999999999998e-05 2024-09-21T10:32:55.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:55.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.555138+0000", 2024-09-21T10:32:55.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0708e-05 2024-09-21T10:32:55.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:32:55.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.555204+0000", 2024-09-21T10:32:55.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5882000000000004e-05 2024-09-21T10:32:55.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:55.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.559549+0000", 2024-09-21T10:32:55.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043445710000000002 2024-09-21T10:32:55.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:32:55.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.559648+0000", 2024-09-21T10:32:55.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9276000000000003e-05 2024-09-21T10:32:55.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:55.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776279+0000", 2024-09-21T10:32:55.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.21663055000000001 2024-09-21T10:32:55.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:55.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776328+0000", 2024-09-21T10:32:55.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9320000000000002e-05 2024-09-21T10:32:55.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:55.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776366+0000", 2024-09-21T10:32:55.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7744000000000003e-05 2024-09-21T10:32:55.369 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.369 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:55.369 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.369 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46322.0:195640 10.9 10:9d2edc51:::benchmark_data_smithi098_281896_object70615:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T10:32:55.369 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:30:00.559661+0000", 2024-09-21T10:32:55.370 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 174.76161217999999, 2024-09-21T10:32:55.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.216796662, 2024-09-21T10:32:55.370 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:55.370 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:55.370 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:55.371 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46322", 2024-09-21T10:32:55.371 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:55.371 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 195640 2024-09-21T10:32:55.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.371 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:55.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.371 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:55.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.559661+0000", 2024-09-21T10:32:55.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:55.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.559661+0000", 2024-09-21T10:32:55.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:55.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.559660+0000", 2024-09-21T10:32:55.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:32:55.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:55.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.559667+0000", 2024-09-21T10:32:55.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3549999999999999e-06 2024-09-21T10:32:55.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:55.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.559676+0000", 2024-09-21T10:32:55.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2400000000000007e-06 2024-09-21T10:32:55.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:55.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.559679+0000", 2024-09-21T10:32:55.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7740000000000002e-06 2024-09-21T10:32:55.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:55.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.559715+0000", 2024-09-21T10:32:55.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5809999999999998e-05 2024-09-21T10:32:55.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:55.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.559799+0000", 2024-09-21T10:32:55.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3230000000000001e-05 2024-09-21T10:32:55.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:32:55.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.559868+0000", 2024-09-21T10:32:55.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9027000000000002e-05 2024-09-21T10:32:55.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:32:55.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.563779+0000", 2024-09-21T10:32:55.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039112770000000003 2024-09-21T10:32:55.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:55.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.564847+0000", 2024-09-21T10:32:55.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001068484 2024-09-21T10:32:55.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:55.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776398+0000", 2024-09-21T10:32:55.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.21155064600000001 2024-09-21T10:32:55.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:55.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776431+0000", 2024-09-21T10:32:55.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2870000000000002e-05 2024-09-21T10:32:55.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:55.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776458+0000", 2024-09-21T10:32:55.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7163000000000001e-05 2024-09-21T10:32:55.376 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.376 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:55.376 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.377 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46322.0:195637 10.3 e379/361 10:cd271909:::benchmark_data_smithi098_281896_object70612:head v 379'12340, mlcod=0'0)", 2024-09-21T10:32:55.377 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:30:00.565066+0000", 2024-09-21T10:32:55.377 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 174.756207437, 2024-09-21T10:32:55.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.21147101800000001, 2024-09-21T10:32:55.377 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:55.377 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:55.377 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:55.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:55.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.565066+0000", 2024-09-21T10:32:55.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:55.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.565066+0000", 2024-09-21T10:32:55.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:55.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.565066+0000", 2024-09-21T10:32:55.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:32:55.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:55.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.565073+0000", 2024-09-21T10:32:55.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8499999999999994e-06 2024-09-21T10:32:55.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:55.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.565079+0000", 2024-09-21T10:32:55.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4249999999999997e-06 2024-09-21T10:32:55.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:55.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.565082+0000", 2024-09-21T10:32:55.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9299999999999999e-06 2024-09-21T10:32:55.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:55.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.565119+0000", 2024-09-21T10:32:55.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7064000000000003e-05 2024-09-21T10:32:55.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:55.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.565140+0000", 2024-09-21T10:32:55.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0828000000000001e-05 2024-09-21T10:32:55.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:55.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776215+0000", 2024-09-21T10:32:55.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.21107503999999999 2024-09-21T10:32:55.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:55.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776537+0000", 2024-09-21T10:32:55.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00032236299999999999 2024-09-21T10:32:55.381 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.381 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:55.381 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.381 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46322.0:195645 10.9 10:96a1b65f:::benchmark_data_smithi098_281896_object70620:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T10:32:55.381 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:30:00.565625+0000", 2024-09-21T10:32:55.381 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 174.75564851499999, 2024-09-21T10:32:55.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.219768823, 2024-09-21T10:32:55.381 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:55.381 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:55.381 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:55.382 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46322", 2024-09-21T10:32:55.382 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:55.382 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 195645 2024-09-21T10:32:55.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.382 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:55.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:55.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.565625+0000", 2024-09-21T10:32:55.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:55.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.565625+0000", 2024-09-21T10:32:55.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:55.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.565624+0000", 2024-09-21T10:32:55.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:32:55.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:55.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.565629+0000", 2024-09-21T10:32:55.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2889999999999997e-06 2024-09-21T10:32:55.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:55.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.565638+0000", 2024-09-21T10:32:55.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.799e-06 2024-09-21T10:32:55.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:55.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.565642+0000", 2024-09-21T10:32:55.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7189999999999999e-06 2024-09-21T10:32:55.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:55.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.565682+0000", 2024-09-21T10:32:55.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0250999999999998e-05 2024-09-21T10:32:55.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:55.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.565782+0000", 2024-09-21T10:32:55.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9897999999999998e-05 2024-09-21T10:32:55.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:32:55.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.565889+0000", 2024-09-21T10:32:55.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107297 2024-09-21T10:32:55.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:55.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776241+0000", 2024-09-21T10:32:55.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.21035234899999999 2024-09-21T10:32:55.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:55.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776567+0000", 2024-09-21T10:32:55.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00032547700000000001 2024-09-21T10:32:55.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:32:55.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.785289+0000", 2024-09-21T10:32:55.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0087221509999999992 2024-09-21T10:32:55.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:55.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.785342+0000", 2024-09-21T10:32:55.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3399999999999997e-05 2024-09-21T10:32:55.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:55.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.785394+0000", 2024-09-21T10:32:55.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1427000000000001e-05 2024-09-21T10:32:55.387 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.387 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:55.387 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.387 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46322.0:195639 10.3 e379/361 10:c5c325f2:::benchmark_data_smithi098_281896_object70614:head v 379'12341, mlcod=0'0)", 2024-09-21T10:32:55.387 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:30:00.568416+0000", 2024-09-21T10:32:55.387 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 174.752857017, 2024-09-21T10:32:55.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.20814490999999999, 2024-09-21T10:32:55.387 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:55.387 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:55.388 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:55.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:55.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.568416+0000", 2024-09-21T10:32:55.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:55.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.568416+0000", 2024-09-21T10:32:55.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:55.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.568416+0000", 2024-09-21T10:32:55.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:32:55.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:55.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.568425+0000", 2024-09-21T10:32:55.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2669999999999995e-06 2024-09-21T10:32:55.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:55.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.568433+0000", 2024-09-21T10:32:55.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2900000000000002e-06 2024-09-21T10:32:55.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:55.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.568438+0000", 2024-09-21T10:32:55.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4580000000000003e-06 2024-09-21T10:32:55.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:55.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.569211+0000", 2024-09-21T10:32:55.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00077365599999999995 2024-09-21T10:32:55.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:55.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.569236+0000", 2024-09-21T10:32:55.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4258000000000001e-05 2024-09-21T10:32:55.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:55.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776541+0000", 2024-09-21T10:32:55.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.207305604 2024-09-21T10:32:55.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:55.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776561+0000", 2024-09-21T10:32:55.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0152e-05 2024-09-21T10:32:55.391 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.391 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:55.391 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.391 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46322.0:195644 10.6 e379/348 10:60a40b40:::benchmark_data_smithi098_281896_object70619:head v 379'12305, mlcod=379'12303)", 2024-09-21T10:32:55.391 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:30:00.569208+0000", 2024-09-21T10:32:55.392 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 174.75206529799999, 2024-09-21T10:32:55.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.207049549, 2024-09-21T10:32:55.392 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:55.392 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:55.392 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:55.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:55.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.569208+0000", 2024-09-21T10:32:55.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:55.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.569208+0000", 2024-09-21T10:32:55.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:55.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.569207+0000", 2024-09-21T10:32:55.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:32:55.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:55.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.569214+0000", 2024-09-21T10:32:55.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2389999999999999e-06 2024-09-21T10:32:55.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:55.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.569223+0000", 2024-09-21T10:32:55.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3270000000000007e-06 2024-09-21T10:32:55.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:55.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.569228+0000", 2024-09-21T10:32:55.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5730000000000001e-06 2024-09-21T10:32:55.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:55.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.569270+0000", 2024-09-21T10:32:55.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1965999999999998e-05 2024-09-21T10:32:55.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:55.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.569287+0000", 2024-09-21T10:32:55.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7407000000000001e-05 2024-09-21T10:32:55.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:55.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776228+0000", 2024-09-21T10:32:55.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.20694114599999999 2024-09-21T10:32:55.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:55.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776258+0000", 2024-09-21T10:32:55.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9641999999999998e-05 2024-09-21T10:32:55.395 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.395 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:55.395 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.46322.0:195645 10.9 e379/351 ondisk, result = 0)", 2024-09-21T10:32:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:30:00.569992+0000", 2024-09-21T10:32:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 174.75128117599999, 2024-09-21T10:32:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.206251084, 2024-09-21T10:32:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:32:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.569992+0000", 2024-09-21T10:32:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.569992+0000", 2024-09-21T10:32:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.569992+0000", 2024-09-21T10:32:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:32:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.569996+0000", 2024-09-21T10:32:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6959999999999996e-06 2024-09-21T10:32:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.570006+0000", 2024-09-21T10:32:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7929999999999992e-06 2024-09-21T10:32:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.570011+0000", 2024-09-21T10:32:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5000000000000001e-06 2024-09-21T10:32:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776226+0000", 2024-09-21T10:32:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.206215484 2024-09-21T10:32:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776233+0000", 2024-09-21T10:32:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9709999999999998e-06 2024-09-21T10:32:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776243+0000", 2024-09-21T10:32:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0326000000000001e-05 2024-09-21T10:32:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46322.0:195646 10.7 e379/361 10:eabf7495:::benchmark_data_smithi098_281896_object70621:head v 379'12029, mlcod=379'12025)", 2024-09-21T10:32:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:30:00.571448+0000", 2024-09-21T10:32:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 174.749825222, 2024-09-21T10:32:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.21636483300000001, 2024-09-21T10:32:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.571448+0000", 2024-09-21T10:32:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.571448+0000", 2024-09-21T10:32:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.571448+0000", 2024-09-21T10:32:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:32:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.571454+0000", 2024-09-21T10:32:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8800000000000002e-06 2024-09-21T10:32:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776090+0000", 2024-09-21T10:32:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.204635548 2024-09-21T10:32:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776107+0000", 2024-09-21T10:32:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6860000000000001e-05 2024-09-21T10:32:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776152+0000", 2024-09-21T10:32:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5487e-05 2024-09-21T10:32:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776182+0000", 2024-09-21T10:32:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9274000000000001e-05 2024-09-21T10:32:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.787738+0000", 2024-09-21T10:32:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011556326 2024-09-21T10:32:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.787813+0000", 2024-09-21T10:32:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5240999999999993e-05 2024-09-21T10:32:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.46322.0:195635 10.9 e379/351 ondisk, result = 0)", 2024-09-21T10:32:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:30:00.581466+0000", 2024-09-21T10:32:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 174.73980769100001, 2024-09-21T10:32:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.19490321599999999, 2024-09-21T10:32:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:32:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.581466+0000", 2024-09-21T10:32:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.581466+0000", 2024-09-21T10:32:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.581465+0000", 2024-09-21T10:32:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:32:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.581472+0000", 2024-09-21T10:32:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5480000000000004e-06 2024-09-21T10:32:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776151+0000", 2024-09-21T10:32:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.19467864800000001 2024-09-21T10:32:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776159+0000", 2024-09-21T10:32:55.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7419999999999997e-06 2024-09-21T10:32:55.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:55.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776267+0000", 2024-09-21T10:32:55.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010837599999999999 2024-09-21T10:32:55.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:55.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776274+0000", 2024-09-21T10:32:55.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6050000000000003e-06 2024-09-21T10:32:55.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:55.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:00.776369+0000", 2024-09-21T10:32:55.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5413000000000003e-05 2024-09-21T10:32:55.408 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.408 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:55.408 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.408 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46322.0:201752 10.9 10:90baaaa4:::benchmark_data_smithi098_281896_object76727:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T10:32:55.408 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:30:09.574690+0000", 2024-09-21T10:32:55.408 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 165.74658351400001, 2024-09-21T10:32:55.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.27707227899999998, 2024-09-21T10:32:55.408 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:55.408 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:55.408 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:32:55.409 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46322", 2024-09-21T10:32:55.409 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:32:55.409 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 201752 2024-09-21T10:32:55.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.409 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:55.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:55.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.574690+0000", 2024-09-21T10:32:55.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:55.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.574690+0000", 2024-09-21T10:32:55.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:55.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.574689+0000", 2024-09-21T10:32:55.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:32:55.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:55.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.574694+0000", 2024-09-21T10:32:55.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0549999999999999e-06 2024-09-21T10:32:55.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:55.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.574703+0000", 2024-09-21T10:32:55.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4369999999999996e-06 2024-09-21T10:32:55.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:55.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.574707+0000", 2024-09-21T10:32:55.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9709999999999998e-06 2024-09-21T10:32:55.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:55.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.574740+0000", 2024-09-21T10:32:55.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2212000000000003e-05 2024-09-21T10:32:55.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:55.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.574824+0000", 2024-09-21T10:32:55.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4633000000000001e-05 2024-09-21T10:32:55.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:32:55.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.574915+0000", 2024-09-21T10:32:55.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0890999999999994e-05 2024-09-21T10:32:55.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:32:55.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.578667+0000", 2024-09-21T10:32:55.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0037514359999999999 2024-09-21T10:32:55.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:55.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.579263+0000", 2024-09-21T10:32:55.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00059672900000000005 2024-09-21T10:32:55.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:55.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.590056+0000", 2024-09-21T10:32:55.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010792234 2024-09-21T10:32:55.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:55.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.590087+0000", 2024-09-21T10:32:55.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1198999999999997e-05 2024-09-21T10:32:55.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:55.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.851762+0000", 2024-09-21T10:32:55.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.26167555300000001 2024-09-21T10:32:55.414 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.414 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:55.414 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.414 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.46322.0:201752 10.9 e380/351 ondisk, result = 0)", 2024-09-21T10:32:55.414 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:30:09.589987+0000", 2024-09-21T10:32:55.414 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 165.73128601600001, 2024-09-21T10:32:55.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.26178268500000001, 2024-09-21T10:32:55.415 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:55.415 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-21T10:32:55.415 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:55.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:55.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.589987+0000", 2024-09-21T10:32:55.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:55.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.589987+0000", 2024-09-21T10:32:55.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:55.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.589986+0000", 2024-09-21T10:32:55.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:32:55.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:55.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.589992+0000", 2024-09-21T10:32:55.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0290000000000002e-06 2024-09-21T10:32:55.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:55.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.590006+0000", 2024-09-21T10:32:55.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3327e-05 2024-09-21T10:32:55.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:55.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.590012+0000", 2024-09-21T10:32:55.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3300000000000004e-06 2024-09-21T10:32:55.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:55.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.590046+0000", 2024-09-21T10:32:55.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4091999999999998e-05 2024-09-21T10:32:55.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:55.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.590052+0000", 2024-09-21T10:32:55.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4990000000000002e-06 2024-09-21T10:32:55.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:55.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.851770+0000", 2024-09-21T10:32:55.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.26171851200000001 2024-09-21T10:32:55.418 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.418 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:55.418 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.418 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46322.0:201762 10.3 e380/361 10:c0dfe06a:::benchmark_data_smithi098_281896_object76737:head v 380'12711, mlcod=0'0)", 2024-09-21T10:32:55.418 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:30:09.595886+0000", 2024-09-21T10:32:55.418 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 165.72538716700001, 2024-09-21T10:32:55.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.26560369900000003, 2024-09-21T10:32:55.419 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:55.419 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:55.419 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:55.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:55.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.595886+0000", 2024-09-21T10:32:55.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:55.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.595886+0000", 2024-09-21T10:32:55.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:55.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.595885+0000", 2024-09-21T10:32:55.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:32:55.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:55.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.595894+0000", 2024-09-21T10:32:55.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1600000000000004e-06 2024-09-21T10:32:55.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:55.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.851740+0000", 2024-09-21T10:32:55.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.25584540300000003 2024-09-21T10:32:55.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:55.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.851746+0000", 2024-09-21T10:32:55.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9429999999999997e-06 2024-09-21T10:32:55.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:55.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.851797+0000", 2024-09-21T10:32:55.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1078999999999997e-05 2024-09-21T10:32:55.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:55.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.851824+0000", 2024-09-21T10:32:55.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7313000000000001e-05 2024-09-21T10:32:55.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:55.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.861428+0000", 2024-09-21T10:32:55.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0096035559999999992 2024-09-21T10:32:55.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:55.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.861490+0000", 2024-09-21T10:32:55.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2317999999999995e-05 2024-09-21T10:32:55.422 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.422 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:55.422 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.423 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.46322.0:201765 10.a e380/348 10:529f12b0:::benchmark_data_smithi098_281896_object76740:head v 380'12808, mlcod=380'12806)", 2024-09-21T10:32:55.423 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:30:09.596678+0000", 2024-09-21T10:32:55.423 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 165.724595255, 2024-09-21T10:32:55.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.27115971100000003, 2024-09-21T10:32:55.423 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:32:55.423 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:55.423 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:32:55.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:32:55.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.596678+0000", 2024-09-21T10:32:55.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:32:55.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.596678+0000", 2024-09-21T10:32:55.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:32:55.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:32:55.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.596678+0000", 2024-09-21T10:32:55.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:32:55.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:32:55.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.596684+0000", 2024-09-21T10:32:55.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4760000000000003e-06 2024-09-21T10:32:55.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:32:55.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.851726+0000", 2024-09-21T10:32:55.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.25504187099999998 2024-09-21T10:32:55.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:32:55.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.851741+0000", 2024-09-21T10:32:55.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5054e-05 2024-09-21T10:32:55.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:32:55.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.851806+0000", 2024-09-21T10:32:55.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4917000000000002e-05 2024-09-21T10:32:55.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:32:55.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.851824+0000", 2024-09-21T10:32:55.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8138000000000002e-05 2024-09-21T10:32:55.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:32:55.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.867798+0000", 2024-09-21T10:32:55.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015973566000000002 2024-09-21T10:32:55.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:32:55.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:32:55.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:32:55.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.867838+0000", 2024-09-21T10:32:55.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0417000000000003e-05 2024-09-21T10:32:55.427 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.427 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:55.427 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.427 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:32:55.427 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:32:55.427 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:32:55.537 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:55 smithi098 ceph-mon[100914]: pgmap v1418: 83 pgs: 83 active+clean; 4.6 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:32:55.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:55 smithi098 ceph-mon[97580]: pgmap v1418: 83 pgs: 83 active+clean; 4.6 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:32:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:55 smithi196 ceph-mon[84784]: pgmap v1418: 83 pgs: 83 active+clean; 4.6 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:32:56.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:32:56.447 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T10:32:56.672 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:32:56.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:56 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3925430435' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:32:56.821 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:56 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3925430435' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:32:56.821 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:56 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3925430435' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:32:57.196 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 1 1 2024-09-21T10:32:57.443 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:57.716 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:57 smithi098 ceph-mon[97580]: pgmap v1420: 83 pgs: 83 active+clean; 4.6 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:32:57.716 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:57 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3925430435' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:32:57.716 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:57 smithi098 ceph-mon[97580]: osdmap e411: 8 total, 8 up, 8 in 2024-09-21T10:32:57.717 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:57 smithi098 ceph-mon[100914]: pgmap v1420: 83 pgs: 83 active+clean; 4.6 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:32:57.717 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:57 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3925430435' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:32:57.717 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:57 smithi098 ceph-mon[100914]: osdmap e411: 8 total, 8 up, 8 in 2024-09-21T10:32:57.834 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:57 smithi196 ceph-mon[84784]: pgmap v1420: 83 pgs: 83 active+clean; 4.6 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:32:57.834 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:57 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3925430435' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:32:57.835 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:57 smithi196 ceph-mon[84784]: osdmap e411: 8 total, 8 up, 8 in 2024-09-21T10:32:58.199 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:32:58.201 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:32:58.201 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-21T10:32:58.202 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:32:58.750 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:32:58.993 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:32:59.454 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-21T10:32:59.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:59 smithi196 ceph-mon[84784]: pgmap v1422: 83 pgs: 83 active+clean; 4.6 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:32:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:59 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2122295395' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:32:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:32:59 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:32:59.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:59 smithi098 ceph-mon[97580]: pgmap v1422: 83 pgs: 83 active+clean; 4.6 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:32:59.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:59 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2122295395' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:32:59.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:32:59 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:32:59.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:59 smithi098 ceph-mon[100914]: pgmap v1422: 83 pgs: 83 active+clean; 4.6 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:32:59.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:59 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2122295395' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:32:59.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:32:59 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:33:00.178 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T10:33:00.422 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:00.455 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:33:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:33:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:33:00.712 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:00 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:33:00.713 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:00 smithi196 ceph-mon[84784]: osdmap e412: 8 total, 8 up, 8 in 2024-09-21T10:33:00.721 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:33:00.721 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:33:00.722 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-21T10:33:00.722 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-21T10:33:00.722 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:33:00.794 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:33:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:33:00] "GET /metrics HTTP/1.1" 200 34164 "" "Prometheus/2.51.0" 2024-09-21T10:33:00.794 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:00 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:33:00.794 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:00 smithi098 ceph-mon[97580]: osdmap e412: 8 total, 8 up, 8 in 2024-09-21T10:33:00.794 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:00 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:33:00.795 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:00 smithi098 ceph-mon[100914]: osdmap e412: 8 total, 8 up, 8 in 2024-09-21T10:33:01.401 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:33:01.515 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:01 smithi098 ceph-mon[97580]: pgmap v1424: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:01.515 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:01 smithi098 ceph-mon[100914]: pgmap v1424: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:01.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:01 smithi196 ceph-mon[84784]: pgmap v1424: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:01.648 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:33:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T10:33:02.772 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:02 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/53064244' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:33:02.773 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:02 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/53064244' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:33:02.849 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:02 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/53064244' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:33:03.185 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 2 1 2024-09-21T10:33:03.218 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:33:03.218 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-21T10:33:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-21T10:33:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:33:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181812 9.1 9:8268420f:::benchmark_data_smithi098_254407_object49081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T10:33:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.268689+0000", 2024-09-21T10:33:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 499.92495295999998, 2024-09-21T10:33:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1774718389999999, 2024-09-21T10:33:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181812 2024-09-21T10:33:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268689+0000", 2024-09-21T10:33:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268689+0000", 2024-09-21T10:33:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268688+0000", 2024-09-21T10:33:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268695+0000", 2024-09-21T10:33:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6420000000000007e-06 2024-09-21T10:33:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268710+0000", 2024-09-21T10:33:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4959e-05 2024-09-21T10:33:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268716+0000", 2024-09-21T10:33:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0290000000000002e-06 2024-09-21T10:33:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.273397+0000", 2024-09-21T10:33:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046805370000000002 2024-09-21T10:33:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.273407+0000", 2024-09-21T10:33:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6970000000000005e-06 2024-09-21T10:33:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.292290+0000", 2024-09-21T10:33:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0188830849999999 2024-09-21T10:33:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.292305+0000", 2024-09-21T10:33:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4841000000000001e-05 2024-09-21T10:33:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.432527+0000", 2024-09-21T10:33:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1402229049999999 2024-09-21T10:33:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.432648+0000", 2024-09-21T10:33:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000120538 2024-09-21T10:33:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,4,7", 2024-09-21T10:33:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.432762+0000", 2024-09-21T10:33:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114478 2024-09-21T10:33:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.440164+0000", 2024-09-21T10:33:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0074017559999999998 2024-09-21T10:33:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.440418+0000", 2024-09-21T10:33:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025371399999999999 2024-09-21T10:33:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.440476+0000", 2024-09-21T10:33:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8146999999999997e-05 2024-09-21T10:33:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.445996+0000", 2024-09-21T10:33:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055200129999999998 2024-09-21T10:33:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.446054+0000", 2024-09-21T10:33:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8400999999999999e-05 2024-09-21T10:33:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.446161+0000", 2024-09-21T10:33:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000106586 2024-09-21T10:33:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181813 9.1 9:877e8199:::benchmark_data_smithi098_254407_object49082:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T10:33:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.268726+0000", 2024-09-21T10:33:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 499.92491590200001, 2024-09-21T10:33:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1922379680000001, 2024-09-21T10:33:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181813 2024-09-21T10:33:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268726+0000", 2024-09-21T10:33:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268726+0000", 2024-09-21T10:33:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268726+0000", 2024-09-21T10:33:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:33:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268728+0000", 2024-09-21T10:33:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7499999999999999e-06 2024-09-21T10:33:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268735+0000", 2024-09-21T10:33:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8700000000000003e-06 2024-09-21T10:33:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.268739+0000", 2024-09-21T10:33:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8419999999999998e-06 2024-09-21T10:33:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.273431+0000", 2024-09-21T10:33:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046914460000000002 2024-09-21T10:33:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.273438+0000", 2024-09-21T10:33:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5809999999999999e-06 2024-09-21T10:33:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.292343+0000", 2024-09-21T10:33:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.018904467 2024-09-21T10:33:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.292352+0000", 2024-09-21T10:33:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0280000000000006e-06 2024-09-21T10:33:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.436686+0000", 2024-09-21T10:33:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1443347740000001 2024-09-21T10:33:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.436801+0000", 2024-09-21T10:33:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011475499999999999 2024-09-21T10:33:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,4,7", 2024-09-21T10:33:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.436931+0000", 2024-09-21T10:33:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012945200000000001 2024-09-21T10:33:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.440424+0000", 2024-09-21T10:33:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003493515 2024-09-21T10:33:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.444139+0000", 2024-09-21T10:33:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0037150170000000001 2024-09-21T10:33:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.454482+0000", 2024-09-21T10:33:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010343140000000001 2024-09-21T10:33:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.460725+0000", 2024-09-21T10:33:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0062427510000000004 2024-09-21T10:33:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.460845+0000", 2024-09-21T10:33:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119721 2024-09-21T10:33:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.460964+0000", 2024-09-21T10:33:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119419 2024-09-21T10:33:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212859 9.1 9:81de2bc9:::benchmark_data_smithi098_254407_object80128:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-21T10:33:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.362389+0000", 2024-09-21T10:33:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 464.83125325999998, 2024-09-21T10:33:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.029750589, 2024-09-21T10:33:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:33:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212859 2024-09-21T10:33:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362389+0000", 2024-09-21T10:33:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362389+0000", 2024-09-21T10:33:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362387+0000", 2024-09-21T10:33:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362395+0000", 2024-09-21T10:33:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6410000000000002e-06 2024-09-21T10:33:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362409+0000", 2024-09-21T10:33:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3872e-05 2024-09-21T10:33:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362415+0000", 2024-09-21T10:33:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0650000000000004e-06 2024-09-21T10:33:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362480+0000", 2024-09-21T10:33:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5152000000000002e-05 2024-09-21T10:33:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362601+0000", 2024-09-21T10:33:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000121327 2024-09-21T10:33:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,4,7", 2024-09-21T10:33:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.362727+0000", 2024-09-21T10:33:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012533800000000001 2024-09-21T10:33:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.369166+0000", 2024-09-21T10:33:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006439274 2024-09-21T10:33:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.369292+0000", 2024-09-21T10:33:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012620200000000001 2024-09-21T10:33:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.369926+0000", 2024-09-21T10:33:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00063336299999999996 2024-09-21T10:33:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.374921+0000", 2024-09-21T10:33:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049951869999999999 2024-09-21T10:33:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.374935+0000", 2024-09-21T10:33:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3872e-05 2024-09-21T10:33:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.392097+0000", 2024-09-21T10:33:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0171621260000001 2024-09-21T10:33:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.392140+0000", 2024-09-21T10:33:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2700000000000001e-05 2024-09-21T10:33:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212886 9.1 9:80fef226:::benchmark_data_smithi098_254407_object80155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:33:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.391416+0000", 2024-09-21T10:33:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 464.80222659399999, 2024-09-21T10:33:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.049473646, 2024-09-21T10:33:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212886 2024-09-21T10:33:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391416+0000", 2024-09-21T10:33:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391416+0000", 2024-09-21T10:33:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391414+0000", 2024-09-21T10:33:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391422+0000", 2024-09-21T10:33:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2369999999999992e-06 2024-09-21T10:33:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391441+0000", 2024-09-21T10:33:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9313999999999998e-05 2024-09-21T10:33:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391448+0000", 2024-09-21T10:33:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8399999999999997e-06 2024-09-21T10:33:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391577+0000", 2024-09-21T10:33:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012900699999999999 2024-09-21T10:33:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.391596+0000", 2024-09-21T10:33:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9448000000000001e-05 2024-09-21T10:33:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.392163+0000", 2024-09-21T10:33:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0005668080000001 2024-09-21T10:33:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.392226+0000", 2024-09-21T10:33:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2736000000000004e-05 2024-09-21T10:33:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:33:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.392302+0000", 2024-09-21T10:33:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6459999999999996e-05 2024-09-21T10:33:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.427385+0000", 2024-09-21T10:33:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.035082852999999997 2024-09-21T10:33:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440761+0000", 2024-09-21T10:33:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013375398 2024-09-21T10:33:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440797+0000", 2024-09-21T10:33:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6238000000000001e-05 2024-09-21T10:33:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440845+0000", 2024-09-21T10:33:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7781999999999998e-05 2024-09-21T10:33:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440889+0000", 2024-09-21T10:33:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4515999999999999e-05 2024-09-21T10:33:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212893 9.1 9:867ca339:::benchmark_data_smithi098_254407_object80162:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:33:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.401222+0000", 2024-09-21T10:33:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 464.79242020800001, 2024-09-21T10:33:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.039903265, 2024-09-21T10:33:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212893 2024-09-21T10:33:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401222+0000", 2024-09-21T10:33:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401222+0000", 2024-09-21T10:33:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401220+0000", 2024-09-21T10:33:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401228+0000", 2024-09-21T10:33:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3100000000000003e-06 2024-09-21T10:33:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401245+0000", 2024-09-21T10:33:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7504999999999999e-05 2024-09-21T10:33:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401252+0000", 2024-09-21T10:33:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5679999999999996e-06 2024-09-21T10:33:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401303+0000", 2024-09-21T10:33:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1232e-05 2024-09-21T10:33:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.401313+0000", 2024-09-21T10:33:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0023999999999999e-05 2024-09-21T10:33:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.395122+0000", 2024-09-21T10:33:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99380883799999997 2024-09-21T10:33:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.395230+0000", 2024-09-21T10:33:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107917 2024-09-21T10:33:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:33:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.395302+0000", 2024-09-21T10:33:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1963000000000002e-05 2024-09-21T10:33:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.427393+0000", 2024-09-21T10:33:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.032091732999999997 2024-09-21T10:33:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440948+0000", 2024-09-21T10:33:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013554639 2024-09-21T10:33:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441060+0000", 2024-09-21T10:33:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011248599999999999 2024-09-21T10:33:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441101+0000", 2024-09-21T10:33:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0738999999999997e-05 2024-09-21T10:33:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441125+0000", 2024-09-21T10:33:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4076000000000001e-05 2024-09-21T10:33:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212896 9.1 9:84f721c8:::benchmark_data_smithi098_254407_object80165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:33:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.404604+0000", 2024-09-21T10:33:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 464.78903781600002, 2024-09-21T10:33:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0367745399999999, 2024-09-21T10:33:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212896 2024-09-21T10:33:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404604+0000", 2024-09-21T10:33:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404604+0000", 2024-09-21T10:33:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404603+0000", 2024-09-21T10:33:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404611+0000", 2024-09-21T10:33:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.619e-06 2024-09-21T10:33:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404629+0000", 2024-09-21T10:33:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8158000000000001e-05 2024-09-21T10:33:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404636+0000", 2024-09-21T10:33:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8249999999999999e-06 2024-09-21T10:33:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404685+0000", 2024-09-21T10:33:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8498000000000002e-05 2024-09-21T10:33:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.404693+0000", 2024-09-21T10:33:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7309999999999999e-06 2024-09-21T10:33:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.397879+0000", 2024-09-21T10:33:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99318610699999998 2024-09-21T10:33:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.398010+0000", 2024-09-21T10:33:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013017000000000001 2024-09-21T10:33:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:33:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.398133+0000", 2024-09-21T10:33:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012345499999999999 2024-09-21T10:33:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.427399+0000", 2024-09-21T10:33:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.029265835 2024-09-21T10:33:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440996+0000", 2024-09-21T10:33:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013597122 2024-09-21T10:33:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441297+0000", 2024-09-21T10:33:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00030115399999999998 2024-09-21T10:33:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441343+0000", 2024-09-21T10:33:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5518e-05 2024-09-21T10:33:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441379+0000", 2024-09-21T10:33:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6149999999999998e-05 2024-09-21T10:33:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212935 9.1 9:83432cba:::benchmark_data_smithi098_254407_object80204:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:33:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.443322+0000", 2024-09-21T10:33:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 464.75032042100003, 2024-09-21T10:33:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99820324299999996, 2024-09-21T10:33:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212935 2024-09-21T10:33:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443322+0000", 2024-09-21T10:33:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443322+0000", 2024-09-21T10:33:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443320+0000", 2024-09-21T10:33:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999976 2024-09-21T10:33:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443328+0000", 2024-09-21T10:33:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0760000000000003e-06 2024-09-21T10:33:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443345+0000", 2024-09-21T10:33:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7479999999999999e-05 2024-09-21T10:33:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443351+0000", 2024-09-21T10:33:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2179999999999996e-06 2024-09-21T10:33:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443503+0000", 2024-09-21T10:33:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015199300000000001 2024-09-21T10:33:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.443524+0000", 2024-09-21T10:33:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0154000000000001e-05 2024-09-21T10:33:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.401733+0000", 2024-09-21T10:33:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.95820901400000003 2024-09-21T10:33:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.401836+0000", 2024-09-21T10:33:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000103812 2024-09-21T10:33:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:33:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.401955+0000", 2024-09-21T10:33:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011872899999999999 2024-09-21T10:33:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433786+0000", 2024-09-21T10:33:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.031831163000000003 2024-09-21T10:33:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441037+0000", 2024-09-21T10:33:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0072507400000000003 2024-09-21T10:33:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441450+0000", 2024-09-21T10:33:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00041282899999999997 2024-09-21T10:33:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441491+0000", 2024-09-21T10:33:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1603999999999997e-05 2024-09-21T10:33:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441525+0000", 2024-09-21T10:33:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3592e-05 2024-09-21T10:33:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212946 9.1 9:86b77d80:::benchmark_data_smithi098_254407_object80215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:33:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.455455+0000", 2024-09-21T10:33:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 464.73818674199998, 2024-09-21T10:33:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.98628469600000002, 2024-09-21T10:33:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212946 2024-09-21T10:33:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455455+0000", 2024-09-21T10:33:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455455+0000", 2024-09-21T10:33:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455454+0000", 2024-09-21T10:33:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455462+0000", 2024-09-21T10:33:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6349999999999995e-06 2024-09-21T10:33:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455479+0000", 2024-09-21T10:33:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6521e-05 2024-09-21T10:33:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455485+0000", 2024-09-21T10:33:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5289999999999997e-06 2024-09-21T10:33:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455531+0000", 2024-09-21T10:33:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5420000000000002e-05 2024-09-21T10:33:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.455539+0000", 2024-09-21T10:33:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3769999999999996e-06 2024-09-21T10:33:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.404976+0000", 2024-09-21T10:33:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94943732199999997 2024-09-21T10:33:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.405084+0000", 2024-09-21T10:33:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010813400000000001 2024-09-21T10:33:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:33:03.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.405217+0000", 2024-09-21T10:33:03.276 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013250100000000001 2024-09-21T10:33:03.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.276 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.276 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:03.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433790+0000", 2024-09-21T10:33:03.276 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.028573111000000002 2024-09-21T10:33:03.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.276 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.276 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441178+0000", 2024-09-21T10:33:03.276 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0073877259999999998 2024-09-21T10:33:03.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.276 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.277 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.277 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441667+0000", 2024-09-21T10:33:03.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00048886199999999996 2024-09-21T10:33:03.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.277 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:03.277 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441706+0000", 2024-09-21T10:33:03.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8828000000000002e-05 2024-09-21T10:33:03.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.277 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:03.277 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441740+0000", 2024-09-21T10:33:03.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4625999999999997e-05 2024-09-21T10:33:03.277 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.278 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:03.278 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.278 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212950 9.1 9:83ffba2d:::benchmark_data_smithi098_254407_object80219:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:33:03.278 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.459261+0000", 2024-09-21T10:33:03.278 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 464.73438078200002, 2024-09-21T10:33:03.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.98262656500000001, 2024-09-21T10:33:03.278 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:03.278 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:03.278 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:03.278 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:03.278 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:03.279 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212950 2024-09-21T10:33:03.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.279 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:03.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:03.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459261+0000", 2024-09-21T10:33:03.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:03.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459261+0000", 2024-09-21T10:33:03.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:03.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459260+0000", 2024-09-21T10:33:03.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:03.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:03.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459267+0000", 2024-09-21T10:33:03.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7970000000000001e-06 2024-09-21T10:33:03.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:03.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459289+0000", 2024-09-21T10:33:03.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1185000000000001e-05 2024-09-21T10:33:03.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:03.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459295+0000", 2024-09-21T10:33:03.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5130000000000002e-06 2024-09-21T10:33:03.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459340+0000", 2024-09-21T10:33:03.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4555000000000003e-05 2024-09-21T10:33:03.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:03.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.459349+0000", 2024-09-21T10:33:03.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4150000000000005e-06 2024-09-21T10:33:03.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.408479+0000", 2024-09-21T10:33:03.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94912973899999997 2024-09-21T10:33:03.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:03.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.408589+0000", 2024-09-21T10:33:03.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109972 2024-09-21T10:33:03.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:33:03.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.408712+0000", 2024-09-21T10:33:03.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012358900000000001 2024-09-21T10:33:03.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:03.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433794+0000", 2024-09-21T10:33:03.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.025081782 2024-09-21T10:33:03.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441575+0000", 2024-09-21T10:33:03.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.007781123 2024-09-21T10:33:03.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441795+0000", 2024-09-21T10:33:03.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000219268 2024-09-21T10:33:03.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:03.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441841+0000", 2024-09-21T10:33:03.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6286999999999999e-05 2024-09-21T10:33:03.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:03.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441888+0000", 2024-09-21T10:33:03.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7077999999999999e-05 2024-09-21T10:33:03.285 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.285 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:03.285 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.285 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212956 9.1 9:85c97ada:::benchmark_data_smithi098_254407_object80225:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:33:03.285 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.465260+0000", 2024-09-21T10:33:03.285 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 464.72838189999999, 2024-09-21T10:33:03.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97679754200000002, 2024-09-21T10:33:03.285 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:03.285 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:03.285 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:03.286 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:03.286 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:03.286 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212956 2024-09-21T10:33:03.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.286 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:03.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.286 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:03.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465260+0000", 2024-09-21T10:33:03.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.286 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:03.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465260+0000", 2024-09-21T10:33:03.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:03.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465258+0000", 2024-09-21T10:33:03.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:03.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:03.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465268+0000", 2024-09-21T10:33:03.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5109999999999992e-06 2024-09-21T10:33:03.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:03.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465290+0000", 2024-09-21T10:33:03.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2013000000000001e-05 2024-09-21T10:33:03.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:03.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465297+0000", 2024-09-21T10:33:03.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.311e-06 2024-09-21T10:33:03.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465456+0000", 2024-09-21T10:33:03.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000159347 2024-09-21T10:33:03.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:03.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.465488+0000", 2024-09-21T10:33:03.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1739000000000002e-05 2024-09-21T10:33:03.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.411613+0000", 2024-09-21T10:33:03.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94612508699999998 2024-09-21T10:33:03.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:03.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.411728+0000", 2024-09-21T10:33:03.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011490200000000001 2024-09-21T10:33:03.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.290 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:33:03.290 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.411855+0000", 2024-09-21T10:33:03.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000126971 2024-09-21T10:33:03.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.290 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:03.290 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433798+0000", 2024-09-21T10:33:03.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.021942560999999999 2024-09-21T10:33:03.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.290 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.290 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441610+0000", 2024-09-21T10:33:03.291 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0078126959999999992 2024-09-21T10:33:03.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.291 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.291 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.291 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441984+0000", 2024-09-21T10:33:03.291 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000373956 2024-09-21T10:33:03.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.291 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.291 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:03.291 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442024+0000", 2024-09-21T10:33:03.291 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9459999999999998e-05 2024-09-21T10:33:03.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.291 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.291 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:03.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442058+0000", 2024-09-21T10:33:03.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4010000000000001e-05 2024-09-21T10:33:03.292 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.292 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:03.292 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.292 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212974 9.1 9:8bf7ce4b:::benchmark_data_smithi098_254407_object80243:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:33:03.292 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.484850+0000", 2024-09-21T10:33:03.292 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 464.708792039, 2024-09-21T10:33:03.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.95736883500000003, 2024-09-21T10:33:03.292 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:03.292 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:03.293 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:03.293 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:03.293 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:03.293 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212974 2024-09-21T10:33:03.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.293 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:03.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:03.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.484850+0000", 2024-09-21T10:33:03.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:03.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.484850+0000", 2024-09-21T10:33:03.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:03.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.484848+0000", 2024-09-21T10:33:03.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:03.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:03.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.484857+0000", 2024-09-21T10:33:03.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1589999999999999e-06 2024-09-21T10:33:03.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:03.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.484883+0000", 2024-09-21T10:33:03.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5681999999999999e-05 2024-09-21T10:33:03.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:03.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.484890+0000", 2024-09-21T10:33:03.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4200000000000001e-06 2024-09-21T10:33:03.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.485094+0000", 2024-09-21T10:33:03.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00020403399999999999 2024-09-21T10:33:03.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:03.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.485127+0000", 2024-09-21T10:33:03.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2221000000000002e-05 2024-09-21T10:33:03.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.414677+0000", 2024-09-21T10:33:03.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.929550023 2024-09-21T10:33:03.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:03.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.414781+0000", 2024-09-21T10:33:03.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104038 2024-09-21T10:33:03.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:33:03.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.414897+0000", 2024-09-21T10:33:03.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011673899999999999 2024-09-21T10:33:03.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:03.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433801+0000", 2024-09-21T10:33:03.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018903169000000001 2024-09-21T10:33:03.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.441942+0000", 2024-09-21T10:33:03.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0081409929999999991 2024-09-21T10:33:03.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442168+0000", 2024-09-21T10:33:03.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00022609 2024-09-21T10:33:03.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:03.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442198+0000", 2024-09-21T10:33:03.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.042e-05 2024-09-21T10:33:03.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.299 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:03.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442219+0000", 2024-09-21T10:33:03.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0837000000000001e-05 2024-09-21T10:33:03.299 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.299 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:03.299 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.299 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212980 9.1 9:8e228128:::benchmark_data_smithi098_254407_object80249:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:33:03.299 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.495564+0000", 2024-09-21T10:33:03.299 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 464.69807817200001, 2024-09-21T10:33:03.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.946840558, 2024-09-21T10:33:03.299 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:03.299 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:03.300 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:03.300 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:03.300 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:03.300 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212980 2024-09-21T10:33:03.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.300 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:03.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:03.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495564+0000", 2024-09-21T10:33:03.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:03.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495564+0000", 2024-09-21T10:33:03.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:03.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495562+0000", 2024-09-21T10:33:03.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:03.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:03.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495569+0000", 2024-09-21T10:33:03.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.781e-06 2024-09-21T10:33:03.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:03.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495580+0000", 2024-09-21T10:33:03.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0601999999999999e-05 2024-09-21T10:33:03.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:03.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495586+0000", 2024-09-21T10:33:03.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9750000000000004e-06 2024-09-21T10:33:03.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495640+0000", 2024-09-21T10:33:03.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3989000000000001e-05 2024-09-21T10:33:03.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:03.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495651+0000", 2024-09-21T10:33:03.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1012000000000001e-05 2024-09-21T10:33:03.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.304 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.417846+0000", 2024-09-21T10:33:03.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.92219546299999999 2024-09-21T10:33:03.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.304 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:03.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.417972+0000", 2024-09-21T10:33:03.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000125695 2024-09-21T10:33:03.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:33:03.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.418097+0000", 2024-09-21T10:33:03.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012517 2024-09-21T10:33:03.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:03.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433804+0000", 2024-09-21T10:33:03.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015706759000000001 2024-09-21T10:33:03.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442128+0000", 2024-09-21T10:33:03.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0083244349999999998 2024-09-21T10:33:03.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442338+0000", 2024-09-21T10:33:03.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00020961600000000001 2024-09-21T10:33:03.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:03.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442372+0000", 2024-09-21T10:33:03.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4573999999999998e-05 2024-09-21T10:33:03.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:03.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442405+0000", 2024-09-21T10:33:03.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2186e-05 2024-09-21T10:33:03.307 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.307 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:03.307 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.307 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:212982 9.1 9:8bbf57c4:::benchmark_data_smithi098_254407_object80251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:33:03.307 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.495812+0000", 2024-09-21T10:33:03.307 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 464.69782977900002, 2024-09-21T10:33:03.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94673693199999998, 2024-09-21T10:33:03.307 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:03.307 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:03.307 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:03.307 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:03.308 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:03.308 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212982 2024-09-21T10:33:03.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.308 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:03.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:03.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495812+0000", 2024-09-21T10:33:03.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:03.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495812+0000", 2024-09-21T10:33:03.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:03.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495812+0000", 2024-09-21T10:33:03.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:33:03.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:03.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495816+0000", 2024-09-21T10:33:03.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3039999999999998e-06 2024-09-21T10:33:03.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:03.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495824+0000", 2024-09-21T10:33:03.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5120000000000002e-06 2024-09-21T10:33:03.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:03.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495828+0000", 2024-09-21T10:33:03.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6240000000000001e-06 2024-09-21T10:33:03.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495870+0000", 2024-09-21T10:33:03.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1554999999999998e-05 2024-09-21T10:33:03.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:03.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.495887+0000", 2024-09-21T10:33:03.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7122999999999999e-05 2024-09-21T10:33:03.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.311 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.420943+0000", 2024-09-21T10:33:03.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.92505649000000001 2024-09-21T10:33:03.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.311 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:03.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.421059+0000", 2024-09-21T10:33:03.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000115108 2024-09-21T10:33:03.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:33:03.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.421167+0000", 2024-09-21T10:33:03.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108929 2024-09-21T10:33:03.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:03.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433807+0000", 2024-09-21T10:33:03.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012639262 2024-09-21T10:33:03.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442271+0000", 2024-09-21T10:33:03.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0084638600000000001 2024-09-21T10:33:03.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442444+0000", 2024-09-21T10:33:03.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017356899999999999 2024-09-21T10:33:03.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:03.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442515+0000", 2024-09-21T10:33:03.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1299999999999998e-05 2024-09-21T10:33:03.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:03.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442549+0000", 2024-09-21T10:33:03.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3889e-05 2024-09-21T10:33:03.314 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.314 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:03.314 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.314 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:213024 9.1 9:8ab0c9ed:::benchmark_data_smithi098_254407_object80293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:33:03.314 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.545957+0000", 2024-09-21T10:33:03.314 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 464.647684701, 2024-09-21T10:33:03.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.896714071, 2024-09-21T10:33:03.314 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:03.314 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:03.314 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:03.315 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:03.315 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:03.315 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 213024 2024-09-21T10:33:03.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.315 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:03.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:03.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.545957+0000", 2024-09-21T10:33:03.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:03.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.545957+0000", 2024-09-21T10:33:03.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:03.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.545956+0000", 2024-09-21T10:33:03.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:03.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:03.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.545964+0000", 2024-09-21T10:33:03.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6629999999999999e-06 2024-09-21T10:33:03.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:03.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.545986+0000", 2024-09-21T10:33:03.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1220999999999999e-05 2024-09-21T10:33:03.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:03.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.545993+0000", 2024-09-21T10:33:03.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.396e-06 2024-09-21T10:33:03.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.546156+0000", 2024-09-21T10:33:03.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016294899999999999 2024-09-21T10:33:03.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:03.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.546178+0000", 2024-09-21T10:33:03.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1971999999999999e-05 2024-09-21T10:33:03.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.424032+0000", 2024-09-21T10:33:03.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.87785439099999996 2024-09-21T10:33:03.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:03.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.424140+0000", 2024-09-21T10:33:03.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107507 2024-09-21T10:33:03.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.319 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:33:03.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.424258+0000", 2024-09-21T10:33:03.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011840700000000001 2024-09-21T10:33:03.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.319 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:03.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.437127+0000", 2024-09-21T10:33:03.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012869234 2024-09-21T10:33:03.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.319 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442310+0000", 2024-09-21T10:33:03.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051825589999999998 2024-09-21T10:33:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442599+0000", 2024-09-21T10:33:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00028935099999999997 2024-09-21T10:33:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442639+0000", 2024-09-21T10:33:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9762000000000003e-05 2024-09-21T10:33:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:03.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442672+0000", 2024-09-21T10:33:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2480000000000001e-05 2024-09-21T10:33:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:213027 9.1 9:8d848b4e:::benchmark_data_smithi098_254407_object80296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:33:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.549492+0000", 2024-09-21T10:33:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 464.64415015999998, 2024-09-21T10:33:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.89346427100000003, 2024-09-21T10:33:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:03.321 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 213027 2024-09-21T10:33:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549492+0000", 2024-09-21T10:33:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549492+0000", 2024-09-21T10:33:03.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:03.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549490+0000", 2024-09-21T10:33:03.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:03.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:03.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549498+0000", 2024-09-21T10:33:03.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4209999999999995e-06 2024-09-21T10:33:03.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:03.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549518+0000", 2024-09-21T10:33:03.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0057000000000001e-05 2024-09-21T10:33:03.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:03.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549526+0000", 2024-09-21T10:33:03.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4730000000000003e-06 2024-09-21T10:33:03.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549593+0000", 2024-09-21T10:33:03.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7409000000000005e-05 2024-09-21T10:33:03.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:03.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.549603+0000", 2024-09-21T10:33:03.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5810000000000003e-06 2024-09-21T10:33:03.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.427438+0000", 2024-09-21T10:33:03.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.87783562999999998 2024-09-21T10:33:03.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:03.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.427559+0000", 2024-09-21T10:33:03.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000120377 2024-09-21T10:33:03.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.326 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:33:03.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.427666+0000", 2024-09-21T10:33:03.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107382 2024-09-21T10:33:03.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.326 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:03.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.437138+0000", 2024-09-21T10:33:03.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0094713769999999996 2024-09-21T10:33:03.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.326 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442725+0000", 2024-09-21T10:33:03.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.005586995 2024-09-21T10:33:03.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442870+0000", 2024-09-21T10:33:03.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014551399999999999 2024-09-21T10:33:03.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:03.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442920+0000", 2024-09-21T10:33:03.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0371999999999998e-05 2024-09-21T10:33:03.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:03.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442956+0000", 2024-09-21T10:33:03.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5805e-05 2024-09-21T10:33:03.328 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.328 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:03.328 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.328 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:213030 9.1 9:8c3d5a31:::benchmark_data_smithi098_254407_object80299:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:33:03.328 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.556579+0000", 2024-09-21T10:33:03.329 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 464.637062867, 2024-09-21T10:33:03.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.88649370000000005, 2024-09-21T10:33:03.329 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:03.329 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:03.329 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:03.329 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:03.329 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:03.329 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 213030 2024-09-21T10:33:03.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.329 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:03.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:03.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556579+0000", 2024-09-21T10:33:03.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:03.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556579+0000", 2024-09-21T10:33:03.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:03.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556578+0000", 2024-09-21T10:33:03.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:03.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:03.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556585+0000", 2024-09-21T10:33:03.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6159999999999996e-06 2024-09-21T10:33:03.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:03.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556600+0000", 2024-09-21T10:33:03.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.454e-05 2024-09-21T10:33:03.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:03.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556607+0000", 2024-09-21T10:33:03.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8900000000000001e-06 2024-09-21T10:33:03.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556650+0000", 2024-09-21T10:33:03.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3727000000000003e-05 2024-09-21T10:33:03.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:03.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.556658+0000", 2024-09-21T10:33:03.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5079999999999999e-06 2024-09-21T10:33:03.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.430408+0000", 2024-09-21T10:33:03.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.87375053700000005 2024-09-21T10:33:03.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:03.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.430547+0000", 2024-09-21T10:33:03.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013809400000000001 2024-09-21T10:33:03.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:33:03.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.430679+0000", 2024-09-21T10:33:03.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013244900000000001 2024-09-21T10:33:03.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:03.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440488+0000", 2024-09-21T10:33:03.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0098090959999999998 2024-09-21T10:33:03.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442765+0000", 2024-09-21T10:33:03.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002277276 2024-09-21T10:33:03.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443000+0000", 2024-09-21T10:33:03.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000234682 2024-09-21T10:33:03.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:03.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443042+0000", 2024-09-21T10:33:03.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2126000000000003e-05 2024-09-21T10:33:03.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:03.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443073+0000", 2024-09-21T10:33:03.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0825999999999999e-05 2024-09-21T10:33:03.335 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.335 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:03.335 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.336 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:213040 9.1 9:8e7e94d6:::benchmark_data_smithi098_254407_object80309:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:33:03.336 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.575656+0000", 2024-09-21T10:33:03.336 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 464.61798601499999, 2024-09-21T10:33:03.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.86753614499999998, 2024-09-21T10:33:03.336 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:03.336 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:03.336 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:03.336 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:03.336 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:03.336 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 213040 2024-09-21T10:33:03.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.336 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:03.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:03.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575656+0000", 2024-09-21T10:33:03.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:03.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575656+0000", 2024-09-21T10:33:03.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:03.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575654+0000", 2024-09-21T10:33:03.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:03.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:03.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575664+0000", 2024-09-21T10:33:03.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7340000000000003e-06 2024-09-21T10:33:03.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:03.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575682+0000", 2024-09-21T10:33:03.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8410000000000002e-05 2024-09-21T10:33:03.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:03.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575691+0000", 2024-09-21T10:33:03.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3559999999999993e-06 2024-09-21T10:33:03.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575742+0000", 2024-09-21T10:33:03.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0698999999999997e-05 2024-09-21T10:33:03.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:03.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.575752+0000", 2024-09-21T10:33:03.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0288999999999999e-05 2024-09-21T10:33:03.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433825+0000", 2024-09-21T10:33:03.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.85807313799999996 2024-09-21T10:33:03.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:03.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.433937+0000", 2024-09-21T10:33:03.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011184900000000001 2024-09-21T10:33:03.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:33:03.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.434061+0000", 2024-09-21T10:33:03.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012429100000000001 2024-09-21T10:33:03.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:03.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440716+0000", 2024-09-21T10:33:03.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006655134 2024-09-21T10:33:03.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442806+0000", 2024-09-21T10:33:03.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002089835 2024-09-21T10:33:03.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443118+0000", 2024-09-21T10:33:03.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00031147500000000002 2024-09-21T10:33:03.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:03.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443158+0000", 2024-09-21T10:33:03.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0670000000000002e-05 2024-09-21T10:33:03.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:03.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443192+0000", 2024-09-21T10:33:03.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4109999999999997e-05 2024-09-21T10:33:03.342 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.342 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:03.342 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.343 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:213046 9.1 9:8035648d:::benchmark_data_smithi098_254407_object80315:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-21T10:33:03.343 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:25:18.597939+0000", 2024-09-21T10:33:03.343 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 464.59570309100002, 2024-09-21T10:33:03.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.84537171499999997, 2024-09-21T10:33:03.343 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:03.343 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:03.343 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:03.343 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:03.343 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:03.343 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 213046 2024-09-21T10:33:03.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.343 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:03.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:03.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.597939+0000", 2024-09-21T10:33:03.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:03.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.597939+0000", 2024-09-21T10:33:03.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:03.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.597938+0000", 2024-09-21T10:33:03.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:03.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.345 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:03.345 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.597945+0000", 2024-09-21T10:33:03.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5959999999999998e-06 2024-09-21T10:33:03.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.345 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:03.345 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.597959+0000", 2024-09-21T10:33:03.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3986e-05 2024-09-21T10:33:03.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.345 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:03.345 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.597964+0000", 2024-09-21T10:33:03.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0289999999999996e-06 2024-09-21T10:33:03.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.598015+0000", 2024-09-21T10:33:03.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0376e-05 2024-09-21T10:33:03.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:03.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:18.598024+0000", 2024-09-21T10:33:03.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8899999999999996e-06 2024-09-21T10:33:03.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.437180+0000", 2024-09-21T10:33:03.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83915633099999998 2024-09-21T10:33:03.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:03.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.437294+0000", 2024-09-21T10:33:03.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114388 2024-09-21T10:33:03.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:33:03.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.437423+0000", 2024-09-21T10:33:03.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000128768 2024-09-21T10:33:03.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:03.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.440953+0000", 2024-09-21T10:33:03.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0035302250000000001 2024-09-21T10:33:03.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.348 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.442834+0000", 2024-09-21T10:33:03.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0018801919999999999 2024-09-21T10:33:03.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.348 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443242+0000", 2024-09-21T10:33:03.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00040894199999999998 2024-09-21T10:33:03.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:03.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443282+0000", 2024-09-21T10:33:03.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9504e-05 2024-09-21T10:33:03.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:03.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:25:19.443311+0000", 2024-09-21T10:33:03.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8816999999999999e-05 2024-09-21T10:33:03.349 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.349 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:03.349 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.349 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46322.0:201765 10.a 10:529f12b0:::benchmark_data_smithi098_281896_object76740:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T10:33:03.350 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:30:09.596371+0000", 2024-09-21T10:33:03.350 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 173.597270893, 2024-09-21T10:33:03.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.271914936, 2024-09-21T10:33:03.350 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:03.350 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:03.350 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:03.350 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46322", 2024-09-21T10:33:03.350 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:33:03.350 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 201765 2024-09-21T10:33:03.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.350 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:03.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:03.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.596371+0000", 2024-09-21T10:33:03.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:03.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.596371+0000", 2024-09-21T10:33:03.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:03.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.596370+0000", 2024-09-21T10:33:03.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:33:03.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:03.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.596376+0000", 2024-09-21T10:33:03.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3299999999999998e-06 2024-09-21T10:33:03.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:03.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.596386+0000", 2024-09-21T10:33:03.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0589e-05 2024-09-21T10:33:03.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:03.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.596391+0000", 2024-09-21T10:33:03.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2800000000000003e-06 2024-09-21T10:33:03.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.596431+0000", 2024-09-21T10:33:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9425000000000003e-05 2024-09-21T10:33:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.596521+0000", 2024-09-21T10:33:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9707e-05 2024-09-21T10:33:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:33:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.596599+0000", 2024-09-21T10:33:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8545000000000001e-05 2024-09-21T10:33:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.599658+0000", 2024-09-21T10:33:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003058851 2024-09-21T10:33:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.612435+0000", 2024-09-21T10:33:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0127774 2024-09-21T10:33:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.868159+0000", 2024-09-21T10:33:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.25572400000000001 2024-09-21T10:33:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.868249+0000", 2024-09-21T10:33:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0026999999999997e-05 2024-09-21T10:33:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.868286+0000", 2024-09-21T10:33:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6776000000000002e-05 2024-09-21T10:33:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46322.0:201770 10.a 10:59860302:::benchmark_data_smithi098_281896_object76745:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T10:33:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:30:09.614381+0000", 2024-09-21T10:33:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 173.579261003, 2024-09-21T10:33:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.26075808099999997, 2024-09-21T10:33:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46322", 2024-09-21T10:33:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:33:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 201770 2024-09-21T10:33:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.614381+0000", 2024-09-21T10:33:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.614381+0000", 2024-09-21T10:33:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.614379+0000", 2024-09-21T10:33:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.614392+0000", 2024-09-21T10:33:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2628e-05 2024-09-21T10:33:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.614411+0000", 2024-09-21T10:33:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9582e-05 2024-09-21T10:33:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.614420+0000", 2024-09-21T10:33:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4519999999999997e-06 2024-09-21T10:33:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.614481+0000", 2024-09-21T10:33:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1288e-05 2024-09-21T10:33:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.614623+0000", 2024-09-21T10:33:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014227 2024-09-21T10:33:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:33:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.614710+0000", 2024-09-21T10:33:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6445000000000003e-05 2024-09-21T10:33:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.618831+0000", 2024-09-21T10:33:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041211169999999997 2024-09-21T10:33:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.636596+0000", 2024-09-21T10:33:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017765171999999999 2024-09-21T10:33:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.875045+0000", 2024-09-21T10:33:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.23844873899999999 2024-09-21T10:33:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.875098+0000", 2024-09-21T10:33:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3338999999999999e-05 2024-09-21T10:33:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:30:09.875139+0000", 2024-09-21T10:33:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1137000000000003e-05 2024-09-21T10:33:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.362 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:03.362 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.362 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:03.362 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:03.362 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:33:03.428 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:03.794 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:03 smithi098 ceph-mon[97580]: pgmap v1425: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:03.794 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:03 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/53064244' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:33:03.795 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:03 smithi098 ceph-mon[97580]: osdmap e413: 8 total, 8 up, 8 in 2024-09-21T10:33:03.795 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:03 smithi098 ceph-mon[100914]: pgmap v1425: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:03.795 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:03 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/53064244' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:33:03.795 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:03 smithi098 ceph-mon[100914]: osdmap e413: 8 total, 8 up, 8 in 2024-09-21T10:33:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:03 smithi196 ceph-mon[84784]: pgmap v1425: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:03 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/53064244' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:33:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:03 smithi196 ceph-mon[84784]: osdmap e413: 8 total, 8 up, 8 in 2024-09-21T10:33:04.161 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:33:04.406 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:33:05.473 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-21T10:33:05.536 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:05 smithi196 ceph-mon[84784]: pgmap v1427: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:05.536 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:05 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1467474910' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:33:05.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:05 smithi098 ceph-mon[97580]: pgmap v1427: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:05.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:05 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1467474910' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:33:05.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:05 smithi098 ceph-mon[100914]: pgmap v1427: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:05.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:05 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1467474910' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:33:06.048 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:33:06.048 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:33:06.048 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-21T10:33:06.048 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:33:06.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T10:33:06.439 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:06.618 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:06 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1467474910' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:33:06.618 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:06 smithi196 ceph-mon[84784]: osdmap e414: 8 total, 8 up, 8 in 2024-09-21T10:33:06.642 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:33:06.811 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:06 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1467474910' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:33:06.812 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:06 smithi098 ceph-mon[97580]: osdmap e414: 8 total, 8 up, 8 in 2024-09-21T10:33:06.812 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:06 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1467474910' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:33:06.812 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:06 smithi098 ceph-mon[100914]: osdmap e414: 8 total, 8 up, 8 in 2024-09-21T10:33:06.886 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:33:07.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:07 smithi196 ceph-mon[84784]: pgmap v1429: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:07.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:07 smithi098 ceph-mon[97580]: pgmap v1429: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:07.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:07.784 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:07 smithi098 ceph-mon[100914]: pgmap v1429: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:07.784 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:08.512 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:33:08.512 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-21T10:33:08.512 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-21T10:33:08.512 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-21T10:33:08.512 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:33:09.149 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:33:09.393 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:33:09.496 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T10:33:09.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:09 smithi098 ceph-mon[97580]: pgmap v1430: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:09.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:09 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3953326375' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:33:09.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:09 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:33:09.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:09 smithi098 ceph-mon[100914]: pgmap v1430: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:09.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:09 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3953326375' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:33:09.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:09 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:33:09.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:09 smithi196 ceph-mon[84784]: pgmap v1430: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:09.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:09 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3953326375' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:33:09.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:09 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:33:10.268 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 3 1 2024-09-21T10:33:10.501 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:33:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:33:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:33:10.518 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:10.756 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:10 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:33:10.756 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:10 smithi196 ceph-mon[84784]: osdmap e415: 8 total, 8 up, 8 in 2024-09-21T10:33:10.763 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:10 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:33:10.763 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:10 smithi098 ceph-mon[97580]: osdmap e415: 8 total, 8 up, 8 in 2024-09-21T10:33:10.763 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:33:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:33:10] "GET /metrics HTTP/1.1" 200 34162 "" "Prometheus/2.51.0" 2024-09-21T10:33:10.763 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:10 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:33:10.763 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:10 smithi098 ceph-mon[100914]: osdmap e415: 8 total, 8 up, 8 in 2024-09-21T10:33:11.082 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-21T10:33:11.082 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-21T10:33:11.082 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-21T10:33:11.083 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-21T10:33:11.083 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.083 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181814 9.e 9:74c3af90:::benchmark_data_smithi098_254407_object49083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T10:33:11.083 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.273477+0000", 2024-09-21T10:33:11.083 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 507.805010106, 2024-09-21T10:33:11.083 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.160252872, 2024-09-21T10:33:11.083 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:11.083 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-21T10:33:11.083 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:11.083 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:11.083 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:11.084 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181814 2024-09-21T10:33:11.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.084 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:11.084 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.084 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:11.084 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.273477+0000", 2024-09-21T10:33:11.084 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.084 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.084 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:11.084 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.273477+0000", 2024-09-21T10:33:11.084 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.085 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.085 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:11.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.273475+0000", 2024-09-21T10:33:11.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:11.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.085 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.085 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:11.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.273486+0000", 2024-09-21T10:33:11.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0481e-05 2024-09-21T10:33:11.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.085 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.085 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:11.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.275927+0000", 2024-09-21T10:33:11.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024413059999999999 2024-09-21T10:33:11.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.086 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:11.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.275937+0000", 2024-09-21T10:33:11.086 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0273e-05 2024-09-21T10:33:11.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.086 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277114+0000", 2024-09-21T10:33:11.086 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011773440000000001 2024-09-21T10:33:11.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.086 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:11.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277227+0000", 2024-09-21T10:33:11.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000112969 2024-09-21T10:33:11.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.087 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.087 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:33:11.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.277336+0000", 2024-09-21T10:33:11.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108162 2024-09-21T10:33:11.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.087 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.087 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:11.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.286117+0000", 2024-09-21T10:33:11.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0087815770000000005 2024-09-21T10:33:11.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.087 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.294408+0000", 2024-09-21T10:33:11.088 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0082903879999999992 2024-09-21T10:33:11.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:11.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.294415+0000", 2024-09-21T10:33:11.088 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2869999999999999e-06 2024-09-21T10:33:11.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.297546+0000", 2024-09-21T10:33:11.088 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0031308510000001 2024-09-21T10:33:11.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.089 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.089 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:11.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.297561+0000", 2024-09-21T10:33:11.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5545000000000001e-05 2024-09-21T10:33:11.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.089 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.089 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.425226+0000", 2024-09-21T10:33:11.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1276647829999999 2024-09-21T10:33:11.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.089 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.089 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for missing object", 2024-09-21T10:33:11.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.425366+0000", 2024-09-21T10:33:11.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000139529 2024-09-21T10:33:11.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.090 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.090 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.090 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433657+0000", 2024-09-21T10:33:11.090 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0082909950000000007 2024-09-21T10:33:11.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.090 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.090 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:11.090 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433729+0000", 2024-09-21T10:33:11.090 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2879000000000005e-05 2024-09-21T10:33:11.090 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.090 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:11.091 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.091 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.091 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181817 9.e 9:7fac4b1e:::benchmark_data_smithi098_254407_object49086:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T10:33:11.091 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.275953+0000", 2024-09-21T10:33:11.091 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 507.80253314499998, 2024-09-21T10:33:11.091 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1579490689999998, 2024-09-21T10:33:11.091 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:11.091 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:11.091 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:11.091 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:11.091 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:11.091 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181817 2024-09-21T10:33:11.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.092 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:11.092 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.092 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:11.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.275953+0000", 2024-09-21T10:33:11.092 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.092 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.092 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:11.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.275953+0000", 2024-09-21T10:33:11.092 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.092 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.092 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:11.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.275952+0000", 2024-09-21T10:33:11.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:33:11.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:11.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.275958+0000", 2024-09-21T10:33:11.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4840000000000003e-06 2024-09-21T10:33:11.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:11.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.275965+0000", 2024-09-21T10:33:11.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2110000000000001e-06 2024-09-21T10:33:11.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.094 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:11.094 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.275969+0000", 2024-09-21T10:33:11.094 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2560000000000004e-06 2024-09-21T10:33:11.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.094 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.094 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.094 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.294432+0000", 2024-09-21T10:33:11.094 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018462203999999999 2024-09-21T10:33:11.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.094 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.094 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:11.094 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.294437+0000", 2024-09-21T10:33:11.094 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.271e-06 2024-09-21T10:33:11.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.095 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.095 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.298237+0000", 2024-09-21T10:33:11.095 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.003800574 2024-09-21T10:33:11.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.095 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.095 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:11.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.298263+0000", 2024-09-21T10:33:11.095 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5819000000000001e-05 2024-09-21T10:33:11.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.095 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.095 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.425691+0000", 2024-09-21T10:33:11.096 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.127427889 2024-09-21T10:33:11.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.096 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:11.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.425818+0000", 2024-09-21T10:33:11.096 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012697000000000001 2024-09-21T10:33:11.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.096 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1,4", 2024-09-21T10:33:11.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.425958+0000", 2024-09-21T10:33:11.096 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014029900000000001 2024-09-21T10:33:11.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.096 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:11.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433272+0000", 2024-09-21T10:33:11.097 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0073139520000000003 2024-09-21T10:33:11.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.098 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.098 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433315+0000", 2024-09-21T10:33:11.098 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2237999999999998e-05 2024-09-21T10:33:11.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.098 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.098 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.098 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433345+0000", 2024-09-21T10:33:11.098 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0508999999999999e-05 2024-09-21T10:33:11.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.098 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.098 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.098 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433759+0000", 2024-09-21T10:33:11.099 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00041381300000000002 2024-09-21T10:33:11.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.099 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.099 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:11.099 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433797+0000", 2024-09-21T10:33:11.099 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7857e-05 2024-09-21T10:33:11.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.099 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.099 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:11.099 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433903+0000", 2024-09-21T10:33:11.099 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105803 2024-09-21T10:33:11.099 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.099 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:11.099 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.100 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.100 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181827 9.e 9:752f527b:::benchmark_data_smithi098_254407_object49096:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:33:11.100 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.295979+0000", 2024-09-21T10:33:11.100 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 507.78250808899998, 2024-09-21T10:33:11.100 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.138117657, 2024-09-21T10:33:11.100 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:11.100 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:11.100 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:11.100 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:11.100 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:11.100 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181827 2024-09-21T10:33:11.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.100 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:11.100 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.100 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:11.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.295979+0000", 2024-09-21T10:33:11.101 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.101 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.101 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:11.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.295979+0000", 2024-09-21T10:33:11.101 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.101 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.101 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:11.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.295977+0000", 2024-09-21T10:33:11.101 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:11.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.102 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:11.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.295982+0000", 2024-09-21T10:33:11.102 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0239999999999997e-06 2024-09-21T10:33:11.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.102 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:11.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.295991+0000", 2024-09-21T10:33:11.102 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3080000000000008e-06 2024-09-21T10:33:11.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.102 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:11.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.295995+0000", 2024-09-21T10:33:11.103 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5410000000000002e-06 2024-09-21T10:33:11.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.103 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.103 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.296052+0000", 2024-09-21T10:33:11.103 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6929000000000003e-05 2024-09-21T10:33:11.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.105 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.106 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:11.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.296082+0000", 2024-09-21T10:33:11.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0080999999999999e-05 2024-09-21T10:33:11.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.106 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.298286+0000", 2024-09-21T10:33:11.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.002203583 2024-09-21T10:33:11.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.107 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.107 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:11.107 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.298293+0000", 2024-09-21T10:33:11.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9820000000000002e-06 2024-09-21T10:33:11.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.429659+0000", 2024-09-21T10:33:11.108 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.131366028 2024-09-21T10:33:11.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:11.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.429795+0000", 2024-09-21T10:33:11.109 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000136594 2024-09-21T10:33:11.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.109 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1,4", 2024-09-21T10:33:11.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.429931+0000", 2024-09-21T10:33:11.109 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000135807 2024-09-21T10:33:11.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.110 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:11.110 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433279+0000", 2024-09-21T10:33:11.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0033479209999999998 2024-09-21T10:33:11.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.110 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.110 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.110 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433374+0000", 2024-09-21T10:33:11.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5358000000000004e-05 2024-09-21T10:33:11.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.111 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433935+0000", 2024-09-21T10:33:11.111 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00056065800000000001 2024-09-21T10:33:11.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.111 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.433965+0000", 2024-09-21T10:33:11.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0321e-05 2024-09-21T10:33:11.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.112 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:11.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.434007+0000", 2024-09-21T10:33:11.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1180000000000002e-05 2024-09-21T10:33:11.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:11.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.434096+0000", 2024-09-21T10:33:11.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9605999999999996e-05 2024-09-21T10:33:11.113 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.113 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:11.113 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.113 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181829 9.e 9:7ab751bc:::benchmark_data_smithi098_254407_object49098:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:33:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.296018+0000", 2024-09-21T10:33:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 507.78246889100001, 2024-09-21T10:33:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1587496179999999, 2024-09-21T10:33:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:11.115 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181829 2024-09-21T10:33:11.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.115 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:11.115 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.115 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:11.115 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.296018+0000", 2024-09-21T10:33:11.115 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.296018+0000", 2024-09-21T10:33:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.296017+0000", 2024-09-21T10:33:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:33:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.296020+0000", 2024-09-21T10:33:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1300000000000001e-06 2024-09-21T10:33:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.118 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:11.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.296028+0000", 2024-09-21T10:33:11.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7500000000000003e-06 2024-09-21T10:33:11.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.118 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:11.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.296032+0000", 2024-09-21T10:33:11.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0980000000000004e-06 2024-09-21T10:33:11.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.119 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.119 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.119 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.296265+0000", 2024-09-21T10:33:11.119 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000233012 2024-09-21T10:33:11.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.119 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.120 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:11.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.296273+0000", 2024-09-21T10:33:11.120 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0749999999999998e-06 2024-09-21T10:33:11.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.120 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.120 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.298339+0000", 2024-09-21T10:33:11.120 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.002066219 2024-09-21T10:33:11.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.120 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.120 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:11.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.298347+0000", 2024-09-21T10:33:11.120 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0659999999999998e-06 2024-09-21T10:33:11.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.434117+0000", 2024-09-21T10:33:11.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.135770629 2024-09-21T10:33:11.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:11.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.434218+0000", 2024-09-21T10:33:11.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100985 2024-09-21T10:33:11.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1,4", 2024-09-21T10:33:11.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.434325+0000", 2024-09-21T10:33:11.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107205 2024-09-21T10:33:11.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.122 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.122 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.445178+0000", 2024-09-21T10:33:11.122 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010852288 2024-09-21T10:33:11.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.122 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.122 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:11.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.445186+0000", 2024-09-21T10:33:11.122 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9130000000000003e-06 2024-09-21T10:33:11.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.122 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.122 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.446747+0000", 2024-09-21T10:33:11.123 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015616580000000001 2024-09-21T10:33:11.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.123 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.123 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.454616+0000", 2024-09-21T10:33:11.123 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0078688309999999997 2024-09-21T10:33:11.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.123 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.123 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:11.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.454662+0000", 2024-09-21T10:33:11.123 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5759e-05 2024-09-21T10:33:11.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.124 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.124 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:11.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.454767+0000", 2024-09-21T10:33:11.124 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105568 2024-09-21T10:33:11.124 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.124 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:11.124 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.124 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.124 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181852 9.e 9:74989e04:::benchmark_data_smithi098_254407_object49121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:33:11.124 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.320821+0000", 2024-09-21T10:33:11.124 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 507.75766549600002, 2024-09-21T10:33:11.124 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1341375820000001, 2024-09-21T10:33:11.124 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:11.125 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:11.125 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:11.125 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:11.125 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:11.125 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181852 2024-09-21T10:33:11.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.125 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:11.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.125 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:11.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.320821+0000", 2024-09-21T10:33:11.125 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.126 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:11.126 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.320821+0000", 2024-09-21T10:33:11.126 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.126 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.126 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:11.126 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.320820+0000", 2024-09-21T10:33:11.126 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:33:11.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.126 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.126 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:11.126 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.320824+0000", 2024-09-21T10:33:11.126 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0879999999999997e-06 2024-09-21T10:33:11.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.127 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.127 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:11.127 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.320835+0000", 2024-09-21T10:33:11.127 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.062e-05 2024-09-21T10:33:11.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.127 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.127 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:11.127 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.320840+0000", 2024-09-21T10:33:11.127 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9629999999999997e-06 2024-09-21T10:33:11.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.127 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.128 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.320906+0000", 2024-09-21T10:33:11.128 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5568999999999995e-05 2024-09-21T10:33:11.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.128 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.128 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:11.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.320913+0000", 2024-09-21T10:33:11.128 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5499999999999997e-06 2024-09-21T10:33:11.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.128 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.128 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.298371+0000", 2024-09-21T10:33:11.128 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97745800699999996 2024-09-21T10:33:11.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.129 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.129 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:11.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.298378+0000", 2024-09-21T10:33:11.129 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7039999999999997e-06 2024-09-21T10:33:11.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.129 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.129 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.438127+0000", 2024-09-21T10:33:11.129 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1397487129999999 2024-09-21T10:33:11.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.129 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.129 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:11.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.438232+0000", 2024-09-21T10:33:11.129 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105609 2024-09-21T10:33:11.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.130 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1,4", 2024-09-21T10:33:11.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.438340+0000", 2024-09-21T10:33:11.130 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010830199999999999 2024-09-21T10:33:11.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.130 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:11.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.445191+0000", 2024-09-21T10:33:11.130 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0068508939999999997 2024-09-21T10:33:11.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.131 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.453574+0000", 2024-09-21T10:33:11.131 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0083831280000000001 2024-09-21T10:33:11.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.131 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.131 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.453934+0000", 2024-09-21T10:33:11.131 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00036004800000000002 2024-09-21T10:33:11.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.131 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.131 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.454812+0000", 2024-09-21T10:33:11.131 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00087731799999999998 2024-09-21T10:33:11.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.132 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.132 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:11.132 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.454856+0000", 2024-09-21T10:33:11.132 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3945e-05 2024-09-21T10:33:11.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.132 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.132 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:11.132 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.454959+0000", 2024-09-21T10:33:11.132 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000102971 2024-09-21T10:33:11.132 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.132 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:11.132 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.133 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.133 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:181865 9.e 9:795ade75:::benchmark_data_smithi098_254407_object49134:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T10:33:11.133 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:24:43.337664+0000", 2024-09-21T10:33:11.133 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 507.74082254000001, 2024-09-21T10:33:11.133 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1178443219999998, 2024-09-21T10:33:11.133 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:11.133 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:11.133 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:11.133 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:11.133 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:11.133 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 181865 2024-09-21T10:33:11.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.133 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:11.133 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.134 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:11.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337664+0000", 2024-09-21T10:33:11.134 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.134 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:11.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337664+0000", 2024-09-21T10:33:11.134 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.134 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:11.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337663+0000", 2024-09-21T10:33:11.134 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:11.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.135 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.135 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:11.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337668+0000", 2024-09-21T10:33:11.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0579999999999997e-06 2024-09-21T10:33:11.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.135 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.135 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:11.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337678+0000", 2024-09-21T10:33:11.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.026e-05 2024-09-21T10:33:11.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.135 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.135 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:11.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337683+0000", 2024-09-21T10:33:11.136 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7790000000000003e-06 2024-09-21T10:33:11.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.136 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.136 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.136 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337726+0000", 2024-09-21T10:33:11.136 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3711999999999998e-05 2024-09-21T10:33:11.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.136 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.136 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:11.136 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:43.337733+0000", 2024-09-21T10:33:11.136 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4030000000000004e-06 2024-09-21T10:33:11.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.136 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.136 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.298435+0000", 2024-09-21T10:33:11.137 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.96070259599999996 2024-09-21T10:33:11.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.137 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-21T10:33:11.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:44.298446+0000", 2024-09-21T10:33:11.137 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0958000000000001e-05 2024-09-21T10:33:11.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.137 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.441414+0000", 2024-09-21T10:33:11.137 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.142967466 2024-09-21T10:33:11.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.138 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:11.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.441533+0000", 2024-09-21T10:33:11.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119294 2024-09-21T10:33:11.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.138 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.138 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,1,4", 2024-09-21T10:33:11.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.441673+0000", 2024-09-21T10:33:11.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014025100000000001 2024-09-21T10:33:11.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.138 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.138 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:11.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.446755+0000", 2024-09-21T10:33:11.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0050815249999999999 2024-09-21T10:33:11.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.139 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.139 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.453610+0000", 2024-09-21T10:33:11.139 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0068551269999999999 2024-09-21T10:33:11.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.139 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.139 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.453962+0000", 2024-09-21T10:33:11.139 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00035222299999999998 2024-09-21T10:33:11.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.139 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.139 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.455371+0000", 2024-09-21T10:33:11.139 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001408802 2024-09-21T10:33:11.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.140 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.140 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:11.140 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.455415+0000", 2024-09-21T10:33:11.140 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3591000000000003e-05 2024-09-21T10:33:11.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.140 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.140 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:11.140 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:24:45.455508+0000", 2024-09-21T10:33:11.140 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3774e-05 2024-09-21T10:33:11.140 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.140 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:11.140 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.141 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.141 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:265403 9.d 9:b0362428:::benchmark_data_smithi098_254407_object132672:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-09-21T10:33:11.141 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:16.862158+0000", 2024-09-21T10:33:11.141 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.21632886100002, 2024-09-21T10:33:11.141 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.187899645, 2024-09-21T10:33:11.141 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:11.141 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:11.141 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:11.141 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:11.141 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:11.141 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 265403 2024-09-21T10:33:11.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.141 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:11.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:11.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862158+0000", 2024-09-21T10:33:11.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:11.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862158+0000", 2024-09-21T10:33:11.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:11.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862156+0000", 2024-09-21T10:33:11.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:11.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.143 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:11.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862163+0000", 2024-09-21T10:33:11.143 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9249999999999998e-06 2024-09-21T10:33:11.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.144 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:11.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862176+0000", 2024-09-21T10:33:11.144 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3096e-05 2024-09-21T10:33:11.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.144 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.144 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:11.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862182+0000", 2024-09-21T10:33:11.144 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5219999999999998e-06 2024-09-21T10:33:11.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.144 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.144 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862418+0000", 2024-09-21T10:33:11.144 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00023681599999999999 2024-09-21T10:33:11.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.145 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:11.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862530+0000", 2024-09-21T10:33:11.145 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000111387 2024-09-21T10:33:11.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.145 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:33:11.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862638+0000", 2024-09-21T10:33:11.145 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010864 2024-09-21T10:33:11.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.145 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.874856+0000", 2024-09-21T10:33:11.146 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012217356 2024-09-21T10:33:11.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.146 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.146 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:11.146 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.874862+0000", 2024-09-21T10:33:11.146 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7790000000000001e-06 2024-09-21T10:33:11.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.146 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.146 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.146 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.049954+0000", 2024-09-21T10:33:11.146 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17509197300000001 2024-09-21T10:33:11.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.146 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.147 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:11.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.050012+0000", 2024-09-21T10:33:11.147 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8332999999999999e-05 2024-09-21T10:33:11.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.147 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.147 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:11.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.050057+0000", 2024-09-21T10:33:11.147 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5529999999999999e-05 2024-09-21T10:33:11.147 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.147 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:11.147 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.148 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.148 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:265404 9.d 9:be84246b:::benchmark_data_smithi098_254407_object132673:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-09-21T10:33:11.148 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:16.862588+0000", 2024-09-21T10:33:11.148 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.21589816800002, 2024-09-21T10:33:11.148 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.20599171599999999, 2024-09-21T10:33:11.148 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:11.148 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:11.148 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:11.148 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:11.148 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:11.148 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 265404 2024-09-21T10:33:11.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.149 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:11.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.149 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:11.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862588+0000", 2024-09-21T10:33:11.149 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.149 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:11.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862588+0000", 2024-09-21T10:33:11.149 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:11.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862588+0000", 2024-09-21T10:33:11.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:33:11.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:11.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862594+0000", 2024-09-21T10:33:11.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7509999999999997e-06 2024-09-21T10:33:11.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:11.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862602+0000", 2024-09-21T10:33:11.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4489999999999999e-06 2024-09-21T10:33:11.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:11.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862607+0000", 2024-09-21T10:33:11.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3240000000000002e-06 2024-09-21T10:33:11.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.871168+0000", 2024-09-21T10:33:11.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0085602720000000007 2024-09-21T10:33:11.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:11.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.871284+0000", 2024-09-21T10:33:11.152 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000116607 2024-09-21T10:33:11.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.152 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.152 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:33:11.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.871398+0000", 2024-09-21T10:33:11.152 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000113458 2024-09-21T10:33:11.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.152 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.152 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.878846+0000", 2024-09-21T10:33:11.153 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0074478829999999998 2024-09-21T10:33:11.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.153 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.153 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:11.153 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.878898+0000", 2024-09-21T10:33:11.153 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1996000000000002e-05 2024-09-21T10:33:11.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.153 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.153 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.153 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.068450+0000", 2024-09-21T10:33:11.153 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.189552424 2024-09-21T10:33:11.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.153 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.154 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:11.154 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.068524+0000", 2024-09-21T10:33:11.154 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3747999999999999e-05 2024-09-21T10:33:11.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.154 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.154 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:11.154 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.068580+0000", 2024-09-21T10:33:11.154 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6515999999999999e-05 2024-09-21T10:33:11.154 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.154 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:11.154 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.154 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.155 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:265406 9.7 9:e704644c:::benchmark_data_smithi098_254407_object132675:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-09-21T10:33:11.155 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:16.862782+0000", 2024-09-21T10:33:11.155 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.21570432499999, 2024-09-21T10:33:11.155 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.20006702200000001, 2024-09-21T10:33:11.155 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:11.155 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:11.155 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:11.155 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:11.155 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:11.155 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 265406 2024-09-21T10:33:11.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.155 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:11.155 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.155 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:11.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862782+0000", 2024-09-21T10:33:11.156 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.156 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.156 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:11.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862782+0000", 2024-09-21T10:33:11.156 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.156 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.156 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:11.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862782+0000", 2024-09-21T10:33:11.156 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999995 2024-09-21T10:33:11.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.157 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.157 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:11.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862785+0000", 2024-09-21T10:33:11.157 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4759999999999998e-06 2024-09-21T10:33:11.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.157 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.157 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:11.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862795+0000", 2024-09-21T10:33:11.157 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5950000000000005e-06 2024-09-21T10:33:11.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.158 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.158 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:11.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862799+0000", 2024-09-21T10:33:11.158 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2150000000000003e-06 2024-09-21T10:33:11.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.158 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.158 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862842+0000", 2024-09-21T10:33:11.158 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2454999999999999e-05 2024-09-21T10:33:11.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.158 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.159 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:11.159 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.862944+0000", 2024-09-21T10:33:11.159 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010283100000000001 2024-09-21T10:33:11.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.159 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.159 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:33:11.159 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.863048+0000", 2024-09-21T10:33:11.159 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104096 2024-09-21T10:33:11.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.159 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.159 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:11.159 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.873531+0000", 2024-09-21T10:33:11.159 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010482742 2024-09-21T10:33:11.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.160 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.160 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.160 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.873574+0000", 2024-09-21T10:33:11.160 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3056000000000002e-05 2024-09-21T10:33:11.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.160 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.160 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.160 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.062724+0000", 2024-09-21T10:33:11.160 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18914968600000001 2024-09-21T10:33:11.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.160 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.160 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:11.160 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.062805+0000", 2024-09-21T10:33:11.160 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0932000000000004e-05 2024-09-21T10:33:11.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.161 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.161 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:11.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.062849+0000", 2024-09-21T10:33:11.161 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4487999999999998e-05 2024-09-21T10:33:11.161 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.161 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:11.161 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.161 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.161 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:265407 9.7 9:eb620111:::benchmark_data_smithi098_254407_object132676:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-09-21T10:33:11.161 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:16.864645+0000", 2024-09-21T10:33:11.162 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.21384180500002, 2024-09-21T10:33:11.162 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.19854269299999999, 2024-09-21T10:33:11.162 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:11.162 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:11.162 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:11.162 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:11.162 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:11.162 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 265407 2024-09-21T10:33:11.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.162 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:11.162 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.162 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:11.162 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864645+0000", 2024-09-21T10:33:11.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.163 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:11.163 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864645+0000", 2024-09-21T10:33:11.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.163 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:11.163 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864643+0000", 2024-09-21T10:33:11.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:11.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:11.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864649+0000", 2024-09-21T10:33:11.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9560000000000002e-06 2024-09-21T10:33:11.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:11.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864667+0000", 2024-09-21T10:33:11.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7668000000000001e-05 2024-09-21T10:33:11.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:11.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.864673+0000", 2024-09-21T10:33:11.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3319999999999997e-06 2024-09-21T10:33:11.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.866735+0000", 2024-09-21T10:33:11.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020622549999999998 2024-09-21T10:33:11.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:11.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.866851+0000", 2024-09-21T10:33:11.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000115458 2024-09-21T10:33:11.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:33:11.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.866969+0000", 2024-09-21T10:33:11.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000118417 2024-09-21T10:33:11.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:11.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.874132+0000", 2024-09-21T10:33:11.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0071623679999999997 2024-09-21T10:33:11.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.876314+0000", 2024-09-21T10:33:11.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021818670000000001 2024-09-21T10:33:11.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.168 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.063081+0000", 2024-09-21T10:33:11.168 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18676781200000001 2024-09-21T10:33:11.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.168 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.168 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:11.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.063151+0000", 2024-09-21T10:33:11.168 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9803999999999998e-05 2024-09-21T10:33:11.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.168 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.168 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:11.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.063188+0000", 2024-09-21T10:33:11.168 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6402999999999997e-05 2024-09-21T10:33:11.168 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.169 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:11.169 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.169 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.169 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:265415 9.d 9:bc500964:::benchmark_data_smithi098_254407_object132684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-09-21T10:33:11.169 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:16.872895+0000", 2024-09-21T10:33:11.169 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.20559159800001, 2024-09-21T10:33:11.169 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.19584159200000001, 2024-09-21T10:33:11.169 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:11.169 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:11.169 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:11.169 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:11.169 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:11.169 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 265415 2024-09-21T10:33:11.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.170 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:11.170 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.170 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:11.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.872895+0000", 2024-09-21T10:33:11.170 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.170 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.170 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:11.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.872895+0000", 2024-09-21T10:33:11.170 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.171 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:11.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.872894+0000", 2024-09-21T10:33:11.171 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:11.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.171 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:11.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.872900+0000", 2024-09-21T10:33:11.171 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5719999999999999e-06 2024-09-21T10:33:11.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.171 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:11.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.872909+0000", 2024-09-21T10:33:11.172 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.064e-06 2024-09-21T10:33:11.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.172 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.172 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:11.172 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.872914+0000", 2024-09-21T10:33:11.172 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1810000000000003e-06 2024-09-21T10:33:11.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.172 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.172 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.172 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.874897+0000", 2024-09-21T10:33:11.172 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001983369 2024-09-21T10:33:11.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.172 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.172 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:11.173 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.875001+0000", 2024-09-21T10:33:11.173 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104219 2024-09-21T10:33:11.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.173 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.173 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:33:11.173 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.875102+0000", 2024-09-21T10:33:11.173 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010056300000000001 2024-09-21T10:33:11.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.173 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.173 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.173 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.878884+0000", 2024-09-21T10:33:11.173 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0037825889999999998 2024-09-21T10:33:11.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.173 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.173 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:11.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.878921+0000", 2024-09-21T10:33:11.174 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6433999999999997e-05 2024-09-21T10:33:11.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.174 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.174 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.068610+0000", 2024-09-21T10:33:11.174 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.189689575 2024-09-21T10:33:11.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.174 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.174 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:11.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.068704+0000", 2024-09-21T10:33:11.174 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3576e-05 2024-09-21T10:33:11.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.175 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.175 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:11.175 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.068737+0000", 2024-09-21T10:33:11.175 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2762000000000002e-05 2024-09-21T10:33:11.175 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.175 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:11.175 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.175 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.175 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:265428 9.7 9:e73ca6e3:::benchmark_data_smithi098_254407_object132697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-09-21T10:33:11.175 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:16.889473+0000", 2024-09-21T10:33:11.175 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.18901396899997, 2024-09-21T10:33:11.176 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.173842887, 2024-09-21T10:33:11.176 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:11.176 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:11.176 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:11.176 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:11.176 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:11.176 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 265428 2024-09-21T10:33:11.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.176 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:11.176 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.176 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:11.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.889473+0000", 2024-09-21T10:33:11.176 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.176 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.177 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:11.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.889473+0000", 2024-09-21T10:33:11.177 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.177 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.177 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:11.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.889471+0000", 2024-09-21T10:33:11.177 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:11.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.177 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.177 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:11.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.889479+0000", 2024-09-21T10:33:11.178 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4619999999999999e-06 2024-09-21T10:33:11.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.178 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.178 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:11.178 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.889493+0000", 2024-09-21T10:33:11.178 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4270999999999999e-05 2024-09-21T10:33:11.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.178 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.178 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:11.178 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.889499+0000", 2024-09-21T10:33:11.178 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4760000000000003e-06 2024-09-21T10:33:11.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.178 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.179 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.889535+0000", 2024-09-21T10:33:11.179 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5768e-05 2024-09-21T10:33:11.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.179 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.179 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:11.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.889613+0000", 2024-09-21T10:33:11.179 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7492999999999998e-05 2024-09-21T10:33:11.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.179 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.179 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:33:11.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.889679+0000", 2024-09-21T10:33:11.179 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6655000000000005e-05 2024-09-21T10:33:11.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.180 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.180 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:11.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.892647+0000", 2024-09-21T10:33:11.180 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029680240000000001 2024-09-21T10:33:11.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.180 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.180 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.892973+0000", 2024-09-21T10:33:11.180 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00032576099999999999 2024-09-21T10:33:11.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.180 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.180 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.063219+0000", 2024-09-21T10:33:11.180 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17024564 2024-09-21T10:33:11.181 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.181 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.181 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:11.181 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.063284+0000", 2024-09-21T10:33:11.181 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4943000000000005e-05 2024-09-21T10:33:11.181 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.181 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.181 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:11.181 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.063316+0000", 2024-09-21T10:33:11.181 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1937000000000003e-05 2024-09-21T10:33:11.181 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.181 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:11.181 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.181 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.182 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.182 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46025.0:265429 9.d 9:b5f6dec9:::benchmark_data_smithi098_254407_object132698:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-09-21T10:33:11.182 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:26:16.890079+0000", 2024-09-21T10:33:11.182 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 414.18840784000002, 2024-09-21T10:33:11.182 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18602902499999999, 2024-09-21T10:33:11.182 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:11.182 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:11.182 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:11.182 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46025", 2024-09-21T10:33:11.182 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/2363119492", 2024-09-21T10:33:11.182 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 265429 2024-09-21T10:33:11.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.182 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:11.183 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.183 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:11.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.890079+0000", 2024-09-21T10:33:11.183 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.183 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.183 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:11.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.890079+0000", 2024-09-21T10:33:11.183 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.183 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.183 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:11.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.890078+0000", 2024-09-21T10:33:11.184 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:33:11.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.184 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.184 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:11.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.890083+0000", 2024-09-21T10:33:11.184 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9799999999999998e-06 2024-09-21T10:33:11.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.184 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.184 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:11.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.890096+0000", 2024-09-21T10:33:11.184 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2683e-05 2024-09-21T10:33:11.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.184 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.185 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:11.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.890101+0000", 2024-09-21T10:33:11.185 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7089999999999999e-06 2024-09-21T10:33:11.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.185 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.185 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.890144+0000", 2024-09-21T10:33:11.185 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2836000000000001e-05 2024-09-21T10:33:11.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.185 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.185 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:11.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.890229+0000", 2024-09-21T10:33:11.185 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5389999999999994e-05 2024-09-21T10:33:11.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.186 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.186 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:33:11.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.890294+0000", 2024-09-21T10:33:11.186 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4559000000000003e-05 2024-09-21T10:33:11.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.186 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.186 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:11.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.892970+0000", 2024-09-21T10:33:11.186 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026759370000000002 2024-09-21T10:33:11.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.186 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.186 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:16.893239+0000", 2024-09-21T10:33:11.187 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00026952699999999998 2024-09-21T10:33:11.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.187 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.187 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.187 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.075955+0000", 2024-09-21T10:33:11.187 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18271579900000001 2024-09-21T10:33:11.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.187 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.187 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:11.187 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.076065+0000", 2024-09-21T10:33:11.187 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109869 2024-09-21T10:33:11.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.187 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.187 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:11.188 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:26:17.076108+0000", 2024-09-21T10:33:11.188 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2698999999999999e-05 2024-09-21T10:33:11.188 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.188 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:11.188 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.188 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.188 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46322.0:115193 10.c 10:3afafc88:::benchmark_data_smithi098_281896_object115192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-21T10:33:11.188 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:28:03.255737+0000", 2024-09-21T10:33:11.188 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 307.82274926299999, 2024-09-21T10:33:11.188 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.11935325500000001, 2024-09-21T10:33:11.188 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:11.188 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:11.188 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:11.189 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46322", 2024-09-21T10:33:11.189 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:33:11.189 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 115193 2024-09-21T10:33:11.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.189 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:11.189 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.189 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:11.189 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.255737+0000", 2024-09-21T10:33:11.189 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.189 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.189 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:11.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.255737+0000", 2024-09-21T10:33:11.190 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.190 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.190 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:11.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.255736+0000", 2024-09-21T10:33:11.190 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:11.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.191 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.191 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:11.191 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.255828+0000", 2024-09-21T10:33:11.191 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2367999999999994e-05 2024-09-21T10:33:11.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.191 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.191 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:11.192 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.255842+0000", 2024-09-21T10:33:11.192 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3935e-05 2024-09-21T10:33:11.192 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.192 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.192 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:11.192 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.255847+0000", 2024-09-21T10:33:11.192 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8250000000000004e-06 2024-09-21T10:33:11.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.193 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.193 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.193 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.255888+0000", 2024-09-21T10:33:11.193 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0448999999999998e-05 2024-09-21T10:33:11.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.194 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.194 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:11.194 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.255980+0000", 2024-09-21T10:33:11.194 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2737000000000003e-05 2024-09-21T10:33:11.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.194 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.194 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:33:11.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.256074+0000", 2024-09-21T10:33:11.195 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3426999999999998e-05 2024-09-21T10:33:11.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.195 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.195 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:11.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.261505+0000", 2024-09-21T10:33:11.195 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.005431015 2024-09-21T10:33:11.196 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.196 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.196 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.196 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.261537+0000", 2024-09-21T10:33:11.196 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2450000000000003e-05 2024-09-21T10:33:11.196 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.197 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.197 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.197 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.374986+0000", 2024-09-21T10:33:11.197 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.11344886899999999 2024-09-21T10:33:11.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.197 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.198 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:11.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.375046+0000", 2024-09-21T10:33:11.198 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9871000000000003e-05 2024-09-21T10:33:11.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.198 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.198 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:11.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.375091+0000", 2024-09-21T10:33:11.199 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.464e-05 2024-09-21T10:33:11.199 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.199 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:11.199 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.199 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.199 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.200 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46322.0:115194 10.c 10:36de5723:::benchmark_data_smithi098_281896_object115193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-21T10:33:11.200 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:28:03.256882+0000", 2024-09-21T10:33:11.200 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 307.821604589, 2024-09-21T10:33:11.200 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.118306798, 2024-09-21T10:33:11.200 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:11.200 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:11.201 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:11.201 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46322", 2024-09-21T10:33:11.201 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:33:11.201 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 115194 2024-09-21T10:33:11.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.201 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:11.201 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.202 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:11.202 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.256882+0000", 2024-09-21T10:33:11.202 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.202 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.203 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:11.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.256882+0000", 2024-09-21T10:33:11.203 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.203 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.203 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:11.204 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.256881+0000", 2024-09-21T10:33:11.204 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.999999 2024-09-21T10:33:11.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.204 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.205 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:11.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.256949+0000", 2024-09-21T10:33:11.205 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8158999999999996e-05 2024-09-21T10:33:11.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.205 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.205 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:11.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.256960+0000", 2024-09-21T10:33:11.206 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0475000000000001e-05 2024-09-21T10:33:11.206 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.206 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.206 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:11.206 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.256964+0000", 2024-09-21T10:33:11.206 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7500000000000001e-06 2024-09-21T10:33:11.207 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.207 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.207 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.207 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.258751+0000", 2024-09-21T10:33:11.207 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0017875580000000001 2024-09-21T10:33:11.208 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.208 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.208 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:11.208 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.258833+0000", 2024-09-21T10:33:11.208 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1308000000000001e-05 2024-09-21T10:33:11.208 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.208 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.209 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:33:11.209 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.258931+0000", 2024-09-21T10:33:11.209 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8045000000000001e-05 2024-09-21T10:33:11.209 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.209 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.209 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:11.209 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.261642+0000", 2024-09-21T10:33:11.210 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027117729999999998 2024-09-21T10:33:11.210 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.210 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.210 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.210 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.264241+0000", 2024-09-21T10:33:11.210 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025984179999999999 2024-09-21T10:33:11.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.211 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.211 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.211 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.375123+0000", 2024-09-21T10:33:11.211 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.110881718 2024-09-21T10:33:11.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.212 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.212 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:11.212 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.375160+0000", 2024-09-21T10:33:11.212 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7440000000000001e-05 2024-09-21T10:33:11.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.212 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.212 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:11.213 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:03.375189+0000", 2024-09-21T10:33:11.213 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8898999999999999e-05 2024-09-21T10:33:11.213 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.213 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:11.213 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.214 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.214 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46322.0:141177 10.c 10:35bed12b:::benchmark_data_smithi098_281896_object16152:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-21T10:33:11.214 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:28:40.807186+0000", 2024-09-21T10:33:11.214 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 270.27130103600001, 2024-09-21T10:33:11.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.136343505, 2024-09-21T10:33:11.214 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:11.214 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:11.215 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:11.215 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46322", 2024-09-21T10:33:11.215 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:33:11.215 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 141177 2024-09-21T10:33:11.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.215 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:11.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.216 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:11.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.807186+0000", 2024-09-21T10:33:11.216 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.216 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:11.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.807186+0000", 2024-09-21T10:33:11.216 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.217 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:11.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.807184+0000", 2024-09-21T10:33:11.217 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:11.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.217 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.217 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:11.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.807193+0000", 2024-09-21T10:33:11.217 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9940000000000006e-06 2024-09-21T10:33:11.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.217 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.217 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:11.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.807205+0000", 2024-09-21T10:33:11.217 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1576000000000001e-05 2024-09-21T10:33:11.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.218 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.218 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:11.218 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.807210+0000", 2024-09-21T10:33:11.218 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7829999999999998e-06 2024-09-21T10:33:11.218 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.218 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.218 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.218 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.807252+0000", 2024-09-21T10:33:11.218 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2020000000000001e-05 2024-09-21T10:33:11.218 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.218 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.218 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:11.218 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.807332+0000", 2024-09-21T10:33:11.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0029000000000002e-05 2024-09-21T10:33:11.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.219 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.219 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:33:11.219 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.807400+0000", 2024-09-21T10:33:11.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8283999999999999e-05 2024-09-21T10:33:11.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.219 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.219 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:11.219 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.810860+0000", 2024-09-21T10:33:11.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034603360000000001 2024-09-21T10:33:11.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.219 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.219 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.811430+0000", 2024-09-21T10:33:11.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00056996800000000002 2024-09-21T10:33:11.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.943442+0000", 2024-09-21T10:33:11.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13201190500000001 2024-09-21T10:33:11.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:11.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.943490+0000", 2024-09-21T10:33:11.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8007999999999998e-05 2024-09-21T10:33:11.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.221 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:11.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.943529+0000", 2024-09-21T10:33:11.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8911000000000001e-05 2024-09-21T10:33:11.221 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.221 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:11.221 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.221 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46322.0:141180 10.c 10:3d593fe7:::benchmark_data_smithi098_281896_object16155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-21T10:33:11.221 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:28:40.813316+0000", 2024-09-21T10:33:11.221 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 270.26517024499998, 2024-09-21T10:33:11.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17292048700000001, 2024-09-21T10:33:11.221 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:11.222 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:11.222 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:11.222 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46322", 2024-09-21T10:33:11.222 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:33:11.222 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 141180 2024-09-21T10:33:11.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.222 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:11.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:11.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.813316+0000", 2024-09-21T10:33:11.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:11.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.813316+0000", 2024-09-21T10:33:11.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:11.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.813315+0000", 2024-09-21T10:33:11.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:11.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:11.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.813324+0000", 2024-09-21T10:33:11.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2269999999999999e-06 2024-09-21T10:33:11.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:11.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.813338+0000", 2024-09-21T10:33:11.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3951000000000001e-05 2024-09-21T10:33:11.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:11.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.813344+0000", 2024-09-21T10:33:11.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6409999999999999e-06 2024-09-21T10:33:11.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.813382+0000", 2024-09-21T10:33:11.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8318999999999997e-05 2024-09-21T10:33:11.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:11.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.813497+0000", 2024-09-21T10:33:11.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000115387 2024-09-21T10:33:11.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:33:11.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.813607+0000", 2024-09-21T10:33:11.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011003800000000001 2024-09-21T10:33:11.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.226 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:11.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.817642+0000", 2024-09-21T10:33:11.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0040343280000000002 2024-09-21T10:33:11.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.226 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.817863+0000", 2024-09-21T10:33:11.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00022094000000000001 2024-09-21T10:33:11.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.226 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.986127+0000", 2024-09-21T10:33:11.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.168264527 2024-09-21T10:33:11.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.227 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:11.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.986196+0000", 2024-09-21T10:33:11.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8460000000000005e-05 2024-09-21T10:33:11.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.227 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:11.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.986237+0000", 2024-09-21T10:33:11.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1356000000000002e-05 2024-09-21T10:33:11.227 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.227 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:11.228 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.228 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.228 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46322.0:141196 10.c 10:36a81655:::benchmark_data_smithi098_281896_object16171:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-21T10:33:11.228 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:28:40.833528+0000", 2024-09-21T10:33:11.228 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 270.24495828099998, 2024-09-21T10:33:11.228 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16630146700000001, 2024-09-21T10:33:11.228 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:11.228 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:11.228 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:11.228 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46322", 2024-09-21T10:33:11.229 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:33:11.229 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 141196 2024-09-21T10:33:11.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.229 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:11.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:11.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.833528+0000", 2024-09-21T10:33:11.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:11.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.833528+0000", 2024-09-21T10:33:11.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:11.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.833527+0000", 2024-09-21T10:33:11.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:11.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:11.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.833534+0000", 2024-09-21T10:33:11.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1620000000000002e-06 2024-09-21T10:33:11.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:11.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.833549+0000", 2024-09-21T10:33:11.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5021000000000001e-05 2024-09-21T10:33:11.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:11.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.833555+0000", 2024-09-21T10:33:11.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6000000000000003e-06 2024-09-21T10:33:11.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.833603+0000", 2024-09-21T10:33:11.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7586999999999997e-05 2024-09-21T10:33:11.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:11.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.833722+0000", 2024-09-21T10:33:11.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119306 2024-09-21T10:33:11.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:33:11.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.833840+0000", 2024-09-21T10:33:11.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000117861 2024-09-21T10:33:11.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:11.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.837869+0000", 2024-09-21T10:33:11.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0040292380000000001 2024-09-21T10:33:11.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.838114+0000", 2024-09-21T10:33:11.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00024487000000000001 2024-09-21T10:33:11.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.999732+0000", 2024-09-21T10:33:11.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.161618176 2024-09-21T10:33:11.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:11.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.999787+0000", 2024-09-21T10:33:11.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4877000000000003e-05 2024-09-21T10:33:11.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.234 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:11.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:28:40.999830+0000", 2024-09-21T10:33:11.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2453000000000002e-05 2024-09-21T10:33:11.234 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.234 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:11.234 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.234 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.46322.0:244327 10.c 10:3c49acf6:::benchmark_data_smithi098_281896_object119302:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-21T10:33:11.234 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-21T10:31:09.686482+0000", 2024-09-21T10:33:11.234 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 121.392004773, 2024-09-21T10:33:11.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.142227613, 2024-09-21T10:33:11.234 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-21T10:33:11.235 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:11.235 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-21T10:33:11.235 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.46322", 2024-09-21T10:33:11.235 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.98:0/1156070446", 2024-09-21T10:33:11.235 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 244327 2024-09-21T10:33:11.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.235 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-21T10:33:11.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-21T10:33:11.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:31:09.686482+0000", 2024-09-21T10:33:11.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-21T10:33:11.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:31:09.686482+0000", 2024-09-21T10:33:11.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-21T10:33:11.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-21T10:33:11.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:31:09.686480+0000", 2024-09-21T10:33:11.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:11.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-21T10:33:11.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:31:09.686486+0000", 2024-09-21T10:33:11.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4969999999999998e-06 2024-09-21T10:33:11.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-21T10:33:11.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:31:09.686503+0000", 2024-09-21T10:33:11.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6444e-05 2024-09-21T10:33:11.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.237 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-21T10:33:11.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:31:09.686510+0000", 2024-09-21T10:33:11.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6390000000000003e-06 2024-09-21T10:33:11.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.237 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-21T10:33:11.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:31:09.686633+0000", 2024-09-21T10:33:11.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012397499999999999 2024-09-21T10:33:11.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.238 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-21T10:33:11.238 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:31:09.686746+0000", 2024-09-21T10:33:11.238 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000112862 2024-09-21T10:33:11.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.238 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.238 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:33:11.238 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:31:09.686857+0000", 2024-09-21T10:33:11.238 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000110764 2024-09-21T10:33:11.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.238 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.238 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-21T10:33:11.238 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:31:09.691166+0000", 2024-09-21T10:33:11.238 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043085040000000003 2024-09-21T10:33:11.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:31:09.696137+0000", 2024-09-21T10:33:11.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049709580000000001 2024-09-21T10:33:11.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:11.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:31:09.828631+0000", 2024-09-21T10:33:11.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13249414000000001 2024-09-21T10:33:11.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-21T10:33:11.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:31:09.828676+0000", 2024-09-21T10:33:11.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5114999999999997e-05 2024-09-21T10:33:11.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-21T10:33:11.240 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-21T10:33:11.240 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-21T10:33:11.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-21T10:31:09.828709+0000", 2024-09-21T10:33:11.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3649000000000003e-05 2024-09-21T10:33:11.240 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.240 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:11.240 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.240 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-21T10:33:11.240 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-21T10:33:11.240 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-21T10:33:11.759 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:11 smithi098 ceph-mon[97580]: pgmap v1432: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:11.760 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:11 smithi098 ceph-mon[100914]: pgmap v1432: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:11.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:11 smithi196 ceph-mon[84784]: pgmap v1432: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-21T10:33:12.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:12 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1972045846' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:33:12.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:12 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1972045846' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:33:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:12 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1972045846' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:33:13.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T10:33:13.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:13 smithi098 ceph-mon[97580]: pgmap v1433: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:13.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:13 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1972045846' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:33:13.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:13 smithi098 ceph-mon[97580]: osdmap e416: 8 total, 8 up, 8 in 2024-09-21T10:33:13.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:13 smithi098 ceph-mon[100914]: pgmap v1433: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:13.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:13 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1972045846' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:33:13.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:13 smithi098 ceph-mon[100914]: osdmap e416: 8 total, 8 up, 8 in 2024-09-21T10:33:13.598 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:13.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:13 smithi196 ceph-mon[84784]: pgmap v1433: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:13 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1972045846' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:33:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:13 smithi196 ceph-mon[84784]: osdmap e416: 8 total, 8 up, 8 in 2024-09-21T10:33:14.726 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:14 smithi098 ceph-mon[97580]: pgmap v1435: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:14.726 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:14 smithi098 ceph-mon[100914]: pgmap v1435: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:14 smithi196 ceph-mon[84784]: pgmap v1435: 83 pgs: 83 active+clean; 4.6 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:15.532 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T10:33:15.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:15 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/39294757' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:33:15.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:15 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:33:15.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:15 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/39294757' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:33:15.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:15 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:33:15.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:15 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/39294757' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:33:15.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:15 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:33:16.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 4 1 2024-09-21T10:33:16.537 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:16.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:16 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:33:16.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:16 smithi098 ceph-mon[97580]: osdmap e417: 8 total, 8 up, 8 in 2024-09-21T10:33:16.597 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:16 smithi098 ceph-mon[97580]: pgmap v1437: 83 pgs: 83 active+clean; 4.6 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:16.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:16 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:33:16.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:16 smithi098 ceph-mon[100914]: osdmap e417: 8 total, 8 up, 8 in 2024-09-21T10:33:16.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:16 smithi098 ceph-mon[100914]: pgmap v1437: 83 pgs: 83 active+clean; 4.6 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:16 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:33:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:16 smithi196 ceph-mon[84784]: osdmap e417: 8 total, 8 up, 8 in 2024-09-21T10:33:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:16 smithi196 ceph-mon[84784]: pgmap v1437: 83 pgs: 83 active+clean; 4.6 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:18.734 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-21T10:33:19.018 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:18 smithi098 ceph-mon[97580]: pgmap v1438: 83 pgs: 83 active+clean; 4.6 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:19.019 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:18 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/127632334' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:33:19.019 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:18 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:33:19.019 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:18 smithi098 ceph-mon[100914]: pgmap v1438: 83 pgs: 83 active+clean; 4.6 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:19.019 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:18 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/127632334' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:33:19.019 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:18 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:33:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:18 smithi196 ceph-mon[84784]: pgmap v1438: 83 pgs: 83 active+clean; 4.6 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:18 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/127632334' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:33:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:18 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:33:19.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T10:33:19.616 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:20.015 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:19 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:33:20.015 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:19 smithi098 ceph-mon[97580]: osdmap e418: 8 total, 8 up, 8 in 2024-09-21T10:33:20.015 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:19 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:33:20.015 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:19 smithi098 ceph-mon[100914]: osdmap e418: 8 total, 8 up, 8 in 2024-09-21T10:33:20.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:19 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:33:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:19 smithi196 ceph-mon[84784]: osdmap e418: 8 total, 8 up, 8 in 2024-09-21T10:33:20.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:33:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:33:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:33:20.941 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:20 smithi098 ceph-mon[97580]: pgmap v1440: 83 pgs: 83 active+clean; 4.6 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:20.941 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:33:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:33:20] "GET /metrics HTTP/1.1" 200 34162 "" "Prometheus/2.51.0" 2024-09-21T10:33:20.941 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:20 smithi098 ceph-mon[100914]: pgmap v1440: 83 pgs: 83 active+clean; 4.6 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:21.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:20 smithi196 ceph-mon[84784]: pgmap v1440: 83 pgs: 83 active+clean; 4.6 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:21.750 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T10:33:21.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:21 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/6715993' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:33:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:21 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:33:22.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:21 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/6715993' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:33:22.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:21 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:33:22.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:21 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/6715993' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:33:22.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:21 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:33:22.752 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 5 1 2024-09-21T10:33:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:23.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:22 smithi098 ceph-mon[97580]: pgmap v1441: 83 pgs: 83 active+clean; 4.6 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:23.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:22 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:33:23.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:22 smithi098 ceph-mon[97580]: osdmap e419: 8 total, 8 up, 8 in 2024-09-21T10:33:23.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:23.032 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:22 smithi098 ceph-mon[100914]: pgmap v1441: 83 pgs: 83 active+clean; 4.6 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:23.032 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:22 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:33:23.032 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:22 smithi098 ceph-mon[100914]: osdmap e419: 8 total, 8 up, 8 in 2024-09-21T10:33:23.032 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:22 smithi196 ceph-mon[84784]: pgmap v1441: 83 pgs: 83 active+clean; 4.6 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:22 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:33:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:22 smithi196 ceph-mon[84784]: osdmap e419: 8 total, 8 up, 8 in 2024-09-21T10:33:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:25.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:24 smithi098 ceph-mon[97580]: pgmap v1443: 83 pgs: 83 active+clean; 4.6 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:25.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:24 smithi098 ceph-mon[100914]: pgmap v1443: 83 pgs: 83 active+clean; 4.6 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:25.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:24 smithi196 ceph-mon[84784]: pgmap v1443: 83 pgs: 83 active+clean; 4.6 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:25.758 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-21T10:33:26.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:25 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3866823047' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:33:26.088 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:25 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:33:26.088 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:25 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3866823047' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:33:26.089 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:25 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:33:26.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:25 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3866823047' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:33:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:25 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:33:26.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T10:33:26.775 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:27.013 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:26 smithi098 ceph-mon[97580]: pgmap v1444: 83 pgs: 83 active+clean; 4.6 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:27.013 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:26 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:33:27.014 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:26 smithi098 ceph-mon[97580]: osdmap e420: 8 total, 8 up, 8 in 2024-09-21T10:33:27.014 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:26 smithi098 ceph-mon[100914]: pgmap v1444: 83 pgs: 83 active+clean; 4.6 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:27.014 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:26 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:33:27.014 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:26 smithi098 ceph-mon[100914]: osdmap e420: 8 total, 8 up, 8 in 2024-09-21T10:33:27.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:26 smithi196 ceph-mon[84784]: pgmap v1444: 83 pgs: 83 active+clean; 4.6 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:26 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:33:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:26 smithi196 ceph-mon[84784]: osdmap e420: 8 total, 8 up, 8 in 2024-09-21T10:33:28.030 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:27 smithi098 ceph-mon[97580]: osdmap e421: 8 total, 8 up, 8 in 2024-09-21T10:33:28.030 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:27 smithi098 ceph-mon[97580]: 1.a deep-scrub starts 2024-09-21T10:33:28.030 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:27 smithi098 ceph-mon[97580]: 1.a deep-scrub ok 2024-09-21T10:33:28.031 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:27 smithi098 ceph-mon[100914]: osdmap e421: 8 total, 8 up, 8 in 2024-09-21T10:33:28.031 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:27 smithi098 ceph-mon[100914]: 1.a deep-scrub starts 2024-09-21T10:33:28.031 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:27 smithi098 ceph-mon[100914]: 1.a deep-scrub ok 2024-09-21T10:33:28.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:27 smithi196 ceph-mon[84784]: osdmap e421: 8 total, 8 up, 8 in 2024-09-21T10:33:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:27 smithi196 ceph-mon[84784]: 1.a deep-scrub starts 2024-09-21T10:33:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:27 smithi196 ceph-mon[84784]: 1.a deep-scrub ok 2024-09-21T10:33:28.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:28 smithi098 ceph-mon[100914]: pgmap v1447: 83 pgs: 83 active+clean; 4.6 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:28.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:28 smithi098 ceph-mon[97580]: pgmap v1447: 83 pgs: 83 active+clean; 4.6 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:28 smithi196 ceph-mon[84784]: pgmap v1447: 83 pgs: 83 active+clean; 4.6 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:29.774 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T10:33:30.080 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:29 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2266862448' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:33:30.080 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:29 smithi098 ceph-mon[100914]: 1.1 scrub starts 2024-09-21T10:33:30.080 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:29 smithi098 ceph-mon[100914]: 1.1 scrub ok 2024-09-21T10:33:30.081 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:29 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2266862448' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:33:30.081 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:29 smithi098 ceph-mon[97580]: 1.1 scrub starts 2024-09-21T10:33:30.081 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:29 smithi098 ceph-mon[97580]: 1.1 scrub ok 2024-09-21T10:33:30.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:29 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2266862448' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:33:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:29 smithi196 ceph-mon[84784]: 1.1 scrub starts 2024-09-21T10:33:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:29 smithi196 ceph-mon[84784]: 1.1 scrub ok 2024-09-21T10:33:30.470 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 6 1 2024-09-21T10:33:30.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:33:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:33:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:33:30.719 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:30.775 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:33:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:33:30] "GET /metrics HTTP/1.1" 200 34162 "" "Prometheus/2.51.0" 2024-09-21T10:33:31.084 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:30 smithi098 ceph-mon[97580]: pgmap v1448: 83 pgs: 83 active+clean; 4.4 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 17 KiB/s rd, 37 KiB/s wr, 10 op/s 2024-09-21T10:33:31.084 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:30 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2266862448' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:33:31.085 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:30 smithi098 ceph-mon[97580]: osdmap e422: 8 total, 8 up, 8 in 2024-09-21T10:33:31.085 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:30 smithi098 ceph-mon[97580]: 1.8 scrub starts 2024-09-21T10:33:31.085 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:30 smithi098 ceph-mon[97580]: 1.8 scrub ok 2024-09-21T10:33:31.085 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:30 smithi098 ceph-mon[100914]: pgmap v1448: 83 pgs: 83 active+clean; 4.4 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 17 KiB/s rd, 37 KiB/s wr, 10 op/s 2024-09-21T10:33:31.085 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:30 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2266862448' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:33:31.085 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:30 smithi098 ceph-mon[100914]: osdmap e422: 8 total, 8 up, 8 in 2024-09-21T10:33:31.085 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:30 smithi098 ceph-mon[100914]: 1.8 scrub starts 2024-09-21T10:33:31.085 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:30 smithi098 ceph-mon[100914]: 1.8 scrub ok 2024-09-21T10:33:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:30 smithi196 ceph-mon[84784]: pgmap v1448: 83 pgs: 83 active+clean; 4.4 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 17 KiB/s rd, 37 KiB/s wr, 10 op/s 2024-09-21T10:33:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:30 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2266862448' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:33:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:30 smithi196 ceph-mon[84784]: osdmap e422: 8 total, 8 up, 8 in 2024-09-21T10:33:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:30 smithi196 ceph-mon[84784]: 1.8 scrub starts 2024-09-21T10:33:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:30 smithi196 ceph-mon[84784]: 1.8 scrub ok 2024-09-21T10:33:32.786 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-21T10:33:33.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:32 smithi098 ceph-mon[97580]: pgmap v1450: 83 pgs: 83 active+clean; 4.4 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 23 KiB/s rd, 49 KiB/s wr, 13 op/s 2024-09-21T10:33:33.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:32 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/4069640801' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:33:33.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:32 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:33:33.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:32 smithi098 ceph-mon[100914]: pgmap v1450: 83 pgs: 83 active+clean; 4.4 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 23 KiB/s rd, 49 KiB/s wr, 13 op/s 2024-09-21T10:33:33.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:32 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/4069640801' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:33:33.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:32 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:33:33.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:32 smithi196 ceph-mon[84784]: pgmap v1450: 83 pgs: 83 active+clean; 4.4 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 23 KiB/s rd, 49 KiB/s wr, 13 op/s 2024-09-21T10:33:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:32 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/4069640801' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:33:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:32 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:33:33.591 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T10:33:33.836 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:34.070 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:33 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:33:34.070 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:33 smithi098 ceph-mon[97580]: osdmap e423: 8 total, 8 up, 8 in 2024-09-21T10:33:34.071 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:33 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:33:34.071 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:33 smithi098 ceph-mon[100914]: osdmap e423: 8 total, 8 up, 8 in 2024-09-21T10:33:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:33 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:33:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:33 smithi196 ceph-mon[84784]: osdmap e423: 8 total, 8 up, 8 in 2024-09-21T10:33:35.007 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:34 smithi098 ceph-mon[97580]: pgmap v1452: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:35.007 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:34 smithi098 ceph-mon[100914]: pgmap v1452: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:34 smithi196 ceph-mon[84784]: pgmap v1452: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:35.800 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T10:33:36.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:35 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3305703628' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:33:36.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:35 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:33:36.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:35 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3305703628' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:33:36.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:35 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:33:36.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:35 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3305703628' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:33:36.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:35 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:33:36.532 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 7 1 2024-09-21T10:33:36.775 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:37.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:36 smithi098 ceph-mon[97580]: pgmap v1453: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:37.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:36 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:33:37.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:36 smithi098 ceph-mon[97580]: osdmap e424: 8 total, 8 up, 8 in 2024-09-21T10:33:37.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:36 smithi098 ceph-mon[100914]: pgmap v1453: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:37.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:36 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:33:37.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:36 smithi098 ceph-mon[100914]: osdmap e424: 8 total, 8 up, 8 in 2024-09-21T10:33:37.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:36 smithi196 ceph-mon[84784]: pgmap v1453: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:36 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:33:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:36 smithi196 ceph-mon[84784]: osdmap e424: 8 total, 8 up, 8 in 2024-09-21T10:33:38.082 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:37 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:38.083 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:37 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:38.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:37 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:38.814 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-21T10:33:39.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:38 smithi098 ceph-mon[97580]: pgmap v1455: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:39.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:38 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/912581623' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:33:39.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:38 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:33:39.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:38 smithi098 ceph-mon[100914]: pgmap v1455: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:39.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:38 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/912581623' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:33:39.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:38 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:33:39.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:38 smithi196 ceph-mon[84784]: pgmap v1455: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:38 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/912581623' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:33:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:38 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:33:39.566 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-21T10:33:39.566 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-21T10:33:39.567 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-21T10:33:39.567 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-21T10:33:39.567 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T10:33:39.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T10:33:39.813 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:40.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:39 smithi098 ceph-mon[97580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:33:40.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:39 smithi098 ceph-mon[97580]: osdmap e425: 8 total, 8 up, 8 in 2024-09-21T10:33:40.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:39 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]} : dispatch 2024-09-21T10:33:40.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:39 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-21T10:33:40.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:39 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]} : dispatch 2024-09-21T10:33:40.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:39 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-09-21T10:33:40.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:39 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-21T10:33:40.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:39 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2024-09-21T10:33:40.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:39 smithi098 ceph-mon[97580]: osdmap e426: 8 total, 8 up, 8 in 2024-09-21T10:33:40.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:39 smithi098 ceph-mon[100914]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:33:40.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:39 smithi098 ceph-mon[100914]: osdmap e425: 8 total, 8 up, 8 in 2024-09-21T10:33:40.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:39 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]} : dispatch 2024-09-21T10:33:40.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:39 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-21T10:33:40.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:39 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]} : dispatch 2024-09-21T10:33:40.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:39 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-09-21T10:33:40.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:39 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-21T10:33:40.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:39 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2024-09-21T10:33:40.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:39 smithi098 ceph-mon[100914]: osdmap e426: 8 total, 8 up, 8 in 2024-09-21T10:33:40.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:39 smithi196 ceph-mon[84784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:33:40.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:39 smithi196 ceph-mon[84784]: osdmap e425: 8 total, 8 up, 8 in 2024-09-21T10:33:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:39 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]} : dispatch 2024-09-21T10:33:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:39 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-21T10:33:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:39 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]} : dispatch 2024-09-21T10:33:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:39 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-09-21T10:33:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:39 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-21T10:33:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:39 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2024-09-21T10:33:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:39 smithi196 ceph-mon[84784]: osdmap e426: 8 total, 8 up, 8 in 2024-09-21T10:33:40.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:33:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:33:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:33:40.925 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:40 smithi098 ceph-mon[97580]: pgmap v1457: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:40.925 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:40 smithi098 ceph-mon[97580]: osdmap e427: 8 total, 8 up, 8 in 2024-09-21T10:33:40.926 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:33:40 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:33:40] "GET /metrics HTTP/1.1" 200 34157 "" "Prometheus/2.51.0" 2024-09-21T10:33:40.926 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:40 smithi098 ceph-mon[100914]: pgmap v1457: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:40.926 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:40 smithi098 ceph-mon[100914]: osdmap e427: 8 total, 8 up, 8 in 2024-09-21T10:33:41.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:40 smithi196 ceph-mon[84784]: pgmap v1457: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:41.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:40 smithi196 ceph-mon[84784]: osdmap e427: 8 total, 8 up, 8 in 2024-09-21T10:33:41.653 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:33:41.653 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":427,"fsid":"26a15512-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:45:33.813487+0000","modified":"2024-09-21T10:33:40.808892+0000","last_up_change":"2024-09-21T10:02:13.080936+0000","last_in_change":"2024-09-21T10:32:36.047316+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:48:34.781592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"407","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"311","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:51:03.408228+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":78,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:52:19.135018+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:52:19.606631+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"579f13de-565d-4801-b65e-f27d1cde297a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":82,"up_from":86,"up_thru":409,"down_at":83,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1869651717},{"type":"v1","addr":"172.21.15.98:6803","nonce":1869651717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1869651717},{"type":"v1","addr":"172.21.15.98:6805","nonce":1869651717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1869651717},{"type":"v1","addr":"172.21.15.98:6809","nonce":1869651717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1869651717},{"type":"v1","addr":"172.21.15.98:6807","nonce":1869651717}]},"public_addr":"172.21.15.98:6803/1869651717","cluster_addr":"172.21.15.98:6805/1869651717","heartbeat_back_addr":"172.21.15.98:6809/1869651717","heartbeat_front_addr":"172.21.15.98:6807/1869651717","state":["exists","up"]},{"osd":1,"uuid":"6d31ac4e-dc1a-4051-8266-ba821106242f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":87,"up_from":91,"up_thru":405,"down_at":88,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":4249036577},{"type":"v1","addr":"172.21.15.98:6811","nonce":4249036577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":4249036577},{"type":"v1","addr":"172.21.15.98:6813","nonce":4249036577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4249036577},{"type":"v1","addr":"172.21.15.98:6817","nonce":4249036577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":4249036577},{"type":"v1","addr":"172.21.15.98:6815","nonce":4249036577}]},"public_addr":"172.21.15.98:6811/4249036577","cluster_addr":"172.21.15.98:6813/4249036577","heartbeat_back_addr":"172.21.15.98:6817/4249036577","heartbeat_front_addr":"172.21.15.98:6815/4249036577","state":["exists","up"]},{"osd":2,"uuid":"62e22789-74be-4098-a9b9-62671fb4cac1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":92,"up_from":96,"up_thru":402,"down_at":93,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1274766038},{"type":"v1","addr":"172.21.15.98:6819","nonce":1274766038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1274766038},{"type":"v1","addr":"172.21.15.98:6821","nonce":1274766038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1274766038},{"type":"v1","addr":"172.21.15.98:6825","nonce":1274766038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1274766038},{"type":"v1","addr":"172.21.15.98:6823","nonce":1274766038}]},"public_addr":"172.21.15.98:6819/1274766038","cluster_addr":"172.21.15.98:6821/1274766038","heartbeat_back_addr":"172.21.15.98:6825/1274766038","heartbeat_front_addr":"172.21.15.98:6823/1274766038","state":["exists","up"]},{"osd":3,"uuid":"77f653a6-69b0-437a-960d-c3fb036708be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":97,"up_from":101,"up_thru":426,"down_at":98,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3950952422},{"type":"v1","addr":"172.21.15.98:6827","nonce":3950952422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3950952422},{"type":"v1","addr":"172.21.15.98:6829","nonce":3950952422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3950952422},{"type":"v1","addr":"172.21.15.98:6833","nonce":3950952422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3950952422},{"type":"v1","addr":"172.21.15.98:6831","nonce":3950952422}]},"public_addr":"172.21.15.98:6827/3950952422","cluster_addr":"172.21.15.98:6829/3950952422","heartbeat_back_addr":"172.21.15.98:6833/3950952422","heartbeat_front_addr":"172.21.15.98:6831/3950952422","state":["exists","up"]},{"osd":4,"uuid":"28ec4928-7f00-46a9-b8c5-95f0b7e6be75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":426,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6801","nonce":158468112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6803","nonce":158468112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6807","nonce":158468112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6805","nonce":158468112}]},"public_addr":"172.21.15.196:6801/158468112","cluster_addr":"172.21.15.196:6803/158468112","heartbeat_back_addr":"172.21.15.196:6807/158468112","heartbeat_front_addr":"172.21.15.196:6805/158468112","state":["exists","up"]},{"osd":5,"uuid":"cdcc44eb-e27f-4d23-9ecb-03bebdc82c96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":420,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6809","nonce":2341878784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6811","nonce":2341878784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6815","nonce":2341878784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6813","nonce":2341878784}]},"public_addr":"172.21.15.196:6809/2341878784","cluster_addr":"172.21.15.196:6811/2341878784","heartbeat_back_addr":"172.21.15.196:6815/2341878784","heartbeat_front_addr":"172.21.15.196:6813/2341878784","state":["exists","up"]},{"osd":6,"uuid":"566cfb44-1b21-45fc-b053-7205b1c10d4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":402,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6817","nonce":3059702155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6819","nonce":3059702155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6823","nonce":3059702155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6821","nonce":3059702155}]},"public_addr":"172.21.15.196:6817/3059702155","cluster_addr":"172.21.15.196:6819/3059702155","heartbeat_back_addr":"172.21.15.196:6823/3059702155","heartbeat_front_addr":"172.21.15.196:6821/3059702155","state":["exists","up"]},{"osd":7,"uuid":"228404c1-3910-4933-b023-55aa8b01aee4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":425,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6825","nonce":2155020234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6827","nonce":2155020234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6831","nonce":2155020234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6829","nonce":2155020234}]},"public_addr":"172.21.15.196:6825/2155020234","cluster_addr":"172.21.15.196:6827/2155020234","heartbeat_back_addr":"172.21.15.196:6831/2155020234","heartbeat_front_addr":"172.21.15.196:6829/2155020234","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:47:43.263977+0000","dead_epoch":82},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:06.738746+0000","dead_epoch":87},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:30.883578+0000","dead_epoch":92},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:54.216866+0000","dead_epoch":97},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:49:15.599276+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:49:38.581033+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:01.504502+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:25.964220+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.9","mappings":[{"from":4,"to":0}]},{"pgid":"3.16","mappings":[{"from":7,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/227493886":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/4208010999":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/493934303":"2024-09-22T09:56:36.894927+0000","172.21.15.196:0/2330152142":"2024-09-22T09:55:58.115206+0000","172.21.15.98:0/3466051265":"2024-09-22T09:54:48.070235+0000","172.21.15.98:0/900860885":"2024-09-22T09:54:48.070235+0000","172.21.15.98:0/675672768":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/4139436413":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/3656555554":"2024-09-22T09:45:54.237657+0000","172.21.15.98:6800/1387543382":"2024-09-22T09:54:48.070235+0000","172.21.15.196:0/79302526":"2024-09-22T09:55:58.115206+0000","172.21.15.196:0/1564192582":"2024-09-22T09:55:58.115206+0000","172.21.15.196:6835/2652695583":"2024-09-22T09:55:58.115206+0000","172.21.15.98:6801/3394444334":"2024-09-22T09:56:36.894927+0000","172.21.15.98:0/262018661":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/2075804383":"2024-09-22T09:45:54.237657+0000","172.21.15.98:0/2144473621":"2024-09-22T09:45:54.237657+0000","172.21.15.98:6801/440385038":"2024-09-22T09:46:24.467967+0000","172.21.15.98:6800/440385038":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/2546189353":"2024-09-22T09:56:36.894927+0000","172.21.15.98:0/2611845896":"2024-09-22T09:56:36.894927+0000","172.21.15.98:6801/1387543382":"2024-09-22T09:54:48.070235+0000","172.21.15.98:0/3628356893":"2024-09-22T09:56:36.894927+0000","172.21.15.98:6800/1427334238":"2024-09-22T09:45:54.237657+0000","172.21.15.196:0/1546057694":"2024-09-22T09:55:58.115206+0000","172.21.15.98:6801/89887073":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/881612124":"2024-09-22T09:50:59.815356+0000","172.21.15.98:6800/3394444334":"2024-09-22T09:56:36.894927+0000","172.21.15.98:6801/1427334238":"2024-09-22T09:45:54.237657+0000","172.21.15.98:6800/89887073":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/1093274465":"2024-09-22T09:54:48.070235+0000","172.21.15.98:0/1268404616":"2024-09-22T09:56:36.894927+0000","172.21.15.98:0/271851122":"2024-09-22T09:56:36.894927+0000","172.21.15.196:0/2663325110":"2024-09-22T09:55:58.115206+0000","172.21.15.98:0/1829140501":"2024-09-22T09:54:48.070235+0000","172.21.15.196:6834/2652695583":"2024-09-22T09:55:58.115206+0000","172.21.15.98:0/3737889208":"2024-09-22T09:46:24.467967+0000","172.21.15.196:0/1036032960":"2024-09-22T09:55:58.115206+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":11,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T10:33:42.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:41 smithi098 ceph-mon[97580]: 1.2 scrub starts 2024-09-21T10:33:42.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:41 smithi098 ceph-mon[97580]: 1.2 scrub ok 2024-09-21T10:33:42.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:41 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3850004709' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:33:42.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:41 smithi098 ceph-mon[100914]: 1.2 scrub starts 2024-09-21T10:33:42.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:41 smithi098 ceph-mon[100914]: 1.2 scrub ok 2024-09-21T10:33:42.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:41 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3850004709' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:33:42.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:41 smithi196 ceph-mon[84784]: 1.2 scrub starts 2024-09-21T10:33:42.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:41 smithi196 ceph-mon[84784]: 1.2 scrub ok 2024-09-21T10:33:42.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:41 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3850004709' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:33:42.399 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T10:33:42.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T10:33:42.644 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:43.063 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:42 smithi098 ceph-mon[97580]: pgmap v1460: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:43.063 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:42 smithi098 ceph-mon[100914]: pgmap v1460: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:43.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:42 smithi196 ceph-mon[84784]: pgmap v1460: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:44.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:43 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:33:44.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:43 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:33:44.249 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:43 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:33:44.656 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:33:44.656 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":427,"fsid":"26a15512-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:45:33.813487+0000","modified":"2024-09-21T10:33:40.808892+0000","last_up_change":"2024-09-21T10:02:13.080936+0000","last_in_change":"2024-09-21T10:32:36.047316+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:48:34.781592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"407","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"311","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:51:03.408228+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":78,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:52:19.135018+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:52:19.606631+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"579f13de-565d-4801-b65e-f27d1cde297a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":82,"up_from":86,"up_thru":409,"down_at":83,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1869651717},{"type":"v1","addr":"172.21.15.98:6803","nonce":1869651717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1869651717},{"type":"v1","addr":"172.21.15.98:6805","nonce":1869651717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1869651717},{"type":"v1","addr":"172.21.15.98:6809","nonce":1869651717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1869651717},{"type":"v1","addr":"172.21.15.98:6807","nonce":1869651717}]},"public_addr":"172.21.15.98:6803/1869651717","cluster_addr":"172.21.15.98:6805/1869651717","heartbeat_back_addr":"172.21.15.98:6809/1869651717","heartbeat_front_addr":"172.21.15.98:6807/1869651717","state":["exists","up"]},{"osd":1,"uuid":"6d31ac4e-dc1a-4051-8266-ba821106242f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":87,"up_from":91,"up_thru":405,"down_at":88,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":4249036577},{"type":"v1","addr":"172.21.15.98:6811","nonce":4249036577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":4249036577},{"type":"v1","addr":"172.21.15.98:6813","nonce":4249036577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4249036577},{"type":"v1","addr":"172.21.15.98:6817","nonce":4249036577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":4249036577},{"type":"v1","addr":"172.21.15.98:6815","nonce":4249036577}]},"public_addr":"172.21.15.98:6811/4249036577","cluster_addr":"172.21.15.98:6813/4249036577","heartbeat_back_addr":"172.21.15.98:6817/4249036577","heartbeat_front_addr":"172.21.15.98:6815/4249036577","state":["exists","up"]},{"osd":2,"uuid":"62e22789-74be-4098-a9b9-62671fb4cac1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":92,"up_from":96,"up_thru":402,"down_at":93,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1274766038},{"type":"v1","addr":"172.21.15.98:6819","nonce":1274766038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1274766038},{"type":"v1","addr":"172.21.15.98:6821","nonce":1274766038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1274766038},{"type":"v1","addr":"172.21.15.98:6825","nonce":1274766038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1274766038},{"type":"v1","addr":"172.21.15.98:6823","nonce":1274766038}]},"public_addr":"172.21.15.98:6819/1274766038","cluster_addr":"172.21.15.98:6821/1274766038","heartbeat_back_addr":"172.21.15.98:6825/1274766038","heartbeat_front_addr":"172.21.15.98:6823/1274766038","state":["exists","up"]},{"osd":3,"uuid":"77f653a6-69b0-437a-960d-c3fb036708be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":97,"up_from":101,"up_thru":426,"down_at":98,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3950952422},{"type":"v1","addr":"172.21.15.98:6827","nonce":3950952422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3950952422},{"type":"v1","addr":"172.21.15.98:6829","nonce":3950952422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3950952422},{"type":"v1","addr":"172.21.15.98:6833","nonce":3950952422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3950952422},{"type":"v1","addr":"172.21.15.98:6831","nonce":3950952422}]},"public_addr":"172.21.15.98:6827/3950952422","cluster_addr":"172.21.15.98:6829/3950952422","heartbeat_back_addr":"172.21.15.98:6833/3950952422","heartbeat_front_addr":"172.21.15.98:6831/3950952422","state":["exists","up"]},{"osd":4,"uuid":"28ec4928-7f00-46a9-b8c5-95f0b7e6be75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":426,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6801","nonce":158468112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6803","nonce":158468112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6807","nonce":158468112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":158468112},{"type":"v1","addr":"172.21.15.196:6805","nonce":158468112}]},"public_addr":"172.21.15.196:6801/158468112","cluster_addr":"172.21.15.196:6803/158468112","heartbeat_back_addr":"172.21.15.196:6807/158468112","heartbeat_front_addr":"172.21.15.196:6805/158468112","state":["exists","up"]},{"osd":5,"uuid":"cdcc44eb-e27f-4d23-9ecb-03bebdc82c96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":420,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6809","nonce":2341878784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6811","nonce":2341878784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6815","nonce":2341878784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2341878784},{"type":"v1","addr":"172.21.15.196:6813","nonce":2341878784}]},"public_addr":"172.21.15.196:6809/2341878784","cluster_addr":"172.21.15.196:6811/2341878784","heartbeat_back_addr":"172.21.15.196:6815/2341878784","heartbeat_front_addr":"172.21.15.196:6813/2341878784","state":["exists","up"]},{"osd":6,"uuid":"566cfb44-1b21-45fc-b053-7205b1c10d4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":402,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6817","nonce":3059702155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6819","nonce":3059702155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6823","nonce":3059702155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":3059702155},{"type":"v1","addr":"172.21.15.196:6821","nonce":3059702155}]},"public_addr":"172.21.15.196:6817/3059702155","cluster_addr":"172.21.15.196:6819/3059702155","heartbeat_back_addr":"172.21.15.196:6823/3059702155","heartbeat_front_addr":"172.21.15.196:6821/3059702155","state":["exists","up"]},{"osd":7,"uuid":"228404c1-3910-4933-b023-55aa8b01aee4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":425,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6825","nonce":2155020234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6827","nonce":2155020234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6831","nonce":2155020234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2155020234},{"type":"v1","addr":"172.21.15.196:6829","nonce":2155020234}]},"public_addr":"172.21.15.196:6825/2155020234","cluster_addr":"172.21.15.196:6827/2155020234","heartbeat_back_addr":"172.21.15.196:6831/2155020234","heartbeat_front_addr":"172.21.15.196:6829/2155020234","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:47:43.263977+0000","dead_epoch":82},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:06.738746+0000","dead_epoch":87},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:30.883578+0000","dead_epoch":92},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:48:54.216866+0000","dead_epoch":97},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:49:15.599276+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:49:38.581033+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:01.504502+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:50:25.964220+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.9","mappings":[{"from":4,"to":0}]},{"pgid":"3.16","mappings":[{"from":7,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/227493886":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/4208010999":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/493934303":"2024-09-22T09:56:36.894927+0000","172.21.15.196:0/2330152142":"2024-09-22T09:55:58.115206+0000","172.21.15.98:0/3466051265":"2024-09-22T09:54:48.070235+0000","172.21.15.98:0/900860885":"2024-09-22T09:54:48.070235+0000","172.21.15.98:0/675672768":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/4139436413":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/3656555554":"2024-09-22T09:45:54.237657+0000","172.21.15.98:6800/1387543382":"2024-09-22T09:54:48.070235+0000","172.21.15.196:0/79302526":"2024-09-22T09:55:58.115206+0000","172.21.15.196:0/1564192582":"2024-09-22T09:55:58.115206+0000","172.21.15.196:6835/2652695583":"2024-09-22T09:55:58.115206+0000","172.21.15.98:6801/3394444334":"2024-09-22T09:56:36.894927+0000","172.21.15.98:0/262018661":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/2075804383":"2024-09-22T09:45:54.237657+0000","172.21.15.98:0/2144473621":"2024-09-22T09:45:54.237657+0000","172.21.15.98:6801/440385038":"2024-09-22T09:46:24.467967+0000","172.21.15.98:6800/440385038":"2024-09-22T09:46:24.467967+0000","172.21.15.98:0/2546189353":"2024-09-22T09:56:36.894927+0000","172.21.15.98:0/2611845896":"2024-09-22T09:56:36.894927+0000","172.21.15.98:6801/1387543382":"2024-09-22T09:54:48.070235+0000","172.21.15.98:0/3628356893":"2024-09-22T09:56:36.894927+0000","172.21.15.98:6800/1427334238":"2024-09-22T09:45:54.237657+0000","172.21.15.196:0/1546057694":"2024-09-22T09:55:58.115206+0000","172.21.15.98:6801/89887073":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/881612124":"2024-09-22T09:50:59.815356+0000","172.21.15.98:6800/3394444334":"2024-09-22T09:56:36.894927+0000","172.21.15.98:6801/1427334238":"2024-09-22T09:45:54.237657+0000","172.21.15.98:6800/89887073":"2024-09-22T09:50:59.815356+0000","172.21.15.98:0/1093274465":"2024-09-22T09:54:48.070235+0000","172.21.15.98:0/1268404616":"2024-09-22T09:56:36.894927+0000","172.21.15.98:0/271851122":"2024-09-22T09:56:36.894927+0000","172.21.15.196:0/2663325110":"2024-09-22T09:55:58.115206+0000","172.21.15.98:0/1829140501":"2024-09-22T09:54:48.070235+0000","172.21.15.196:6834/2652695583":"2024-09-22T09:55:58.115206+0000","172.21.15.98:0/3737889208":"2024-09-22T09:46:24.467967+0000","172.21.15.196:0/1036032960":"2024-09-22T09:55:58.115206+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":11,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T10:33:45.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:44 smithi196 ceph-mon[84784]: pgmap v1461: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:44 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/493948138' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:33:45.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:44 smithi098 ceph-mon[97580]: pgmap v1461: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:45.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:44 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/493948138' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:33:45.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:44 smithi098 ceph-mon[100914]: pgmap v1461: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:45.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:44 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/493948138' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:33:45.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T10:33:45.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T10:33:45.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T10:33:45.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T10:33:45.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T10:33:45.397 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T10:33:45.397 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T10:33:45.398 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T10:33:46.089 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:46.090 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:46.090 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:46.102 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:46.105 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:46.106 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:46.106 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:47.158 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:46 smithi098 ceph-mon[97580]: pgmap v1462: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:47.159 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:46 smithi098 ceph-mon[100914]: pgmap v1462: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:47.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:46 smithi196 ceph-mon[84784]: pgmap v1462: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:48.728 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:48 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:33:48.729 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:48 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:33:48.729 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:48 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:33:48.729 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:48 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:33:48.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:48 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:33:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:48 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:33:49.316 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:49.773 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:49 smithi196 ceph-mon[84784]: pgmap v1463: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:49.793 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:49 smithi098 ceph-mon[100914]: pgmap v1463: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:49.794 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:49 smithi098 ceph-mon[97580]: pgmap v1463: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:50.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:33:50 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:33:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:33:51.004 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:33:50 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:33:50] "GET /metrics HTTP/1.1" 200 34157 "" "Prometheus/2.51.0" 2024-09-21T10:33:51.818 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:51 smithi098 ceph-mon[97580]: pgmap v1464: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:51.819 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:51 smithi098 ceph-mon[100914]: pgmap v1464: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:51.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:51 smithi196 ceph-mon[84784]: pgmap v1464: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:52.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:52 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:52.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:52 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:52.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:52 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:53.767 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:53 smithi098 ceph-mon[97580]: pgmap v1465: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:53.767 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:53 smithi098 ceph-mon[100914]: pgmap v1465: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:53.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:53 smithi196 ceph-mon[84784]: pgmap v1465: 83 pgs: 83 active+clean; 4.6 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:55.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:55 smithi098 ceph-mon[97580]: pgmap v1466: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:55.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:55 smithi098 ceph-mon[100914]: pgmap v1466: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:55.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:55 smithi196 ceph-mon[84784]: pgmap v1466: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:55.994 INFO:teuthology.orchestra.run.smithi098.stdout:158913790488 2024-09-21T10:33:55.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T10:33:56.265 INFO:teuthology.orchestra.run.smithi098.stdout:369367187866 2024-09-21T10:33:56.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T10:33:57.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:57 smithi098 ceph-mon[97580]: pgmap v1467: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:57.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:57 smithi098 ceph-mon[100914]: pgmap v1467: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:57.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:57 smithi196 ceph-mon[84784]: pgmap v1467: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:59.314 INFO:teuthology.orchestra.run.smithi098.stdout:390842024332 2024-09-21T10:33:59.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T10:33:59.563 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:33:59 smithi098 ceph-mon[97580]: pgmap v1468: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:59.563 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:33:59 smithi098 ceph-mon[100914]: pgmap v1468: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:33:59.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:33:59 smithi196 ceph-mon[84784]: pgmap v1468: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:00.638 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:34:00 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:34:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:34:00.811 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:34:00 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:34:00] "GET /metrics HTTP/1.1" 200 34162 "" "Prometheus/2.51.0" 2024-09-21T10:34:01.819 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:01 smithi098 ceph-mon[97580]: pgmap v1469: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:01.819 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:01 smithi098 ceph-mon[100914]: pgmap v1469: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:01.850 INFO:teuthology.orchestra.run.smithi098.stdout:184683594259 2024-09-21T10:34:01.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T10:34:01.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:01 smithi196 ceph-mon[84784]: pgmap v1469: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:03.540 INFO:teuthology.orchestra.run.smithi098.stdout:433791697281 2024-09-21T10:34:03.540 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T10:34:03.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:03 smithi098 ceph-mon[97580]: pgmap v1470: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:03.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:03 smithi098 ceph-mon[100914]: pgmap v1470: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:03 smithi196 ceph-mon[84784]: pgmap v1470: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:04.068 INFO:teuthology.orchestra.run.smithi098.stdout:210453398031 2024-09-21T10:34:04.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T10:34:04.679 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:34:04.720 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:04 smithi098 ceph-mon[100914]: pgmap v1471: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:04.720 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:04 smithi098 ceph-mon[97580]: pgmap v1471: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:04 smithi196 ceph-mon[84784]: pgmap v1471: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:05.420 INFO:teuthology.orchestra.run.smithi098.stdout:412316860808 2024-09-21T10:34:05.421 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T10:34:06.140 INFO:teuthology.orchestra.run.smithi098.stdout:133143986719 2024-09-21T10:34:06.140 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T10:34:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:34:07.064 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:06 smithi098 ceph-mon[97580]: pgmap v1472: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:07.064 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:06 smithi098 ceph-mon[100914]: pgmap v1472: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:07.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:06 smithi196 ceph-mon[84784]: pgmap v1472: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:07.263 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:34:07.295 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:34:07.943 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:34:07.944 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:34:08.046 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:34:08.046 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:34:08.046 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:07 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:08.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:34:08.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:34:08.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:07 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:08.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:34:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:34:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:07 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:08.851 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:34:09.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:08 smithi098 ceph-mon[97580]: pgmap v1473: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:09.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:34:09.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:34:09.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:08 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:34:09.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:08 smithi098 ceph-mon[100914]: pgmap v1473: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:09.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:34:09.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:34:09.098 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:08 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:34:09.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:08 smithi196 ceph-mon[84784]: pgmap v1473: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:34:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:34:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:08 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' 2024-09-21T10:34:10.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:34:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:34:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:34:10.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:10 smithi098 ceph-mon[97580]: pgmap v1474: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:10.971 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:34:10 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:34:10] "GET /metrics HTTP/1.1" 200 34157 "" "Prometheus/2.51.0" 2024-09-21T10:34:10.971 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:10 smithi098 ceph-mon[100914]: pgmap v1474: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:11.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:10 smithi196 ceph-mon[84784]: pgmap v1474: 83 pgs: 83 active+clean; 4.6 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:13.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:12 smithi098 ceph-mon[97580]: pgmap v1475: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:13.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:12 smithi098 ceph-mon[100914]: pgmap v1475: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:13.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:12 smithi196 ceph-mon[84784]: pgmap v1475: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:13.438 INFO:teuthology.orchestra.run.smithi098.stdout:433791697283 2024-09-21T10:34:14.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:13 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3949841191' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T10:34:14.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:13 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3949841191' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T10:34:14.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:13 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3949841191' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T10:34:14.620 INFO:tasks.cephadm.ceph_manager.ceph:need seq 433791697281 got 433791697283 for osd.3 2024-09-21T10:34:14.620 DEBUG:teuthology.parallel:result is None 2024-09-21T10:34:14.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:14 smithi098 ceph-mon[97580]: pgmap v1476: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:14.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:14 smithi098 ceph-mon[100914]: pgmap v1476: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:15.091 INFO:teuthology.orchestra.run.smithi098.stdout:133143986721 2024-09-21T10:34:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:14 smithi196 ceph-mon[84784]: pgmap v1476: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:15.747 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986719 got 133143986721 for osd.4 2024-09-21T10:34:15.747 DEBUG:teuthology.parallel:result is None 2024-09-21T10:34:16.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:15 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1258026927' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T10:34:16.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:15 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1258026927' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T10:34:16.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:15 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1258026927' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T10:34:16.919 INFO:teuthology.orchestra.run.smithi098.stdout:184683594262 2024-09-21T10:34:17.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:16 smithi098 ceph-mon[97580]: pgmap v1477: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:17.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:16 smithi098 ceph-mon[100914]: pgmap v1477: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:17.118 INFO:teuthology.orchestra.run.smithi098.stdout:369367187871 2024-09-21T10:34:17.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:16 smithi196 ceph-mon[84784]: pgmap v1477: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:17.483 INFO:teuthology.orchestra.run.smithi098.stdout:412316860810 2024-09-21T10:34:17.744 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:34:18.095 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:17 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/3423592340' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T10:34:18.095 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:17 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/432361654' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T10:34:18.095 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:17 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/217984923' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T10:34:18.095 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:17 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/3423592340' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T10:34:18.095 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:17 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/432361654' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T10:34:18.095 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:17 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/217984923' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T10:34:18.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:17 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/3423592340' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T10:34:18.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:17 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/432361654' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T10:34:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:17 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/217984923' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T10:34:18.929 INFO:tasks.cephadm.ceph_manager.ceph:need seq 412316860808 got 412316860810 for osd.2 2024-09-21T10:34:18.929 DEBUG:teuthology.parallel:result is None 2024-09-21T10:34:19.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:18 smithi098 ceph-mon[97580]: pgmap v1478: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:19.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:18 smithi098 ceph-mon[100914]: pgmap v1478: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:18 smithi196 ceph-mon[84784]: pgmap v1478: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:19.531 INFO:teuthology.orchestra.run.smithi098.stdout:158913790492 2024-09-21T10:34:20.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:19 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/368163467' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T10:34:20.204 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:19 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/368163467' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T10:34:20.204 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:19 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/368163467' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T10:34:20.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:34:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:34:20.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:34:20.987 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:20 smithi098 ceph-mon[97580]: pgmap v1479: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:20.987 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:34:20 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:34:20] "GET /metrics HTTP/1.1" 200 34157 "" "Prometheus/2.51.0" 2024-09-21T10:34:20.987 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:20 smithi098 ceph-mon[100914]: pgmap v1479: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:21.013 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790488 got 158913790492 for osd.5 2024-09-21T10:34:21.013 DEBUG:teuthology.parallel:result is None 2024-09-21T10:34:21.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:20 smithi196 ceph-mon[84784]: pgmap v1479: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:22.255 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683594259 got 184683594262 for osd.6 2024-09-21T10:34:22.255 DEBUG:teuthology.parallel:result is None 2024-09-21T10:34:22.606 INFO:teuthology.orchestra.run.smithi098.stdout:210453398035 2024-09-21T10:34:22.876 INFO:tasks.cephadm.ceph_manager.ceph:need seq 369367187866 got 369367187871 for osd.0 2024-09-21T10:34:22.876 DEBUG:teuthology.parallel:result is None 2024-09-21T10:34:23.106 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:22 smithi098 ceph-mon[97580]: pgmap v1480: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:23.106 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:22 smithi098 ceph-mon[97580]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:23.106 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:22 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/1920332610' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T10:34:23.106 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:22 smithi098 ceph-mon[100914]: pgmap v1480: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:23.106 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:22 smithi098 ceph-mon[100914]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:23.107 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:22 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/1920332610' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T10:34:23.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:22 smithi196 ceph-mon[84784]: pgmap v1480: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:22 smithi196 ceph-mon[84784]: from='mgr.26492 172.21.15.98:0/2486376001' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:22 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/1920332610' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T10:34:23.463 INFO:teuthology.orchestra.run.smithi098.stdout:390842024337 2024-09-21T10:34:24.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:23 smithi196 ceph-mon[84784]: from='client.? 172.21.15.98:0/2712445535' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T10:34:24.285 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:23 smithi098 ceph-mon[97580]: from='client.? 172.21.15.98:0/2712445535' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T10:34:24.285 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:23 smithi098 ceph-mon[100914]: from='client.? 172.21.15.98:0/2712445535' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T10:34:24.312 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453398031 got 210453398035 for osd.7 2024-09-21T10:34:24.312 DEBUG:teuthology.parallel:result is None 2024-09-21T10:34:24.972 INFO:tasks.cephadm.ceph_manager.ceph:need seq 390842024332 got 390842024337 for osd.1 2024-09-21T10:34:24.973 DEBUG:teuthology.parallel:result is None 2024-09-21T10:34:24.973 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T10:34:24.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:34:25.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:24 smithi196 ceph-mon[84784]: pgmap v1481: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:25.225 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:34:25.346 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:24 smithi098 ceph-mon[97580]: pgmap v1481: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:25.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:24 smithi098 ceph-mon[100914]: pgmap v1481: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:27.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:26 smithi098 ceph-mon[97580]: pgmap v1482: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:27.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:26 smithi098 ceph-mon[100914]: pgmap v1482: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:27.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:26 smithi196 ceph-mon[84784]: pgmap v1482: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:27.233 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:34:27.234 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:34:28.050 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1482,"stamp":"2024-09-21T10:34:25.691564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5983,"num_read_kb":13006,"num_write":907,"num_write_kb":12623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":10252,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2132,"ondisk_log_size":2132,"up":249,"acting":249,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":252,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7404740,"kb_used_data":484268,"kb_used_omap":35,"kb_used_meta":6920348,"kb_avail":742556476,"statfs":{"total":767960285184,"available":760377831424,"internally_reserved":0,"allocated":495890432,"data_stored":476572198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36353,"internal_metadata":7086436863},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002940"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":159,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814761+0000","last_change":"2024-09-21T10:32:37.077344+0000","last_active":"2024-09-21T10:33:40.814761+0000","last_peered":"2024-09-21T10:33:40.814761+0000","last_clean":"2024-09-21T10:33:40.814761+0000","last_became_active":"2024-09-21T10:32:37.077009+0000","last_became_peered":"2024-09-21T10:32:37.077009+0000","last_unstale":"2024-09-21T10:33:40.814761+0000","last_undegraded":"2024-09-21T10:33:40.814761+0000","last_fullsized":"2024-09-21T10:33:40.814761+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:45.790900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":457,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.813606+0000","last_change":"2024-09-21T10:24:44.305198+0000","last_active":"2024-09-21T10:33:39.813606+0000","last_peered":"2024-09-21T10:33:39.813606+0000","last_clean":"2024-09-21T10:33:39.813606+0000","last_became_active":"2024-09-21T10:24:44.297653+0000","last_became_peered":"2024-09-21T10:24:44.297653+0000","last_unstale":"2024-09-21T10:33:39.813606+0000","last_undegraded":"2024-09-21T10:33:39.813606+0000","last_fullsized":"2024-09-21T10:33:39.813606+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:19.320833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":293,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822909+0000","last_change":"2024-09-21T10:28:00.294917+0000","last_active":"2024-09-21T10:33:39.822909+0000","last_peered":"2024-09-21T10:33:39.822909+0000","last_clean":"2024-09-21T10:33:39.822909+0000","last_became_active":"2024-09-21T10:28:00.294134+0000","last_became_peered":"2024-09-21T10:28:00.294134+0000","last_unstale":"2024-09-21T10:33:39.822909+0000","last_undegraded":"2024-09-21T10:33:39.822909+0000","last_fullsized":"2024-09-21T10:33:39.822909+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:29:05.780212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":453,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.424250+0000","last_change":"2024-09-21T10:32:54.793377+0000","last_active":"2024-09-21T10:33:41.424250+0000","last_peered":"2024-09-21T10:33:41.424250+0000","last_clean":"2024-09-21T10:33:41.424250+0000","last_became_active":"2024-09-21T10:32:54.793082+0000","last_became_peered":"2024-09-21T10:32:54.793082+0000","last_unstale":"2024-09-21T10:33:41.424250+0000","last_undegraded":"2024-09-21T10:33:41.424250+0000","last_fullsized":"2024-09-21T10:33:41.424250+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:20:50.891853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":742,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.424657+0000","last_change":"2024-09-21T10:28:00.297783+0000","last_active":"2024-09-21T10:33:41.424657+0000","last_peered":"2024-09-21T10:33:41.424657+0000","last_clean":"2024-09-21T10:33:41.424657+0000","last_became_active":"2024-09-21T10:28:00.297024+0000","last_became_peered":"2024-09-21T10:28:00.297024+0000","last_unstale":"2024-09-21T10:33:41.424657+0000","last_undegraded":"2024-09-21T10:33:41.424657+0000","last_fullsized":"2024-09-21T10:33:41.424657+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:24.382523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.823600+0000","last_change":"2024-09-21T10:33:39.823280+0000","last_active":"2024-09-21T10:33:39.823600+0000","last_peered":"2024-09-21T10:33:39.823600+0000","last_clean":"2024-09-21T10:33:39.823600+0000","last_became_active":"2024-09-21T10:33:39.822965+0000","last_became_peered":"2024-09-21T10:33:39.822965+0000","last_unstale":"2024-09-21T10:33:39.823600+0000","last_undegraded":"2024-09-21T10:33:39.823600+0000","last_fullsized":"2024-09-21T10:33:39.823600+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":426,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:39:40.734884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":166,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.816073+0000","last_change":"2024-09-21T10:33:40.816012+0000","last_active":"2024-09-21T10:33:40.816073+0000","last_peered":"2024-09-21T10:33:40.816073+0000","last_clean":"2024-09-21T10:33:40.816073+0000","last_became_active":"2024-09-21T10:33:40.815701+0000","last_became_peered":"2024-09-21T10:33:40.815701+0000","last_unstale":"2024-09-21T10:33:40.816073+0000","last_undegraded":"2024-09-21T10:33:40.816073+0000","last_fullsized":"2024-09-21T10:33:40.816073+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:51:26.944089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":399,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822240+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:33:39.822240+0000","last_peered":"2024-09-21T10:33:39.822240+0000","last_clean":"2024-09-21T10:33:39.822240+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:33:39.822240+0000","last_undegraded":"2024-09-21T10:33:39.822240+0000","last_fullsized":"2024-09-21T10:33:39.822240+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":448,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.814794+0000","last_change":"2024-09-21T10:24:44.325196+0000","last_active":"2024-09-21T10:33:39.814794+0000","last_peered":"2024-09-21T10:33:39.814794+0000","last_clean":"2024-09-21T10:33:39.814794+0000","last_became_active":"2024-09-21T10:24:44.299614+0000","last_became_peered":"2024-09-21T10:24:44.299614+0000","last_unstale":"2024-09-21T10:33:39.814794+0000","last_undegraded":"2024-09-21T10:33:39.814794+0000","last_fullsized":"2024-09-21T10:33:39.814794+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:14.558663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":179,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822773+0000","last_change":"2024-09-21T10:26:41.305571+0000","last_active":"2024-09-21T10:33:39.822773+0000","last_peered":"2024-09-21T10:33:39.822773+0000","last_clean":"2024-09-21T10:33:39.822773+0000","last_became_active":"2024-09-21T10:26:41.305135+0000","last_became_peered":"2024-09-21T10:26:41.305135+0000","last_unstale":"2024-09-21T10:33:39.822773+0000","last_undegraded":"2024-09-21T10:33:39.822773+0000","last_fullsized":"2024-09-21T10:33:39.822773+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:49:55.452467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":477,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.813917+0000","last_change":"2024-09-21T10:32:37.086045+0000","last_active":"2024-09-21T10:33:39.813917+0000","last_peered":"2024-09-21T10:33:39.813917+0000","last_clean":"2024-09-21T10:33:39.813917+0000","last_became_active":"2024-09-21T10:32:37.075889+0000","last_became_peered":"2024-09-21T10:32:37.075889+0000","last_unstale":"2024-09-21T10:33:39.813917+0000","last_undegraded":"2024-09-21T10:33:39.813917+0000","last_fullsized":"2024-09-21T10:33:39.813917+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:58:14.277307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":156,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.855976+0000","last_change":"2024-09-21T10:33:40.855976+0000","last_active":"2024-09-21T10:33:40.855976+0000","last_peered":"2024-09-21T10:33:40.855976+0000","last_clean":"2024-09-21T10:33:40.855976+0000","last_became_active":"2024-09-21T10:33:40.855672+0000","last_became_peered":"2024-09-21T10:33:40.855672+0000","last_unstale":"2024-09-21T10:33:40.855976+0000","last_undegraded":"2024-09-21T10:33:40.855976+0000","last_fullsized":"2024-09-21T10:33:40.855976+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:16.228416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":375,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.814330+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:33:39.814330+0000","last_peered":"2024-09-21T10:33:39.814330+0000","last_clean":"2024-09-21T10:33:39.814330+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:33:39.814330+0000","last_undegraded":"2024-09-21T10:33:39.814330+0000","last_fullsized":"2024-09-21T10:33:39.814330+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":410,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822478+0000","last_change":"2024-09-21T10:32:37.073962+0000","last_active":"2024-09-21T10:33:39.822478+0000","last_peered":"2024-09-21T10:33:39.822478+0000","last_clean":"2024-09-21T10:33:39.822478+0000","last_became_active":"2024-09-21T10:32:37.073651+0000","last_became_peered":"2024-09-21T10:32:37.073651+0000","last_unstale":"2024-09-21T10:33:39.822478+0000","last_undegraded":"2024-09-21T10:33:39.822478+0000","last_fullsized":"2024-09-21T10:33:39.822478+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:07:15.433004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":128,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822669+0000","last_change":"2024-09-21T10:33:26.769245+0000","last_active":"2024-09-21T10:33:39.822669+0000","last_peered":"2024-09-21T10:33:39.822669+0000","last_clean":"2024-09-21T10:33:39.822669+0000","last_became_active":"2024-09-21T10:33:26.768656+0000","last_became_peered":"2024-09-21T10:33:26.768656+0000","last_unstale":"2024-09-21T10:33:39.822669+0000","last_undegraded":"2024-09-21T10:33:39.822669+0000","last_fullsized":"2024-09-21T10:33:39.822669+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:54:32.081451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":778,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.423999+0000","last_change":"2024-09-21T10:32:37.084484+0000","last_active":"2024-09-21T10:33:41.423999+0000","last_peered":"2024-09-21T10:33:41.423999+0000","last_clean":"2024-09-21T10:33:41.423999+0000","last_became_active":"2024-09-21T10:32:37.084156+0000","last_became_peered":"2024-09-21T10:32:37.084156+0000","last_unstale":"2024-09-21T10:33:41.423999+0000","last_undegraded":"2024-09-21T10:33:41.423999+0000","last_fullsized":"2024-09-21T10:33:41.423999+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:38:59.721832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"82'100","reported_seq":548,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.812748+0000","last_change":"2024-09-21T10:28:00.285919+0000","last_active":"2024-09-21T10:33:39.812748+0000","last_peered":"2024-09-21T10:33:39.812748+0000","last_clean":"2024-09-21T10:33:39.812748+0000","last_became_active":"2024-09-21T10:28:00.285324+0000","last_became_peered":"2024-09-21T10:28:00.285324+0000","last_unstale":"2024-09-21T10:33:39.812748+0000","last_undegraded":"2024-09-21T10:33:39.812748+0000","last_fullsized":"2024-09-21T10:33:39.812748+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.728195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.728195+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:00.841610+0000","scrub_duration":0.0069698140000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":149,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814247+0000","last_change":"2024-09-21T10:28:00.303125+0000","last_active":"2024-09-21T10:33:40.814247+0000","last_peered":"2024-09-21T10:33:40.814247+0000","last_clean":"2024-09-21T10:33:40.814247+0000","last_became_active":"2024-09-21T10:28:00.293684+0000","last_became_peered":"2024-09-21T10:28:00.293684+0000","last_unstale":"2024-09-21T10:33:40.814247+0000","last_undegraded":"2024-09-21T10:33:40.814247+0000","last_fullsized":"2024-09-21T10:33:40.814247+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:04.495913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":126,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.814164+0000","last_change":"2024-09-21T10:28:00.283456+0000","last_active":"2024-09-21T10:33:39.814164+0000","last_peered":"2024-09-21T10:33:39.814164+0000","last_clean":"2024-09-21T10:33:39.814164+0000","last_became_active":"2024-09-21T10:28:00.282881+0000","last_became_peered":"2024-09-21T10:28:00.282881+0000","last_unstale":"2024-09-21T10:33:39.814164+0000","last_undegraded":"2024-09-21T10:33:39.814164+0000","last_fullsized":"2024-09-21T10:33:39.814164+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:43.681481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"82'100","reported_seq":29,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822679+0000","last_change":"2024-09-21T10:33:29.952265+0000","last_active":"2024-09-21T10:33:39.822679+0000","last_peered":"2024-09-21T10:33:39.822679+0000","last_clean":"2024-09-21T10:33:39.822679+0000","last_became_active":"2024-09-21T10:33:26.770295+0000","last_became_peered":"2024-09-21T10:33:26.770295+0000","last_unstale":"2024-09-21T10:33:39.822679+0000","last_undegraded":"2024-09-21T10:33:39.822679+0000","last_fullsized":"2024-09-21T10:33:39.822679+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:33:29.952194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:33:29.952194+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:01:43.700109+0000","scrub_duration":0.0024670159999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":461,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822533+0000","last_change":"2024-09-21T10:32:37.073933+0000","last_active":"2024-09-21T10:33:39.822533+0000","last_peered":"2024-09-21T10:33:39.822533+0000","last_clean":"2024-09-21T10:33:39.822533+0000","last_became_active":"2024-09-21T10:32:37.073604+0000","last_became_peered":"2024-09-21T10:32:37.073604+0000","last_unstale":"2024-09-21T10:33:39.822533+0000","last_undegraded":"2024-09-21T10:33:39.822533+0000","last_fullsized":"2024-09-21T10:33:39.822533+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:43:20.583916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":136,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.814462+0000","last_change":"2024-09-21T10:32:37.075432+0000","last_active":"2024-09-21T10:33:39.814462+0000","last_peered":"2024-09-21T10:33:39.814462+0000","last_clean":"2024-09-21T10:33:39.814462+0000","last_became_active":"2024-09-21T10:32:37.075087+0000","last_became_peered":"2024-09-21T10:32:37.075087+0000","last_unstale":"2024-09-21T10:33:39.814462+0000","last_undegraded":"2024-09-21T10:33:39.814462+0000","last_fullsized":"2024-09-21T10:33:39.814462+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:06:43.955465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":196,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.424371+0000","last_change":"2024-09-21T10:25:39.872680+0000","last_active":"2024-09-21T10:33:41.424371+0000","last_peered":"2024-09-21T10:33:41.424371+0000","last_clean":"2024-09-21T10:33:41.424371+0000","last_became_active":"2024-09-21T10:25:39.872312+0000","last_became_peered":"2024-09-21T10:25:39.872312+0000","last_unstale":"2024-09-21T10:33:41.424371+0000","last_undegraded":"2024-09-21T10:33:41.424371+0000","last_fullsized":"2024-09-21T10:33:41.424371+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:54.942046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":136,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.814525+0000","last_change":"2024-09-21T10:32:37.077131+0000","last_active":"2024-09-21T10:33:39.814525+0000","last_peered":"2024-09-21T10:33:39.814525+0000","last_clean":"2024-09-21T10:33:39.814525+0000","last_became_active":"2024-09-21T10:32:37.076783+0000","last_became_peered":"2024-09-21T10:32:37.076783+0000","last_unstale":"2024-09-21T10:33:39.814525+0000","last_undegraded":"2024-09-21T10:33:39.814525+0000","last_fullsized":"2024-09-21T10:33:39.814525+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:05:20.632915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'100","reported_seq":29,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.823020+0000","last_change":"2024-09-21T10:33:26.873794+0000","last_active":"2024-09-21T10:33:39.823020+0000","last_peered":"2024-09-21T10:33:39.823020+0000","last_clean":"2024-09-21T10:33:39.823020+0000","last_became_active":"2024-09-21T10:33:26.769799+0000","last_became_peered":"2024-09-21T10:33:26.769799+0000","last_unstale":"2024-09-21T10:33:39.823020+0000","last_undegraded":"2024-09-21T10:33:39.823020+0000","last_fullsized":"2024-09-21T10:33:39.823020+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:33:26.873733+0000","last_deep_scrub":"82'100","last_deep_scrub_stamp":"2024-09-21T10:33:26.873733+0000","last_clean_scrub_stamp":"2024-09-21T10:33:26.873733+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:22.217151+0000","scrub_duration":0.0016179689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":159,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814342+0000","last_change":"2024-09-21T10:32:37.078156+0000","last_active":"2024-09-21T10:33:40.814342+0000","last_peered":"2024-09-21T10:33:40.814342+0000","last_clean":"2024-09-21T10:33:40.814342+0000","last_became_active":"2024-09-21T10:32:37.077192+0000","last_became_peered":"2024-09-21T10:32:37.077192+0000","last_unstale":"2024-09-21T10:33:40.814342+0000","last_undegraded":"2024-09-21T10:33:40.814342+0000","last_fullsized":"2024-09-21T10:33:40.814342+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:21.833793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":59,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.018999+0000","last_change":"2024-09-21T10:32:37.083366+0000","last_active":"2024-09-21T10:33:41.018999+0000","last_peered":"2024-09-21T10:33:41.018999+0000","last_clean":"2024-09-21T10:33:41.018999+0000","last_became_active":"2024-09-21T10:32:37.082694+0000","last_became_peered":"2024-09-21T10:32:37.082694+0000","last_unstale":"2024-09-21T10:33:41.018999+0000","last_undegraded":"2024-09-21T10:33:41.018999+0000","last_fullsized":"2024-09-21T10:33:41.018999+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:40.298562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":529,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.812832+0000","last_change":"2024-09-21T10:23:18.790100+0000","last_active":"2024-09-21T10:33:39.812832+0000","last_peered":"2024-09-21T10:33:39.812832+0000","last_clean":"2024-09-21T10:33:39.812832+0000","last_became_active":"2024-09-21T10:23:14.677610+0000","last_became_peered":"2024-09-21T10:23:14.677610+0000","last_unstale":"2024-09-21T10:33:39.812832+0000","last_undegraded":"2024-09-21T10:33:39.812832+0000","last_fullsized":"2024-09-21T10:33:39.812832+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:18.790017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:18.790017+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:56.562241+0000","scrub_duration":0.0080400790000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":621,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.821691+0000","last_change":"2024-09-21T10:26:41.279374+0000","last_active":"2024-09-21T10:33:39.821691+0000","last_peered":"2024-09-21T10:33:39.821691+0000","last_clean":"2024-09-21T10:33:39.821691+0000","last_became_active":"2024-09-21T10:26:41.278994+0000","last_became_peered":"2024-09-21T10:26:41.278994+0000","last_unstale":"2024-09-21T10:33:39.821691+0000","last_undegraded":"2024-09-21T10:33:39.821691+0000","last_fullsized":"2024-09-21T10:33:39.821691+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:49:57.821175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":482,"num_read_kb":1581,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":146,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814226+0000","last_change":"2024-09-21T10:28:00.295436+0000","last_active":"2024-09-21T10:33:40.814226+0000","last_peered":"2024-09-21T10:33:40.814226+0000","last_clean":"2024-09-21T10:33:40.814226+0000","last_became_active":"2024-09-21T10:28:00.295135+0000","last_became_peered":"2024-09-21T10:28:00.295135+0000","last_unstale":"2024-09-21T10:33:40.814226+0000","last_undegraded":"2024-09-21T10:33:40.814226+0000","last_fullsized":"2024-09-21T10:33:40.814226+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:14:03.027395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":413,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822016+0000","last_change":"2024-09-21T10:24:44.308008+0000","last_active":"2024-09-21T10:33:39.822016+0000","last_peered":"2024-09-21T10:33:39.822016+0000","last_clean":"2024-09-21T10:33:39.822016+0000","last_became_active":"2024-09-21T10:24:44.307595+0000","last_became_peered":"2024-09-21T10:24:44.307595+0000","last_unstale":"2024-09-21T10:33:39.822016+0000","last_undegraded":"2024-09-21T10:33:39.822016+0000","last_fullsized":"2024-09-21T10:33:39.822016+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:33:49.448676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"82'100","reported_seq":60,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.018623+0000","last_change":"2024-09-21T10:32:37.081585+0000","last_active":"2024-09-21T10:33:41.018623+0000","last_peered":"2024-09-21T10:33:41.018623+0000","last_clean":"2024-09-21T10:33:41.018623+0000","last_became_active":"2024-09-21T10:32:37.080879+0000","last_became_peered":"2024-09-21T10:32:37.080879+0000","last_unstale":"2024-09-21T10:33:41.018623+0000","last_undegraded":"2024-09-21T10:33:41.018623+0000","last_fullsized":"2024-09-21T10:33:41.018623+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:32:26.379765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:32:26.379765+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:45.658670+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":863,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.812748+0000","last_change":"2024-09-21T10:32:40.086328+0000","last_active":"2024-09-21T10:33:39.812748+0000","last_peered":"2024-09-21T10:33:39.812748+0000","last_clean":"2024-09-21T10:33:39.812748+0000","last_became_active":"2024-09-21T10:32:40.085287+0000","last_became_peered":"2024-09-21T10:32:40.085287+0000","last_unstale":"2024-09-21T10:33:39.812748+0000","last_undegraded":"2024-09-21T10:33:39.812748+0000","last_fullsized":"2024-09-21T10:33:39.812748+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:15.146359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":772,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.424432+0000","last_change":"2024-09-21T10:28:00.296831+0000","last_active":"2024-09-21T10:33:41.424432+0000","last_peered":"2024-09-21T10:33:41.424432+0000","last_clean":"2024-09-21T10:33:41.424432+0000","last_became_active":"2024-09-21T10:28:00.296527+0000","last_became_peered":"2024-09-21T10:28:00.296527+0000","last_unstale":"2024-09-21T10:33:41.424432+0000","last_undegraded":"2024-09-21T10:33:41.424432+0000","last_fullsized":"2024-09-21T10:33:41.424432+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:18.024868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":754,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.424009+0000","last_change":"2024-09-21T10:32:37.080391+0000","last_active":"2024-09-21T10:33:41.424009+0000","last_peered":"2024-09-21T10:33:41.424009+0000","last_clean":"2024-09-21T10:33:41.424009+0000","last_became_active":"2024-09-21T10:32:37.080065+0000","last_became_peered":"2024-09-21T10:32:37.080065+0000","last_unstale":"2024-09-21T10:33:41.424009+0000","last_undegraded":"2024-09-21T10:33:41.424009+0000","last_fullsized":"2024-09-21T10:33:41.424009+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:12:42.262043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"82'100","reported_seq":426,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.572568+0000","last_change":"2024-09-21T10:33:40.572529+0000","last_active":"2024-09-21T10:33:40.572568+0000","last_peered":"2024-09-21T10:33:40.572568+0000","last_clean":"2024-09-21T10:33:40.572568+0000","last_became_active":"2024-09-21T10:33:39.818529+0000","last_became_peered":"2024-09-21T10:33:39.818529+0000","last_unstale":"2024-09-21T10:33:40.572568+0000","last_undegraded":"2024-09-21T10:33:40.572568+0000","last_fullsized":"2024-09-21T10:33:40.572568+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":426,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:33:40.572474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:33:40.572474+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:20.210668+0000","scrub_duration":0.0014184460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":1549,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.855130+0000","last_change":"2024-09-21T10:28:03.346064+0000","last_active":"2024-09-21T10:33:40.855130+0000","last_peered":"2024-09-21T10:33:40.855130+0000","last_clean":"2024-09-21T10:33:40.855130+0000","last_became_active":"2024-09-21T10:28:03.345313+0000","last_became_peered":"2024-09-21T10:28:03.345313+0000","last_unstale":"2024-09-21T10:33:40.855130+0000","last_undegraded":"2024-09-21T10:33:40.855130+0000","last_fullsized":"2024-09-21T10:33:40.855130+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:32.097521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":59,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.018514+0000","last_change":"2024-09-21T10:32:37.076203+0000","last_active":"2024-09-21T10:33:41.018514+0000","last_peered":"2024-09-21T10:33:41.018514+0000","last_clean":"2024-09-21T10:33:41.018514+0000","last_became_active":"2024-09-21T10:32:37.075328+0000","last_became_peered":"2024-09-21T10:32:37.075328+0000","last_unstale":"2024-09-21T10:33:41.018514+0000","last_undegraded":"2024-09-21T10:33:41.018514+0000","last_fullsized":"2024-09-21T10:33:41.018514+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:44:29.546239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":59,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.018943+0000","last_change":"2024-09-21T10:32:37.076492+0000","last_active":"2024-09-21T10:33:41.018943+0000","last_peered":"2024-09-21T10:33:41.018943+0000","last_clean":"2024-09-21T10:33:41.018943+0000","last_became_active":"2024-09-21T10:32:37.075758+0000","last_became_peered":"2024-09-21T10:32:37.075758+0000","last_unstale":"2024-09-21T10:33:41.018943+0000","last_undegraded":"2024-09-21T10:33:41.018943+0000","last_fullsized":"2024-09-21T10:33:41.018943+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:46:11.681207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"82'100","reported_seq":175,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822549+0000","last_change":"2024-09-21T10:33:28.904517+0000","last_active":"2024-09-21T10:33:39.822549+0000","last_peered":"2024-09-21T10:33:39.822549+0000","last_clean":"2024-09-21T10:33:39.822549+0000","last_became_active":"2024-09-21T10:33:26.769976+0000","last_became_peered":"2024-09-21T10:33:26.769976+0000","last_unstale":"2024-09-21T10:33:39.822549+0000","last_undegraded":"2024-09-21T10:33:39.822549+0000","last_fullsized":"2024-09-21T10:33:39.822549+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:33:28.904449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:33:28.904449+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:51.562677+0000","scrub_duration":0.001795643,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":736,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.823140+0000","last_change":"2024-09-21T10:33:26.770227+0000","last_active":"2024-09-21T10:33:39.823140+0000","last_peered":"2024-09-21T10:33:39.823140+0000","last_clean":"2024-09-21T10:33:39.823140+0000","last_became_active":"2024-09-21T10:33:26.769475+0000","last_became_peered":"2024-09-21T10:33:26.769475+0000","last_unstale":"2024-09-21T10:33:39.823140+0000","last_undegraded":"2024-09-21T10:33:39.823140+0000","last_fullsized":"2024-09-21T10:33:39.823140+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:00:40.325730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":116,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.813699+0000","last_change":"2024-09-21T10:24:44.299492+0000","last_active":"2024-09-21T10:33:39.813699+0000","last_peered":"2024-09-21T10:33:39.813699+0000","last_clean":"2024-09-21T10:33:39.813699+0000","last_became_active":"2024-09-21T10:24:44.299187+0000","last_became_peered":"2024-09-21T10:24:44.299187+0000","last_unstale":"2024-09-21T10:33:39.813699+0000","last_undegraded":"2024-09-21T10:33:39.813699+0000","last_fullsized":"2024-09-21T10:33:39.813699+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:58.500386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":59,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.018932+0000","last_change":"2024-09-21T10:32:37.083724+0000","last_active":"2024-09-21T10:33:41.018932+0000","last_peered":"2024-09-21T10:33:41.018932+0000","last_clean":"2024-09-21T10:33:41.018932+0000","last_became_active":"2024-09-21T10:32:37.082902+0000","last_became_peered":"2024-09-21T10:32:37.082902+0000","last_unstale":"2024-09-21T10:33:41.018932+0000","last_undegraded":"2024-09-21T10:33:41.018932+0000","last_fullsized":"2024-09-21T10:33:41.018932+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:20:49.762214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"82'100","reported_seq":291,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822333+0000","last_change":"2024-09-21T10:28:00.294997+0000","last_active":"2024-09-21T10:33:39.822333+0000","last_peered":"2024-09-21T10:33:39.822333+0000","last_clean":"2024-09-21T10:33:39.822333+0000","last_became_active":"2024-09-21T10:28:00.294595+0000","last_became_peered":"2024-09-21T10:28:00.294595+0000","last_unstale":"2024-09-21T10:33:39.822333+0000","last_undegraded":"2024-09-21T10:33:39.822333+0000","last_fullsized":"2024-09-21T10:33:39.822333+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:30.569982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:30.569982+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:01.422939+0000","scrub_duration":0.0051309349999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":224,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822911+0000","last_change":"2024-09-21T10:28:00.294966+0000","last_active":"2024-09-21T10:33:39.822911+0000","last_peered":"2024-09-21T10:33:39.822911+0000","last_clean":"2024-09-21T10:33:39.822911+0000","last_became_active":"2024-09-21T10:28:00.294552+0000","last_became_peered":"2024-09-21T10:28:00.294552+0000","last_unstale":"2024-09-21T10:33:39.822911+0000","last_undegraded":"2024-09-21T10:33:39.822911+0000","last_fullsized":"2024-09-21T10:33:39.822911+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:47.292125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":886,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.018870+0000","last_change":"2024-09-21T10:32:40.086133+0000","last_active":"2024-09-21T10:33:41.018870+0000","last_peered":"2024-09-21T10:33:41.018870+0000","last_clean":"2024-09-21T10:33:41.018870+0000","last_became_active":"2024-09-21T10:32:40.085561+0000","last_became_peered":"2024-09-21T10:32:40.085561+0000","last_unstale":"2024-09-21T10:33:41.018870+0000","last_undegraded":"2024-09-21T10:33:41.018870+0000","last_fullsized":"2024-09-21T10:33:41.018870+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:54.245378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":59,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.018668+0000","last_change":"2024-09-21T10:32:37.081781+0000","last_active":"2024-09-21T10:33:41.018668+0000","last_peered":"2024-09-21T10:33:41.018668+0000","last_clean":"2024-09-21T10:33:41.018668+0000","last_became_active":"2024-09-21T10:32:37.081499+0000","last_became_peered":"2024-09-21T10:32:37.081499+0000","last_unstale":"2024-09-21T10:33:41.018668+0000","last_undegraded":"2024-09-21T10:33:41.018668+0000","last_fullsized":"2024-09-21T10:33:41.018668+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:23.388943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'100","reported_seq":301,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.821561+0000","last_change":"2024-09-21T10:28:00.295277+0000","last_active":"2024-09-21T10:33:39.821561+0000","last_peered":"2024-09-21T10:33:39.821561+0000","last_clean":"2024-09-21T10:33:39.821561+0000","last_became_active":"2024-09-21T10:28:00.294016+0000","last_became_peered":"2024-09-21T10:28:00.294016+0000","last_unstale":"2024-09-21T10:33:39.821561+0000","last_undegraded":"2024-09-21T10:33:39.821561+0000","last_fullsized":"2024-09-21T10:33:39.821561+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:24.536711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:24.536711+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:28:49.984470+0000","scrub_duration":0.0047525650000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":59,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.018835+0000","last_change":"2024-09-21T10:32:37.077779+0000","last_active":"2024-09-21T10:33:41.018835+0000","last_peered":"2024-09-21T10:33:41.018835+0000","last_clean":"2024-09-21T10:33:41.018835+0000","last_became_active":"2024-09-21T10:32:37.076786+0000","last_became_peered":"2024-09-21T10:32:37.076786+0000","last_unstale":"2024-09-21T10:33:41.018835+0000","last_undegraded":"2024-09-21T10:33:41.018835+0000","last_fullsized":"2024-09-21T10:33:41.018835+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:53:45.261483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":849,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.814414+0000","last_change":"2024-09-21T10:24:44.334847+0000","last_active":"2024-09-21T10:33:39.814414+0000","last_peered":"2024-09-21T10:33:39.814414+0000","last_clean":"2024-09-21T10:33:39.814414+0000","last_became_active":"2024-09-21T10:24:44.301321+0000","last_became_peered":"2024-09-21T10:24:44.301321+0000","last_unstale":"2024-09-21T10:33:39.814414+0000","last_undegraded":"2024-09-21T10:33:39.814414+0000","last_fullsized":"2024-09-21T10:33:39.814414+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:37.628940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":607,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.424527+0000","last_change":"2024-09-21T10:32:54.862663+0000","last_active":"2024-09-21T10:33:41.424527+0000","last_peered":"2024-09-21T10:33:41.424527+0000","last_clean":"2024-09-21T10:33:41.424527+0000","last_became_active":"2024-09-21T10:32:54.862435+0000","last_became_peered":"2024-09-21T10:32:54.862435+0000","last_unstale":"2024-09-21T10:33:41.424527+0000","last_undegraded":"2024-09-21T10:33:41.424527+0000","last_fullsized":"2024-09-21T10:33:41.424527+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:30.412498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"82'100","reported_seq":116,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.814388+0000","last_change":"2024-09-21T10:24:44.296758+0000","last_active":"2024-09-21T10:33:39.814388+0000","last_peered":"2024-09-21T10:33:39.814388+0000","last_clean":"2024-09-21T10:33:39.814388+0000","last_became_active":"2024-09-21T10:24:44.296403+0000","last_became_peered":"2024-09-21T10:24:44.296403+0000","last_unstale":"2024-09-21T10:33:39.814388+0000","last_undegraded":"2024-09-21T10:33:39.814388+0000","last_fullsized":"2024-09-21T10:33:39.814388+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.105746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.105746+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:20:12.502081+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":149,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814877+0000","last_change":"2024-09-21T10:28:00.300952+0000","last_active":"2024-09-21T10:33:40.814877+0000","last_peered":"2024-09-21T10:33:40.814877+0000","last_clean":"2024-09-21T10:33:40.814877+0000","last_became_active":"2024-09-21T10:28:00.293337+0000","last_became_peered":"2024-09-21T10:28:00.293337+0000","last_unstale":"2024-09-21T10:33:40.814877+0000","last_undegraded":"2024-09-21T10:33:40.814877+0000","last_fullsized":"2024-09-21T10:33:40.814877+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:58.409761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":633,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.019064+0000","last_change":"2024-09-21T10:32:40.087217+0000","last_active":"2024-09-21T10:33:41.019064+0000","last_peered":"2024-09-21T10:33:41.019064+0000","last_clean":"2024-09-21T10:33:41.019064+0000","last_became_active":"2024-09-21T10:32:40.086869+0000","last_became_peered":"2024-09-21T10:32:40.086869+0000","last_unstale":"2024-09-21T10:33:41.019064+0000","last_undegraded":"2024-09-21T10:33:41.019064+0000","last_fullsized":"2024-09-21T10:33:41.019064+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:11:40.134565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":146,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814283+0000","last_change":"2024-09-21T10:28:00.293864+0000","last_active":"2024-09-21T10:33:40.814283+0000","last_peered":"2024-09-21T10:33:40.814283+0000","last_clean":"2024-09-21T10:33:40.814283+0000","last_became_active":"2024-09-21T10:28:00.293494+0000","last_became_peered":"2024-09-21T10:28:00.293494+0000","last_unstale":"2024-09-21T10:33:40.814283+0000","last_undegraded":"2024-09-21T10:33:40.814283+0000","last_fullsized":"2024-09-21T10:33:40.814283+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:37.098068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"82'100","reported_seq":162,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814909+0000","last_change":"2024-09-21T10:32:43.908359+0000","last_active":"2024-09-21T10:33:40.814909+0000","last_peered":"2024-09-21T10:33:40.814909+0000","last_clean":"2024-09-21T10:33:40.814909+0000","last_became_active":"2024-09-21T10:28:00.290981+0000","last_became_peered":"2024-09-21T10:28:00.290981+0000","last_unstale":"2024-09-21T10:33:40.814909+0000","last_undegraded":"2024-09-21T10:33:40.814909+0000","last_fullsized":"2024-09-21T10:33:40.814909+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:32:43.908293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:32:43.908293+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:31:13.366375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":799,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.424311+0000","last_change":"2024-09-21T10:32:37.089513+0000","last_active":"2024-09-21T10:33:41.424311+0000","last_peered":"2024-09-21T10:33:41.424311+0000","last_clean":"2024-09-21T10:33:41.424311+0000","last_became_active":"2024-09-21T10:32:37.080136+0000","last_became_peered":"2024-09-21T10:32:37.080136+0000","last_unstale":"2024-09-21T10:33:41.424311+0000","last_undegraded":"2024-09-21T10:33:41.424311+0000","last_fullsized":"2024-09-21T10:33:41.424311+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:08.980579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":653,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822500+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:33:39.822500+0000","last_peered":"2024-09-21T10:33:39.822500+0000","last_clean":"2024-09-21T10:33:39.822500+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:33:39.822500+0000","last_undegraded":"2024-09-21T10:33:39.822500+0000","last_fullsized":"2024-09-21T10:33:39.822500+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":116,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.813077+0000","last_change":"2024-09-21T10:24:44.303781+0000","last_active":"2024-09-21T10:33:39.813077+0000","last_peered":"2024-09-21T10:33:39.813077+0000","last_clean":"2024-09-21T10:33:39.813077+0000","last_became_active":"2024-09-21T10:24:44.302058+0000","last_became_peered":"2024-09-21T10:24:44.302058+0000","last_unstale":"2024-09-21T10:33:39.813077+0000","last_undegraded":"2024-09-21T10:33:39.813077+0000","last_fullsized":"2024-09-21T10:33:39.813077+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:30.371229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":136,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.813768+0000","last_change":"2024-09-21T10:32:37.074825+0000","last_active":"2024-09-21T10:33:39.813768+0000","last_peered":"2024-09-21T10:33:39.813768+0000","last_clean":"2024-09-21T10:33:39.813768+0000","last_became_active":"2024-09-21T10:32:37.074496+0000","last_became_peered":"2024-09-21T10:32:37.074496+0000","last_unstale":"2024-09-21T10:33:39.813768+0000","last_undegraded":"2024-09-21T10:33:39.813768+0000","last_fullsized":"2024-09-21T10:33:39.813768+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:07.169160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":126,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.812878+0000","last_change":"2024-09-21T10:24:44.321421+0000","last_active":"2024-09-21T10:33:39.812878+0000","last_peered":"2024-09-21T10:33:39.812878+0000","last_clean":"2024-09-21T10:33:39.812878+0000","last_became_active":"2024-09-21T10:24:44.297082+0000","last_became_peered":"2024-09-21T10:24:44.297082+0000","last_unstale":"2024-09-21T10:33:39.812878+0000","last_undegraded":"2024-09-21T10:33:39.812878+0000","last_fullsized":"2024-09-21T10:33:39.812878+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:35:37.704881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":59,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.018613+0000","last_change":"2024-09-21T10:32:37.076103+0000","last_active":"2024-09-21T10:33:41.018613+0000","last_peered":"2024-09-21T10:33:41.018613+0000","last_clean":"2024-09-21T10:33:41.018613+0000","last_became_active":"2024-09-21T10:32:37.075491+0000","last_became_peered":"2024-09-21T10:32:37.075491+0000","last_unstale":"2024-09-21T10:33:41.018613+0000","last_undegraded":"2024-09-21T10:33:41.018613+0000","last_fullsized":"2024-09-21T10:33:41.018613+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:28:02.453017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":490,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.814180+0000","last_change":"2024-09-21T10:28:00.296771+0000","last_active":"2024-09-21T10:33:39.814180+0000","last_peered":"2024-09-21T10:33:39.814180+0000","last_clean":"2024-09-21T10:33:39.814180+0000","last_became_active":"2024-09-21T10:28:00.285336+0000","last_became_peered":"2024-09-21T10:28:00.285336+0000","last_unstale":"2024-09-21T10:33:39.814180+0000","last_undegraded":"2024-09-21T10:33:39.814180+0000","last_fullsized":"2024-09-21T10:33:39.814180+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:06:50.848492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":217,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822401+0000","last_change":"2024-09-21T10:32:37.078245+0000","last_active":"2024-09-21T10:33:39.822401+0000","last_peered":"2024-09-21T10:33:39.822401+0000","last_clean":"2024-09-21T10:33:39.822401+0000","last_became_active":"2024-09-21T10:32:37.077852+0000","last_became_peered":"2024-09-21T10:32:37.077852+0000","last_unstale":"2024-09-21T10:33:39.822401+0000","last_undegraded":"2024-09-21T10:33:39.822401+0000","last_fullsized":"2024-09-21T10:33:39.822401+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:11.370577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":145,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814445+0000","last_change":"2024-09-21T10:28:00.294338+0000","last_active":"2024-09-21T10:33:40.814445+0000","last_peered":"2024-09-21T10:33:40.814445+0000","last_clean":"2024-09-21T10:33:40.814445+0000","last_became_active":"2024-09-21T10:28:00.294044+0000","last_became_peered":"2024-09-21T10:28:00.294044+0000","last_unstale":"2024-09-21T10:33:40.814445+0000","last_undegraded":"2024-09-21T10:33:40.814445+0000","last_fullsized":"2024-09-21T10:33:40.814445+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:43:30.999988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":742,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.424453+0000","last_change":"2024-09-21T10:28:00.296935+0000","last_active":"2024-09-21T10:33:41.424453+0000","last_peered":"2024-09-21T10:33:41.424453+0000","last_clean":"2024-09-21T10:33:41.424453+0000","last_became_active":"2024-09-21T10:28:00.296562+0000","last_became_peered":"2024-09-21T10:28:00.296562+0000","last_unstale":"2024-09-21T10:33:41.424453+0000","last_undegraded":"2024-09-21T10:33:41.424453+0000","last_fullsized":"2024-09-21T10:33:41.424453+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:06.863310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":166,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.164508+0000","last_change":"2024-09-21T10:33:41.164439+0000","last_active":"2024-09-21T10:33:41.164508+0000","last_peered":"2024-09-21T10:33:41.164508+0000","last_clean":"2024-09-21T10:33:41.164508+0000","last_became_active":"2024-09-21T10:33:41.164178+0000","last_became_peered":"2024-09-21T10:33:41.164178+0000","last_unstale":"2024-09-21T10:33:41.164508+0000","last_undegraded":"2024-09-21T10:33:41.164508+0000","last_fullsized":"2024-09-21T10:33:41.164508+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:21.525842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":410,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822414+0000","last_change":"2024-09-21T10:32:37.073911+0000","last_active":"2024-09-21T10:33:39.822414+0000","last_peered":"2024-09-21T10:33:39.822414+0000","last_clean":"2024-09-21T10:33:39.822414+0000","last_became_active":"2024-09-21T10:32:37.073507+0000","last_became_peered":"2024-09-21T10:32:37.073507+0000","last_unstale":"2024-09-21T10:33:39.822414+0000","last_undegraded":"2024-09-21T10:33:39.822414+0000","last_fullsized":"2024-09-21T10:33:39.822414+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:52.244604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":159,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814034+0000","last_change":"2024-09-21T10:32:37.079936+0000","last_active":"2024-09-21T10:33:40.814034+0000","last_peered":"2024-09-21T10:33:40.814034+0000","last_clean":"2024-09-21T10:33:40.814034+0000","last_became_active":"2024-09-21T10:32:37.079625+0000","last_became_peered":"2024-09-21T10:32:37.079625+0000","last_unstale":"2024-09-21T10:33:40.814034+0000","last_undegraded":"2024-09-21T10:33:40.814034+0000","last_fullsized":"2024-09-21T10:33:40.814034+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:43:45.397882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":759,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.424690+0000","last_change":"2024-09-21T10:28:00.297304+0000","last_active":"2024-09-21T10:33:41.424690+0000","last_peered":"2024-09-21T10:33:41.424690+0000","last_clean":"2024-09-21T10:33:41.424690+0000","last_became_active":"2024-09-21T10:28:00.296631+0000","last_became_peered":"2024-09-21T10:28:00.296631+0000","last_unstale":"2024-09-21T10:33:41.424690+0000","last_undegraded":"2024-09-21T10:33:41.424690+0000","last_fullsized":"2024-09-21T10:33:41.424690+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:22:16.633581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":59,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.018528+0000","last_change":"2024-09-21T10:32:37.083327+0000","last_active":"2024-09-21T10:33:41.018528+0000","last_peered":"2024-09-21T10:33:41.018528+0000","last_clean":"2024-09-21T10:33:41.018528+0000","last_became_active":"2024-09-21T10:32:37.082725+0000","last_became_peered":"2024-09-21T10:32:37.082725+0000","last_unstale":"2024-09-21T10:33:41.018528+0000","last_undegraded":"2024-09-21T10:33:41.018528+0000","last_fullsized":"2024-09-21T10:33:41.018528+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:22.604264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":159,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814677+0000","last_change":"2024-09-21T10:32:37.080087+0000","last_active":"2024-09-21T10:33:40.814677+0000","last_peered":"2024-09-21T10:33:40.814677+0000","last_clean":"2024-09-21T10:33:40.814677+0000","last_became_active":"2024-09-21T10:32:37.079766+0000","last_became_peered":"2024-09-21T10:32:37.079766+0000","last_unstale":"2024-09-21T10:33:40.814677+0000","last_undegraded":"2024-09-21T10:33:40.814677+0000","last_fullsized":"2024-09-21T10:33:40.814677+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:13:29.691366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":136,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.813154+0000","last_change":"2024-09-21T10:32:37.075224+0000","last_active":"2024-09-21T10:33:39.813154+0000","last_peered":"2024-09-21T10:33:39.813154+0000","last_clean":"2024-09-21T10:33:39.813154+0000","last_became_active":"2024-09-21T10:32:37.074655+0000","last_became_peered":"2024-09-21T10:32:37.074655+0000","last_unstale":"2024-09-21T10:33:39.813154+0000","last_undegraded":"2024-09-21T10:33:39.813154+0000","last_fullsized":"2024-09-21T10:33:39.813154+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:46.280140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":410,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.854979+0000","last_change":"2024-09-21T10:32:37.074510+0000","last_active":"2024-09-21T10:33:40.854979+0000","last_peered":"2024-09-21T10:33:40.854979+0000","last_clean":"2024-09-21T10:33:40.854979+0000","last_became_active":"2024-09-21T10:32:37.074070+0000","last_became_peered":"2024-09-21T10:32:37.074070+0000","last_unstale":"2024-09-21T10:33:40.854979+0000","last_undegraded":"2024-09-21T10:33:40.854979+0000","last_fullsized":"2024-09-21T10:33:40.854979+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:26.357339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":748,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.424506+0000","last_change":"2024-09-21T10:24:44.299989+0000","last_active":"2024-09-21T10:33:41.424506+0000","last_peered":"2024-09-21T10:33:41.424506+0000","last_clean":"2024-09-21T10:33:41.424506+0000","last_became_active":"2024-09-21T10:24:44.299356+0000","last_became_peered":"2024-09-21T10:24:44.299356+0000","last_unstale":"2024-09-21T10:33:41.424506+0000","last_undegraded":"2024-09-21T10:33:41.424506+0000","last_fullsized":"2024-09-21T10:33:41.424506+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:58.867008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":146,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.813572+0000","last_change":"2024-09-21T10:32:37.075907+0000","last_active":"2024-09-21T10:33:39.813572+0000","last_peered":"2024-09-21T10:33:39.813572+0000","last_clean":"2024-09-21T10:33:39.813572+0000","last_became_active":"2024-09-21T10:32:37.075581+0000","last_became_peered":"2024-09-21T10:32:37.075581+0000","last_unstale":"2024-09-21T10:33:39.813572+0000","last_undegraded":"2024-09-21T10:33:39.813572+0000","last_fullsized":"2024-09-21T10:33:39.813572+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:53.334817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":158,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822550+0000","last_change":"2024-09-21T10:33:26.770588+0000","last_active":"2024-09-21T10:33:39.822550+0000","last_peered":"2024-09-21T10:33:39.822550+0000","last_clean":"2024-09-21T10:33:39.822550+0000","last_became_active":"2024-09-21T10:33:26.769758+0000","last_became_peered":"2024-09-21T10:33:26.769758+0000","last_unstale":"2024-09-21T10:33:39.822550+0000","last_undegraded":"2024-09-21T10:33:39.822550+0000","last_fullsized":"2024-09-21T10:33:39.822550+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:59.264432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":126,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.813221+0000","last_change":"2024-09-21T10:28:00.286009+0000","last_active":"2024-09-21T10:33:39.813221+0000","last_peered":"2024-09-21T10:33:39.813221+0000","last_clean":"2024-09-21T10:33:39.813221+0000","last_became_active":"2024-09-21T10:28:00.285327+0000","last_became_peered":"2024-09-21T10:28:00.285327+0000","last_unstale":"2024-09-21T10:33:39.813221+0000","last_undegraded":"2024-09-21T10:33:39.813221+0000","last_fullsized":"2024-09-21T10:33:39.813221+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:34.985354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":141,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822783+0000","last_change":"2024-09-21T10:33:26.769225+0000","last_active":"2024-09-21T10:33:39.822783+0000","last_peered":"2024-09-21T10:33:39.822783+0000","last_clean":"2024-09-21T10:33:39.822783+0000","last_became_active":"2024-09-21T10:33:26.768810+0000","last_became_peered":"2024-09-21T10:33:26.768810+0000","last_unstale":"2024-09-21T10:33:39.822783+0000","last_undegraded":"2024-09-21T10:33:39.822783+0000","last_fullsized":"2024-09-21T10:33:39.822783+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:19:53.205774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":145,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814438+0000","last_change":"2024-09-21T10:28:00.293054+0000","last_active":"2024-09-21T10:33:40.814438+0000","last_peered":"2024-09-21T10:33:40.814438+0000","last_clean":"2024-09-21T10:33:40.814438+0000","last_became_active":"2024-09-21T10:28:00.292723+0000","last_became_peered":"2024-09-21T10:28:00.292723+0000","last_unstale":"2024-09-21T10:33:40.814438+0000","last_undegraded":"2024-09-21T10:33:40.814438+0000","last_fullsized":"2024-09-21T10:33:40.814438+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:01:45.933570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":146,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814868+0000","last_change":"2024-09-21T10:28:00.285515+0000","last_active":"2024-09-21T10:33:40.814868+0000","last_peered":"2024-09-21T10:33:40.814868+0000","last_clean":"2024-09-21T10:33:40.814868+0000","last_became_active":"2024-09-21T10:28:00.285203+0000","last_became_peered":"2024-09-21T10:28:00.285203+0000","last_unstale":"2024-09-21T10:33:40.814868+0000","last_undegraded":"2024-09-21T10:33:40.814868+0000","last_fullsized":"2024-09-21T10:33:40.814868+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:41.982228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":136,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.812764+0000","last_change":"2024-09-21T10:32:37.077513+0000","last_active":"2024-09-21T10:33:39.812764+0000","last_peered":"2024-09-21T10:33:39.812764+0000","last_clean":"2024-09-21T10:33:39.812764+0000","last_became_active":"2024-09-21T10:32:37.077161+0000","last_became_peered":"2024-09-21T10:32:37.077161+0000","last_unstale":"2024-09-21T10:33:39.812764+0000","last_undegraded":"2024-09-21T10:33:39.812764+0000","last_fullsized":"2024-09-21T10:33:39.812764+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:33.823372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":734,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.424413+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:33:41.424413+0000","last_peered":"2024-09-21T10:33:41.424413+0000","last_clean":"2024-09-21T10:33:41.424413+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:33:41.424413+0000","last_undegraded":"2024-09-21T10:33:41.424413+0000","last_fullsized":"2024-09-21T10:33:41.424413+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":10252,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5983,"num_read_kb":13006,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1100,"ondisk_log_size":1100,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453398036,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1050956,"kb_used_data":153732,"kb_used_omap":1,"kb_used_meta":897214,"kb_avail":92694196,"statfs":{"total":95995035648,"available":94918856704,"internally_reserved":0,"allocated":157421568,"data_stored":154994726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918747480},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594264,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":906956,"kb_used_data":5628,"kb_used_omap":1,"kb_used_meta":901310,"kb_avail":92838196,"statfs":{"total":95995035648,"available":95066312704,"internally_reserved":0,"allocated":5763072,"data_stored":3344295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922941784},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790494,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":914368,"kb_used_data":10292,"kb_used_omap":1,"kb_used_meta":904062,"kb_avail":92830784,"statfs":{"total":95995035648,"available":95058722816,"internally_reserved":0,"allocated":10539008,"data_stored":8124478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":925759832},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986723,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":747848,"kb_used_data":157952,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":92997304,"statfs":{"total":95995035648,"available":95229239296,"internally_reserved":0,"allocated":161742848,"data_stored":159320024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":101,"seq":433791697285,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":696184,"kb_used_data":6192,"kb_used_omap":5,"kb_used_meta":689978,"kb_avail":93048968,"statfs":{"total":95995035648,"available":95282143232,"internally_reserved":0,"allocated":6340608,"data_stored":3929932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":706538256},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860812,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1037508,"kb_used_data":9972,"kb_used_omap":8,"kb_used_meta":1027511,"kb_avail":92707644,"statfs":{"total":95995035648,"available":94932627456,"internally_reserved":0,"allocated":10211328,"data_stored":7797078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":1052172140},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024338,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":984620,"kb_used_data":134884,"kb_used_omap":7,"kb_used_meta":849720,"kb_avail":92760532,"statfs":{"total":95995035648,"available":94986784768,"internally_reserved":0,"allocated":138121216,"data_stored":135721591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":870113842},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187873,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1066300,"kb_used_data":5616,"kb_used_omap":8,"kb_used_meta":1060663,"kb_avail":92678852,"statfs":{"total":95995035648,"available":94903144448,"internally_reserved":0,"allocated":5750784,"data_stored":3340074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":1086119921},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:34:28.053 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 26a15512-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:34:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:27 smithi196 ceph-mon[84784]: from='client.46850 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:34:28.300 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:34:28.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:27 smithi098 ceph-mon[97580]: from='client.46850 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:34:28.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:27 smithi098 ceph-mon[100914]: from='client.46850 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:34:29.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:28 smithi098 ceph-mon[97580]: pgmap v1483: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:29.097 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:28 smithi098 ceph-mon[100914]: pgmap v1483: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:29.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:28 smithi196 ceph-mon[84784]: pgmap v1483: 83 pgs: 83 active+clean; 4.6 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:30.166 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-21T10:34:30.167 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-21T10:34:30.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:34:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:34:30.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:34:30.885 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1484,"stamp":"2024-09-21T10:34:29.692495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5983,"num_read_kb":13006,"num_write":907,"num_write_kb":12623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":10252,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2132,"ondisk_log_size":2132,"up":249,"acting":249,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":252,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7385540,"kb_used_data":465068,"kb_used_omap":35,"kb_used_meta":6920348,"kb_avail":742575676,"statfs":{"total":767960285184,"available":760397492224,"internally_reserved":0,"allocated":476229632,"data_stored":456911398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36353,"internal_metadata":7086436863},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002733"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":159,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814761+0000","last_change":"2024-09-21T10:32:37.077344+0000","last_active":"2024-09-21T10:33:40.814761+0000","last_peered":"2024-09-21T10:33:40.814761+0000","last_clean":"2024-09-21T10:33:40.814761+0000","last_became_active":"2024-09-21T10:32:37.077009+0000","last_became_peered":"2024-09-21T10:32:37.077009+0000","last_unstale":"2024-09-21T10:33:40.814761+0000","last_undegraded":"2024-09-21T10:33:40.814761+0000","last_fullsized":"2024-09-21T10:33:40.814761+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:45.790900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":457,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.813606+0000","last_change":"2024-09-21T10:24:44.305198+0000","last_active":"2024-09-21T10:33:39.813606+0000","last_peered":"2024-09-21T10:33:39.813606+0000","last_clean":"2024-09-21T10:33:39.813606+0000","last_became_active":"2024-09-21T10:24:44.297653+0000","last_became_peered":"2024-09-21T10:24:44.297653+0000","last_unstale":"2024-09-21T10:33:39.813606+0000","last_undegraded":"2024-09-21T10:33:39.813606+0000","last_fullsized":"2024-09-21T10:33:39.813606+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:19.320833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":293,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822909+0000","last_change":"2024-09-21T10:28:00.294917+0000","last_active":"2024-09-21T10:33:39.822909+0000","last_peered":"2024-09-21T10:33:39.822909+0000","last_clean":"2024-09-21T10:33:39.822909+0000","last_became_active":"2024-09-21T10:28:00.294134+0000","last_became_peered":"2024-09-21T10:28:00.294134+0000","last_unstale":"2024-09-21T10:33:39.822909+0000","last_undegraded":"2024-09-21T10:33:39.822909+0000","last_fullsized":"2024-09-21T10:33:39.822909+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:29:05.780212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":453,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.424250+0000","last_change":"2024-09-21T10:32:54.793377+0000","last_active":"2024-09-21T10:33:41.424250+0000","last_peered":"2024-09-21T10:33:41.424250+0000","last_clean":"2024-09-21T10:33:41.424250+0000","last_became_active":"2024-09-21T10:32:54.793082+0000","last_became_peered":"2024-09-21T10:32:54.793082+0000","last_unstale":"2024-09-21T10:33:41.424250+0000","last_undegraded":"2024-09-21T10:33:41.424250+0000","last_fullsized":"2024-09-21T10:33:41.424250+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:20:50.891853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":742,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.424657+0000","last_change":"2024-09-21T10:28:00.297783+0000","last_active":"2024-09-21T10:33:41.424657+0000","last_peered":"2024-09-21T10:33:41.424657+0000","last_clean":"2024-09-21T10:33:41.424657+0000","last_became_active":"2024-09-21T10:28:00.297024+0000","last_became_peered":"2024-09-21T10:28:00.297024+0000","last_unstale":"2024-09-21T10:33:41.424657+0000","last_undegraded":"2024-09-21T10:33:41.424657+0000","last_fullsized":"2024-09-21T10:33:41.424657+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:12:24.382523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.823600+0000","last_change":"2024-09-21T10:33:39.823280+0000","last_active":"2024-09-21T10:33:39.823600+0000","last_peered":"2024-09-21T10:33:39.823600+0000","last_clean":"2024-09-21T10:33:39.823600+0000","last_became_active":"2024-09-21T10:33:39.822965+0000","last_became_peered":"2024-09-21T10:33:39.822965+0000","last_unstale":"2024-09-21T10:33:39.823600+0000","last_undegraded":"2024-09-21T10:33:39.823600+0000","last_fullsized":"2024-09-21T10:33:39.823600+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":426,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:39:40.734884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":166,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.816073+0000","last_change":"2024-09-21T10:33:40.816012+0000","last_active":"2024-09-21T10:33:40.816073+0000","last_peered":"2024-09-21T10:33:40.816073+0000","last_clean":"2024-09-21T10:33:40.816073+0000","last_became_active":"2024-09-21T10:33:40.815701+0000","last_became_peered":"2024-09-21T10:33:40.815701+0000","last_unstale":"2024-09-21T10:33:40.816073+0000","last_undegraded":"2024-09-21T10:33:40.816073+0000","last_fullsized":"2024-09-21T10:33:40.816073+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:51:26.944089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":399,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822240+0000","last_change":"2024-09-21T09:59:54.266091+0000","last_active":"2024-09-21T10:33:39.822240+0000","last_peered":"2024-09-21T10:33:39.822240+0000","last_clean":"2024-09-21T10:33:39.822240+0000","last_became_active":"2024-09-21T09:59:54.265626+0000","last_became_peered":"2024-09-21T09:59:54.265626+0000","last_unstale":"2024-09-21T10:33:39.822240+0000","last_undegraded":"2024-09-21T10:33:39.822240+0000","last_fullsized":"2024-09-21T10:33:39.822240+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:44.147961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":448,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.814794+0000","last_change":"2024-09-21T10:24:44.325196+0000","last_active":"2024-09-21T10:33:39.814794+0000","last_peered":"2024-09-21T10:33:39.814794+0000","last_clean":"2024-09-21T10:33:39.814794+0000","last_became_active":"2024-09-21T10:24:44.299614+0000","last_became_peered":"2024-09-21T10:24:44.299614+0000","last_unstale":"2024-09-21T10:33:39.814794+0000","last_undegraded":"2024-09-21T10:33:39.814794+0000","last_fullsized":"2024-09-21T10:33:39.814794+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:14.558663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":179,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822773+0000","last_change":"2024-09-21T10:26:41.305571+0000","last_active":"2024-09-21T10:33:39.822773+0000","last_peered":"2024-09-21T10:33:39.822773+0000","last_clean":"2024-09-21T10:33:39.822773+0000","last_became_active":"2024-09-21T10:26:41.305135+0000","last_became_peered":"2024-09-21T10:26:41.305135+0000","last_unstale":"2024-09-21T10:33:39.822773+0000","last_undegraded":"2024-09-21T10:33:39.822773+0000","last_fullsized":"2024-09-21T10:33:39.822773+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:49:55.452467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":477,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.813917+0000","last_change":"2024-09-21T10:32:37.086045+0000","last_active":"2024-09-21T10:33:39.813917+0000","last_peered":"2024-09-21T10:33:39.813917+0000","last_clean":"2024-09-21T10:33:39.813917+0000","last_became_active":"2024-09-21T10:32:37.075889+0000","last_became_peered":"2024-09-21T10:32:37.075889+0000","last_unstale":"2024-09-21T10:33:39.813917+0000","last_undegraded":"2024-09-21T10:33:39.813917+0000","last_fullsized":"2024-09-21T10:33:39.813917+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:58:14.277307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":156,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.855976+0000","last_change":"2024-09-21T10:33:40.855976+0000","last_active":"2024-09-21T10:33:40.855976+0000","last_peered":"2024-09-21T10:33:40.855976+0000","last_clean":"2024-09-21T10:33:40.855976+0000","last_became_active":"2024-09-21T10:33:40.855672+0000","last_became_peered":"2024-09-21T10:33:40.855672+0000","last_unstale":"2024-09-21T10:33:40.855976+0000","last_undegraded":"2024-09-21T10:33:40.855976+0000","last_fullsized":"2024-09-21T10:33:40.855976+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:50:16.228416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":375,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.814330+0000","last_change":"2024-09-21T09:52:20.370642+0000","last_active":"2024-09-21T10:33:39.814330+0000","last_peered":"2024-09-21T10:33:39.814330+0000","last_clean":"2024-09-21T10:33:39.814330+0000","last_became_active":"2024-09-21T09:52:20.370284+0000","last_became_peered":"2024-09-21T09:52:20.370284+0000","last_unstale":"2024-09-21T10:33:39.814330+0000","last_undegraded":"2024-09-21T10:33:39.814330+0000","last_fullsized":"2024-09-21T10:33:39.814330+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:35.625046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":410,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822478+0000","last_change":"2024-09-21T10:32:37.073962+0000","last_active":"2024-09-21T10:33:39.822478+0000","last_peered":"2024-09-21T10:33:39.822478+0000","last_clean":"2024-09-21T10:33:39.822478+0000","last_became_active":"2024-09-21T10:32:37.073651+0000","last_became_peered":"2024-09-21T10:32:37.073651+0000","last_unstale":"2024-09-21T10:33:39.822478+0000","last_undegraded":"2024-09-21T10:33:39.822478+0000","last_fullsized":"2024-09-21T10:33:39.822478+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:07:15.433004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":128,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822669+0000","last_change":"2024-09-21T10:33:26.769245+0000","last_active":"2024-09-21T10:33:39.822669+0000","last_peered":"2024-09-21T10:33:39.822669+0000","last_clean":"2024-09-21T10:33:39.822669+0000","last_became_active":"2024-09-21T10:33:26.768656+0000","last_became_peered":"2024-09-21T10:33:26.768656+0000","last_unstale":"2024-09-21T10:33:39.822669+0000","last_undegraded":"2024-09-21T10:33:39.822669+0000","last_fullsized":"2024-09-21T10:33:39.822669+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:54:32.081451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":778,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.423999+0000","last_change":"2024-09-21T10:32:37.084484+0000","last_active":"2024-09-21T10:33:41.423999+0000","last_peered":"2024-09-21T10:33:41.423999+0000","last_clean":"2024-09-21T10:33:41.423999+0000","last_became_active":"2024-09-21T10:32:37.084156+0000","last_became_peered":"2024-09-21T10:32:37.084156+0000","last_unstale":"2024-09-21T10:33:41.423999+0000","last_undegraded":"2024-09-21T10:33:41.423999+0000","last_fullsized":"2024-09-21T10:33:41.423999+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:38:59.721832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"82'100","reported_seq":548,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.812748+0000","last_change":"2024-09-21T10:28:00.285919+0000","last_active":"2024-09-21T10:33:39.812748+0000","last_peered":"2024-09-21T10:33:39.812748+0000","last_clean":"2024-09-21T10:33:39.812748+0000","last_became_active":"2024-09-21T10:28:00.285324+0000","last_became_peered":"2024-09-21T10:28:00.285324+0000","last_unstale":"2024-09-21T10:33:39.812748+0000","last_undegraded":"2024-09-21T10:33:39.812748+0000","last_fullsized":"2024-09-21T10:33:39.812748+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:22.728195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:22.728195+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:00.841610+0000","scrub_duration":0.0069698140000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":149,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814247+0000","last_change":"2024-09-21T10:28:00.303125+0000","last_active":"2024-09-21T10:33:40.814247+0000","last_peered":"2024-09-21T10:33:40.814247+0000","last_clean":"2024-09-21T10:33:40.814247+0000","last_became_active":"2024-09-21T10:28:00.293684+0000","last_became_peered":"2024-09-21T10:28:00.293684+0000","last_unstale":"2024-09-21T10:33:40.814247+0000","last_undegraded":"2024-09-21T10:33:40.814247+0000","last_fullsized":"2024-09-21T10:33:40.814247+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:04.495913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":126,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.814164+0000","last_change":"2024-09-21T10:28:00.283456+0000","last_active":"2024-09-21T10:33:39.814164+0000","last_peered":"2024-09-21T10:33:39.814164+0000","last_clean":"2024-09-21T10:33:39.814164+0000","last_became_active":"2024-09-21T10:28:00.282881+0000","last_became_peered":"2024-09-21T10:28:00.282881+0000","last_unstale":"2024-09-21T10:33:39.814164+0000","last_undegraded":"2024-09-21T10:33:39.814164+0000","last_fullsized":"2024-09-21T10:33:39.814164+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:43.681481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"82'100","reported_seq":29,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822679+0000","last_change":"2024-09-21T10:33:29.952265+0000","last_active":"2024-09-21T10:33:39.822679+0000","last_peered":"2024-09-21T10:33:39.822679+0000","last_clean":"2024-09-21T10:33:39.822679+0000","last_became_active":"2024-09-21T10:33:26.770295+0000","last_became_peered":"2024-09-21T10:33:26.770295+0000","last_unstale":"2024-09-21T10:33:39.822679+0000","last_undegraded":"2024-09-21T10:33:39.822679+0000","last_fullsized":"2024-09-21T10:33:39.822679+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:33:29.952194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:33:29.952194+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:01:43.700109+0000","scrub_duration":0.0024670159999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":461,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822533+0000","last_change":"2024-09-21T10:32:37.073933+0000","last_active":"2024-09-21T10:33:39.822533+0000","last_peered":"2024-09-21T10:33:39.822533+0000","last_clean":"2024-09-21T10:33:39.822533+0000","last_became_active":"2024-09-21T10:32:37.073604+0000","last_became_peered":"2024-09-21T10:32:37.073604+0000","last_unstale":"2024-09-21T10:33:39.822533+0000","last_undegraded":"2024-09-21T10:33:39.822533+0000","last_fullsized":"2024-09-21T10:33:39.822533+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:43:20.583916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":136,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.814462+0000","last_change":"2024-09-21T10:32:37.075432+0000","last_active":"2024-09-21T10:33:39.814462+0000","last_peered":"2024-09-21T10:33:39.814462+0000","last_clean":"2024-09-21T10:33:39.814462+0000","last_became_active":"2024-09-21T10:32:37.075087+0000","last_became_peered":"2024-09-21T10:32:37.075087+0000","last_unstale":"2024-09-21T10:33:39.814462+0000","last_undegraded":"2024-09-21T10:33:39.814462+0000","last_fullsized":"2024-09-21T10:33:39.814462+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:06:43.955465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":196,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.424371+0000","last_change":"2024-09-21T10:25:39.872680+0000","last_active":"2024-09-21T10:33:41.424371+0000","last_peered":"2024-09-21T10:33:41.424371+0000","last_clean":"2024-09-21T10:33:41.424371+0000","last_became_active":"2024-09-21T10:25:39.872312+0000","last_became_peered":"2024-09-21T10:25:39.872312+0000","last_unstale":"2024-09-21T10:33:41.424371+0000","last_undegraded":"2024-09-21T10:33:41.424371+0000","last_fullsized":"2024-09-21T10:33:41.424371+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:54.942046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":136,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.814525+0000","last_change":"2024-09-21T10:32:37.077131+0000","last_active":"2024-09-21T10:33:39.814525+0000","last_peered":"2024-09-21T10:33:39.814525+0000","last_clean":"2024-09-21T10:33:39.814525+0000","last_became_active":"2024-09-21T10:32:37.076783+0000","last_became_peered":"2024-09-21T10:32:37.076783+0000","last_unstale":"2024-09-21T10:33:39.814525+0000","last_undegraded":"2024-09-21T10:33:39.814525+0000","last_fullsized":"2024-09-21T10:33:39.814525+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:05:20.632915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'100","reported_seq":29,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.823020+0000","last_change":"2024-09-21T10:33:26.873794+0000","last_active":"2024-09-21T10:33:39.823020+0000","last_peered":"2024-09-21T10:33:39.823020+0000","last_clean":"2024-09-21T10:33:39.823020+0000","last_became_active":"2024-09-21T10:33:26.769799+0000","last_became_peered":"2024-09-21T10:33:26.769799+0000","last_unstale":"2024-09-21T10:33:39.823020+0000","last_undegraded":"2024-09-21T10:33:39.823020+0000","last_fullsized":"2024-09-21T10:33:39.823020+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:33:26.873733+0000","last_deep_scrub":"82'100","last_deep_scrub_stamp":"2024-09-21T10:33:26.873733+0000","last_clean_scrub_stamp":"2024-09-21T10:33:26.873733+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:22.217151+0000","scrub_duration":0.0016179689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":159,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814342+0000","last_change":"2024-09-21T10:32:37.078156+0000","last_active":"2024-09-21T10:33:40.814342+0000","last_peered":"2024-09-21T10:33:40.814342+0000","last_clean":"2024-09-21T10:33:40.814342+0000","last_became_active":"2024-09-21T10:32:37.077192+0000","last_became_peered":"2024-09-21T10:32:37.077192+0000","last_unstale":"2024-09-21T10:33:40.814342+0000","last_undegraded":"2024-09-21T10:33:40.814342+0000","last_fullsized":"2024-09-21T10:33:40.814342+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:21.833793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":59,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.018999+0000","last_change":"2024-09-21T10:32:37.083366+0000","last_active":"2024-09-21T10:33:41.018999+0000","last_peered":"2024-09-21T10:33:41.018999+0000","last_clean":"2024-09-21T10:33:41.018999+0000","last_became_active":"2024-09-21T10:32:37.082694+0000","last_became_peered":"2024-09-21T10:32:37.082694+0000","last_unstale":"2024-09-21T10:33:41.018999+0000","last_undegraded":"2024-09-21T10:33:41.018999+0000","last_fullsized":"2024-09-21T10:33:41.018999+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:40.298562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"82'100","reported_seq":529,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.812832+0000","last_change":"2024-09-21T10:23:18.790100+0000","last_active":"2024-09-21T10:33:39.812832+0000","last_peered":"2024-09-21T10:33:39.812832+0000","last_clean":"2024-09-21T10:33:39.812832+0000","last_became_active":"2024-09-21T10:23:14.677610+0000","last_became_peered":"2024-09-21T10:23:14.677610+0000","last_unstale":"2024-09-21T10:33:39.812832+0000","last_undegraded":"2024-09-21T10:33:39.812832+0000","last_fullsized":"2024-09-21T10:33:39.812832+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:18.790017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:18.790017+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:56.562241+0000","scrub_duration":0.0080400790000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"78'152","reported_seq":621,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.821691+0000","last_change":"2024-09-21T10:26:41.279374+0000","last_active":"2024-09-21T10:33:39.821691+0000","last_peered":"2024-09-21T10:33:39.821691+0000","last_clean":"2024-09-21T10:33:39.821691+0000","last_became_active":"2024-09-21T10:26:41.278994+0000","last_became_peered":"2024-09-21T10:26:41.278994+0000","last_unstale":"2024-09-21T10:33:39.821691+0000","last_undegraded":"2024-09-21T10:33:39.821691+0000","last_fullsized":"2024-09-21T10:33:39.821691+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:49:57.821175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000235615,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":482,"num_read_kb":1581,"num_write":131,"num_write_kb":6279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":146,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814226+0000","last_change":"2024-09-21T10:28:00.295436+0000","last_active":"2024-09-21T10:33:40.814226+0000","last_peered":"2024-09-21T10:33:40.814226+0000","last_clean":"2024-09-21T10:33:40.814226+0000","last_became_active":"2024-09-21T10:28:00.295135+0000","last_became_peered":"2024-09-21T10:28:00.295135+0000","last_unstale":"2024-09-21T10:33:40.814226+0000","last_undegraded":"2024-09-21T10:33:40.814226+0000","last_fullsized":"2024-09-21T10:33:40.814226+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:14:03.027395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":413,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822016+0000","last_change":"2024-09-21T10:24:44.308008+0000","last_active":"2024-09-21T10:33:39.822016+0000","last_peered":"2024-09-21T10:33:39.822016+0000","last_clean":"2024-09-21T10:33:39.822016+0000","last_became_active":"2024-09-21T10:24:44.307595+0000","last_became_peered":"2024-09-21T10:24:44.307595+0000","last_unstale":"2024-09-21T10:33:39.822016+0000","last_undegraded":"2024-09-21T10:33:39.822016+0000","last_fullsized":"2024-09-21T10:33:39.822016+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:33:49.448676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"82'100","reported_seq":60,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.018623+0000","last_change":"2024-09-21T10:32:37.081585+0000","last_active":"2024-09-21T10:33:41.018623+0000","last_peered":"2024-09-21T10:33:41.018623+0000","last_clean":"2024-09-21T10:33:41.018623+0000","last_became_active":"2024-09-21T10:32:37.080879+0000","last_became_peered":"2024-09-21T10:32:37.080879+0000","last_unstale":"2024-09-21T10:33:41.018623+0000","last_undegraded":"2024-09-21T10:33:41.018623+0000","last_fullsized":"2024-09-21T10:33:41.018623+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:32:26.379765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:32:26.379765+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:45.658670+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"78'126","reported_seq":863,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.812748+0000","last_change":"2024-09-21T10:32:40.086328+0000","last_active":"2024-09-21T10:33:39.812748+0000","last_peered":"2024-09-21T10:33:39.812748+0000","last_clean":"2024-09-21T10:33:39.812748+0000","last_became_active":"2024-09-21T10:32:40.085287+0000","last_became_peered":"2024-09-21T10:32:40.085287+0000","last_unstale":"2024-09-21T10:33:39.812748+0000","last_undegraded":"2024-09-21T10:33:39.812748+0000","last_fullsized":"2024-09-21T10:33:39.812748+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:15.146359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010407249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":595,"num_read_kb":1696,"num_write":119,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":772,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.424432+0000","last_change":"2024-09-21T10:28:00.296831+0000","last_active":"2024-09-21T10:33:41.424432+0000","last_peered":"2024-09-21T10:33:41.424432+0000","last_clean":"2024-09-21T10:33:41.424432+0000","last_became_active":"2024-09-21T10:28:00.296527+0000","last_became_peered":"2024-09-21T10:28:00.296527+0000","last_unstale":"2024-09-21T10:33:41.424432+0000","last_undegraded":"2024-09-21T10:33:41.424432+0000","last_fullsized":"2024-09-21T10:33:41.424432+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:18.024868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":754,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.424009+0000","last_change":"2024-09-21T10:32:37.080391+0000","last_active":"2024-09-21T10:33:41.424009+0000","last_peered":"2024-09-21T10:33:41.424009+0000","last_clean":"2024-09-21T10:33:41.424009+0000","last_became_active":"2024-09-21T10:32:37.080065+0000","last_became_peered":"2024-09-21T10:32:37.080065+0000","last_unstale":"2024-09-21T10:33:41.424009+0000","last_undegraded":"2024-09-21T10:33:41.424009+0000","last_fullsized":"2024-09-21T10:33:41.424009+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:12:42.262043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"82'100","reported_seq":426,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.572568+0000","last_change":"2024-09-21T10:33:40.572529+0000","last_active":"2024-09-21T10:33:40.572568+0000","last_peered":"2024-09-21T10:33:40.572568+0000","last_clean":"2024-09-21T10:33:40.572568+0000","last_became_active":"2024-09-21T10:33:39.818529+0000","last_became_peered":"2024-09-21T10:33:39.818529+0000","last_unstale":"2024-09-21T10:33:40.572568+0000","last_undegraded":"2024-09-21T10:33:40.572568+0000","last_fullsized":"2024-09-21T10:33:40.572568+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":426,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:33:40.572474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:33:40.572474+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:20.210668+0000","scrub_duration":0.0014184460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"78'159","reported_seq":1549,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.855130+0000","last_change":"2024-09-21T10:28:03.346064+0000","last_active":"2024-09-21T10:33:40.855130+0000","last_peered":"2024-09-21T10:33:40.855130+0000","last_clean":"2024-09-21T10:33:40.855130+0000","last_became_active":"2024-09-21T10:28:03.345313+0000","last_became_peered":"2024-09-21T10:28:03.345313+0000","last_unstale":"2024-09-21T10:33:40.855130+0000","last_undegraded":"2024-09-21T10:33:40.855130+0000","last_fullsized":"2024-09-21T10:33:40.855130+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:32.097521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202598,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1296,"num_read_kb":2706,"num_write":130,"num_write_kb":1396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":59,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.018514+0000","last_change":"2024-09-21T10:32:37.076203+0000","last_active":"2024-09-21T10:33:41.018514+0000","last_peered":"2024-09-21T10:33:41.018514+0000","last_clean":"2024-09-21T10:33:41.018514+0000","last_became_active":"2024-09-21T10:32:37.075328+0000","last_became_peered":"2024-09-21T10:32:37.075328+0000","last_unstale":"2024-09-21T10:33:41.018514+0000","last_undegraded":"2024-09-21T10:33:41.018514+0000","last_fullsized":"2024-09-21T10:33:41.018514+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:44:29.546239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":59,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.018943+0000","last_change":"2024-09-21T10:32:37.076492+0000","last_active":"2024-09-21T10:33:41.018943+0000","last_peered":"2024-09-21T10:33:41.018943+0000","last_clean":"2024-09-21T10:33:41.018943+0000","last_became_active":"2024-09-21T10:32:37.075758+0000","last_became_peered":"2024-09-21T10:32:37.075758+0000","last_unstale":"2024-09-21T10:33:41.018943+0000","last_undegraded":"2024-09-21T10:33:41.018943+0000","last_fullsized":"2024-09-21T10:33:41.018943+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:46:11.681207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"82'100","reported_seq":175,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822549+0000","last_change":"2024-09-21T10:33:28.904517+0000","last_active":"2024-09-21T10:33:39.822549+0000","last_peered":"2024-09-21T10:33:39.822549+0000","last_clean":"2024-09-21T10:33:39.822549+0000","last_became_active":"2024-09-21T10:33:26.769976+0000","last_became_peered":"2024-09-21T10:33:26.769976+0000","last_unstale":"2024-09-21T10:33:39.822549+0000","last_undegraded":"2024-09-21T10:33:39.822549+0000","last_fullsized":"2024-09-21T10:33:39.822549+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:33:28.904449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:33:28.904449+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:51.562677+0000","scrub_duration":0.001795643,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"78'100","reported_seq":736,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.823140+0000","last_change":"2024-09-21T10:33:26.770227+0000","last_active":"2024-09-21T10:33:39.823140+0000","last_peered":"2024-09-21T10:33:39.823140+0000","last_clean":"2024-09-21T10:33:39.823140+0000","last_became_active":"2024-09-21T10:33:26.769475+0000","last_became_peered":"2024-09-21T10:33:26.769475+0000","last_unstale":"2024-09-21T10:33:39.823140+0000","last_undegraded":"2024-09-21T10:33:39.823140+0000","last_fullsized":"2024-09-21T10:33:39.823140+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:00:40.325730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031410200000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":861,"num_read_kb":1179,"num_write":85,"num_write_kb":236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":116,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.813699+0000","last_change":"2024-09-21T10:24:44.299492+0000","last_active":"2024-09-21T10:33:39.813699+0000","last_peered":"2024-09-21T10:33:39.813699+0000","last_clean":"2024-09-21T10:33:39.813699+0000","last_became_active":"2024-09-21T10:24:44.299187+0000","last_became_peered":"2024-09-21T10:24:44.299187+0000","last_unstale":"2024-09-21T10:33:39.813699+0000","last_undegraded":"2024-09-21T10:33:39.813699+0000","last_fullsized":"2024-09-21T10:33:39.813699+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:58.500386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":59,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.018932+0000","last_change":"2024-09-21T10:32:37.083724+0000","last_active":"2024-09-21T10:33:41.018932+0000","last_peered":"2024-09-21T10:33:41.018932+0000","last_clean":"2024-09-21T10:33:41.018932+0000","last_became_active":"2024-09-21T10:32:37.082902+0000","last_became_peered":"2024-09-21T10:32:37.082902+0000","last_unstale":"2024-09-21T10:33:41.018932+0000","last_undegraded":"2024-09-21T10:33:41.018932+0000","last_fullsized":"2024-09-21T10:33:41.018932+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:20:49.762214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"82'100","reported_seq":291,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822333+0000","last_change":"2024-09-21T10:28:00.294997+0000","last_active":"2024-09-21T10:33:39.822333+0000","last_peered":"2024-09-21T10:33:39.822333+0000","last_clean":"2024-09-21T10:33:39.822333+0000","last_became_active":"2024-09-21T10:28:00.294595+0000","last_became_peered":"2024-09-21T10:28:00.294595+0000","last_unstale":"2024-09-21T10:33:39.822333+0000","last_undegraded":"2024-09-21T10:33:39.822333+0000","last_fullsized":"2024-09-21T10:33:39.822333+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:30.569982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:30.569982+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:01.422939+0000","scrub_duration":0.0051309349999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":224,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822911+0000","last_change":"2024-09-21T10:28:00.294966+0000","last_active":"2024-09-21T10:33:39.822911+0000","last_peered":"2024-09-21T10:33:39.822911+0000","last_clean":"2024-09-21T10:33:39.822911+0000","last_became_active":"2024-09-21T10:28:00.294552+0000","last_became_peered":"2024-09-21T10:28:00.294552+0000","last_unstale":"2024-09-21T10:33:39.822911+0000","last_undegraded":"2024-09-21T10:33:39.822911+0000","last_fullsized":"2024-09-21T10:33:39.822911+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:47.292125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"78'187","reported_seq":886,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.018870+0000","last_change":"2024-09-21T10:32:40.086133+0000","last_active":"2024-09-21T10:33:41.018870+0000","last_peered":"2024-09-21T10:33:41.018870+0000","last_clean":"2024-09-21T10:33:41.018870+0000","last_became_active":"2024-09-21T10:32:40.085561+0000","last_became_peered":"2024-09-21T10:32:40.085561+0000","last_unstale":"2024-09-21T10:33:41.018870+0000","last_undegraded":"2024-09-21T10:33:41.018870+0000","last_fullsized":"2024-09-21T10:33:41.018870+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:54.245378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036544700000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1141,"num_read_kb":2035,"num_write":188,"num_write_kb":1135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":59,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.018668+0000","last_change":"2024-09-21T10:32:37.081781+0000","last_active":"2024-09-21T10:33:41.018668+0000","last_peered":"2024-09-21T10:33:41.018668+0000","last_clean":"2024-09-21T10:33:41.018668+0000","last_became_active":"2024-09-21T10:32:37.081499+0000","last_became_peered":"2024-09-21T10:32:37.081499+0000","last_unstale":"2024-09-21T10:33:41.018668+0000","last_undegraded":"2024-09-21T10:33:41.018668+0000","last_fullsized":"2024-09-21T10:33:41.018668+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:23.388943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'100","reported_seq":301,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.821561+0000","last_change":"2024-09-21T10:28:00.295277+0000","last_active":"2024-09-21T10:33:39.821561+0000","last_peered":"2024-09-21T10:33:39.821561+0000","last_clean":"2024-09-21T10:33:39.821561+0000","last_became_active":"2024-09-21T10:28:00.294016+0000","last_became_peered":"2024-09-21T10:28:00.294016+0000","last_unstale":"2024-09-21T10:33:39.821561+0000","last_undegraded":"2024-09-21T10:33:39.821561+0000","last_fullsized":"2024-09-21T10:33:39.821561+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:24.536711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:24.536711+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:28:49.984470+0000","scrub_duration":0.0047525650000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":59,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.018835+0000","last_change":"2024-09-21T10:32:37.077779+0000","last_active":"2024-09-21T10:33:41.018835+0000","last_peered":"2024-09-21T10:33:41.018835+0000","last_clean":"2024-09-21T10:33:41.018835+0000","last_became_active":"2024-09-21T10:32:37.076786+0000","last_became_peered":"2024-09-21T10:32:37.076786+0000","last_unstale":"2024-09-21T10:33:41.018835+0000","last_undegraded":"2024-09-21T10:33:41.018835+0000","last_fullsized":"2024-09-21T10:33:41.018835+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:53:45.261483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'81","reported_seq":849,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.814414+0000","last_change":"2024-09-21T10:24:44.334847+0000","last_active":"2024-09-21T10:33:39.814414+0000","last_peered":"2024-09-21T10:33:39.814414+0000","last_clean":"2024-09-21T10:33:39.814414+0000","last_became_active":"2024-09-21T10:24:44.301321+0000","last_became_peered":"2024-09-21T10:24:44.301321+0000","last_unstale":"2024-09-21T10:33:39.814414+0000","last_undegraded":"2024-09-21T10:33:39.814414+0000","last_fullsized":"2024-09-21T10:33:39.814414+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:37.628940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025421599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":640,"num_read_kb":1556,"num_write":71,"num_write_kb":1069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":607,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.424527+0000","last_change":"2024-09-21T10:32:54.862663+0000","last_active":"2024-09-21T10:33:41.424527+0000","last_peered":"2024-09-21T10:33:41.424527+0000","last_clean":"2024-09-21T10:33:41.424527+0000","last_became_active":"2024-09-21T10:32:54.862435+0000","last_became_peered":"2024-09-21T10:32:54.862435+0000","last_unstale":"2024-09-21T10:33:41.424527+0000","last_undegraded":"2024-09-21T10:33:41.424527+0000","last_fullsized":"2024-09-21T10:33:41.424527+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:30.412498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"82'100","reported_seq":116,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.814388+0000","last_change":"2024-09-21T10:24:44.296758+0000","last_active":"2024-09-21T10:33:39.814388+0000","last_peered":"2024-09-21T10:33:39.814388+0000","last_clean":"2024-09-21T10:33:39.814388+0000","last_became_active":"2024-09-21T10:24:44.296403+0000","last_became_peered":"2024-09-21T10:24:44.296403+0000","last_unstale":"2024-09-21T10:33:39.814388+0000","last_undegraded":"2024-09-21T10:33:39.814388+0000","last_fullsized":"2024-09-21T10:33:39.814388+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:23:21.105746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:23:21.105746+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:20:12.502081+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":149,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814877+0000","last_change":"2024-09-21T10:28:00.300952+0000","last_active":"2024-09-21T10:33:40.814877+0000","last_peered":"2024-09-21T10:33:40.814877+0000","last_clean":"2024-09-21T10:33:40.814877+0000","last_became_active":"2024-09-21T10:28:00.293337+0000","last_became_peered":"2024-09-21T10:28:00.293337+0000","last_unstale":"2024-09-21T10:33:40.814877+0000","last_undegraded":"2024-09-21T10:33:40.814877+0000","last_fullsized":"2024-09-21T10:33:40.814877+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:58.409761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"78'126","reported_seq":633,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.019064+0000","last_change":"2024-09-21T10:32:40.087217+0000","last_active":"2024-09-21T10:33:41.019064+0000","last_peered":"2024-09-21T10:33:41.019064+0000","last_clean":"2024-09-21T10:33:41.019064+0000","last_became_active":"2024-09-21T10:32:40.086869+0000","last_became_peered":"2024-09-21T10:32:40.086869+0000","last_unstale":"2024-09-21T10:33:41.019064+0000","last_undegraded":"2024-09-21T10:33:41.019064+0000","last_fullsized":"2024-09-21T10:33:41.019064+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:11:40.134565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047287799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":403,"num_write":94,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":146,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814283+0000","last_change":"2024-09-21T10:28:00.293864+0000","last_active":"2024-09-21T10:33:40.814283+0000","last_peered":"2024-09-21T10:33:40.814283+0000","last_clean":"2024-09-21T10:33:40.814283+0000","last_became_active":"2024-09-21T10:28:00.293494+0000","last_became_peered":"2024-09-21T10:28:00.293494+0000","last_unstale":"2024-09-21T10:33:40.814283+0000","last_undegraded":"2024-09-21T10:33:40.814283+0000","last_fullsized":"2024-09-21T10:33:40.814283+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:37.098068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"82'100","reported_seq":162,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814909+0000","last_change":"2024-09-21T10:32:43.908359+0000","last_active":"2024-09-21T10:33:40.814909+0000","last_peered":"2024-09-21T10:33:40.814909+0000","last_clean":"2024-09-21T10:33:40.814909+0000","last_became_active":"2024-09-21T10:28:00.290981+0000","last_became_peered":"2024-09-21T10:28:00.290981+0000","last_unstale":"2024-09-21T10:33:40.814909+0000","last_undegraded":"2024-09-21T10:33:40.814909+0000","last_fullsized":"2024-09-21T10:33:40.814909+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'100","last_scrub_stamp":"2024-09-21T10:32:43.908293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:48:34.850872+0000","last_clean_scrub_stamp":"2024-09-21T10:32:43.908293+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:31:13.366375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":799,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.424311+0000","last_change":"2024-09-21T10:32:37.089513+0000","last_active":"2024-09-21T10:33:41.424311+0000","last_peered":"2024-09-21T10:33:41.424311+0000","last_clean":"2024-09-21T10:33:41.424311+0000","last_became_active":"2024-09-21T10:32:37.080136+0000","last_became_peered":"2024-09-21T10:32:37.080136+0000","last_unstale":"2024-09-21T10:33:41.424311+0000","last_undegraded":"2024-09-21T10:33:41.424311+0000","last_fullsized":"2024-09-21T10:33:41.424311+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:35:08.980579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"78'46","reported_seq":653,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822500+0000","last_change":"2024-09-21T10:01:41.361079+0000","last_active":"2024-09-21T10:33:39.822500+0000","last_peered":"2024-09-21T10:33:39.822500+0000","last_clean":"2024-09-21T10:33:39.822500+0000","last_became_active":"2024-09-21T10:01:41.360706+0000","last_became_peered":"2024-09-21T10:01:41.360706+0000","last_unstale":"2024-09-21T10:33:39.822500+0000","last_undegraded":"2024-09-21T10:33:39.822500+0000","last_fullsized":"2024-09-21T10:33:39.822500+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:51:04.120483+0000","last_clean_scrub_stamp":"2024-09-21T09:51:04.120483+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.714378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028748599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":443,"num_read_kb":1850,"num_write":28,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":116,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.813077+0000","last_change":"2024-09-21T10:24:44.303781+0000","last_active":"2024-09-21T10:33:39.813077+0000","last_peered":"2024-09-21T10:33:39.813077+0000","last_clean":"2024-09-21T10:33:39.813077+0000","last_became_active":"2024-09-21T10:24:44.302058+0000","last_became_peered":"2024-09-21T10:24:44.302058+0000","last_unstale":"2024-09-21T10:33:39.813077+0000","last_undegraded":"2024-09-21T10:33:39.813077+0000","last_fullsized":"2024-09-21T10:33:39.813077+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:30.371229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":136,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.813768+0000","last_change":"2024-09-21T10:32:37.074825+0000","last_active":"2024-09-21T10:33:39.813768+0000","last_peered":"2024-09-21T10:33:39.813768+0000","last_clean":"2024-09-21T10:33:39.813768+0000","last_became_active":"2024-09-21T10:32:37.074496+0000","last_became_peered":"2024-09-21T10:32:37.074496+0000","last_unstale":"2024-09-21T10:33:39.813768+0000","last_undegraded":"2024-09-21T10:33:39.813768+0000","last_fullsized":"2024-09-21T10:33:39.813768+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:07.169160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":126,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.812878+0000","last_change":"2024-09-21T10:24:44.321421+0000","last_active":"2024-09-21T10:33:39.812878+0000","last_peered":"2024-09-21T10:33:39.812878+0000","last_clean":"2024-09-21T10:33:39.812878+0000","last_became_active":"2024-09-21T10:24:44.297082+0000","last_became_peered":"2024-09-21T10:24:44.297082+0000","last_unstale":"2024-09-21T10:33:39.812878+0000","last_undegraded":"2024-09-21T10:33:39.812878+0000","last_fullsized":"2024-09-21T10:33:39.812878+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:35:37.704881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":59,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.018613+0000","last_change":"2024-09-21T10:32:37.076103+0000","last_active":"2024-09-21T10:33:41.018613+0000","last_peered":"2024-09-21T10:33:41.018613+0000","last_clean":"2024-09-21T10:33:41.018613+0000","last_became_active":"2024-09-21T10:32:37.075491+0000","last_became_peered":"2024-09-21T10:32:37.075491+0000","last_unstale":"2024-09-21T10:33:41.018613+0000","last_undegraded":"2024-09-21T10:33:41.018613+0000","last_fullsized":"2024-09-21T10:33:41.018613+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:28:02.453017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":490,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.814180+0000","last_change":"2024-09-21T10:28:00.296771+0000","last_active":"2024-09-21T10:33:39.814180+0000","last_peered":"2024-09-21T10:33:39.814180+0000","last_clean":"2024-09-21T10:33:39.814180+0000","last_became_active":"2024-09-21T10:28:00.285336+0000","last_became_peered":"2024-09-21T10:28:00.285336+0000","last_unstale":"2024-09-21T10:33:39.814180+0000","last_undegraded":"2024-09-21T10:33:39.814180+0000","last_fullsized":"2024-09-21T10:33:39.814180+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:06:50.848492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":217,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822401+0000","last_change":"2024-09-21T10:32:37.078245+0000","last_active":"2024-09-21T10:33:39.822401+0000","last_peered":"2024-09-21T10:33:39.822401+0000","last_clean":"2024-09-21T10:33:39.822401+0000","last_became_active":"2024-09-21T10:32:37.077852+0000","last_became_peered":"2024-09-21T10:32:37.077852+0000","last_unstale":"2024-09-21T10:33:39.822401+0000","last_undegraded":"2024-09-21T10:33:39.822401+0000","last_fullsized":"2024-09-21T10:33:39.822401+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:11.370577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":145,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814445+0000","last_change":"2024-09-21T10:28:00.294338+0000","last_active":"2024-09-21T10:33:40.814445+0000","last_peered":"2024-09-21T10:33:40.814445+0000","last_clean":"2024-09-21T10:33:40.814445+0000","last_became_active":"2024-09-21T10:28:00.294044+0000","last_became_peered":"2024-09-21T10:28:00.294044+0000","last_unstale":"2024-09-21T10:33:40.814445+0000","last_undegraded":"2024-09-21T10:33:40.814445+0000","last_fullsized":"2024-09-21T10:33:40.814445+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:43:30.999988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":742,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.424453+0000","last_change":"2024-09-21T10:28:00.296935+0000","last_active":"2024-09-21T10:33:41.424453+0000","last_peered":"2024-09-21T10:33:41.424453+0000","last_clean":"2024-09-21T10:33:41.424453+0000","last_became_active":"2024-09-21T10:28:00.296562+0000","last_became_peered":"2024-09-21T10:28:00.296562+0000","last_unstale":"2024-09-21T10:33:41.424453+0000","last_undegraded":"2024-09-21T10:33:41.424453+0000","last_fullsized":"2024-09-21T10:33:41.424453+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:06.863310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":166,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.164508+0000","last_change":"2024-09-21T10:33:41.164439+0000","last_active":"2024-09-21T10:33:41.164508+0000","last_peered":"2024-09-21T10:33:41.164508+0000","last_clean":"2024-09-21T10:33:41.164508+0000","last_became_active":"2024-09-21T10:33:41.164178+0000","last_became_peered":"2024-09-21T10:33:41.164178+0000","last_unstale":"2024-09-21T10:33:41.164508+0000","last_undegraded":"2024-09-21T10:33:41.164508+0000","last_fullsized":"2024-09-21T10:33:41.164508+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:21.525842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":410,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822414+0000","last_change":"2024-09-21T10:32:37.073911+0000","last_active":"2024-09-21T10:33:39.822414+0000","last_peered":"2024-09-21T10:33:39.822414+0000","last_clean":"2024-09-21T10:33:39.822414+0000","last_became_active":"2024-09-21T10:32:37.073507+0000","last_became_peered":"2024-09-21T10:32:37.073507+0000","last_unstale":"2024-09-21T10:33:39.822414+0000","last_undegraded":"2024-09-21T10:33:39.822414+0000","last_fullsized":"2024-09-21T10:33:39.822414+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:52.244604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":159,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814034+0000","last_change":"2024-09-21T10:32:37.079936+0000","last_active":"2024-09-21T10:33:40.814034+0000","last_peered":"2024-09-21T10:33:40.814034+0000","last_clean":"2024-09-21T10:33:40.814034+0000","last_became_active":"2024-09-21T10:32:37.079625+0000","last_became_peered":"2024-09-21T10:32:37.079625+0000","last_unstale":"2024-09-21T10:33:40.814034+0000","last_undegraded":"2024-09-21T10:33:40.814034+0000","last_fullsized":"2024-09-21T10:33:40.814034+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:43:45.397882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":759,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.424690+0000","last_change":"2024-09-21T10:28:00.297304+0000","last_active":"2024-09-21T10:33:41.424690+0000","last_peered":"2024-09-21T10:33:41.424690+0000","last_clean":"2024-09-21T10:33:41.424690+0000","last_became_active":"2024-09-21T10:28:00.296631+0000","last_became_peered":"2024-09-21T10:28:00.296631+0000","last_unstale":"2024-09-21T10:33:41.424690+0000","last_undegraded":"2024-09-21T10:33:41.424690+0000","last_fullsized":"2024-09-21T10:33:41.424690+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:22:16.633581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":59,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.018528+0000","last_change":"2024-09-21T10:32:37.083327+0000","last_active":"2024-09-21T10:33:41.018528+0000","last_peered":"2024-09-21T10:33:41.018528+0000","last_clean":"2024-09-21T10:33:41.018528+0000","last_became_active":"2024-09-21T10:32:37.082725+0000","last_became_peered":"2024-09-21T10:32:37.082725+0000","last_unstale":"2024-09-21T10:33:41.018528+0000","last_undegraded":"2024-09-21T10:33:41.018528+0000","last_fullsized":"2024-09-21T10:33:41.018528+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:22.604264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":159,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814677+0000","last_change":"2024-09-21T10:32:37.080087+0000","last_active":"2024-09-21T10:33:40.814677+0000","last_peered":"2024-09-21T10:33:40.814677+0000","last_clean":"2024-09-21T10:33:40.814677+0000","last_became_active":"2024-09-21T10:32:37.079766+0000","last_became_peered":"2024-09-21T10:32:37.079766+0000","last_unstale":"2024-09-21T10:33:40.814677+0000","last_undegraded":"2024-09-21T10:33:40.814677+0000","last_fullsized":"2024-09-21T10:33:40.814677+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:13:29.691366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":136,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.813154+0000","last_change":"2024-09-21T10:32:37.075224+0000","last_active":"2024-09-21T10:33:39.813154+0000","last_peered":"2024-09-21T10:33:39.813154+0000","last_clean":"2024-09-21T10:33:39.813154+0000","last_became_active":"2024-09-21T10:32:37.074655+0000","last_became_peered":"2024-09-21T10:32:37.074655+0000","last_unstale":"2024-09-21T10:33:39.813154+0000","last_undegraded":"2024-09-21T10:33:39.813154+0000","last_fullsized":"2024-09-21T10:33:39.813154+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:46.280140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":410,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.854979+0000","last_change":"2024-09-21T10:32:37.074510+0000","last_active":"2024-09-21T10:33:40.854979+0000","last_peered":"2024-09-21T10:33:40.854979+0000","last_clean":"2024-09-21T10:33:40.854979+0000","last_became_active":"2024-09-21T10:32:37.074070+0000","last_became_peered":"2024-09-21T10:32:37.074070+0000","last_unstale":"2024-09-21T10:33:40.854979+0000","last_undegraded":"2024-09-21T10:33:40.854979+0000","last_fullsized":"2024-09-21T10:33:40.854979+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:26.357339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":748,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.424506+0000","last_change":"2024-09-21T10:24:44.299989+0000","last_active":"2024-09-21T10:33:41.424506+0000","last_peered":"2024-09-21T10:33:41.424506+0000","last_clean":"2024-09-21T10:33:41.424506+0000","last_became_active":"2024-09-21T10:24:44.299356+0000","last_became_peered":"2024-09-21T10:24:44.299356+0000","last_unstale":"2024-09-21T10:33:41.424506+0000","last_undegraded":"2024-09-21T10:33:41.424506+0000","last_fullsized":"2024-09-21T10:33:41.424506+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:58.867008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":146,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.813572+0000","last_change":"2024-09-21T10:32:37.075907+0000","last_active":"2024-09-21T10:33:39.813572+0000","last_peered":"2024-09-21T10:33:39.813572+0000","last_clean":"2024-09-21T10:33:39.813572+0000","last_became_active":"2024-09-21T10:32:37.075581+0000","last_became_peered":"2024-09-21T10:32:37.075581+0000","last_unstale":"2024-09-21T10:33:39.813572+0000","last_undegraded":"2024-09-21T10:33:39.813572+0000","last_fullsized":"2024-09-21T10:33:39.813572+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:53.334817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":158,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822550+0000","last_change":"2024-09-21T10:33:26.770588+0000","last_active":"2024-09-21T10:33:39.822550+0000","last_peered":"2024-09-21T10:33:39.822550+0000","last_clean":"2024-09-21T10:33:39.822550+0000","last_became_active":"2024-09-21T10:33:26.769758+0000","last_became_peered":"2024-09-21T10:33:26.769758+0000","last_unstale":"2024-09-21T10:33:39.822550+0000","last_undegraded":"2024-09-21T10:33:39.822550+0000","last_fullsized":"2024-09-21T10:33:39.822550+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:59.264432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":126,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.813221+0000","last_change":"2024-09-21T10:28:00.286009+0000","last_active":"2024-09-21T10:33:39.813221+0000","last_peered":"2024-09-21T10:33:39.813221+0000","last_clean":"2024-09-21T10:33:39.813221+0000","last_became_active":"2024-09-21T10:28:00.285327+0000","last_became_peered":"2024-09-21T10:28:00.285327+0000","last_unstale":"2024-09-21T10:33:39.813221+0000","last_undegraded":"2024-09-21T10:33:39.813221+0000","last_fullsized":"2024-09-21T10:33:39.813221+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:34.985354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":141,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.822783+0000","last_change":"2024-09-21T10:33:26.769225+0000","last_active":"2024-09-21T10:33:39.822783+0000","last_peered":"2024-09-21T10:33:39.822783+0000","last_clean":"2024-09-21T10:33:39.822783+0000","last_became_active":"2024-09-21T10:33:26.768810+0000","last_became_peered":"2024-09-21T10:33:26.768810+0000","last_unstale":"2024-09-21T10:33:39.822783+0000","last_undegraded":"2024-09-21T10:33:39.822783+0000","last_fullsized":"2024-09-21T10:33:39.822783+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:19:53.205774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":145,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814438+0000","last_change":"2024-09-21T10:28:00.293054+0000","last_active":"2024-09-21T10:33:40.814438+0000","last_peered":"2024-09-21T10:33:40.814438+0000","last_clean":"2024-09-21T10:33:40.814438+0000","last_became_active":"2024-09-21T10:28:00.292723+0000","last_became_peered":"2024-09-21T10:28:00.292723+0000","last_unstale":"2024-09-21T10:33:40.814438+0000","last_undegraded":"2024-09-21T10:33:40.814438+0000","last_fullsized":"2024-09-21T10:33:40.814438+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:01:45.933570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":146,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:40.814868+0000","last_change":"2024-09-21T10:28:00.285515+0000","last_active":"2024-09-21T10:33:40.814868+0000","last_peered":"2024-09-21T10:33:40.814868+0000","last_clean":"2024-09-21T10:33:40.814868+0000","last_became_active":"2024-09-21T10:28:00.285203+0000","last_became_peered":"2024-09-21T10:28:00.285203+0000","last_unstale":"2024-09-21T10:33:40.814868+0000","last_undegraded":"2024-09-21T10:33:40.814868+0000","last_fullsized":"2024-09-21T10:33:40.814868+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:41.982228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":136,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.812764+0000","last_change":"2024-09-21T10:32:37.077513+0000","last_active":"2024-09-21T10:33:39.812764+0000","last_peered":"2024-09-21T10:33:39.812764+0000","last_clean":"2024-09-21T10:33:39.812764+0000","last_became_active":"2024-09-21T10:32:37.077161+0000","last_became_peered":"2024-09-21T10:32:37.077161+0000","last_unstale":"2024-09-21T10:33:39.812764+0000","last_undegraded":"2024-09-21T10:33:39.812764+0000","last_fullsized":"2024-09-21T10:33:39.812764+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:20.350159+0000","last_clean_scrub_stamp":"2024-09-21T09:52:20.350159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:33.823372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":734,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T10:33:41.424413+0000","last_change":"2024-09-21T10:01:41.361426+0000","last_active":"2024-09-21T10:33:41.424413+0000","last_peered":"2024-09-21T10:33:41.424413+0000","last_clean":"2024-09-21T10:33:41.424413+0000","last_became_active":"2024-09-21T10:01:41.360922+0000","last_became_peered":"2024-09-21T10:01:41.360922+0000","last_unstale":"2024-09-21T10:33:41.424413+0000","last_undegraded":"2024-09-21T10:33:41.424413+0000","last_fullsized":"2024-09-21T10:33:41.424413+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:52:19.342687+0000","last_clean_scrub_stamp":"2024-09-21T09:52:19.342687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:45.206286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":10252,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18419,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5983,"num_read_kb":13006,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1100,"ondisk_log_size":1100,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453398037,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1041356,"kb_used_data":144132,"kb_used_omap":1,"kb_used_meta":897214,"kb_avail":92703796,"statfs":{"total":95995035648,"available":94928687104,"internally_reserved":0,"allocated":147591168,"data_stored":145164326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918747480},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594265,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":906956,"kb_used_data":5628,"kb_used_omap":1,"kb_used_meta":901310,"kb_avail":92838196,"statfs":{"total":95995035648,"available":95066312704,"internally_reserved":0,"allocated":5763072,"data_stored":3344295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922941784},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790495,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":914368,"kb_used_data":10292,"kb_used_omap":1,"kb_used_meta":904062,"kb_avail":92830784,"statfs":{"total":95995035648,"available":95058722816,"internally_reserved":0,"allocated":10539008,"data_stored":8124478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":925759832},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986724,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":738248,"kb_used_data":148352,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93006904,"statfs":{"total":95995035648,"available":95239069696,"internally_reserved":0,"allocated":151912448,"data_stored":149489624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":101,"seq":433791697286,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":696184,"kb_used_data":6192,"kb_used_omap":5,"kb_used_meta":689978,"kb_avail":93048968,"statfs":{"total":95995035648,"available":95282143232,"internally_reserved":0,"allocated":6340608,"data_stored":3929932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5360,"internal_metadata":706538256},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":96,"seq":412316860813,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1037508,"kb_used_data":9972,"kb_used_omap":8,"kb_used_meta":1027511,"kb_avail":92707644,"statfs":{"total":95995035648,"available":94932627456,"internally_reserved":0,"allocated":10211328,"data_stored":7797078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8340,"internal_metadata":1052172140},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":91,"seq":390842024338,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":984620,"kb_used_data":134884,"kb_used_omap":7,"kb_used_meta":849720,"kb_avail":92760532,"statfs":{"total":95995035648,"available":94986784768,"internally_reserved":0,"allocated":138121216,"data_stored":135721591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7630,"internal_metadata":870113842},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":86,"seq":369367187873,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1066300,"kb_used_data":5616,"kb_used_omap":8,"kb_used_meta":1060663,"kb_avail":92678852,"statfs":{"total":95995035648,"available":94903144448,"internally_reserved":0,"allocated":5750784,"data_stored":3340074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8207,"internal_metadata":1086119921},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4302,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6503,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5926,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2334,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:34:30.888 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T10:34:30.889 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rbd.sh) on smithi098 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2024-09-21T10:34:31.052 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3bf7d22f47f24f0db0d4c4d748368a11 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rbd.sh) on smithi098 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2024-09-21T10:34:31.054 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-21T10:34:31.064 INFO:tasks.cephadm:Teardown begin 2024-09-21T10:34:31.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T10:34:31.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:30 smithi098 ceph-mon[97580]: pgmap v1484: 83 pgs: 83 active+clean; 4.6 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:31.093 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:30 smithi098 ceph-mon[97580]: from='client.36978 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:34:31.093 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:34:30 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: ::ffff:172.21.15.196 - - [21/Sep/2024:10:34:30] "GET /metrics HTTP/1.1" 200 34162 "" "Prometheus/2.51.0" 2024-09-21T10:34:31.094 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:30 smithi098 ceph-mon[100914]: pgmap v1484: 83 pgs: 83 active+clean; 4.6 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:31.094 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:30 smithi098 ceph-mon[100914]: from='client.36978 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:34:31.112 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T10:34:31.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:30 smithi196 ceph-mon[84784]: pgmap v1484: 83 pgs: 83 active+clean; 4.6 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:34:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:30 smithi196 ceph-mon[84784]: from='client.36978 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:34:31.163 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-21T10:34:31.164 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T10:34:31.179 DEBUG:teuthology.orchestra.run.smithi196:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T10:34:31.222 INFO:tasks.cephadm:Stopping all daemons... 2024-09-21T10:34:31.222 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-21T10:34:31.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-26a15512-77fe-11ef-baf6-efdc52797490@mon.a 2024-09-21T10:34:31.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:31 smithi098 systemd[1]: Stopping Ceph mon.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:34:31.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:31 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a[97576]: 2024-09-21T10:34:31.551+0000 7f655ba04640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:34:31.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:31 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a[97576]: 2024-09-21T10:34:31.551+0000 7f655ba04640 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-09-21T10:34:31.848 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:31 smithi098 podman[327280]: 2024-09-21 10:34:31.787053072 +0000 UTC m=+0.376289128 container died 0a2195915f653b39d02facf97187ed42b1411801de2b8544e408d25b7814e465 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:34:31.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:34:31 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:10:34:31] ENGINE Bus STOPPING 2024-09-21T10:34:31.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:34:31 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:10:34:31] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T10:34:31.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:34:31 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:10:34:31] ENGINE Bus STOPPED 2024-09-21T10:34:31.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:34:31 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:10:34:31] ENGINE Bus STARTING 2024-09-21T10:34:31.848 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:34:31 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:10:34:31] ENGINE Serving on http://:::9283 2024-09-21T10:34:31.849 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:34:31 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: [21/Sep/2024:10:34:31] ENGINE Bus STARTED 2024-09-21T10:34:32.590 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:32 smithi098 podman[327280]: 2024-09-21 10:34:32.286427922 +0000 UTC m=+0.875663974 container cleanup 0a2195915f653b39d02facf97187ed42b1411801de2b8544e408d25b7814e465 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, CEPH_POINT_RELEASE=) 2024-09-21T10:34:32.590 INFO:journalctl@ceph.mon.a.smithi098.stdout:Sep 21 10:34:32 smithi098 bash[327280]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-a 2024-09-21T10:34:32.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@mon.a.service' 2024-09-21T10:34:32.768 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:34:32.768 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-21T10:34:32.768 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-21T10:34:32.768 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-26a15512-77fe-11ef-baf6-efdc52797490@mon.c 2024-09-21T10:34:32.847 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:32 smithi098 systemd[1]: Stopping Ceph mon.c for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:34:33.347 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:33 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-c[100910]: 2024-09-21T10:34:33.058+0000 7f246711c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:34:33.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:33 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-c[100910]: 2024-09-21T10:34:33.058+0000 7f246711c640 -1 mon.c@2(peon) e4 *** Got Signal Terminated *** 2024-09-21T10:34:33.348 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:33 smithi098 podman[327410]: 2024-09-21 10:34:33.171429228 +0000 UTC m=+0.216409300 container died ef4452ba8cf09dd0181ce0bcb93eea8c284f12f160a55987aa8a7b3826c8dd5d (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-c, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-21T10:34:33.845 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:33 smithi098 podman[327410]: 2024-09-21 10:34:33.550761479 +0000 UTC m=+0.595741547 container cleanup ef4452ba8cf09dd0181ce0bcb93eea8c284f12f160a55987aa8a7b3826c8dd5d (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-c, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-21T10:34:33.845 INFO:journalctl@ceph.mon.c.smithi098.stdout:Sep 21 10:34:33 smithi098 bash[327410]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-c 2024-09-21T10:34:33.972 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@mon.c.service' 2024-09-21T10:34:34.005 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:34:34.005 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-21T10:34:34.006 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-21T10:34:34.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-26a15512-77fe-11ef-baf6-efdc52797490@mon.b 2024-09-21T10:34:34.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:34 smithi196 systemd[1]: Stopping Ceph mon.b for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:34:34.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:34 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-b[84780]: 2024-09-21T10:34:34.311+0000 7f637f904640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:34:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:34 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-b[84780]: 2024-09-21T10:34:34.311+0000 7f637f904640 -1 mon.b@1(peon) e4 *** Got Signal Terminated *** 2024-09-21T10:34:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:34 smithi196 podman[226702]: 2024-09-21 10:34:34.548385533 +0000 UTC m=+0.332440041 container died 2776b693f5c027514541e559e3dbc2a7a04cb3bc9f46292910262e9598772bd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-b, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-09-21T10:34:35.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:35 smithi196 podman[226702]: 2024-09-21 10:34:35.114060481 +0000 UTC m=+0.898115025 container cleanup 2776b693f5c027514541e559e3dbc2a7a04cb3bc9f46292910262e9598772bd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-21T10:34:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 10:34:35 smithi196 bash[226702]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-mon-b 2024-09-21T10:34:35.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@mon.b.service' 2024-09-21T10:34:35.627 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:34:35.627 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-21T10:34:35.627 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-21T10:34:35.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-26a15512-77fe-11ef-baf6-efdc52797490@mgr.y 2024-09-21T10:34:35.924 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:34:35 smithi098 systemd[1]: Stopping Ceph mgr.y for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:34:36.267 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:34:35 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y[88523]: 2024-09-21T10:34:35.921+0000 7f6c95f20640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:34:36.267 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:34:35 smithi098 podman[327534]: 2024-09-21 10:34:35.963505187 +0000 UTC m=+0.146476291 container died da66347fe5a2e6fa8228f978c8e137dc7d89510aa3e4454193e5aa2dba1f7d49 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:34:36.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:34:36 smithi098 podman[327534]: 2024-09-21 10:34:36.456553623 +0000 UTC m=+0.639524769 container cleanup da66347fe5a2e6fa8228f978c8e137dc7d89510aa3e4454193e5aa2dba1f7d49 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y, GIT_BRANCH=HEAD, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-21T10:34:36.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Sep 21 10:34:36 smithi098 bash[327534]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-y 2024-09-21T10:34:36.882 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@mgr.y.service' 2024-09-21T10:34:36.916 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:34:36.917 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-21T10:34:36.917 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-21T10:34:36.917 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-26a15512-77fe-11ef-baf6-efdc52797490@mgr.x 2024-09-21T10:34:37.259 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 10:34:36 smithi196 systemd[1]: Stopping Ceph mgr.x for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:34:37.537 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 10:34:37 smithi196 podman[226827]: 2024-09-21 10:34:37.257595629 +0000 UTC m=+0.140178995 container died d4969d635bcd65334e4e29736a0a044e98c46ee74eabca0cc2298ab14ae641bc (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, RELEASE=main-62da54b, org.label-schema.schema-version=1.0) 2024-09-21T10:34:37.889 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 10:34:37 smithi196 podman[226827]: 2024-09-21 10:34:37.72696553 +0000 UTC m=+0.609548885 container cleanup d4969d635bcd65334e4e29736a0a044e98c46ee74eabca0cc2298ab14ae641bc (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-21T10:34:37.889 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 21 10:34:37 smithi196 bash[226827]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-mgr-x 2024-09-21T10:34:38.167 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@mgr.x.service' 2024-09-21T10:34:38.201 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:34:38.201 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-21T10:34:38.202 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-21T10:34:38.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.0 2024-09-21T10:34:38.382 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 10:34:38 smithi098 systemd[1]: Stopping Ceph osd.0 for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:34:38.847 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 10:34:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0[107900]: 2024-09-21T10:34:38.501+0000 7fb642ef7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:34:38.847 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 10:34:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0[107900]: 2024-09-21T10:34:38.501+0000 7fb642ef7640 -1 osd.0 427 *** Got signal Terminated *** 2024-09-21T10:34:38.847 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 10:34:38 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0[107900]: 2024-09-21T10:34:38.502+0000 7fb642ef7640 -1 osd.0 427 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:34:40.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:34:40 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:34:40.211Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"26a15512-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:34:44.449 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 10:34:44 smithi098 podman[327654]: 2024-09-21 10:34:44.109704721 +0000 UTC m=+5.718835765 container died 9c8da4094eee81a38b2328339c2d8d60a0a33c15f3e0d571d5680b262ce64507 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-21T10:34:44.847 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 10:34:44 smithi098 podman[327654]: 2024-09-21 10:34:44.618719673 +0000 UTC m=+6.227850709 container cleanup 9c8da4094eee81a38b2328339c2d8d60a0a33c15f3e0d571d5680b262ce64507 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, io.buildah.version=1.37.2, ceph=True) 2024-09-21T10:34:44.847 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 10:34:44 smithi098 bash[327654]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0 2024-09-21T10:34:45.347 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 10:34:44 smithi098 podman[327681]: 2024-09-21 10:34:44.981167456 +0000 UTC m=+0.869431555 container remove 9c8da4094eee81a38b2328339c2d8d60a0a33c15f3e0d571d5680b262ce64507 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, RELEASE=main-62da54b, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-21T10:34:45.669 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 10:34:45 smithi098 podman[327768]: 2024-09-21 10:34:45.283144689 +0000 UTC m=+0.015670864 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:34:45.670 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 10:34:45 smithi098 podman[327768]: 2024-09-21 10:34:45.390466568 +0000 UTC m=+0.122992780 container create 2e786963023319de476ae5df8308141626e7d8f15d403048e6482ebea5cfcedc (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-21T10:34:46.001 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 10:34:45 smithi098 podman[327768]: 2024-09-21 10:34:45.668463464 +0000 UTC m=+0.400989635 container init 2e786963023319de476ae5df8308141626e7d8f15d403048e6482ebea5cfcedc (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:34:46.001 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 10:34:45 smithi098 podman[327768]: 2024-09-21 10:34:45.672488986 +0000 UTC m=+0.405015155 container start 2e786963023319de476ae5df8308141626e7d8f15d403048e6482ebea5cfcedc (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-21T10:34:46.002 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 10:34:45 smithi098 podman[327768]: 2024-09-21 10:34:45.77658649 +0000 UTC m=+0.509112702 container attach 2e786963023319de476ae5df8308141626e7d8f15d403048e6482ebea5cfcedc (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True) 2024-09-21T10:34:46.002 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 10:34:46 smithi098 podman[327768]: 2024-09-21 10:34:46.000085341 +0000 UTC m=+0.732611524 container died 2e786963023319de476ae5df8308141626e7d8f15d403048e6482ebea5cfcedc (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=main-62da54b, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-21T10:34:46.645 INFO:journalctl@ceph.osd.0.smithi098.stdout:Sep 21 10:34:46 smithi098 podman[327791]: 2024-09-21 10:34:46.630690931 +0000 UTC m=+0.624758111 container remove 2e786963023319de476ae5df8308141626e7d8f15d403048e6482ebea5cfcedc (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-21T10:34:46.648 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.0.service' 2024-09-21T10:34:46.686 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:34:46.686 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-21T10:34:46.686 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-21T10:34:46.687 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.1 2024-09-21T10:34:47.027 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:34:46 smithi098 systemd[1]: Stopping Ceph osd.1 for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:34:47.347 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:34:47 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1[112969]: 2024-09-21T10:34:47.024+0000 7f0efb379640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:34:47.347 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:34:47 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1[112969]: 2024-09-21T10:34:47.024+0000 7f0efb379640 -1 osd.1 427 *** Got signal Terminated *** 2024-09-21T10:34:47.347 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:34:47 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1[112969]: 2024-09-21T10:34:47.024+0000 7f0efb379640 -1 osd.1 427 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:34:52.879 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:34:52 smithi098 podman[327872]: 2024-09-21 10:34:52.614907833 +0000 UTC m=+5.702436921 container died c84cfb0f2fd7a854e6c42a645b8b3b6c4ad3b416379d92322e82c24bc587f4ea (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-09-21T10:34:53.347 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:34:53 smithi098 podman[327872]: 2024-09-21 10:34:53.050446062 +0000 UTC m=+6.137975136 container cleanup c84cfb0f2fd7a854e6c42a645b8b3b6c4ad3b416379d92322e82c24bc587f4ea (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-62da54b, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:34:53.347 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:34:53 smithi098 bash[327872]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1 2024-09-21T10:34:53.796 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:34:53 smithi098 podman[327888]: 2024-09-21 10:34:53.394421676 +0000 UTC m=+0.779269336 container remove c84cfb0f2fd7a854e6c42a645b8b3b6c4ad3b416379d92322e82c24bc587f4ea (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-62da54b, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:34:54.097 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:34:53 smithi098 podman[327976]: 2024-09-21 10:34:53.697039016 +0000 UTC m=+0.015600535 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:34:54.097 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:34:53 smithi098 podman[327976]: 2024-09-21 10:34:53.814737986 +0000 UTC m=+0.133299505 container create f62ee66e66fa9a7484fbba985aee27c2a064af838b78f31f71d2a1ab5104e104 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2) 2024-09-21T10:34:54.360 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:34:54 smithi098 podman[327976]: 2024-09-21 10:34:54.09842688 +0000 UTC m=+0.416988413 container init f62ee66e66fa9a7484fbba985aee27c2a064af838b78f31f71d2a1ab5104e104 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-21T10:34:54.360 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:34:54 smithi098 podman[327976]: 2024-09-21 10:34:54.101875998 +0000 UTC m=+0.420437520 container start f62ee66e66fa9a7484fbba985aee27c2a064af838b78f31f71d2a1ab5104e104 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_BRANCH=HEAD, RELEASE=main-62da54b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-21T10:34:54.361 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:34:54 smithi098 podman[327976]: 2024-09-21 10:34:54.181734952 +0000 UTC m=+0.500296490 container attach f62ee66e66fa9a7484fbba985aee27c2a064af838b78f31f71d2a1ab5104e104 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, io.buildah.version=1.37.2) 2024-09-21T10:34:54.361 INFO:journalctl@ceph.osd.1.smithi098.stdout:Sep 21 10:34:54 smithi098 podman[327976]: 2024-09-21 10:34:54.310632275 +0000 UTC m=+0.629193797 container died f62ee66e66fa9a7484fbba985aee27c2a064af838b78f31f71d2a1ab5104e104 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, RELEASE=main-62da54b, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:34:55.011 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.1.service' 2024-09-21T10:34:55.050 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:34:55.050 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-21T10:34:55.050 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-21T10:34:55.051 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.2 2024-09-21T10:34:55.347 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:34:55 smithi098 systemd[1]: Stopping Ceph osd.2 for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:34:55.847 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:34:55 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2[118221]: 2024-09-21T10:34:55.439+0000 7fa27f179640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:34:55.847 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:34:55 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2[118221]: 2024-09-21T10:34:55.439+0000 7fa27f179640 -1 osd.2 427 *** Got signal Terminated *** 2024-09-21T10:34:55.847 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:34:55 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2[118221]: 2024-09-21T10:34:55.439+0000 7fa27f179640 -1 osd.2 427 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:35:01.216 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:35:00 smithi098 podman[328090]: 2024-09-21 10:35:00.905506112 +0000 UTC m=+5.616949677 container died 47ba7450689130a5ab1b9a26257396d4534bb1fe1977d052cc962639ed7b3df3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:35:01.597 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:35:01 smithi098 podman[328090]: 2024-09-21 10:35:01.41498724 +0000 UTC m=+6.126430863 container cleanup 47ba7450689130a5ab1b9a26257396d4534bb1fe1977d052cc962639ed7b3df3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-21T10:35:01.597 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:35:01 smithi098 bash[328090]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2 2024-09-21T10:35:01.940 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:35:01 smithi098 podman[328110]: 2024-09-21 10:35:01.633172547 +0000 UTC m=+0.727866922 container remove 47ba7450689130a5ab1b9a26257396d4534bb1fe1977d052cc962639ed7b3df3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2, RELEASE=main-62da54b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-21T10:35:02.213 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:35:01 smithi098 podman[328197]: 2024-09-21 10:35:01.841735913 +0000 UTC m=+0.015571659 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:35:02.214 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:35:01 smithi098 podman[328197]: 2024-09-21 10:35:01.950835756 +0000 UTC m=+0.124671530 container create 797b6b78b3203301a3871595f5c4b0023215fc1940ca3229ae2b81a0ef703b88 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-21T10:35:02.214 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:35:02 smithi098 podman[328197]: 2024-09-21 10:35:02.212251522 +0000 UTC m=+0.386087263 container init 797b6b78b3203301a3871595f5c4b0023215fc1940ca3229ae2b81a0ef703b88 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-21T10:35:02.597 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:35:02 smithi098 podman[328197]: 2024-09-21 10:35:02.215754187 +0000 UTC m=+0.389589927 container start 797b6b78b3203301a3871595f5c4b0023215fc1940ca3229ae2b81a0ef703b88 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=main-62da54b, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-09-21T10:35:02.597 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:35:02 smithi098 podman[328197]: 2024-09-21 10:35:02.287044403 +0000 UTC m=+0.460880135 container attach 797b6b78b3203301a3871595f5c4b0023215fc1940ca3229ae2b81a0ef703b88 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-21T10:35:02.598 INFO:journalctl@ceph.osd.2.smithi098.stdout:Sep 21 10:35:02 smithi098 podman[328197]: 2024-09-21 10:35:02.405789294 +0000 UTC m=+0.579625038 container died 797b6b78b3203301a3871595f5c4b0023215fc1940ca3229ae2b81a0ef703b88 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-62da54b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:35:03.224 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.2.service' 2024-09-21T10:35:03.270 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:35:03.270 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-21T10:35:03.270 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-21T10:35:03.271 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.3 2024-09-21T10:35:03.579 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:35:03 smithi098 systemd[1]: Stopping Ceph osd.3 for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:35:03.847 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:35:03 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3[123636]: 2024-09-21T10:35:03.576+0000 7f487d8aa640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:35:03.847 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:35:03 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3[123636]: 2024-09-21T10:35:03.576+0000 7f487d8aa640 -1 osd.3 427 *** Got signal Terminated *** 2024-09-21T10:35:03.847 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:35:03 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3[123636]: 2024-09-21T10:35:03.576+0000 7f487d8aa640 -1 osd.3 427 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:35:05.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:04 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:04.744+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:05.768 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:05 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4[37445]: 2024-09-21T10:35:05.499+0000 7fcc16806640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.219772+0000 front 2024-09-21T10:34:41.219940+0000 (oldest deadline 2024-09-21T10:35:05.319788+0000) 2024-09-21T10:35:05.768 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:05 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:05.714+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:05.847 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:35:05 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3[123636]: 2024-09-21T10:35:05.460+0000 7f4875eb7640 -1 osd.3 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.863663+0000 front 2024-09-21T10:34:39.863691+0000 (oldest deadline 2024-09-21T10:35:05.163680+0000) 2024-09-21T10:35:06.138 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:05 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:05.766+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:06.806 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:06 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4[37445]: 2024-09-21T10:35:06.536+0000 7fcc16806640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.219772+0000 front 2024-09-21T10:34:41.219940+0000 (oldest deadline 2024-09-21T10:35:05.319788+0000) 2024-09-21T10:35:06.807 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:06 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:06.684+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:06.847 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:35:06 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3[123636]: 2024-09-21T10:35:06.415+0000 7f4875eb7640 -1 osd.3 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.863663+0000 front 2024-09-21T10:34:39.863691+0000 (oldest deadline 2024-09-21T10:35:05.163680+0000) 2024-09-21T10:35:07.138 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:06 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:06.805+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:07.523 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:07 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:07.208+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.528440+0000 front 2024-09-21T10:34:41.528650+0000 (oldest deadline 2024-09-21T10:35:06.828418+0000) 2024-09-21T10:35:07.748 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:35:07 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3[123636]: 2024-09-21T10:35:07.422+0000 7f4875eb7640 -1 osd.3 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.863663+0000 front 2024-09-21T10:34:39.863691+0000 (oldest deadline 2024-09-21T10:35:05.163680+0000) 2024-09-21T10:35:07.799 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:07 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:07.704+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:07.799 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:07 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4[37445]: 2024-09-21T10:35:07.521+0000 7fcc16806640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.219772+0000 front 2024-09-21T10:34:41.219940+0000 (oldest deadline 2024-09-21T10:35:05.319788+0000) 2024-09-21T10:35:08.138 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:07 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:07.796+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:08.534 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:08 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:08.190+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.528440+0000 front 2024-09-21T10:34:41.528650+0000 (oldest deadline 2024-09-21T10:35:06.828418+0000) 2024-09-21T10:35:08.767 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:35:08 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3[123636]: 2024-09-21T10:35:08.392+0000 7f4875eb7640 -1 osd.3 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.863663+0000 front 2024-09-21T10:34:39.863691+0000 (oldest deadline 2024-09-21T10:35:05.163680+0000) 2024-09-21T10:35:08.888 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:08 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4[37445]: 2024-09-21T10:35:08.532+0000 7fcc16806640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.219772+0000 front 2024-09-21T10:34:41.219940+0000 (oldest deadline 2024-09-21T10:35:05.319788+0000) 2024-09-21T10:35:08.889 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:08 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:08.773+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:08.889 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:08 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:08.723+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:09.347 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:35:09 smithi098 podman[328300]: 2024-09-21 10:35:09.065855519 +0000 UTC m=+5.611235002 container died 98d8757e4343a5166a83b992ad05a410f9a789e371b4490f417df2338060e6b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=main-62da54b, org.label-schema.build-date=20240731) 2024-09-21T10:35:09.532 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:09 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:09.170+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.528440+0000 front 2024-09-21T10:34:41.528650+0000 (oldest deadline 2024-09-21T10:35:06.828418+0000) 2024-09-21T10:35:09.847 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:35:09 smithi098 podman[328300]: 2024-09-21 10:35:09.544426281 +0000 UTC m=+6.089805768 container cleanup 98d8757e4343a5166a83b992ad05a410f9a789e371b4490f417df2338060e6b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=main-62da54b) 2024-09-21T10:35:09.847 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:35:09 smithi098 bash[328300]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3 2024-09-21T10:35:09.888 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:09 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4[37445]: 2024-09-21T10:35:09.530+0000 7fcc16806640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.219772+0000 front 2024-09-21T10:34:41.219940+0000 (oldest deadline 2024-09-21T10:35:05.319788+0000) 2024-09-21T10:35:09.889 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:09 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:09.768+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:09.889 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:09 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:09.754+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:10.206 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:35:09 smithi098 podman[328316]: 2024-09-21 10:35:09.845727568 +0000 UTC m=+0.769254436 container remove 98d8757e4343a5166a83b992ad05a410f9a789e371b4490f417df2338060e6b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, ceph=True) 2024-09-21T10:35:10.388 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:10.126+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.528440+0000 front 2024-09-21T10:34:41.528650+0000 (oldest deadline 2024-09-21T10:35:06.828418+0000) 2024-09-21T10:35:10.457 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:35:10 smithi098 podman[328415]: 2024-09-21 10:35:10.108348616 +0000 UTC m=+0.015800783 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:35:10.457 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:35:10 smithi098 podman[328415]: 2024-09-21 10:35:10.230182957 +0000 UTC m=+0.137635116 container create d4a5307734bfe341520b0dc6bd46e330018e3ab1b4b0c429d3f18f64b6f4a2ba (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-09-21T10:35:10.457 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:35:10 smithi098 podman[328415]: 2024-09-21 10:35:10.455467467 +0000 UTC m=+0.362919645 container init d4a5307734bfe341520b0dc6bd46e330018e3ab1b4b0c429d3f18f64b6f4a2ba (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-21T10:35:10.783 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4[37445]: 2024-09-21T10:35:10.523+0000 7fcc16806640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.219772+0000 front 2024-09-21T10:34:41.219940+0000 (oldest deadline 2024-09-21T10:35:05.319788+0000) 2024-09-21T10:35:10.847 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:35:10 smithi098 podman[328415]: 2024-09-21 10:35:10.458735515 +0000 UTC m=+0.366187674 container start d4a5307734bfe341520b0dc6bd46e330018e3ab1b4b0c429d3f18f64b6f4a2ba (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-62da54b, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:35:10.848 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:35:10 smithi098 podman[328415]: 2024-09-21 10:35:10.539064651 +0000 UTC m=+0.446516828 container attach d4a5307734bfe341520b0dc6bd46e330018e3ab1b4b0c429d3f18f64b6f4a2ba (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-62da54b) 2024-09-21T10:35:10.848 INFO:journalctl@ceph.osd.3.smithi098.stdout:Sep 21 10:35:10 smithi098 podman[328415]: 2024-09-21 10:35:10.647983815 +0000 UTC m=+0.555435974 container died d4a5307734bfe341520b0dc6bd46e330018e3ab1b4b0c429d3f18f64b6f4a2ba (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=main-62da54b, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:35:11.138 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:10.781+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:11.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:10 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:10.785+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:11.308 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.3.service' 2024-09-21T10:35:11.348 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:35:11.348 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-21T10:35:11.348 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-21T10:35:11.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.4 2024-09-21T10:35:11.402 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:11.142+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.528440+0000 front 2024-09-21T10:34:41.528650+0000 (oldest deadline 2024-09-21T10:35:06.828418+0000) 2024-09-21T10:35:11.672 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:11 smithi196 systemd[1]: Stopping Ceph osd.4 for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:35:11.672 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4[37445]: 2024-09-21T10:35:11.567+0000 7fcc16806640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.219772+0000 front 2024-09-21T10:34:41.219940+0000 (oldest deadline 2024-09-21T10:35:05.319788+0000) 2024-09-21T10:35:11.672 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:35:11.560Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.98:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.98:8765: connect: connection refused" 2024-09-21T10:35:11.673 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:35:11.562Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.98:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.98:8765: connect: connection refused" 2024-09-21T10:35:11.673 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:35:11.565Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.98:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.98:8765: connect: connection refused" 2024-09-21T10:35:11.673 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:35:11.565Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.98:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.98:8765: connect: connection refused" 2024-09-21T10:35:11.673 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:35:11.565Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.98:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.98:8765: connect: connection refused" 2024-09-21T10:35:11.673 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:35:11.568Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.98:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.98:8765: connect: connection refused" 2024-09-21T10:35:11.673 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:35:11.568Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=smb msg="Unable to refresh target groups" err="Get \"http://172.21.15.98:8765/sd/prometheus/sd-config?service=smb\": dial tcp 172.21.15.98:8765: connect: connection refused" 2024-09-21T10:35:12.138 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:11.786+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:12.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:11.790+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:12.139 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4[37445]: 2024-09-21T10:35:11.670+0000 7fcc1c1f2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:35:12.139 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4[37445]: 2024-09-21T10:35:11.670+0000 7fcc1c1f2640 -1 osd.4 427 *** Got signal Terminated *** 2024-09-21T10:35:12.140 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:11 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4[37445]: 2024-09-21T10:35:11.670+0000 7fcc1c1f2640 -1 osd.4 427 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:35:12.618 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:12 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:12.184+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.528440+0000 front 2024-09-21T10:34:41.528650+0000 (oldest deadline 2024-09-21T10:35:06.828418+0000) 2024-09-21T10:35:12.888 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:12 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4[37445]: 2024-09-21T10:35:12.615+0000 7fcc16806640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.219772+0000 front 2024-09-21T10:34:41.219940+0000 (oldest deadline 2024-09-21T10:35:05.319788+0000) 2024-09-21T10:35:12.889 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:12 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:12.763+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:12.889 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:12 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:12.764+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:12.889 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:12 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:12.764+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:49.772961+0000 front 2024-09-21T10:34:49.772885+0000 (oldest deadline 2024-09-21T10:35:12.672931+0000) 2024-09-21T10:35:13.589 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:13 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:13.200+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.528440+0000 front 2024-09-21T10:34:41.528650+0000 (oldest deadline 2024-09-21T10:35:06.828418+0000) 2024-09-21T10:35:13.889 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:13 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4[37445]: 2024-09-21T10:35:13.588+0000 7fcc16806640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.219772+0000 front 2024-09-21T10:34:41.219940+0000 (oldest deadline 2024-09-21T10:35:05.319788+0000) 2024-09-21T10:35:13.889 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:13 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4[37445]: 2024-09-21T10:35:13.588+0000 7fcc16806640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:48.820810+0000 front 2024-09-21T10:34:48.820798+0000 (oldest deadline 2024-09-21T10:35:12.920687+0000) 2024-09-21T10:35:13.889 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:13 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:13.748+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:13.890 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:13 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:13.727+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:13.890 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:13 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:13.727+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:49.772961+0000 front 2024-09-21T10:34:49.772885+0000 (oldest deadline 2024-09-21T10:35:12.672931+0000) 2024-09-21T10:35:14.585 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:14 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:14.210+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.528440+0000 front 2024-09-21T10:34:41.528650+0000 (oldest deadline 2024-09-21T10:35:06.828418+0000) 2024-09-21T10:35:14.888 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:14 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4[37445]: 2024-09-21T10:35:14.583+0000 7fcc16806640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.219772+0000 front 2024-09-21T10:34:41.219940+0000 (oldest deadline 2024-09-21T10:35:05.319788+0000) 2024-09-21T10:35:14.889 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:14 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4[37445]: 2024-09-21T10:35:14.583+0000 7fcc16806640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:48.820810+0000 front 2024-09-21T10:34:48.820798+0000 (oldest deadline 2024-09-21T10:35:12.920687+0000) 2024-09-21T10:35:14.889 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:14 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:14.784+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:14.889 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:14 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:14.784+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:14.889 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:14 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:14.696+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:14.889 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:14 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:14.696+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:49.772961+0000 front 2024-09-21T10:34:49.772885+0000 (oldest deadline 2024-09-21T10:35:12.672931+0000) 2024-09-21T10:35:15.587 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:15 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:15.198+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.528440+0000 front 2024-09-21T10:34:41.528650+0000 (oldest deadline 2024-09-21T10:35:06.828418+0000) 2024-09-21T10:35:15.888 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:15 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:15.825+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:15.889 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:15 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:15.825+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:15.890 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:15 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:15.719+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:15.890 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:15 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:15.719+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:49.772961+0000 front 2024-09-21T10:34:49.772885+0000 (oldest deadline 2024-09-21T10:35:12.672931+0000) 2024-09-21T10:35:15.890 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:15 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4[37445]: 2024-09-21T10:35:15.586+0000 7fcc16806640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.219772+0000 front 2024-09-21T10:34:41.219940+0000 (oldest deadline 2024-09-21T10:35:05.319788+0000) 2024-09-21T10:35:15.890 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:15 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4[37445]: 2024-09-21T10:35:15.586+0000 7fcc16806640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:48.820810+0000 front 2024-09-21T10:34:48.820798+0000 (oldest deadline 2024-09-21T10:35:12.920687+0000) 2024-09-21T10:35:16.632 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:16 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:16.214+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.528440+0000 front 2024-09-21T10:34:41.528650+0000 (oldest deadline 2024-09-21T10:35:06.828418+0000) 2024-09-21T10:35:16.888 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:16 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:16.728+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:16.889 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:16 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:16.728+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:49.772961+0000 front 2024-09-21T10:34:49.772885+0000 (oldest deadline 2024-09-21T10:35:12.672931+0000) 2024-09-21T10:35:16.889 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:16 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4[37445]: 2024-09-21T10:35:16.629+0000 7fcc16806640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.219772+0000 front 2024-09-21T10:34:41.219940+0000 (oldest deadline 2024-09-21T10:35:05.319788+0000) 2024-09-21T10:35:16.889 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:16 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4[37445]: 2024-09-21T10:35:16.629+0000 7fcc16806640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:48.820810+0000 front 2024-09-21T10:34:48.820798+0000 (oldest deadline 2024-09-21T10:35:12.920687+0000) 2024-09-21T10:35:16.889 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:16 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:16.796+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:16.890 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:16 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:16.796+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:17.388 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:17 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:17.225+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.528440+0000 front 2024-09-21T10:34:41.528650+0000 (oldest deadline 2024-09-21T10:35:06.828418+0000) 2024-09-21T10:35:17.389 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:17 smithi196 podman[226965]: 2024-09-21 10:35:17.109531513 +0000 UTC m=+5.564539597 container died 1693914f1a8bd3751710000c28aae3c9f1eab80bf54d6cb38de263477374d2b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-fbd1a6a, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-21T10:35:17.745 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:17 smithi196 podman[226965]: 2024-09-21 10:35:17.594858545 +0000 UTC m=+6.049866672 container cleanup 1693914f1a8bd3751710000c28aae3c9f1eab80bf54d6cb38de263477374d2b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=quincy-fbd1a6a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-21T10:35:17.746 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:17 smithi196 bash[226965]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4 2024-09-21T10:35:18.138 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:17 smithi196 podman[226982]: 2024-09-21 10:35:17.888110283 +0000 UTC m=+0.754119973 container remove 1693914f1a8bd3751710000c28aae3c9f1eab80bf54d6cb38de263477374d2b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=quincy-fbd1a6a, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-21T10:35:18.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:17 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:17.744+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:18.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:17 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:17.744+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:49.772961+0000 front 2024-09-21T10:34:49.772885+0000 (oldest deadline 2024-09-21T10:35:12.672931+0000) 2024-09-21T10:35:18.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:17 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:17.840+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:18.140 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:17 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:17.840+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:18.476 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:18.179+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.528440+0000 front 2024-09-21T10:34:41.528650+0000 (oldest deadline 2024-09-21T10:35:06.828418+0000) 2024-09-21T10:35:18.477 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:18 smithi196 podman[227070]: 2024-09-21 10:35:18.110023218 +0000 UTC m=+0.015740919 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T10:35:18.477 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:18 smithi196 podman[227070]: 2024-09-21 10:35:18.230925197 +0000 UTC m=+0.136642885 container create ae80859485229dd11386263fb323869c12fc94995b5a607b029dc9ab959dde06 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a, GIT_BRANCH=HEAD) 2024-09-21T10:35:18.477 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:18 smithi196 podman[227070]: 2024-09-21 10:35:18.475063759 +0000 UTC m=+0.380781457 container init ae80859485229dd11386263fb323869c12fc94995b5a607b029dc9ab959dde06 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-21T10:35:18.730 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:18 smithi196 podman[227070]: 2024-09-21 10:35:18.478523356 +0000 UTC m=+0.384241048 container start ae80859485229dd11386263fb323869c12fc94995b5a607b029dc9ab959dde06 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-21T10:35:18.730 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:18 smithi196 podman[227070]: 2024-09-21 10:35:18.58345539 +0000 UTC m=+0.489173081 container attach ae80859485229dd11386263fb323869c12fc94995b5a607b029dc9ab959dde06 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=quincy-fbd1a6a, ceph=True) 2024-09-21T10:35:18.730 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 10:35:18 smithi196 podman[227070]: 2024-09-21 10:35:18.705737738 +0000 UTC m=+0.611455434 container died ae80859485229dd11386263fb323869c12fc94995b5a607b029dc9ab959dde06 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-4-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=quincy-fbd1a6a, org.label-schema.build-date=20240731, ceph=True) 2024-09-21T10:35:19.005 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:18.727+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:19.005 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:18.727+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:49.772961+0000 front 2024-09-21T10:34:49.772885+0000 (oldest deadline 2024-09-21T10:35:12.672931+0000) 2024-09-21T10:35:19.005 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:18.815+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:19.005 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:18 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:18.815+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:19.305 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:19 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:19.169+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.528440+0000 front 2024-09-21T10:34:41.528650+0000 (oldest deadline 2024-09-21T10:35:06.828418+0000) 2024-09-21T10:35:19.306 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:19 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:19.169+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.829947+0000 front 2024-09-21T10:34:50.829858+0000 (oldest deadline 2024-09-21T10:35:18.430347+0000) 2024-09-21T10:35:19.325 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.4.service' 2024-09-21T10:35:19.365 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:35:19.365 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-21T10:35:19.365 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-21T10:35:19.365 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.5 2024-09-21T10:35:19.638 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:19 smithi196 systemd[1]: Stopping Ceph osd.5 for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:35:20.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:19 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:19.770+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:20.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:19 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:19.770+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:20.139 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:19 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:19.689+0000 7f2b39db0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:35:20.139 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:19 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:19.689+0000 7f2b39db0640 -1 osd.5 427 *** Got signal Terminated *** 2024-09-21T10:35:20.139 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:19 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:19.689+0000 7f2b39db0640 -1 osd.5 427 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:35:20.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:19 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:19.757+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:20.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:19 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:19.757+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:49.772961+0000 front 2024-09-21T10:34:49.772885+0000 (oldest deadline 2024-09-21T10:35:12.672931+0000) 2024-09-21T10:35:20.638 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:20.142+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.528440+0000 front 2024-09-21T10:34:41.528650+0000 (oldest deadline 2024-09-21T10:35:06.828418+0000) 2024-09-21T10:35:20.639 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:20.142+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.829947+0000 front 2024-09-21T10:34:50.829858+0000 (oldest deadline 2024-09-21T10:35:18.430347+0000) 2024-09-21T10:35:21.138 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:20.744+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:21.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:20.744+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:21.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:20.800+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:21.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:20 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:20.800+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:49.772961+0000 front 2024-09-21T10:34:49.772885+0000 (oldest deadline 2024-09-21T10:35:12.672931+0000) 2024-09-21T10:35:21.638 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:21 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:21.162+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.528440+0000 front 2024-09-21T10:34:41.528650+0000 (oldest deadline 2024-09-21T10:35:06.828418+0000) 2024-09-21T10:35:21.638 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:21 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:21.162+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.829947+0000 front 2024-09-21T10:34:50.829858+0000 (oldest deadline 2024-09-21T10:35:18.430347+0000) 2024-09-21T10:35:22.131 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:21 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:21.807+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:22.132 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:21 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:21.807+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:49.772961+0000 front 2024-09-21T10:34:49.772885+0000 (oldest deadline 2024-09-21T10:35:12.672931+0000) 2024-09-21T10:35:22.132 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:21 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:21.764+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:22.132 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:21 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:21.764+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:22.388 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:22 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:22.129+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.528440+0000 front 2024-09-21T10:34:41.528650+0000 (oldest deadline 2024-09-21T10:35:06.828418+0000) 2024-09-21T10:35:22.389 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:22 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:22.129+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.829947+0000 front 2024-09-21T10:34:50.829858+0000 (oldest deadline 2024-09-21T10:35:18.430347+0000) 2024-09-21T10:35:22.389 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:22 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:22.129+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:58.430904+0000 front 2024-09-21T10:34:58.430888+0000 (oldest deadline 2024-09-21T10:35:21.330660+0000) 2024-09-21T10:35:23.138 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:22 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:22.804+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:23.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:22 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:22.804+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:23.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:22 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:22.786+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:23.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:22 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:22.786+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:49.772961+0000 front 2024-09-21T10:34:49.772885+0000 (oldest deadline 2024-09-21T10:35:12.672931+0000) 2024-09-21T10:35:23.638 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:23 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:23.137+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.528440+0000 front 2024-09-21T10:34:41.528650+0000 (oldest deadline 2024-09-21T10:35:06.828418+0000) 2024-09-21T10:35:23.639 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:23 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:23.137+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.829947+0000 front 2024-09-21T10:34:50.829858+0000 (oldest deadline 2024-09-21T10:35:18.430347+0000) 2024-09-21T10:35:23.639 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:23 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:23.137+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:58.430904+0000 front 2024-09-21T10:34:58.430888+0000 (oldest deadline 2024-09-21T10:35:21.330660+0000) 2024-09-21T10:35:24.106 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:23 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:23.844+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:24.106 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:23 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:23.844+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:24.107 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:23 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:23.798+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:24.107 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:23 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:23.798+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:49.772961+0000 front 2024-09-21T10:34:49.772885+0000 (oldest deadline 2024-09-21T10:35:12.672931+0000) 2024-09-21T10:35:24.389 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:24 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:24.104+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.528440+0000 front 2024-09-21T10:34:41.528650+0000 (oldest deadline 2024-09-21T10:35:06.828418+0000) 2024-09-21T10:35:24.389 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:24 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:24.104+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.829947+0000 front 2024-09-21T10:34:50.829858+0000 (oldest deadline 2024-09-21T10:35:18.430347+0000) 2024-09-21T10:35:24.389 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:24 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5[43160]: 2024-09-21T10:35:24.104+0000 7f2b363c8640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:58.430904+0000 front 2024-09-21T10:34:58.430888+0000 (oldest deadline 2024-09-21T10:35:21.330660+0000) 2024-09-21T10:35:25.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:24 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:24.812+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:25.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:24 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:24.812+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:25.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:24 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:24.773+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:25.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:24 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:24.773+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:49.772961+0000 front 2024-09-21T10:34:49.772885+0000 (oldest deadline 2024-09-21T10:35:12.672931+0000) 2024-09-21T10:35:25.576 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:25 smithi196 podman[227174]: 2024-09-21 10:35:25.269399818 +0000 UTC m=+5.686268085 container died cb8e4afb2db73798d8286d2fddb4adea1a86804c67fadbd604e36f2c4c60dbf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2) 2024-09-21T10:35:25.838 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:25 smithi196 podman[227174]: 2024-09-21 10:35:25.65642326 +0000 UTC m=+6.073291526 container cleanup cb8e4afb2db73798d8286d2fddb4adea1a86804c67fadbd604e36f2c4c60dbf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-fbd1a6a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.build-date=20240731) 2024-09-21T10:35:25.839 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:25 smithi196 bash[227174]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5 2024-09-21T10:35:25.839 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:25 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:25.769+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:25.839 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:25 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:25.769+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:49.772961+0000 front 2024-09-21T10:34:49.772885+0000 (oldest deadline 2024-09-21T10:35:12.672931+0000) 2024-09-21T10:35:25.839 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:25 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:25.769+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:54.973746+0000 front 2024-09-21T10:34:54.973765+0000 (oldest deadline 2024-09-21T10:35:25.574307+0000) 2024-09-21T10:35:26.138 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:25 smithi196 podman[227189]: 2024-09-21 10:35:25.933037595 +0000 UTC m=+0.654893443 container remove cb8e4afb2db73798d8286d2fddb4adea1a86804c67fadbd604e36f2c4c60dbf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-fbd1a6a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-21T10:35:26.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:25 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:25.836+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:26.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:25 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:25.836+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:26.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:25 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:25.836+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:56.572454+0000 front 2024-09-21T10:34:56.572468+0000 (oldest deadline 2024-09-21T10:35:25.372700+0000) 2024-09-21T10:35:26.530 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:26 smithi196 podman[227280]: 2024-09-21 10:35:26.16073356 +0000 UTC m=+0.015653459 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T10:35:26.530 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:26 smithi196 podman[227280]: 2024-09-21 10:35:26.292261196 +0000 UTC m=+0.147181084 container create 09a251b7f918e72ee0595d11b5b3f9fd9ddc98bb84ecb61173d95be94607df5e (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-21T10:35:26.530 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:26 smithi196 podman[227280]: 2024-09-21 10:35:26.528223823 +0000 UTC m=+0.383143722 container init 09a251b7f918e72ee0595d11b5b3f9fd9ddc98bb84ecb61173d95be94607df5e (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:35:26.875 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:26 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:26.746+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:26.876 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:26 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:26.746+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:49.772961+0000 front 2024-09-21T10:34:49.772885+0000 (oldest deadline 2024-09-21T10:35:12.672931+0000) 2024-09-21T10:35:26.876 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:26 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:26.746+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:54.973746+0000 front 2024-09-21T10:34:54.973765+0000 (oldest deadline 2024-09-21T10:35:25.574307+0000) 2024-09-21T10:35:26.876 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:26 smithi196 podman[227280]: 2024-09-21 10:35:26.531700187 +0000 UTC m=+0.386620077 container start 09a251b7f918e72ee0595d11b5b3f9fd9ddc98bb84ecb61173d95be94607df5e (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5-deactivate, RELEASE=quincy-fbd1a6a, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-21T10:35:26.876 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:26 smithi196 podman[227280]: 2024-09-21 10:35:26.611684502 +0000 UTC m=+0.466604403 container attach 09a251b7f918e72ee0595d11b5b3f9fd9ddc98bb84ecb61173d95be94607df5e (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=quincy-fbd1a6a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-21T10:35:26.876 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 10:35:26 smithi196 podman[227280]: 2024-09-21 10:35:26.705054666 +0000 UTC m=+0.559974564 container died 09a251b7f918e72ee0595d11b5b3f9fd9ddc98bb84ecb61173d95be94607df5e (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, RELEASE=quincy-fbd1a6a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-21T10:35:27.138 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:26 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:26.873+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:27.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:26 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:26.873+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:27.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:26 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:26.873+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:56.572454+0000 front 2024-09-21T10:34:56.572468+0000 (oldest deadline 2024-09-21T10:35:25.372700+0000) 2024-09-21T10:35:27.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.5.service' 2024-09-21T10:35:27.380 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:35:27.381 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-21T10:35:27.381 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-21T10:35:27.381 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.6 2024-09-21T10:35:27.638 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:27 smithi196 systemd[1]: Stopping Ceph osd.6 for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:35:28.138 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:27.838+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:28.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:27.838+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:28.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:27.838+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:56.572454+0000 front 2024-09-21T10:34:56.572468+0000 (oldest deadline 2024-09-21T10:35:25.372700+0000) 2024-09-21T10:35:28.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:27.726+0000 7f18628a5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:35:28.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:27.726+0000 7f18628a5640 -1 osd.6 427 *** Got signal Terminated *** 2024-09-21T10:35:28.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:27.726+0000 7f18628a5640 -1 osd.6 427 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:35:28.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:27.774+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:28.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:27.774+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:49.772961+0000 front 2024-09-21T10:34:49.772885+0000 (oldest deadline 2024-09-21T10:35:12.672931+0000) 2024-09-21T10:35:28.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:27 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:27.774+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:54.973746+0000 front 2024-09-21T10:34:54.973765+0000 (oldest deadline 2024-09-21T10:35:25.574307+0000) 2024-09-21T10:35:29.138 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:28.790+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:29.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:28.790+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:49.772961+0000 front 2024-09-21T10:34:49.772885+0000 (oldest deadline 2024-09-21T10:35:12.672931+0000) 2024-09-21T10:35:29.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:28.790+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:54.973746+0000 front 2024-09-21T10:34:54.973765+0000 (oldest deadline 2024-09-21T10:35:25.574307+0000) 2024-09-21T10:35:29.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:28.855+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:29.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:28.855+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:29.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:28 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:28.855+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:56.572454+0000 front 2024-09-21T10:34:56.572468+0000 (oldest deadline 2024-09-21T10:35:25.372700+0000) 2024-09-21T10:35:30.138 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:29 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:29.835+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:30.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:29 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:29.835+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:49.772961+0000 front 2024-09-21T10:34:49.772885+0000 (oldest deadline 2024-09-21T10:35:12.672931+0000) 2024-09-21T10:35:30.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:29 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:29.835+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:54.973746+0000 front 2024-09-21T10:34:54.973765+0000 (oldest deadline 2024-09-21T10:35:25.574307+0000) 2024-09-21T10:35:30.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:29 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:29.870+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:30.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:29 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:29.870+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:30.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:29 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:29.870+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:56.572454+0000 front 2024-09-21T10:34:56.572468+0000 (oldest deadline 2024-09-21T10:35:25.372700+0000) 2024-09-21T10:35:31.138 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:30.852+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:31.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:30.852+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:49.772961+0000 front 2024-09-21T10:34:49.772885+0000 (oldest deadline 2024-09-21T10:35:12.672931+0000) 2024-09-21T10:35:31.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:30.852+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:54.973746+0000 front 2024-09-21T10:34:54.973765+0000 (oldest deadline 2024-09-21T10:35:25.574307+0000) 2024-09-21T10:35:31.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:30.902+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:31.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:30.902+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:31.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:30.902+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:56.572454+0000 front 2024-09-21T10:34:56.572468+0000 (oldest deadline 2024-09-21T10:35:25.372700+0000) 2024-09-21T10:35:31.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:30 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:30.902+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6830 osd.3 since back 2024-09-21T10:35:08.273744+0000 front 2024-09-21T10:35:08.273688+0000 (oldest deadline 2024-09-21T10:35:29.874225+0000) 2024-09-21T10:35:32.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:31 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:31.839+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:39.871363+0000 front 2024-09-21T10:34:39.871335+0000 (oldest deadline 2024-09-21T10:35:04.571194+0000) 2024-09-21T10:35:32.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:31 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:31.839+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:49.772961+0000 front 2024-09-21T10:34:49.772885+0000 (oldest deadline 2024-09-21T10:35:12.672931+0000) 2024-09-21T10:35:32.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:31 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:31.839+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:54.973746+0000 front 2024-09-21T10:34:54.973765+0000 (oldest deadline 2024-09-21T10:35:25.574307+0000) 2024-09-21T10:35:32.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:31 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6[48640]: 2024-09-21T10:35:31.839+0000 7f18606c0640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.98:6830 osd.3 since back 2024-09-21T10:35:05.574944+0000 front 2024-09-21T10:35:05.574922+0000 (oldest deadline 2024-09-21T10:35:30.874770+0000) 2024-09-21T10:35:32.140 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:31 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:31.912+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:32.140 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:31 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:31.912+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:32.140 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:31 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:31.912+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:56.572454+0000 front 2024-09-21T10:34:56.572468+0000 (oldest deadline 2024-09-21T10:35:25.372700+0000) 2024-09-21T10:35:32.141 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:31 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:31.912+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6830 osd.3 since back 2024-09-21T10:35:08.273744+0000 front 2024-09-21T10:35:08.273688+0000 (oldest deadline 2024-09-21T10:35:29.874225+0000) 2024-09-21T10:35:33.388 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:33 smithi196 podman[227381]: 2024-09-21 10:35:33.182159788 +0000 UTC m=+5.588385787 container died 8ba9647073b2e5e657a688cf24fe11e10522f6b34c219675308797a6975bc60c (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-21T10:35:33.389 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:32 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:32.957+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:33.389 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:32 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:32.957+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:33.389 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:32 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:32.957+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:56.572454+0000 front 2024-09-21T10:34:56.572468+0000 (oldest deadline 2024-09-21T10:35:25.372700+0000) 2024-09-21T10:35:33.389 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:32 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:32.957+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6830 osd.3 since back 2024-09-21T10:35:08.273744+0000 front 2024-09-21T10:35:08.273688+0000 (oldest deadline 2024-09-21T10:35:29.874225+0000) 2024-09-21T10:35:33.712 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:33 smithi196 podman[227381]: 2024-09-21 10:35:33.526034479 +0000 UTC m=+5.932260467 container cleanup 8ba9647073b2e5e657a688cf24fe11e10522f6b34c219675308797a6975bc60c (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-fbd1a6a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-21T10:35:33.712 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:33 smithi196 bash[227381]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6 2024-09-21T10:35:34.112 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:33 smithi196 podman[227399]: 2024-09-21 10:35:33.710725794 +0000 UTC m=+0.524912901 container remove 8ba9647073b2e5e657a688cf24fe11e10522f6b34c219675308797a6975bc60c (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, RELEASE=quincy-fbd1a6a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-21T10:35:34.113 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:33 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:33.921+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:34.113 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:33 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:33.921+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:34.113 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:33 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:33.921+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:56.572454+0000 front 2024-09-21T10:34:56.572468+0000 (oldest deadline 2024-09-21T10:35:25.372700+0000) 2024-09-21T10:35:34.113 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:33 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:33.921+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6830 osd.3 since back 2024-09-21T10:35:08.273744+0000 front 2024-09-21T10:35:08.273688+0000 (oldest deadline 2024-09-21T10:35:29.874225+0000) 2024-09-21T10:35:34.389 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:34 smithi196 podman[227487]: 2024-09-21 10:35:34.013776921 +0000 UTC m=+0.015706429 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T10:35:34.389 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:34 smithi196 podman[227487]: 2024-09-21 10:35:34.111603227 +0000 UTC m=+0.113532735 container create 6a9ce2ce59f29cd1c1bad5a2ac410773deb7e9d841fc077fbdf94992be7f8054 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=quincy-fbd1a6a, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-21T10:35:34.389 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:34 smithi196 podman[227487]: 2024-09-21 10:35:34.330613887 +0000 UTC m=+0.332543396 container init 6a9ce2ce59f29cd1c1bad5a2ac410773deb7e9d841fc077fbdf94992be7f8054 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=quincy-fbd1a6a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-21T10:35:34.389 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:34 smithi196 podman[227487]: 2024-09-21 10:35:34.333848751 +0000 UTC m=+0.335778259 container start 6a9ce2ce59f29cd1c1bad5a2ac410773deb7e9d841fc077fbdf94992be7f8054 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-21T10:35:34.802 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:34 smithi196 podman[227487]: 2024-09-21 10:35:34.420149254 +0000 UTC m=+0.422078779 container attach 6a9ce2ce59f29cd1c1bad5a2ac410773deb7e9d841fc077fbdf94992be7f8054 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=quincy-fbd1a6a, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-21T10:35:34.802 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 10:35:34 smithi196 podman[227487]: 2024-09-21 10:35:34.511014032 +0000 UTC m=+0.512943541 container died 6a9ce2ce59f29cd1c1bad5a2ac410773deb7e9d841fc077fbdf94992be7f8054 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-21T10:35:35.086 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:34 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:34.958+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:35.086 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:34 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:34.958+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:35.086 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:34 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:34.958+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:56.572454+0000 front 2024-09-21T10:34:56.572468+0000 (oldest deadline 2024-09-21T10:35:25.372700+0000) 2024-09-21T10:35:35.086 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:34 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:34.958+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6830 osd.3 since back 2024-09-21T10:35:08.273744+0000 front 2024-09-21T10:35:08.273688+0000 (oldest deadline 2024-09-21T10:35:29.874225+0000) 2024-09-21T10:35:35.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.6.service' 2024-09-21T10:35:35.143 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:35:35.143 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-21T10:35:35.143 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-21T10:35:35.143 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.7 2024-09-21T10:35:35.388 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:35 smithi196 systemd[1]: Stopping Ceph osd.7 for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:35:35.888 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:35.487+0000 7efe64fe8640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:35:35.888 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:35.487+0000 7efe64fe8640 -1 osd.7 427 *** Got signal Terminated *** 2024-09-21T10:35:35.889 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:35.487+0000 7efe64fe8640 -1 osd.7 427 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:35:36.388 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:35.971+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:36.389 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:35.971+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:36.389 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:35.971+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:56.572454+0000 front 2024-09-21T10:34:56.572468+0000 (oldest deadline 2024-09-21T10:35:25.372700+0000) 2024-09-21T10:35:36.389 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:35 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:35.971+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6830 osd.3 since back 2024-09-21T10:35:08.273744+0000 front 2024-09-21T10:35:08.273688+0000 (oldest deadline 2024-09-21T10:35:29.874225+0000) 2024-09-21T10:35:37.388 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:36 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:36.930+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:37.389 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:36 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:36.930+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:37.389 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:36 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:36.930+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:56.572454+0000 front 2024-09-21T10:34:56.572468+0000 (oldest deadline 2024-09-21T10:35:25.372700+0000) 2024-09-21T10:35:37.389 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:36 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:36.930+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6830 osd.3 since back 2024-09-21T10:35:08.273744+0000 front 2024-09-21T10:35:08.273688+0000 (oldest deadline 2024-09-21T10:35:29.874225+0000) 2024-09-21T10:35:38.389 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:37 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:37.908+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:38.389 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:37 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:37.908+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:38.389 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:37 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:37.908+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:56.572454+0000 front 2024-09-21T10:34:56.572468+0000 (oldest deadline 2024-09-21T10:35:25.372700+0000) 2024-09-21T10:35:38.389 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:37 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:37.908+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6830 osd.3 since back 2024-09-21T10:35:08.273744+0000 front 2024-09-21T10:35:08.273688+0000 (oldest deadline 2024-09-21T10:35:29.874225+0000) 2024-09-21T10:35:39.138 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:38 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:38.860+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:39.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:38 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:38.860+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:39.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:38 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:38.860+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:56.572454+0000 front 2024-09-21T10:34:56.572468+0000 (oldest deadline 2024-09-21T10:35:25.372700+0000) 2024-09-21T10:35:39.139 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:38 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:38.860+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6830 osd.3 since back 2024-09-21T10:35:08.273744+0000 front 2024-09-21T10:35:08.273688+0000 (oldest deadline 2024-09-21T10:35:29.874225+0000) 2024-09-21T10:35:40.302 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:39 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:39.899+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-09-21T10:34:41.370673+0000 front 2024-09-21T10:34:41.370589+0000 (oldest deadline 2024-09-21T10:35:04.870693+0000) 2024-09-21T10:35:40.302 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:39 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:39.899+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-09-21T10:34:50.171678+0000 front 2024-09-21T10:34:50.171621+0000 (oldest deadline 2024-09-21T10:35:14.271540+0000) 2024-09-21T10:35:40.302 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:39 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:39.899+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-09-21T10:34:56.572454+0000 front 2024-09-21T10:34:56.572468+0000 (oldest deadline 2024-09-21T10:35:25.372700+0000) 2024-09-21T10:35:40.302 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:39 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:39.899+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.98:6830 osd.3 since back 2024-09-21T10:35:08.273744+0000 front 2024-09-21T10:35:08.273688+0000 (oldest deadline 2024-09-21T10:35:29.874225+0000) 2024-09-21T10:35:40.302 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:39 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7[54652]: 2024-09-21T10:35:39.899+0000 7efe62e03640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-21T10:35:14.475358+0000 front 2024-09-21T10:35:14.475401+0000 (oldest deadline 2024-09-21T10:35:39.775316+0000) 2024-09-21T10:35:41.331 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:41 smithi196 podman[227587]: 2024-09-21 10:35:41.062965138 +0000 UTC m=+5.698009926 container died 540c98a3c43623ab8665e4fe1854be3f09118dc827f193658550c920bcca6f3a (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=quincy-fbd1a6a, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-21T10:35:41.638 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 10:35:41 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a[73474]: logger=cleanup t=2024-09-21T10:35:41.367664564Z level=info msg="Completed cleanup jobs" duration=13.381649ms 2024-09-21T10:35:41.639 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:41 smithi196 podman[227587]: 2024-09-21 10:35:41.428935706 +0000 UTC m=+6.063980490 container cleanup 540c98a3c43623ab8665e4fe1854be3f09118dc827f193658550c920bcca6f3a (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-21T10:35:41.639 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:41 smithi196 bash[227587]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7 2024-09-21T10:35:42.087 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:41 smithi196 podman[227613]: 2024-09-21 10:35:41.73000375 +0000 UTC m=+0.663828670 container remove 540c98a3c43623ab8665e4fe1854be3f09118dc827f193658550c920bcca6f3a (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7, RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-09-21T10:35:42.389 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:42 smithi196 podman[227704]: 2024-09-21 10:35:41.988166273 +0000 UTC m=+0.015620847 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T10:35:42.389 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:42 smithi196 podman[227704]: 2024-09-21 10:35:42.105952548 +0000 UTC m=+0.133407161 container create 929b8bcaa26f06205bf6010ab029f294e2fb2f0a78cc59f9c2ed289ce4d11d13 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=quincy-fbd1a6a, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-21T10:35:42.389 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:42 smithi196 podman[227704]: 2024-09-21 10:35:42.325375813 +0000 UTC m=+0.352830393 container init 929b8bcaa26f06205bf6010ab029f294e2fb2f0a78cc59f9c2ed289ce4d11d13 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=quincy-fbd1a6a, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True) 2024-09-21T10:35:42.389 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:42 smithi196 podman[227704]: 2024-09-21 10:35:42.328737267 +0000 UTC m=+0.356191846 container start 929b8bcaa26f06205bf6010ab029f294e2fb2f0a78cc59f9c2ed289ce4d11d13 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-21T10:35:42.788 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:42 smithi196 podman[227704]: 2024-09-21 10:35:42.400127767 +0000 UTC m=+0.427582339 container attach 929b8bcaa26f06205bf6010ab029f294e2fb2f0a78cc59f9c2ed289ce4d11d13 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=quincy-fbd1a6a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-21T10:35:42.789 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 10:35:42 smithi196 podman[227704]: 2024-09-21 10:35:42.505456744 +0000 UTC m=+0.532911316 container died 929b8bcaa26f06205bf6010ab029f294e2fb2f0a78cc59f9c2ed289ce4d11d13 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-osd-7-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, RELEASE=quincy-fbd1a6a) 2024-09-21T10:35:43.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@osd.7.service' 2024-09-21T10:35:43.140 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:35:43.140 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-21T10:35:43.141 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-21T10:35:43.141 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-26a15512-77fe-11ef-baf6-efdc52797490@prometheus.a 2024-09-21T10:35:43.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:43 smithi196 systemd[1]: Stopping Ceph prometheus.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:35:43.775 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:43 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:35:43.490Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T10:35:43.775 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:43 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:35:43.490Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-21T10:35:43.775 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:43 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:35:43.490Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-21T10:35:43.775 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:43 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:35:43.490Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-21T10:35:43.775 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:43 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:35:43.490Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T10:35:43.775 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:43 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:35:43.490Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-21T10:35:43.776 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:43 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:35:43.490Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T10:35:43.776 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:43 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:35:43.490Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-21T10:35:43.776 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:43 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:35:43.490Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-21T10:35:43.776 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:43 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:35:43.492Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T10:35:43.776 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:43 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:35:43.492Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-21T10:35:43.776 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:43 smithi196 ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a[75783]: ts=2024-09-21T10:35:43.492Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-21T10:35:43.776 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:43 smithi196 podman[227806]: 2024-09-21 10:35:43.50044712 +0000 UTC m=+0.143529297 container died b30b25036dc770698c359850128bd31911b0a40808e1921ae30d90e2ee267d70 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T10:35:44.138 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:43 smithi196 podman[227806]: 2024-09-21 10:35:43.891359953 +0000 UTC m=+0.534442133 container cleanup b30b25036dc770698c359850128bd31911b0a40808e1921ae30d90e2ee267d70 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T10:35:44.139 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 10:35:43 smithi196 bash[227806]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T10:35:44.325 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-26a15512-77fe-11ef-baf6-efdc52797490@prometheus.a.service' 2024-09-21T10:35:44.358 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:35:44.358 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-21T10:35:44.358 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 26a15512-77fe-11ef-baf6-efdc52797490 --force --keep-logs 2024-09-21T10:35:46.097 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 10:35:45 smithi098 systemd[1]: Stopping Ceph alertmanager.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:35:46.577 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 10:35:46 smithi098 ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a[86918]: ts=2024-09-21T10:35:46.217Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-21T10:35:46.577 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 10:35:46 smithi098 podman[328818]: 2024-09-21 10:35:46.268001019 +0000 UTC m=+0.154585279 container died 72b57f9247443847d472287276d478b2da08bc1bf69a8c0d280cf384956c539e (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T10:35:46.847 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 10:35:46 smithi098 podman[328818]: 2024-09-21 10:35:46.742430925 +0000 UTC m=+0.629015166 container cleanup 72b57f9247443847d472287276d478b2da08bc1bf69a8c0d280cf384956c539e (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T10:35:46.847 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 10:35:46 smithi098 bash[328818]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a 2024-09-21T10:35:47.347 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 10:35:47 smithi098 podman[328831]: 2024-09-21 10:35:47.010423542 +0000 UTC m=+0.746936755 container remove 72b57f9247443847d472287276d478b2da08bc1bf69a8c0d280cf384956c539e (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T10:35:47.347 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 10:35:47 smithi098 podman[328831]: 2024-09-21 10:35:47.086094866 +0000 UTC m=+0.822608079 volume remove 9416b1407029b682e0f8d9e2952c1ded7b0e30de155db3f7c555a050715d560b 2024-09-21T10:35:47.347 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 10:35:47 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-21T10:35:47.347 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 10:35:47 smithi098 systemd[1]: Stopped Ceph alertmanager.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:35:47.348 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Sep 21 10:35:47 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@alertmanager.a.service: Consumed 2.709s CPU time. 2024-09-21T10:35:47.847 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 10:35:47 smithi098 systemd[1]: Stopping Ceph node-exporter.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:35:47.848 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 10:35:47 smithi098 podman[328939]: 2024-09-21 10:35:47.641449532 +0000 UTC m=+0.109827983 container died d587eb6669b7b053736a42d8ce28947df2b55e96c9c45bb1d3c1eecf9d6829c9 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T10:35:48.317 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 10:35:47 smithi098 podman[328939]: 2024-09-21 10:35:47.972546027 +0000 UTC m=+0.440924478 container cleanup d587eb6669b7b053736a42d8ce28947df2b55e96c9c45bb1d3c1eecf9d6829c9 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T10:35:48.317 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 10:35:47 smithi098 bash[328939]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a 2024-09-21T10:35:48.597 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 10:35:48 smithi098 podman[328952]: 2024-09-21 10:35:48.315615745 +0000 UTC m=+0.673891875 container remove d587eb6669b7b053736a42d8ce28947df2b55e96c9c45bb1d3c1eecf9d6829c9 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T10:35:48.597 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 10:35:48 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-21T10:35:48.597 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 10:35:48 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-21T10:35:48.598 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 10:35:48 smithi098 systemd[1]: Stopped Ceph node-exporter.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:35:48.598 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Sep 21 10:35:48 smithi098 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@node-exporter.a.service: Consumed 6.880s CPU time. 2024-09-21T10:35:56.004 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 26a15512-77fe-11ef-baf6-efdc52797490 --force --keep-logs 2024-09-21T10:35:57.815 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 10:35:57 smithi196 systemd[1]: Stopping Ceph grafana.a for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:35:58.096 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 10:35:57 smithi196 podman[228209]: 2024-09-21 10:35:57.8127805 +0000 UTC m=+0.119179734 container died 7724661af1a8c1859052fa5a9e7c770b3302a3e3e0403f990a6d8a09a43d4f4d (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a, distribution-scope=public, build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=239.1651231664, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., maintainer=Paul Cuzner , com.redhat.component=ubi8-container, io.openshift.expose-services=, version=8.5) 2024-09-21T10:35:58.388 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 10:35:58 smithi196 podman[228209]: 2024-09-21 10:35:58.168643538 +0000 UTC m=+0.475042759 container cleanup 7724661af1a8c1859052fa5a9e7c770b3302a3e3e0403f990a6d8a09a43d4f4d (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a, distribution-scope=public, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, vcs-type=git, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, version=8.5, vendor=Red Hat, Inc., release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-09-21T10:35:58.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 10:35:58 smithi196 bash[228209]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a 2024-09-21T10:35:58.764 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 10:35:58 smithi196 podman[228222]: 2024-09-21 10:35:58.453141687 +0000 UTC m=+0.641773259 container remove 7724661af1a8c1859052fa5a9e7c770b3302a3e3e0403f990a6d8a09a43d4f4d (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-grafana-a, maintainer=Paul Cuzner , release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, com.redhat.component=ubi8-container, io.buildah.version=1.32.0, vcs-type=git, io.openshift.expose-services=, version=8.5, distribution-scope=public, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com) 2024-09-21T10:35:58.764 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 10:35:58 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-21T10:35:58.765 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 10:35:58 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@grafana.a.service: Failed with result 'exit-code'. 2024-09-21T10:35:58.765 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 10:35:58 smithi196 systemd[1]: Stopped Ceph grafana.a for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:35:58.765 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 10:35:58 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@grafana.a.service: Consumed 25.862s CPU time. 2024-09-21T10:35:59.138 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 10:35:58 smithi196 systemd[1]: Stopping Ceph node-exporter.b for 26a15512-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:35:59.139 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 10:35:59 smithi196 podman[228329]: 2024-09-21 10:35:59.008439254 +0000 UTC m=+0.108215843 container died 7a8564adbc711b7285fb6f712a3c6e0f619405530f16ffc330a0e95546238461 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T10:35:59.516 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 10:35:59 smithi196 podman[228329]: 2024-09-21 10:35:59.322566211 +0000 UTC m=+0.422342841 container cleanup 7a8564adbc711b7285fb6f712a3c6e0f619405530f16ffc330a0e95546238461 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T10:35:59.517 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 10:35:59 smithi196 bash[228329]: ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b 2024-09-21T10:35:59.888 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 10:35:59 smithi196 podman[228341]: 2024-09-21 10:35:59.51542872 +0000 UTC m=+0.510575916 container remove 7a8564adbc711b7285fb6f712a3c6e0f619405530f16ffc330a0e95546238461 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-26a15512-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T10:35:59.888 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 10:35:59 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-21T10:35:59.889 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 10:35:59 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-21T10:35:59.889 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 10:35:59 smithi196 systemd[1]: Stopped Ceph node-exporter.b for 26a15512-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:35:59.889 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 21 10:35:59 smithi196 systemd[1]: ceph-26a15512-77fe-11ef-baf6-efdc52797490@node-exporter.b.service: Consumed 6.465s CPU time. 2024-09-21T10:36:03.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T10:36:03.299 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T10:36:03.327 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-21T10:36:03.329 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/crash to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913661/remote/smithi098/crash 2024-09-21T10:36:03.330 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/crash -- . 2024-09-21T10:36:03.383 INFO:teuthology.orchestra.run.smithi098.stderr:tar: /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T10:36:03.384 INFO:teuthology.orchestra.run.smithi098.stderr:tar: Error is not recoverable: exiting now 2024-09-21T10:36:03.386 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/crash to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913661/remote/smithi196/crash 2024-09-21T10:36:03.387 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/crash -- . 2024-09-21T10:36:03.424 INFO:teuthology.orchestra.run.smithi196.stderr:tar: /var/lib/ceph/26a15512-77fe-11ef-baf6-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T10:36:03.424 INFO:teuthology.orchestra.run.smithi196.stderr:tar: Error is not recoverable: exiting now 2024-09-21T10:36:03.425 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-21T10:36:03.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v OSD_DOWN | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v MON_DOWN | egrep -v 'out of quorum' | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v OSDMAP_FLAGS | egrep -v PG_AVAILABILITY | head -n 1 2024-09-21T10:36:03.503 INFO:teuthology.orchestra.run.smithi098.stdout:2024-09-21T10:20:00.006249+0000 mon.a (mon.0) 1324 : cluster [WRN] pg 8.0 is stuck undersized for 78s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,5] 2024-09-21T10:36:03.504 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-09-21T10:36:03.504 INFO:tasks.cephadm:Compressing logs... 2024-09-21T10:36:03.504 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T10:36:03.507 DEBUG:teuthology.orchestra.run.smithi196:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T10:36:03.542 INFO:teuthology.orchestra.run.smithi196.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-21T10:36:03.543 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.7.log 2024-09-21T10:36:03.546 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-volume.log 2024-09-21T10:36:03.546 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.4.log 2024-09-21T10:36:03.546 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph.audit.log 2024-09-21T10:36:03.546 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph.cephadm.log 2024-09-21T10:36:03.546 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.6.log 2024-09-21T10:36:03.546 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.5.log 2024-09-21T10:36:03.546 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph.log 2024-09-21T10:36:03.546 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-mgr.x.log 2024-09-21T10:36:03.547 INFO:teuthology.orchestra.run.smithi196.stderr:gzip/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph.log: -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-mds.foo.smithi196.ugblah.log 2024-09-21T10:36:03.547 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-mon.b.log 2024-09-21T10:36:03.547 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-mds.foo.smithi196.ugblah.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T10:36:03.547 INFO:teuthology.orchestra.run.smithi196.stderr: 62.6% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-mds.foo.smithi196.ugblah.log.gz 2024-09-21T10:36:03.563 INFO:teuthology.orchestra.run.smithi098.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.0.log 2024-09-21T10:36:03.563 INFO:teuthology.orchestra.run.smithi098.stderr:: No such file or directory 2024-09-21T10:36:03.563 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.1.log 2024-09-21T10:36:03.563 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-volume.log 2024-09-21T10:36:03.564 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph.audit.log 2024-09-21T10:36:03.566 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-mds.foo.smithi098.stcwsu.log 2024-09-21T10:36:03.566 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-mon.c.log 2024-09-21T10:36:03.567 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-mds.foo.smithi098.stcwsu.log: gzip -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph.cephadm.log 2024-09-21T10:36:03.567 INFO:teuthology.orchestra.run.smithi098.stderr: 82.0% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-mds.foo.smithi098.stcwsu.log.gz 2024-09-21T10:36:03.567 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-mon.a.log 2024-09-21T10:36:03.567 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.2.log 2024-09-21T10:36:03.567 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-mgr.y.log 2024-09-21T10:36:03.567 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph.log 2024-09-21T10:36:03.567 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.3.log 2024-09-21T10:36:03.567 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T10:36:03.574 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.3.log: /var/log/ceph/cephadm.log: 92.7% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph.log.gz 2024-09-21T10:36:03.837 INFO:teuthology.orchestra.run.smithi098.stderr: 91.1% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-volume.log.gz 2024-09-21T10:36:03.900 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-mon.b.log: /var/log/ceph/cephadm.log: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T10:36:03.904 INFO:teuthology.orchestra.run.smithi196.stderr: 84.2% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph.cephadm.log.gz 2024-09-21T10:36:03.994 INFO:teuthology.orchestra.run.smithi196.stderr: 91.3% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-volume.log.gz 2024-09-21T10:36:04.054 INFO:teuthology.orchestra.run.smithi098.stderr: 91.0% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph.cephadm.log.gz 2024-09-21T10:36:04.234 INFO:teuthology.orchestra.run.smithi098.stderr: 93.8% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph.audit.log.gz 2024-09-21T10:36:04.243 INFO:teuthology.orchestra.run.smithi098.stderr: 95.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T10:36:04.244 INFO:teuthology.orchestra.run.smithi196.stderr: 89.8% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph.audit.log.gz 2024-09-21T10:36:04.256 INFO:teuthology.orchestra.run.smithi196.stderr: 87.0% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph.log.gz 2024-09-21T10:36:04.479 INFO:teuthology.orchestra.run.smithi196.stderr: 90.1% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-mgr.x.log.gz 2024-09-21T10:36:05.881 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-mgr.y.log.gz 2024-09-21T10:36:12.709 INFO:teuthology.orchestra.run.smithi196.stderr: 92.3% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-mon.b.log.gz 2024-09-21T10:36:14.910 INFO:teuthology.orchestra.run.smithi098.stderr: 91.4% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-mon.c.log.gz 2024-09-21T10:36:52.341 INFO:teuthology.orchestra.run.smithi098.stderr: 90.2% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-mon.a.log.gz 2024-09-21T10:38:08.984 INFO:teuthology.orchestra.run.smithi196.stderr: 93.0% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.7.log.gz 2024-09-21T10:38:41.629 INFO:teuthology.orchestra.run.smithi196.stderr: 93.6% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.4.log.gz 2024-09-21T10:39:23.134 INFO:teuthology.orchestra.run.smithi098.stderr: 93.2% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.3.log.gz 2024-09-21T10:39:45.157 INFO:teuthology.orchestra.run.smithi196.stderr: 94.0% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.5.log.gz 2024-09-21T10:40:19.258 INFO:teuthology.orchestra.run.smithi196.stderr: 93.5% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.6.log.gz 2024-09-21T10:40:19.260 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-09-21T10:40:19.260 INFO:teuthology.orchestra.run.smithi196.stderr:real 4m15.738s 2024-09-21T10:40:19.260 INFO:teuthology.orchestra.run.smithi196.stderr:user 4m23.943s 2024-09-21T10:40:19.260 INFO:teuthology.orchestra.run.smithi196.stderr:sys 0m11.620s 2024-09-21T10:55:55.076 INFO:teuthology.orchestra.run.smithi098.stderr: 93.1% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.2.log.gz 2024-09-21T10:57:18.191 INFO:teuthology.orchestra.run.smithi098.stderr: 92.7% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.1.log.gz 2024-09-21T10:58:03.200 INFO:teuthology.orchestra.run.smithi098.stderr: 92.4% -- replaced with /var/log/ceph/26a15512-77fe-11ef-baf6-efdc52797490/ceph-osd.0.log.gz 2024-09-21T10:58:03.203 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-21T10:58:03.203 INFO:teuthology.orchestra.run.smithi098.stderr:real 21m59.683s 2024-09-21T10:58:03.203 INFO:teuthology.orchestra.run.smithi098.stderr:user 8m34.401s 2024-09-21T10:58:03.203 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m25.062s 2024-09-21T10:58:03.203 INFO:tasks.cephadm:Archiving logs... 2024-09-21T10:58:03.204 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913661/remote/smithi098/log 2024-09-21T10:58:03.205 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T10:59:23.294 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913661/remote/smithi196/log 2024-09-21T10:59:23.295 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T10:59:59.765 INFO:tasks.cephadm:Removing cluster... 2024-09-21T10:59:59.765 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 26a15512-77fe-11ef-baf6-efdc52797490 --force 2024-09-21T11:00:00.836 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 26a15512-77fe-11ef-baf6-efdc52797490 --force 2024-09-21T11:00:01.690 INFO:tasks.cephadm:Removing cephadm ... 2024-09-21T11:00:01.690 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T11:00:01.710 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T11:00:01.726 INFO:tasks.cephadm:Teardown complete 2024-09-21T11:00:01.726 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-21T11:00:01.742 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-21T11:00:01.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T11:00:01.754 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T11:00:01.843 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T11:00:01.852 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T11:00:01.852 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T11:00:01.855 DEBUG:teuthology.orchestra.run.smithi196:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T11:00:01.869 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-09-21T11:00:01.870 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-09-21T11:00:01.884 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T11:00:01.884 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-09-21T11:00:01.884 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv01.front.sepia.ceph.com 3 6 377 41 -212us[ -213us] +/- 56ms 2024-09-21T11:00:01.884 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 35 +221us[ +221us] +/- 49ms 2024-09-21T11:00:01.884 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 +26us[ +25us] +/- 67ms 2024-09-21T11:00:01.885 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T11:00:01.933 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T11:00:01.934 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-09-21T11:00:01.934 INFO:teuthology.orchestra.run.smithi196.stdout:^* hv01.front.sepia.ceph.com 3 6 377 41 -224us[ -221us] +/- 56ms 2024-09-21T11:00:01.934 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 42 +209us[ +212us] +/- 49ms 2024-09-21T11:00:01.934 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 +6936ns[+9901ns] +/- 67ms 2024-09-21T11:00:01.934 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T11:00:01.935 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T11:00:01.944 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T11:00:01.945 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T11:00:01.970 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T11:00:02.121 DEBUG:teuthology.orchestra.run.smithi196:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T11:00:02.206 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-21T11:00:02.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:02.237 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-21T11:00:02.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:02.264 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T11:00:02.274 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T11:00:02.299 INFO:teuthology.task.internal:Duration was 4860.362373 seconds 2024-09-21T11:00:02.299 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T11:00:02.312 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T11:00:02.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T11:00:02.314 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T11:00:02.369 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T11:00:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T11:00:02.700 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T11:00:02.700 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-09-21T11:00:02.701 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T11:00:02.734 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi196.front.sepia.ceph.com 2024-09-21T11:00:02.734 DEBUG:teuthology.orchestra.run.smithi196:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T11:00:02.761 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T11:00:02.762 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T11:00:02.776 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T11:00:02.832 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T11:00:02.832 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T11:00:02.851 DEBUG:teuthology.orchestra.run.smithi196:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T11:00:04.571 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T11:00:04.581 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T11:00:04.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T11:00:04.617 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T11:00:04.647 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T11:00:04.657 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T11:00:04.660 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T11:00:04.691 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-09-21T11:00:04.719 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = core 2024-09-21T11:00:04.746 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T11:00:04.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:04.764 DEBUG:teuthology.orchestra.run.smithi196:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T11:00:04.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:04.801 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T11:00:04.811 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T11:00:04.812 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913661/remote/smithi098 2024-09-21T11:00:04.812 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T11:00:04.886 DEBUG:teuthology.misc:Transferring archived files from smithi196:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913661/remote/smithi196 2024-09-21T11:00:04.886 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T11:00:04.930 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T11:00:04.930 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T11:00:04.932 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T11:00:04.978 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T11:00:04.989 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T11:00:04.989 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T11:00:05.019 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T11:00:05.019 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T11:00:05.022 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T11:00:05.037 INFO:teuthology.orchestra.run.smithi196.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 11:00 /home/ubuntu/cephtest 2024-09-21T11:00:05.061 INFO:teuthology.orchestra.run.smithi098.stdout: 265336 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 21 11:00 /home/ubuntu/cephtest 2024-09-21T11:00:05.061 INFO:teuthology.orchestra.run.smithi098.stdout: 417162 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 21 10:31 /home/ubuntu/cephtest/mnt.0 2024-09-21T11:00:05.061 INFO:teuthology.orchestra.run.smithi098.stdout: 417163 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 21 10:31 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T11:00:05.061 INFO:teuthology.orchestra.run.smithi098.stdout: 660391 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 10:31 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-21T11:00:05.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:00:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-21T11:00:05.063 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-21T11:00:05.068 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T11:00:05.078 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T11:00:05.118 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi098 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-21T11:00:05.121 INFO:teuthology.run:Summary data: description: upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} duration: 4860.362372636795 failure_reason: 'Command failed (workunit test cls/test_cls_rbd.sh) on smithi098 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=3bf7d22f47f24f0db0d4c4d748368a11 status: fail success: false 2024-09-21T11:00:05.122 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T11:00:05.212 INFO:teuthology.run:FAIL